Fragmentation of CTI: The Deck is Stacked Against the Defenders

Okay, so we are back. Um, the next passwords con track is ready to start. So, we'd like to welcome everybody to this track. Uh, a reminder about our sponsors, uh, Diamond sponsors, Prisma Cloud and Vant, and our gold sponsors, Drop Zone AI and Simgrip, and them along with all the other sponsors and volunteers. Again, thanking for uh all the uh sponsorship and the work and the effort that goes into producing this conference um and passwords con being possible as well. Just a reminder about cell phones, please put them on silent so that we can uh pay full attention to what will be presented this afternoon. And then also uh when it comes to photos, so the the B

size Las Vegas photo policies, no photographs of any presenters or uh uh content or attendees unless the presenters give permission. So these presenters don't mind if you take photos of them or their content. So you can, but please no photos of any of the attendees uh in this room or anywhere else. And there's no need to record. It is being recorded for us already. So, hazard analysis of military AI systems using SCPA sec a systems theoretic approach to secure and assured autonomy by Dr. Josh Hargus and Chris Ward. They're now up. Let's see what it's about. >> Thank you. Thanks everyone. Um, so my name is Josh Hargus. Uh, so myself and Chris Ward run

a small consultancy called Fire Mountain Labs. uh we do AI and AI security consulting. Uh something that we've been working on is trying to to uh kind of adapt some of the frameworks that are out there for other purposes to use for AI security and AI safety. Uh my background, we just started this back in April. Um I was the chief of AI security at Cranium and AI security company. Uh before that MITER, before that Navy research, been doing AI for almost two decades. I'll let Chris do it. >> Hey, Chris Ward. Welcome. Um I'm a Navy veteran. That's where I got my start in security. I was like a radar guy, but you can't be in the military and not

learn some security. Uh, so got out of the Navy, got a degree in electrical engineering. That's where I started thinking about like processes and and system analysis. Uh, met up with Josh at a lab called Spay War, which is now defunct and called Naval Information Warfare Center. Uh, but that's where we started looking into AI security and how to apply this kind of stuff to AI systems. >> Cool. All right. So, if you haven't noticed yet, this is has nothing to do with passwords. Um, so hopefully that's uh clear to everyone right off the beginning. >> If we get to the end, you're like, I know how this connects to passwords. Let's talk about it.

>> We would love to know. >> So, bluff, so AI systems can fail without actually breaking in kind of traditional ways. Um, so what we're looking at is is uh you know, ways that are that are a little bit non-traditional. So, traditional fail analysis is not sufficient for AI because of this. So, as STPA sec, we'll get into that. Identifies hidden and systemic AI risks. Um, and then it also uh proactively applies controls uh for safe AI. So, quick outline. So, we're going to get into the overview uh of of STPA, some of that background, um STPA SEC, so the security kind of extension of that. uh we'll go through a case study, look at identifying hazards for for these

case studies, applying controls, and then we'll we'll leave you with some uh takeaways and a little bit of time I I hope at the end for Q&A. If you have any questions while we're going though, please feel free to uh interrupt. So, STPA is a system theoretic process analysis. Uh so, software's role in this complex uh kind of kind of systems is definitely growing. So STPA is is really uh kind of uh before software. So it's been around for a very long time. Uh these traditional safety methods are kind of outdated for these modern systems. This isn't even including AI. So that's that's kind of why this this second one exists. Um and then STPA offers kind of

a unique system level safety analysis that can be applied to systems extended to AI systems, which is what we're looking at here. Um and then the other key part of this is that it can identify hazards beyond component failures. So we'll get into what those hazards are and how to how to define those. So software uh safety challenges. So accidents can stem from very complex interactions as as I'm sure you know if you're from the software world. Um software flare flaws are systematic, not random. Uh traditional methods lack that ability to to you know make software safely uh and and analyze it. So and then formal verification is not really enough uh to to take us to where we need

to go for this. Um and so STPA is what really helps us link uh system and software safety. So stamp is another uh uh kind of process out there. Uh so it's system theoretic accident model and processes. So that's stamp. Um so stamp itself views safety as a system control problem. So if you're from a controls background, some of this might look familiar. Uh accidents result from inadequate controls um and interactions. Whereas STPA a little different uh it's hazard analysis from MIT identifies causal factors behind component failures. Uh and then really the goal is to prevent hazards. That's really the whole purpose of STPA. So some of the key concepts so it's a hierarchical approach uh for system

analysis uh we'll get into some of these terms but some of the key terms are loss hazard unsafe control action very important part of this that we'll get to later um these unsafe control actions can lead to hazards and so they're very important to identify and try to uh mitigate um and then there's four types of these unsafe control actions that we've identified uh within STPA and that's these four here. So control action required for safety is not provided. So they just didn't put it in. Uh an unsafe uh control action is provided. So it is it is there. It's it's known. Um safe control action provided too early or too late. Uh and then the last one is safe control action

is stopped too soon. So it's it falls into one of these categories when you when you build your kind of unsafe control actions for these hazards. Um and so you you figure out how to kind of c categorize that. Okay. So kind of wrapping up this section. So benefits and applications. So uh you know really focuses on the interactions among the components. Uh identifies missing feedback or decisions. Uh proactive risk development or risk uh management in development. So that's one of the key aspects of this is proactive versus uh you know after the fact. um versatile across uh diverse software intensive systems including AI. So obviously it has to kind of be extended to AI but the whole point of

this is to capture very complex uh a software systems. Um if you know so some camps out there that just say AI is software there's there's some truth to that. So we're extending some of this uh work for for capturing that. Uh and then there is an open source handbook and tool support available out there. There's a there's a ton of literature kind of on this topic. So moving into overview of STPA sec. So this is this is that extension from STPA into uh security. So again you know we're we're noticing the software's increasing role into new safety challenges. Uh kind of a repeat of some of the things but traditional safety analysis is is you know often uh

insufficient for this new complex you know software uh system. um cyber cyber security risks are also rising in complex systems. Uh and then finally STPA sec integrates security into safety analysis. So it's trying to bridge those two things together. So some of the key concepts we mentioned these earlier. So loss is anything of value to stakeholders. So we we'll get into specific ones but you can imagine this is where you really want to focus your energy. uh if there's something that you are are you know viable to lose uh that is very important to you uh SBA also expands losses uh to include security uh concerns so not just things of value but actual uh you know just

purely security items and then a hazard which we'll talk about some examples is a system condition leading to a loss so there's a connection there between hazard and loss um and then finally the the framing the problem framing defines analysis purpose and context. So these are kind of the the main concepts. Um now we'll get into some of the scenarios and talk about a little bit more specifically how this applies to kind of AI systems. So there's the UCAs and scenarios. So so again this is that um I already lost my place. So UCAs are are the controllers actions that are that are leading to these uh hazards. So four types we already we already covered

those types. um focuses on unsecured control actions. And really what we're trying to get at is these scenarios drive how these UCAs arrive. That's that's really the key part. And then you can translate, you know, some of what we build here into your own systems and systems that you're going to be working on in the future. Um accidents stem from these uh systematic failures and interactions.

So some of the benefits and uses um so it really focuses on these complex interactions we've kind of talked about that um it's proactive uh for the development side uh broad applicability across diverse systems. Uh we talk about this open source handbook uh that's also available. So, STBA SEC is also welldeveloped, well known out there and and now we'll kind of talk about how you bridge that piece into into these AI uh systems. So, case study, we'll we'll dig into a little bit more more detail here for uh AI decision support systems. Um we've got two that we'll we'll go into. So, challenges of AI systems. I'm sure everybody in the room is is really

starting to think about what these challenges are. It's not magic. it's not going to solve all of our problems. Uh it is very powerful. Uh but you know what are what are the things that we need to be concerned with? What are the things we need to start analyzing from a security perspective? Um so modern systems are increasingly software intensive and complex uh clearly uh traditional safety analysis again uh in the AI space uh falls short and then there's new you know emergent behaviors. There's there's all sort like uh hallucinations. There's all sorts of things that are that are new from this AI space um that are a little different from kind of your traditional software

security uh behaviors. Um and then our focus here on these case studies is this AI decision uh support system. Uh so Josh had a really great point there. We're really thinking about emergent behavior. So this is a planning tool. Um and it's focused on the mission and business level, not component level failures. So, if a hard drive goes bad, what happens to the AI? We're not thinking about that here. Um, we're not thinking about like an AI system necessarily that has levers like training a model, right? We're thinking about what happens when you take an AI model and you put it out in the world with uh in the wild where there's other uh objects, people, you know, weather,

environmental conditions interacting with uh either this autonomous system or artificial intelligence agent. Um so that's where STPA really thrives. Uh is looking at these emergent and complex interactions which is perfect for AI. Uh it's totally living at this mission business level, right? We have mission objectives. Let's see if an AI can handle it. We fire up some chat GPT throw all our documents in it and something good maybe comes out. That's as coupled as you can get to mission business level uh with a technology. Uh so it's very important that we do analysis like this. Uh if you look at technical readiness levels uh which is maybe an arcane topic but uh it goes from like a transition level one to nine

um and then you do different kinds of testing all the way up at the ladder. So lower levels of the ladder you're doing uh lab testing and then you take a operational capability out in the world you need to do operational testing. Um so STP is a way to do that. It's a way to project and then we can also talk about how you might use it for red teaming and blue teaming. So quick quick quick thing I'll add is that um an interesting aspect of AI that's that's kind of new for for some of these uh systems that are out there especially at the operational kind of mission level uh is that humans are

going to start relying on AI to augment their role. So it's kind of like a human plus technology which is a little different from from the way we've kind of analyzed systems in the past. Um so having some sort of methodology like this that helps us do that analysis is is really important. Uh good segue. Like how many of you have seen this disclaimer like really really recently? It's in everything that has a an AI prompt, right? Uh in the DoD, you don't get to have bad days. You don't get to have an AI decision uh model like give you bad recommendations and then operator acts on those. So control actions here, right? I wanted to find an

image that really like told the story of control actions. So you have a human operator, you have some kind of autonomous bird in the sky that's taking these footage. That's a controller. Um, and then if you add AI into the mix, maybe an AI model is making a recommendation to this controller. Controller may not trust it and may not take action. That could lead to a loss. So that makes it unsafe control action. The operator could take the wrong action and you could have a greater loss. You could have a loss of life, loss of a military asset, loss of military secrets. Um, those are huge compromises that aren't acceptable in a DoD mission space.

Okay, so we've got a couple case studies we'll go through. We'll we'll kind of show the system diagram modeling of of what this looks like from a high level. I'll go into some of the hazards, uh, some of the use cases. Um, so this first one here is about an AI coding system. So, uh, seemingly something you would want if you're software engineer, you want, you know, some augmentation of your workflow. Um, so clearly LLMs are going to be a very critical part of of something like that in an AI system. Um, so we want to automate, you know, kind of the goal here is really automate repetitive coding tasks, generate code snippets. There's all sorts of other

things you could imagine you'd want to do with this uh AI coding system, but that's that's a clear uh benefit there. Uh, you might also see proactive bug and vulnerability detection. Um, some of you might be cringing at the fact that that's probably not true. Um, but it is a thing that you could imagine being able to do versus having an engineer, you know, just sit there at the terminal and and, you know, type Python. You might be able to introduce some of these things. Um, so bunch of bunch of system losses you can think about here. Um, one of them is just inoperable or buggy code. >> Who's doing like vibe coding type stuff

or real coding with AI? >> Scaffolding. >> Okay. >> Scaffolding. [laughter] Scaffolding. >> Just one, >> two. Wow. Okay. >> Okay. Okay. Okay, we got a couple. All right, half dozen. >> Anybody sell their vibe going company for a million dollars yet? >> Okay, I see you there. [laughter] >> Um, so kind of from a high level again, uh, a simple control diagram of of this AI coding system. Um so you know some of the components here controller would be developer another controller would be your agent plugin um IDE uh and then you get to uh kind of where where things live in the in the uh lower parts where you're doing log logging you're doing

some feedback store and then you're finally kind of doing a training pipeline and so you might send a prompt to uh the agent plugin that's that's your control you might get a AI response back that's the feedback from that agent plugin um kind of and so on and you see you know existing code generated code coming from that uh agent plugin to the IDE um and so on. So this is kind of how you would start analyzing uh an AI coding system. Obviously you could get much more detailed than this but this is kind of a from a high level what you what you should be considering as a kind of starting blocks. I'll let you do this one.

[snorts] Okay. Okay, so this one's uh kind of predicated on like a government analyst or a DoD analyst using an LLM to gather open source intel. Uh so LLMs are really fantastic at parsing through data. It's really easy them easy to point them at Google or like a rag database or something and say make sense of this for me. Tell me what I need to worry about in this giant pool of open source intelligence. Um, well, if you've ever used an LLM, they are gaslighting machines and will outright lie to you. Uh, could be a problem for an analyst that needs to do something actionable, right? Um, so immediately we can see kind of losses that that can occur.

Is that good? Okay. >> Um, here's a control loop diagram uh for such a system. It's exemplary. Um but what you have is again these controllers at various different points. Um this one has nice color coding. So the blue lines are control signals and the return lines are like sensor signals. If you think about it from terms of like an actuator uh you maybe you have a pos excuse me position sensor and an actuator. Uh you sense a position it returns a position the act controller says I need you to move here. the actuator moves, feedback circle happens, and then when the actuator gets to the sensor's position, uh, it stops. So that's the whole

control loop. So that's what all this is modeling, control signals, feedback loops, and the thing that does the action.

Okay, so kind [clears throat] of summary slide. So you know, why why we'd want to apply STPAC for AI systems. So it actually does uh start to address this emergent behavior, non-failure risks in AI that are a little bit beyond the sort of normal complexities of of software. Uh reveal systemat uh systemic risks from misaligned recommendations and feedback. We saw that in the in the diagrams. Uh we can explore diverse causal factors in AI system workflows. uh we can proactively integrate safety and security into the system design uh by looking at those diagrams and we'll see next uh how we uh dive into hazards um and then also it's applicable across kind of various uh complex AI enabled

systems. So you know we just gave two case studies you can you can kind of go through the same process for sort of any AI system. So identifying hazards. So hazard identification and STPA. So just from a from a high level back to uh basics of STPA. So uh it's vital for complex systems uh because these hazards are are really what you're you're after. Uh traditional methods again fall short of of this for complex systems. Um STPA is a modern approach that helps us uh integrate hazards and loss into into what we're trying to do. uh and it views it as a control uh safety as a control problem. So defining losses. So uh this all comes

back to stakeholder values. So what what are uh the things that the stakeholders actually care about? What are they trying to protect? What are the things that they can't lose? Um start by defining what are the unacceptable negative outcomes. Um, you've probably done this in some sort of risk management uh type of way before. Um, but this is kind of a a different way of of thinking about it. Um, so losses, you know, especially for, you know, DoD missions as Chris was mentioning. You know, that can include harm to life, property, and environment. So, um, could be, you know, really, really large, uh, things like that. Um, but it could be things that are just important to your

organization. So, modern systems, uh, add mission and data losses. Um and then we really want to use this to uh include the specific uh security lo uh losses. So not only thinking about kind of what the stakeholders value um but also putting in things that are very specific to security. So defining system hazards. Um so these are conditions that lead to loss. So that's really what the hazard is all about. So you want to start with defining these losses. U hazards exist within the systems control boundary. Um so you you can't really build a hazard for something that's outside of your system. You're trying to focus on what is covered by your system. Um what can

you do something about? Um focus on controllable system states for design influence. Um and then you know there's some examples. We're obviously going to continue with the case studies that we had. Um, but they can include security and operational violations in addition to uh, you know, things that you're trying to protect from a a stakeholder perspective. So, moving this to AI systems. So, we want to establish control structure of the AI system. Um, so we want to model commands, feedback, decisions between humans and the AI. So, we saw some examples of that. um especially in like the o OSM one the OSENT uh where there's an engineer uh somewhere in the mix that's a human and then there's a system

an LLM so interacting so that's that's the scenario that we really want to kind of examine uh closely uh and then we want to identify the power button [laughter] is it coming back all Right. Yes. All right. Uh and then we want to identify critical system hazards. Um like inaccurate intelligence. I think we had, you know, kind of an example of that. Um but what are the really uh critical hazards? Uh could go back uh to some of the losses, but that's that's really what we're looking at there. Um and then you want to analyze these these UCAs, the unsafe uh control actions um even from non-failures. So even in a case where it's not a drastic failure,

you know, I think the gaslighting is a is a good example. I think generating incorrect uh code is is one that could be very subtle, might not be seen as a failure. Um so you want to exam kind of kind of past that. [snorts] Okay. Uh so let's look at this from that AI coding system perspective. Um so one system loss that we can consider that we talked about before is this inoperable or or buggy code. Um we might have inaccurate code generation and workflow uh uh errors as key hazards. Um so if we're relying on the system to provide us with code that we're actually going to use on this real system um then you

know incorrect inaccurate code is is going to be uh a pretty bad hazard for us. Um and then AI unpredictability obviously creates this emergent and non-failure risks uh that are that are key to this analysis. So the control diagram uh we'll revisit that real quick. So we've got the developer agent plugin the IDE all the way down to training pipeline. Um we kind of see uh from a high level you know what the interactions between these things would be. And now we're going to take a look at what that translates into from a hazard perspective. So these are not necessarily prioritized but these are these are some of the identified hazards. Um so agent generates incorrect vulnerable or

malicious code that the developer unknownly accepts. Uh feedback and logging data expose sensitive or proprietary code. Obviously one uh that's very important. Uh prompt context includes misleading, ambiguous or incomplete information degrading the model performance. Obviously bad for future versions of that model. Um updates we have cursors specifically but you know any any sort of AI coding uh system. Uh degrades the functionality introduces regressions or misalign with with what the user is actually trying to do. Uh telemetry collection could fail or is manipulated. uh and then the training pipeline might retrain on biased or lowquality feedback data from from the process that you're actually in instantiating. So all realistic hazards uh that need to be kind of enumerated uh

to understand how to protect against these. >> Sure. So I actually want to go back a slide because there's a really interesting attack that just came out. Um, so if we added GitHub on here, which uh these kind of tools have access to GitHub now, they can pull repos, they can make commits, um, they can clone repos. Uh, there's an attack vector where a readme in a git repository has an adversarial prompt. And so if you tell your AI coding app to pull that repo, clone it, you've now essentially jailbroken your AI coding assistant and now you don't know what it's doing. So it could be passing API tokens, sensitive information, making unauthorized commits, uh super fun

stuff. So what that would look like is uh probably another controller that says GitHub hooked onto this IDE um and then ingest the repo adds it into your codebase. So without that extra box, you can't really do the analysis like Josh was saying. So we'd have to add that on. That becomes an extension of our system. Um so it's very important to capture that because uh that shows you parts of your attack surface uh that are available to the adversary. Um I don't want to spend too much time on this osent LLM. Um but you also need to oh we don't even have the control loop here. So you would need to go back to the control loop and that's how you

start building out hazards. Um and then the next step which Josh is going to talk about is start identifying the unsafe control actions. they're going to lead to those hazards. So, these could be because of accidental operation in the environment. It could be um unsafe control actions induced by an adversary to lead to a loss or it might be just something that you're considering like how do we put a control on this and how do we safeguard it so adversary can't come in and tamper with this control. >> Yeah, I think I think some of these are self-explanatory. you you can see you know one loss when might might be the inability to categorize data accurately

which you know when when the analyst is actually relying on that information that may not be uh you know something that that you you want to happen. Um so you know there's going to be lots of different hazards that kind of come out of that system. Um so modeling uh control structure. So uh you know we want to look at this from a hierarchical point of view. Uh try to understand the system components and their interactions. Um define these control actions and feedback loops. And then we really want to model uh these AI components explicitly as controllers. And so we've seen that in the in the past couple slides um where these AI pieces do actually control things. they

actually have uh functionality beyond you know just receiving signals. So identifying unsafe control actions. Um so we've we've gone over those four types before. Uh but we really want to define these clearly for the systems that we're talking about here and and for those hazards. Um and we want to do that beyond the component failures. As that slide uh Chris showed a while back. Um we're not just operating at the component level. But we're not just operating at the system level. We're at kind of the mission operational level. So for the AI coding system, um kind of related back to those those hazards, um so we have some identified ones here. So we have a control action and then an

unsafe when some some something is done. So developer accepts uh AI generated code. Um that could be fine. uh but it's unsafe without proper review leading to the deployment of insecure or incorrect logic. So makes sense. Um but you kind of go down and you enumerate these. Um so we have for example you know we have a suggestion being returned uh that could be fine but if it's based on uh you know undated or relevant in info then that might not be uh you know what you want. um all the way down to things like developer feedback um you know is applied to training uh could be great but but if you're not auditing that

feedback if you're if you're not controlling who has access to give that feedback to the model um then it could be incorrect feedback it could be fe feedback that ends up in bias in the training data um etc u so these are all potential uh uh unsafe control actions based on some of the hazards that we identified but you can see that as you kind of blow this up and start to build out the system diagram to you know really control uh use use everything uh that's in your system um as as a as a control point and a and kind of feedback um similarly sorry similarly for the uh osent uh you know we have some uh

non-failure that leads to an inaccurate intelligence we've gone through some examples of that where you might get uh inaccurate intelligence or completely hallucinated intelligence um may not be uh a failure in a in a usual sense, but that that might lead to a failure uh down the way. Uh we might actually see a situation where we're missing feedback uh that causes a misalignment between the model outputs. Um and we're really trying to do this in a proactive way so that we can identify these risks as as soon as we can. Okay. So the loss causal scenarios. So we want to identify what these causal factors are for unsafe control actions. So we're not just kind of identifying

hazards, identifying these UCAs and then you know calling it calling it good. We actually want to know what what are the causal factors? How do you go from one thing to the next? Um so inadequate control algorithms can also lead to uh UCAs. Um inconsistent process models definitely that can be a significant factor. So you want to make sure those are correct and and uh detailed. Um missing feedback or incorrect uh inputs can also cause these issues. Um specifically to AI LLM, uh insufficient training data can impact that. So uh that could be insufficient from a collection point of view, specific to what you're trying to do. It could just be training data uh as a whole

insufficiency. Um, and then lack of confidence uh really uh affects this intelligence uh workflow. So if you can't trust the data that's coming to you, then obviously that's going to have a big impact on how you do your your workflow. So STPA for AI. So some of the benefits so it addresses AI's unique and emergent hazards. Um provides comprehensive causal analysis. uh enables proactive design and integration of safety applicable across you know diverse complex AI systems uh by extending it all the way to to AI uh we can do this um and then it also integrates with various safety and security standards you can bring in now you know what is what is NIST RM you know AI RMF say like

what do what do we have to what do we have to think about when we're designing these systems security by design a great great tool to use in handinhand with something like

Okay. Um, so we're we're now at the place where we're going to be applying controls based on these these UCAs. >> So yeah, so we're applying so we're going to transition from this hazard identification to risk mitigation. So so we've identified the hazards, we're we're happy with the things that we've we've we've built there. Um but now we want to try to mitigate against those. So we want to apply controls to prevent these occurrences. So we want to integrate safety and security into the engineering process. So feed this back to the engineering team. Um and then we want to develop uh inherently safe and secure systems as close to the left as we can possibly get.

So core principles. So we want to enforce constraints. So safety is a control problem not just uh component failures. Uh accidents result from inadequate control or uninforced uh constraints. Uh controls enforce these constraints preventing the hazardous states and then safety constraints are critical boundaries for the system behavior. So you want to have uh some sort of constraints within the system of how you're going to uh do this analysis. So deriving requirements. So now we have these unsafe actions. We have we have these things enumerated. We want to derive a requirements. What does our system need to do in order to uh keep itself safe uh from these actions. So we want to formalize the safety and

security constraints. Those are the constraints from the last side. We want to create those boundaries. Uh we want to guide system design and implementation based on this information. And then we want to prevent hazards through these concrete requirements.

And then formalizing the requirements. Um we want to formalize any sort of informal safety requirements that we have. Um use formal methods like LTL for verification. That's for another talk. Um but then ensure rigorous software safety verification. So any place that we have these these hazards identified um where can we do something to ensure uh that we're doing the right things from a safety and security perspective uh in those hazards. Uh and then we want to bridge the gap to formal verification approaches. So we don't have to just end here. We can actually move what we've done analysis wise into formal verification. And then safe behavior modeling. So we want to model this software behavior for

safety. So we can construct uh with designers and analysts. We can do this collaboratively. Uh we can focus on safety related variables. We can visualize using UL other other types of uh you know state chart uh notification, state machines, however you want to do it. Uh and then we incorporate these STPA derived safety requirements back into uh the the uh full plan. And we've done this kind of thing with people even in kind of a tabletop exercise kind of point of view. So even even something as informal as that can really inform a lot of of a lot of what this is uh doing. So generating safety test cases um so we can automate this uh uh we can do this

generation uh from an automated point of view. We can convert the model uh into testing tool input language. We can do uh suitability test coverage uh generate test data, remove duplicative tests, uh all sorts of things in order to generate these test cases. Um I might add we can use AI to do this as long as we're doing this in a safely and secure manner. We can we can utilize AI for this software verification and execution. So we can verify the safe behavior model uh against those STPA requirements that we've formed. uh we can assure that the model satisfies all of that safety criteria that we've we've set. Uh convert abstract ta test cases into

executable scripts. So now we're moving over into the actionable uh side of this. Run the scripts to validate software implementation and then link those test cases that we've built um back to the requirements for traceability.

Uh so fun little image but uh you know one one thing you can do with SCPA sec that's kind of building up to this is that you know wargaming mitigations. So addressing kind of intentional threats from adversaries. So that's that's kind of going back a few slides but thinking about security as as really one of the hazards um or one of the one of the uh uh what you really want to protect from a stakeholder point of view um against uh adversaries. So if there's intentional threats trying to model those uh wargaming is a is a great uh mitigation tactic for that. Um you can involve your red and blue teams uh to build this from a kind of simulated

emulated point of view. Um and then you assess the effects and costs of those countermeasures and then you're going to bring what you learn from that back into your your system design and analysis. system level interventions. Uh, so I'm running a little low on time, so I might run through this. So, we want to address hazards beyond the component failures, feedback paths. Um, really what we're getting to at the at the bottom here is we want clear accountability for any harms that might come to the system or to the users of the system. And then, as always, continuous improvement and integration. So we want to iterate this throughout the entire system life cycle. Not just at the

beginning, not just in the middle, not just at the end. You want to continue to do this um refine safety and security measures continuously. Uh we want to integrate with uh software engineering standards and with those those teams um and support uh uh sorry few future work uh involves tool support and automation. So um as this gets more adapted into the AI space, you can expect to see more tools uh being developed in this in the space. >> So a place where AI is definitely helping in STPA analysis is uh just enumerating and coming up with hazards that could cause an unsafe control action. That's usually a stage where you need like a few humans in a room like

smashing their head, getting really creative. Um AI really helps you kind of like go through a bunch of plausables and and see if you can get to these actions. and the hallucinatory as aspect of it was kind of a benefit in that regard.

Okay, couple quick slides on takeaways and then we'll open up for questions. So, um I think we've beat this over everybody's head so maybe there's nothing to to continue on here. Um but one one is this risk compliance and governance. So, uh, sometimes a complicated space that's very different from system design, um, but still needs to be integrated back into your your workflow. Uh, you can you can use this methodology to to bring those things in earlier. Um, and so now early in your system design, you're thinking about risk, compliance, and governance instead of towards the end. Um, so you can derive detailed safety requirements from these UCAs. So that helps connect back um and then address intentional security

threats back to kind of that red teaming piece um and then enhance uh compliance with these industry standards that are out there. And with that we'll open it up for questions.

>> I thought it was interesting the point that you raised where the system's not trustworthy. I thought it was interesting the point that you raised that if the operator finds a system to be untrustworthy, not only are you then going to have the risks of the system causing damage due to bad logic, but you're going to have the risk of the operator failing to allow the system to actually do its job. >> Yep. >> So, >> yeah, I can give you a couple concrete examples. Uh, the radar system I worked on, it didn't have AI, but it had like automatic gain control, which is very new at the time. And what it was supposed to do is uh adapt the filter of

the radar automatically to filter clutter out. Well, if you turn this on, um it sees a lot of clutter, so it lowers the gain all the way to the bottom. Nice clean screen, but you're not going to detect an inbound missile with a radar cross-section like smaller than a B. Uh because your filter is so high. Um we just turn it off and not use it. So, it didn't save us any time and it cost the Navy a whole lot of money. um another system uh computer vision uh physical security camera detecting intruders, right? Um so the the DoD sponsor paid us to come in and do an analysis of how could this go wrong.

There's a couple of specific adversarial attack vectors that we want to do analysis for. Um so one was injecting adversarial artifacts into an image stream. Another was creating these adversarial patches that you can like put on a wall and if the camera sees it, it suppresses object detections. Very effective. We do the analysis. Not only can you suppress detections, but you can actually create so many false positives that the user instantly turns the system off. So you just undermine the entire security system. >> When situations like that arose, what did you do to regain trust in the operator once you had repaired the or improved the problem? So for that one uh it kind of underscored that these

adversarial patches are a huge problem in AI space and you have to control them somehow. There were no controls for that before. So if your system encountered this in a wild, it's a new novel threat. There's no control mechanism to suppress it or even deal with it. Maybe a human user would see it and be like that's a weird pattern. I don't know why it's making all these false positives. Um so now there's controls you can do training on your model against those patterns. you can detect the patterns, things like that to like alert an operator. You can um say that we have really low confidence in this detection. So maybe like don't use the system right now,

things like that. >> One one thing I'll add to that is that another thing we learned is that the system just wasn't robust enough. um it wasn't it wasn't built to not only not withstand these these patches that we built the adversarial pieces but once you started to dig into the system itself uh you could give it you know kind of noisy samples and it would fail miserably so it just it all the way came all the way back to like good TE V&V um so they needed to get back to the drawing board any other questions we've got time for one

No, it's all. Thank you very much. Thanks everyone for joining. Thank you.