The Human-AI Handshake: A Framework to Build Trust and Unlock Innovation in Security Operations

Welcome, welcome. Come on in and sit down. Uh, I'm here today to present the human AI handshake, a framework to build and unlock innovation in modern security operations, but the important thing about this is that it's a human-centric approach to what we're doing. AI is a very inhuman system. So, uh, that's the framework we're going to kind of walk through and talk about today. Uh, my name is Michael Rajie. I'm currently a principal threat response analyst at CrowdStrike, but today I'm just Michael um not affiliated with CrowdStrike, just a guy from New York. Really excited to be at Bides NYC. So, thank you guys for coming out. So, we're going to start with kind of an

image, uh kind of a story. So, imagine a chaotic universe in which we've heard a lot of metaphors about AI and how it is a transformative technology. It's a printing press. It's the internet itself. It's going to change absolutely everything we do. It's the big bang. I'm tired of this metaphor. Uh I've heard it a lot. Uh so I thought I would try and do a little bit of an anti- metaphor. Talk about the downsides of a big bang and how that might create problems at scale in both a universe and an enterprise environment. So imagine this chaotic universe where stars are scattering across the heavens. New things like dark matter and black holes are not understood and all of a sudden

are getting in the way of everyone trying to travel around space and planets that previously didn't exist have to grapple with new laws like physics and gravity. Now cut over that metaphor to an enterprise environment where new AI wrapped tools are scattering across an enterprise stack. APIs aren't talking to each other. Dark matter and black holes are resulting in time waste, resource waste, and APIs unavailable across different applications. And AI is being brought down to Earth by things like legal frameworks and technical limitations. Things like new gravities that previously we didn't understand. But imagine we could direct the Big Bang. Imagine we could exist in an orderly universe and deploy very deliberately these things that are

supposed to transform the world as we know it. Imagine a centralized AI interface where you can map the stars and how they talk to each other, knowing what is going to communicate across which APIs in your environment. Imagine if you had a collective scope and purpose for AI rather than just deploying all of this matter into the universe and watching what all of the different teams or galaxies do with it. And then imagine the creation of your own gravity, a governance for an AI framework that defines both who builds AI and how it is deployed throughout your environment. So that's kind of the fundamental of this human AI handshake. So we're going to start with an

allegory, which we already did. I'm going to talk about the framework itself and kind of plot it across a graph. Uh I'm going to talk about some of the prerequisites for uh automation and AI reviews, building a central interface, telling AI what not to do, which is probably the most important thing about using AI. And then winning champions for AI adoption, and how to specialize AI and security operations specifically. Uh and then finally at the core of this is really the importance of feedback and listening and uh keeping in mind that we're deploying an inhuman system in enterprises which are human organizations and constructs first and foremost. So let's dive into the framework itself really quick with the definition. A

framework is a mental structure underlying a concept or a system. So what system are we describing? The system is AI and security operations. In order to understand the system, we need to map functions onto it to describe how it works. And then we talk about how these functions interrelate. So let's make a graph to make this simple. This is a graph with nothing on it. Uh it is an x-axis and a y-axis. We're going to start with the easy part, the title AI deployment and security operations. So this is a uh xaxis and a yaxis. We're talking about the system first and foremost trying to define how to move AI across a given sec ops

environment. We're going to want to talk about adoption. And adoption is easy. It's something that can be measured in percentage. It could be measured in numbers. So uh thinking about measuring AI adoption, this could be percentage of users in your environment, percentage of teams utilizing a particular uh application that utilizes AI or a particular uh agentic workflows that are being impacted by AI. That one's fairly straightforward and that's the y-axis characterizing growth. The x-axis is more challenging. And I've talked to a lot of people about this and they seem to always want to put two things on the x- axis to measure how effective AI adoption is. Those two things are time and cost saved. And each of those pose

an unique problem. AI when you put it in terms of time saved automatically creates this binary between eliminating labor to produce cost savings. So, by putting both time and cost, which are outcomes of AI adoption, at the bottom, we're forgetting the benefits that AI might have on a security operations workflow, we're preventing the ability for AI itself to be measured in adoption to know how effective it is, and we're putting the actual outcomes on the graph itself. This isn't a good idea. So, I would say neither time nor cost are what we put there. But what do we put down there? I mentioned enterprises are a human framework. Humans are being confronted with technology in a new way

that requires trust. So I would put this in terms of trust generated. Trust generated in the applications that we're using, trust generated in the models that we're utilizing, and trust generated in the teams themselves that are actually developing and releasing this AI in an environment. So, uh, really quick, we're going to walk through non, uh, numerical measures of this, but rather qualitative measures across your security enterprise life cycle that will tell you where you're at in your environment, uh, for AI deployment and talk about what level of adoption you're experiencing in the larger enterprise environment. So, we're going to go through 10 of these and then there's going to be slides at the end to

cover this again. So, don't stress if you don't get them as they're flying at you, uh, like little spaceships. The first one we're going to cover is a technology and process audit. This is at a point before you've selected or procured a technology. This isn't really about if you're using anthropic or chat GBT. It's really about the technique. So before you're utilizing uh AI in your environment, you want to understand what technologies you're going to use and combined with AI and what processes you're trying to transform with AI. This is kind of mapping the universe intentionally. Knowing what AI is going to be for rather than deploying it across your enterprise and letting people novelly develop what they want to

do with it is going to put you in a place to succeed. And at this point in time, you only really want the kind of innovators in your environment utilizing AI. After this point in time, you need to ask what are you going to feed to your AI and your security operations. You have wikis, you have confluences, you could just download all of it into an AI environment and see what the AI returns when queried without particular filters, without particular segmentation of data and without particular uh sanitation of data. But what you're going to get is a mile wide, an inch deep, and it's going to conflate many many things. So you need to understand

what your AI knowledge repository looks like, scrub it for high quality data, and make sure you have particular use cases in mind as you're developing that. Once you have that, you need to ask and answer the question, how are we accessing AI as a security operations organization? How are our sock people accessing it? How are our threat intelligence people accessing it? How are our incident responders? And then how are the people who are just procuring technology accessing it? And I would make the argument that a unified AI interface, not necessarily one tool, but a particular repository like a git or Bitbucket repo for maintaining your knowledge files as well as a unified application for interfacing with that to

send requests in and get requests back out for multiple teams is a key indicator of success for this particular framework. And the reason for that is if you don't have a single interface, you need to cultivate trust in every single tool that you're implementing with AI, which is a larger lift than doing it with one interface unified across your environment. Okay, after that point in time, we're still in the pre-production phase. What we need to do is we need to introduce the laws of gravity, the laws of physics. We need to talk about AI governance. We need to talk about who is building AI, what it can be used for, and what your AI agents are actually

going to be allowed to do. Telling them what not to do, what data not to return in answers, what PII not to return from the database that they have access to is going to be key for governing how AI is going to then be scaled across your enterprise. So now you're at about 20%, you have a couple of pilot teams utilizing AI. you have not gone production yet and now we're at a point where we're ready to productionize and we need to start winning influence in our enterprises, winning trust amongst our users. So there are two primary forms of AI that we deal with in enterprise settings. Agentic AI and uh generative AI. Think generative AI is the chat GPT

that we're all very familiar with. Agentic AI is a passive form of AI that's operating on passive systems. A lot of people would start with the fun thing, which is the little chat bot that you can ask weird questions to about arcane video games or whatever you want. But instead of that, I would caution to start with the agentic because the way that you win support is by making it seem easy amongst people in your enterprise. Winning champions by automating the menial tasks in their day-to-day workflows and making life seem so much more doable because of these AI tools. And once you have won over these trust networks, these trusted people in aentic workflows, they're

going to be the people that drive through human networks the adoption of AI in your enterprise. Otherwise, you're going to be stuck with these human roadblocks for very normal and ordinary reasons that I'll cover in a little bit. Uh, okay. So, we're at about 30% now that we've automated our agentic workflows. We're now ready to push into a deeper adoption set uh with uh generative AI agents using these trusted teams as people designing and deploying AI. We're not going to be able to build this for every function of our enterprise. You're going to need to find trusted outposts of expertise to give them the ability to utilize that single interface, that system for building AI

as well as disseminating AI. So basically, how do you guys do your job? How do you do this on a day-to-day basis? Can you create a unique underlying knowledge repository and a very smart agent to assist you in your day-to-day workflow because I don't know how to do what you do, but use my system to do that. And you can also inherit all of the trust that we've built across this organization and make this the primary platform for AI adoption and security operations. So now once we've done this, we're at a pretty good level of adoption. We're at 40 to 50%. Uh now we really need to start thinking about all of the stars that we've scattered across the sky and

how messy this is getting. We have a lot of smart people building a lot of smart tools really rapidly and our system is getting a little bit sloppy. We're really thankful that we had AI governance at the beginning that said this is how we do it. This is who does it, these are the checks that we do and this is the testing that we do for different types of AI. Um, and at this point in time, you're going to want an orchestrator to be built on that knowledge funnel at the front end that says, "We now have 15, 20, 35, 50, 110 specialized agents in our enterprise environment being used. We need to put an orchestrator agentic agent at the top

to redirect those requests to the right generative AI bot or agent that has been developed. That's going to allow us to specialize and go deeper with the knowledge repositories that lie underneath redirecting the right queries to the right places. And then once we kind of get to this point, we have some latestage life cycle stuff that's just going to drive our growth. Uh things like metrics tracking, the ability to have feedback, the ability to have uh a mature environment that trusts our outputs and validates our outputs so that we can actually use those AI outputs for further automation in our environment. And then finally, it's going to be the life cycle cycling through this the stasis of maintaining

the underlying systems, maintaining the underlying knowledge repository and updating the processes as they're important. So anyone who might have mistakenly gone to business school for like a year and a half like I did um might recognize this curve. This is the technology adoption curve. Um basically what we're looking at here is different segments for the adoption of technology life cycle. We see innovators at the beginning early adopters and early majority through the experimentation government uh governance and then early agentic workflows and then our later majority comes when we start allowing our workforce to specialize in the uh generative AI specialized systems that they're working through. And then once we get into the late stage, the lagards,

that's when you want to start automating on AI itself to push adoption through the far reaches of your security operations environment. Okay, so this slide's about to go away. If anybody wants it, now's the time. Uh I'm going to hit this a couple of different times uh on the next few slides to dive deeper on thoughts. Okay, I think we're ready. So key points for success that I'm going to dig deeper on in this technology and process audit. This is at the very beginning before we've deployed productionized AI in any sense. We want to know what technology is available to integrate with AI. That's specifically what APIs have access to, what restrictions exist on those data, what limitations you're

going to encounter when you combine those APIs with other technologies specific to your AI environment. uh and then basically trying to figure out how you want to go about doing that. Um you want to know what processes are suited for initial roll out of AI and what processes are not suited. So if there are compliance things that suggested is not going to be a good AI road to go down because there is validation levels that can't be achieved with AI, that's not a good technology use for AI. You're going to end up wasting a lot of cycles. So, um, this allows us to know what natural stewards of AI exist in our environment and being able to answer

where and what teams are going to be really good partners for developing AI. And we want to know this before we start building the AI in general. Um, I talked about this at length, so I'll just hit it one more time. Building a knowledge funnel. This is building a single source and repository for your AI data sets as well as an application interface to cultivate trust behind rather than allowing a million different rappers for security tools. Um, tools have their difficulties and people have their biases about tools. So, whatever tool in your environment that you're rolling and wrapping AI in is going to inherit the biases that exist in the previous tool. By cultivating trust in a centralized

manner in a single tool in your environment in the right way, uh you're going to be able to do this process once rather than having to do it again and again and having your user base wonder is this a black box? What is behind this particular AI data set or knowledge base that I'm operating on? uh providing a level of transparency to the underlying files that exist in GitHub and Bitbucket repos allows you to empower your analysts, your different teams, your different functions to answer the question and get over the human bias of I don't know what it's doing and therefore I won't use it. I can't trust it. Okay, this is one I want to hit really

hard. Telling AI what not to do. Um this is the core function of what us as security practitioners need to do in the adoption of AI. This is telling what teams are responsible for building AI and uh telling our organizations to collaborate with those teams specifically. It's defining AI governance for deployment and it's defining AI governance at a tool level for a generative AI agent itself. So when you build an AI agent, you're telling it what is its identity. You're telling it what it's expected to receive from its users. You're telling it what type of prompts it should return. And you're telling it, prompt your users to interact further with that prompt. But what you're most core telling it, don't

do this insane thing that we're not allowed to do for a legal reason, for a data reason. You're not allowed to validate false positives or true positives. You're not allowed to generate novel code combining separate code sets that are in our different repos. If you don't provide these guard rails and test these guardrails, you're going to end up with different hallucinations that are not productionizable and it's going to create a headache on a liability, on a legal, on an operational level for your security operations, for your incident response, for your threat intelligence conclusions. So making sure that you have really strong guardrails for your AI and a mechanism for building them is really core in security operations

before you roll out these different agents and making sure that all of the different teams are adhering to these core functions for building generative AI agents. Um okay this really gets to the core issue of trust that I mentioned is the fundamental measurement of adoption for AI winning friends and influencing models. Um, you want to map the trust networks in your enterprise. And when I say trust networks, we all have people we work with who you ask questions to because you trust them and they just know things deeper and better than you could. Mapping these trust networks and winning over these individuals by asking them deep questions about what they do and how we can enable them to do that

better is how we automate via agentic AI to win trust in our organization. you tap into pre-existing networks and on a human level cultivate what you need to have people comfortable in this technology transformation. So taking input and being honest and transparent about what the AI does, what it's built on, and what your intentions are for it. So if you're automating processes, you're automating jobs, being transparent is key here. It's uncomfortable and it is a new thing that we're all dealing with and have fears about but it's happening and being honest about it is going to allow us to not have to overcome a major hurdle for trust. Uh I think I covered this one but it's a

good slide just to sum up the agentic and uh generative functions. Agentic is the on the early end. It is a passive autonomous multi-step automation process for menial tasks not meaningful tasks. We never want to be in a position in our enterprises where we're taking away core actions of security teams that provide identity that provide uh impact and that provide the ability to cohhere as a team. Automate the manual, not the meaningful is what I say. And uh the generative AI is going to be an active co-pilot prompt-based uh that's meant to kind of 10x the ability of an analyst and really unlock creativity and impact in a given environment. Okay. Um, so kind of moving through uh

this concept, the specialization in orchestration for latestage AI adoption. Uh, being able to leverage AI to direct inquiries to those agents is really key. Uh, I covered this kind of a bit earlier, but these specialized agents are going to perform better than the generalized agents trained on all of the data that you have. and uh creating a front-end orchestrator to make sure the right queries are getting directed to the most educated agent that you have to answer it is going to be really key to just keeping an organized library of agents in your environment. Uh and then from a metrics, feedback and active listening perspective, um we need to provide mechanisms for feedback on a

human level. And that's going to include listening. That's going to include listening to anxieties. That's going to include listening to technical issues. That's going to include listening to legal compliance frameworks and figuring out why all of these things are human obstacles to AI adoption. collaboratively keeping a space and a process for constantly taking feedback in those ways is going to cultivate the trust that allows this to get pushed across an enterprise. Otherwise, we're going to be dealing with 9 to 18 month delays in pushing through these tools as well as a ton of turnover of really key talent that we could have won and understood and built collaboratively with. So having metrics to understand that is really really key. Collaboration

builds trust. Okay, last slide, final slide here. The key takeaways from this talk. The fundamentals of AI adoption is trust. These are human systems being deployed across human enterprises. Um, AI is something that needs to be built collaboratively and that depends on trust. We want to start by automating non-human processes with agentic AI in a way that is repeatable and cultivates trust in our key and most talented technology groups. And we want to implement a cross-domain knowledge funnel so that we can cultivate trust behind it organizing the knowledge files underneath it in a single source for uh ingestion of queries from our multiple security teams in an enterprise. Uh we want to create specialized AI agents and

orchestrators to keep this uh kind of growing and scaling in a way that is manageable. And then finally fostering collaboration in AI development is key. automate the mundane, not the meaningful, and we'll be in a position to succeed with AI adoption, keeping trust central to AI adoption and security enterprises. And that's what I have for you guys today. Thank you. [applause]

Think we probably have time for two questions. Uh so if anybody's got a question, happy to take it or feel free to hit me up out in the hallway at the end. >> Uh in the back. Yeah. Sure.

So, um, trust is really a human metric of operating with technology. So, there's the trust of the tool to be reliable. There's the trust of the underlying data that exists with that. And there's the trust in the future ability to contribute to an enterprise. So the trust in the tool, it is a tool that is not built by the teams that are using it. They're being mandated to use it and it's a disruption to their workflow on a day-to-day basis. For anyone who has ever worked in a sock or uh different security operations functions, Excel and Notepad++, you can pry those out of people's dead hands. Like they will not give that to you. having them trust you enough to say,

"This is a new tool that's going to revolutionize your workflow." Your day-to-day is going to suck for three weeks because you're going to have to learn how to do do the new thing. You're going to be disrupted. Building trust that the outcome is powerful enough to do that is a really important part of it. From the data perception, they if it is a blackbox system where they're not able to examine the underlying knowledge files, there's an inherent human capability to be skeptical of what we're working on. Uh, and so we want to be able to validate what we're looking on, seeing the underlying data sets for that. And at a human level, we want to

understand that we're able to continue to contribute, that we're not going to be automated out of existence, and that by using this, I'm going to write myself out of a job. So being transparent about what we're doing is really, really important in that way of where we're going on it. So yeah, with that, I think that's my time. So yeah, feel free to come up at the end and hit me up in the hallway. Thank you guys. [applause]

Thank you.

>> My question was I think >> Oh, I got