Microsoft Security Co-Pilot: Do you really need AI in your SOC?

Microsoft security co-pilot do you really need AI in your sock if you're putting AI in your socks uh wow those are some high-tech socks um our speaker today is Thomas Deng um currently works as a security architect for the Yukon government in Northern Canada so he knows even more about cold weather than we do uh he's also currently pursuing uh MSC cyber security as uh as a cyber security fellow in your NYU Tannon School of Engineering uh formerly served as the youngest member of uh uh the legislature here in Alberta uh so you know politician jokes are perfectly valid uh as a security architect he's led the Yukon government in his deployment of Microsoft dour based

security platform running a lean and it's Yukon there's of course everything's going to be really lean uh lean team organization the organization has undergone significant overhaul in the last year utilizing Sentinel as a Sim sore integration with on-premises security applications and vulnerability management has been a priority of the modernization of the government security system Thomas has experience in both red and blue team Technologies holder of a cism and ocp certifications and not to mention one hell of sexy beast so take it away Thomas thanks Brad um okay so that's basically covered the whole intro part but uh like he said I'm Thomas uh I am the security architect for the Yukon government um and to give you a

bit of the lay of the land I guess we're an organization that's a mediumsized Enterprise right we've got 5,000 plus employees we like to say it's 5,500 licenses if you're one of my vendors it's 4500 4,000 actually um but uh yeah so I've got that software computer science background um and and my interest in our security team has been of course the modernization and overhaul of our systems we were running a relatively Legacy system for a long time in the Yukon um but it's also in automating the boring stuff right like if you're a security analyst you're a security team um lead you're doing so much boring stuff every single day right every day you're you're sitting there

going okay I've got another fish that went out to 3,000 users and I've got to go pull it out of their inboxes I've got another um alert over somebody clicked on something and typed their password in and now I got to call the help desk and tell the help desk to reset their password and whatever right there there's tons of boring stuff that goes on every single day um so what we have in the Yukon is probably similar to what a lot of medium-sized Enterprises have in their environments as well small organizations on their in their Enterprises we have a really small team it's it's me and three analysts I've got a ceso as well and then we've

got a network team that helps out where they can adds up to six people right like we're we're supporting thousands of V points across um about a dozen departments um thousands of servers all over um private cloud and Azure Cloud trying to do the best we can right and and it's really hard because you've got um in government when you have about a dozen departments it means that there's a dozen different groups of people who consider themselves C Level right and if anybody ever had to support se- level uh Executives they know that's kind of like as well right so you have this really high competing priority list you have this really high uh uh group of admins

right we've got we've got um hundreds of admins really right um who you're trying to Wrangle in a sense and say like how can we get everybody on the same page how can we make sure security is working um our team does run um 247 for availability um I put an Ask there because 247 means there's the four or five of us uh and every day we take the on call phone for the evening right um we use an MDR service for our triage so we have a we have a third party um that does our monitoring and in the evenings if something happens that they call one of us right um and that's probably

really Sim to what a lot of your organizations if you're running 24-hour monitoring um are doing as well you have you have a third party who does the evening stuff because frankly most people don't have the bodies right we don't have the the bodies to do 24 hours in house um but yeah so we're mediumsized but we're here today to talk about uh co-pilot for security right in our organization the Yukon government we were actually piloted onto co-pilot for security uh before it went General availability so so we piloted this um starting in December of last year um which was about six months I think before it went went GA uh and they sold us on this um idea that it was

going to radically improve our our workflows right it was going to significantly reduce analyst time it was going to significant reduce the amount of triage we had to do it was going to significantly reduce the amount of uh response we had to do and and they even told us that in in the future we'd be able to completely eliminate our third party sock service right we'd be able to not need a sock at all uh we wouldn't need 24-hour monitoring security pull pilot would do it um our analyst would just wake up in the morning come into work look at what happened overnight and respond and and that was the pitch right um so we're going to answer a few

questions today which is like what does cop actually do what does security copilot do in your environment right now how does it help with your response and really who is this for who's going to benefit the most from this but let's look at what it does right so so I did load up the sales video they gave us uh let's see if I can get it to actually click maybe oh always test your demos before you go

live my screen's gone black over here is it plain

I can't see what I'm doing up up oh play button there you

go I got it I swear I work in tech oh maybe the reception going to hate us today but I'm going to talk to you a little bit about what it does for us in our environment this screen that screen is not working I don't know why but I will talk to you a little bit about our environment and what it what it did for us so so basically you're given access to this uh like chat gbts it's backed by chat GPT is the is the back end for the engine um screen that's on their custom portal right it's it's the secure uh I think it's Security copil on microsoft.com or whatever it is um so you go to that screen and you and

you get access to these uh playbooks these list books and and the playbooks are able to basically inject a bunch of prompts into it automatically right so they they've pre-written some you can pre-write your own uh doesn't look like this wants to play so I'll just go back to my slides probably there we go yeah so um you can see the screenshot of some of it there as well um and basically you type in your prompts you like select the incident number from your sim and you go incident number 5 4368 I need you to tell me what happened tell me if there's any other things in my environment that look like it whatever right so it you basically

type that all in and or you hit the button and it and it spits out a bunch of responses might even spit out some kql for you so you can put it back into your logs and you can search through them um those types of things right so it's able to enumerate your entire environment right it's able to look through your entro ID logs it's able to look through your Sentinel your Defender um all those types of things and and and like chat gbt like any other large language model we're seeing today it uses those normal semantic queries right so you can ask it when was the last time this user logged in uh what IP address

was this uh user using when they logged in right you can you can do all those types of things you would expect to be able to do with a with a large language model and and it it works in a sense right like it it it does that for you it does that uh if I was just to ask an an a question I would expect that analyst to go and spend 5 10 20 minutes finding out the answer and giving me the answer it's able to do that for you right it's able to go through your logs and find that solution um but the question really becomes like how does it help right what

what are the advantages right um and it uses those playbooks to automatically analyze incident so you can schedule them or you can you can set it up so that if one was triggered or an incident happens an event happens it goes through and runs A playbook for you automatically and and pre-compiled information so that when you go into your sim and you look at it you're going to see that information right away it allows you to rapidly search through that large amount of data um that's in your sim because if if anyone else is running a a relatively large Sim um I like to think we're running a relatively large Sim uh our tendo inest fees freak

me out every time I look at them um but I've heard there organizations spend 10x is what I do right so if you're running a relatively large stim you have a lot of data in there right you have a lot of tables you have a lot of things in your common security log you have a lot of information it is able to help you um sort that information relatively quickly and and the really nice thing about security pull PT is that unlike if you're using a traditional large language model that's just publicly available you go to chat gp.com or whatever um you don't have to standardize your data right you're not worried about is my data going to be fed

into some training model is my data going to be Tre fed into something that's going to leak later right you're not sanitizing your data you're just saying hey what's the what's happening in my environment like tell me right now and and go through my logs and I'm not worried about any of that stuff um so I guess the the question becomes like who is this really for right and and we had a really hard time answering this question in YG we we piloted for six months uh we went uh we were licensed till June um and maybe I'll give you a little bit of a spoiler alert like didn't renew our license right so we're we're not currently

licensed for security c um which is why I'm not doing a live demo for you frankly um but the question becomes like who is this for and and I have a few questions if if you want to answer do you want security C Val organization the first question is what size is your security team right if you're five people like like we are um then you're a reasonably sized organization right you have a reasonably sized security team most or they're lucky if they have one one in in many cases right um but the size of your security team is going to make a big difference on how you answer this question because does your analyst

have time to go and actually respond to every incident they get or are they bogg down in vulnerability management all day like or are they uh or are you an organization that has multiple teams of security people where you have a vulnerability team you have a incident Response Team you have a threat management team like do you have teams those are those are big questions because organizations that are slightly larger with larger teams that have more things going on they might see more benefit from this right because they'll benefit from being able to uh consolidate their data they're going to benefit from being able to not have to go back and forth between as many people

they're going to benefit from being able to uh simplify that workflow but the other question is do you have 247 monitoring right if you're not running 247 monitoring then being able to respond after hours becomes really difficult right if nobody likes getting a 2 am wakeup call uh nobody likes getting a 2 am wake up call because president got fished um and the reality is that if you're not running 247 monitoring um then you need to be a little bit more efficient right you need to be a little bit faster you need to be a little bit quicker because you need to be able to get up in the morning and see what are all the things that happened

overnight what are all the things that I missed while I was off right um and and those are big questions and then and then of course the the other question is do you have a contracted sock right like do you have MDR do you have somebody who's looking at alerts for you looking at incidents for you looking at all these types of issues for you because that's going to be a big deal as well because are you going to give them access to your to your security c um and and we'll talk a little bit about licensing but currently security Hile the way they license it is they called security compute units um and security

compute units are uh processing units they call them and it basically is kind of a proxy for how uh how much you're using the model um your your minimum buy is going to be one secur unit a mon month um they recommend at least three uh and those are continuous you continuously have to pay for them right so if you have a lot of analysts on the MDR team if you have a lot of in house you're buying more scus right and if you're buying more use it obviously cost more money right so is your s going to get benefit out of this are they going to use this those are those are big questions

right so what did we find we found that there were actually quite a few limitations um that security cop while the claim can do a lot of things Microsoft says it does all the magic things for you um that for example it doesn't take any action on your environment um right now it's not set up to take any action on your environment which means that it'll triage for you it'll answer questions for you it'll uh tell you what it thinks is happening it'll tell you what it thinks you should do even but it will never take an action on your envirment and you can't even write an Automation and make it take action on your environment right like

you really cannot they really don't want you using co-pilot in an active way they want it to be a passive Observer they want it to be reading and reviewing and having a human um in the loop not on the loop right they want a human to be able to take action every single time and that's um I think probably a good thing uh in general uh I don't think that's a bad thing the AI models aren't as good as an analyst not as good as somebody making a judgment call on what the actual issue is but um that is a limitation in the sense of if you're tell if they're selling it as a 247

solution a solution that's going to replace your sock replace your 24-hour monitoring um and it can't take action in your environment um it leaves a big gap right it leaves a big gap in your after hours response leaves a big gap in your um automations and it leaves a big gap right so I guess the question like was it worth it for us uh I gave you bit of a spoiler earlier but we didn't renew our license right it was frankly pretty expensive um but how did we actually try to use it right and let me let me tell you a bit about how we tried to use it we we used it in House of course my team

used it directly but we also gave access to our sock right so our sock used it um quite a bit more than us because they they triage all their incidents they see a lot more incidents um and they said for them it was beneficial in the sense it was a little bit faster they were able to be a little bit more efficient they would maybe save instead of saying 20 minutes on triaging incident they would spend 15 10 minutes right so they'd say half the time 5 10 minutes an incident um which we're escalating about n incidents a day uh they're investigating about 20 of them right so if you're if you're looking at that yeah

it saves a few hours every day for your analysts which isn't which isn't nothing right it's not nothing um but for us we found we asked them what are you actually using it for right how does it save you that time what what is making it effective for you I said well really they're using it to right kql they're using it to say I need to search through my sim find these results or whatever find these how many times this user logged in those types of things right so they're using it to write queries that they would have had open in their notepad next to them all day on their second monitor or whatever um instead

now they're just typing it into into security copilot and it spits out the result instead right so uh yeah it saves a few minutes in terms of they're not copying pasting trying to fix the query a little bit change the user ID whatever right see in bit of times uh like that um but really otherwise and and you don't have to sanitize your data if you do want to ask large language model um for advice that was what the analyst found as well they didn't have to sanze they like that um but really they said beyond that there wasn't um for them as as big of a value because they said our anal do this every day not every client

for for for our stock had security coot and and we were early early access of course so most of their clients didn't have security cot so they said our anst are very used to writing kql they're very used to searching through our Sim already searching through our our our logs and analytics um they're pretty good at it right so when we heard that we said okay so really your Tong me saves you five minutes every incident but for every other client that doesn't do it you're doing the same work you would be doing anyways um you're not finding this like really huge Improvement and it doesn't really affect our escalation Times by for us we

thought a significant amount but okay we were like okay so we're not finding this huge value we're certainly not finding an and um I didn't load up the price page but it's $3,900 plus dollar per seu per month uh Canadian which works out to Just Around 12K once you add in taxes um if you pay taxes uh as an or um once you do that it's all about 12K a month just at the base level the three scus of what Microsoft recommends right to run to run this right so we're like is it worth $122,000 a month or or more I mean some organizations if you're a larger organization maybe you're buying half a million dollars in scus right like the

money adds up really fast uh and and we said we're probably not paying for we're probably not gain $122,000 a month worth of time out of this right because if you're talking $112,000 a month that's for most ORS probably two analysts right two two or more analysts right so you're probably going to be able to get more out of hiring two people than than running uh than running security cot right uh two people are going to be able to do a lot of things that that saves more than five minutes times 10 20 incidents a day um so that was a big consideration for us but we thought okay we we do like it though right we we use

it for things like if I have to write a report for example and then you have to write a lot of reports when you're in the government if they report I like to say like oh can you just summarize the incident for me and not having to sanitize the data or anything just say summarize the incident give me the template for putting this into a word doc and I'll just like copy paste it and and edit it and then I look it over that's something that actually saves me maybe 20 30 minutes every time I have read an intive report right so what what alternatives could exist right how could we do this in a more cost- effective way

what's what's a different way we could do this well it turns out obviously if you're using Sentinel if you're using a Sim they usually have some sort of automation system Sentinel uses uh what they call playbooks uh runbooks the logic app system is how they're integrated into Sentinel um and they have an Azure open AI uh endpoint right that that's built into Microsoft as well right um and you can go generate your open AI key you pay your 03 cents a token or whatever and you uh write your own Playbook and then you say okay so I know that I needed to be able to query the last five related logs the last 10 sign-ins for any user

um that's related to it anything like that right and then you go in and you generate your little logic app and you say okay go and get the all the alerts that are associated with this incident run the query um pull all the text Data out pump it into the Azure open AI endpoint uh respond and create a comment and Tack it onto your incident and then when I open my incidents now it says oh this incident had these things I recommend you do those things and um and I'll yeah there's a little screenshot there um of what that looks like ignore the button that I've take out but um the uh and I've sanitized

this data here but uh from an incident that happened today um it just says like look you can this incident is characterized by malicious email attachment that was quarantined by Defender the email was tagged as whatever and had this uh initial access category the IP adjust associated with had an abuse iptb score um this was a recipient this was the sender this was the IP address um here's the Playbook I think you should use here are my recommendations for containment um and I just addit it as a clment right so I just open it up and and I see that and then we we thought to ourselves okay so so let's compare this like home brew DIY

I hacked it together in a day solution um to what we're buying for that $12,000 plus dollars a month right and let's say so the home R solution is cheap it it's really not that difficult to implement if you're familiar with logic apps you're familiar with Sentinel and and and playbooks um but it's more limited in what it can do right it's not automatically writing all this qql and curing through your environment you have to do a lot of that yourself you have to be able to think about um what types of things you want to search for before you uh before as you write your playbook um and and it's still going to be usage

based pricing of course it's it's relatively cheap usage based pricing it's it's maybe hundreds of dollars a month depending on how many incidents you're pushing through it compared to tens of thousands um and if you really wanted to you could even have it take action environment right by writing the Playbook to take a response from open Ai and then and then have it do whatever you want right you can you can take that response and and add it as an action and have your PlayBook take that action right so you could have it take action in your environment of course all the caveats included be extremely careful um but let's compare that to to what we

were getting for this Enterprise level solution right so security Co what other the nice about that it's built in you don't have to write anything right it just it just works you click the link in it and it does the thing um it has some nice Integrations right and and actually uh Microsoft wrote a bunch of Integrations for playbooks even to to security Cil so you can you can write a Playbook and say send the output of this Playbook to copilot and then get the response back as text and you can do that in a Playbook um which is great but it's almost exactly the same as what I I I wrote uh on the open AI endpoint for

40 cents of requests instead of $1,000 a month right um it's easier to use I'll give to that like it's it's got this nice UI you just go in and you type in your inent ID and it even recommends playbooks for you all those all those things and it's fully supported right when we were onboarding to it uh we had time with Engineers we had time with uh Microsoft uh Professional Services and and they sat down and said like here are all the ways we recommend you do it here's like the repository we have of recommended scripts and playbooks and and here are the use cases we're seeing other clients use and like here's all

the things you can do and it's nice and big and supported um but it's also cost that they cost that money right and and it's got that consumption based seu pricing and and it can't take action on your environment right so so so we look at this and we said do we really need this in our environment um and would other people need this in environment and and and we have some colleagues in other jurisdictions and and other governments and organizations uh that we talked to who are also uh Early Access for security Co pilot right and one of our partner ORS told us that they loved it right they they they they're a small

team as well they I believe they have two analysts for security period um so they're smaller than us in a sense um and they said like they're an Enterprise SI work as well they're a similar number of seats to us they are a similar number of uh uh they have a larger security budget than us actually but they're um a similar number of teats to us and they've had a big incident a few years ago which leads to having a lar security budget than me um but uh what happened is they said they use it all the time right they they use it to constantly review every single incident they pump every single thing that goes into their

Sim through it um and and they spent all their time basically just running through the playbooks um and when you were a small small team with a large org with 10 departments 15 departments whatever when you're a small team with or um and you can't handle the load that you're having um maybe that's where it starts to make sense right maybe that's where something starts to make sense a slightly larger team not a lot larger just two three more people on on my team we were finding it's not giving us that benefit right so when you're only two people and you're trying to do vul management you're trying to do security awareness training you're trying to do

um patches maybe maybe you're you're hounding admins to do patch their things whatever right like you're you're trying to do everything with with with two admins um maybe having one person just in the morning just click through all the co-pilot buttons that's a huge win for them right uh but for us when we have a MDR service we have our team just goes in and as we get escalations we go in and we and we make those changes and we we do the type of work that needs to be done whether that's containment or that's a forensics whatever it is we just go in and do it as as as necessary um and and we always know that if

something like serious were to happen we could we could always bring in Professional Services right uh we we have that flexibility we say like you know what for us as a a slightly larger team who's doing slightly more things with slightly more people um we're finding less value to it right so I think the big question becomes how do you determine if this is valuable for your order right and and it's really hard now because Microsoft has put this really high cost of entry in right they've put in this recommended 12,000 really would be a minimum about $4,000 entry fee um if you bought one seu so they put this really high cost of entry

in um that now you have to think about is that something that we want a pilot for a month as an organization and I I I think if you were a security um Team lead and you're looking at your budget and you're saying can I burn 12 Grand in one month to Pilot co-pilot that's a pretty hard sell uh frankly for me right like if I looked at my budget and said I got a burn 12 Grand at to Pilot for one month um that's pretty hard to sell um but I I I think if I was a team where I said okay I've got that amount of money and I need to decide between hiring two

analysts or buying co-pilot I think it becomes pretty clear pretty quickly that the amount of work two analyst is going to get done is going to be more than what coil is going to do right is going to do for you but if you're an or that for some reason geographically like it's where we are it's really hard to hire um two people uh or or it's really hard to find two people that are willing to come work that are skilled uh and trained and you don't have to invest I don't know one two three years of training into um then maybe that that changes the equation too right um for us like I said

we chose not to Ren C right we chose not to because we thought frankly it just wasn't giving us enough of that value we were to find other products that filled the niche we were able to invest the money in other areas buy different software products byy different solutions um but um we do still continue to use S our Sim we do continue to use the the home brew Integrations we continue to use all those types of uh tools that that came with it um that we were seeing benefit from we were we still do the things that kopai was doing we just wrote them in house right we just did them in house and I've included

on the on the next page there G to be a link to the gist for you can find the logic app yourself so you can copy it and paste it into your uh into your Azure environment um but I'm not here to slag on on C I mean I I've been saying well we chose not to buy it's too expensive for like an hour now 40 minutes whatever it is but um I'm not here to say on clot I'm here to say that if you're an or who needs something off the shelf fast and needs to be able to improve your response times improve that and you have a very limited uh Team size that that is trying to do Everything at

Once which kind of feels like how our team is sometimes um then there there probably is going to be some value for you um but what I can't tell you today is or I couldn't tell you without like knowing the Deep details of your org is whether that value is worth it over trying to go do recruitment whether that value is worth it for you over trying to do something else uh with that money right um in terms of whether uh I think that the future of security Co pilot is going to do more things right like they keep saying Microsoft keeps saying it's going to replace your sock it's going to replace your sock that's what they

that's what they were pitching us on during Early Access that's what they pitched us on when they tried when we tried to renew it they said if you keep buying it right now it's going to replace your sock in a year it's going to replace your sock in two years it's going to replace your sock that's what they were telling us when we when we uh early accessed it that's what they pitched us on when we tried to renew it and I'm here to tell you I frankly don't think it's going to replace your sock uh I don't think um AI is going to replace analyst I don't think AI is going to replace your Juniors because um even if

it could do and it does in in in some ways and in many ways it does do the equivalent work of a lot of the Juniors um what it's not going to replace is that you're going to need those Juniors to turn into seniors at some point right you're going to need those people to continue to grow in your organization so that they can go and lead a security team in the future you're going to need someone that would hopefully you retire they're going to be able to run the org and and and do the security operations right so if if you rely too heavily on these types of tools and and if the industry in general relies to heav on

these types of tools we're going to run into a problem where we have this complete skills gap between um Juniors and seniors and we're going to run into a situation where it becomes almost impossible to to to fill that Niche right um and when we look at uh these products and and what they're siging to you I I I also just don't think that we're ever going to really trust as as an as analysts as security operations people um I don't think we're ever going to trust the AI to take action in our environments right like uh I know the automations I write myself lock me out of my systems constantly because I do something stupid or I do something silly

that one of my automations picks up and now my account's locked I have to go call someone else to unlock me or whatever right like I I do that constantly already um now you throw AI into the mix uh and it's going to be taking a ton of actions that you don't really understand why they're being taken you don't really understand what the decision logic is going on in it uh and when you have to explain to your insurance when you have to explain to your to your sea level person why this decision was made to lock out another sea level at 2: a.m. in the morning when they were working in China and trying to

ask us an email that was essential for that business um when you're trying to make that uh justification my laptop goes to sleep of course but when you're trying to make that justification um then you're going to have to say well we pumped it into an AI model and the AI model decided that your business meeting in Asia wasn't as important as we thought you might be compromised even though we knew you were in Asia right like that's a decision you have to be able to answer when you write yourn report that's decision that you have to be answer as a security person and it's going to become more and more difficult as we rely more and more on

these AI tools so I think that's um going to be a big problem um there is a bit of a QR code all the caveats about scanning unknown QR codes apply uh I promise you I'm probably not doing anything bad with your data trust me don't trust me I wouldn't um but the this will be published as well uh we'll be sending the slides out uh for all the attendees um there is uh a guide that I followed or I saw that was um last year somebody had integrated open AI into their senal environment as well I I linked that there and then I wrote I've linked my code as well and and and my

website um but there I think there's about 10 minutes left if anybody's got any questions I'm happy to take some or or comments or concerns or or just want to tell me that I'm a jerk I don't know whatever uh feel free you're a jerk no I I just wanted to comment that uh where I see a lot of value is in the natural language processing for queries because not every body's aware of like a particular product Syntax for doing a query or anything like that so if you want some Junior person or somebody who's not always security you need you know extra hands on deck they can ask when was the last time this user logged

in without having to figure out you know some weird you know SQL esque kind of thing also good if you have like uh outside parties coming in to help with an incident or something like that they don't have to know your exact product they can just natural language yeah I think that's the that's what the sock was saying that they found was easier for for their process is that they didn't have to learn our tables and learn how our Sim was ingesting logs um but we were also still finding of course if you're just doing a when was this person logged into entra like a very simple one if you just type that in the

regular chat GPT um it'll give you the same answer you just have to copy and paste the username in yourself after right so so it really depends on how complex is that query going to be and how complex do you think co-pilot's capable of the questions uh this is just about like what the value of the product actually is and like from your perspective having actually used it what was like the major significant difference between the free to use language models and then what would you value the cost at that you would pay for for your organization like is it $2,000 a month is it because you said it's 12K a month which is significant yeah so um that's a that's

pretty good question um in terms of the the difference between public models and and and private models like like the one that Microsoft is doing is that theirs is very tuned and like they have really good uh prompts to be able to say like how do you work through a Microsoft environment right like how do you look at kql how do you look at tables um and they they've designed that already so there's no fangling around with your public model being like oh that kql didn't work can you change it in whatever way right like that's the that's the big advantage of using the private model that Microsoft developed um in terms of what the value of that is

compared to a public model I I think it becomes a really hard question because right now you can get chpd Pro for I know 25 bucks us or whatever um you can get a bunch of other models for around the same price right um some that people have tuned for security purposes as well um and when you consider that and you think about that you go well okay so if I'm per analyst paying $25 a month for a public model that they have to go and spend five extra minutes to sanitize value your analyst time at I don't know whatever your 100 bucks an hour whatever right say let's say just to make it easy

like you're talking only a few hundred doar right you're not actually talking that much money in terms of the time you're saving with 5 minutes per incident 10 minutes per incident right so if it's 500 bucks an analyst let say a month like if you're going to have to look at how these models going to be used in the future and and and what they're going to cost I think that the the current model of what they're trying to sell us isn't sustainable uh for most organizations oh wait down

there um do you think that uh co-pilot could take somebody who's operating at a very low analyst level and bring them up a couple of notches yes the short answer is yes right um the the short answer is yes um the longer answer is um does it do the things you want in that sense though right like does it teach the analyst how to be that Junior to a mid-grade higher level analyst does it actually teach them anything about that or are they just pumping in semantic queries and asking it questions all day and and depending on it and learning to depend on on this model right so if the question is does it let them operate at

a higher level absolutely right it's faster it's more efficient it's it gives them access to things like they may not know kql and now they don't have to know kql right it giv them access to those things but are they going to learn kql so that they can write their own analytic rule in a year no they're not going to do that either because they're they're they're depending on the model right so it does make you more effective um but there are limitations in terms of it's going to cost you that in employee training employee growth and and and all those things that you're going to have to invest in anyways right so now you're

paying for extra Microsoft training you're paying for extra time off whatever to go to those training certifications um so they can learn those type of things as

well great presentation Thomas uh pran here just a quick one in general not in particular to your POC on the co-pilot what kind of skill sets would you see in future which the sock analyst would have to learn to adopt these kind of Technologies because at some point there'll be a critical m and these technologies will creep in into sock or other areas so what would it be prompt engineering would it be learning the models would it be more understanding how to write the right queries to get the right results so what in your opinion would be the right skill set to learn I well that's a really deep question um I think cret engineering is

for sure going to be a big part of it you have to understand like what do I ask the model and what's going to give me a good response um that that is something that your analysis always going to need to know um but I think the bigger question around that is actually going to be do your analy and your secur human General your security as an organization in general um understand what is happening behind the scenes in a sense or what what is the limitation of the model right so as this as the technology becomes more widely adopted as it becomes more widely available and and we are approaching that like critical mass level um your organization

your team is going to need to understand what it can't do for you uh more than what it can do for you because it's going to be able to do all these amazing things that they claim it's going to do um but you also have to understand where it's going to fail where it's going to accidentally lock someone out at 2 a.m. or whatever and and and that's the thing that I think is going to be most important for your analyst in the future does it give any critical information decision making information for example you oh sorry uh for example of server logs uh you feed the server locks and can you tell which vulnerability it

exploited what was the payload and how what happened before that and what happened after that yeah so so the question is can can it tell you like if it if you're feeding your server logs into it can it tell you um critical information like what happened what what happened before what happened after all those types of things um and the short answer is yes the longer answer is it's it's not always going to be 100% reliable right you can't you can't depend on it you have still have to review its work um for example we're going to see things where like we did see things where for example said oh we watched the Powershell script execute on

the device and it said here's the username here's um the connections we saw before here's connections we saw after whatever right um but it was completely mistaken on what the script did right um it it didn't really understand what the context of that was in that case actually it was Defender running it's like onboarding script um and it was like oh all these really weird enumerations are happening this environment is being enumerated it's checking who the admins are it's checking uh what SMB Shares are attached but it was the defender onboarding script right so it can tell you these really scary things that are all factually accurate but it doesn't have the context information about your

environment and that is context in the sense of like what you know about how you're running your environment and that's a part where the models are still very limited got one more here I uh do you have any sense of like if future versions of uh co-pilot would automate response or would that be too much like exposure in terms of like liability given like open AI is like unsupervised um Microsoft when we were working with them basically told us not to do that um they they actually explicitly told us they don't recommend that uh in their in their automations to to directly allow open AI to take action in your REM um they say again that it

will replace your sock in however many years right so I'm sure someone's trying to work towards that uh I don't think it's coming anytime soon much to the uh uh pleasure of the various sock analysts that are going to maintain their job so thank you very much Thomas um y maybe you got and you're you're hanging around for a bit to so it'll be around for other questions out in the hall