Automating Analysis Without Automating the Analyst

I'm just a simple five-step process to automation and that process is this first we identify the opportunity right it seems kind of like a no-brainer you find out what the opportunities are for automation what the big problems are the big challenges and what you need to kind of attack first you take that workflow where that task and you deconstruct it into its various subtasks or sub parts you consider the return on your investment for automating each of those individual subtasks you think about what kind of automation you're going to have to apply to that problem and then finally you measure what you've done you evaluate the impact of that automation and what we're going for here

is measurable improvement right we don't want to go back at the end of the day or week or month or year and say we built this huge awesome widget and it's super cool and you know it exists and that's basically all you can say about it right we want measurable improvement kind of calling back to the keynote from earlier this morning we're gonna be able to speak the right language right we don't want to be building widgets or scripts or programs just for the sake of doing it when you be able to speak the language to people who are not in the sock or on your team or in your group so they understand you know what they're

getting back for their investment in your time and the capability they're getting just by way of a disclaimer you know I think for many of you who raised your hands a few minutes ago who are kind of doing automation today whether that's kind of like a micro automation where maybe you've written a script that goes and pulls some sort of information you know corollary information or gets you context or does some research for you or maybe you've got you know automated email generation those kinds of sort of micro automations they may not require some big five step process or some detailed analysis to execute right you might just be whipping that was out you know in an afternoon and

that's fine that's a big part of the job that's part of what makes it fun but for a larger more complicated initiatives particularly with kind of more complex tool sets we get into like machine learning for example those types of things where you're making a significant investment in time and effort those things really do require more deliberate analysis and planning and so that's kind of what we're gonna focus on and try to exemplify today in our talk okay so let's talk about this this five-step process and kind of how you walk through it identifying the opportunity requires something that is horrifying - I think most of us which is we're going to have to go talk to people have

conversations and you know if the team that you're helping whether it's your own team or another team if they're not bought into what you're doing they're not going to use it they might use it for a little while and then let it go by the wayside and all your effort is wasted right so it's really important to take that first step of gathering requirements talking to your users making sure that you're identifying things that are going to help them in their day-to-day and use that as an input to the automation process the second thing you're gonna do is deconstruct that workflow into various subtasks and need to ask yourself some questions about those subtasks right

questions like is the tasks repetitive or is it variable is it something that's gonna happen the same way every single time or is it something that's gonna change depending on the situation or certain inputs or certain scenarios is the task independent or is it interactive so is it something that I can just trigger a process and just let it go and give me the output or get to a certain end state that I desire or is it something that's going to require some user inputs perhaps some interaction in order to kind of decide what tack to take or what actions to take what information to get or tasks to perform and then finally I think this is

particularly relevant in defense is it verbal or is it electronic in terms of orientation of the tasks so does it require you interacting and talking to another group or a customer or a user or is it something that is you know fully kind of programmatic can just happen without any kind of back-and-forth interaction so these factors are really going to influence whether or not a task is even appropriate for automation and it's going to impact how difficult the task is to automate and where and your priorities when we talk about measuring automation I think this is where a lot of us kind of tend to have challenges because it's something that if you don't do it correctly it can be

arbitrary and it can be very subjective I can say that hey I wrote this script and it helps me do this thing a whole lot faster right but I think what we've seen is that that's that's maybe not always the case right some of that can can vary from individual to individual so you can't really just measure things based on you know time savings you need to really look at the tasks and think about what improving that task really gets you so let's step away from the automation part of it and just talk about improving a task because really that's what this is is all about at the end of the day right improving the tasks that we're doing and

it's really kind of a scale if you think about it in terms of you know very minimally improving a task can have negative value which despite the use of the term negative it's not a bad thing it just means you can improve a task so much but once you improve the thing there's really not anything else you can do with that so in an Operations context an example might be doing Quality Assurance on a ticket that you would open right or a case that you open making sure it's complete making sure it's accurate you know that task has relatively negative value because once I do those things there's not a whole lot more I'm gonna

do to improve a ticket being complete and accurate either it is or it is it isn't going up the scale a task improvement can have constant value it can have incremental value this really just describes the amount of value that I can get from improving a task more than one time so for example if I'm getting better visibility in my environment if I'm a an analyst a sake analyst every time I improve my visibility I'm gonna get more value out of it and really that's kind of a sliding scale that continues to go up over time I always want to be making those improvements right I'm always going to get value out of expanding and extending

my visibility you know intelligence is another example if I'm getting more accurate better more specific actionable intelligence you know every time I add to that or improve that process I'm probably gonna get more value out of that and then kind of at the top of the scale is this exponential value where maybe I'm detecting you know something previously unknown or something that's really sophisticated you know and I'm that's exponential value because every time I make a change there I'm gonna be doing something that you know is rare or enables some kind of creative performance so that that's kind of the sliding scale and we talk about return on improved performance that we get through automation that's how we

measure and the relative value of the improvements were making not all automation is equal not all automation is the same so in the case studies that Brandon's going to talk about in a few minutes we're going to be focused more on robotic process automation which is kind of the more standard like programmatic automation so you're writing a script or you're doing a custom integration or you're writing a program or using a platform to automate a set of tasks but there's also cognitive automation where it requires user input to determine kind of which path you're going to go down and what automation you're going to do there's collaborative social kind of robotic automation which is more collaborative

intelligence artificial intelligence where the entire process might change depending on you know certain variables or certain imports over time right again today we're gonna be focused mostly on robotic process automation I think that's probably what most of us deal with day-to-day and the kind of automation that we're working with and then finally we're all about results right so we want to measure the impacts of the automation are we substituting a series of tasks in an automated way for something a human normally does manually are we oughtn't accumulate you know adding more information you know adding more value to that set of tasks are we saving time so that new work and new value can be created right we're getting

things off of our plates so we can focus on other things maybe that are more valuable or that are more suited to humans doing the task first is doing them programmatically right so that kind of summarizes that that five-step process and how we step through that so now I'm going to hand it over to Brandon he's going to walk through some examples of applying this process in an operational environment Thanks so we we have two case studies and we kind of chose two case studies based off of where our analysts felts actually we got the most impact and in return as well as to make a little bit more relatable so our first one is alert

context and again we we want to talk to our analysts we want to figure out what what is tedious in their daily lives what do they just absolutely hate having to do every single time as well as where can we potentially make the best improvement so we found that it was tedious and time-consuming you know we this happens almost every single time you have to triage an alert you have to go do these steps you have to find your historical context learn from yourself unfortunately many analysts will forget that process until they go down the deep dark rabbit hole and then all of a sudden realize look that got written up about four hours ago

and now all that time is wasted so and we need to gather identity information so who is this who owns this device what type of device is it can we even determine whether this threat is even applicable and also find out related events so when working with our analyst we kind of came up with a high-level workflow and it's intended to you know to be a little bit more simplistic and everyone's gonna be a little bit different they saw for your processes based off of you the tools that you have and how your analysts work within your environment this is what we had we went to our analysts and we were able to identify seven different tasks out of

that entire workflow that we could chunk down and make more manageable so we could start with just these ones determine what their potential impact would be what our return would be and then start with those first before we look at this entire process and just think let's just go create an entire automation for everything where you try to do too much too fast so walking through the first one it's a repetitive independent electronic type of task where we're doing the we're pulling the similar information we're basically just changing the metadata that we're searching and our next task is basically a way for us to potentially escape the entire rest of that data collection process right so

we when we working with our analysts we actually even had to reorganize this a little bit to make this most sense where we can get the biggest bang for our buck where if this was true where we've already seen this there's enough fields to say that we've already seen this we've already termed as a false positive or we've already taken action on this we can skip the entire rest of the data collection process save ourselves some automation resources as well as save us of our own human resources because you know automation resources still something we have to take in consideration when we're doing automation because it still has a cost so that wasn't is variable independent

electronic working down through the rest of it where we're trying to identify the device owner who who's affected as well as what are they potentially vulnerable to gather that information so we can provide it to our analysts which kind of leads into the next part right so where we have a is this threat even applicable do we have an attack against a system that's not vulnerable for it or do we have an attack against the system that's not even applicable again try to escape that process as soon as possible so we can try to save ourselves some time on the back end and then now we have our you know typical we gather a log data

from the Sam or log store and we determine we that's when we start to to actually do some of that analysis so you can see we've tried to front-load a lot of what we can and push the analysis a little bit further down so now we can can we determine some false positive what do we need to do with it before we actually go to the hosts and start gathering logs memory captures or whatever the case may be to help help perform the completion of that analysis so we're able to identify four tasks that are really good for automation the repetitive they're independent and electronic we can potentially take these tasks and code them out and provide a

lot more value to our analysts and then we have one where we need to put a little bit more rules bounds some logic around that before we continue with it just to make sure that we don't automate something that we're not supposed to and always err on the side of let's send it through the rest of the workflow just in case because we don't want to make that mistake where we're trying to do too much and our human breakpoints and this one's really really important to identify all of your human break points where it's absolutely necessary within your process because that's the whole point we're trying to provide the information for the analysts to make the decision there

are some cases where you know we start talking with the analysts that they're pretty much making the same decision based off the same set of results you can try to automate that portion of it this is where we really determine that we need to actually stop at that point and let the human make the decision so we've broken down all of our tasks we have seven different tasks that we might be able to automate and we want to start adding information and determining our potential ROI P on it to start helping prioritize of where we want to start so we have our historical context you know it allows us to do a little more timely

response the biggest one is allows us to reduce duplication of effort right and unfortunately you know it does require some of a P access to some of our tools so it's important to capture those dependencies so you know you might have to go grab those dependencies and solve those first before you can start on this and it helps some with your prioritization have we observed this before if you know the dependency is we have to have task one done first so if we go down our automation process this has to be done before we even consider the rest of it it allows us to be a little more efficient and reduce some of that duplication of effort again device

identity as you know kind of pre talked about a little bit is it's basically a way for us to gather information on what actually who's actually potentially affected and again we require some tool access and when we were talking with our analysts and again you know this is gonna be different for everybody depending on the tool set you have how many logins you have sometimes do login a single tool you gotta log into your SSO and then you got to log into the tool itself in some cases so this was determined high difficulty based on the tool sets they had to access is a threat applicable and this is our human breakpoint right so we have to make sure

that everything up until task 3 is completed before we even consider this but again this is one of those things where we have to make sure that the human is actually inserted into that process at that point then we want to gather a log data again we need some tool a PI accesses we get some incremental value some more context makes us allows us to make more efficient and consistent decisions based off of the data and the information that we're getting and I think the big one for this one to point out is that we're getting consistent data set for analysis so analysts will typically query a sim in three or four or five different ways

across your team this will allow you to present information to your analysts in a clear and effective way where it's consistent through every single alert versus who knows what they ran and if they self to gather information that's you know so be it and then is it false positive against us one of our human break points but we identified you know before we get to this point we need sufficient context analysis before we can continue down our workflow then our lot and lastly we have our you know gathering the host data you know typically this is a little bit more intrusive because we have to utilize an endpoint tool or scripts or some other fashion of gathering that information

and for us it was a little bit more difficult because we had to actually execute some of those scripts and some of those and access some of those tools and sometimes it's not as clear as it should be so we break it down you know one by one and I and we can kind of start to see that there's a lot of information that goes into this there's a lot of things to consider there's a lot of things that kind of make this seemingly simple process relatively complex so this is one of the reason I'm Airi reasons why you want to kind of go through this exercise a little bit so we can start to determine

where do we want to start and where are we going to get the biggest return as well as you know think about your analysts what's gonna make them happy as well because you know as Mark mentioned a little earlier we need buy-in if they don't like the tool how can you use the tool at the end of the day and we have to take that into consideration that we're doing something for them and not saying here's a finished product enjoy right so we're able to determine historical context you know kind of going back to us saying a little before was learned from ourselves if we've our done the work why do we do it again and

unfortunately this happens all too often an illness process where you get so far down there because it's something cool you really want to look at it but then you realize someone else already did it or we already had information on it and you just forgot to go get it so that's where we get the biggest return is we can potentially exit our process as quickly as possible save on human resources save on automation resources and we have you know the rest of them that we put down and lists so you know device identity gathering our log data having observed some four and then gathering our host data so now we have identify we kind of

go back to our graph here of a workflow we've identified our seven tasks and then we've identified down from that five tasks that we can actually automate and we've broken apart we've determined where we want to start and now we want to start determining how are we actually going to automate that so there's the four tasks out of those five that we can use robotic process automation on so we can put these through basically feed it data and we get data out we don't necessarily have to do too much logic or too much things in between there and it makes makes it easy and then we have our cognitive automation where we're again we'll probably to talk a little bit more

to our analysts figure out how are we what kind of logic what kind of bounds rules do we need to put in place again with erring on the side of being safe of always sending it back through the workflow to make sure we're not saying something's it's all false positive from previously when it really wasn't and this one kind of this leads into like the next the next step is what is our impact of automation how what is the return we're getting you know we need to be able to present this in a manner that is ingestible that is we have to speak the same language so we can condense down our big workflow into basically

some some simple steps where we've taken all of our data collection all of it assumedly automated is now in one step and then we have our cognitive automation under Horst Oracle context we still want to take that in consideration we still have that breakpoint in there to again exit the process as soon as can and then we have our analysis or our human has to do some of the review of that data and analyze the actual data there and we can see how much some more simple that process becomes with implementing some of this automation now we have to assess and make actually sure that we're actually getting what we expect from the actual automation that

we're putting in place talk to your analysts figure out how long it takes them to complete that task and kind of gets an idea so we can start putting some metrics in place of you know again make sure that we're actually getting some of that what we expected automation and one of the things I think you know tend to get forgotten is you know what what augments of a human can would do where if we're providing all this context up front they don't have to go gather from the tool not only are we saving that time of them gathering it but we're also helping them give that information so they don't have to go see

five or six different tools and correlate themselves or gather that information dump it onto a CSV and like make pretty pivot tables and add more information and context later right we're giving that information to the analyst ingestible format so that's going to save them time on just performing that analysis in general so reducing time and effort and they want to know we're reducing the analyst fatigue we're saving the time that they have to go and find that information and see me a lot easier for them to work through that workflow so kind of is a little bit bonus I kind of wanted to show what we what we were able to do and how we kind of present it to the analyst

a little bit we're talking about a lot of data ingestion we're talking about a lot of things and you still have to consider what this actually looks like a little bit so kind of quickly walk through you know this is where we kind of is where we end up storing our historical context and the various different tabs and we wanted to summarize it there on the front for the analyst again summarizing as much information as possible from our automation so they can see it straight up front and then our event context and the other is a thread actually applicable and so they can make a quick decision depending on what the actual type of work that they're working

through it's kind of an example pointing back to how me how we feel that alert context and what you've already done is so critical where we can see in the beginning we had a not blocked trigger artis is blocked Higgins azim you see at the top and it was already remediated same IP and then we can also see that we have an assigned alert that came in a little bit before us and that one has the same W script message line there it hasn't been resolved yet so we need to work with that analyst so we're not working in a bubble right so we can reduce that effort we don't want him to perform the analysis and then us to

perform the analysis and then now we're duplicating that effort and then same thing unfortunately it happens all too often in the real world where we go to ask have a email removed because we found something malicious unfortunately doesn't take you know takes more than 5 or 10 minutes in some cases and it already clicked the link but I tell me have it removed well we get out there all that information in that context straight up front so the analyst can move forward with their investigation so our second case study is run intelligence research you know this is again something extremely common that all of us have dealt with if you're if you've been in the analyst world or in a

sock and it kind of say similar thing to what the alert context was where you know it's a similar step so we're performing every single time we're going through the same set of open source tools or paid subscriptions we're dumping in an IP we're trying to find the same similar data coming back we're trying to gather the context of that indicator of compromise we are trying to determine what is that threat you know maybe some attribution in there if we can get it or in big one is you know what is the capabilities so what what is it capable of doing to the affected user stealing passwords so on and so forth so again you know everyone's process is

going to look a little bit different this is what we what we call non you together with our analysts a semi high-level so that way we could start can deconstructing the tasks and determining where do we want to start with our automation and make it manageable so we were able to determine we had six potential steps that had that fell into the fell into potential automation with our first task wheel determinist repetitive independent electronic you know this is gathering if you have process intelligence if you have an intelligence team or if you have an IOC database you know and the other one here out to the left is gonna be things you've already researched why

burn your API three or four or five more times if you already have that data cached from five ten minutes ago maybe an hour ago search that data first find out make sure that you search it and he could save yourself some API calls and then you know his context actually true we actually find anything if anything useful and then this is where our first analyst breakpoint was where we had to actually analyze the data we got back is this is the annal is the data that we pulled back sufficient enough to continue on with the analysis process or do we need to go gather some more data from open source and some of our paid

subscriptions and it's important to keep in mind this this process here is basically we broke it down to be one single IOC that goes through this each and every time and we have to iterate through this process every time and the idea is is that we're gathering from our paid subscriptions from our open source and though even those have some limitations on what what is this thread about and then a lot down at the bottom when we started talking to analysts you know it's a natural progression to when you find an ayah when you find an item see you go search for it and then you find maybe five hashes or five domains associated with that and then you go and

search those and you keep going down that rabbit hole and that's where a lot of times we get lost and when we put this specifically in here to address some of that was that we wanted to is sufficient information there did we go sufficiently deep notice do we have three resources that say it's all this same malware and it's all malicious and all have similar risk scores there's three separate ones they're not referencing each other okay we can move on and not burn the rest of our API calls as necessary so when we're finding some of the related infrastructure we can ensure that we're being as efficient as possible and then lastly we have our

you know our actual analysis the is it a false positive do we need to continue on with it can we go ahead and exit our process so as we did before we want to break down and add in all of that information to help prioritize where we will actually want to start so here you know we determined it was an incremental value we still have to have API access and we still have to have some of that historical and processed intelligence in some way shape or form you know its context found with we have dependency on the tax on task one for that one you know it's a fishing context found and this is going to be where analyst

actually has to step in and it's a break point where we have to make sure before we go burn our api's before we go run all that information do we actually need to do it do we have enough information that we that we can save ourselves on that and then gather in the actual context you know this does require some subscription API is if you if you're if you have those and free and open source API is there's a ton of open source stuff it's just a lot of times you have to hit two or three of them instead of just the one one or two and this allows us to you know the the threat context

allows more informed responses and it reduces the tools the analyst would actually have to access you know sometimes it's five or ten and sometimes have to use that one that they only use here and there and then it's a sufficient death check kind of a little bit of what as mentioned before was allows us to extrapolate on indicator relationships and hopefully go down deep enough in our particular workflow that we worked on we found that to you typically only need to go another layer deep to actually determine is this is this actually malicious so you can start that response process and then checking if false positive is our last step where the actual analysis comes into play and

again you know it's it's seemingly a simple process but as you start to put all this extra stuff into it to help you prioritize it becomes a little bit more complicated to know where do where do you actually want to start so we were able to determine you know our analysts actually kind of really wanted to gather the context and indicators but that always comes at a resource cost so something that we had to taking consideration where we ended up gathering our gnome context first again why search for something if you already have it in your database from an hour or two ago or maybe a little bit longer you have to kind of set that threshold for

whatever you're comfortable with you know more context allows us to make the better decisions and remove some of the variability so as we did before we kind of wanted to I'd start to identify what type of automation we want to put in place for these particular tasks and we identified these particular tasks we can do some robotic automation basically static we can feed the same thing in get the same thing out and then give that to the analyst and then we need to put a little bit of bounds around this you know if - two layers deep isn't deep enough for what you're looking for then you might want to put in maybe even some

logic around that to determine I want to go one layer deeper based upon these five five rules or five pieces of logic and then we try to simplify it so we can see start to see where what kind of value we can actually get of this what the actual impact is so you know taking in consideration everything that we want to automate we can throw basically all the data collection again into one little bubble and with that depth check to make sure and with that human stop point of sufficient context and then you know so we talked again with our analysts you know when we started going through this process to determine how much time they

spend on this so we can actually add some metrics to it and we you know reduces some effort and reduces our time to respond to threats because we can find out instead of an analyst taking fifteen or twenty minutes to perform a task after they get their hands on an alert they have that data there waiting for them and they can potentially respond to that threat quicker than is if they had to go and do the research themselves so reduce it reduces our time there and again just kind of a little bit of example of what that might look like to an analyst and how powerful this information can really kind of be so we'll dump are known context again so on

to the other related records and we're going to be focusing more on the Oh scent that we're getting from our various different tasks and the tools that we have access to so it's a lot of information to kind of look at at first but one of the things that we quickly discovered that I was actually extremely beneficial to our analyst so it's kind of just grouping things into various different columns right and key into our key fields so as we can see here you know we have a couple different i/o sees that came back as trick bot in this particular case but we still want to look into those and just you know as an

analyst just to make sure we got plenty of plenty of hashes to go research if we wanted to research and you know we've identified that it's certainly these indicators are certainly suspicious if not definitely malicious so it gives us an opportunity to quickly identify this and start the response process as we dig in deeper and with that so you know these workflows and these tasks may or may not be representative of the kind of work that you all are doing on a daily basis but you know I think some some common pitfalls that we've run into aside from some of the ones that Brandon mentioned during the case studies you know you go through the trouble of

applying some automation you know writing a script creating you know a custom app or a custom integration and a lot of times you know you go through that entire exercise you think you know what you're doing you think you know what kind of impact it's gonna have only to find after the fact that it's okay well the team's not really using it or they don't use it every time sometimes they just go you know do do it manually so it only has you know partial kind of value add or you know it really doesn't provide the kind of return on investment that we expected to it doesn't help us do this one thing faster or more

effectively or more consistently and in those cases if you're not doing some analysis upfront and you're not breaking it down kind of into its sub parts and and doing some of that legwork and some of that homework before you undertake that initiative it can be really hard to go back through and say well why is that right why isn't the team adopting it the way that we anticipated they would why is it not consistently doing the thing that we expected it to do so hopefully some of the the case studies Illustrated kind of the value in doing that analysis and doing those breakdowns ahead of time so that if you need to measure and adjust

later it's a little bit easier to do as opposed to just creating you know some black box if any of you have done any work with like artificial intelligence there's this concept of explained ability right and transparency people have to understand what the thing is doing and how it's working and that plays into not only kind of managing that automation or managing that that application on an ongoing basis but also training your staff training the users to understand you know when they click this button or you know when they execute that that automated task what actually is happening under the hood so they can give you the kind of detailed feedback you're gonna need to make sure

that you're enabling them another kind of risk in doing any kind of automation that that people don't often think about is aside from the upfront work of actually building the thing there's that ongoing engineering support and maintenance right so I'm sure many of us have things in our organization that some really smart person built or did and now they're long gone and you just kind of don't touch it and stay away from it and you know don't ask too many questions about how it works it just kind of does right so you got to think about like that ongoing ownership and maintenance and also I know no one in this room would ever do this but

technical people sometimes can like go down really deep rabbit hole right we can like find a cool problem or like a cool platform and we can take like four times as much you know time as we anticipated kind of doing that so again part of the reason for doing some of this analysis and breaking down some of these tasks this way and looking at the problems this way is to avoid doing too much and we'll avoid trying to take on kind of too much and letting you know perfect be the enemy of good when all you're doing is trying to add incremental value so hopefully some takeaways from this talk again you know automation should be a process it should

be a way that you improve value it should not be a prod that you buy and Jam into your environment to do a thing you know maybe a product as part of your strategy and automating maybe it's part of that process but it's not kind of the end in and of itself repeatable agile processes like I'm a huge fan Brandon's a huge fan you know they help us avoid those rabbit holes they help us make sure that we're getting return on our investment whether that's investment in a paid service that we're using where we have to be judicious about how we you know hit those api's whether it's return on people investment that we're getting out

of the the folks that were relying on to build some of these integrations priorities are really important not all improvements provide equal value so sometimes we get so wrapped up in understanding that you know we're doing a cool thing or we'll but we're building a widget that's gonna do you know this automation that we weren't doing before we don't always ask the hard questions about you know is that gonna be more valuable than focusing on you know a different widget or focusing on just you know handling this task manually for now because we have other priorities right so so priorities are important one thing that you know if you wanted to leave here this weekend and go back into work

on Monday morning you know and try to apply some of this again it doesn't take some sort of formal rigorous process to start to ask some of these questions and start to apply some of this stuff you know take any automation initiative either that you already have operationally in your environment or that you're thinking about or planning or working on try to to go through this exercise try to go through those steps try to quantify kind of the value that improving you know whatever that task has for the organization try to break it down into its subtasks try to really understand you know and make sure that that all that stuff is aligned right see

if maybe there's something that's been missed or overlooked you know and that's kind of even just a mental exercise that you can go through and apply some of these concepts so that takes us to the end of our talk thank you very much both Brandon and I are on Twitter Brandon's a lurker mostly but you can find us both there and with that happy to answer any questions or talk to you after but thank you [Applause]

I think we're done within five minutes we're closed yeah did you see the guy with the sign