Automating Analysis Without Automating the Analyst

besides DC would like to thank all of our sponsors and a special thank you to all of our speakers volunteers and organizers all right good morning everyone thank you very much for coming out today again my name is Mark Orlando this is Brendon Denker today we're talking about automating analysis without automating the analysts just by way of background and introduction both Brandon and I have an Operations background working in around building security operations teams doing managed services doing internal security teams threat hunting malware analysis forensics automation if it happens in a sock chances are we've done it at one point or another in our career and we wanted to give this talk today because I think we're getting to a

point now where there's a misconception that automation particularly in the defensive context is a product or it's a specific initiative and I think both of us firmly believe that it is a process and should be a process to be applied in solving problems of all kinds and increasing the value that you can generate for your organization or your team so that's what we're gonna talk about today we're going to talk about kind of a repeatable process that will enable you to do that but before we get into that you know I wanted to kind of step back a little bit and and talk about sort of how we got here and I refer to it as the automation

imperative you know why is it so important and I think for anyone who's doing this kind of work whether it's defensive or offensive today understands that you have to have some level of automation you can't do everything yourself and I think back when you know socks were really coming into vogue the the rallying cry was weed all the data right whatever you have any log you can possibly generate we need to get it and today you know I think it's a very different story every is online now so many more business processes are electronic or app driven and there's so much data obviously it's very difficult to make any kind of sensitive of it or make use of it

without some level of automation I think most of us will accept that so now you know we're standing particularly on the defensive side standing atop this mountain very very important things very very important data we have all these things particularly in the stock that we just have to do right you've got to do monitoring you've got to do analysis you've got to do instant response probably has to do you know some level of malware analysis file analysis and oh by the way you know answer the phones and answer the emails and whatever else happens right so suffice to say we do have an imperative now I think to automate what we're doing in order to

scale in order to be effective okay so that being said I think we're still maybe a few years back in our history lesson now so aren't we for the most part automating today on some level how many of you in the audience in your day-to-day work are doing some sort of automation or taking advantage of an automated task of some sort most of us right great so so why are we up here talking about you know the need for automation and the process to do it and I included this quote from Harvard Business Review article and really what it's saying is you know part of the problem with getting the value may be that were not quite getting out of

automation at this point is that we tend to mechanize the old way of doing things and I know particularly on the defensive side again something that I've seen is that we're doing a lot of automation of old kind of broken processes we're just doing them faster right but we're not necessarily adding a lot of value we're just looking to say well we've always done a and then B and then C so now let me go buy something or let me go code something that's just gonna do a B and C you know faster maybe without me hitting a button to do B and C right that doesn't necessarily get you where you need to go and I think the key

message there is that all automation is not equal just because you can do something in an automated way or do it faster doesn't mean you're gonna be generating value doesn't mean it's worth the time and the effort to put in so you know for example automating some tasks might reduce risk right they might help you produce some more quality products they might reduce variance and make things more consistent but again that doesn't mean that all those things are providing equal value to your team or your organization so over the next couple of minutes and and few slides I'm gonna talk a little bit about this kind of repeatable approach to identifying tasks to be automated to

prioritizing them to executing that automation as for measuring what you're doing in automating a given task Brandon's gonna come up and talk through some scenarios some case studies that that he's been through where these processes have been successfully applied and we'll be drawing quite a bit especially on the process side from this book reinventing jobs which really is a it's a text about you know all kinds of work automation but I think it's particularly relevant to the work that we all do on a daily basis so I just wanted to make sure I'm referencing that and and process-wise we're really talking about kind of just a simple five-step process to automation and that process is this first we identify the

opportunity right it seems kind of like a no-brainer you find out what the opportunities are for automation what the big problems are the big challenges and what you need to kind of attack first you take that workflow or that task and you deconstruct it into its various subtasks or sub parts you consider the return on your investment for automating each of those individual subtasks you think about what kind of automation you're going to have to apply to that problem and then finally you measure what you've done you evaluate the impact of that all animation and what we're going for here is measurable improvement right we don't want to go back at the end of the day or

week or month or year and say we built this huge awesome widget and it's super cool and you know it exists and that's basically all you can say about it right we want measurable improvement kind of calling back to the keynote from earlier this morning we're gonna be able to speak the right language right we don't want to be building widgets or scripts or programs just for the sake of doing it we need to be able to speak the language to people who are not in the sock or on your team or in your group so they understand you know what they're getting back for their investment in your time and the capability they're getting just by way of a disclaimer you

know I think for many of you who raised your hands a few minutes ago who are kind of doing automation today whether that's kind of like a micro automation where maybe you've written a script that goes and pulls some sort of information you know corollary information or gets you context or does some research for you or maybe you've got you know automated email generation those kinds of sort of micro automations they may not require some big five step process or some detailed analysis to execute right you might just be whipping those out you know in an afternoon and that's fine that's a big part of the job that's part of what makes it fun but for a

larger more complicated initiatives particularly with kind of more complex tool sets we get into like machine learning for example those types of things where you're making a significant investment in time and effort those things really do require more deliberate analysis and planning and so that's kind of what we're gonna focus on and try to exemplify today in our talk I mean so let's talk about this this five-step process and kind of how you walk through it identifying the opportunity requires something that is horrifying - I think most of us which is we're gonna have to go talk to people have conversations and you know if the team that you're helping whether it's your own team or another

team if they're not bought into what you're doing they're not going to use it they might use it for a little while and then let it go by the wayside and all your effort is wasted right so it's really important to take that first step of gathering requirements talking to your users making sure that you're identifying things that are going to help them in their day to day and use that as an input to the automation process the second thing you're gonna do is deconstruct that workflow into various subtasks and need to ask yourself some questions about those subtasks right questions like is the task repetitive or is it variable is it something that's gonna happen the same

way every single time or does something that's gonna change depending on the situation or certain inputs or certain scenarios is the task independent or is it interactive so is it something that I can just trigger a process and just let it go and give me the output or get to a certain end state that I desire or is it something that's gonna require some user inputs perhaps an interaction in order to kind of decide what tap to take or what actions to take what information to get or tasks to perform and then finally and I think this is particularly relevant in defense is it verbal or is it electronic in terms of orientation of the tasks so does it require you

interacting and talking to another group or a customer or a user or is it something that is you know fully kind of programmatic can just happen without any kind of back-and-forth interaction so these factors are really going to influence whether or not a task is even appropriate for automation and it's gonna impact how difficult the task is to automate and where it should land in your priorities when we talk about measuring automation I think this is where a lot of us kind of tend to have challenges because it's something that if you don't do it correctly it can be arbitrary and it can be very subjective I can say that hey I wrote this script

and it helps me do this thing a whole lot faster right but I think what we've seen is that that's that's maybe not always the case right some of that can can vary from individual to individual so you can't really just measure things based on you know time savings you need to really look at the tasks and think about what improving that task really gets you so let's step away from the automate part of it and just talk about improving a task because really that's what this is is all about at the end of the day right improving the tasks that we're doing and it's really kind of a scale if you think about it in terms of you know

very minimally improving a task can have negative value which despite the use of the term negative it's not a bad thing it just means you can improve a task so much but once you improve the thing there's really not anything else you can do with that so in an Operations context an example might be doing Quality Assurance on a ticket that you would open right or a case that you would open making sure it's complete making sure it's accurate you know that task has relatively negative value because once I do those things there's not a whole lot more I'm gonna do to improve a ticket being complete and accurate either it is or it is it isn't going up the scale a

task improvement can have constant value it can have incremental value this really just describes the amount of value that I can get from improving a task more than one time so for example if I'm getting better visibility in my environment if I'm an analyst a stock analyst every time I improve my visibility I'm gonna get more value out of it and really that's kind of a sliding scale that continues to go up over time I always want to be making those improvements right I'm always gonna get value out of expanding and extending my visibility you know intelligence is another example if I'm getting more accurate better more specific actionable intelligence you know every time I add to that or improve

that process I'm probably gonna get more value out of that and then kind of at the top of this scale is this exponential value where maybe I'm detecting you know something previously unknown or something that's really sophisticated you know and that's exponential value because every time I make a change there I'm gonna be doing something that you know is rare or enables some kind of creative performance so that that's kind of the sliding scale and we talk about return on improved performance that we get through automation that's how we and the relative value of the improvements were making okay not all automation is equal not all automation is the same so in the case studies that Brandon's going to talk

about in a few minutes we're gonna be focused more on robotic process automation which is kind of the more standard like programmatic automation so you're writing a script or you're doing a custom integration or you're writing a program or using a platform to automate a set of tasks but there's also cognitive automation where it requires user input to determine kind of which path they're going to go down and what automation you're going to do there's collaborative social kind of robotic automation which is more collaborative intelligence artificial intelligence where the entire process might change depending on you know certain variables or certain imports over time right again today we're gonna be focused mostly on robotic process

automation I think that's probably what most of us deal with day-to-day and the kind of automation that we're working with and then finally we're all about results right so we want to measure the impacts of the automation are we substituting a series of tasks in an automated way for something a human normally does manually are we all men ting what a human is doing manually you know adding more information you know adding more value to to that set of tasks are we saving time so that new work and new value can be created right are we getting things off of our plates so we can focus on other things maybe that are more valuable or that are more

suited to humans doing the task versus doing them programmatically right so that kind of summarizes that that five-step process and how we step through that so now I'm going to hand it over to Brandon he's gonna walk through some examples of applying this process in an operational environment Thanks so we have two case studies and we kind of chose the case studies based off of where our analysts felts actually where we got the most impact and in return as well as to make it a little bit more relatable so our first one is alert context and again we we want to talk to our analysts we want to figure out what what is tedious in their daily lives what do

they just absolutely hate having to do every single time as well as where can we potentially make the best improvement so we found that it was tedious and time-consuming you know we this happens almost every single time you have to triage an alert you have to go do these steps you have to find your historical context learn from yourself unfortunately many analysts will forget that process until they go down the deep dark rabbit hole and then all of a sudden realize look that got written up about four hours ago and now all that time is wasted so and we need to gather identity information so who is this who owns this device what type of device is it can we even

determine whether this threat is even applicable and also find out related events so when working with our analyst we kind of came up with a high-level workflow and it's intended to you know to be a little bit more simplistic and everyone's gonna be a little bit different based off of your processes based off of your the tools that you have and how your analysts work within your environment this is what we had we went to our analysts and we were able to identify seven different tasks out of that entire workflow that we could chunk down and make more manageable so we could start with just these ones determine what their potential impact would be what our return would be and

then start with those first before we look at this entire process and just think let's just go create an entire automation for everything where you try to do too much too fast so walking through the first one it's a repetitive independent electronic type of task where we're doing the we're pulling the similar information we're basically just changing the metadata that we're searching and our next task is basically a way for us to potentially escape the entire rest of that data collection process right so we when we work with our analysts we actually even had to reorganize this a little bit to make this most sense where we can get the biggest bang for our buck where if this

was true where we've already seen this there's enough feels to say that we've already seen this we've already termed as a false positive or we've already taken action on this we can skip the entire rest of the data process save ourselves some automation resources as well as save us of our own human resources because you know automation resources is still something we have to take in consideration when we're doing automation because it still has a cost so that wasn't is variable independent electronic working down through the rest of it where we're trying to identify the device owner who who's affected as well as what are they potentially vulnerable to gather that information so we can provide it to our

analysts which kind of leads into the next part right so we're we have a is this thread even applicable do we have an attack against a system that's not vulnerable for it or do we have an attack against a system that's not even applicable again try to escape that process as soon as possible so we can try to save ourselves some time on the backend and then now we have our you know typical we gather a log data from the Sam or logs store and we determine we that's when we start to to actually do some of that analysis so we can see we've tried to front-load a lot of what we can and push the analysis a little

bit further down so now we can can we determine false positive what do we need to do with it before we actually go to the hosts of start gathering logs memory captures or whatever the case may be to help help perform the completion of that analysis so we're able to identify four tasks that are really good for automation the repetitive they're independent and electronic we can potentially take these tasks and code them out and provide a lot more value to our analysts and then we have one where we need to put a little bit more rules bounds some logic around that before we continue with it just to make sure that we don't automate something that we're not supposed to and

always err on the side of let's send it through the rest of the workflow just in case because we don't want to make that mistake where we're trying to do too much and our human breakpoints and this one's really really important to identify all of your human break points where it's absolutely necessary within your process because that's the whole point we're trying to provide the information for the analysts to make the decision there are some cases where you know when you start talking with the analyst that they're pretty much making the same decision based off the same set of results you can try to automate that portion of it this is where we really determine that we need to actually stop

point and let the human make the decision so we've broken down all of our tasks we have seven different tasks that we might be able to automate and we want to start adding information and determining our potential our IP on it to start helping prioritize of where we want to start so we have our historical context you know it allows us to do a little more timely response the biggest one is allows us to reduce duplication of effort right and unfortunately you know it does require some of ap access to some of our tools so it's important to capture those dependencies so you know you might have to go grab those dependencies and solve those first

before you can start on this and it helps some with your prioritization have we observed this before if you know the dependency is we have to have test1 done first so if we go down our automation process this has to be done before we even consider the rest of it it allows us to be a little more efficient and reduce some of that duplication of effort again device identity as you kind of talked about a little bit is it's basically a way for us to gather information on what actually who's actually potentially affected and again we require some tool access and when we're talking with our analysts and again you know this is gonna be

different for everybody depending on the tools that you have how many logins you have sometimes do you log in a single tool you gotta log into your SSO and then you got to log into the tool itself in some cases so this is determined high difficulty based on the tool sets that they had to access is a threat applicable and this is our human breakpoint right so we have to make sure that everything up until task 3 is completed before we even consider this but again this is one of those things where we have to make sure that the human is actually inserted into that process at that point then we want to gather a log data again we need some

tool a P I accesses we get some incremental value some more context makes this allows us to make more efficient and consistent decisions based off of the data and the information that we're getting and I think the big one for this one to point out is that we're getting consistent data set for analysis so analysts will typically query a SEM in 3 or 4 or 5 different ways across team this will allow you to present information to your analysts in a clear and effective way where it's consistent through every single alert versus who knows what they ran and if they sell together information that's you know so be it and then is it false positive against

this one of our human breakpoints but we identified it you know before we get to this point we need sufficient context and analysis before it can continue down our workflow then our lie and lastly we have our you know gathering the host data you know typically this is a little bit more intrusive because we have to utilize an endpoint tool or scripts or some other fashion of gathering that information and for us it was a little bit more difficult because we had to actually execute some of those scripts and some of those and access some of those tools and sometimes that's not as clear as it should be so we break it down you know one by one and I and we

can kind of start to see that there's a lot of information that goes into this there's a lot of things to consider there's a lot of things that kind of make this seemingly simple process relatively complex so this is one of the reason I'm Airi reasons why you want to kind of go through this exercise a little bit so we can start to determine where do we want to start and where are we going to get the biggest return as well as you know think about your analysts what's gonna make them happy as well because you know as Mark mentioned a little earlier we need buy-in if they don't like the tool how can you use the

tool at the end of the day and we have to take that into consideration that we're doing something for them and not saying here's a finished product enjoy right so we're able to determine historical context you know kind of going back to us and a little before was learned from ourselves if we've already done the work why do we do it again and unfortunately this happens all too often an endless process where you get so far down there because it's something cool you really want to look at it but then you realize someone else already did it or we already had information on it and you just forgot to go get it so that's where we get the biggest return is we

can potentially exit our process as quickly as possible save on human resources save on automation resources and when you know the rest of them that we put down there list so you know the device identity gathering our log data have we observe before and then gathering our host data so now that we've identified we kind of go back to our graph here of a workflow we've identified our seven tasks and then we've identified down from that five tasks that we can actually automate and we've broken apart we've determined where we want to start and now we wants to start determining how are we actually going to automate that so there's the four tasks out of

those five that we can use robotic process automation on so we can put these through basically feed it data and we get data out we don't necessarily have to do too much logic or too much things in between there and it makes makes it easy and then we have our cognitive automation where we're again we'll probably talk a little bit more to our analysts figure out how are we what kind of logic what kind of bounds rules do we need to put in place again with erring on the side of being safe of always sending it back through the workflow to make sure we're not saying something's it's all false positive from previously when it really wasn't and

this one kind of this leads into like the next the next step is what is our impact of that automation how what is the return we're getting you know we need to be able to present this in a manner that is ingestible that is we have to speak the same language so we can condense down our big workflow into basically some some simple steps where we've taken all of our data collection all of it assumedly automated is now in one step and then we have our cognitive automation under Horst Oracle context we still want to take into consideration we still have that breakpoint in there to again exit the process as soon as we can

and then we have our animal analysis or our human has to do some of the review of that data and analyze the actual data there and we can see how much some more simple that process becomes with implementing some of this automation now we have to assess and make actually sure that we're actually getting what we expect from the actual automation that we're putting in place talk to your analysts figure out how long it takes them to complete that task and kind of gets an idea so we can start putting some metrics in place of you know again make sure that we're actually getting some of that what we expected that automation and one of the things I

think you know tends to get forgotten is you know what what augments of a human can would do where if we're providing all this context upfront they don't have to go gather from the tool not only are we saving that time of them gathering it but we're also helping them give that information so they don't have to go see five or six different tools and correlate themselves or gather that information dump it onto a CSV and like make pretty pivot tables and add more information and context later right we're giving that information to the analyst and an ingestible format so that's going to save them time on just performing that analysis in general so reducing time and effort and they won't

know we're reducing the analyst fatigue we're saving the time that they have to go and find that information and see me a lot easier for them to work through that workflow so kind of is a little bit bonus I kind of wanted to show what we what we were able to do and how we kind of present it to the analyst a little bit we're talking about a lot of data ingestion we're talking about a lot of things and you still have to consider what this actually looks like a little bit so kind of quickly walk through you know this is where we kind of stood where we end up store in our historical context and the various different tabs

and we wanted to summarize it down the front for the analyst again summarizing as much information as possible from our automation so they can see it straight up front and then our event context and the other is a threat actually applicable and so they can make a quick decision depending on what the actual type of work that they're working through and it's kind of an example pointing back to how me how we feel that alert context and what you've already done is so critical where we can see in the beginning we had a not blocked trigger ours is blocked again seeing you see at the top and it was already remediated same IP and then we can also

see that we have an assigned alert that came in a little bit before and that one has the same W script message line there it hasn't been resolved yet so we need to work with that analyst so we're not working in a bubble right so we can reduce that effort we don't want him to perform the analysis and then us to perform the analysis and then now we're duplicating that effort and then same thing unfortunately it happens all too often in the real world where we go to ask have a email removed because we found something malicious unfortunately doesn't take you know it takes more than five or ten minutes in some cases and it

already clicked the link but tell me have it removed well we get out there all that information of that context straight up front so the analysts can move forward with their investigation so our second case study is run intelligence research you know this is again something extremely common that all of us have dealt with if you're if you've been in the analyst world or in a stock and it kind of similar thing to what the alert context was where you know it's a similar steps were performing every single time we're going through the same set of open source tools or paid subscriptions we're dumping in an IP we're trying to find the same similar data coming back we're

trying to gather the context of that indicator of compromise we are trying to determine what is that threat you know maybe some attribution in there if we can get it or in big one is you know what is the capabilities so what what is it capable of doing to the affected user stealing passwords so on and so forth so again you know everyone's process is going to look a little bit different this is what we what we call menu together with our analysts a semi high-level so that way we could start can deconstructing the tasks and determining where do we want to start with our automation and make it manageable so we were able to determine

we had six potential steps that had that fell into the fell into potential automation with our first task wheel determinist repetitive in depend on ik you know this is gathering if you have process intelligence if you have an intelligence team or if you have an IOC database you know and the other one here out to the left is going to be things you've already researched why burn your API three or four or five more times if you already have that data cached from five ten minutes ago maybe an hour ago search that data first find out make sure that you search it and he could save yourself some API calls and then you know this context actually true we

actually find anything if anything useful and then this is where our first analyst breakpoint was where we had to actually analyze the data we got back is this is the annal is the data that we pulled back sufficient enough to continue on with the analysis process or do we need to go gather some more data from open source and some of our paid subscriptions and it's important to keep in mind this this process here is basically we broke it down to be one single IOC that goes through this each and every time and we have to iterate through this process every time and the idea is is that we're gathering from our paid subscriptions from our open source

and though even those have some limitations on what what is this thread about and then a lot down at the bottom when we start talking to analysts you know it's a natural progression to when you find an i/o when you find the 9 OC you go search for it and then you find maybe five hashes or five domains associated with that and then you go and search those and you keep going down that rabbit hole and that's where a lot of times we get lost and when we put this specifically in here to address some of that was that we wanted to is sufficient information there did we go sufficiently deep notice do we have

three resources that say it's all this same malware and it's all malicious and all have similar risk scores there's three separate ones they're not referencing each other okay we can move on and not burn the rest of our API calls as necessary so when we're finding some of the related infrastructure we can ensure that we're being as efficient as possible and then lastly we have our you know our actual analysis the is it a false positive you need to continue on with it can we go ahead and exit process so as we did before we want to break down and add in all of that information to help prioritize where we actually want to start so

you know we determined it was an incremental value we still have to have API access and we still have to have some of that historical and processed intelligence in some way shape or form you know its context found with we have a dependency on the tax on task one for that one you know it's a fishing context found and this is going to be where our analyst actually has to step in and it's a breakpoint where we have to make sure before we go burn our API is before we go run all that information do we actually need to do it do we have enough information that we that we can save ourselves on that and then gather in the

actual context you know this does require some subscription API is if you if you're if you have those and free and open source API is there's a ton of open source stuff it's just a lot of times you have to hit two or three of them instead of just the one one or two and this allows us to you know the threat context allows more informed responses and reduces the tools the analyst would actually have to access you know sometimes it's five or ten or sometimes have to use that one that they only use here and there and then this sufficient death check kind of a little bit of what as mentioned before was allows us to

extrapolate on indicator relationships and hopefully go down deep enough in our particular workflow that we worked on we found that two was actually sufficient and if you had enough resources that you were checking where if you're checking five or ten different resources you typically only need to go another layer deep to actually determine is this is this actually malicious so you can start that response process and then checking is a false positive is our last step where the actual analysis comes into play and again you know it's it's seemingly a simple process but as you start to put all this extra stuff into it to help you prioritize it becomes a little bit more complicated to know

where do where do you actually want to start so we were able to determine you know our analysts actually kind of really wanted to gather the context and indicators but that always comes at a resource cost something that we had a taking consideration where we ended up gathering our known context first again why search for something if you already have it in your database from an hour or two ago or maybe a little bit longer you have to kind of set that threshold for whatever you're comfortable with you know more context allows us to make the better decisions and then remove some of the variability so as we did before we kind of wanted to I'd start to

identify what type of automation we want to put in place for these particular tasks and we identified these particular tasks we can do some robotic automation basically static we can feed the same thing in get the same thing out and then give that to the analyst and then we need to put a little bit of bounds around this you know if to two layers deep isn't deep enough for what you're looking for then you might want to put in maybe even some logic around that to determine I want to go one layer deeper based upon these five five rules or five pieces of logic and then we try to simplify it so we can see start to see

where what kind of value we can actually get of this what the actual impact is so you know taking to consideration everything that we want to automate we can throw basically all the data collection again into one little bubble and with that depth check to make sure and with that human stop point of sufficient context and then you know so we talked again with our analysts you know when we started going through this process to determine how much time they spend on this so we can actually add some metrics to it and we you know reduces some effort and reduces our time to respond to threats because we can find out instead of an analyst taking

fifteen or twenty minutes to perform a task after they get their hands on an alert they have that data there waiting for them and they can potentially respond to that threat quicker than is if they had to go and do the research themselves so reduce it reduces our time there and again just kind of a little bit of example of what that might look like to an analyst and how powerful this information can really kind of be so we'll dump Arnone context again to the other related records and we're going to be focusing more on the Oh scent that we're getting from our various different tasks and that tools that we have access to so it's a lot of

information to kind of look at at first but one of the things that we've quickly discovered that I was actually extremely beneficial to our analysts was kind of just grouping things into various different columns right and key into our key fields so as we can see here you know we have a couple different i/o sees that came back as trick bot in this particular case but we still want to look into those and just you know as an analyst just to make sure we got plenty of plenty of hashes to go research if we wanted to research and you know we've identified that it's certainly these indicators are certainly suspicious if not definitely malicious so it gives us

an opportunity to quickly identify this and start the response process as we dig in deeper and with that it's pretty so you know these workflows and these tasks may or may not be representative of the kind of work that you all are doing on a daily basis but you know I think some some common pitfalls that we've run into aside from some of the ones that Brandon mentioned during the case studies you know you go through the trouble of applying some automation you know writing a script creating you know a custom map or a custom integration and a lot of times you go through that entire exercise you think you know what you're doing you think you know what kind of

impact it's gonna have only to find after the fact that it's okay well the team's not really using it or they don't use it every time sometimes they just go you know to do it manually so it only has you know partial kind of value added or you know it really doesn't provide the kind of return on investment that we expected to it doesn't help us do this one thing faster or more effectively or more consistently and in those cases if you're not doing some analysis upfront and you're not breaking it down kind of Intuit subparts and doing some of that legwork and some of that homework before you undertake that initiative it can be really hard to go

back through and say well why is that why isn't the team adopting it the way that we anticipated they would why is it not consistently doing the thing that we expected it to do so hopefully some of the the case studies Illustrated kind of the value in doing that analysis and doing those breakdowns ahead of time so that if you need to measure and adjust later it's a little bit easier to do as opposed to just creating you know some black box if any of you have done any work with like artificial intelligence there's this concept of explained ability right transparency people have to understand what the thing is doing and how it's working and that plays into

not only kind of managing that automation or managing that that application on an ongoing basis but also training your staff training the users to understand you know when they click this button or you know when they execute that that automated task what actually is happening under the hood so they can give you the kind of detailed feedback you're gonna need to make sure that you're enabling them another kind of risk in doing any kind of automation that that people don't often think about is aside from the upfront work of actually building the thing there's that ongoing engineering support and maintenance right so I'm sure many of us have things in our organization that some really smart person built or did

and now they're long gone and you just kind of don't touch it and stay away from it and you know don't ask too many questions about how it works it just kind of does right so you got to think about like that ongoing ownership and maintenance and also I know no one in this room would ever do this but technical people sometimes can like go down really deep rabbit hole right we can like find a cool problem or like a cool platform we can take like four times as much you know time as we anticipated kind of doing that so again part of the reason for doing some of this analysis and breaking down some of

these tasks this way and looking at the problems this way is to avoid doing too much and to avoid trying to take on kind of too much and letting you know perfect be the enemy of good when all you're doing is trying to add incremental value so hopefully some takeaways from this talk again you know automation should be a process it should be a way that you improve value it should not be a product that you buy and Jam into your environment to do a thing you know may be a product as part of your strategy and automating maybe it's part of that process but it's not kind of the end in and of itself

repeatable agile processes like I'm a huge fan Brandon's a huge fan you know they help us avoid those rabbit holes they help us make sure that we're getting return on our investment whether that's investment in a paid service that we're using where we have to be judicious about how we you know hit those api's whether it's return on people investment that we're getting out of the folks that were relying on to build some of these integrations priorities are really important not all improvements provide equal value so sometimes we get so wrapped up in understanding that you know we're doing a cool thing or we'll but we're building a widget that's gonna do you know this

automation that we weren't doing before we don't always ask the hard questions about you know is that gonna be more valuable than focusing on you know a different widget or focusing on just you know handling this task manually for now because we have other priorities right so so priorities are important one thing that you know if you wanted to leave here this weekend and go back into work on Monday morning you know and try to apply some of this again it doesn't take some sort of formal rigorous process to start to ask some of these questions and start to apply some of this stuff you know take any automation initiative either that you already have

operationally in your environment or that you're thinking about or planning or working on try to to go through this exercise try to go through those steps try to quantify kind of the value that improving you know whatever that task has for the organization try to break it down into its subtasks try to really understand you know and make sure that that all that stuff is aligned right see if maybe there's something that's been missed or overlooked you know and that's kind of even just a mental exercise that you can go through and apply some of these concepts so that takes us in the end of our talk thank you very much both Brandon and I are

Twitter Brandon's a lurker mostly but you can find us both there and with that happy to answer any questions or talk to you after but thank you [Applause]