Data Visualization in Security: Still Home of the WOPR?

please welcome hey guys thanks for coming out my name is Mathew Park I am a user experience lead over at endgame real quick and game they just it's a ETR EPP platform the context of this is we're building an attack timeline visualization which I'll be talking about during this talk today so this is come something we're currently part of typing within the product and what our company does is they build endpoint protection across enterprise-wide kind of stuff well my team does in general is what we do and what we care about or kind of building thoughtful and practical work flows like visualizations and general experiences for cyber security analysts so we really care what the users dough do 100% of the time that

image to the left is it's not supposed to make sense it's it's a joke defining what UI and UX is but I got a lot to go through so let's just start it out

all right I have to do that I would be totally miss if I didn't actually put a video of actual war games in there for you guys that aren't familiar with what war games are it's a film back in like 1983 basically they the Americans created this thing called the whopper that like kind of go through these wartime simulations during the latter portion of the Cold War and it's supposed to like have us have a leg up and the nihilo arms race at this point of the movie the whopper can no longer tell between reality and kind of fiction or and it's trying to route force its way into NORAD and getting the codes so

Matthew Broderick is kind of like yelling at it trying to understand it to understand the futility of war by kind of like this tic-tac-toe simulations so it's it's very surreal I highly recommend watching it but it's kind of ridiculous but I'm gonna segue into that by saying it's ridiculous for something totally unrelated to it for those of you who've seen the movie sock analysts and the actual like generals make the majority of their main decisions and driving components primarily based on all the Whopper visualization I can honestly count like maybe between two fingers the amount of times a sock analyst has like a UNIX shell open versus when they're like mouths agates in front of like all these types of

visualizations out there and you can actually see like in a typical sock obviously you have kind of visualizations up on the board but in their individual monitors over there they actually have just the same things just kind of strewn about or on the stock floor and I kind of came to this conclusion after watching this movie for the fifth time leading up to this talk that the only way that this kind of makes sense to me is I guess that these the this visualization was pretty practical for the 1983 version of NORAD out there the designs are like pretty minimal simplistic relatively easy to understand I guess it had the stock analyst mindset of a Norris

and NORAD analysts which is kind of contradictory to what this talk is about or what it was framed as still home of the whopper not exactly because visualizations that we as we know today don't really have the main like driving decision force at the whopper had in this movie analysts generally don't like dictate their workflows based on what they're gonna be seeing in like some sort of pew pew map so it was really never home with a whopper whatsoever it's been plot twists as we know analysts today kind of are more comfortable sifting through like a stream of events through like typical heavy list views and they're more comfortable doing that and they form these patterns around how they're

taking the metadata from those individual list views and what they generally view visualizations as of what I found it's kind of bogging them down and being a little bit more time-consuming and generally useless so I took this quote from this guy named Raphael Marty he wrote a book called applied security visualizations which is pretty good and he talked about the general problems plaguing security visualizations today he says they're either the work of designers with no background in security whatsoever or the work of security professionals who don't really understand data visualizations one is generally pretty beautiful but not practical and getting work done and the other is relatively effective but it's kind of non-intuitive it requires the average analyst to do a

lot more work in actually piecing together the story and generally as an industry we kind of lean towards the latter of these visualizations because a they're easy to build you can just take them straight out of like a d3 library and it's kind of a disappointment because we tend to lean on something that's kind of familiar to what we've done previously rather than make a focused effort towards usability in general which leads me into attack timelines this is the outline of the talk right a whopper right there so this is a talk around attack timelines in general it's actually disguised as usability talk around tech timelines surprise so at the end of this talk I'll be kind of

discussing and showing a couple concepts around some prototypes that we're currently currently putting together through endgame but these kind of designs aren't really the key takeaway I want like this group to leave with it's it's more around my hope is as I discussed the approach of how we came to this conclusion of what we should be doing for our attack timelines you guys can take these C principles back and broaden your own perspectives about how you should be approaching your own set of visualizations whether or not it's attack timelines or anything else in particular I firmly believe in order to introduce any sort of visualization in general that's functional and can be usable to like whoever the people that

build it first need to understand like do the proper research into design structures and understand their user base to make their corrupt decisions so going into it surprised I am easier experienced designer so we went through a very user centric approach to discover what exactly is needed for attack timelines and we kind of split this into three individual stages the first being discovery coming from endgame we had certain biases coming into what the attack timeline should be because people within our organization have worked previously with them so we wanted to confirm whether or not those are actual things that were currently out in the wild we had a opportunity to test a large amount of participants through

actual stock organizations as well as our own internal so we ran a couple user of research testing and to capture kind of the organs a organizational workflows from them as well as creating a new persona paradigm around specifically around alert triage and attack timelines the next phase is concept in so through all the data that we collected we eventually came up with a couple design requirements that gave us the guidelines to formed what the version of the attack timeline was and we revisited some tried-and-true all basic foundations of what design patterns should be around and then the last phase which is part is typing user testing this is something we're currently in I'm not gonna be

demoing like an actual working thing rather explaining and showing some designs that we've been going and actually are currently building with our dev teams now the hope again is to take that back out into the wild and test it among the users that we previously research with or whoever wants to get their hands on it so getting it straight into the discovery phase I'm going to walk through each one of these things as an organization we wanted to define exactly what in the TAC timeline is to have like a basic understanding or a consensus of what everyone thought what we're trying to accomplish and I'll just read this straight off as we wrote this down so the goal of an attack timeline

or an alert triage visualization however you want to put it is to allow an analyst to quickly assess the relative severity of an alert so they can dismiss remediate or escalate or escalate to another analyst it really serves as a means to communicate a story of an attack because it's generally just a story like lot of visualizations and should be used as a platform for data manipulation and general exploration like I said before coming from an organization that had prior experience this kind of stuff we also had a couple other biases going into it so we found that you know large groups of users generally like tier 1 users generally lack domain experience in the security

field as well as well as platform experience on a lot of other security platform products which made a lot of the visualizations that are currently up just in the ecosphere of security products kind of difficult to navigate because some of them are kind of overly complicated they return a lot of information back to the user and they really provide no context of where they should be going or how they should be mediating it up or down the chain so that's one the one is the lack of time and this is just a general problem among Sox in general repeated at toys they have like 5 to 10 minutes to make a decision on an alert so typically

they're looking at streams of like hundreds or thousands of different types of data so without context without knowing what they should be looking for they have to make a split decision whether or not gonna they're gonna be sending it up the chain or they're gonna be archiving this is something that's nothing they nothing they really need to be looking at the last one was kind of our own thing ports endgame we wanted to build something it was kind of a bit of more of a differentiator we wanted to build something that kind of enhanced the analyst workflow something that they've actually used and not be useless eye candy so the list of different things that we added to that were

showing response actions being able to gather on large amounts of data and make like different types of investigations off of it building some context to alerts already taken or so actions already taken on particular alerts and IP changed if you killed a process if if you suspended a process and general collaboration and commenting kind of something that we found and I'll get into this a little bit more is as people are communicating up and down the chain within a sock it's it's very difficult for them to kind of show what they're actually working on as well as communicate whether or not alerts are actually bad so knowing these biases we came into the study where we got like

three uh participant groups and we wanted to be able to capture different types of team dynamics among the workers of roles within the cirrie organizations we wanted to form our own unique personas off of all of them as well as kind of see how they work through alert triage in general we've done this multiple times on multiple features but this one was kind of segmented directly off of alert triage and attack timelines I'll walk through this really quickly but the first one was a traditional flock team so we got 20 people to participate asked him a set of general broad questions over our attack Alert triage and attack timelines to give us kind of a bucket of

responses so we can see where they should be going where we should be going with the feature the next one was a version of our own a bee testing so we had an actual livestock team that was participating in a red vs. blue exercise so we mimicked the exact same thing or tried to mimic the exact same thing internally and that included like general dis side by side monitoring with other researchers at endgame data Sciences product people we had some user interviews that kind of hit on the same questions that we asked user group a and we had a larger retrospective at the end were like a two-hour session were these people about to air their grievances on

where they felt attacked online should be going how they were dealing with alert triage what they wish was there like within the product at that point in time so that was how we collected all this data back and it was a lot and this is a wallet text but this is of the basic four types of personas that we came out of this group and I'll go for this through this really quickly as well it's so a Tier one analysts again has generally no experience within the domain or a security platform in general they're more comfortable working through the UI they are the first line of defense within osakan organization they either they look at the alerts coming in

and they have to make a split decision whether or not to send this up to a Tier three or a forensic hunter or to just like throw it down into a queue that doesn't really matter the tier threes take this and they actually have the authority or the power to make actionable responses on top of this these guys are both of these guys don't really use the UI too much they're more comfortable working in the command line they just want to rip the api's out and like work directly off of that so they generally regard visualizations as kind of tiresome and like a little bit overbearing and the last one is a sock manager so

with sock manager cause that's the the tempo schedules they really care more about collaboration and reporting purposes they want high to low level of visibility what's going on day-to-day within individual alerts or trends at a hole so this was kind of the spectrum of different users we were trying to cater towards when we were building the attack timeline we came with we boiled down to a couple perspectives of what we found throughout the entire group so in general the first two were more variables and a representation of time when we were opening the attack time visualization they there was a general consensus that a lots there was a lot of data streaming through and whatever we

were going to build and whatever people currently had previous that had previous experience working in attack timelines they generally thought that we couldn't stream the amount of data that a an individual centric sensor could provide back to the platform so that was a worry that was a worry among a lot of different types of analysts there is a trepidation representation of time a quote I got from one of these guys is time is kind of relatively abstract concepts so it's not inherently visible in general and we force this not inherently visible perspective in a very linear plane so when visualizations come out there a either force in a linear perspective or it's just removed completely

it's just based off connection patterns so analysts like to know obviously when an event first started to when it triggered but they would like to know they would generally like to know kind of the connection patterns of time between those two events which is currently lacking in a couple of attack time visualizations out there the next two was lack of expertise and lack of time these are our two biases coming into it and we kind of got these confirmed this is more on the tier one perspective they wanted more context of why they should be remediating alert through a type of visualization they wanted to they wanted the platform to point them in one way or another and

they felt force to make like these uninformed decisions under a stressful amount of time so they're hoping that whatever we built would help them through both of these scenarios the next one were kind of new ones that we serviced out these were from a Tier three perspective so the first one is increased working memory so they wanted to build a a better way to enhance detection and recognition with lower end users a large portion of their days since there's no real great onboarding process or a way to teach the lower end guys like what they should be looking for they're mentoring these guys and they're spending a substantial portion of their day showing them what

they should be looking for and one that's done over and over again they're not necessarily telling these guys how they should be looking two alerts they're just kind of saying what key metadata points they should be looking at like md5 hashes or whatever file path so it's not really teaching them in any one way or another so they were hoping that whatever platform we did we put together or whatever feature we put together in the platform would allow them to kind of break that apart and the last one is facilitate Scarberry in search so if this was being built in this scenario a lot of the tier ones would be flooding up a whole slew of

alerts back up to the tier 3 so they wanted a way to help them without moving to different types of products or the command line or anywhere else that could focus in on key areas that they thought were suspicious or anomalous within their environment so based on kind of these six things that we boiled down from all of our user interviews we both us down into two different design requirements which we are trying to accomplish during the entirety of our prototyping phase right now so the first is the visualization the attack timeline should be really used as a tool to enhance the typical typical analyst workflow by providing obviously a high to low level visibility and context on

granular data just in general so if created correctly and purposefully it would be able to shine when large when mapping large quantities of data at scale over time the next one is something I didn't get into in the last slide but this was more for the stock managers and kind of for the tier ones a little bit the visualization should be used as a tool for collaboration reporting clear visual visualization representations in general don't require really anyone to be like a high-level security domain expert really anyone can understand trends our content distribution in general what we found or what we were starting into where humans are generally just very naturally visible people and they tend to process

data at a faster at-a-glance rate which if again done correctly opens the accessibility of an attack time line visualization to a larger array of users which are talk managers in tier once you don't have that typical domain experience to it the next phase which is the concept phase so we had a whole slew of different design principles that we were kind of looking into one was around perception that I toast the the one that I like the most in it's a around this guy named ben shneiderman it's his information information seeking mantra and I'll go through this quickly basically he found that the most powerful visualizations kind of share the same similar traits and this might

be just a kind of thing for the rest of you but it's worth noting and thinking about as you're building a certain type of visualization so the first one is you want to build an overview first so it's it's kind of the first thing the user will see and it'll guide them through other parts of the product for further exploration the overview when carefully planned to highlight important parts of the story it gives it should be giving lesser weight to like not so critical parts as well this is also the one that's going to be the most visual so you should be opening whatever overview - bored whatever - the most critical amount of user testing the next one is

zoom and filter so once you have all your data presented on an overview section you want to give that user obviously the ability to focus on on particular areas of interest this involves like zooming and filtering down on data which is like zooming scrolling panning drill down legends range lectures what-have-you this is particularly very important for complex visualizations because the zoom and filtering functionality will be should be designed in a way where users can't get lost throughout the visualizations attack timelines in general are pretty expansive node beasts so if you leave it up to like a Google map kind of like find and select and zoom generally if you open too many notes it'll be hard to navigate back to

where you came from and the last one is design on detail being able to give the minutiae of detail back to the user so this is for the tier 3 case getting as close to the raw data as possible this third level would be less visible but it's more text having kind of focused on accurate information rather than trends and I'm running again for the clock so I'm gonna go through our actual prototype in phases a little bit quickly so this was our first iterations of what we wanted the attack timeline to be these were two examples of many many ones we put together it was kind of the wrong way of going about it when we mapped it on like

a 2d linear perspective it kind of confined the amount of data that we were putting up on the screen it also again map time to singular axis we had the correct amount of overview section kind of at a glance stuff there but it was difficult doubt it was difficult to zoom and filter down on corresponding areas of that node map so we were researching other types of other structures that could house all the information that we were gathering from the the user feedback as well as all the the rest of the design principles that we wanted to do so we kind of went to something called spatial temporal structures and these are really meant to map they're

traditionally used for a kind of mapping 3d geography in general a particular use case around it is mapping hurricane patterns of like where a path came from to where I'm I'm running long in I I'm over speakers okay sorry we apologize this was a short talk and we can take about one or two minutes at most for questions so that we can get ready for the next talk all right sorry yeah interviews and the split testing amongst the different groups yes sort of um how do you bring like I guess a hard numerical side or like empirical to to the testing itself like you can interview a person and they say this is way better than the last

thing but you guys are probably trying to squeeze the most improvement out so how do you really we try to not be as biased as possible by like leading them into a certain path of like hey this is like it obviously it's going to be better every time the iteration that we come out with is so we try to show them a couple other designs to kind of move the needle back into the middle to see if if this is actually something that they are actually like an improvement on the last iteration not currently yet we are doing some implementation within the product to do that kind of third-party testing yeah I think we're gonna have to

cut it now and see if you'd like to continue the conversation go ahead and follow Matthew on purist okay thank you very much I