GT - 101 ways to fail at getting value out of your investments in security analytics … and how not

hello everyone hopefully you can hear me as the presentation title gives away there are lots and lots of ways to fail a security analytics and in this presentation I'm going to share some thoughts frameworks and ideas on how we can stop that happening by thinking about and managing things up like make products we're going to look at how those frameworks can help us navigate the challenges associated with two critical dependencies in security analytics the first of which is actually having the data available that we need and the second of which is being able to move that data to where we need it to go and in my experience if we don't do this one of two things is going to happen

either we are going to fail totally or we all deliver a level and consistency of value that is unmoving to the people that are paying our salaries now to manage data like a product we're going to need a crack team that can focus on joining all the dots and picking up all the things that fall between the cracks of any serious effort to use data thank you it's use data to best protect our business that's likely to be a team of teams it's not going to be a specific group and it's going to draw people from all these areas of skillsets to come together to solve the problems we face and while their mission in our case is

security specific as this fantastic paper makes clear it's actually no different to any other analytics mission that takes place in business today and that mission can be handily summed up in a single paragraph it is to facilitate relationships and closely connect all the people that need to come together to collect data process it analyze it and put the results of that analysis to good use and this isn't just about kind of machine learning and anomaly detection for threat a staff this is actually about a bigger issue which is how different security teams collect and manage their data and then deliver inputs and receive outputs from other teams to make their life easier so this is a talk in four acts in the first act

I'm going to describe why security data operations is a necessary function for any security team today in the second and third act I'm going give some ideas about it I said some structured approaches to dealing with our critical dependencies for analytics success namely these ones and then I'm going to describe the change that security data operations often has to try and bring to a business in terms of the data that security teams have access to and why that is we're going to go through a lot of material very fast today so don't try and absorb everything on the slides look at this as a field guide for all the issues we're going to run into when we go and do security data

operations and if you want to get access to the slide they're already pinned on Twitter here so you can get a hold of them with full slide notes and absorb them at your leisure so for now sit back relax and enjoy the ride ladies and gentlemen welcome to cyber plant systems you are here because you are afraid so you should be sales of hooded sweatshirts are increasing exponentially which can mean only one thing the tide of cyber crime is rising and while you're bored of or it only you know the true meaning of the horror the complexity of your systems indescribable the threats you face innumerable the competency of your team questionable the challenges you must solve impossible

which is why today I am proud to announce our new product our normal user because in a world like this what would want to be different a normal loser ha this is the power of very clever math but not just any math math so devious in its power so complete in its knowledge that it knows what the threat is and who the threat is at all times and that threat it's good Gabe is printing documents two floors above and hundreds of meters away from his team in a vain effort to hide the fact that he is exfiltrating your intellectual property in physical form Gabe is your data thief that threat it's hot Hoff's credentials are being used to

access systems that deviate from his baseline through a foreign country Hoffa is collaborating with the Russian mafia Hoth is giving them access to your systems Hoth is your mole the threat later is downloading gigabytes of data to a cluster laptop in a manner of the disturbed regis but all access to her systems must be shut off immediately and automatically but enough of these trivialities as our comprehensive demonstration has shown you there is only one answer to your cyber crime needs that answer is a normal user that answer is 43 no well I cannot promise you our system is a hundred percent accurate what fool would do that what I can promise you is if before buying our

software you had fear uncertainty and doubt when we are done with you you will have fear uncertainty doubt and a massive invoice that we will use to pay for bizarre neoclassical fusion art double hang all the walls of our West Coast offices so that when you visit us you are in no doubt about where your money has gone now the only question that remains for you to answer today is are you ready to bomb

[Music] [Applause] so thank you for indulging me in that that's a condensed version of fairly much every pitch you get in security analytics as you begin to slide down a vendor sales pipeline and while yes I'm sure you'll show me something in 30 days possibly how difficult it is to get data into your product there's a really serious point here about the challenges that face businesses that want to use data analytics and automation to improve their business thank you change my um so you know I would argue ten years ago it was a simpler time sorry so ten years ago is a simpler time um he's sure we had sales people like this pitching us for kind of IDs

firewalls and stuff but within a year we could know roughly what the tech did and how difficult it was to get us to do it however today sales pitches are filled with more confusing terminology like random trees and exponential forests that are somehow kind of got automagically you're going to solve our problems and while it's very easy to embarrass this new generation of salesperson if you have a math degree when they spout this kind of stuff in meetings it's really hard with or without a master's degree to really know what kind of value a product can give you in the long term and that's the case because when you go and speak to peers in the marketplace and they say negative

things about analytic solutions you can probably attribute that just as much to failures of deployment as you can product failures themselves and that kind of means you've got no option with this stuff but to just give it a try and that can be really daunting because when you go and speak to people that have committed to a vendor technology and feel they're getting value this is the kind of thing they say the products great as long as you don't believe the marketing right we have to invest five years and two analysts to build the skills to use it and if they leave we're screwed so if one of our jobs in security data and operations is to avoid

wasting time on pocs that won't deliver value another one of our jobs is to short insta to ensure that if we do commit to something that we think lines up we know what we're getting into so it's handy to have a set of questions to do this here are mine um my personal favourites are number five and number eight to date only three vendors that I've and to have given me what I would consider double thumbs up answers to these without missing a beat and all that those people were CEOs of their company so that's Alex Pinto from middle its Rob Lee from dragos and Kelly white from risk recon all great people and all great products

that I would definitely recommend checking out if you have time now as helpful that these questions are you may well be asking them while you are already in an express elevator to hell going down and what you find when you get there is the vendor data pool designed of course to look amazing so amazing that in the heat of the moment one of your team may turn to the vendor and say sideways to you as well as the skeptic bought into the room to put them through their paces you mean this could be our single pane of glass to which you a sigh but don't we have five those already because if you spend long enough looking at this

space what you find is there are a load of very specific niche use cases that vendors are selling on platforms that restrict both your ability to ask the data further questions that you have to put into these platforms as well as use the platform's themselves to deal to do what you want I must say like not all niche solutions are bad so you know Alex Alex from little corpse and solution is a great example very specific data sex very targeted fields within those data sets and if you feel that you know they can tell you very specifically what value they can give you but if you're going to a vendor who's tackling a kind of wider problem set for example like

you EBA what you can find is there's a massive imbalance in terms of what it takes to get this stuff working and then the long-term value that you're going to be able to extract from it and that's especially the case in being able to get data out of these things because in some cases once you've got it in there you can either feel push out what you the the the date of the solution needs to process or the data that it generates which you then want to enrich and correlate with other sources and that's really annoying now of course in vendors don't want to tell you this because if they did you might stare a year into the

future and realize that after investing all this time what you're going to have is a capability that's limited and at best the challenge you now need to solve is on the roadmap favorite favorite term of any product manager and and you're stuck in an awkward place without paddle but yet this is the dynamic that 90% of the buy-side face and why is that well if you think about it in terms of who develops analytics and who develops platforms there are actually very very few companies very few private sector firms that can afford to develop their own platform and develop analytics on that platform and if they decide to do that they have to commit to it for the long term because

there's not really much incentive for a vendor to build analytics on a private sector companies platform which they get full visibility into because vendors like to develop intellectual property they can resell there's also very few firms that can or want to work with unproven startups which is basically outsourcing both the building of the platform and the doing of the analytics to companies that need but don't have data to develop their IP which means that you're probably and if you are what you've got is generally applicable narrow use cases on a platform that limits your capabilities now side note I really hate the term use case because while it is possible to find a use for

almost everything that is not necessarily the problem that you need to solve however we must stare into those darkest of mirrors as it would be wrong of me to suggest the only challenge we face in security data operations is vendor selling a stuff we don't need if you ever heard a C so say something like and the dashboard should look like Minority Report you have a fair idea of the story which I'm about to tell the cry goes out we must have all the data teams of data engineers and data scientists are hired morale is high they are sold a vision of extensible platforms a network of things working together to solve all these team's

problems but if you fast-forward a few years into the future millions burn through in budget teams that funded this central effort asking for their money back what you find is everyone got a little of what they wanted and no one got enough of what they needed till they are just because you can does not mean you should so this brings us neatly to the summary of our first act ultimately in business we are responsible when we're running programs like this to make the best decision we can in often complicated environments in imperfect conditions and that means that going to have to make trade-offs in terms of what platforms we use some are going to be open some are going to be

proprietary some are going to be things we develop ourselves some are going to be things we absolutely rely on our vendor partners to deliver and what that means is that we need to really understand as we go into this to deliver the both long term and short term value for our our businesses from analytics what does the system of processes look like that's going to make these things work together effectively so that we don't have loads of silos of data that can't talk to each other and loads of people having to look into loads of different panes of glass this is where security data operations really comes into play it's about knitting together all these

solutions and it's about focusing on the two critical dependencies that we're going to have regardless pretty much of any technology we choose which are namely the data features we have available to feed our analytics and how we're going to get that data from where it is at the moment to where it's going to be analyzed so getting the data is going to be our first key key focus in the next act and then that after that we're going to focus on getting the data and talk about pipeline and bandwidth so at - let's jump in if you're a seaso and you want to make an argument for why security need to get their hands on data a good

pitch might be something like this we need timely visibility into our environment to know if things are working more or less as they should and obviously to find risks discover bad staff and hey like data gives us that so give us the data please and if you speak to CIOs and cxos they don't disagree they don't need monthly powerpoints they need current data to run their business better however as the selective attention test makes clear just having data just having visibility is a very far cry from being able to recognize the things we need to even if they're in plain sight Alex from Netflix has a fantastic blog on sockless detection which I'd highly recommend you read if

only for this line the creation and/or collection of data should be justified by a quantitative reduction in risk to the organization in dollar terms but that can be difficult to forecast accurately if you had until you have a chance to explore the data now obviously our pitch this cio can't be late you know hey sport we're in a bit of a catch-22 a battle this so if you could just hand everything over on this list we'll get back to you about what value it has because even handing it over can be incredibly expensive and neither can we or should we make expertise base claims about what certain data sources will tell us NetFlow dns as we may well find

as we've gone to all the expense of centralizing these that they don't actually tell us what they think that means to develop a structured and reasoned business case for getting the data that we need we need to do two things first of all we need to answer all the questions that tell us about the inventory of data sources we have to work with both now and in the future and the second thing we need to do is we need to figure out what the answers to these questions mean for what we're going to be able to recognize the best thinkI framework that I have found to Reason through this is the cyber defense matrix which is the brainchild of Sunil

Hugh what it gives us is a way to understand what data is provided by what technologies in relation to the operational entities that our business basically relies on to operate so users devices networks and data and so on and it gives us a way to do that in relation to the key security activity of domains of identified protect detect exception we can do this for specific technologies that we have an iron in our environment to understand what data they are providing us based on how we've configured them and moving on from that we can also use the matrix to understand what data we have to execute detections on business critical assets using the data we have today versus the data that

we might actually need if we want to deliver that detection properly so here's an example of how to do that let's say we've done some appropriate threat modeling and decided that we want to detect super users who are trying to exfiltrate our intellectual property we can use the cyber defense matrix to say right well what are the data sources that we need in order to do that so we obviously need to identify the users we might need to identify the accounts for they use we might reasonably want to know what devices they use when they sue or pseudo in to a server we may want to know if we have a privileged access management solution in place what was

the reason that someone checked out privileged access and does that correspond to the logs we're seeing on the device that they're logging into that means that we'll need logs from servers and devices they log in to potentially applications and we might want to see if there are any network sources showing us movement of data from A to B once we've gone through that kind of exercise we can then zero in on the priority data sources let's say we decide that those are PowerShell logs and web proxy data what we can do now is we can for those data sources start to map our capabilities to make use of them starting with our dependency tree here

and I've gone with a sliding scale if we have no capability weak capability good or optimized down the bottom I'm not sure if you can see that in the back but that's what the bottom scale is so let's take an example let's say we do this for web proxy data tech we've bought great tech we've gone top right-hand quadrant we've got the best unfortunately we've configured it really badly or not at all so config wise we're going to need some work we do have a data pipeline that's great but unfortunately we don't have much bandwidth in that pipeline so we can't shift the logs at speed we're gonna have to batch them we're not sure

what that means for the SLA on the detection that we're going to be running we've got a great platform to put them in we can ETL them no problem but even if we do that there's some weak team processes in terms of the capability to get these logs and while there are some skills we're gonna have to train people up to understand the date from the alerts in our saw so it becomes pretty clear pretty quickly when you do this what you're going to need to shift to the right and this is where we're going to have to focus to understand what kind of effort and money is going to be involved in this and what kind of change

in our business processes might be involved and who are the teams who are going to help us side note we can also ask what happens if a technology gets swapped out or disappears so if the web proxy solution that you're using is going to be replaced with another will you get the same fidelity out of data that you did with the previous one and if not what does that mean for what you're going to be able to Tec detect in the data that you can get from it so if you do this for all your detection goals and then you do it for all the data sources that are relevant to those detection goals which sounds like a lot

of work what you get is a cross rest first set of your minimum viable data sets that you're going to DES need you need to deliver the detections that you have decided to prioritize based on threat model and respond so here's an example of what a road map might look like once you've gone through this now this isn't necessarily right but it's just to show how you can begin to pull this journey of acquiring datasets into showing business and showing other teams the kind of logic and the kind of progression we're going through and why we're asking them to ultimately give us the data and go through the effort of pulling it out of their systems and

putting it into somewhere else it also helps us show the business who often kind of want to get straight to Lego technics they're like yeah we want this super advanced thing it's like I think we need to start a do play I think we need to start somewhere a little bit simpler then we'll build up to Lego and maybe we'll get the Lego technics but it allows us to show the business that we're on a path to laying down those fundamental building blocks for the capabilities they're after if we're lucky enough not to have a flat network we could also segment this by different environments and what that does is further de-risk the type of data that

we're going to be looking at and hopefully helps us focus in on more specific detections on particular choke points in our business so good news we have now got our business case for the data that we want we've got our roadmap for the dates we want and it's time to go get that data any serious effort to centralize security logs is going to run into the challenge of bandwidth and more specifically than just bandwidth the Headroom that is available within your existing bandwidth and that is because when your business put down the pipes that it thought it needed to shift data from A to B large amounts of security logs were probably not on the top of the

laundry list this means when the cry goes out to have the data different teams will hear this message in different ways the network's team for example may feel that this his risk to their SLA so you're very quickly going to run in to a bunch of constraints and that means you're going to have to make a lot of decisions around those constraints about what data is stored where for how long what is moved where for how long and what if actually gets to your end state platform those decisions are going to be have to be made across what I call the three pillars of getting the data hosts and observers that generator intermediaries collection points which you may have to

use before you send it on to the platform where your analysis is running the decision you're going to make are more or less are going to break down like this and the reason that these decisions become important is as you begin to filter data out as it gets to your platform you have less rich content available to you for example in the event of needing it for an investigation and as charity major says in this wonderful comment if you don't follow charity well worth following on Twitter loads and loads of insights um you net you might end up with a situation where not only is the data you have not what you need but all context has been

stripped away when you go and look for that narrative in the data about what's happened in that incident and so that becomes really important when you start thinking about local storage what's available and what you're going to mandate or put in policy or request for those teams they're running these devices that are generating the data now it'd be lovely if you just had like one or two types of collectors one or two types of hosts but the reality is you have loads and not only you gonna have loads you're gonna have them across loads of different environments some are going to be in multiple on-prem data centers all of which have structured differently some are going to be in the

cloud some are going to be in the hybrid cloud which no one can really explain it's rather sounds like someone's just a third party managing a Dyson for you and then some will be in an environment they don't always like just stay away from there until the transformation is complete it's a very politically charged environment and doing anything in that'll be really bad so what we end up with when we go through these decisions is more like a capability that some multi-headed squid with tentacles reaching in to lots of data as opposed to having the power of one ring to rule them all and what that means is we need to break down the problem and we need to

do this for three reasons first of all we need to understand the complexity we're dealing with which is by what I call kind of finding out where the body's buried and why certain teams are kind of repaving their patio every three weeks um and a second is what is the second hopefully I haven't got my nose here

don't bother with a second and the third is figuring out well the system of processes is that you're going to have to use once you join this stuff together so when we when we go do this we're going to run into a bunch of teams that we need to go and ask for help and to do that it really helps to have thought through all the issues that those teams are going to face because we're going to run into stakeholders that may for some reason or other be unsupportive of our efforts and we need to be able to show them that we're doing as much work as we can for them to make their lives easier

rather than giving them a huge list of things to do and a huge load of problems that we want them to go and figure out it also does help on occasion to make sure that you have much better questions than people you go speak to so you can just get them get map your way so what we're going to do now is run through the key dimensions that we need to think about for hosts and observers for collectors and for platforms and then we're going to get into the final act of our presentation so let's dive in host first five things to really consider here number one the mechanism of the logging that you're asking for key point

here it's really important when we go and ask for data we're really specific about what we're looking for and the reason for that is if we say hey we want Windows logs we could be talking about native Windows loads we could be talking about antivirus alerts from agents installed on Windows servers we could be talking about the function that Windows servers provide like the HTTP servers and if we're not super clear and if we don't have a good way of structuring that not only will teams not be entirely sure what we're asking for but we will have no way to track over time what we have asked for and what we're getting across all these different environments

so we need a matrix of some sort to do that volume we're not going to be able to send data anywhere until we've worked out what kind of volumes we're talking about here is a handy calculation to be able to figure that out but the key thing we're going to have to work on with these teams that we work with is what is the time period of interest that we want to calculate volume for now it may well be a kind of something general like hey we want to know over seven days for each hour during those seven days how much this firewall produced by way of log party or it might be based on a

constraint that way what games run into for example how long a particular server can or a certain amount of logs before we have to write and batch it off to get to our platform we might also be further downstream we might be worried about how contended our collection server is going to be from all these hosts and observers at the 95th percentile when they're bombarding our collection server with logs sidenote extrapolation can be difficult different types of devices even at the same type produce different data and perhaps do that inconsistently so it's very easy to get this wrong don't do that it's also important in terms of how much data you think your collector is

going to be processing because if the team that hosts it puts it in an environment where it can't scale or can't actually deal with true volume of the logs you're collecting you're up a difficult creek without a paddle and sidenote side notes and there are actually lots of time periods to consider for example at the moment from data being generated that indicates something bad to it getting to the sock to deliver an alert or an automated message to someone they all need consideration and that takes a lot of time to think through collection mode we can push pull directly from indirectly from the key point here is sometimes you're going to be able to send logs

directly to your collection and server and platform from the place they were generated sometimes you're going to be dealing with an intermediary decentralized collection point and that depending on nature of your infrastructure can impact where you have to put your collectors to be able to move them to where your central platform is hosted be that in the cloud or on Prem that needs thought and it also needs understanding of what the traffic routes for all these different data volumes will look like you also need to figure out what's being collected where so if that feed stops you know what went wrong where in your collection infrastructure format there are lots of them the key thing here is understanding

what your system of analytics is going to have to process or transform by way of different formats so that they become consumable and usable by the analytics you're running in the back end and finally capacity capacity is really important because if as you will you decide at some point to turn the logs up to eleven you need to make sure that you can do that on the bits of kit on which your for example domain controllers have been built without melting the CPU and memory collector dimensions five things to think about here - now we've already talked a lot about store-and-forward fidelity so we don't need to go over this and too much depth except to say that if at the

point of collector you decide to filter data out keep track of what you've filtered out so you know what you can filter back in later if you need to forwarding agents the choice of this may be made already for you depending on the platform you've decided to use but the key thing here is just to understand will the collector that you're using be able to handle the volume of logs that you want to throw at it in the long term or not can it drink from the fire hose spec again with your collectors if you turn logs up to eleven they've got to be able to handle that here's a simple way to figure out if your collector is going

to be able to handle the volumes of data that you're throwing at it pick a top talking host turn all the locks tweak the EPS input to the collector of our config figure out where the CPU the memory the net the disk i/o ops collapses at increase sorry at fail increase spots required and then test and retest the bonus with this is if you've already got your forwarder installed you can test what kind of compression and what forwarding rate you're getting on those logs as well which is a double bound then you just rinse repeat keep going resilience I'm not going to cover this other than to say this is when it's really handy to have an awesome

Enterprise Architect on your team who really understands how to build resilient or her high high availability infrastructure and then throttling now it may be that you're not able to send all the logs to the place you want at the time you want them and that might mean that you have to batch now batch has great Vantage's to compression in terms of the volume of death for your you're dealing with but that may also affect the SLA that we can give on our analytics it may be fine that an analytic runs once every 24 hours but then again there might be other analytics where the SLA is more like matters of minutes like 15 minutes the

business would definitely want to know if that thing happened if you're streaming it's really important that you understand who has their hand on the lever of what logs are moving where and if for any reason they need to shut that off who gets notified if they do and what's the plan in order to then shift those logs to the central place that you want them so that you can catch up on the amount of logs you want and that's also the case on failover so if your network highly constrained and there's a failure during one of the few times that you can shift logs during the day can you ever catch up before those Lord get written

to cold storage over it nor deleted platform dimensions so I'm not going to dwell too much on platforms because it's not say and frankly what you'll find is that people have very strong opinions about platforms and those opinions are not always same so suffice to say one of the reasons that people don't like certain commercial platforms is the pricing model themes very expensive this is a great Fred you know every spunky install I've seen is utilized some method usually syslog-ng in front of the data collectors to throttle prune the logs defeats the whole point of log are archeology but as an alternative answer to this thread makes clear these problems are agnostic of platforms I

mean don't kid yourself if you're going to do this properly it doesn't matter whether you L do sumos blunt it's gonna be expensive you're gonna have to pay the piper somewhere and as this gentleman as this gentleman says these blind spots exist with lots of solutions not just with Splunk buyers often forget to plan budget for a lot of direct indirect costs especially fully tuned deployment 100% right the card a lot of ground we've gone quite quickly I appreciate this is the morning first track of the day it's maybe been there rushed it's alright we're just gonna recap reset and then we're going to get into our last act so we've talked about the need for a

team to manage data like a product and the need for them to do that management across all the dimensions that are going to contribute to what value we're getting of making our data available to ask it questions we've talked about the fact that the first of those challenge is going to be identifying and prioritizing what kind of data we want and developing a good business case for that we've talked about the second issue of then actually getting that data and having to make trade-offs and decisions of what we move where and what's stored for how long we've talked about the way that we can do that using the cyber defense framework to through the different data sets we

didn't prioritize them and we've also come up with a way to look at the dependencies that might prevent us reaching our goal and help us uncover the places that we're going to have to spend money if we want to make the shift to the right that's going to be needed potentially to differ lots and lots of different benefits through analytics not only to security but to other parts of the business as well we've come up with our roadmap which we now have we can show to other parts of the business we know what we're going after we've come up with an understanding across our different environments of what this is going to mean for our different collection

pillars we thought through all the angles and we're ready for any objections who might face so I want to finish the presentation with kind of a view on what why does all this matter to detect functions especially but also to wider security teams because if you go and ask the majority of farmers today what drives data collection the likelihood is what they're going to tell you is compliance or whatever came out of the box and if you've ever looked inside the cold dark heart of the data pipeline to your sim platform you like many others would have found it to be wanting so not many organizations in my experience go to the sock to incident

response and say what is the user need that you have and what system of processes does that user need existence day and what change would you like to bring because in my experience they don't say I need another 50 thousand alerts with no idea what systems those alerts are relevant to and no ability to triage them other than via loads of work what they say is I'd like to be able to search when I want so what is the status quo that has led to the situation that we find ourselves in as security teams trying to use data to make our lives better so to explain this I'd like to imagine that these buckets represent all

potential data that we could have across various different types security technology into networking devices like surfs and workstations apps and various enrichment data obviously we're going to have different types of data that all these things that are generating and within those buckets we're only going to be generating a percentage from zero to a hundred percent of everything that we might want to generate that we could get a hands on there might be some blips in that so for example maybe someone's doing some ops troubleshooting and bob has a syslog server on to his desk where he pulls data recursion efface troubleshooting needs but generally it's going to be fairly static though let's not forget about that server gets

important later on so there are a couple of implications of this first of all there's a ton of lost potential of what we could be generating the security realm and second of all we absolutely know that we're not going to be centralizing all of it so if we look at what that picture is over time in aggregate if we pulled all those buckets together it would look roughly like this some lost potential load stuff that's not centralized some stuff of questionable value they centralized and a few blips here and there for ops dip tests so let's narrow in on what we are collecting let's look at a time period of the last 90 days we're going to

probably have some assumptions about what's definitely available either locally or in a centralized collection point or where logs only might be available it gets a little more blurry and then it happens the incident the moment of boom and now you'll have to ask questions of the logs that we have available because it's time forensic eight the experts are being called in they know the data sources that have potential signals that we might need to look at in order to determine what the adversary did and where they went now those signals aren't necessarily going to be relevant but those are the data sets that we're going to need to go and get at short notice to figure out if

they are this is where horror of Horrors we discover we're not even collecting or locally for 30 days because during an incident a few months ago during troubleshooting someone turned off log collection because there was no policy that said it had to be turned on they just never reactivated it and no one picked up on that until the moment of boom but we can check our sim right so we run to the sim what we find is that unfortunately the sim only keeps data directly associated with alerts for 90 days all other data is stripped away at the intermediary decentralized collection points before it's sent to be processed so there's no time to sadface here because now is the

time for prep do those logs exist anywhere else our prayers are answered by Bob someone finds out about his syslog server and indeed there is the data that we need thank you Bob we won't let this happen again we've learnt our lesson we're gonna roll this data source into our sim so that next time we get the alerts set we want because what stings us this time is probably on the basis of sods law not gonna be what's things that's next time and even if it is it's not going to be in the same place with the data that Bob happens to be collecting it's going to be somewhere else in our infrastructure this is what

is perpetuating for many security teams the cycle of boom rinse and repeat so if you've invested in a hunting and you're wondering why your hunt team is failing to deliver on this amazing vision statement that sounds super exciting then look no further than what you've got side that this is also why any vendor that suggests this abomination in terms of running analytics to help you detect stuff should be instantly cast out of your office never to return so I think in security data operations we need to adopt a philosophy Aled Jim Collins that shoots for the stars and our starting point should be collect everything that we reasonably need by default figure out what that

data can tell us about the narratives that we need to know about from the point of view of the threats and risks in our business and then filter and it's by exception because how can we know if the field in a data source is relevant if we've never enabled it and never collected it so let's look at what justifies this so why do I come to this conclusion ultimately in security we need better correlation of nuggets that's what we're after and sure as craig chamberlain makes the point here there are going to be some things some some high fidelity alerts that we can get with just one data source and we should absolutely grab those with both

hands however deciding to try and dial down data collection is the wrong the right answer the wrong problem because what we need in order to develop better alerts that give us better activity from our sort is actually more data not less because that's how we're going to be able to tune our understanding of our environment over time and as Craig says devise intelligent alerts and that becomes clear when we look at the ways that we need to comprehend data in relation to what it can tell us about threat activity so sequences fairly obviously before we heard the boom of the Thunder we could have seen the strike of lightning it's really challenging to do that if the logs are

only connect collected in a decentralized way because that what that means is lower downstream when you see things that aren't high fidelity alerts they'll probably be either not got to because there's a 50,000 alerts and people are trying to get to or they will be just be categorizes are well that doesn't look that threatening doesn't know that bad is probably a false positive structure you know sometimes sequence isn't enough sometimes even with a sequence we can't for sure say that's malicious as opposed to just suspicious or benign and so we're going to need to dive into the structure of a thing that was alerted on and we're going to be able to have to make sure that data hasn't been deleted

in the manner in the time that we want to kind of grab it and look at it and finally says because if we think about different data sets so more or less the same kind of timestamp and how we may see indicators of threat activity in certain instances in different data sets we're going to see exactly the same signal and what is effectively the same time and that gives us a great point on which to triage but for other things we're only going to see it in one day source that's the only place that's going to show up and in other places at the same time we may see different signals of activity so we really need to

be able to pull that together through this method of collect everything that we need by default understand what threat activity looks like across that data and then filter exclude by exception because if history has taught us anything it's that we discard at risk that which didn't seem relevant at the time in isolation and later on turns out to be absolutely vital so with that thank you very much so to the punch line at the talk go manage data like a product I think we need to focus more on the data that technology gives us as opposed to the technologies and the use cases that we're being sold the science and the frameworks to do this now exist

so with things like mitre attack we can now really begin to categorize what we can protect what we can detect across vectors and exploits and really understand what we're able to do from the point of view of fast response to either new new new techniques in vectors that we're monitoring or not monitoring and new exploits that come out um this isn't just about ml this is about how the whole security team uses data you'll be super surprised when you go and do this in your business as I have for the last seven months all the other teams that come out of the woodwork and go like at last someone who's looking this does we really won that data too and we

want to help you do this and it's amazing um how you know you find loads of people are struggling with the same issues as we are insecurity about the quality of data that we have so if there is a battlecry go do this go start a security data Operations Group in your business go bring people together across these disciplines and figure out how we can all improve our lives so a big thank you to all the people that have helped me with this talk and contributed to the ideas over the last few years and thanks to you for listening if there's one I guess you know message that I can leave you with Keep Calm make the world a

better place and if you like to talk about how to do that with data um I always love to tell about this stuff so thank you very much indeed and enjoy the rest of the conference [Applause] all right thank you very much John that was wonderful as any questions anything anybody would like to cover I will bring you a mic thank you John that was a very excellent talk one question at the end you mentioned security data operations why do you think this is a function of security and not a function of IT in general um that's that's a great question um I think it should be a function of IT but and if you look at

what companies are trying to do as they shift so in big companies the shift that I see is people are trying to move from old things being done in old ways to new things being done in new ways which is typically called digital which is about changing the way that data flows between various systems and having a better understanding of that IT has been stuck in a lot of firms being IT management and they've been severely constrained with very little budget very little support and so often when you come in there isn't the drive to do that NIT so when you do get the opportunity to come in and deliver that kind of transformation doing it through security

and showing people that this isn't just about analytics this is about how we use data is a great way to do it and yeah maybe you know one day will be a night in in some firms it probably is already I just haven't seen that in the majority of ways so this is kind of the hack to do that and by getting other people on board with it it tends to work at least in my experience I think the biggest thing I've noticed when it comes to collecting all the data is PII questions you know when as a developer not in security but we would always have to have everything filtered out so you know a lot of that data is just lost

intentionally particularly as a GDP are coming up to is like no we don't want to keep that data because that's a toxic waste dump for us your thoughts on that yeah great question so actually I in one of the in that slide of the team of teams legal super important stakeholder um you can get access to that data you can process it you just need good justification for why you're doing that and what the risks to the business is and if you have those conversations and you're very open about it usually you can get to a good negotiation point and then in terms of how you then go on to protect that data and wherever you're

storing it so I found that that can be done and despite the regulatory requirements as long as that's explained in terms of risk not in terms of like security really bad stuff might happen and like loads of fear-mongering so that's been my experience at any rate and but I'd be keen to understand for anyone else you know what they've run into if they've run into other problems but I haven't I haven't so I can't really talk to it that's how I got around it anyway don't take legal for a coffee as one of the first things you do and make super good friends with them can you go back to your slide reference for the slides the Twitter one I may be

some time um was that the the Splunk one

right cxo sidekick on twitter um so if you search for that CXO sidekick then you'll find me and the slides are pinned right at the top sorry I can oh there we are that's an easy way to do

yeah like comments are on as well on that deck oh sorry it's not coming up now I've broken it so yeah comments are open this is hosted on Dropbox so if you've got things you'd like to contribute ideas critiques of the things I've written things you think could be improved like just dump the comments in there try not to be rude or I'll have to shut them off but if you do you know if you find this useful if you'd like to collaborate around it that was part of the point of doing the talk was to kind of Press print on a load of things that I wish I'd had as ways to think about this

problem so hopefully that's helpful but if there's anything wrong or that you disagree with I super like to know about it cool that looks like it to lunch oh no one more yeah notice what I think was between act 1 and act to you bet you had almost like a pretty in-depth kind of like a vendor survey is what I would call it questions however I did is that like a second tier because I think that I understand the in-depth but do you have like a secondary one like a some of your top 5 did you ask so I mean so they even get to that vendor vendor vendor list so that's what it was it look like

something that you were like a vendor so this is this is the questions right yes so so the points of these is so what with these what I found is first of all you need to figure out if you're speaking to a vendor that wants to partner with you or is in just scalar sell to anyone we can Dan the consequences and like if we get lucky maybe we get to flog you some professional services to deal with like the horrendous install that will go horribly wrong because you can't get access to the data that's kind of like one way to filter these out because you get really bad answers to these or they give non-answers good vendors will

either give you a really crisp answer to them that's like hey if you don't have this this and this don't bother doing business with us to do won't be able to give you any help other vendors will go I can't answer that question but let me put you in front of our chief data scientist and they will answer those questions and if they do that that's not that's another good sign so this is kind of a proxy for figuring out how much do these people want to help us because you will need help with this if you're just buying a product and you don't have a team like the one I run which is whose

job it is to go make this stuff work go and reach out to all the different teams you need to get you know the date if you EBA is like a classic example right there's a lot of stuff you've got to get in there and if it's not just going to be security who own the data see me like five or six teams so that's kind of the intent with the questions does that answer what you were asking those two like those two

so what attack means and what the tax services you cover if they can't very concisely articulate that game over the specific data sources if you can't tell me exactly what you need and the fields in those sources then you're just asking for loads of data and we're hoping we're going to tune it later to my mind anyway and then the next one about the visibility of the logic if I can't understand how you've built your machine learning how do I know what it was trained on and that there wasn't either bad stuff or wrong stuff on the training model if I can't get insight into that I'm basically buying a black box and I don't want to try and detect badness

through a black box because I can't understand how the thing works so those three are to my mind like pretty much if they're not willing to give you very frank honest answers about that then the flag goes up so yeah I'm losing my voice okay thank y'all very much Thanks [Applause]