How to Make Metrics and Influence People

thank you so I'm Luna I'm a data scientist at panacea I'll start my thing thanks for showing up I know it's been a long day so I appreciate you coming along so metrics security metrics are meant to help security tiers they're meant to help them prioritize actions we work out what the best cost action is and also provide evidence for security leaders to justify the decisions they make these shouldn't just be something we do for the sake of it just to measure something however over the past couple of years I've been working with and speaking to security teams at big financials and that's not what they've told me that metrics are to them from them I've heard that

metrics are at best useless and it was a massive drain on people's precious time so since we're in Sin City I thought I'd try to take you on a journey from analysis hell to metrics heaven so if they I'm going to kick off by talking about the poor soul stuck in analysis health the people so the people are weren't we had spoken to over the last couple of years we're going to hear from them about the problems they have with metrics why they're not meeting their needs and what they would really prefer to be getting from them then I'm going to move on to talking about the seven deadly metric sins and these are things

we've encountered when working with teams to build new metrics and yeah just kind of present you some of the the things that's good to avoid if possible then I'm going to move on to how we get on the path to metrics and how we avoid these deadly sins how's that I'll try not to maybe I suppose I'm paced around and okay so then we'll move on to looking at how we can start getting towards metrics heaven I'll present a case study some metrics we worked on with one of the teams and just kind of share some lessons learned things we tried things that worked and I'd love to get your kind of feedback at the end on what

you've done and how how you approach the same problem and then I thought I'd leave you with nice existential doubts to consider over your beers tonight and I'm going to pose the question how much influence should metrics have anyway so let's get started with the the people in analysis hell so the people we have people like I see so I spoke to who finds it really hard not to get stuck talking about numbers of vulnerabilities to risk committee but that's not really that useful doesn't really provide any strategic information then we have something like the bull manager she spent months just getting infrastructure and IT - so infrastructure and security to agree on the way they could report on patching

everyone wanted to plot different things I want to measure different things there was no agreement we have a nun list - spends hours now is producing a risk report but has no idea if anyone looks at it or even cares no engagement this is common if you're the person building the metrics you sometimes get no feedback as to whether any was actually using them or how they might you might improve them for for that person we then have someone like the head of security who kind of gets a bunch of information out of new tools allowed more alerts for the analyst to deal with we've heard about this kind of problem in some the machine learning course earlier today

doesn't really help and ask analysts already swamped in this case the date is not great because it's simply not practical they can't deal with all the sexual information anyway I know they have people like a CFO who just doesn't have the right information to kind of justify the the risk posture to governance I have no evidence on the data so these are kind of some of the things we've heard some the things have come up over the past couple of years people with all these various issues that need them to kind of end up in analysis hell basically not get much value from their metrics then we have someone else Camille on the optimistic data scientist who was possibly like me

a couple years ago and says hey guys don't worry about it we'll just do some really like robust math and it'll all be fine and this person is soon to you face a feeling of the thing I want to say today is that the road to analysis hell is paved with well-intentioned metrics we all know what we kind of want to get from this I'm going to be talking about the next couple of things from the point of view of myself as a scientist person kind of producing the metrics but I think that what I say should apply to anyone whether you're consuming metrics you're requesting them it's just all way about looking looking at what you want to get

from the analysis I'm kind of thinking a bit more deeply before we dive in and get the data and so next I want to talk to you about the seven daily metrics and this is how we got into an asset hell in the first place so I'm just going to take you through these again these are just a bunch of observations that I've seen when working with teams front about the metrics stuff I've maybe potentially you know since a pitcher committed myself once or twice and things when I think hey let's avoid doing that so I hope sharing them is useful to you guys so the first first deadly metrics in that's exactly the chart they asked for

so often people when you go to team and they ask you to build a metric they'll be really clear they want a plot of X versus Y color coded by Zed and so you could go away and build it in my experience you come back you show it to them and they hate it I think well you wanted that the thing is people will often describe a chart as a kind of proxy for the information they'd like to receive but the chart they describe isn't necessarily best way about seeing the question they have and either they'd scientist or unlist someone building metrics if you just go in immediately build the exact plot someone asked for

you're essentially just crossing your fingers and hoping that they've chanced upon the right visualization what you really need to do is pinpoint the information they need to receive and then it's our job to work out the best way to measure that and visualize it so we're gonna go back to drawing boards we're going to wake up while the person actually wants to know we're gonna go back to the data this leads quite nice answer next deadly sin of metrics just take my base you know personally I find this thing fascinating this is a classic data scientist era guilty as charged you get a dataset you think glasses the magazine can find all this interesting stuff and you go down a

bit of a rabbit hole well we have to remember is that this kind of project is is this kind of work it's not a pet project right there's a team that needs insight to do something with it and in general things like terms like fascinating interesting what does that even mean it's gonna be different for different people and so to present some in insight that's interesting to them it's often about thinking of things from their perspective phrasing it in a way that is kind of works with the the language they're used to the the context they have and beyond all that the really important thing is measuring something that somebody can change there's no point coming out with a bunch

of interesting stuff if no one can do anything about it because you know the whole purpose of this securitymetrics program we're trying to build is to is to drive change drive improvement I've got that Debbie metrics in three so great well it was gonna be actionable okay now you've got this thing like it's actionable insight they're gonna love it and the thing is making actionable is absolutely crucial but it's not about just finding every possible thing that a security team can change because what we have to remember for that team is a time is money they have got limited budget resources people to actually do stuff so there's no point just providing them with a massive list of potential

actionable insights and this goes back to what the quote we saw on the head of security before I don't want another big list of alerts my analyst to work I want know what to do the other thing is you need to get engagement right you to get buy-in so which actionable thing will actually kind of get traction more people actually start to work on this is quite tricky and it usually comes down to aligning the the action with the kind of key goals of the security team or the organization so you're gonna want to try really work out what is the best cost action what can the team do that will get them closest to their goal for the least use of

resources I mean that's pretty that's a pretty big topic so I'll come back to that a bit later but essentially actual insight although its massive buzzword is not the be-all and end-all deadly better seen than before okay so we've got an actual metric that addresses something that someone cares about great the next thing we need to make sure is that it's going to respond predictably to action that people take to to change the metric to improve the situation so metrics must clearly reflect improvement otherwise if you make a decision based on a metric the team goes away it goes away and acts on it and the metric just kind of oscillates wildly or whatever goes up when it should go down

it's no good but you'd be able to track these things so to give you an example suppose we want to kind of track our patching success and we want to patch as many bonds as possible we want to patch them fully and we want to do this as quickly as possible so we decide to look at for example average number of days to fully patch a vulnerability on this day by month and the team go and do a bunch of stuff we look this number goes down I think hey brain that's good isn't it I mean yeah potentially what we have to remember is that there's sometimes extra context around one measurement so we all

we have to remember this is a specific population of the data this is just looking at vulnerabilities that we've fully patched what about the ones that we might not have fully patched so if we look at a metric that might give us some information about that like this one number of bombs that were fully patched commit this has also gone bad so in this case we've kind of moved resources from one problem to another so we're patching more quickly but we're fully patching less I mean is that we're going to achieve it might well have been but the point is to be aware of it right because if you're basing all your all your kind

of your success and you're choosing your actions based on just one the chart on the left you're not nasty getting the full picture so it's really important to get the context around a certain metric think about what filters you've effectively applied today to what population you're looking at so it's really key it's really important to clearly define a target the action to achieve that and then work out what successful look like on what measurements you need all together to get that whole picture okay metric so number five okay so the opposite metrics are focusing on this stuff over here because they understand that but see so Julien for this stuff over here so you know who metrics

focusing on this now I'm not saying you can't do this but it's a bit it's very challenge because really what you want is the whole security team pulling in the same direction and if you're measuring a bunch of stuff over here for one team and bunch of stuff over here for the the person at the top who's going to potentially making their strategic decisions actions that are taken to kind of school well on the operational metrics well necessary of reflect in in the expected way in a high level metrics that are being used for strategy so you need some kind of common threads through the information now that's not to say that these these people are going to look at the exact

same piece of information exact same chart right they need different very different information so to give you a quick example of that you might have something like if we're talking about looking at antivirus deployed on assets we want to see okay how many are kind of scanning as frequently as they should how many are updating their virus signatures as frequently as they should kind of see so at least like a program manager level you might have something like this a percentage of assets that a policy broken down by region and device type I'll talk about like why we split it like that later on and then for the people actually going to kind of check

on these machines that are out of policy you would have much lower level information so basically you're almost getting down to raw records the specific devices so if they go and act on these machines the top level metrics will respond predictably and actually there was some some comments around this kind of thing in the talk earlier on day two NASA data visualization for for security around different user users different personas and highly needed different information and this idea of going from you know high level down to almost war information so through you I'm talking about the point is that the high level metrics are actually affected by the actions taken on this low level data so

deadly metrics in number six oh yeah we're like really confident that they dis complete this also applies to things like we're really confident that our AV like the previous example is running it's installed it's scanning it's come on do you software we don't need to check that it also applies to things like our patching process is really consistent you don't we don't need to review that it's fine and essentially the point of this is I guess it's kind of the basis of all the data analysis is don't trust verify so I would say measure everything but obviously if you're starting out with a metrics program you can't actually measure everything there's so much data insecurity so many tools but if you're

going to focus on developing metrics for a specific area then verify all the information going into making those metrics at least so again it's a specific example of this looking at kind of completeness of data stats there's that classic thing of do you have a up-to-date asset inventory and in the kind of maybe consulting approach it's going to be a one-off measurement do you have one yes but we should probably like check that a bit more frequently you know we want to look at these things continuously so it might be a five six months ago what's it like now and in a more kind of data-driven approach actually we look at some analysis we did

where we do across source analysis so you go and look at all the devices that show in your vault candidate the devices that show up in your AV and a bunch of other controls and you compare those to get all devices in your asset Ventre and you can start to see if there's gaps in your control coverage right but there's machines in my device infantry they don't show up in AV why not some better gonna fix it and you can start off assuming actually that your your asset inventory is great good quality it's like a we'd call it golden sauce right it's got all your devices in it and these other sources things like vol IV whatever may be silver sources

you might expect some devices not to be in a ether because you know you're only AV standing Windows machines or whatever or they're just missing however often when you go and measure this you find a picture more like this where the Gold's was ignited on source and we find one example this was about a third of all devices were not recorded correctly or missing from the CMDB which is considered to be actually really really robust and it's not a surprise I mean working on what's on the network is really hard but it's just like don't assume anything if you're going to use that date at the benchmark something go and check it so we are on the 730

metrics in pie charts I I have to admit I saw put this in Gia's because I wanted seven but I mean there's there is kind of a real point around this as well and that's you can do all the robust analysis you like and then if you don't think about how to get trapped with metrics that's goes to waste if you do the robust analysis and I think about how to get traction and then you put it in a ridiculous chart something like these courtesy of Twitter you kind of lose the impact and potentially it's quite it's quite problematic right because if you look at the chart on the left it's actually really misleading you know

this is a really great example you can see the column charts here showing the actual distribution in these categories and I mean by oh I certainly can't tell difference in those pie charts no if you're basing like strategic decision or something like this and look how different the data is underneath this is like you know quite quite serious point the chart on the right yeah I don't know but I'm not gonna say much about is in this talks we've had a bunch of more detailed talks about visualizations and what works what doesn't but you know just have to get in there so I just put up the slide with all the deadly metric sins on it as a kind of so you can see

them all together and so that's that's some of things I've noticed when I wake on to grab the metrics kind of pitfalls that it's easy to fall into and things that you should really try to avoid and so now I'm gonna try and helpfully tell you how you can avoid those and start to ascend to metrics heaven and I'm going to tell you the punchline right away so you're not like on the edge of your seat for the next five minutes the series of metrics Heaven starts with understanding security or IT process it's like really catchy and essentially what I want to say with this is that you need to understand the process that underpins

what you're measuring before you measure it when I say process here what I'm talking about is really things like you know we touched on kind of vulnerability and patching before so I say the process what I mean is first of all I guess like a policy or guidelines we need to patch vulnerabilities within X days than being first detected but then it also goes out further it's like who's responsible for that who who owns that who takes action on it what's the time scales involved who's reporting to who what's the kind of channels of information going around the organization so that whole thing that spreads out like how does this actually get done who's doing what and

who's in charge so some of the advantages of aligning with map metrics with process now this is just this is just how I see it right so this is just from experience what I found worked and some of the benefits you can get from doing this but I'd be interested to hear your feedback on at the end first of all if you line metrics of process it's much easier to identify the question that a stakeholder wants answered they've got a job to do you've group that now that there's this information flowing around they need to do X they need to report this information to Y so you get away from that whole you know do

me this plot of you know X versus Y color-coded by said you kind of know what the information they need because you've asked secondly insight is typically actionable if it's focused on process we're talking about process as I said it's who's doing what so if we start to to measure that normally what comes out of that is also actionable and it's really hard to change the way things are done in organizations right sure you're you know or well aware of that so we start on what how things are going at the moment how things work at the moment and try to move gradually away from that if required this is much easier much more palatable the next

point is it can help get people on the same page so we hear the quote at the beginning about a role manager taking months to get people just to agree on how to report stuff we wrote everything in process hopefully everyone agrees what the process is and so at least we're starting to talk the same language and get people on the same page and because hopefully process is in place to get us towards our high-level goals either for the security team of the organization that means that the metrics then will map towards map map to those goals so we're kind of worked out what people are trying to achieve and every to be pulling in the same direction so

if you remember the go back to 7 Deadly metrics since briefly essentially what we've done there is kind of by aligning with business process we've avoided about four of them the others are really about how you can sort the metric and just the whole pie chart thing so yeah but we managed to get away from four of them if we start to think about everything in terms of the process so key information to note down I give you quick intro to what I was talking about when I said process so use a bunch of things that are useful to to know that you can use to inform how you kind of measure things later in how you

visualize things later first of all who owns it is it vulnerability manager who's above them maybe their reporting into CIO or C so who takes action infrastructure teams maybe I mean you know is internal team who's in charge of them how do they break down who reports to who a lot of metrics will be for the purpose of reporting some of it will be for people to actually do their job and some of it is just to report that they did their job what timescales are involved so this is both things like we need to patch stuff within X days and also things like I need to produce a report like a week or whatever and then exceptions like oh

yeah we need to patch this within X days but not those two servers they have this own their own set of rules or whatever and you know you're you Metra's gonna have to reflect all these different different exceptions to the rules if they're going to be usable in that organization so now a couple of kind of tips or things I found useful when trying to understand intersect process so I've not worked in like a patching team I see my background is the the data analysis side so when I come in I'm working on these metrics I need to make sure I learn about and understand the process that the people in the SCOOTER team are undertaking and the first thing

that comes to mind typically is to go and like ask for a document where you know either where's the source of data through this process which is fine but the process is actually implemented might sometimes be a bit different to what's written down so my first suggestion is definitely ask somebody else what the process is how things operate all the things I noted down before then I would suggest you ask someone else and if really I don't really useful to kind of ask people at different levels so we talked about the metrics having that kind of common thread between like a CIO seaso and like a sac ops team so go and ask all the people that you're

going to be trying to produce metrics for how they view process you might find different perspectives different information team in one region if you're working with a global organization might do things a bit differently to a team in another and if they both need to use information then you need to know the kind of subtle several differences between the two next point is question of some and by this I mean like my assumptions sir someone's describing this process it consignment Libre and then you're kind of going and IO and them that probably happens we need to you need to really ask I found or asking people what they did like they've kind of daily actions

how they reported their often be a bunch of hidden information the people wouldn't think to mention because to them it's second nature it's like if you ask me how I do data analysis I'm sure I forget a lot of small details because I don't even think of them as things I do anymore so it's basically yeah really great to kind of get reading into details and work out exactly what's going on and just question everything you know and that's to help me understand what the process is and then just basically ask for because this when you talk to use first of all asking a bunch of questions saves a lot of time later otherwise you go and do all the analysis

you cannot the symmetric just sharing like are kind of use this because whatever expose out so ask now and save time later so in terms of with your collects all that information how you can kind of put that together and we understand what's going on start to map it to metrics there's a there's lots of different kind of frameworks and things out there so I just wanted to highlight this one gqm gold question metric I'm not really going to talk in detail about it except to say it's handy and there's a talk by heads in a moment from RSA a couple of years ago which is there isn't a lot of detail about this that's really

great essentially that the basics are you start off with a stakeholder or you know in this case we've got the C so you work out what their goal is so in this example we've got right reduce Chris by minimizing toilet vulnerabilities are open on those staked you come up with a question around that so how are we going to work out if we're going towards that goal well we can ask how many vulnerabilities are already been open for longer than that amount of days or or the inverse probably like and then the metric is is what you're going to measure to kind of answer your question and so this is really useful because I said before you you really trying to get

in order to have this common thread through metrics from operational data to high level you need everand have the same goal even if they look at slightly different information so when what we did then is kind of shoehorned a pee in there so process and so in there I've just put in this example okay fun ability you should be remediated by the infrastructure in we're the next days after face detection and this all typically be there should be this box should be actually pretty detailed if you've asked all the questions because all those little details like which team is doing it servers and workstations of different Americans do it different to UK that will make the information much more

usable when you present the final results so as I said before I'm going to talk about a case study so just a piece of analysis we actually did and tried to put what I've said into some some context so sticking with the vulnerability theme in this case we have plate let's see so they are trying to every nurse bring down the number of vulnerabilities on their state and when of challenges they were facing it was really hard to pinpoint what was driving their changes in that number so they take a bunch of actions targeted patching whatever following you know the process they put in place the number would increase or decrease but it was really hard to work out if however they

were effecting that action and so what we did with them is kind of sat down went through this policy stuff and came up with saying like this is probably like quite small on the screen and especially what I want to show you is that we just have the goal process question matter that we had before extra information and then we started asking other questions like things like there's anything outside this process what happens when something has been there for more than X days so we're trying to work out here is essentially does anything lie outside a process and what we found was in this case yes when something had gone through its patching cycle and had not been fixed it

just sat there there is no more process to do with it so that's really interesting it's not surprising then that kind of looking at that total number of vulnerabilities becomes really complicated because there's a lot of factors driving that total number so in order to kind of break this down a bit we decide to break up the kind of vulnerabilities into two populations so you've got the age distribution of detections on mistake and you've got all this stuff here which is kind of covered by the Pathfinder patching cycle this is stuff coming in new it's been there less than X days wherever that might be say 30 days we said like let's call out the

influx that's the new stuff coming in you can't really a fact that that's just new vulnerabilities being published it's still within your policy fine then we classified everything they've gone beyond that as backlog this is essentially stuff that was not covered by process it's gone through the same patch patch a cycle it's not being fixed it's just gonna sit around now and before they were measuring everything together now it's no wonder it's confusing cuz it's back what they're just going to grow and whatever you're doing in terms of improving your your standard patching policy is not gonna affect this right these are two completely separate populations and you're never going to be able to pin

down what's going on unless you identify you two separate the back so we've got influx and backlog we kind of just yeah when I decided to call it like this and actually having these simple names is also really useful because everyone would just refer to it like that oh that's it that's new influx think oh that's the backlog problem so I think that a use of language is really important as well one other thing we found with the go back to the influx for a moment is two types of influx if you like another area where they attend she wasn't policy sorry process covering it so basically this stuff is coming in new like this not being detected on the

estate it's only been first set in this state recently however what we found was in in that population there was one of us is that been published recently and vulnerabilities have been published say over a year ago right this is reintroducing old stuff onto the estate that is not as a completely different mechanism just stuff that's being published recently and that's kind of thing where you're making more work for yourself and again it's a category you need to track separately another it's the third category and if you lump all these together you're going to have no idea what's going on so to summarize we essentially identify two populations of Varner which there was really no protists in

place this influx old so old venerable phone abilities publish maybe more than a year the year ago yeah exact number there isn't really too important that would be reintroducing to your state and the backlog vulnerabilities that was still present after the patent cycle so I'm going to focus just on the backlog in terms of using this an example of how are you kind of built a per metric and I think one thing is interesting to note is that it's not just about measuring process it's about working out which things aren't covered by a process so you've got a goal and it not all your goals have a process to kind of match them then they're never

gonna be met so how do we align a metric with process if we look at the backlog we decided to look at the percentage of all run ability detected on the estate that are actually part of this backlog population we've it this is quite small to read so I'm just gonna kind of zoom in on this one section to talk about it in a bit more detail so what we've done here as you can see from that the larger chart is while we've broken it down by region and by device type the reason for this is the region relates to ownership I talked about before who owns the problem like who who does the person in

charge go and complain to when there's when something's not looking good on the metrics so in this case there was regional regional vulnerability managers so you have to break out by region so there's a clear line of reporting there in terms of the breaking down by device type that was do with the timescales so there was different different policies around patching for servers and workstations so again you have to split those out in both cases as well who takes action was also a factor in this so we had regional infrastructure teams and different teams fir for servers and workstations so there's actually worked out quite nicely because it you had basically two breakdowns that covered up off a bunch

of the points I mentioned earlier so it's very important to separate out all these things separate out timescales separate out the ownership the action and break down the data as much as possible so it's actually directly usable and just a quick note around a couple of the things on on in terms of the disease but as I say you know we've had a lot of talks on visualization already we use color to kind of show if agreed targets have been met nearly point one to make about this is that in something like this percentage of detections there in the backlog which as I said is essentially the percentage of detections that are not now covered

by a patching process I didn't want this to be zero but that's not the right answer because as it approaches zero we're probably gonna squeeze our thresholds right we're going to say okay that's actually now let's try and patched up within less than 30 days not throw stuff is tensile open security is you get close to the goal you move the goalposts so what we did was worked out with the team what what kind of percentages were were they comfortable with and then you can use color coding you know the the infamous wag status to see if your measurement is within the agreed threshold so green somewhere between Amber or if it's what you consider to be kind of you know high

risk now so it's red and because this is all about tracking and improving we're always showing the like a sparkline to show the variation and then the arrow there just shows the total change in percentage points over the period show I think they were kind of showing around three months of data but that's that's flexible it's important to show like where you are now on the way you've come from so okay that's great we've got this metric the next thing to do and this is obviously a high level metric but probably maybe Cesar is looking at they say write work stations in the Americas these are bad they know exactly which person to go to and ask about that they

can go with you infrastructure team that's patching the work stations say hey guys go and get onto this and of course underneath all this as I showed you before the a/v example is essentially they're all records so it should be directly linked with the action needs to be taken but as we talked about in the beginning having just a bunch of stuff to up is not always that useful so you want to start to move towards helping work out what type of action words would kind of push these numbers in the right direction so what we did was went down to the device level data and if we look at this here we have the new more

vulnerabilities detected on a device versus device rank so basically taken all the in this case this is the service all the service ranked them by the number of detections in the backlog that they have and stick them on this chart the color coding is software type it's actually quite interesting if you look at this these kind of plots a little bit crazy because I think there's back to put about 200 devices on here produced for a bunch of devices you'd probably expect if the build of device is the same initially and patching was consistent you probably expect the colors to look kind of the same but often they're really quite inconsistent such that can black and highlight

something interesting as well to check if around the consistency of process but in this case why we're doing this is to work out essentially if this isn't out by a problem do we have a couple of devices but for whatever reason have not been patched as well as the others or is it a systemic problem we've got a couple of old and spread over all the devices and as you can see from the kind of distribution here isn't that like a problem right we have these few servers that have way more vulnerabilities and everything else whatever reason so now I'm not saying this is like you know this is not the final word and trying to

work out what's the next best action but it starts to do the hint right we've got outlier problems so your next best option is probably to go and fix those devices and so when the team did that afterwards it looks like this and the number of critical vulnerabilities on the servers went down massively so just by identifying there's this whole backlog problem or population for which there's no process working out that it was an outlier problem with a few devices and then setting out to create a kind of campaign of action to go and target those great so this is obviously just let's say one case study to kind of illustrate the kind of points I've raised and this is

just based on the process in one organization I'll see process in every organization is going to differ slightly but what if I'm really interesting is that when you apply some a type of analysis to a much larger dataset like in the DBA are you see a similar thing so here you have a pop among the appendices in the dir and you have the percentage of vulnerability detections that been fixed on me Y and on the X you have the weeks taken to do that what you can see is that in every industry this this plateaus right so after about 12 weeks it just flattens off so this is kind of saying maybe this backlog things stuff falling off the the

back of a patching cycle and not really being addressed potentially it is a bit more it's a you know kind widespread problem and interestingly there's a great part in here as well which also looks at that kind of is an outlier is it systematic so looking at is it one device with many vulnerabilities or one vulnerability across many devices and you can see both examples here actually you can see so sorry to explain that the y-axis is the busy verbal sections and the X's devices so it kind of a column of doctors basically one device and so here you can see an example of both you have one device here's kind of purple column it's got way more vulnerabilities

than a lot of the others in here and you can also see this band of stuff here which is a bunch of vulnerabilities that peer across the loads of devices so you can see again of both examples so things really interesting that kind of aligning with policy in one specific organization or there'll be some details that will differ maybe on a you know there's sort of larger factors that are the same across organizations okay that's the end of the case study now I just want to move on to the final section which is the existential doubts so once you've done all that stuff and you feel like hey I've got some metrics people are finding it useful they

managed to take some action get close towards their goal that's cool sometimes you might do all that and that doesn't work there's a bunch of other problems that come up which as a scientist or analyst or whoever you are making metrics it's kind of beyond your most like beyond your your power to to change so I thought I'd tell you those okay so the first one is I'll only accept pie charts in a word doc now some teams organizations are really wedded to the way they report and it's not actually their fault right there can be a lot of internal politics about this you know again came out that Vaughn managed that took months just getting

people to agree on like a a chart and so it could be really painful and slow to change these things and sometimes people will want to but for whatever reason they can't management says no we have to have this standardized reporting across all departments or whatever and this is challenging I think maybe it's oh I don't have the answer to any of these by the way just I think in this in this case potentially over time as people start to see values from different ways of presenting things perhaps they can move over gradually but this is this is quite challenging the second point is we can't show any report with red on it and again kind of a bit of a politics thing

people don't really want to report on stuff if you have their you know the ragas there's no thing we don't want anything be red because I don't know then we're not secure and the problem with this is if you kind of end up massaging those top-level metrics that go to the board or whoever they're not no longer going to be linked to the operational data right so you do some great stuff to improve it but as max you'd hidden the fat that was a problem in the first place so I mean I get that we can't send stuff to board with like you know skull and crossbones are like nuclear waste signs all over but I'd

know I kind of feel like there's got to be some common ground maybe my hope is that as we start to move towards maybe more robust metrics and people can see that they can illustrate a problem but then be pretty confident that they can identify the action that will fix it they'll be more open to expressing that because at the moment from what I've hear people tend to find it really hard to pinpoint the action to take so then obviously you don't want it don't reveal a problem if you don't have to fix it so maybe this war maybe this'll improve final one is my gut feeling says its analysis is wrong so this part was

already my talk but then I saw talk gessie by John Knight about the the kind of human factor about the psychology behind decision-making and this is great cuz it's like someone has talked about all the actual research behind this point I'm said she wanted took away from his talk was that despite a bunch of information being provided when you have to make a decision your emotions take over and yeah you can see that right so when you show someone some like really rubbish analysis if it agrees with what they already thought they'll probably accept it no questions asked if you show some some really rigorous analysis but it goes against what they thought they'll often you know really scrutinize

and potentially just deny it completely so just say like climate change and leave that one there so I don't really know what the answers to these are but I'll be interested to hear your thoughts them afterwards but I think like thinking about these things leave me too it's kind of question or raise a question what is the philosophy around data in a team or organization I what influence that they actually want data to have so you can do data analysis one of two ways you can either use it to monitor and tweak your existing processes and that's kind of pretty inherently limiting you're saying we're gonna do it this way but I'll just kind of measure a bunch of stuff around the

edge this is the this guy on the left here mmm oh you can say you know what I want to be completely open till Dave's gonna tell me really think that's that's doing a way to get maximum value right to to accept that it might try to do things differently and I would question why would we invest heavily in data analysis for InfoSec if we don't actually want to reveal things we didn't already know if we know the answers then then why bother so the question I posed to you and think about your teams and organizations is do you want to be data justified or would you actually want to be data driven so I

that's the end of the talk just you can think about that over drinks if you want to know any more you can get me on Twitter you can also follow Panettiere who I work for I'm around until Friday afternoons so happy to chide loved to hear his thoughts and if you want to hear a bit more from me about the actual like math stuff I did talk last year which is on YouTube which actually has numbers in it thank you very much

are you open for any audience questions we still have a little time yeah yeah are you familiar with Gary Henson's work pragmatics securitymetrics and what do you think about it if you are I'm not no why don't you give me a summary basically he takes a look at a number of security metrics and tries to measure their maturity against a pragmatic framework is it actionable as a private eye is it practical is you spells out what pragmatic means in terms of each other stands for something that you want out of the metric and try try to give our security professionals the idea that they can mature the metrics over time yeah I mean I think that's that's a

really good point set of metrics that is useful for tracking what you do on day once the first time you measure stuff you're gonna find probably there's a lot of outliers a lot of hidden poms you didn't know about once you clear all those up so in the example I show you that backlog metric once you fix your backlog I mean you probably ought to check on it now and then just make sure it's not getting out of hand but it's not gonna be the thing you're targeting so absolutely I think Derby metrics for reaching a steady state and then metrics are appropriate for maintaining a steady state but thanks I'll I'll look at that reference thank

you thank you for you talk I just wanted to ask I suppose about metrics as it relates to sort of reporting up to the board and that relationship I guess between reporting and and metrics and metrics I guess being a part of a wider story what what have you seen work well before in that sort of more board level reporting so I think you showed some really good examples of you know how size owes neither a high or high level view and with you know the board really needing a high level view again yeah how does that sort of wrap into what you've seen work well in the past yes I think we've had pretty good success in terms

of some of the metrics talked about venomoth you being reported to levels like the risk committee then going up literally to the board is still something we're working on nothing else that's quite that's pretty challenging I think the key is what it seems to get traction is when you can present a problem and you can show clearly that you you know how you're going to act on it and then you come back for the next meeting and what you said is going to happen has happened right so I think it's about the people in management who are going forward to a board meeting with this information being really confident in the numbers and they know

that they can back it up in terms of the what you actually show yeah it's got to be it's got to be high level I think at the moment we're kind of we've kind of got to pipe maybe kind of Risk Committee level reporting but yeah the next stage is the next challenge I would say

maturity in what sense like using a framework to sort of measure maturity of oh I'm not yeah like something like nice but with a capability maturity model okay no I mean so certainly people are we see more and more trying to align all the metrics finesse and we're looking at that as well at the moment I think you know that's the way things are going property gives it a framework to put that in I guess it's what we've kind of talked today at the moment is making sure things are actionable and I think frameworks are going to be more like increasingly important possibly even compulsory so yeah it's definitely an important thing to consider so I have a

statement and a question so I'm actually a senior executive c-level and right now one of the biggest challenges that we have at least getting information from people is if you abstract it out enough that it becomes meaningless it's nothing that executives can actually act upon the statement I'd say is before this position I was managing the continuous diagnostics and mitigation program for federal agency and working with the dashboard this is supposed to be their panacea to report of all ability and configurations and stuff like that and one of the core issues with that is while the tools that gather that information are extremely detailed and it were extremely useful for the the first-line responders what has been

designed in the framework to roll up eventually if your report to the White House and DHS and so forth has been extracted out and determined basically meaningless it's a meaningless metric and I would advise if anyone's in that role in the private sector is to avoid the pitfalls and problems that of the federal government has created for themselves in that respect so I guess take that as a lessons learned but this was very good but I'd say if you wanted a counter-argument is to look at that program okay great thank you do you have any specific examples of a sort of metric that became meaningless interested well generally it as the data rolled up from the tools it began to

strict more and more detail that was relevant it got into just things like counts so as you're talking about with the way the graphs were you're getting these these meaningless graphs and trends that had numbers assigned to them that didn't tie back to their sources so you couldn't actually make an argument that you know basically causation and there are no ways to basically if someone asked a question of why is this doing that you couldn't actually trace that back into where that was generated and then act upon that yeah so as I went through the multiple levels you just had that inability for that person receiving that information to basically make a decision and then have an action upon it

yeah yeah it's definitely challenge to keep keep meaning as you go off I think so yeah it's interesting to hear that's gonna poke me also as well like in the additional questions we would like to thank Leila power for an excellent presentation [Applause]