Speaking Metrics to Executives

and he is speaking on speaking metrics to executives so Michael okay so just quickly about this is not gonna work so we'll drop that I had breakfast this morning and other things going on it's like okay here we go and it works but

one level said as we get started first of all if you're looking for this you're at the wrong track there's a lot of really good stuff at bsides and an awful lot of it is technical this isn't that session but the reality is sometimes we're going to have to take all that good work that's happening on the technical side and be able to communicate that to leaders so they know what we're doing so they can see the accomplishments and honestly so they can fund it because otherwise we don't get the tools we need to do the job and so if you were thinking the hacker track get anywhere else around you'll probably find it this morning we're talking about

leadership in talking with leadership and that's kind of the whole focus of here briefly about me I've been doing information security for 25-30 years I started in the defense industry then join the Internet boom and kind of wound my way through different organizations along the way I got into that thing called management and I really enjoyed the opportunity to get people moved up and equip them and part of that was learning how to talk to the senior leaders in the organizations to support the program so a lot of this is going to be focused that direction so let's start into it when you go talking with leadership to be very frank they are not interested in the technical details or

tactics that you did they don't spend time with hackers they focus on strategy and so we have to be that bridge between what's happening on the technical level and the next step up we have to be able to speak and and hold ourselves in a manner that allows that communication to happen and so this session is really intended to work on one of the things I find a common tripping stone something I struggled with a lot which is how to take metrics numbers that we're very used to and get them to communicate a meaningful message to our leaders I'm going to show you some examples some bad some better and as we go through in the

talk not necessarily what my numbers are they're not all that relevant but how you think about presenting them so you can prove whatever is you're going to present so with this let's start here please don't do this this slide or something fairly close to it to protect the guilty this image literally in the last six months was put in front of a board of directors it didn't do any good it's large meaningless numbers and they got no reaction from the board they thought this is going to be brilliant they got no reaction from the ward other than so what's your strategy we'll see you again in three months and we want to your strategy not not the effect now we

know the solution for that obviously you don't put the graphics up you just give them this instead please don't show raw numbers and I call them this way because these these sort of charts have a numbing effect on business leaders eyes just glazed right over and there's no business context you lose them and you spent time gathering this but it's worthless to them the CEO and the other senior leaders that you're going to interact with when you get that opportunity vix they expect you the security leader to address the daily security issues all that operational stuff and so for you this is valuable valuable information for them not so much they expect you to accomplish

commission whatever that takes and so you've got to kind of separate that out and when you bring it forward bring forward the business issues which these metrics represent typically that's going to be graphs what sort of graph it's up to you there's all sorts of different ways of graphically presenting numbers but it's probably going to be in that form and as you do that it's okay to summarize and it's not quite as detailed and precise but make sure you know these numbers behind the graphs when you do present because some of those people are pretty darn sharp and if they ask you have to be ready for it with that maybe you give them something like this instead here's your

typical presentation not too bad but there's some pit balls we want to avoid on this chart so quickly this chart still gives no real information to an executive how much spam didn't hit the company it doesn't provide any insight not by itself is there a call to action here that you want me the board member to do so you get some support to get some real business impact we're going to have to explain a little bit more about email remember the business doesn't care about how you did it what the external factors are that the business can't control not that much and they really don't care exactly how much they want to know about how much they do care is

email being delivered is the spam and the malware generally being blocked it's minimal or none they like none thank you very much and what are the trends so we're try again we'll give them maybe this slide a little bit better quickly taking a look at this we now have a little bit of a story forming up because you can see flow this tells that if I didn't have good Melva filtering in place the volume of mail hitting the individual users would be double and to be a whole lot of garbage in there okay and that doubling of email half of which is spam would not help the business so now I start to appreciate your spam filtering I care now not

because I care about your spam filtering but I care that you're cleaning up something you're you're doing a business valued function to make the business more efficient so we're starting to talk in business terms but we can do better than this there are several flaws with this it probably would best look maybe like this talk through a few points first of all let's get the colors right delivered mail is green that's good shorthand it for them because it's not that they're stupid but they don't have much time so make it real accessible the stuff that we want to say is worrisome is in yellow with a nice red border around it those are cautionary colors

we've also thought about on this chart the differentiation of how its presented on this slide it looks great what if I print it in black and white that previous one dark green dark purple will come out dark gray and dark gray this will actually print out on a black and white printer still be readable Oh what if they see it on a tablet more often than not if you're going to the top leaders think about how look on a tablet this morning I had a wonderful experience i had my laptop all ready to go and I said all right here we go to be sides wait a second my laptop has hdmi I wonder if they have hdmi

I grab my iPad guess what they they don't have a whole lot of hdmi interfaces guess what happened to my slides yeah they got they got a little munch so I had to go fix them so think in advance how your material is going to present on the form factor that they're going to look at doesn't matter how good it looked at beforehand matters what they see okay enough on that we're not really here to talk about slides in email in particular we're here to talk about what I call numbers this is you got it this is the picture this is your executive there is your data and he is enjoying it because you present it in a

very palatable way for him to consume it and use it right away all the information has been brought into context everything is pointing to a trend or towards a solution that you're recommending in a nice type form it's not a puzzle for him to go figure out that's not what he likes to do and you got kind of this immediate consumption approach these are the key steps you've got to walk through when you look at whatever you're bringing to your leadership is it actionable that means it creates an urge a push to do something now the leaders are going to be driven by two things they have a long-term strategy and they're concerned about risk at a

tactical level because it messes with their long-term strategy and that's it make sure that whatever you're doing delivers towards those ends it has to be presented in a clear form has to be easy for them to absorb everything in the detail level has been summarized when I showed you these charts the complexity is removed that doesn't mean it's stupid sometimes you have to work these very hard and I'll show you kind of a harder one in a moment but we've gotten the complexity out of the presentation so we can talk about the idea anything that we've got is should be trend focused you gather data / x consistent periods consistent way of measuring you know how

it's done up they ask you and now when they start looking at that they're going to look at the trend lines they're going to look for variances that's how they think they have this accounting mindset and as they drill through that you'll present in a way that shows them trends so the conversation is easy and finally everything's got a story behind it the things that you probably remember about field trips as a kid what happened in family history all these things have stories around it so make sure that that you can take the essence of what you're talking about pack it into a story so it delivers easily and they can take and process it I don't just throw the slide

up there and say oh email we got problems spam I need more money you're probably going to talk a little bit about how it impacts the business and when you wrap it they'll resonate with a story and they'll go after that ultimately these are meaningful metrics they're successfully delivered they're successfully filtered how we get it ready and and then we go forward with flows and trends and we haven't asked so I want to go into something that I found challenging to explain it's a little more complex and this is our favorite thing to deal with perhaps if you if you've led a team where you deal with the server admins and that's called vulnerabilities it's so easy to go ask

the CEO to help solve your vulnerability problem the challenge is the security team us we're going out there and and you know what it is out there you've all seen it and the sysadmin to like don't waste my time you come you make me patch i patch it's a waste of my time next month you want me to patch again I don't understand you people it's a waste I've got better things to do and manage me has to try to sit and listen to these two arguing sides and figure out what to do with it so we're gonna we're going to lift this whole argument up and make it a lot better because otherwise you will

end up like one of my other four friends here that just never managed to get across this gap okay I want to enable you to show and if necessary dr patching success if that's the thing you want to go do two very hard activities one of them is about the patching itself getting people to do that and the other one is the configuration there's a whole lot of detail and other people that can talk much better and give you lots of great ideas about that I'm just going to talk about how you gonna get the board to fix the problem we're going to focus on risk not the technical what the Volm is it doesn't matter ms whatever a loss

whatever we're going to drill down and make sure before we present anything that you get your charts together and you share them with the system admins and you share them with a CIO because no surprises is good and once you've explained and you've argued through the methodology and I go well yeah I guess it's a fair way to represent the numbers then when you present the numbers and the CIO still doesn't like it it's too bad he at least agreed to the methodology here's the approach I took because I've seen what spits out of the tool it's usually a number that says you have 17,000 150 vulnerabilities okay what I do that we're going to take a

little differently to break it down I thought with some help the team's I work with to compute a risk score that's not so abstract we know that the risk scores often are calculated on on how how big the vulnerability is maybe there's different scales and they give you a number out of it I like grading grading is it's like school we all kind of get that we either painfully remember our grades or maybe if we've got kids we're painfully dealing with grades but we get grades they're pretty easy to understand they they communicate a story very quickly and clearly and they're discreet you got a through f and you're done so if you put things in Nice

discrete groups you can count them and you can start determining who our low performers are in the classroom and we're going to solve that problem let's march into it here's what I did you can disagree your own formula go right ahead but basic idea run our scanner tool comes back and says here's I found so many criticals highs mediums lows I'm going to assign each weight somewhat arbitrary you can play with them the fun thing with this is I guarantee you then no matter how much you play with the waiting's if you've got problem servers they'll still get enough go ahead and play with these numbers I don't care you work at to tune it until you satisfy

your sis admins this one came out actually not bad and while you might want a perfect server in the real world okay 1.high not happy about that medium low not bad get to be okay fair enough and maybe you've got these servers laying around this isn't a whole lot worse but it's got a critical on there we know what that means the board doesn't button this scaling score this is an F this is a problem server now if we start taking all our servers we score them and then we say how many do we have that RS how many d's how many bees we can get something that looks like this the specific remediation actions we can

talk with this the engineers about but this is a leadership discussion we're having since we've counted them top leaders look at this they see trend lines and they can decide whether this threatens their business strategy or not right they only care about two things and vulnerability scores ain't one of them but here they can look at this and go huh okay and be sure when you present this that you're ready to explain what are the resources left to fix it because they're going to ask you and you have to be able to say okay I'm going to need this many man hours or project alpha that we're working on it's taking up all the time the system engineers honestly

if we could just slide that 30 days and give me full focus we will nail this whatever the resource requirements time people money licenses make sure you know what that is before you go and stand before these leaders because gonna say what do you need and when you answer crisply they're going to turn to the CIO this is the fun part they return to the CIO it's a hmm we've all you told me months ago we couldn't solve patching and now look at this trend line we're getting pretty good here even though the number of servers is increasing in the environment right we can see the height of the bars we the board of decided we don't want

any more expertise in the environment so in the next 30 days no read on this chart and by next quarter we want to see the the everything kind of in the scene above we're targeting we're happy with a be and you look across the CIO and smile okay so there's an awful lot of stories you can tell with metrics okay I'm giving it to there's a whole lot more I would love to hear from you I'd like all of us to be talking with each other about meaningful metrics that drive action of the business you have to try to change your mindset when you get ready to walk to the other side of the hall and think

about stories and business impact but if you're doing that you'll see your program suddenly surge forward because you're talking their language the CIO will no longer argue with you because every time you bring a new metric to the table it's just so painful not to comply with your wishes think about the numbers you can collect you've got to be able to collect something it's got to be something collect in cycles and it has to be collected consistently so that it's defensible and you have to be able to explain it and then when you get all those numbers turn them around over and over try different ways bar charts whatever talk to other people go talk to

marketing they're so great and making things look so pretty but get their numbers in a form that the story comes out and drives the direction you want it to drive and then just get it down real tight because you're gonna get about five minutes in front of the board if you ever do talk to the board put the slide up give a few bullet points and in two and a half minutes you're done and you've addressed a very significant piece and when they kind of sit back into chair and ago hmm so what's it going to take you win so that's the direction make sure people want your numbers make them tasty make him presentable and when you

do this you will start to influence your company's actions you'll have more success with your program and I think you'll find it much more satisfying to be in this very challenging industry that we're in so with that you can find me on Twitter I'm just one ping I would love to hear from any of you if you've got some ideas about other metrics that we might want to talk about I'd love to continue the discussion on Twitter thank you

I think it's on just little little I go ahead okay thank you talk thanks for the suit the lambert Iseman doesn't talk tomorrow at five o'clock and go to steeper into scales and weights and all that stuff so you guys interested 90 deeper question to you is how do you deal with executives you want to skip dash we were all the metrics of blood screen so forth yet telling a story with it seems to complete with the idea

dashboards yeah dashboards are real challenging because they do boil it down until you end up with mush we need to be very thoughtful about which metrics we pick out because when you pick out a metric to present at a high level there's so much you're going to lose we're going to pick a very narrow one that just too narrow look at what drives your company look for the metric that matches it if I were I work in hospitality so in that space I might look for metrics that either reflects satisfaction like if Wi-Fi for our guests isn't working that's a technical problem but it matters in the business I would avoid something real esoteric that

that I can't easily peg back to the business and then maybe when they drill down the next level you'll have opportunity to open it up think also about things like reputation reputation is really hard to quantify but an awful lot of what we do can quantify that okay thanks thank you and the other talk to you any talk that goes into metrics how to get deeper into it not go after those talks and see what you can learn yeah really good what are your thoughts on GQ m approach to metrics with executives I'm going to beg ignorance a goal question metric approach where you start with the goal develop a set of questions for that goal

and then metrics that answer the questions

the question tell you set goals I think if your goals if you're talking very broad like business goals and it talked security goals how do you get your security goals lined up to the business in the first place if you do that when you're when you're setting your goals and thinking about what metrics you're going to use to support them I think it works back down the other way but maybe that's not what you're asking about I found that in the security space we're not very mature yet and how we talk to top leadership and so if you've got a good methodology try it if it's not working be willing to change it and reach outside of info SEC people I

remember very serious talk to marketing talk to someone on one of the business lines and and see if what you're doing makes sense

okay there's gonna be a lot of great speakers around today as you're going to these different sessions every time you hear numbers out there you're hearing it from some of the smartest people around grab that but then ask yourself is this one of those numbers I can turn into a number and then this serve it up see how it goes thank you very much [Applause]