Metrics That Matter: How To Choose Cloud Security KPIs For Your Business

um hi everyone uh thank you for taking to the time to join me today um I know it's late afternoon I'm trying to be fast as fast as possible to go through a lot of information um I hope you all enjoy the the the the conference so far today um I heard that there were a lot of uh fantastic talks so first of all um who am I and why I'm here today so I'm my name is Emma and I'm A Cloud security architect at epam um so as a consultant I love design and architect everything in uh in Cloud for my clients eam is a sec a security well is a technology consultancy I would like to say security

but specialize in uh in software engineering uh product design and we help our clients to accelerate their digital transformation programs um alongside my work I also volunteered rering at um women in cyber security wies UK and Ireland affiliate and I'm also one of the um um elit uh and volunteering as elit um in the in in the in the community I'm also a uh Advisory Board committees for the computer science faculty at University of um backing so today I'm talking about um metrics so first uh before I go into my agenda today I would like to um you know I would like to set up three questions that I'm aiming to answer today why are

metrics important in the conversation um about Cloud security what metrics matters to you and your business how can you establish effective and meaningful uh Cloud security kpis so please keep these questions um in in your mind by the end of the talk I would like you to tell me whether my talk answer those questions so this is the agenda for today um so we firstly we will be discussing about the challenges that we all exper experience about metrix as a developer and Cloud practitioners then I will move on to uh some techniques and best practices and I will share a couple of examples from my past projects uh where those techniques has been used to

establish key kpis and finally I will wrap up with an overview of the prioritization approach to one beity remediation uh this is the yeah this is the um the a quote that I got from somewhere by just Google search what is cloud security metrics um to be honest Cloud we we we now have um you know predicted to be 60% 64% in the public Cloud maybe higher so when we're talking about um Cloud security metrics it actually covers multiple domains um instance response identity access management software development and so on so I don't think there is a specific definition for the cloud security metrix but just keep in mind that's everything to do with cyber security because we are

now living on the cloud so what are the big metric problems so in my experience with our 20 customers in my you know my project lifetime um so nearly 90% of them have PR PR ize with and priortize security metrics as part of the continuous improvement process but they all struggle to take an action especially the Senior Management visibility so why this is often often the case so I've outlined some of the uh the this the the the business challenges to do with metrics so first of for um we often encountered the misalignment with the business goals and sometimes business have um limited resource to deploy to uh to meure and collect the metrics so what are the gaming metric so

gaming metric is something that you know you track the metric to make your posture look good so it's it's not really yeah it's not really representating representative of your Sho environment and um and another another challenge is we have evolving uh St landscape so um metric that matters today might not matters tomorrow and um another another challenge that we have is we have a lot of tools in our systems and this can complicate our metric um and the measurement of our security so these are all the factors contributing to the um the failure of getting the metrics done in the correct way so I'm trying to I'm not trying to solve this issue today but um I I want

to talk about it and uh talk about a few approaches that we can adopt at least to start getting our metric correct so um so why do we track metrics um these are the you know the business drivers that I given on my page I'm not going each one going down each one of them but I want to emphasize the point that metrics we want to measure Effectiveness we want to improve the our security posture we want to identify gaps in our uh security team uh we want to identify the gaps in our uh security toolings and so on and finally you want to check your security posture uh Trend over time over time it's not just now it's not tomorrow it's

over a per of time to see how much you have improved in your environment so so why why do we do this it all comes down to business risk reduction so we want to reduce business risk so think about metrix security metric is not just security it's all about business right at the end of the day um your business pay for pay for the security to to get security uh to get the a risk remediated so it's all linked back to the the business I'll talk about that in a minute um so before we deep uh dive deeper I want to clear up some confusions around metrics and kpis maybe some of you already know what are the

differences but um metric is uh simply a measurement a snapshot of what happening right now however kpi on the other hand is go goal oriented so they track your progress towards a Pacific goals um think of think of them like uh uh you know the fitness tracker telling you how close you are to your daily goal um so for example meantime remediate is a a metrix and it's just a number but um if you say that reduce that number by 25% within the next quarter that would become a kpi so that's the important difference between those two um so there are a number of ways to design your metric program on the page here I've

outl three h seven sorry seven steps it could be different for each organization but I think this kind of seven steps of life cycle can get you started to to start thinking about what is your security metric life cycle look like in your organization so I will go down into details of each of those steps in this session because I don't have enough time because yeah so um the bottom line is I I would recommend to investigate and into further into how how you're going to Define and follow a metric programs okay so how to identify metrics then uh there there are a few question that you can ask your business to start with who are the target stakeholders

what value are the metrics bring to your teams and the board um who to share the metric with if if you sharing the metrix with a cloud Ops could be different to sharing the metrics to a reporting to the board and actionable outcome you want metrics that don't just measure but also provides clear insight into where security can be improved and what are the tools you have right now and that you can utilize to you know to collect and track your metrics from your current two set so so and and then um you need to think about okay we we have defined those metric right now can can we adapt those metric to a future uh Trend

because we have because we talk about the the threats because threats changing all the time so we don't want to fixate on one metrix and then it doesn't work for tomorrow and finally um it's very important that our metric tells a story um because we want to ensure that um the metric reflect our Cloud security posture okay um over the past years a few key metric criterias have been developed some of them are generic and not just for security metrics but these are the framework that I kind of research into um but I think they can all be adopted depending on you know depending on the preference of your company so I just going to grab them on

my page here um but feel free to look into them so um how to establish key metrics which slash kpis um so I have to approach I mean yes I investigated into two approaches first of all is the top down approach so you started with um thinking about the bigger picture uh What are the business goals and drivers and what do you want to achieve Revenue a customer satisfactory or Regulatory Compliance and with the information then you can defin uh what are what to measure and then you can defin what are the metrics um so and at the end of it you identifying the the the tools that you can use to collect and your metrics so

what is a bottom up approach is um is basically the other way WR so you look at the different data points in your tools and then you identifying okay what are the data that I can generate from those uh from those tools and what other the metrics I can generate from those tools and then you work your way up to your business gos and then you ident at the end of it you identify the relationship between those metrics eventually identifying what are the key metrics and kpis so for this one I've been using um The Casual Loop diagram which which helps me to identify what are the you know the key uh the key kpis

for my company I like this quote that um um that that I got from sneak uh a white paper um there is a a research has been conducted by this by sneak to build a metric model of marking relationships and differences between different different data Elements which is the C Loop diagram I'm going to go through it okay it shows everything okay um so so why why C Loop diagram is important in identifying your key uh kpis so relationship between the metrics can help um to understand the the dynamic between them so go so you can go beyond and looking at different uh different individual metrix and explore how they interact and influence each other so this is a visualized tool that

I can use to establish those relationship so just to go go through some Basics um a plus sign in the in the in in between the two two different data points uh means that there is a direct uh relationship which means the increase of One Direction leads to increase of another Direction and on the other hand the minus sign it means that there is an inverse relationship which means one increase in one point uh could lead to a decrease of another it might look quite complicated to start with um so so I I also feel the same but let's take a you know a closer look at the um the diagram here so the connections between the um

the DAT the the number of security uh instance and the meantime to detect indicating a balancing Loop so why why is that because a high in uh instance rate could lead to the detection rate um to be uh longer because it could affect the detection however this could change over time when security teams um increase their efficiency this could lead to a faster detection rate so you can play this game you know forever because because the situation change over time but the the the idea is the Casual Loop diagram is used to represent the the um to to use to represent the relationship between the two data points so um on this page I'm looking

I'm going to look at how to use the Casual Loop diagram to prioritize our metric um on the left hand I have a typical example of infrastructure code security pipeline workflow so some some of you might be familiarized with this if you don't it's fine um it's it's kind of a simple and high level at the at at in the diagram so the the infrastructure scod or polic code templates are normally stored in a GitHub report or any report and then you use that to automate your uh infrastructure provisioning so every time when you spin up your infrastructure the the policy is applied to ensure your environment is consistent and compliant and security scan is used to scan the template for

availability if detected the resource owner will be notified to fix the issue so how does it apply to a a cat Loop diagram so I collected some of the key metrics from that diagram for example number of vulnerability detected number of uh vulnerability fixed and so on but if you notice on my uh casual Loop diagram policy violation rate and average time to remediate they all have multiple um multiple uh data going into their Direction but they have zero or one outgoing relationship which means they might be the you know the best candidate for your key uh kpis and and they could representing the you know the final the final the final uh kpi that you would like to focus

on now I would like to talk about the protection level agreement so um jumping from the c c Loop diagram to the protection level agreement is actually the other approach so protection uh protection level agreement is kind of like top down approach because it it kind of uh make you think about your business your business um needs and your business requirements and then identifying the the risk uh your your risk tolerance and the cost of protection so so what are the you you you to think about what are dri drivers um that that you can drive towards the security outcome what other outcomes that you want to you know you want to have in your business so um yeah so this

technique was researched and published by Gartner uh is a tool provides a way to translate security into uh the language of business it's like a service level agreement if you want to put it in in in a more familiarized way um but an agree but this is an agreement of protection between the security team and the business stakeholders or leaders so the business designed amount of budget they would like to in invest into to uh align their efforts to and um the security deciding what other level of protection they can offer for that budget so this is crucial this is crucial because it's used to balance this business needs and the security requirements essentially what would you

like to protect how how much would you would you willing to invest into protecting our um our business right so on the next page I'm going to dive down into um the different stages of the uh of the protection level agreement how how it can be implemented to to establish your kpis so this is uh five step approach and so firstly you want to figure out what really matters to the business so you think about the business driver what are the what are the absolute masses how much risk are we comfortable with knowing these answers set the direction for everything else next we can look at what are This Cloud access that we want to protect and what are their

criticality normally you you can use a inventory tool to identify those uh mentry and and then you identify what are those um Crown juice so you can decide what are the protection level that you want for each of those risk level for those um inventory so this is not a one-size fe or situation so you need to think about that you know criticality a little bit more then you map the security controls with the identifying protection level for each of those risk level it seems to be quite complicated but actually it's really easy to you know to understand at the end of the day because think about it is it's just you just want to

establish different protection level for different assets different CR for the for the different criticality of asset and at the end of it you want to establish the pla using the outcome driven metrics um now I'm giving an example of how this works so in this slides I use PR way to uh prioritize remediation investments in this example consumer markets um have um different uh kind of systems you have critical system you have other systems you know internal systems so for each type of system you have different um different patch patch management um you know day to patch uh Target for example for the critical system you want to aim for 35 days or 30 days of PA that is your protection level

and then what are the observed outcome so the the the number in the r here indicating that we actually observing 40 days of um or 40 days were actually required to patch the vulnerability in this distance uh so the in this systems so this indicating a few gaps so what do we do now we identifying that the current security budget is not enough uh to patch the system within the agreed time line and then we have decisions that we can make available to the business right we can think about can we increase security budgets can we adjust the protection level to uh you know to to 30 to 40 40 Days by accepting the risk so now you have a plenty of you

know options to think about that is a conversation that needs to be happened between your security team and the business right let's go down to the case studies I think this session I mean this you know this section of the talk should be quite fun I hope it's fun but I it does have a lot of information on it so the first use case the first uh case study I would like to present is um is one of the project I used to work for um it's a global energy Supply U operations company so it's a global company in a global setting so they have they are critical infrastructure they are doing uh energy Supply um they use Cloud

deployment but they are in still in the Greenfield Cloud migration they're migrating to the cloud uh from the start and they they will they they have also have a lot of um on premises environment so the thing about this uh the challenge that they facing here is they don't have a clear governance and also they have over 1 metrics security metrics that that they identified in their project and then they they will struggle to strug struggle to uh identify the key kpis so so that is what they ask us to help them to you know to to identify the key kpis so we asked them what so what what other the story you want to share so what do you want to

you know what do you want to do with your kpis and and then they said okay we want to measure our team efficiency uh we are we are uh migrating to the cloud of course we want to track the cloud security posture and most importantly is we want to communicate security risk to the board so okay they want to do security risk to the board and they don't they don't want to feed too much kpis to the board because they wouldn't be interested in too much so that is our goal we want to identify those um metrics for the for the Bard and then during during this project that we have um we have um review a list

of kpis that they have identified so the company inti initially focus on metrics like the ones that you see on the slides they seems to be valid uh metrics right but actually when you look into those metrics they are not outcome driven they you can't really do anything about it you know number of instance vulnerability alert detected what do you do with them you know what what are the outcome if the higher higher number of incident doesn't doesn't representing an improvement of the SEC security posture and for example another example is um modity metrics based on the CVSs Bas um risk scoring and although it seems to be uh legit I mean it's good it's of course

it's legit but simply using the CVSs score alone can be Mis misleading because sometimes not all the viabilities are you know are having the same uh likelihood of exploitation so now you want to kind of look further other approach to to mitigate that mitigate that uh Gap and then finally you know percentage and number of threats blocked by XYZ tools and it seems to be a good one but in fact a lot of um you know a lot of noises that we have experiencing or experiencing in our tools especially Cloud security tools there's so much noises and we want to we we we and and there so much work that we need to do to in order to oh what in order to

drill down those at in order to solve those issues um so these are the metrics and kpis that I um that kind of we we propose to this company um please do not read into them because they might be they might be good metrics or kpis for this company but they might not be the good one or right one for your company right so I just want to explain a few facts about this um so you we asked the story that the security team wants to share with their Business Leaders so we talk about it with them and understand that that they wants to prioritize um they understanding their priortize business prior priorities and the risk Torance

level and we also identify a few outcome driven metrics from those list so for example the board wants to gain like a you know a visibility of how security post should look like so we proposed the cspn score I know cspn sco could be uh quite cont cont yeah but you know could be quite wrong because yeah because um because sometimes it's um it might not be representing the true posture of your company but the thing is you want to you want to speci you want to um separate them by subscription or accounts or project depending on the cloud provider that you use and then uh you want to set the Target that is realistic and for

each different project they have different criticality you want to set um maybe uh 80% of Target for one one project maybe a 60% minimum for another project but also at the end of it you want to also Define a baseline for the entire company so you have a consistency that you want to match for example 60% is your Baseline you can't go below that and you want to maintain that something like that and then um things like the instance management instance response management metrics they are the good one to you know to look into but um you need to think about you know whether you can align with the industry benchmarks uh what other what are the similar business

that you um that how do they operate what is the a meain time to remediate and whether you are vulnerable to run somewhere attack so some something like that you just need to know what are the you know what are the business driver behind those metrics and then um you can think about you know public Cloud workloads how can we how can we achieve how how can we um achieve the compliance of the uh of the cloud workload production in the production environment and then you want to want to set a Target that's realistic for that uh approach so um yeah so the the bottom line is that you want to identify the the the the the metrics and

kpis that actually apply for your organization um this is the second example that I have um this another project so this this company is uh quite different to the comp the previous company is a payment Service Company is highly regulated they are subject to PCI DSS compliance requirements but they are a small company they are is um in this sense so they have a dedicated stock team um but they they they um they they don't have a dedicated soof team they are relying on the external manage manag service provider to operate that so what what matters to them is the Innovation so Innovation comes to application application development and they wants to use they are using Cloud native environments they

wants to go fully Cloud native in terms of Cl uh software development so what we have what we have suggesting here is that we want to they they uh we they they can they can shift left they can Implement shift left security mindsets and then they can leverage the different uh tool sets such as the sus um infrastructure code scans and d and backlog management everything to leverage those tools to establish their metrics they don't have any metrics to start with so these are the metrics that we have proposed to the for them so we think about the data that come out from those processes think about you know the delivery pipelines what can what can you

identify from that the pipeline in terms of metrics and and also how how does instance respond process of the organization get involved are there any data elements that could fit in our to in into our metrics and of um into our measurement of our the security posture and what is actually relevant to what we are doing here so because they are adopting the shft left so we focus a lot uh of the metrics on the software development so for example the production cach passed a automatic security test is used is is used to demonstrate that the code vulnerability are being being addressed early in the dep development life cycle uh the infrastructure ASCO uh coverage rate

allow also allows the company to um understand how much infrastructure has been is is compliance and also we have dropped uh some of the um identity um the the IM the IM based um uh metrics here for example the risky uh signning of risky user for cloud identity and the percentage of the privilege accounts that have uh the pin uh privileg um identity management policy assigned something like that so they have experienced a few instance in the past so they are very conscious about you know how so whether our the whether our users are risky in the environment so so those so those um and also they want to demonstrate the cost of um you know the

cost of fixing the security bugs early in the in the life cycle so so that's why we put down the the cost of security bags remediation at the staging or production um stage I know some of the metrics might be difficult to measure but the idea is to get the mindsets into your business into your uh developers in order to help to to ask them to you know to collaborate with security teams um okay so that's my case study I hope there's not too much um of information um but now I would like to talk about epss um so this is for yeah so this the epss is for V ability of prioritization so um as we all know that

c CVSs score uh creates a blind spot because we we we don't get actual um actual probability of um the vulnerability being exploited and small and even small amounts of vity are actually exploited so that's why epss approach could be used to prioritize your uh remediation effort so maybe some of you already heard about it or apply already appli in uh to your organization but this is essentially representing the the uh the the likelihood of your of the exploitation so sometimes you want to balance the uh efficiency of the team and the epss coverage for example the higher the the epss thresold the the the higher efficiency and the lower of the courage that you want to remediate so you only

want to focus on those ones that have higher um likelihood of being exploited so yeah so that that's that's kind of like the the choice between you know between your restaurant and your courage and on the on this slides I've uh I had a few examples of you know of uh how epss can be used into uh use in your uh kpi so um I just need to run a little bit more faster okay so where where can you collect your metrics from um I've listed a few tools um they you know I don't I don't put down the vendor names here but in the next page but these are all the security tools that you can can uh you

can identify your metrics and collect your metrics on this page I've um I've put down some example of the tools but you know I I I want to be vend vendor neutral so it's just some suggestions I mean it's some of the tools that I I have used in the past um for example the you want you wants to enable the obser observability for your Cloud native security and you want to use the um the the DPM or a a workflow uh platform to manage your inventory and and a security posture and so on and you you can use use use a few um visualization tools to to present your metrics okay this is our key takeaways

um just a few tips to get you started on the metric uh program um so firstly kpi is not just for security teams they are business uh decisions so you need to align your metrics with your organization specific goals and resturants and remember there is no one-size fit all uh solution you want to customize your metrics to meet your unique environment and uh requirements and also you can use the protection level agreement to shift responsibility from security to the organization's business uh leaders then and please you can start with something small and something essential don't go big don't don't don't create 70 metrics to start with because that could that could mess with your mindset and then um don't get lost in

the in the numbers look at Trends and um and patterns because they are there are more actionable insights that from that can uh that can you can get from those Trends and patterns so uh finally is that um you know don't work silos we want uh security and the developers to work together and collaborate and uh so the the most important thing is you want to identify the metrics that truly matters for your business not truly matters for everything right yeah okay so that's my talk um yeah I've got my link in here um feel free to connect with me but yeah uh any question do we have time yeah got a couple of minutes okay great

one question any

questions dispar systems mention most moving towards centralized try use centralized obviously that takes time yeah what's your recommendation to what you do in the middle there your experience oh your question is about how so if we want to move from you know from separating tools into centralized to into a centralized solution what do you do in in the meanwhile right so I think I don't think there is a you know like a good answer to this one because already depending on you know your timeline and how how how how much time you have in to move to the centralized Solutions but in the meanwhile I think it's when you manage your wers it's difficult to you

know to get rid of one Wenders because you might have uh you know a contract in place right so and you you might want to uh finish that contract but the thing is you wants to identify what the capability that you already have and what do you want to achieve and and then you want to identify the right tools that is going to be centralized and how can you integrate different tools to that centralized tool ciz is not about getting rid of multiple tools it's all about feeding the data into a centralized tool that you can manage it in a more efficient way that's that answers your question yeah thanks than you okay great thank you