Secrets, Lies, and Half Truths in Cyber Defense

hello everyone and welcome to my talk secrets lies and half truths in cyber defense my name is Mark Orlando and I'm thrilled to be here with you all virtually at B-side Charlotte to give a talk that I'm quite passionate about hopefully that'll come through we have a lot to discuss really quick about me I have uh 21 years experience give or take in cyber defense who's counting uh I've been a sock analyst I've been a security manager I've been an executive today I'm the co-founder and CEO of a cyber defense company called Bionic I'm also a certified instructor and course author at the Sans Institute I teach SEC 450 which is blue team fundamentals and

security operations and I co-authored management 551 which is a class on building and leading sock teams so working in security operations supporting cyber defense as an executive a manager as an analyst as a Defender something that I've done for many years now and I wanted to give this talk um at what I believe is a fantastic event for Defenders and security folks uh you know of all uh different disciplines is I feel like as a manager as a consultant as a practitioner we're investing a lot arguably more than we ever have in cyber defense but I don't feel like we're improving at the rate that we're making Investments and I think there are a lot of reasons

for that I think we are getting better but I think we still have a long ways to go so I wanted to give this talk basically because I wanted to be the well actually guy who just sits in a meeting and trolls everybody else and tries to tear down the lies and the half truths that we often tell ourselves to make ourselves feel better I wanted to take all of those things really annoying things put them into a talk give it virtually where no one can throw things at me or punch me in the face for being annoying and you know really try to be honest with ourselves about you know some of these lies and and some of

these things that we tell ourselves to to make ourselves feel better but which are quite as effective as we like to believe they are I don't know how I got hacked I had my risk Matrix fully completed and fully color coded I have no idea why we could have been breached right we're going to tear some of those things down just a tiny little bit in this talk more specifically what we'll cover some common myths truths insecurity mostly in operations because that's my background reasons why we get some of these things wrong even when we feel like we're doing them right and we're doing the right things we'll talk about some ways to get to

ground truth we don't want to throw all of these things out entirely there may be some more effective ways that we can look at what we're doing we can improve the things that we're doing do new things right in a better way to get to some measurable and repeatable results for building out executing and managing our cyber defense so that's what we're going to talk about so let's start off by going over some of The Lies We Tell and secrets we keep and I think to some extent these aren't necessarily Secrets or rather they're poorly kept secrets in our industry the first deals with teamwork we have an effective team I've hired really smart people

so obviously they're making the right decisions they're doing the right things when it comes to technology we've spent so much in fact I made it a point to walk the Expo floor at RSA and I bought one of every single shiny Blinky box they have so surely we're in a good shape from a technology perspective right and surely those things work from my metrics perspective well I have comprehensive metrics that I'm tracking I'm measuring my security I'm doing assessments on a regular basis I'm collecting numbers and figures and charts so I'll know when security is not working as I intend for it to work oh those alerts yeah that's just noise we just acknowledge those and move on we

don't really pay attention to those but otherwise I will know when I miss something or something isn't working as it should right so in case you haven't picked up on my sarcasm there are some issues here in how we think about technology in cyber defense how we think about our teams and how they collaborate and work together how we measure and track the work that we're doing on a daily basis so let's dig into some of those starting with teamwork are you an effective team does anyone get this meme one of my favorite Sci-Fi movies of all time Oblivion it's awesome you should check it out but are we truly an effective team having an effective team

doesn't start and end with having smart capable people throwing them together and just kind of hoping that teamwork happens in a lot of cases it does happen because we're all relatively smart capable individuals at least all of us here are right I don't know about other people but we definitely are having really solid teamwork having spaces where we can share ideas and we can share skills kind of freely we can address gaps in each other's knowledge right that kind of effective collaboration it requires deliberate effort it doesn't just happen organically on its own or rather those team relationships don't form in a way that maximizes what's possible so just because I'm smart and you're smart and maybe we're doing good work

individually doesn't necessarily mean that our team is effective as it could be in fact we're much more likely to go off track unless there is some deliberate effort and some intentionality in how our working relationships how our teams are built and cultivated meaning how we train together how we track our interactions how we govern our Communications these things have to be deliberate it requires work teamwork must be measured and we have to measure teamwork how do you measure teamwork it seems kind of weird just as we would capture metrics on the alerts that we're investigating on the cases we're opening uh on the response that we're executing we need to capture metrics on team interactions remember

incident response cyber defense it's very much a team sport and I could be the most capable responder capable analyst in the entire organization I'm still not going to be able to fully contain eradicate recover from an incident all by myself I'm going to rely on other members of my team my team is going to rely on other groups and other functions within the organization to make changes right contain the damage recover those systems to normal working order so we have to track plan track and measure those interactions in that collaboration to make sure that it's happening as smoothly as it needs to in those crisis situations finally remember teamwork is a task work just because individuals are really good

at doing certain tasks by themselves again doesn't mean that the team is going to come together in gel and that those individuals who are great individual contributors are also going to be good teammates I know this comes as a shock to all of you that really smart capable technical people don't always work really well as part of a group I know talking about other people not us okay certainly other people why does this happen why does this require deliverable deliberate Focus especially in cyber security and cyber defense let me clue you in on kind of a not so secret secret in security teams one there's a lot of what we call egocentrism in cyber security

okay not talking about being egotistical that's a different thing surely not a problem in our industry right but egocentrism this feeling that I'm capable I'm smart I can handle this on my own I'm going to tackle this investigation by myself I'm going to respond to this incident by myself there's a lot of that going on and oftentimes our teams are kind of set up to encourage that mentality how many times have you been in an investigation or an instant response and you've got that really capable really knowledgeable teammate that jumps in and starts running the entire effort do you find that you're able to keep up and collaborate effectively with that person or do you kind of step away say

oh so-and-so has got it they're always the one that kind of drives these things so I'm just going to step back I don't know as much as they do I can't do what they can do and we're just gonna let them kind of run with it because they're the loudest voice in the room okay a lot of egocentrism and security cyber security is a complex set of problems really complex ecosystem complex set of problems no one person is going to have all of the the knowledge share the Mind share required to solve all elements of those problems okay if I have a problem in my cloud environment I as a security analyst or as a system administrator or as a

responder or an engineer requires a team coming together in order to effectively remediate that issue and to that point incident response cyber security consists of these multi-team systems it's not just one team it's multiple different teams that have to work together effectively to execute that effective defense okay these different issues these different problems intersect and result in ineffective teams ineffective teams so we have to address these issues we want to incentivize and motivate solid effective teamwork we want to create psychological safety within our teams someone disagrees with the way we're doing something as something to bring into the process we want to create an environment where those new ideas and those sometimes conflicting perspectives are welcome and they're part of the

process in doing that we want to cultivate autonomy belonging competence people want to feel confident that they're able to add to the discussion Be an Effective and productive part of the team so how can we do this one of the things that we can do in a very tactical sort of way to track and measure the collaboration that's happening is putting together something like this this is a multi-team system diagram it's how we can plan and track collaboration not only within our own team but between our teams and other teams in our instant response organization this comes from some research done by Dr Daniel Shore who's a behavioral scientist and a founding member member of a firm called

let's we can they're focused on teamwork and training and a lot of the things that I'm talking about here right but Dr Short and his team created this model essentially as a very simple way to identify and track relationships and team interaction in this case during security incidents so you can see here in this diagram we have a few different functional groups we have our Watch Team our threat intelligence team engineering team and so forth and we're using these arrows to track how much interaction happens between these teams in these crisis scenarios so for example here the watch team in the engineering team always work together very closely and very often during incidents the watch team and the

Intel team and the IR team also work pretty often together the Intel team and the pr Communications team maybe not so much but there needs to be some kind of relationship some kind of contact there so we can map out these relationships and these interactions and we can use this to guide how often these teams train together what kind of trust needs to be built within these teams what kind of cross training we need to do maybe what kind of shadowing we need to do between these teams do our processes and our tools support this really busy high level of interaction between these teams that we've mapped out here so again just a simple useful way to deliberately plan

out collaboration and not just tell yourself that you think the teams are working well together when the reality you have no idea right we want to get to ground truth on that this is a good way to start down that path we can also use Frameworks like the nice framework developed by nist and constituents of the the federal government to Define job roles Define competencies tasks Knowledge and Skills required for those job roles essentially coming up with a set of building blocks to say you know what it's not enough just to hire really smart capable people I've seen way too many really smart capable people put into roles that don't match their skill set don't match their

career goals right what they're interested in what they want to do so we can use models like the nice framework to write specific objective repeatable role descriptions with very clear knowledge and skill statements you have to know the you have to be able to do these things here's how it's going to translate into doing the actual work to make sure that our candidates our people our talent and the roles that they're filling the work that they're doing are well aligned nice framework if you're less familiar with it I put a link in this slide to the nist publication that describes it really really great resource not only for writing those objective comprehensive role descriptions also really good for

building training plans so if you have folks on your team or you yourself feel like you're lacking specific knowledge or certain skills technical skills to do the job effectively we can identify those gaps come up with training Solutions or training plans to help as opposed to simply saying I mean I'm an okay analyst but you know I think I could really be a much better sock analyst if I you know take that underwater basket weaving training class you know because that really interests me uh maybe maybe not but by using standard Frameworks like nice we can make sure that those requirements and the requirements of the job what you really want to do what the

job requires you to do that those are all lined up and again we're not just hoping that by throwing smart people at the problem things will magically get solved and our cyber defense ultimately becomes more effective very little deliberate effort so let's talk truth here about some technology right technology surely there are no half truths or lies about technology in our sector right or in our industry the truth about technology is that tools won't stop a skilled adversary if you think Uber did not purchase any security infrastructure or security tools probably wouldn't want to place money on that bet your security much like their security will be defeated by poor I.T governance creative adversaries who have

anticipated how you're defending the environment or not defending it and they're going to exploit those weaknesses right it's not just about having that fancy Blinky box in fact sometimes having the fancy Blinky box can put you in a really bad spot where you think your security is a lot better than it really is we invest in tools because it's easier than i t governance Real Talk folks eliminating Rogue I.T or Shadow tea buying that stuff's hard can't I just buy a security tool instead right now maybe that's a little bit unfair but that is the prevailing mindset in a lot of organizations that haven't yet tackled many of these fundamental governance issues but think that because

they've invested in security infrastructure and security tools they're solving a lot of their problems and as we know as we've recently seen that just isn't the case so what do we do about that how can we be a little bit more honest with ourselves cut through some of those half truths get technology that really works for us well first off technology starts and ends with requirements right we want to have a plan for our technology our tools should enable our teams to fulfill their Charter meet the requirements of the organization we don't start with a fancy tool and then figure out how we can use it build our processes around that no we have a need we have a thing that

we're trying to do right detect threats at the network layer detect threats you know post exploitation activities at the endpoint what technology lets us do that more effectively and in a scalable way we have to keep an eye on our capacity you can have 10 of the best tools in the world if you only have staff to Monitor and manage three of those tools you're over subscribed you have a capacity problem you're not going to be able to take full advantage of the technology that you're buying if you don't have the capacity to operate it finally when it comes to introducing new technology I'm a huge fan of Frameworks and processes I mean I'm a manager after all

we want to adopt a repeatable process for introducing new technology miter's analysis of Alternatives or AOA process I've put a link here in this slide really really great process for that comes down to identifying your requirements or the opportunity I know what are the things that my tools need to do to meet those requirements let me look at multiple different tools that potentially check the boxes on that criteria compare pick the best one that's it that's it we can also take some lessons from other disciplines into security love doing this there's been some really great writing in the last couple of years by Anton Chewbacca and some other folks at Google and other places um

about taking lessons and approaches from site reliability engineering SRE applying some of those principles and some of those techniques in cyber security a lot of this comes down to adopting an engineering first mindset insecurity with an eye towards making permanent improvements so tell me if you've heard this one before I stand up a sock I buy a bunch of tools I slam them in there I hire a bunch of people tools make noise people handle the noise and that just goes on in perpetuity and the people get burned out along the way and guess what I do I bring in more people they get burned out guess what I do then bring in more people you see where I'm

going with this we want to think about what kind of improvements we can make hello and solvable reduce those male operations uh there's some really great there Google if you Google if you search for Google SRE if you're looking at Anton Chewbacca's blog posts on this topic there's lots of really good write-ups there on kind of the life cycle of automation which I've shown here in this slide and some of those SRE principles that we can bring into cyber Security apply with our technology eliminate some of that burnout repetitive work some of the things that we all kind of know our problems with some of our technology but we just sort of accept it we accept that we're just

going to have to look at the same alerts ad nauseum to Infinity forever until we leave that job or get promoted out that's really not the way it should be let's talk about metrics for a bit I'll be kind and say there are some half truths and security metrics and I want you to take a look at these and let me know if any of these look familiar right that's rhetorical because this is a pre-recorded Talk and of course you can let me know in the Discord where present Mark is currently monitoring and answering questions I'm past Mark speaking to you now okay look at these metrics see if any of them look familiar for detected attacks

we're tracking how many alerts our tools are generated how many uh incidents or tools have detected when it comes to analyst productivity we're looking at number of tickets opened how fast those tickets get closed the number of projects that we're working on it kind of shows our productivity level when it comes to vulnerabilities we're prioritizing patches based on CVSs score of those vulnerabilities when it comes to risk measurement we've got risk matrices where we're tracking attack Impact versus likelihood coming up with some calculations there well this all seems fine right maybe maybe not bottom line is security metrics are hard who knew think about our good friend the risk Matrix which I'm going to beat up on a

little bit right now fair warning common mistakes here that we're making type errors right and that is something that is specific to the risk Matrix I want you all to come along on a journey with me as we walk through these values we express likelihood as a scale of very likely likely possible unlikely Etc and we express impact as non or negligible minor moderate severe significant Etc right so these are qualitative measures and we gauge these measures as low moderate high red yellow green these are not quantitative measures these are labels they're ordinals and nominals that we're using to basically sort things into buckets it's not true quantitative risk measurement we're saying well this thing

is less likely to happen and if it does it's majorly impactful what even does that mean the way that I assess that is going to be different than the way you obsess that assess that excuse me it's not very objective and the problem really comes into play when we take what we assess for severe and we multiply that by very likely to get a overall an overall risk of high so we've basically taken an ordinal low medium high and we've multiplied that by another ordinal what is a High Times a low I don't know what's a yellow times a green that math can't meaningfully be done so the bottom line here is we're taking highly subjective measures highly

subjective classifications we're doing meaningless math with them and we think that that gets us to a meaningful designation for what the risk is unfortunately things just don't really work that way so this is what's called a type error we're applying numeric labels we're just assigning numbers randomly to nominal or ordinal classifications and then we're trying to do computations based on that it can't be meaningfully be done another challenge that we often have when it comes to security metrics is this phenomenon called metrics fixation where we use metrics and numbers and data as a replacement for judgment an experience Talent we'll talk about why this happens here how we can avoid those type errors that I just described using the risk Matrix

one use the right measurement scale if you want to have data if you want to have interval or ratio values you know zero one two three Etc design your metric that way from the beginning don't simply assign numbers because you think that it makes the metric sound more scientific it doesn't doesn't work if you are using ordinals like high medium low red yellow green whatever red yellow green would be a um nominal right if you're using those kinds of measures where there aren't numbers involved that's fine drop the numbers maybe create a mapping that says if this is a low likelihood right then do this don't throw numbers in there and complicate things and try to add math

which doesn't work option three shift to True quantitative analysis there's a fantastic book which I'll reference at the end of the slide presentation how to measure anything in cyber security risk or everything in cyber security risk uh really fantastic reference for doing true quantitative analysis of cyber security risk highly recommend you check that out can we avoid this concept of metrics fixation where we essentially use numbers as a replacement for judgment you know everything has to satisfy the numbers it's a replacement for essentially effective management think about how you're using metrics in your teams today are those metrics how many alerts you looked at how fast you were how fast you were on your instant response okay are

those being used as a diagnostic tool for practitioners or are they being used to assess penalties or incentives there are reams of research that point to any time metrics are used as the basis for a penalty or incentive guess what's going to happen people are going to game the system and there are countless examples of this from higher education to law enforcement to cyber security absolutely there's a great book on this which I'll reference at the end of the deck at the end of the talk here get inputs from the right people your practitioners your Defenders are only going to be on board they're going to be open to feedback if they really believe in the metrics how

they're being measured and then finally recognize the limits of data we can't use metrics and data for everything okay so metrics should inform judgment they should demand judgment they shouldn't replace judgment why do we keep making these mistakes if these are you know not so secret secrets in cyber security why do we keep doing these things over and over and over again in some cases for over a decade we've been doing these things in cyber defense well first off it's easy to assume that correlation is causation if there are two things that seem kind of related may or may not be related we have what sounds like a really bad you know vulnerability and it's got a

high cvsf score we automatically assume that maybe we have to prioritize that for patching before everything else and we assume that there's a causal relationship there um where maybe there isn't one the mere exposure effect wrist Matrix we see something over and over again we assume that that is just the right way to do it when really maybe not quite as meaningful as we think it is numbers make us feel safe and scientific they imply objectivity when maybe there isn't any finally data can sometimes be metrics can sometimes be an easy button in situations where we don't have domain expertise or we don't have a lot of experience has anyone ever been in a situation

where you have a manager or an executive who's brand new doesn't have a lot of history or background maybe doesn't have domain expertise what's the first thing they want to do well I want to look at the numbers I want to make some decisions change some things show me you know how many incidents we've opened or how many vulnerabilities that we've patched right I just want to see the numbers I'll make decisions based on that that is an easy button and cognitively challenging situations where it's really tough to get up to speed or to get all the insights we need sometimes metrics unfortunately become that easy button and it's not just for managers and Executives right we all do

it but we should so how do we get on a path to more honest measurement first we start with top down alignment so think about our goals and our objectives for the security team what services are we offering to the organization what results do we need to see to know whether or not we're successful as a security team are we meeting our commitments We Gather our metrics using consistent measurement without subjective criteria preferably in some cheap you know automated or semi-automated way use that data as an indicator of how things are working not as a method of control unfortunately we're gonna have to use judgment we're gonna have to manage things you can't just rely only on the data

doesn't tell the entire story in all cases finally any metrics that we're using that we're tracking for our teams have to be expressed with at least one unit of measure percentage hours dollars that is the path to True quantitative risk assessment and not just feels like that's a high feels like it's moderately likely may be kind of impactful right that's not particularly useful one example of coming up with meaningful metrics or one approach is the goal question metric processor gqms pioneered originally in software development projects essentially it is a way to come up with useful metrics that are traceable to high level goals or objectives and here's how this works let's say one of the goals of your

security team is minimizing the impacts of supply chain attacks so we want to come up with metrics that tell us whether or not we are in fact minimizing the impact of supply chain attacks so how do we do that first we take that goal we ask clarifying questions about that goal try to get to something very specific and measurable first off how do we Define supply chain attack are we talking about alerts of a certain type are we talking about new vulnerabilities or weaknesses that fall within a certain kind of classification do we have Telemetry do we have data and visibility to monitor for these kinds of threats what does that look like when we fail to minimize the impact of

supply chain attacks what does that look like is there data or is there some tool or some report or I can identify those failures measure how often those occur so some metrics that answer these questions number of a search insertion attacks third party compromises right those are supply chain attack types tracked in our Incident Management System that is a measurable specific data point that I can capture what is the completion rate of my third-party attestations so it's not just about technical data Maybe I track third party you know questionnaires how I'm kind of tracking that that third-party supply chain risk the time to contain supply chain attacks third party compromises based on what's In My

Incident Management System so now I've got three data points three metrics I can use answer my follow-up questions that tell me whether or not I'm achieving that goal minimizing the impacts of supply chain attacks taking this a little bit further do we have roles on our team think about the teamwork that we talked about right the Staffing we talked about cutting through some of that noise do we have roles skills knowledge required to identify and investigate those supply chain attacks do we even know a supply chain attack when we see it do we require some additional training to fill those gaps make sure we know what we're looking at looking for do our tools and our Telemetry enable us

to identify and investigate these kinds of compromises are we using those supply chain metrics to support our judgment or are we using them as a replacement for our judgment are they helping us actually improve that's really the fundamental question here again this is the path to more honest metrics tying it all together think about why your security team exists start with those high-level goals and by the way the goals are not operate our sin Open tickets those are not goals what are you doing for the organization to enable it to operate free from Cyber attack or major outages due to Cyber attack right start with those high level security goals ask clarifying questions to understand

what success looks like and then make sure that the roles of your team the tools you have the processes that you've developed make sure those all support those objectives make sure the team is working together effectively map out and track how your team works together internally and with other teams in this larger multi-team system create one of those MTS interaction diagrams use those maps to train to communicate build trust make sure that teamwork is happening effectively do your tools meet your requirements are they really doing the thing that you need that tool to do for you or are they just creating more busy work eating up your capacity Without Really solving any problems finally in your metrics for those of you

manager leader types out there do you have the right metrics are you using them to diagnose problems to drive improvements or are you simply racking up a bunch of meaningless data to use as a replacement for good judgment good management that's how we bring some of these things together in summary look we're getting much better at Cyber defense by any number of measures we're getting better but we still have to be honest about systemic failures ineffective approaches things that aren't really working for us keeping requirements in mind starting and ending with requirements when your Staffing security when you're implementing new technology when you're measuring success it's absolutely critical do use existing models and Frameworks and processes to add transparency and

consistency but don't rely on faulty deceptive less than honest approaches that obscure hide those systemic issues or those bottlenecks some references for some of the things that I've covered in this talk again how to measure anything in cyber security risk fantastic book by Douglas Hubbard Richard syerson the tyranny of metrics uh really short kind of quick read by Jerry Mueller a really great book on kind of misuses and unintended consequences of using data not specific to security but I think is very relevant for security and of course I'd be remiss if I didn't plug management 551 which goes into a lot of these Topics in far more detail than I've done here written by myself and John Hubbard

with that I want to thank you all very much for attending my talk of course I want to thank the b-sides team for having me back again to do this talk if you have any questions or any comments I've been in the Discord Channel hopefully responding to those by now but uh certainly willing and happy to talk about more of this stuff in detail there so please feel free to reach out and thank you very much enjoy the rest of the show

foreign