Buying Security: A Client's Guide

hi folks oh there we go perfect um thank you for your patience uh thanks to besides for putting this on for having me to all the volunteers and to you for coming this is buying security a client's guide and today we're going to be talking about how to buy and get value from security services vendors specifically around security assessments um hi i'm rami uh i do not work at twilio i'm at a series d health tech unicorn out in new york city i'm based in boston um i am what i call a reformed security consultant so i have experience both buying and selling these assessments uh and i just want to give a quick thanks

to tldrsek and my friend clint who worked with me a lot on this content more on that in one second so quick heads up you do not have to worry about taking notes and photos i made this super dense there are a lot of references there are a lot of call outs these slides will be published after the talk a link will be available at the end and clint and i have put up a long form version of this guide on tl dr sec that you can read at your leisure in the future as well and of course the talk is being recorded um so feel free to take pictures but don't stress about it

so why are we here why do i feel this is important you can't buy security right it's not something you can purchase but security services are a really important part of security programs especially as your company grows you'll need capabilities that you don't have in-house you'll need to scale your program in a flexible way you'll need a third-party validation of the quality of your program and shopping for and deriving value is therefore a core competency for a lot of security engineers but organizations struggle it's a really hard thing to do when you're doing it maybe once a year maybe you've never done it before there's information asymmetry there's difficulty assessing performance of a vendor or quality of your assessment

there's infrequent or limited awareness of just the breadth of this industry and the services on offer there's frankly misleading sales tactics and a lack of standardization or quality on delivery across the board so it's really easy to buy a dud and there are a lot more alternatives nowadays that make it harder and harder to navigate this marketplace so we're gonna dive into it um hard to say better than the survey respondent and a little programming note more on the survey in a second but when you see this logo on the top left that is all sourced from the security community who participated in this survey it's also just a big industry like security services are almost 100 billion

dollars i know in these rooms at these conferences we can feel like a small community and i love that but there's a lot of money in security services and that means there are a lot of competing interests at play so today we're going to talk about buying security assessments as a narrow lens to look at the security services industry as a whole security assessments just in case you you know don't have the definition off hand there's the nist version it's all about assessing architecture implemented controls and really just making sure your security program is accomplishing its goals and meeting your security requirements so what do we get over the next you know 40 or so minutes um

i've synthesized 200 resources on the topic from all across the internet i surveyed over 100 security professionals in collaboration with tldrsec and we've put together a comprehensive guide on dying and buying and getting value from these services my goal is to use this to work together and like really continue the conversation about how we as an industry procure assessments and then use them to measurably reduce risk because that's what it's all about so numerous sources just wanted to throw up some logos quickly we stand on the shoulders of giants there's been lots of writing and thinking on this topic from pen test vendors as content marketing from standards groups government organizations conferences academics a full bibliography and citations will

be up with the guide as well in case you want to dive more into this and about the survey this was circulated throughout the security community slack groups networks tldr sec again we had over 100 respondents 71 of whom were buyers and 68 were sellers with a lot of overlap and it was basically a 20 question survey about buying and a 20 question survey about selling just trying to get some frankly quantitative sampling bias present data from the community at large beyond you know what we were reading and what our personal experiences were and like here's some of what we were surveying right if you look we had representation across industries obviously professional services are really well represented

when you're interviewing folks who work at security consultancies but also a big representation from like the information industry financial industries as well and companies of all size right about half folks worked at companies with more than 500 employees but uh you know a broad array of folks represented and uh procuring when you're under 10 folks is definitely different than when you're over 500 so sort of a broad swath of the industry here and i just wanted to talk about like how much coverage do we get we only talk to 100 folks obviously it's limited but together and this is a logarithmic scale we have coverage of over 1 300 purchase security assessments and over 25 000

sold and that's like especially on the right hand here you have some folks selling 5 000 security assessments which is a bonkers number to me um so i went and checked into it and i will say these are folks who have like 10 to 20 years running a consultancy or sales for one um and they just have this wealth of experience that we were able to tap so like why are we here why am i entering with this presumption that this is hard well i really latched on to this idea from haroon's 2012 2011 talk some of you have seen it others should he comes up with this concept of security pen testing companies as a

market for lemons it's a economics paper that talks about marketplaces and it says basically if you're selling used cars you have good cars which are cherries and bad cars which are lemons and if in the market both exist buyers have to average their expectations they don't want to like be paying cherry money for a lemon and so slowly folks will take cherries off the market and only be selling lemons and this is like how information asymmetry where buyers can't trust that they're getting a quality product can really just degrade the quality of the industry as a whole and i think and haroon does as well or at least did in 2012 that this is emblematic of security services

buyers don't trust they're getting a good quality product and so they average their expectations and so we're slowly moving to sort of the least common denominator product even in 2007 we were talking about this gary mcgraw likes to refer to pen testing security assessments as a badnesso meter right on one hand you have like you and your security program suck and on the other you have well i guess we don't know there's no winning in security assessments you can't prove a negative there's no way to say there are no holes there are no vulnerabilities there is no risk all you can say is that in the target scope with the time allotted we didn't find any um so this is hard this

makes it a really hard like product to buy because is a blank report a good thing or not what about one with a lot of findings right like what are you actually looking for how are you assessing quality and how do people feel about the quality of the marketplace right here's a here's a statement we made and got some you know how much do you agree with this statement metrics um the average answer you know a normal distribution right around to 2.9 this was the same for folks who had done both buying and selling agreed but when you slice the data by buyers and sellers you see a shift buyers actually have a higher confidence

in average quality and frankly i actually think this is a problem i trust the sellers here who are likely more informed on the industry and the offerings obviously with some biases and i think that this is evidence of like some issues where buyers are overconfident in the quality of the services they're receiving or unable to dig into the quality and similarly concerningly um like the output of a security assessment is probably some risks uh and some advice on what your priorities should be and what the risk level is and there isn't like this overwhelming consensus that risks are appropriately graded we'll talk later about how to account for this discrepancy um but you really uh would

hope that we could trust that vendors are working in your business context giving you like well-groomed risks that you can immediately apply and that's just not the case currently so we'll dig into more challenges as we go um but i want to talk about how we can do better at buying security assessments it's certainly generalizable but much easier to carve a slice out and talk about it in specifics here and i stole this diagram from patrick who i believe is somewhere around b-side so if you see him thank him for me but there are a lot of types of security assessments out there he slices it along scope and the level of visibility given to testers

there are a lot of other ways you can slice it but i just like this to show that like when you're buying an assessment there are a lot of things that could mean and you should probably know which one you want before you talk to a vendor here's a different breakdown right we have vulnerability assessments threat modeling red teaming social engineering white box testing is dominant this is actually interesting outside the talk this wasn't always the case and there's been a big push by security services vendors to get more access a lot of clients historically don't want to give you access to source code or their systems through fears of the risk involved but it does let you get much

more efficient thorough comprehensive testing done for cheaper so it was actually good to see that reflected and then you can see there's a long tail of all sorts of assessments represented here and big red slide i don't know if this is a hot take or a warning we'll we'll figure out how you all feel about it but when researching services vendors you're going to stumble across emergent offerings bug bounties and automated pen tests or red teaming are super trendy right now there's a lot of marketing i don't want to digress too much but there are false comparisons out there between these services and a traditional you know consultant-led engagement um and i just don't think they're a

one-to-one drop-in replacement and i think that uh folks who are buying their first pen test now can get you know taken down a rabbit hole of other services to put it another way or as one respondent put it it's gang tough to find something that's not just a dressed up nessa scan this has always been a problem it's a problem with legacy vendors but it's especially a problem when you're looking at the level of automation used in some of these places so let's talk about the process right how do we go from deciding we need an assessment to actually buying something delivering value for our business and then what do we do after we've done this

once i have kind of arbitrarily created this 11 step breakdown i want to note it is a circle even though it doesn't look like it you probably don't want a one-off security assessment you want some sort of program of security that security assessments are part of and so we can start by figuring out our motivation and this is like fairly obvious to some folks but a misstep in a lot of these processes to book a successful engagement you have to have a clear goal it's true of any product management but there are a wide range of reasons to you know book an assessment for a report yours are going to have downstream considerations for how you operate the

assessment what vendor you procure you might want to just do pure risk reduction i want to know what risks are out there i want to fix them maybe you're trying to sell your security program internally and are hoping for a report that shows that you're doing a great job or maybe you're hoping for a report that highlights the need for more resources it's involved in the sales cycle although i will say that for sales wait until you have a client deal contingent on an assessment before procuring one for that reason it will save you some time and pain early on and also post breach and this isn't instant response this is you know after a breach i want to identify how they got

in where they otherwise could get in raise my risk baseline and so you know great to see risk reduction as the top cause of folks buying a penetration test uh security assessment in my mind um that's the most altruistic reason to just holistically improve your program but you'll also see a lot of folks are buying them for compliance or internal evidence and this is interesting because you probably want a different test for each of these reasons and we'll talk about that in a bit but you know you've decided you want a test for risk reduction or for your you know pci compliance um any go about finding vendors this is like the first question right like

what is the list of vendors what are the good ones how do i find them it's surprisingly difficult you can't just like type it into google and trust you'll get a good deal uh you can't tell the wheat from the chaff um and there's a lot of marketing out there trying to uh muddy the waters and so i was gonna give you a taxonomy of like the types of vendors but i don't have five thousand dollars to spend on this talk so we're just gonna give you the types of vendors minus the taxonomy so um there's a broad marketplace there's misleading sales tactics there's evolving terminology and an overwhelming array of vendors and services but

navigating this as a buyer it helps to try and like bucket the vendors you're seeing um into common patterns and you can take this and your goals and then really start to narrow down the possible vendors for your contracts so there's a big range here from multinationals to solo shops you have single service specialists as well as you know low-cost vendors that are competing on price they have their place and then mssps and vars and on the last two i just want to say it's a high trust relationship to have a managed security service provider or a var however if you're buying a security assessment in order to validate the risk reduction they are providing think about

what it means to book the pen tester security assessment through that partner you may want external validation in that case and so here's the first set of challenges we got from survey respondents finding good vendors who provide significant value who are available when you need them who have expertise in your specific subsystem and who can provide consistent quality staff it's true throughout the industry that um you can definitely find a vendor but finding one that meets your needs is still hard and so there are a lot of strategies you can do to find them this image is associated with certifications in case anyone was wondering so you can start with network recommendations this is the first thing

you'll hear generally right like talk to your network find a friend who's gotten an assessment ask them who their vendor was when doing so keep in mind that like it depends on the quality of your network and their ability to vet whether they got a quality assessment a lot of folks will get referrals from people who may not have broad experience with different vendors and have no way of you know saying how good it was or not or folks who haven't been a client at all and are recommending a friend who they trust which has a little less diligence to it i also just kind of like this follow the leader play in finding vendors which is a lot of

marketplaces and integrations for big tech companies require an assessment to get into the marketplace to integrate google's oauth stuff has this facebook workplace has it and they generally name a few vendors they trust to do these assessments and that can be a good way to kind of get a picture of what big tech is using that being said your needs are probably not the same of those of like facebook or google and keep that in mind as well depending on your budget um but there are a lot of ways to do this research quick call out great to go up to conference speakers if they work at security consultancies and hire them as someone who used to do that

and work at a security consultancy but watch out for pay-to-play schemes watch out for sponsored sessions um this is the stack rank of like what other people are using to find their vendors um i would call out that i'm a bit surprised by research being so high research is very cool it contributes a lot to the industry it's not very tightly correlated to like real world assessment competencies and so like going to a conference talk and having a researcher say like a war story about a cool exploit they found and then hiring their company as your pen tester is not one to one to me so i'd argue a little caution there and i'm surprised by the low view of

analyst rankings although not offended by it uh they get a lot of attention especially on twitter a lot of effort paid and apparently we're all in this cohort at least in some consensus that's not how we prefer to make our decisions so we've defined motivation let's get more granular on what you need and what sort of like constraints you have on the vendor you can choose and so scoping is during a security assessment it's like here's what work we're intending to get done um and like let's put some boundaries on it so that our vendor is not just wandering in the woods and it falls under parkinson's law um this is the law that states that like

work expands to fill the time allotted um you scope a project because you're trying to give them just enough time to give like pareto principle 20 of the time 80 percent of the outcomes high return on value if you ask a vendor how long something will take without giving them a tight scope you can expect them to try and take as much time as you can for it and it's all about balance right when you're scoping an assessment whether this be a web app or a network you're trying to do a comprehensive set of tests but a focus on the greatest risk because you're not going to pay for an assessment that covers everything everywhere all the time

so you have to be thoughtful about where you pick your battles and does the client you get ahead by coming into procurement um you know something like half of clients are coming in with a clear scope in mind and this will really help you groom your engagement to provide the most value to your organization there are a lot of things that you should think about when you're scoping like budget is the first and biggest constraint here you can see the folks are saying that you know assessments are a big part of their security program but budget may not be totally there and it's important because if you can't afford everything you want you're going to have to make difficult decisions and

cuts so also think about the motivation and how that might impact scoping if you're getting a pci assessment for example there's some pretty clear guidelines on what you need to include um if you're doing it for sales you may care a lot about what sort of public-facing materials you're allowed to use and say you may have documentation needs like you need a letter of attestation to give to a client or maybe even a public report which is something like pretty complicated and expensive to procure if you have measurement goals like you're trying to test your blue team you're probably talking about a red or purple team engagement which has all sorts of different constraints scoping

is a huge topic of its own i'm linking here to four different guides i found helpful marks from 2007's still holds up so like that's the one i'd read if you read one thing about uh scoping application security so these are some dots they are hopefully animated we'll see how this goes awesome so there are requirements you may have that limit your available pool of vendors and your optionality here if you are requiring a remediation assistant not all vendors will offer that um you should know that in advance so that you can kind of use your time wisely similarly any requirements you place on the assessor whether that be requiring them to be on-site or have

certain citizenship or clearance or methodologies or certifications these all serve to limit your pool of potential vendors fundamentally that's not always a problem it you know reduces your decision space um but it can also make things more expensive because you're limited in your options so you you know you you roughly know i want to test this web app i have a few you know tens of thousands of dollars i have a rough idea of a list of vendors like let's gather some proposals and start figuring out who we're going to use and so there's a few steps right if you're a large company you probably have a formal rfp process smaller organizations it's normally like

call up a few vendors and ask rarely something so formalized generally industry analysts my experience your shortlist should probably be around three organizations it's really time consuming to have all these vendor conversations and there's like limited incremental benefit to adding additional vendors to that pool of selection so cut early get some high confidence in some vendors through your rfp or request for information or some basic scoping and then really drill down into some initial calls and here's where we flash back to motivations right like what are some of the ways that your motivation may influence your vendor selection if you're looking for compliance you may require vendors with certain certifications you also want to have

what i call a balance of substance when you're trying to target compliance with an assessment you want them to provide value you want them to do a good job you also want them to help you pass compliance so there's a balance there on the type of vendor you choose um similarly like there may be auditor relationships um in internal attestations this also may be like depending on your audience you may know that a certain executive has a lot of faith in a certain external party and you can use them mmas are interesting because it ends up boiling down to speed of engagement in a lot of cases but they also have specific characteristics you'll want to search

for and then of course like for sales like i said wait until a client deal is contingent on an assessment because clients will have really strong opinions on who the assessor is if they're at the scale they're trying to throw their weight around here and the worst thing is when you've like just paid for a shiny new assessment and then a client says they don't care and you have to go to the big four it hurts from a budget perspective so like what do you do on this first call i've been on a lot of these calls with clients and with vendors and there's no consistency to like how we talk about an assessment and try and

figure out if the vendor's correct here are some things you should think about from a logistics perspective like how soon can you staff this engagement is a really early way to call vendors especially as you get into q4 calendars get packed generally you're looking at like a week or two lead time for the average vendor in the average period and then in q4 that balloons you're also going to want to focus on experience with organizations like yours and when i say like yours i mean things like tech stack industry vertical risk landscape you want to hear them speak to your industry business logic threat model and needs and not just give you like a you know here's a web app pen test in a

box here's a you know network security assessment here's our standard threat model and you want to talk about engagement model and this really depends on your organization as well and how you like to work but how do they collaborate are they willing to open a slack channel with you will they communicate over email will they get on calls with your pms how do they staff engagements and make sure that you're getting a qualified assessor for your problem space what's their project management methodology and as you get bigger and bigger and are doing more of these this plays an outsized role because you want the vendor to kind of self-direct as much of this as possible because this is why you

are outsourcing and you should take these opportunities to judge vendors on their communication on their responsiveness on their collaboration if they can't field these initial calls effectively in a way that inspires confidence why would you expect the assessment to be any better so you've gathered proposals right you have a presumptive vendor the next step is actually like welcome back to scoping again this time with the vendor involved vendors do their own scoping process um it's in sport it's important you've explicitly communicated your expectations because otherwise you can do a lot of back and forth as vendors revise proposals that are just out of line with what you're intending to accomplish you can request multiple scoping options

most vendors will give you like a big medium or small if you request it and that gives you some flexibility and you should expect that this will be broken down by level of effort coverage or inclusion or exclusion of certain like types of assessment elements deliverables you're going to want this like scope to give clear objectives for the assessment an expected level of effort which ties directly to cost and what they think that gets you in terms of depth of coverage and another little warning here um watch out for vendors that are willing to like write you a quote on the spot based on a metric like number of ip addresses or like number of apps

um it's insufficient to accurately scope and it lends itself to false precision and i consider it like a smell when this is how a vendor's gonna scope for me uh for example when i was on the vendor side we were getting clients come to us and say like i have 200 ip addresses how much will an assessment cost and you say well like let me run a quick nmap scan let me see what's out there you find out that there are 100 ip addresses inside those ip addresses there are three web apps and they're actually asking you to scope three web app pen tests right very different approaches very different levels of scope or scale the difference

between like test a single service if it's a crud app or a static website are different it's important to right size your assessment over scoping and underscoping both have problems over scoping you're paying too much under scoping you're not getting what you need accomplished and you're not identifying risks and there's no consistency in how vendors are going to want to scope your assessment but this is the basic process they'll send you a questionnaire or have a call or ask to like scan your perimeter or maybe like give them the outputs of a specific tool or look at your code a little bit and then they'll have a conversation about you know what you're trying to accomplish and how big your

app is and what kind of screens there are or what services are running on your network or how many employees you have for uh threat modeling and then like maybe they'll ask for a demonstration or some documentation and they throw that on a blender and come out with very different numbers if you've ever run procurement for pen tests like pick five vendors you will get five very different quotes back so that is why it's helpful to come in with your own preconception of what the scope looks like and they're giving you back a quote and it can either be fixed price or time and materials but basically your goal as a buyer is to get detailed pricing and this is so you

actually can compare like for like apples to apples across your various quotes you're receiving because if you just get like a bundled price it can be really hard to tell where your flexibility is what your axes for negotiation are what each is offering often they'll call out earmarked discounts like here's our list price we're giving you 25 because we think you're great um take it with a grain of salt focus on the bottom line number uh payment terms are super interesting with smaller vendors uh this is something that i as a security person didn't think a lot about um but you know whether you get paid 30 60 365 whether you get a percentage up

front or in full um whether you're paying on milestones like for small consultancies getting the cash earlier has real leverage on you know where they're willing to compromise otherwise and something that comes up when you're like working as a security vendor is people ask why the heck it's so expensive uh this is a pretty old reference as well but just gives you a high level of like everything that goes into selling security services that results in the day rate uh you know hitting the level it does especially in this market hiring qualified security people is expensive um utilization like they're not working 24 7 every day they're scheduling constraints there's inconsistent demand there's ongoing professional development

there's burnout prevention which we've heard about already today and there's like the fully loaded cost of employee and overhead of non-assessors so um i'm not defending the pricing model of any specific vendor because i'm not aware of them but i will say that there's a lot more to it than like taking the salary you pay your employee and dividing it by the number of hours and comparing the two so you get a vendor proposal you get a contract great like what goes into it well you might want to negotiate um i personally hate negotiating with vendors is not something i enjoy i know people who like take to it a lot there are a few axes you can do this

along negotiating on pure rate is effectively asking for a discount right you're saying well instead of two thousand dollars what if i pay you 1900. there's not a ton of leverage there scope is often easier to work with vendors on generally this is either taking down the level of effort saying like can i pay you for a few fewer days and we'll accept there'll be some gaps or it's trading depth for breath like we would like to cover fewer services at the same depth or i'm willing to say that like let's do 80 of what you're saying um reporting also comes into play a lot of vendors charge for reporting time the more types of reports deliverables you

want the more expensive it gets you can cut some of those depending on your needs and if you are a big enterprise paying millions of dollars a year you can probably expect the price to go down a little bit but you have to remember project management triad for those that don't know good fast cheap i think normally it's pick two in security sometimes it's like pick one um if you're trying to be more flexible on timeline you can save on cost and quality by like cherry picking a vendor that's willing to work with you on the other hand if you're trying to book in q4 like we talked about before it's tenuous or expensive to get on

calendars and while there are certainly varying prices uh if you're getting a quote that is abnormally cheap be a little skeptical please and this is the same thing that we're hearing about challenges from people in the survey balancing quality price and availability justifying spend or even affording it all is really hard in security services and like vetting the proposals also takes a lot of work because you're going to want to think about comparing multiple proposals like for like is the quality similar is the level of effort similar what are the qualifications ideally you're performing reference checks on your vendors if you are a large enough deal you can request that the vendor put you in touch

with a client of theirs and they might do it you also probably want to pump your network for anyone who has an existing relationship a lot of people will tell you that the consultant matters more than the firm and this is true but unless you're working with a really boutique or solo shop you're going to have a hard time requesting specific consultants so just to warn you a lot of time like first-time assessment buyers really want to review bios pick a specific consultant someone they've seen at a conference for a variety of like valid and less valid reasons big firms are going to probably not play ball unless you are a massive client in which case you can

often get a little more specific there you can also review example reports see if they have public-facing reports really try and get an idea of the quality of their work and then do a bunch of paperwork so i don't want to dive too much into these but i will just say that there is a lot of paperwork involved in security assessments hopefully it is not your problem but someone else at your company's nda's msas sows you're going to see requirements around insurance you're going to see liability and there's a lot of resources out there cure53 is a consultancy that's made their contracts public secureworks has their msa there are a lot of options here for

cutting down on this so how do you prepare and deliver your assessment right like you've picked a vendor they're starting in two weeks um a lot of logistics tips from the surveys right it's gonna take more than you expect you're gonna need technical people on call you're going to want clear lines of communication and you're going to want to use business initiatives to potentially fund the pen test right like these are small disconnected tips here's how i break it down there's kind of three horizons you're working on internal alignment which means getting authorization and buy-in from all the right people that this assessment is going to happen and they will be ready to deal with the results

as well as in some cases blue team collaboration if you are running a red team assessment have a very clear plan of how your blue team will become aware of it if they need to communication channels are important how are you going to internally track progress who's responsible for first applying to vendor questions how are you going to dispatch vulnerabilities internally and what's your escalation policy is there a level of risk where you'll want it reported immediately and what are you going to do if like a critical comes in off this assessment and then this is fairly uncommon but i really advocate for people considering sharing their known risks with vendors if you're getting an assessment share

your previous reports share your threat models share your risk assessments this transparency is uncommon but lets them hit the ground running and build on prior art you've already done so you're not starting from scratch each time and that really speaks directly to your most business critical risks there's also technical preparation the first one feels obvious but like think about resolving anything outstanding before commissioning an assessment if at all possible the worst thing is when you as a client buy an assessment and then are paying them to tell you a bunch of open findings you already know about there's also a lot to do with the test environment assuming you're testing in a test environment setting up integrations

configuration feature flags roles seating data some of these are more common than others but when i was working as a consultant the number of times i walked into a company and they gave me like an empty b2b sas app with nothing set up and i spent my first three to five days just like typing in data entry um it's fine as a consultant if that's where you want to spend your day rates go for it but really think about whether the environment's sufficient for them to express all your functionality same with like network environments um is everything actually like ready to be tested configured in a prod similar way you may need to change freeze if this is

a like production environment something to think about reporting vulnerabilities and then having to like trace them through various versions of applications to see if they exist or not because of moving environments is never fun and consider disabling out of scope controls this is sometimes controversial with clients but if you have a waf if you have ip allow listing if you have risk-based authentication unless you are intending to get an assessment to test the efficacy of those controls just turn them off you probably don't want to pay a consultant to test cloudflare like cloudflare does that so it's better for everyone to just like let them focus on your actual scope and your controls and just to call out that like folks

forget about onboarding depending on the size of your company this may involve like putting a ticket in jira for another team to manage it but there's hardware there's software there's demos and documentation you may need hr paperwork legal paperwork you may need to warn it if they're coming on site there's all sorts of badging and security get this done early so that again you are not paying your vendor to sit around and wait for their stuff to get set up and disappointing for some of you i'm sure but i'm actually not going to talk about the assessment at all there's a lot of great resources out there on like how to do a security assessment there

are a lot of kinds of assessment um i'm focusing on the scaffolding around it so you know voila your vendor comes in they perform their assessment and you're on to the readout off to the races so right the report is the key deliverable for most security assessments um almost everyone based on the survey like does a pdf report and a presentation that's what's involved the report the thing to note about this like breakdown is that reports are designed to be decomposed like if you receive a pdf report you can split it up into sections and send those to different audiences internally um executive summary is named that because you probably want to field it to like

higher level executives well finding summaries are really useful for your security leadership to have a meta level and then of course you can break out like individual findings for line level engineers and a fun fact for me which i disagree with but there you go is that according the survey clients prefer an overall risk rating in their report so if you are a vendor your clients would like you to tell them if they're doing like you know good bad or mediocre across the board in your reports and then like after the assessment you're going to want to do a retrospective with the vendor this is where like in the readout you can take advantage and

get additional value by asking where they'd look next what they'd recommend you do differently what trends they observed systemic mitigations they might recommend areas that were well hardened get some positive feedback but especially like how do i compare to other assessments you've done this year or industry benchmarks or your expectations a massive benefit is that like security services vendors are one of the best ways for you to get aggregate data across the industry on security program health and performance and please make sure your vendor cleans up after themselves i'm just going to point you to this excellent talk from wesley mcgrew which goes into all the various ways that a security assessment can make your

environment more vulnerable not less as a consultant more than once on a security assessment i like stumbled upon findings wrote them up reported it and it turns out that my shell i picked up or escalated access was just left behind by a previous vendor in prod never a good look and this was fun task vendors is like what happens if you have no findings like i said earlier no findings does not necessarily read as all clients to a successful engagement like they're paying you to write up risks so vendors do three kind of buckets of things they manage client expectations note limitations communicate early and often they might write up a really detailed test plan um and try and like do quality

control sandy check the results or they can also talk about like other value add defense in depth additional measures you're at a really high level of maturity but let's talk about what's bleeding edge and then you have to ingest these results um a couple tips use your standard process retriage findings you know your environment best uh you want to do root cause analysis and variant analysis and this is what takes an assessment from like a one-time point in time whack-a-mole on risks to actually scaling your program um determining root cause finding variance block out some security team time to do this after the assessment so that next year you're not like getting a cross-site scripting this year two more

next year three more the year after that like build in some secure defaults security assessments are generally like an expensive way to find a single vulnerability so you want to do everything you can to scale that impact and just to note on personal reporting um if you are a big enough client or many vendors do this by default you can request like a csv of findings and then you can automate piping that into your vulnerability tracking which can save someone some data entry time and when you're remediation planning just a reminder you can always choose to fix mitigator accept these risks in many cases these assessments will report things that you're going to just accept not

everything needs to be fixed all the time it's all about your business's risk appetite so speeding up a little here because i'm down the last couple minutes uh please remediate your vulnerabilities job one fix vulnerabilities job two minimize transactional nature point-in-time assessment maximize your overall value if you have validation or re-testing as part of the contract especially if you're getting client-facing documents make sure to follow through on that and do a retrospective internally this is where we're talking about like how do you tell if you got a good assessment same reason procurement is hard this is hard to do you're going to want to look at like report quality how they handled false negatives and false positives

it's especially insidious but you can use canary bugs if you have open risks and they don't find them you really want to have a conversation about like were there good reasons bad reasons what methodologies were they using why did these latent bugs not show up again in the report and think about vendor communication how did you feel about price for value did you feel prepared were you able to reach them uh did the readout answer any questions and then you have to talk about testing cadence right you're not just doing this once annual is a good starting point but other teams do it quarterly or aligned to development cycles uh continuous testing is certainly gaining momentum as this one quippy

person said uh i'll give you a second because i enjoyed that and you're going to think about scope and vendor for future assessments right talk about breadth and depth did we cover a wide enough scope did we get deep enough what limitations were there do we want to retarget the same surface again to get more depth or a different surface next time and vendor rotation is divisive some people say you should switch vendors some say you shouldn't what i will say is that there is a compromise use the same vendor use different consultants at that vendor pros of rotation is like cross vendor comparison it works if you think firms are fungible quality cons or pros of repetition vendors will

recommend this they probably have their own motives but i will say it does give you like improved continuous context and then you want to scale your program um the bigger you get the more you optimize for product management uh the more you want to standardize procurement the more you want to decide when to staff in-house for high value and basically select a handful of companies to make contracting easy reduce like one-off pen tests with different firms it has high overhead and think about return on investment and with that i will leave up the fact that most people don't uh i am at time so i just want to say think about how hard this is

structure things based on motivation and use assessment to not kill bugs but to kill bug classes if you'd like to dive deeper into this it's available on tl dr sec slides are available as well and thank you all for your time [Applause]