Why Your Tools Suck And What To Do About It

uh the quality of this talk is L be the same as number of slides okay there is zero slides everyone Mark who about stuff children the children so yeah no it's it's p it's so yeah this this talk as you probably guessed from the title is um well it originated in aning r uh actually originally in several in FRS at which point Andrew who I used to work with you s over there said Mark you really need to do a talk about this because holding on to this pain is not doing you any good and uh so I said I'd do it and that was two years ago so I probably get and so that's why I'm here

and yes it's a bit of a rant I've got a loot of stuff I'm going to say that is fairly critical of some things that people put a lot of time and Ed into but hopefully during the course of my runting there will be some things that are um things you can learn from ways you can work around the deficiencies of some of the things I talk about and and strategies to not making the tools suck as much for you as they have for me so maybe if there's a point to this talk is so you can avoid some of the pain that I in and um when I I posted I was doing this talk on on the socials one of my

good friends reached out to me and he said Mark you said that security tools suck what's my job um and their job is building security tools and I said you know actually my point is not that I take security tools actually I couldn't do my job without security tools there isn't really a single security tool but I don't think has some kind of Merit it's just that sometimes it's didn't quite well and so my St Point really is this say that tools are useful they don't all suck all of the time in our jobs we're trying to do work where the work is reproducible it's scalable the the function that we're building grows with the with the business we want to be

able to gather data and act on those data we want to be able to do things in intelligent stainable reproducible ways and we can't do any of those things by ourselves without any tools right so my starting point for this talk really is to make the point that you know even if there are annoying features of these things we can't do without them we got to learn to make peace with them and Simon Z doesn't suck and this talk isn't about you doing the that job okay having said that uh tools are pretty much the pain of my life right and nearly all of the work I do and I work in application security as I said at the beginning

involves Tools near time I can't do my job without them my goodness and the first thing that that I have with is the sheer amount of nonsense that you get from your talks this is not unique I understand to application security fact you know I play with tool inut the domains in security it's pretty much the same picture everywh right I said from the outset we need these tools because we need the metrics we need the intelligence to do our jobs well but the difficulty is that there is just so much nonsense there that it's very hard to do those things effective and those problems really fall into a few different areas and the first is that um

there's a lot of M in the reports you know the signal to noise ratio is not great there's stuff in there the absolutely need to know and that's why you use the tools but a lot of the information they give you is not actually is actually not very good quality dat now um they launched in talk seal years ago yes again and I recommend you watch the talk on YouTube and the talk titled was um is it most what I took from the talk and probably there more in is that we don't really help ourselves as an industry through the quality of a lot of the information that we get right and there are some fairly badly Mis find

incentives in um getting CVS to our names and that sort of thing that mean that you know there are lots of f reports that while people work hard to make them accurate maybe don't either contain all the information they need or maybe don't AG particular right and you heard the expression garbage in garbage out I actually make one slide for this book by the way but one slide and basically there lots of and then some like Loos and then a bit um and and if if you take the body of information on the vulnerabilities in software comps for example and you think is designed and does a very good job identify those being used what a

surprise you didn't get a whole bit stuff out of it and if a lot of the quality of what goes in isn't great then surprise surprise signal noise ratio you using is not be either and in that set this has some UHA flavors right and there there's one that I like to call the not actually a vulnerability thing usually when you look at it in the vulnerability databases it has something like

dis and this come to the fact that very often when people are trying to record a problem they don't have a particularly good concept of what the thing they're using or reporting when is being used and maybe they're not very rigorous in identifying or or articulating a threat model in which theability and I'm going to give you an intentionally B example it I want Searchers coming in saying said my work Rish but I saw a thing relatively recently where people have found a library and that has some features and if you used some of the features a particular way you could create a prototype solution problem in some of the things that have loaded for right

and but to do that you needed to have control over the way that the library that the um utility was used right and and if you're executing code in that context you kind of won anyone right so what what threet model does if iute code ex more code makes sense this right people are laughing that right so that's that's one example and another is that um and this is this is more inherent the nature of the kind of I'm talking about yeah it's really really hard to provide the information or act on available information that tells you whether a particular issue applies to a Contex so give you a really really simple example and I remember sending ages and trying

to work out how we could s out a fendy problem for a phally nasty looking direct rsal and remote combed when I realized we probably couldn't fix it and I needed to work out the exact impact I could reproduc the vulnerability when I got to the really really small print it only affected Windows posts and of course we never any production Windows systems and so you know part of this who Lo to start with part is actually there a really really important bit of context that's missing one at the time and that even if you have it most of the to we have just doesn't have future set way ofing that's the next of issue that are things

where the do a good job in isolation but it makes people py right um anybody is a sec person find that don't like them yeah

security um but you know very quickly if you work in an environment where conly putting things in people um they will sometimes become hesitant to respond to you and one of the is we Happ these security tools particularly with industry cultural changes light everything has to shift as far left as possible and that generally means that you're putting as many things in as many people's way as early as possible and that sometimes means that they end having to so an example of this actually I can give you examples as well um let's say check maybe it checks the secrets being committed maybe it checks for dependen is being to have a problem those things are things you absolutely want to look

for but they're comparatively rare right um do you want to shift as far as you can those issues probably the credential do dependencies well don't know and I can tell you in experience with one has um I found a ground of one instance where a developer was prevented from introducing a vulnerable dependency um and several every week where the tool was too done to realize that they were fixing the vulnerability in producing it and stop them from Computing request say so right and you know itres the problem you know if you're going to introduce something that add a control to the way someone works if you're not really really careful about get that thing right are very very

quickly start and so you know always my practice was introducing children organization especially integration people's workflows put it on in sort of stop mode first get the intelligence start things and and and this of second point is actually really really important because you know we've got Limited time we've got limited ability to execute on things ourselves and one of our most valuable assets when we're working within our organizations is our ability it's the ability to get people to do you cheerfully allows you to carry out work by yourself right the social capital aspect of your role really really important if you told the problem the third thing that I want to talk [Music] about the third thing that I wanted to

talk about is that actually you know what a l of these t a little bit so on tooling Anda analogies there withou Cod tooling analogies with right and we like them as I said from theide because they give us metric we want to build that processes driven so that we can set our course and adjust based on the information we have that becomes really really addicted over time right you got you got information metric and you've got a thing that you can Benchmark something against and immediately you'll focus on that thing and the difficulty with and spending your time chasing with metric is firstly you're not necessarily checking to see whether the data you're acting as the

right data it's data and it might be right is it the right to focus and secondly it tends to make you miss surpass that there are lots of immeasurable things that are incredbly important as well and this becomes really important in context of the kind of problems that I de with in s is because immediately people will want reports on the things they can measure and that would become that you yourself measure let's take the example OFA to start you've got um s projects and it might have a few hundreds of thousands of lines of code and it might have a few hundred direct dependencies third party code and that uh translates to probably a few thousand trans dependencies right

and when you add body of you haven't well you know there load and load be Millions T Millions people of your and really really important that you take the security things in their seriously but have you know which are the most important issues draw a picture Tre you have yeah now one of the difficult things about SCA is that a book the library as demonstrated doesn't necessarily translate to exposure to even if You' got that n c 9 something and C execution really really really bad but if it doesn't apply to you it doesn't matter at all

and get and my experience is that the severity and drops really really quick the deeper you get into the defend so your direct dependencies you got to take very seriously indir fenses most of the time issue some of the time [Music] um first can we make a guess there a lot of problems in a particular tree one of those dependencies is a problem you just high once well um when you dig into it you know those things that work so what else you do and what you learn is that actually what your SCA tool is telling you is how well you did it's not telling you how bad things are now they're telling you how

you did and the real thing you want to fix is dependencies that you rely on directly that you can't respond too quickly there a so you know take yourself to um December a couple of years ago and um you mind your own business is Christmas and something like you drill youy fix is horribly easy to exploit and it's you know catastrophic impact and all that stuff um the real thing you need to be able to do in that situation is fix all stuff quick and so what the tool has encourage to do is to play whack all everything that's not realistic what you really need to do is to take look at what it is

that you've got that you can't upate tomorrow it be eat so don't ignore the first but it turns out that the second is actually a really good way of solving the first problem because what happens when you start that problem is that firstly organically all the stuff you got L around has ages get sorted out you know sort certain the root of Defense tree and rest but more importantly tomorrow when you get the report you've been dading you know that you're in a good place so yeah to Su up what I talked about first signal noise that's horrible security to don't help you a lot with high signal ratio secondly they annoy the people that you really really don't want to

annoy and lose you all with your organization really careful about how use and finally make sure that when you're introducing security you're mindful that not only what it can do for you what it doesn't and think helps so