Tradecraft vs. Toolkit: Who's Truly Important and What's Actually to Blame?

uh this is tradecrafters toolkit who's actually important and what's actually to blame i'd like to give a special thanks to the b-sides team for giving this opportunity to speak with you today this presentation is born from an experience of developing operators and comparing their skills against security products and then relying on people at the end of the day to solve those problems so let's talk about what brings us here for this one how many times have you heard something similar to this that we've invested in the most cutting edge security suite available now we'll be safer what do you mean we had a breach why didn't the system catch it i mean obviously the tools are

ineffective if my new hire can't catch and stop everything from day one well netappy didn't work so um guess the site's secure hopefully you haven't heard the last one too often uh tools don't only become a crutch for the analysts and the operators but they also unjustly receive a lot of blame as to the reason that a team isn't enough isn't effective so why are detections why are mitigations failing in this age of unprecedented technology what should we do whenever the exploit just doesn't work on that pin test in each of these situations the development of each individual's or the team's overall tradecraft that's what will determine the success for the failure so this talk focuses on the importance

of developing people and their abilities to create a more efficient and effective crew we're going to discuss when tool reliance is appropriate and when it's a hindrance now this is not any sort of rebuke on tools or their usage and no tools be called out specifically but this is a discussion on the importance of skilled professionals so getting into this what we'll be going over here is we're going to define tradecraft the necessities of the industry as you can see the bullet points here now this is intended to apply to blue red purple white black gray beige defenders aggressors developers professionals in general we will define tradecraft so i'm not going to spend too much time confusing

everyone on all of that oh yeah looks like i have backed up here sorry tech problem's always great right so i'll also hit on what i'll refer to as the necessities of the cyber security industry these are going to be tools and people and then from there we're going to quickly discuss a few of the challenges that are presented by an overly tool-reliant approach to security now that's going to help set up the next bit where the value of investing in personal professional growth and skill development will be highlighted a fair warning on that it gets a bit so boxy uh don't worry i'll try not to be too preachy whole thing we're going to top off with

kind of a proposed road map to success this is merely suggestion or possibly starting point there's no end-all be-all solution so let's dive in so first who am i well i am wall-e like the robot although sometimes i am the walrus now so i'm a cyber i am a calibrator turned cyber security specialist um i conduct defensive server operations for the united states air force i have several responsibilities in this role some of the most notable of these are cyber threat emulation cyber readiness which is to say network and vulnerability mapping and analytics conducting training and weapons and tactics which sounds much fancier than what it is a lot of documentation but enough rattling on about myself

that's not why any of you are actually here i say tradecraft what am i talking about most people when they hear the word think of spies and who can vote one definition is definitely the techniques and procedures of espionage but obviously this is not what i mean the definition that i'm leveraging is skill acquired through experience in a trade notice that it's not aptitude through training it's skill through experience this is critical to understanding what this discussion is about training is a key component and i don't want to discount the value of it everyone needs foundational knowledge but it's experience that is the solidifier of that skilled personnel are vital to the success of any venture

and they're the greatest asset available so when i speak of industry necessities i'm referencing technology and people we deal with technology so we need it right i will generally use the word tool to reference software platforms technology scripts and so forth tools are absolutely necessary as networks grow in scale complexity and geographic disparity the ability to defend test or attack everything necessary with just a handful of people and the command line becomes less realistic at the very least it becomes less effective we have to be able to leverage and rely upon the hard work of others now while technology is a necessity so to our operators and analysts with solid trade craft one might even argue that they are more

important who makes the who do you think makes the tools we've all seen through reports of breaches and compromise the impact of failing to make people skilled in the art of security for the sake of making them tool jockeys so bar for an old commander of mine we don't want to reduce our operators to push button get banana a tool jockey only knows how to push the button and then becomes paralyzed or panicked when there is no banana or worse doesn't go grow concerned that the banana stopped appearing as long as the button works everything's fine you get the banana what happens when the button doesn't work how do you get the banana now are you even sure the button is giving

you as many as it should or even as many as it could this is where you need your skilled operators these are going to be your people who can fix the tool or get the banana another way or even double check that you're getting the right bananas in the right amount so we're going to break up the idea of skill operators a little bit more here so remember how i said the tradecraft is skill through experience not every skilled operator can do everything there are a few that can but they're rare the skilled analyst takes many forms some of which i'll present here and we're gonna expand on a couple of the most prominent those being the three highlighted on

screen i keep in mind that these are just my interpretations of skill sets and they're in no way definitive or inclusive so starting off some of your most valuable talent are the tool champions they use the tool but they also know how to leverage it more efficiently and more effectively this is the talent that's able to troubleshoot like lightning and tune out the false results they set themselves apart from the tool jockey by understanding how the tool does what it does and by being able to do the same thing temporarily when the tool fails albeit less efficiently you can trust their setup and the results that it produces often your tool champions are the they

that you will tap to train others because you trust their expertise these individuals keep their finger on the pulse of capabilities and can provide expert recommendations on what to use to best accomplish your objectives your tool champions work hand in hand with what i call the concept of depths these are your lead hacksaws you're lethal for instigators this is the gal who reads assembly for fun the them that develops the new covert channels or the guy who solves a memory forensic ctf challenge via them and that is a true story by the way there are also those individuals who can put together a team of complimentary skill sets they can guide them to a 70 solution and

save the day this is who's going to get tapped to provide the expertise to build out technological functionality your concept adapts they're going to be your signature authors they're going to be your exploit writers now i need to stress here you don't have to be an absolute master to be a valuable concept adept filling this role requires a solid conceptual understanding and an ability to implement concepts in real world scenarios now no one ever knows everything right i can't say that i've heard of anyone knowing everything they need without some sort of outside influence or research your training gurus will get your people to where they need to be it's a skilled and trusted role and it's

vital to any long-term success these individuals understand not only the technology but also the people behind and in front of it this is who's going to be trusted to develop and also deliver solid improvement in the people that you and your team rely on we'll get to our road map of success later on but who do you who do you think is a cornerstone of guiding people in the right direction to speak frankly this is often one of the most underappreciated roles while also being one of the most vital so knowing what we're working with what happens when the overall approach is too too reliant well the excuses flow like electrons through cat7 we have too many tools so our operators

aren't able to be effective the technology doesn't work the software isn't doing the job i expect i can't get the data retrieved for analysis we're suffering from alert fatigue so we missed that critical event i couldn't get that metasploit module to work so i just didn't test it so what can be done about this well did you ever look into why you have those tools in the first place was that technology even meant to work the way that you're trying to use it do you have the right thing for this are you sure these are not the droids i mean data sets you're looking for what if you didn't look at everything but then you just looked at the stuff

that mattered and is metasploit really the only way to conduct a pin test so a tool champion could have told you don't use that tool because you're not going to get memory information from a nids your training guru would have taught you or sure hope should have taught you that you should not try to run volatility on a dead disc to retrieve pcap a concept adept could have given you a new or a different exploit to get that critical finding that you needed in that pen test the right script could have easily weeded out some of those known benign alerts from your spreadsheet which would have allowed your analyst to focus more on the actual anomalous behavior

and the potentially malicious

focusing too heavily on the toolkit without the complimentary refinement of tradecraft introduces a myriad of issues how much time is wasted in training to learn a new system even though it does the same job as the tools you already have countless man hours being lost as you train entire departments on something different that just does the same thing in a slightly different way how long does it take to get performance back up to an effective level after introducing a new platform you've lost the time in the training already but it doesn't stop there every system every process it has an accompanying learning curve that still has to be worked through that will slow people down

your experienced analysts become overworked because the work still needs to be done but now those tool jockeys don't have the ability to function because you've yanked to the crutch out from underneath them forcing one or two people to do the work of 20 isn't going to go well for efficiency it won't be effective and even worse is when they're expected to do all of this while they're teaching their peers overburdening skill is the quick a very quick way to lose talent now that you have everyone focused on learning each new piece of tech that is constantly being placed or refreshed or augmented where are your operators gaining the experience that they need to develop that skill

who's carrying your operations into the future you can't rely on the same experienced people forever humans quit go on vacation get fired retire get sick and cease to be able to work for a variety of reasons the replacements they have to come from somewhere so how many non-problems have been solved quote unquote buy new features instead of buying that new av maybe you should look at tuning your edr and your sword to respond to the situations you're worried about there's always a likelihood that you aren't catching events because events aren't even occurring luckily a skilled team can help you figure that out how many of those so-called problems or how many actual problems we're caused

by the new features if you don't have the knowledgeable people you're introducing a greater chance for unexpected errors due to technological incompatibility so what happens when and i do mean when the technology fails if all of your focus was on using the tool and now it's gone what do you do is it just time to pack up and go home while a malfunction an error or a crash can definitely create a work stoppage it shouldn't be the case every time wouldn't it be great if someone was able to continue getting the job done even if not at the same pace as the tool but until that functionality comes back you're still moving forward and then worst of all if all you know

are the tools and nobody knows the concepts underneath well then how do you know that you've found or you've missed bad how do you know that you've tested the vulnerabilities to which the client is most susceptible so someone once asked the question which is worse false positives or false negatives almost universally the response is false negatives i can't say universally because there may have been a troll or two trying to spark an argument but why would so strongly the false negatives come back as the resounding answer because missing the bad thing is much less damaging than investigating the benign thing so you've been trained to id what someone else calls bad but this company you're working with or

the working company you're working for is forced to use those practices legacy systems can sometimes require legacy configurations that newer technology just doesn't understand with all of this it's going to be experience more than anything else that really helps discern the malicious anomalies from the night

so what should you do or what should we do as an industry well invest in personal technical growth if you run the place grow your people they get the job done if you aren't in management or in leadership or i'd even say if you are improve yourself because you also get the job done growth can be fostered in a variety of ways everyone always thinks about going to training but that's not the only solution creating opportunities for yourself and others is one of the greatest methods of refining your skill set these opportunities should include but obviously you're not limited to hands-on experience leadership opportunities and even opportunities to fail allowing failure fosters growth controlled failure leads to fuzzing we

have an entire section of industry dedicated to breaking so we have to try and manage risk not let the fear of it paralyze us the organizational toolkit should include a minimum of numbers and minimum number of skilled personnel in critical roles make sure that the team loadout is postured for success and groom the newer members towards those roles you're going to need long term as security specialists you'll find yourself in a variety of unique situations and faced with problems whose solutions are far from obvious how do i know that my alerts work why didn't the ids catch that what should i do when that exploit doesn't work how do we handle a situation technology hasn't

been designed to cover each of these situations it's the trade craft inherent in the operator and the team that will ultimately determine the success or failure of the operation as you build your personal skill set as you develop your organizational toolkit you'll find yourself better equipped to tackle the complex issues honing your abilities and the abilities of those who work with and for you creates more efficiency generates more effectiveness and raises the overall value in both yourself and your group this does include creating a greater proficiency with the toolkit remember the importance of the tool champion now all of this takes time and on a personal level takes commitment creating a skilled team or building

yourself into the best is a costly endeavor it takes time opportunity and money sometimes opportunities must be created training and technology oftentimes don't come cheap in this industry time is your most valuable asset and there never seems to be enough of it all of these can seem quite daunting but there are some things you should ask yourself what have i gotten out of my previous investments what will i get out of this investment how do i actually see returns are you investing wisely you might have gotten a new tool but does it actually do anything fundamentally different than what you already had it'd be great if you had someone knowledgeable enough to point that out

before spending the money the bells and whistles on that new server sure are something aren't they does it actually perform any better than the one i already own do you still have the expertise on your team to accomplish your goals if not you can try to get it back but it's gonna take time and will prove costly did you ever develop the personal expertise to hold your own in your chosen field if the answer to this is no you do still have time it's not too late you can do it you saved plenty of money overall by buying the software and the hardware instead of sending people to training conferences but was it really worth

the loss of talent that you've been experiencing so what reason is there to invest in creating skilled people or building out your own tradecraft why not just get tools to do things faster or may not even require human interaction technology just keeps getting better so that's where we should focus right i'll answer those questions with question is it technology or people that make the tools you'll find that as you invest the time and the effort into building the trade craft the return on investment is going to come through there's nothing like finding the one piece that was needed to complete the puzzle speaking of the person who seemed to know it all on an intimidating level when you

started and being able to hold an honest technical conversation without getting lost or feeling like you sound like an idiot is super satisfying and a satisfied team is a productive team also considered the most revered professionals are those accomplished in their craft well-developed tradecraft provides flexibility and operations that is invaluable it is the best way for your analysis to be able to adapt and evolve in this ever-changing landscape sometimes that vulnerability is not going to have been published yet there are times where the technology hasn't been developed yet but these don't need to be impassable barriers to success unique problems are going to require unique solutions and in the realm of cyber security there is no shortage

of unique situations the greater your trade craft the more efficient and innovative your solutions can be the road to success is paved with lessons of failure don't give up or get discouraged just because you don't get the concept right away don't feel that you're succeeding because you can't do the job without a particular tool allow people the chance to fail forward try looking at every challenge as a learning opportunity and teaching others to see it too a very rough road map can be laid out something like this so we're going to lay down a quick idea of how we might get where we where we wish to go this isn't groundbreaking but hopefully some of you may find it

enlightening or inspiring most of this is going to be approached from an individual individualized and a personalized kind of viewpoint organizationally and as a community the biggest thing that we can do is allow and enable this journey in ourselves and in our peers so we have a starting point right and we all have to start somewhere somewhere is going to be reliant on learning the operation and learning what matters during this phase the junior analyst will be largely sometimes wholly reliant upon the tools this stage can be overwhelming and potentially seem oppressively long but we've all been there and we all make it through for the leaders out there remember you were once in this stage yourself

you find yourself in this phase pay attention to as many details as you can soak it all in as you have personnel in this stage feed them those details give them the wise give them the house junior analysts should start to learn what it is the technology that you're leveraging is doing do your best to begin learning how it's doing what it does compiling and delivering reports can actually be a great method of building this understanding it can help to highlight what it is the organization and or the customers actually care about it's also a great opportunity to learn the data that matters and also to provide results to compare your own endeavors too no junior analyst should ever be afraid

of starting to dream up better ways to do things the junior analyst has a fresh set of eyes not marred by the that's how we've always done it mentality trying things out safely in a sandbox and not on company infrastructure you should be encouraged

so as you advance in your career you find that it's less about the specific software and becomes more about the concepts and the objectives that edr definitely makes host analysis simpler but you've learned how to leverage powershell to do the same thing with experience we all find ourselves relying less on the technology and instead leveraging it to accomplish our goals the tools are great alone but what if you bridge their functionality then they might become phenomenal taking advantage of growth opportunities such as training conferences meetups and even mistakes the hands-on experience is the solidifier that creates the trade craft instead of just retained knowledge this is this progression is where operators truly start to learn how to apply

as analysts advancing their career you they'll find that it's less about the specific sorry i got completely backed up there so anyways all the solutions that have been dreamt up as a junior analyst how do they start to get implemented so i know it says made it and eventually you'll be able not only to see what needs done but how to do it and what to use to best succeed but how do you know you've succeeded how do you know you've made it you may never feel that way we exist in a space that is filled with imposter syndrome because so many of us know that there is so much that we don't know we can't

worry about that you know so much more than you're giving yourself credit for organizationally success is incredibly subjective personally it's still incredibly subjective but might be slightly easier to generalize i can't lay out what success is to everyone but this proposed roadmap does have some expectations of those who've reached the goal providing technical insights into higher level decision making of whatever organization you're a part of is vital to the continued success of your operations start presenting to your peers even if it's just over beers to a few to a few friends or colleagues and you never present to a local group or ever at a conference share your experiences with others help the community move forward

this road stretches further into the horizon than we can possibly see so it requires the continued development of its travelers the next generation of skill has to come from somewhere you may not be teaching their classes but you should be training them to fill your shoes when you are inevitably gone being able to take security concepts and apply them in a real world situation is the greatest skill that you can develop and the greatest asset that you can bring to the table and so what's the tldr you need the tools need to know what they do and need to know how they do what they do stop blaming the tools ignoring the training stop blaming the training ignoring the

tools you need to invest in yourself you need to invest in your people and never forget the most important asset is you thank you for your time and if there are any questions

all right i guess sounded crickets means all good in the room so um yeah uh contact information on screen and for any questions shooting my way i will make my slides available to the b-sides team however they want that to happen and thank you very much everyone i greatly appreciate it

thank you logan we really appreciate it uh i'm not seeing any questions in the uh in the go to webinar chat if there are any other uh questions feel free to post them in the discord or you can ask them here

oh seems good all right well thank you wally we really appreciate your your talk is very informative and definitely indicative of of what i see you know in my day-to-day work as well that's what sounded very very similar to our to my situation uh well thank you very much yeah um i think regardless of what section of this we fall into whether it be public private red blue however you want to look at it we all kind of stare down the same things going along right so at the end of the day we're the ones who get to make the difference this is true so a lot of folks in the discorder are saying thank you great job

ngl imposter syndrome is real yeah way too much suffering from that even on a personal level i hear you no thank all of you and thank you thank you so much for giving me the time to to actually present as well

you