StephenMathezer

foreign

we're at the closing keynote everyone thank you for hanging around how was it [Applause] well you know what the closing keynote means it means right after this you're going to just go and just chill out I think we're gonna do that too so I'm gonna keep it real short added an overconceiving officer thank you so much for being a closing keynote really appreciate it you bet I think he's hoping I'd pack the room but yeah no uh so thank you everyone for sticking around we'll we'll try to get you out of here in a reasonable amount of time but I do have some stuff I want to talk about and one thing I was going to bring up uh

this I this wasn't planned but I was talking to Quinn earlier uh about uh getting presenters for b-sides and uh you know this year I think what some 35 folks put their hands up to present um but it is still I think as any conference organizer will tell you hard to get people to put their hands up so I've uh said this to folks privately I've never really stood up here and said it so I may regret this but you know it doesn't take much to come up here and talk come on it's not that hard I'm not you know anyways so anyone thinking about doing this in the future I'll put it out there

um if you want a little help a lot of help someone to review slides someone to co-present whatever um look me up I'll be happy to help you out because I think it's the best way uh to get get out there and and have fun meet people learn stuff I teach uh that's probably where I learn the most is my teaching so just encourage everyone of you to to take the plunge and and get up here and give it a shot so I'm I didn't really uh um we didn't put a an introduction or description of this talk out there the call for papers came out on a day when I was just uh let's just say grouchy it's

been one of those days um and you'll see why hopefully as I go through this but uh this this talk totally came out of my mood that day um dealing with uh some challenges around other people troubleshooting really so I think this is really important um concept it's quite overlooked it's something I'll Branch off in other areas uh and uh you know there's a lot of folks new to cyber security here so hopefully this will give you some food for thought and those of you that are troubleshooting experts while you can sit here and and commiserate with me uh as I go through this and and we talk it over I'm just gonna make my speaker notes a

tiny bit bigger so that I can see them without sprinting and oh that's too big there we go all right so our goal here is is really how do you solve problems faster and as I mentioned you know take me from the top picture to the bottom one right like there is nothing worse than listening to someone tell me all the ways they didn't solve a problem because they really maybe didn't approach it in the right way troubleshooting is not the easiest thing in the world some of it comes with experience uh some of it honestly probably requires a little thought um something we can maybe be do a bit better job of teaching and uh

it it really shouldn't be that that hard I want us to resolve things faster make customers or you know customers whoever those customers may be happy and most importantly reduce stress on everyone involved it's it's I don't know I see problem solving troubleshooting is something for a lot of people you're either good at it or you're not but there's a lot of things that we can uh do to help ourselves get better at it and honestly you got to be a little bit careful about being the person who fixes all the problems uh but it is a great way to show your value to get out there to meet other people to learn how things work

and and progress in your career as well so this is a security conference what does troubleshooting have to do with security well where do people go who do they point fingers at right in a security role if if the Network's not to blame well then who's next right it's either Network or the firewaller security or whatever right it's one of those things uh after all when is has it ever been the fault of the vendor or the application or the server or any of those kinds of things that doesn't ever happen does it it is a similar skill set I would argue to instant response threat hunting forensics all of those disciplines of cyber security they're very very they

use the same skills the same capabilities as um troubleshooting you're basically solving problems hunting for things looking for anomalies and of course in an operational security role you tend to have a pretty diverse knowledge and skill set and we'll I'll come back to this a little bit uh as well how important that is in security we we touch Network and web and Os and endpoint and authentication and scripting and logging and just about everything else we have uh lots of visibility visibility that's not a hard word um you know Sim we can scan we can look at Network pockets and logs and everything in between and of course we have the best toys right that is why you

want to work in cyber security because you have a budget I think maybe nowadays and you get all of the best toys and unique you meet friends and make relationships with all of those people that you are solving problems for and that you can get to help you solve problems because those you know I I kind of jest but those relationships especially in a security role are really important um for any number of reasons um especially when most often we're in the position where we know where the problem likely lies or the incident or the issue or whatever but we need someone else to do that thing to get us to where we're going so

those those relationships are pretty important and I would argue that troubleshooting is not really that hard but there sure are ways to do it badly and this is the commiseration part um you can let me know if you recognize any of these and uh hopefully this doesn't trigger any PTSD or anything like that for for any of you folks that have been around as long as I have strategy number one give up hope it goes away you know right like like let's just start with failure right like I this is too hard I can't deal with this who hasn't been there right it's not really my problem you're I'll just end up getting blamed I've got better

things to do I'm busy you know there are definitely days where I want to to take this approach and and give up and hope it goes away but this isn't going to get anybody anywhere right and certainly not to make friends and influence people closely related there's there's several that are related here closely related to give up

is they find someone else to figure it out if if I can't just ignore it and um if if I can't wait for someone else to do it then I'm just gonna go and find somebody else to do it because I really don't want to be I don't I don't want this stuff stuck to me right I don't want to have to uh have to deal with it we can try to actively make it someone else's problem one of the ones that will really I could rant on for a while is is this approach right let's just try a whole bunch of random things and and hope that that whatever it is going on stops and and ideally we don't

cause any other problems right uh you know there's many many issues with this approach first of all there's my beautiful Venn diagram right there's what we throw there's what sticks and there's what works so even all of that stuff that sticks how do you know if that's actually working there's a lot of things that might look like they've stopped look like they've worked but what which of those things actually fixed it what actually did the trick and and for that matter how much mud do you have around to throw at the wall to see if it sticks right like how many ideas how many different strategies are you going to try I have definitely worked with folks who

are so busy trying everything else that they never actually stopped to think about what might work I've had you know someone come up to me trying to solve this problem I've tried this and this and this and this and this is still not working hey have you thought about that and now I'm going to go and try this and this and this and this and and maybe that'll get me closer hey look over there have you tried that and and if that fails I don't know what I'm gonna do like come on uh so there's no um shame in trying multiple things to see what will work but let's do this in a uh somewhat ordered fashion and if it

doesn't have a hope of working then it better be real quick to try it eliminate it move on like don't don't waste your time uh let's let's spend our time on the things that we think might work and I think lastly oh no not lastly second lastly there there's you know oh good another good old standby right oh somebody must have somebody else you know those previous three slides somebody else must fix this somewhere along the way let's just go to the next version and and hope for the best right what could go wrong and yeah hopefully um you know because our vendors all have good track records right I upgrade it's going to fix problems not create any new

problems Maybe and then coming back to the security side there's good old Swiss cheese let's just go with the permit any any are you sure yeah sure why not uh so these are all of the approaches that I see way more than I would like to see um you know some of you that have worked with me in the past have probably know what I'm talking about or who I'm talking about but uh yeah let's uh see if we can come up with uh something a little bit better here but first a quick tangent I know I'm an artist trust me but um why is this such a challenge my wife yeah tangent is doing her PhD in

education um and yeah don't get me started I I can you know um what's the word you know transitory transitory talk about all kinds of things uh to do with our curriculum in education and so on we won't really go there but as I said I came up with this presentation in a fit of rage or Peak or frustration or what it was in a moment um because I was working with someone who was basically stumbling around in the dark doing everything but solving the problem so I was talking to my wife about this talk and she's been doing a lot of research into how we teach uh kids to read like little little kindergarten

grade one kids to read spoiler not everyone's the same not everyone learns the same not everyone approaches these problems the same the example she's given me and this boggles my mind so we we had I don't know when was it how old's my daughter when she was in elementary school so maybe eight ten years ago we had Discovery based learning right sit the kid down expose them to stuff and osmosis happens and they learn right now the pendulum swung back the other way and we are teaching the basics nothing beats the basics so so much so that the example she told me is there are literally classrooms where they are not allowed to use any books and again

like kindergarten grade one they're not allowed to teach kids to read with any books that have any words that aren't pronounced the way they're written so you know what this is awesome for test scores my kid can read anything as long as it is written I mean uh pronounced exactly like the kids have mastered phonics the scores are off the charts but there's so much they need to do to cope in the real world they have to be able to relate what they are reading um to something to an experience to something they're familiar with to a context right if we teach kids and we just tell them stuff and they have no idea why it's relevant they're not going

to learn anything it's a bit of a tortured analogy but the point I'm making here is foundational stuff absolutely matters you have to know the foundation but just learning a thing by wrote is not going to get you anywhere we need to uh come up with ways to um enrich that extend that take it further so we don't want to start with uh and you'll understand where I'm going with this in a second we don't want to start with hey I know every thing about security systems I don't know what's the network what are packets whatever pick your thing uh we need those foundational skills and I think it's it's somewhat reversed in cyber security today we need those

foundational skills but we also need those higher order skills uh like the problem solving some of you may find this hits a little close to home so I'm sorry but we have many two-year technical programs how much can you cover like cyber security how many vendors are there I once put together a slide where I tried to list every separate discipline or kind of cyber security like every product category I was down to eight point fonts in the out of room on the slide right like there are there cyber security is a massive massive um discipline or landscape so in two years what can you cover well you can go about a little steep and

maybe well not even this wide right this wide so we've got a lot of folks who um you know get just that surface level introduction to cyber security um and and honestly very little uh introduction to things like operating systems uh that's that's a big one how systems work how they interact how how does a web server work how do any of these things fit together so then to solve this problem they go and get their masters and they become experts in risk and compliance and still we don't have any practical I don't want to say we don't have any useful skills because that's not true but you get where I'm going right um so we have

bad security decisions that result from that lack of the Practical and the business understanding because you don't get started on the business context as well with security folks who've just gone to school and learned what is right and what is wrong give you a hint in the real world there ain't no right and wrong um and then we depend on the magical tooling right who who hasn't had a vendor or talked to a vendor who's got a tool that will solve all your problems right yeah uh and and it's very hard to troubleshoot or solve some of these problems with the lack of foundation right so what the reason I'm taking this tangent is I just want to get you

thinking about what are you doing especially the folks that are newer to cyber security what are you doing to fill in those gaps right how are you um what are you doing to help yourself uh to further your your learning in the areas that that probably haven't been covered because like I said you need a lot of Knowledge from a lot of different places and I I often talk about imposter syndrome it is very very real I was talking to a friend whose daughter is looking for a job and is convinced she can't do it I'm like you know there's a dirty little secret the bar is low it is very very low your goal is to be about

here you know a couple inches above your audience and the truth is all of us folks standing up here that's about where we are you know the bar is here are just sliding right over here and we look look brilliant at least I hope we look brilliant I don't know um so you some some things to think about you all know something that nobody else knows every single person does and the person sitting next to you knows something that you can learn from right you should be taking every opportunity to learn but recognize that you have some piece of knowledge you've had some experience especially in a field this broad um that you know uh something that they

don't so things for those that are getting started just want to get you thinking you know security change is faster than just about anything else that in this this you know general area that is to your advantage because that means by tomorrow I'm going to be an idiot and you're going to know more than me right like like that's not too untrue uh definitely you need to understand the foundation uh you know if as much as you may not want to do it if you've got an opportunity to take an operating system internals course jump at that opportunity that is knowledge you um we'll use for the rest of your career just the way the the underlying pieces

go together learn how systems networks interfaces applications work learn how to learn of course learn how to solve problems and then apply that knowledge in a security context I I really uh you know I will say the the best additions to our team have been people that have other experience you know 10 years of desktop support server admin whatever it is coming from other places with some of that knowledge and that desire to learn security and build on top of that so I think you know those are really important skills to get somewhere along the way somehow a long way and that doesn't mean go and you know spend your life in school doing that I'm just

saying get some exposure to some of these underlying Concepts so let's get this thing back on track troubleshooting is where we started some key tenets when it comes to troubleshooting simplify yes keep it simple compare everything in troubleshooting is some form and and incidentally for those of you that have have looked at fuzzing fuzzing is just another way well that's actually more sorry fuzzing would be analogous to the picture of the mud against the fence but the point is that you want to do something and compare do one thing compare the results see what happens compare working with not working right process of elimination eliminate you know try some things rule something out do some things to intentionally roll

something out move on of course validate whether you actually fix the problem get under the hood um very rarely have I solved the problem just by interacting with an application it's by digging in looking at source code in some cases looking at log files looking at scripts looking at what's going on in memory anything well and of course my favorite network packets but we'll get there um you sometimes have to dig in a little bit but know when to stop spinning your wheels like wait you know when we've got snow coming tonight right you know when you've got that you've dug that uh trench in the snow spinning your wheels let's let's not get to that point

foreign so being a bit more explicit some approaches we have I I even laugh saying it we can read the documentation uh it look at I'll talk about some of these as we go you can search for help you can ask for help as I said you can enable logging because it might not be there check your logs simplify as I've talked about there's some other approaches like changing your perspective or your approach sleep ing on it is an option that sometimes works wonders talking it through changing different device different browser different network different location different perspective something like that right and of course the process of elimination we can start at one end work our way to the other we can

start in the middle the point is don't randomly start here no that didn't work okay no that didn't work oh that didn't work like we want to do this somewhat methodically so I'll dig into a couple of these just to to um you know provide some more detail I guess starting with everyone's favorite the documentation documentation is awesome right it's easy to read it's clear it tells you exactly what you want to know this is where I'll talk about searching in a second searching is also a skill because you're not reading that documentation from start to finish you're searching for what you need in there but honestly some of the dock I've seen makes me not want

to bother and that's actually a bit of a problem because there are nuggets in there even if it doesn't directly say when this happens you need to do this you get those little pieces of information that tell you how things work you know you you get some insight into the assumptions that the Creator made or or what you might expect to see happen so you know you do have to fight down that that I don't want to bother kind of instinct um not to say though that documentation is going to solve all your problems but you can't fix something if you don't know how it's supposed to work so understanding how it's supposed to work

definitely does have value uh and like I said you might find some clues in there right logging boring right who who has time to sit through well you know once you've finished reading the manual then you can go start reading the logs line by line right um obviously we have system logs they're there they often provide some valuable Clues application logs though they're valuable sometimes they're sitting there you know don't don't go read the documentation to see what logs there are I mean do that but go hunting around because some of these apps leave stuff everywhere you can find lots of Clues as to what they're trying to do what they're failing to do and so on

sometimes you need to enable the log so do that turn on debugging whatever see what comes out you might get some Clues we have external logs firewalls proxies authentication Services whatever client side logs server side logs web server logs there's there's stuff all over the place so don't don't restrict yourself I guess just to the thing that you're working on and generate your own I have JavaScript python whatever it is so don't put some logging in there for yourself I mean let's do it with intent in the right place but certainly there's there's uh you know when all else fails do it yourself right bottom line especially if you're dealing with apps you know the same apps

consistently is figure out how they work and how to enable the logging that you need um it's not necessarily glamorous but one a skill it is a skill to be able to extract useful information from you know a few 100 gigs of logs getting down you know I've I I probably won't tell this story later so I'll just briefly mention it now I call the vendor and said here here is a 500 megabyte packet capture packet number 131 702 is your problem go fix it right like being people the people who can go and get to that level and quickly pull the relevant information out of out of well packet capture or out of a log or something

like that those are incredibly useful skills as well I know aren't we in the best most glamorous industry on the simplify slide side it's on a slide and on the simplify side I mean just simple things right isolate take components isolate them from each other test them each individually focus on the basic functionality you know this is where the whole concept of pinging something came from also don't just rely on that right let's start with basic functionality and work up from there and get one thing working at a time back to the throwing stuff at the wall whatever that stuff may be let's get one thing working at a time the internet that's the solution to

everything you know how do people troubleshoot and solve problems before the internet like I really don't know um first question am I searching effectively another skill set learning how to get the information you want out of a search engine uh because the answer is usually the I'm feeling lucky button is not I mean it's happened but it's not going to get you where you need to go um you know I started out by saying learn but Master Google dorks meaning if you don't know what that is well Google it but uh how how to make Google return useful results I only care about this file expansion I only care about things on this site I

only care about things with this in the URL or this in the the title or whatever you can really eliminate a lot of the crap quickly that way um speaking of Google also Google cyber security search engines um I I was reading a couple of posts recently this week on on Twitter yes uh let's let's take a moment to mourn for the end of Twitter um but uh um Google cyber security search engine there are things you know for searching everything from file hashes to to password hashes to to um I'm I've lost the plot I can't remember the rest but there was a list of 20 different kinds of things um that you can search for that are

non-conventional right like not just stuff on a website historical information I can't tell you how many times I found those manuals at the vendor documentation on archive.org right um vendor knowledge bases vendor communities they they tend to have they're different vendor knowledge base is a very different thing from a community of people who work with a product there tends to be a lot of knowledge there as well but I'll talk a bit more about how you should well on the next slide approach solving problems but don't just say oh it's broken call the vendor they can help me his trust me when I say the bar is low that's about the level that's about where it starts

right yes go to vendors they do have people who are very good at the vendors anyone come in from the vendor Hall I wasn't paying attention anyways um they you know they have very knowledgeable people but don't just default to hey this is broken help me fix it um you need to go with some information try to solve the problem first because I have support tickets that are months old so it's better off for me to I'm better off figuring it out first and when we're talking about the internet I often do searches to find the Search terms that I need to search for to find my problem right like zoom in narrow in so talking about asking for help

one of the golden rules balance right if you come to me

and ask for helps um on this one you've already tried and you tell me nothing then I'm not helping you tell me what you've tried what worked what failed what you've done what's going on um and and then I'll help you so I called the 30-minute rule spend half an hour for obviously this has to be proportional to the size of the problem but spend 30 minutes trying to solve it yourself then come to me and tell me what you try where you're stuck and then we move on from there right so please ask for help don't sit there spinning your tires don't sit there bashing your head against the wall forever but don't come

to me without at least trying to do any of that right so that's that's one of the golden rules the other one is to be prepared like I said tell me what you've already tried or I'll send you back to try again and because not only does that help you but it helps me Focus what we're doing from this point on how I'm going to help you how I can validate what you've already tried and please the uh whether this is someone you know who works with or for me getting me to help with the problem or whether this is a customer or whatever don't tell me it doesn't work that is the least possible piece of information

you wouldn't be here talking to me if it worked so why doesn't it work I want all the details now let me guide you through those details I also don't need my mother explaining to me exactly what she clicked on on in the 10 steps to get here uh and her theory that it's because you know mice used to have chords by the way that the mice cord was actually Tangled and when she Untitled the mice part of the computer started to work better like no but but seriously I do want some of the details um be prepared to still have work to do maybe I don't know the answer maybe I'll choose not to give you the answer maybe

I'll give you some guidance that might be all I've got that might be yes yeah but I thought I'd go off and try this and this no try this I'm saying it for a reason um go and do this and and see where that gets you and then come back if you're still stuck tell me what happened when you did that we'll move on it's red for a reason you came to me to ask for help listen to what I am telling you don't just keep mattering on and and saying oh yeah yeah yeah great idea did you hear a word I said um so you came to me for help stop talking for a second clearly something

I'm not very good at and use all of your energy to process and understand what I'm telling you having said all of that again coming back to imposter syndrome I'm not going to be part of it believe me I'm not going to be right every time so do challenge me tell me why you think I'm crazy um you know let's let's work through this talk it through together and go on so I don't know I have have some feelings about this slide anyways I mentioned perspective earlier taking a break sleeping on it it does work go walk the dog just take a notebook or a phone or something so that when you have that Eureka moment out in

the middle of the park you actually remember it I started writing this presentation in my head while walking the dog and yeah I finished it this morning but that's a different story um but yeah I I had to run home and like oh take some notes so that I knew what I was gonna talk about talk it through uh you know with a colleague someone on your team someone you work with I've solved so many problems that way or teach it to somebody teach it to your kids it doesn't really matter if you learn anything well hopefully they do but or or just verbalize right teach it to your pet your imaginary friend Casper I don't

care oh hey Casper I'm dating myself again anyways um talk it through explaining something makes it much easier um I don't know it does something for me at least and I'm like I I stop mid-sentence and go and like just let me try that oh no okay we'll keep talking and change the the setting and when I mean the setting I don't mean that the use a different device a different system a different OS a different browser right that is so much easier to do now than having to run around and find a piece of Hardware uh that you know foreign each of those criteria now you've got VMS and stuff try different ISP a

different network location uh sorry different network a different office different location log in with different accounts whatever it is in all of this let's go through process of elimination right start at the beginning work your way forward start at the end and work Back start in the middle and work towards the the problem um you know eeny meeny miny mo do something as a process of elimination this is a really quick example I'm not going to spend a lot of time on it but this is and and I'm not saying this explicit but what you're doing should match pretty well with something like this if I asked you to find the seven I know it's right there but

what let's just make this a really long list what's one way to approach it well let's start in the middle is it five well no it's not five so is it bigger or