Testing Your Threat Hunting, Cyber Threat Intel And IR Process And Procedures - Mike Johnson

[Applause] good morning so just a bit about me I am Mike I'm the global cyber threat and incident response manager at verone um the war games is more relevant to a previous organization um with you with me part of their model was looking for potential overe experience and breaking down the um the Cyber skill shortage and also the entry level roles being quite um difficult to achieve so part of the model of the business was bringing in people with zero experience usually from neurod Divergent or um veteran backgrounds that slightly more difficult to employ and giving them the hands-on experience and training that they require to become very competent cyber operators um it's a proven method and it

does work it's also a great way to test your incident response capabilities um and it also means you can offset some of the budget and requirements based on training because it's I um for your compliance things because obviously if you do your yearly tabletop you can kind of offset things so the rationale behind it um Hands-On training is obviously one of the most critical things for a lot of people and you will see job requirements that require years of experiencing a orc different tooling sets things like that um and depending on your background and or experiences it's quite difficult to get that hands-on experience um you might have done a lot of certifications you might have a lot of theoretical

knowledge but it's getting that handson experience and actually developing the skills that you need to grow into your career um it's also a great way to test your processes and procedures uh obviously you might do the yearly tabletop exercise and you'll pull out one of your playbooks and decide we'll do run someware or we'll do a malware outbreak this year and you're ticking that compliance box but it's just more of a it's for the compliance driving it as opposed to the actual enrichment for junior or entry level employees um so this way you're getting everyone to sort of test them processes and procedures and you can identify any gaps um any weaknesses in a slightly more coherent

manner because you're running it through in real time and checking as it goes and again I've mentioned it there's the global cyber skills shortage um by bringing in Junior and entry level people and giving them that Hands-On training um it makes it a lot easier to fill them gaps it also shares the knowledge and reduces skill silos um again if you've got with the skill shortage it's quite um hot market for cyber professionals so as a result you can end up going through quite a few soort of your turnover can be quite High and bringing people in and training them up should be seen as an investment as opposed to a burden um an enrichment and

development uh again bringing in Junior and sort of entry-level employees and getting them that s development and actually investing your time in them which is going to help in the future so again the why training upskilling um it's one of these things that being doing you your war games um it's like a controlled environment that you can train and develop people in but it also gives them that as close to real as possible again as you mature your wargaming process you can um delve further into the sort of how real it is so initially I'd always start off in like a very sandboxed environment and it's almost spoonfed but it's getting that sort of muscle memory in place and

then as you start to expand you can throw like red herrens into your production environment and things and it means it's a little bit more difficult for them to actually do sort of the threat hunting process or you give them sort of a slightly more arbitrary CTI task to then find and discover and build a profile on um and cross training um depending on how your organization is structured um you might have threat hunting functions you might have threat intelligence functions and you might have an instant response function again you might be all one team you might be siloed off um it's great to cross train and get that experience between everyone um reduces skill silos if for example you've got

one guy who does your CTI and he takes a twoe holiday um there's potentially a deficit in your squad and you've got got that experience across the board um and again if you've got entry level and junior level employees you put someone in as a CTI analyst they might want to do a little bit more threat hunting they might find that more interesting and it's about giving people the opportunity to grow their career and give them that sort of experience and examples of What's um what's out there and the other disciplines so who's this targeted at well if you're a team leader or manager and you're sort of having issues hiring or looking at retention or potentially

just you're not hiring Talent that's hitting the ground running um which is why we see these job adverts that need six years experience in anything and you expect it to be an entry-level expert um so again by doing this and planning these war games again depending on your organization you can run them at different stages of the year um and use it as a sort of you have keystones or milestones and you can hit every 3 months depending on how sort of intense you want to do it and you'll watch the individuals grow um again if you're an individual contributor um and you sat there wondering if you can get a little bit more Hands-On you can grow into your

career a little bit more and bring this up with your manager or your lead um again the benefit of doing it as an incident response training process is that you can usually get away with mitigating your yearly compliance requirement so it means it's a little bit easier to get away with sort of the Manpower and the time deficit that you're trying to justify um so planning the war games um again this will be completely unique to most organizations um it's going to be the one sort of try and grow them into the area that's going to be most beneficial for you initially and then potentially broaden out so as an um as an organization if you're using

industrial control systems or something that are quite bespoke to your industry there's not going to be that many people who are at a junior level or an entry level or less senior level who are going to have had experience of exposure to these systems um so you can look to war game around that sort of system and it's going to get people up to speed a lot quicker and also again strengthen the ranks again other organizations have different operating systems you could have IBM I AIX hpu and things like that that again unless you've been in a large Enterprise or a large or organization you're not going to have had much exposure to so getting that training and

getting the hands-on experience of responding to alerts associated with salus Os or even just doing a digital investigation on like a salar host that isn't the norm or isn't the sort of there's not that many documents or you don't read blog posts about how to do IR on salaris it tends to be Linux Windows sort of systems that people favor um and yeah ask where your team are feeling weakest it might be that they want to explore Linux a little bit more of your windows shop and again it might not be completely sort of um beneficial to your organization at the time um but getting that skill and keeping people interested I think is the hugest thing

because if you can keep driving that passion you're always going to have people performing better because it's that motivation and then you've got to find the right moment again this can be somewhat timeconsuming and it can be a time constraint for an organization and depending on how busy say your sock is um you might not have the time to actually sit sit there and go well we need to dedicate a week to this a lot of organizations have started doing Innovation weeks or hack weeks um this would be a great time to do it it's sort of that time where you can develop and sort of enrich yourself uh and this shouldn't be seen as a burden um it's the key to maximize

engagement so if you're doing it you can get stakeholders from all over the business um especially if you're doing the full cycle like IR processes and procedures um you can get the threat hunting aspect done you can get the executives in so when you get a caller 3:00 in the morning because the um there's a ransom we outbreak or you've lost customer data and it's getting that simulation to real as possible so again designing your war games uh initially as I said I'd start off with a nice sort of very controlled sandbox environment um it all again depends on how experienced the team you're trying to train up or get exposure to are um if they are

completely green or sort of very um at the beginning of their journey in their career um I'd say make it as sort of obvious as possible and it's that spoon feeding process because we can get the more information you're instilling in them they'll start to disc discover the journey themselves as they develop uh again the opportunity to build your lab can also be your first upskilling or cross Skilling section um security engineering or devops or systems Administration tasks things like that while it's not maybe necessary necessarily um imperative to the role that they're performing and again it's broadening that skill set it's giving them that slightly different knowledge about systems that they're protecting and looking at them systems in a

slightly more um like they have that knowledge to go behind it um and I'd always advocate for infrastructure as code where possible um once you've spent the time and you've done that big investment at first you can then just spin things up on sort of Q which then mitigates the issues later down the line especially the time issues so the example I'm going to use today is an all hands so we're going to test our threat intelligence functions we're going to train Junior threat Hunters our threat intelligence people and our incident responders um I always like to come up with like a fictitious AP group um it depends on how mature you want to go with it how sort of abstract

you want to be if you've got a sort of specific industry or your sort of attack surfaces in a certain area you can tailor it towards that um and it'll basically be sprinkling some technical indicators that you control or some C2 servers and other technical indicators and creating some ttps that associate to this profile um and then getting your entry level or Junior threat analysts to then build up the profile on it it could be just getting them to write Sigma rules it could be getting them just to start putting the data into sticks and things like that even just entering it into like M or their threat intelligence platform and just getting that hands- on

building the profile and working out sort of the different technical indicators and mapping and understanding the ttps that these threat actors use again attribution was mentioned in the uh previous call if you're an organization that wants to perform attribution which I know many don't because of the time constraint and the costs associated and this would be a great way to start building that um that skill set and getting people using the diamond model or their pyramid of pain and getting them in from the early levels and sort of in that controlled environment where there's no harm no fou type thing it gives people that real time to get into it and understand it uh at this point once the threat

intelligence has been conducted we can start looking at the threat hunting side of things again this will be in a again controlled environment we'll have some file hashes potentially some IP addresses that we do control again you can just sprinkle them about um and you can get your hypothesis written up it could be that you know that there's a AP group that are using Brute Force attacks from stolen credentials and then they're using ransomware that's B shell or poers shell based and you can start building your hypothesis of what to look for you've got a couple of the indicators to compromise could be that you've got an email address and things and you can start to build it and start to hunt from

there and then once You' finished your hunt um you'll you'll start finding the ioc's you'll find these indicators you can move into your incident response mode and you can start to contain and eradicate the threat that's posed to your environment um and again it's that full cycle you could do this as a single team of sock analysts it doesn't necessarily need to be defined as per function um it's just that was the experience I had at my previous organization so the example here I'm going to use AP clumsy Magpie who are a Northeast based threat actor uh we know that their ttps are brute for fishing remote system Discovery and data encryption um so this is the sort of

information we're going to give the CTI guys uh we know that they look for publicly exposed services such as RDP and SSH um and they have got po shell and bash script the reason I choose this sort of attack method for the first one um there's a lot of Open Source bash and Powershell based ransomware that has built in C2 servers um so the barrier to entry and getting this sort of environment spun up is quite low um but it also gives that real slightly more real to life threat Prof file that you can start to work on so from here you've got your hypothesis to then build um it could be that your CTI analysts have started

building this profile they've started mapping the ttps to the attack framework miter they've looked at our attack surface they've worked out potentially where the attacker is going to try and get us um and then you hand it out um again this area is where you can get these Junior guys writing queries searching for things and just really getting into it with again no real time constraints or it's all about that enrichment even just understanding the query and language that you're using and how to pull specific hashes things like that from the files that normally they might have the slightly more senior guys query and writing the queries and doing these sort of [Music] tasks and then you move into your

incident response side of things again this could be one team it could be a dedicated function um and it's it's given that slightly more real tabletop exercise so you're not talking about you're actually doing it you're going in you're analyzing the network connections that are coming out on the host you're looking at what files have been encrypted for instance where it's sending its data and things like that and this all then feeds back into the CTI process and it's that nice full cycle of the systems but again if you uh sat there at 3:00 in the morning and you get a P1 incident and you've lost half your estate to ransomware it's not going to be the junior guy that's going to be

there jumping on and having to dig into it this is that it's giving them the chance again whilst you're going through this you'll have more senior people driving them through giving them the guidance and just keeping it going but when you sat there in the passenger seat you're not learning so when it's that sort of it's that slightly more real to life experience and people will upskill a lot quicker and it does it's just getting that muscle memory and they'll start to look for things that potentially they wouldn't have noticed before because they'd not seen it and then you go through you your after action review again if you're running through like your playbooks that

you've done your tabletop exercises for three years you've run the ransomware one two or three times and yeah you might realize that there's certain processes that just don't make sense there might be things that would make sense now you've got a different tooling set in place and it could just be that well we should do things a little bit differently or we'll skip this step we'll move to this step and it's that real life you've ran through it you've tested it and you know it works or you know that there's deficits it could just be the fact the the process and the order well let's break this out into sort of different sections and Silo a

little bit more or move it into different bits how did the g games go so you're getting that review you're getting that feedback if the sheima sat there going yeah I learned a lot in this past week two three days then you can look to plan to move on to the next one and make it slightly more challenging slightly more difficult and then the biggest one is was there enough support because you essentially you are J just going to drop a load of people who may not have the experience in the exposure into the deep end and it's either swim or drown type thing um but there you've got to give them that gentle encouragement and

support and if they do need anything or they need that pointer guide them in the direction and help them grow themselves and sort of continue on that journey and it does work um again previous organization uh we had a team of eight or nine security analysts that had no prior experience in Tech cyber security anything like that um they' done a couple of online courses uh and then the organization brought them in and within two or three months they were starting to get their feet wet and they were getting quite confident but after six months they were more than happy to jump into incidents the SE thre intelligence guys were building up profiles on AP groups and looking for

attribution um our threat hunting team were tearing into different web tokens and other quite sort of sophisticated um attack methods that were prevalent at the time and within our environment and then the interdent response guys are running these tests they're pulling host data they're looking at reverse engineering malware samples and they're getting into that process of how does it work again a lot of organizations they may have just been you know churn tickets churn tickets pass it up and this is that letting them get in that they're too stuck into it again there's always that safety barrier of a senior member of the team to jump in if there is an issue but people will grow in an

exponential way and once you've got that drive and that sort of hunger for it they'll continue to grow any [Music]

[Music]

questions excellent excellent right questions Who's Got Hand up we'll start at the bottom it's EAS hi great talk by the way um I really agree I do really agree with you on um creating tabletop exercises and lab environment but um as a business leader an experience that we've had is we'll pay thousands of pounds a year and we still do for training and lab environments pay extra for lab environments and they don't get used and you'll have people um some sometimes in the tech team that'll just totally not be engaged with the labs at all they're interested in theory it's a massive problem I thought it was just do um you know these people are good at the jobs

they're not they're not bad um and I spoke to other people in the same situation as me and they're like yeah what is that all about you go to these conferences and they'll say we want lab environments you want practical training you provide it and it's just totally disengaged with so I'm just wondering what your opinion is on that and whether you think it should be worked into kpis or if it's forced is it is it then more disinteresting so for me it was always the um if you've got the right people so again we've got um like my team was built up of a lot of EX veterans and things that are second third maybe even

fourth craes at this point and they've got that hunger they've got that drive and they've got that passion to really grow into an industry that they find fascinating and I think a lot of it is if you've got the again if youve got the right people and they've got that sort of drive and passion they'll they'll be chomping your arm off for these Labs they'll be wanting more they'll be wanting to do different things I mean we used to do digital forensic disc image hunts and I would come up and create different random it was like stealing cupcake recipes and these bad guys have got the world secret cupcake recipe please analyze the disc image and pull

it and they be sat there within the next two to three days they' be like I need another one can you make they take a lot of time to make so it's kind of you're turnning through them but once you've got that passion that drive I think again I've seen the online learning resources in the labs that haven't been used by a lot of companies um and it depends on if you're using it is your yearly compliance training and you're having to drive it that way um but it also depends on the content if the content is good and engaging I do find that people will sit there and spend the time but it's also that if the time's

made out of their own personal time they might not want to spend an extra two hours in a lab a day yeah well it's the right place [Music] then hi um so I and an online education platform for people interested in getting into cyber um and I was wondering if this kind of approach could be useful in a kind of virtualized environment for people where it's not actually a company that's working on it but could be a completely F fictitious scenario completely virtualized potentially run over something like a zoom call yeah it wouldn't be anything dissimilar to like blue team Labs but on a sort of scale where you're multiple people working on the same thing and

mentoring and I think the biggest thing is the mentorship because if you were to give people these scenarios and situations and just drop them in it they do need the guidance and the sort of here's where to look but it's also when to not give them that bit if you can give them that nugget of information and they have to then discover it themselves they're going to discover it a lot quicker or in their own way because again everyone learns in a different manner um it could be that someone really likes getting presentations and speaking and listening um it could be that they are a Hands-On doer and when you're in that process of doing things

people learn and it's it's having the it's every individual is going to be a different use case and it's kind of knowing who your audience are to some extent thank you another question over here I think you raised a a really important point there about the mentorship and kind of the the instructor or kind of the the guidance of the class or the the trainees what are the tips on picking the mentors or developing the mentors I guess in my experience a lot of the really good practitioners almost do this out of an innate or natural ability and and therefore struggle to then pass that process or that skill onto someone else that they you they know what they do but

they don't know how they do so that makes it hard to teach again I think it's all on um it's difficult because I know what you mean there are some incredibly talented technical people but it's kind of they've been doing it that long it's just oh how would you not know this and I think there's that um again work out how the the team likes to learn and it could even just be a case of shadowing okay so I'm going to spend the next two hours doing this do you want to just sit on ask questions as I go and you've got that live process which might be a little bit easier for someone who

doesn't necessarily understand what they're doing is a lot more complicated than they make it look um again I've seen that where you'll sit on a call for three or four hours as you're just doing the generic stuff and people are like why are you doing that why are you doing this what's what's the reason behind this and as you go they're like oh okay I wouldn't thought of that and it's just them little bits of when they're watching it in real time they can go that's how that makes sense and how it's pieced together excellent any more questions for Mike oh right oh you're closer just listening to the previous questions I think maybe it's uh if the

person's on site with you it makes it a lot easier to actually ask the question and the contact is probably the key thing because it breaks down barriers whereas online and in a virtual environment it's hard to pick up the phone or like just say can you look at this for me so I'm not faed to say this but I actually got into this argument on Twitter not too long ago um and it was on about how someone was complaining about the remote world and they can't mentor and they can't their staff is sort of stifled their employees are just not growing and I get a load of rubbish I think fair enough turning around in

the office CH like oh how do I do this it's a little bit easier again it's all about knowing the people that you're targeting so if you've got someone who will sit and read a technical document and they'll sit there and just Eng gross themselves in it and pick it up that's great for them it might be that someone else needs to S on a call with you and have it talked through they take that presentation and I think it is there's no one size fits all and I think there is limitations in doing things remotely in terms of that initial gratification and that initial feedback but if you prepare and plan things in advance you

can sit there and go where are you weakest let's jump on a couple of calls this week we we'll sit down we'll go through it and it's just about that continually growing and trying to nurture the talent but again everyone's different so it could be that someone doesn't want two or three calls a week they just want to get a task to get stuck into and then just they'll ask questions as they go and I think it's that F finding that balance between where it becomes cumbersome to try and customize a sort of training plan for everyone in your team to making it a sort of broaden accessible to everyone but I think just being there like if you're always at the

sort of the end of the team slack what a messaging chat you're using and they can get that initial oh I need this oh well here you go here's where to look or here's who you should speak to and it's just continually sort of communication with them will help yeah and sometimes it's a good idea to pull people that are remote into one place and do one of these things but similarly if you are going to have an incident at 2: in the morning and people normally work all over the country then they should practice being able to do it remotely because they can't be in the same room they're not going to be thank you I'm an apprentice so I'm very

much on the receiving end of all this training so whenever my company decid to trial a new product we always go through a CTF with it to just see how it feels how does ctfs differ from doing war games and such when it comes to this sort of thing it's a similar sort of process um I think the war gameing more for the it's that real real life simulation I know capture the flags can be somewhat real but at the same you're kind of playing a game um CU you're hunting for the flag in a lot of respects and with the war game it's trying to again start off in a sort of spoonfed very controlled very obvious

manner but as you start to mature the process it'll be you've thrown a couple of Rogue servers up in your production environment that have this magic binary that's running and doing things that's again tightly controlled and I think it's just giv that it's trying to simulate it's trying to simulate what the tabletop exercise would do just in a more practical and Technical manner than talking it through and sort of um verbalizing it it's actually doing it excellent thanks very much awesome okay can we get a massive Round of Applause for Mike thank you