How to Develop Cyber Personnel by Alessandro Lovadina & Ryan Irving

Morning

everyone. Welcome to Besides Tampa 2025. Your first you're in the leadership track here. Your uh first presentation is on how to develop cyber personnel and it's going to be uh given by Aleandro Levodina and Ryan Irving. Aleandro is a thirdyear computer science student at USF. He's originally from Italy and his main interests are cyber security and AI. He is a uh he works at the at Cyber Florida as part of SOAP, the security operations center advanced program um and was recently promoted there. Additionally, he uh he is interested in rockets that go multiple thousands of feet in the air. Ryan is the sock manager at Cyber Florida and he's been in information cyber security for over a decade in various roles at

Hillsboro County and at JP Morgan Chase. He uh his specialty is digital forensics and um he holds a master's degree in digital forensics from UCF and various other certifications. Please welcome our speakers.

All right.

Yeah. Well, as Michael said, I'm a undergrad student in computer science. I'm a thirdyear student. I've been recently promoted as a senior security analyst. Uh so I want to thank Ryan for that promotion and for that raise. Um I'm also hold a position as a executive officer in a senior organization. It's called society of an accident rocketry and that's where I make rockets. So um that's where I code rockets and make software for them to just deploy stuff do stuff in the air like um like NASA does. And I guess on Ryan's perspective I'm the Italian James Bond. I I don't I'm not sure why. I think it's the glasses. I changed them. So

All

right. So,

All right.

Check. Check. There we go. All right. No problem. I'll talk away from it. Apologize for that. So, yeah, we want to uh increase the uh engagement, right? We want to get our professionals engaged in practicing how they fight. All right. Um this will improve their knowledge retention. uh we don't want to go into an intrusion or some sort of incident and be like what do I do right so part of this is getting that muscle memory uh working for them so that way when these actions occur we already kind of know what to do and if we already know what to do then we can adapt adequately all right uh enhance practical skills we want to give the

tools and abilities and methods to our students as well as to your potential employees uh to uh essentially work with the tools they have or learn some uh methods with similar tools to execute these certain functions and we want to of course be uh cost effective uh of of what we're working on. We want to consider the cost of those sorts of things and we'll talk about that a little bit and we want to obviously overall improve our security posture. So if we're able to do cyber security better in a training environment and so when we get to the real thing uh we'll be able to uh improve improve our organization company's whatever security posture and be more prepared. Next

slide. Who here has heard of backwards and breaches? About half the room. About half the room. Put the hands up in the air. Um I was thinking maybe a little bit more. This is a great coste effective tool uh for you to do some basic technical tabletop exercises. And so we're going to talk about that because we leverage that in our sock relatively heavily. Uh next next slide. So what do we do? It's engaging interactive. It encompasses teamwork. And to give you uh an idea in a nutshell how we set it up, we have a team of analysts. We typically do teams of four or five. And if we have more than that on a day, we may break it up

into two teams into two different sessions. I'll designate one student as the incident lead. So they're the person responsible for making decisions because at the end of the day, someone has to make a decision, right? We can't just all just kind of pow-wow. We should do this, that, or the other thing, and no decisions are actually made. And then um they work together with the procedure cards and things like that. We're going to talk about how that works together, but at a high level, this will help in their decision-making process and give them the critical thinking skills and will help develop what I've kind of coined and like to say is conversational cyber security. My goal for all of my

students to walk out of the SOCAP program is to have this conver conversational cyber security. And what that means is they may not know all the buttons to push. They may not know every single aspect of how something works, but they should be aware enough of what's going on to where they can either push the discussion forward with their own thoughts or ask relevant questions so they can have a better understanding and push the discussion forward. All right, so that's the ultimate goal of doing these back doors and breaches exercise. It gives them better contextual awareness, helps them uh have a better understanding of cyber concepts and will again will give them better o better overall understanding. Next

slide. So how to play in a nutshell? Um, and if you've played this before and it's repeat, I apologize. Uh, but I think it's important to do. Okay. Essentially, you have an incident master. Typically, that's me. I'm the moderator. I typically will say something. All right. It's a, you know, it's a sunny shiny day at the cyber floor sock and we get an alert from this XYZ domain with a potential binary and we're unsure about it. What do we do? Right? So, I'll kind of start an incident like that. They will have um uh procedure cards uh that that they get to pick certain access from or certain actions from like reviewing firewall logs, reviewing

memory, doing endpoint security. Uh perhaps they can isolate host uh do things like that. But essentially my incident is created from these four cards up here. We have a initial compromise card. We have a lateral movement and escalation card. And we have a persistence card as well as a data xfill card. And I kind of chain together this scenario usually off the top of my head um as far as what's happening and kind of lead them toward these answers so they can pick the correct procedure cards which tell you which tells me um you know which which do I flip if they pick the correct one. Uh next slide, Aleandro. All right. So in a rundown they

basically have 10 turns to figure out the incident. This is essentially how the board looks and you're looking at an online version. You can do this for free today. All right. This is the online back doors and breaches version. We do it over teams. We do it in person. We do it with the physical cards. So they will essentially say, "Okay, we have our four established procedures. That just means they get a plus three modifier." You're like, "Plus three modifier on what?" Well, this plays like a light Dungeons and Dragons style game. There's a d20 involved. And D20s are fun. And I don't care what anyone says. That stuff is fun. All right. So, they

get to roll a d20. And if they get anything that's 11 or higher, then that's considered a successful roll. So on the established procedures, if they roll like an eight, well plus three is 11. That's considered successful. Anything less than 11 is considered a failed roll. So what we'll do in either case, if it's a successful role and like let's say they picked firewall logs, I might say, "Aleandro, you were successful with your firewall log review. Why was that a good pick?" Well, probably because if I was reviewing the firewall logs, I knew what I was doing. So like what would you find in the firewall logs? But I will find like all the indication like I guess DNS

um any post get requests eventually. Yeah. And now if I said all right Alandre you did not roll an 11 you rolled uh a one or a two and you failed the role. Why did you fail your firewall log review? Um that's a good question. So something could be a lack of training. That's something that uh it happens most of the time like somebody cannot like it doesn't know how to use the power or like it doesn't know what to look for so they can miss some logs or maybe the baseline was incorrect so it wasn't actually picking up the right alerts or like right so we talk about the different things that are successful

different things that would fail some of my favorite failures are all right someone forgot to check the logging box on the firewall config oops right or and I've had this happen to I mean, not with a firewall, but with another network device. Um, the last person who knew this password died. Um, and it's like, how do we get into that now, right? Um, so, oh, we had a trial firewall or we had a trial tool, right? So, you kind of see like I usually use the trial excuse for like endpoint security or whatever, but you get the idea. We talk about these real world issues that can cause problems as well as a real world

successes on why you would want to do it. So, we have basically injects anytime you roll a natural one or natural 20 that will add randomness to the game. It could be positive, but most likely it's negative. You'll have some sort of negative action. Like one of the ones they seem to get regularly is, "Hey, your incident responder had a baby and they're on, you know, FMLA now." So, I'm like, "Okay, you instant responder, go leave the room. You're done with the game." You know what I mean? The last one I had, which I was playing a newer deck, um there was a tree that fell on a power source for the building and the

power got knocked out and they were basically driving to an alternate site and they were out of the game for three turns. I'm like, "Huh, that was new and fun." Um, not seen that one before. Uh, then you have a cool down. So, so they can't just keep using the same card over and over regardless of a successful or failure roll. Uh, they have a three turn cool down where they cannot use that card for three turns. All right. Um, next slide. So, one thing I do want to point out is it's an excellent tool for this uh, analyst to really start talking about getting a a conceptual understanding of why firewall logs work. Why do we threat

hunt? Why do we care about memory forensics? Why do we care about endpoint security? How do we do that? What tools do we use? These are all things that we talk about. Additionally, um I also use this as an interview tool. So when I interview uh candidates, I have two round interviews for student candidates. I have a brief phone interview, 15 20 minutes. Do we like the candidate? Do are they good? Are they using chat GPT to give me answers? Which has been happening often now in case you're wondering if you're if you're a hiring manager. Um, and then if they move on to the second round, the back doors and breaches is their second round

interview. I will sit them down and I love it. Like Alisandre came in a full suit and everything tie and I'm like, "All right, we're going to have this interview. It's very laid-back. I have no idea what questions I'm going to ask you, right?" Um, at the time I always kind of did the random incident on my own. Now I kind of pre-make my incident and I make the incidents the same for all the candidates just so I don't have to keep making new scenarios myself. Um, but it allows it allows me and I had my HR sitting in with me and we were seeing how it went and my HR told me, you know,

Ryan, I really like this interview because we can ask question or we can see the answers to questions like tell me about a time when you didn't know something and how did you find the answer to it without asking that exact question because we're asking these questions and they really don't know. Oh, also I bring in support analysts. So I bring in students who are already hired and they act as support. So I and I say the this is your team candidate. You're the incident lead. Use these use these guys or or gals how you see fit. Ask them questions and bring them in. I get to see how fast they ask questions. I get to see how fast they make

decisions. I get to see if they use their team or not. And most importantly there's actually one uh uh student candidate that I had very smart knew his cyber security stuff up and down did not take feedback from team or even myself. Do you think he got hired? No. So, I got to see all those behavior things. The truth of the matter is I really don't care how how much they know about cyber security or how technically apt they are. I really don't care because they're students, number one, but number two, I really care about that personality skill. What personality traits do they have? How well do they work as a team? Are they going to work with our culture

and things like that? Alandro, any comments about your experience in back doors and breaches? Yeah. Well, when I first went in to for my interview, I was like I was super prepared for like answering all the possible cyber security questions, especially since I didn't know what fishing was for my first interview. So after that, I was like, "Okay, I got this." Went with a footuit. But I saw Ryan just sitting there with these cars that I've never seen before. I was like, "Okay, I never seen this in my experience." Um, but overall it was like great because I actually thought about it and I was like since I don't know much about cyber security at least I can show my

like problem solving skills because I know like I have them I have these skills and it was like thanks to Ryan thanks to this method I was able to like show how I could like problem solve and how I could work with other security analysts that were working there at the time and how fit in in the team and also from the perspective of like helping other people in their interview, I was able to see like okay, so how do they other people approach uh the same scenario that I also went through or how um what do they do different and uh um are they more experienced than me and what could have I been been doing

different? So stuff like that that it's like uh it makes you reflect. So I think that's really important. Oh, it make me feel like I'm doing an okay job at work. Thank you. I appreciate that, Allesandro. All right. Um, so let's move on from back doors and breaches. Uh, so instant response, hello, my name is. So I really like this meme because this meme was my day when I worked at the bank. I walked into uh the bank I was working at and I was like, "Oh man, like it's so early. I just need my coffee. I haven't even eaten breakfast yet." And they're like, "Ryan, you're now incident lead for this incident that just

occurred like right now." I'm like, "What?" Like, "Yeah, talk about existential crisis. Like, we want answers and we want them now." And I'm like, "I don't even know what host names or IPs are are compromised. you haven't given me any information yet. You know what I mean? But as a lot of you probably know, that's the reality of when you're walking into some sort of incident or intrusion in your work environment. There's a lot of hysteria going on. People are not sure what's going on. It's like, you know, information and news is kind of fluid. You don't know what's necessarily fact. And you got to get all that stuff down. So, not questions we're going to answer

right now, but questions to think about. What if you had someone new on your team and this was their first time experiencing this, right? What would the outcome be for a brand new person performing a real incident? How would you think that would go? And what can we do to hone and train their skills to be better prepared? So that's going to be kind of the focus of these next set of slides. Uh next slide, Alessandro. All right. So for us, it's welcome to the ARCs range. All right. Now, quick tidbit. Uh cyber Florida arcs range. It stands for the align realistic cyber attack simulation. And it's a cyber range that we have and we offer free

service to public sector entities. So if you are an SLT um you can get access to this. You can either take the CAN training or do kind of the live open world sort of environment where they'll simulate a threat actor attack and you can go and investigate it yourselves. That's what we do in the SOCAP program monthly. We have a monthly exercise of this. Um and but if you're not public sector um and you're private sector like I do have you know some recommendations we can talk on the side about but there's also free and open source solutions too. Next slide. So I call this and my assistant manager call this we call this the hyperbolic

time chamber. Don't be embarrassed. How many Dragon Ball Z fans do we have in the house? Yes. Yes. Only about 20% of us. That's okay. We're the cool ones here. So real quick, who knows what a hyperbolic time chamber is? What's the hyperbolic time chamber, sir? Come here. Explain the hyperbolic time chamber. So, basically, it's an area where time is very slow. So, you get to train. Uh, that's essentially what it was for. So, these guys would do training to be able to fight the next big guy essentially that's there. So, basically, yeah, they would spend one year in the hyperbolic time chamber and it was only one day on the outside. So me and my assistant manager just kind

of came up one day like, "Yeah, this is kind of like the hyperbolic time chamber." Like, it is kind of like the hyperdolic time chamber because we're accelerating the training of these students. And I I don't not going to name names, but we've been told that our students are outperforming other public sector incident response teams and that's based on the actual attacks and exercises in the same environment. Not all, but most, right? So that's a huge kudos one to the cyber range and two to what these students are doing in this program. Uh, next slide. So what do they do? So we do have our own dedicated range environment. It's a small kind of business network. There's

about 30 or so assets on there. Um the cyber range folks will fire off a threat uh a threat actor simulated attack that mimics, you know, Lazarus or an oil rig or some ransomware event, whatever, right? They have a whole catalog of attacks. They'll fire that attack off against the network. We'll get then get prompted with some sort of alert. We do have our own CrowdStrike licenses installed into the cyber range. So we're actually leveraging CrowdStrike as a commercial EDR tool uh in there, but they have a bunch of free tools in there such as Splunk. Sysmon's already installed as well as security onion. So at a very minimum, we have EDR logs, we have proxy logs, we have network IDs

logs, and we have Windows logs to include basic and sysmon Windows logs all to gather from for investigating this um incident. Um so we will utilize crowd strike for targeted evidence acquisition. Who's familiar with real-time responder in the room? Only a few hands. So real-time responder, we will connect to the machine. We will gather targeted evidence such as memory. We'll uh well I I explain that more on the next slide. We'll get into that. Um but then yeah, we use log analysis predominantly with Splunk and as well as the crowd strike logs and then we'll do timeline generation which we'll also get into next slide. All right. So for target evidence acquisition like I said we use

uh CrowdStrike RTR we'll use XME dump to dump the memory. If there is malware running on the system we'll use the mem dump uh function within crowdstrike to dump the process individually. And then we will deploy cape with a um uh that we have hosted in the crowdstrike cloud. And then we have a customized PowerShell script that we deployed to the compromised assets do run cape and then we export all that up to the crowdstrike cloud. And then we pull that back down to our host systems. And then we have forensic tools that we can process uh and do we'll do post forensics on right we have magnet axium bulk as well as blexity which is commercialized

volatility if you're not familiar with that. All right. Um and then if we do any malware identification we'll like I said we'll extract that and we can also do some basic malware analysis too. Uh next slide log analysis. Um so like I said we'll use the crowdstrike console. We'll use the various timeline uh logs from there such as the uh process and the host timeline. Um as well as use some of the nice gooey functionality. Really crowdstrike I call it a cheat code sometimes uh because it really tells you that telemetry in real time. Um and it's really nice. Uh and then the students do get a lot of experience in Splunk which Allesandre will be covering some of that

here in a minute. And then uh we leverage security onion um as well in that. But then we go to Excel and if if you're familiar with forensics at all in any capacity, Excel is the best forensic tool. And we're going to talk about why here. Um next slide. Um Alicandre, did you have anything you want to say about the other tools and things that we've used so far from your experience? Yeah. Um well what I can say from my experience uh like academic experience I I can say that we uh in school we don't really get to learn all these tools uh that we use uh as uh security analysts in these instant response exercises especially like using

crowd strike. Um I haven't never used it in any of my classes and I know it's like a really good tool to learn how to use um especially in school and before actually getting like a real job. Uh but other things um for example Splunk um I knew some queries before but I didn't know how to just start from zero just get through a whole incident by myself create my own queries and just develop my thought process and and going to also the basic tools like Excel like everyone uh thinks they know Excel until they actually create like for example a timeline and you like for example I didn't know how to order it or by like a

specific column I had to ask Ryan for help and every time every incident response exercise we do I'm like oh Ryan can you help me in ordering this time link Alexandre when we're in an incident doing a simulated instant response what is one thing that I always harp on and complain about to everybody about the Excel timeline as uh the timestamps are like that's right I if you get the timestamps out of the format I get mad I'm like stop it that will mess up the entire Yes. Question.

Nope. We're just doing it. We're just doing it. Now, if you do want to talk about AI AI on the side, Aleandro actually did have an AI project where we made we we imported our own cyber security LLM and we created our own offline chatbot for cyber security. um and he made an interface so the students could actually connect to it and ask it questions uh so you don't have to be at the actual place where it's being hosted. So that's a side conversation but he he's actually worked on that too. So right now not not really using AI for the timeline generation or anything like that. Doing it manually which learning how to do things manually

is the good way because then when you do get to automation then you could be like hey that doesn't look right. Right. So timeline generation. So what we do is we take CrowdStrike, we take uh the stuff from our proxy logs in Splunk, our uh network ids logs in Splunk, our Windows logs in Splunk, and we take all that evidence, we find the key artifacts, right? And then we put them in this timeline, and we aggregate them in a chronological order because then we're able to see the story of what is happening, right? We're able to see, okay, we see this user, they clicked on this link, we see a get request to this uh domain that's getting this weird

binary. we see the binary execute. We can see execution in crowdstrike or pre-fetch files or some other source. And then we see, hey, just moments later, a connection to another weird domain. And now we see uh you know regular connections every 15 or 30 seconds to that domain. And oh, there's a post in there every once in a while. These mean bad things are likely happening. All right. So, um Aleandro, what do you think about timelines from uh your perspective? I think it's pretty like the first time I saw them it was a little confusing because it was like 300 logs everything together but after you look into it a little bit you can kind

of see okay you can see the initial compromise you see okay that's how they got into the system you see all these other proxy connections everything that was going on during the incident and so you can kind of get a full story of what h actually happened uh throughout this full attack so I think it's a very useful tool to use uh especially when you're working with like a team and everything is pulling data from different indexes on Splunk for example. Um I think it's like a great summary to make everyone understand what's going on. And one important thing to keep note of this is not an individual effort like we're usually in teams of five to seven

when we do these at minimum. So this is a multi-team effort where we're delegating you know usually a student is labeled the incident lead unless I have like all new students and then me or my assistant manager will be the lead. But we try to make a student the lead. Uh so that way they can get used to being responsible for decision-m, assigning tasks, figuring out workflows, and they'll tell me what to do, right? And then I'll ask them questions that I might know the answers to, but I want them to get there, right? Um so this is definitely a team effort. Next slide. So why do we do it? And I was already kind of getting to that. It's to give

the real hands-on learning. They they leave the hyperbolic time chamber much better than they were uh before they did the incident. And by the way, we do these in 4hour blocks once a month. Uh, usually from like 10 to two or something like that, right? So, they come out with a lot more knowledge in an accelerated format that they wouldn't get anywhere else, I would dare say, because I don't really know of many places that is doing this sort of activity. Um, they get practical skills needed by organizations like you guys, right? They're getting experience with CrowdStrike. They're getting experience with Splunk. They're getting experience with methodology and approach. And that's really the most important thing, methodology and

approach. If you learn a methodology and approach, you can take that to any tool. You might, you know, have Splunk one day, then your organization says, "Hey, Microsoft's great in all the things." So, we're dumping it for Sentinel, right? Because it saves. Oh, did I hit some nerves? Oh, okay. Um, yeah. So, but if you understand the methodology, you at least know what you're looking for. Now, you just got to figure out the Azure Sentinel um uh way of finding things as opposed to the way you found things in Splunk, for example. All right. Um, and it better ties to their academic knowledge to actual practice. I get in and talking to my students are

like I understand these concepts better now because of what we did and that's inc and that's important and ultimately this is for development of the workforce right um do you want someone who's just a graduate with a bachelor's degree with no work experience has never really seen anything that looks real before you're going to have to teach them everything right it's hard enough when you join a new organization that you get used to the organization's processes in of themselves all the administrative stuff how do you get permissions how do I do time off how do I do x y and z how do I log access to your system? Right? That's similar but varies anywhere you go. But

if they can come in with a methodology and approach to understanding how things work technically in the environment, they're going to be much better off and they're going to be much more accelerated and ready. And most importantly, this is a safe place to learn mistakes. Next slide. So, let's talk about mistakes. I like to I think about one day writing this book called Oopsent Response because I've seen a lot of oopsidants in my life. So, we're going to talk about one of these oops right now. And we almost have the script here. Hey, Alessandro, where are the memory images? Um, well, right. I saved everything on the folder on Teams, pretty sure. Excellent. Did you save the cape report anywhere?

Try checking your bin, aka recycle bin. Oh, yeah. Okay. Um, that's a good idea. I'll get back to you on that. But honestly, the the beam was empty. And I already knew that. But I was just trying to save up some time and thinking of what I should have said to Ryan. So he was concerned because he didn't have it. He was like, "I'll just tell him that even though I know I don't have it, divide some time." I love it. All right. Uh meanwhile, still not knowing anything. Hey, Allesandro. Any uh updates on where those images are? Yeah, well, I found one file that says cape on it. Maybe that's the one. I'll send it

to you now. But it's funny. It wasn't and I knew that because it was from like a past instant response exercise. But raise your hands if any of you can relate to something like this, right? Um I when I worked at the bank, if this were to happen, I was not having a good day, right? So, if I was tasked with collecting targeted evidence off machines that were potentially compromised by a nation state actor and now I just don't know where that evidence is because I didn't collect it or to copy it the right way. What do you think would happen? That's a resume generating event. I would say yes. Yes. Yes. So, Alessandro, is it good that we learned this lesson

in this environment? What do we learn from this Alessandro? But we learned that even easy tasks like this should not be like underestimated and especially when stuff is like temporary cuz this like crowd strike for example um has like a seven days period of like after seven days it expires. So, we were like trying to make like a a tutorial for other sock analysts on how to extract memory and he was actually making a recording and everything when he asked me all these questions and I was trying to make up for it somehow. But it didn't really work out but like it was a safe environment so nothing actually happened after because it was just a exercise. Yes, the good thing

about it. Allesandre was not fired. In fact, he was eventually promoted. And in the CrowdStrike cloud, we I don't know if there's a different taring, but we only have seven days of retention. So, after seven days, that evidence is just gone from their cloud. You know what I mean? So, good place to uh to to learn this. And I love that that Goku meme. I didn't save the samples, check an empty bin. That's just ridiculous. All right. Um, next slide. All right. Oops. In number two, Aleandro, why don't you uh talk about this one a little bit? Yeah. So this other opinion is um something related to like um the time zone of the logs. So in our instant response

exercise we use different tools. We use splank we use crowd strike and all of them have different time zones that you can set depending on like your user preferences and before the incident we actually all agree on using a specific time zone usually uh UTC and that's what Splunk has. However, we found out that Crowd Strike resets uh their time zone to Eastern time. So, every time you open a new tab, you have to put it back to UDC or else you're going to see everything shifted by five hours. And what happened in one of our instant response exercises that um I was looking into crowd strike. I didn't set it right time zone and I just extracted all the

logs um from like I think it was the Windows logs and I just sent it over to my coworker. Just telling him, "Oh, you should review this. Uh it probably has some interesting stuff." And they've been looking at that stuff for like half more than half an hour and they didn't find anything. I was like, "Why? What? What's going on?" And then I found out on this, as you can see in this little image, that it was set to eastern. And so I set it back to UDC. And then that's where the actual logs uh came out. So this is something that um I learned from uh these exercises and it's something I I will

not uh repeat especially when it's like um involving other people and wasting other people's time. Yeah. So like on that point like he was sweating bullets in both of these. Um and so I just wanted to say like imagine again if this was like your real job like you were working for where you're because like you're interning at Bank of America, right? I I don't know exactly what you'll be doing at Bank of America, but let's say you're doing this sort of work for them. Like if you were to do that, like that would be a much more I think heart palpitating situation than doing it in the SOCAP program where we're doing a simulated non-real attack,

right? So it's good that we get Oh man, I really need to make sure that UTC is check because CrowdStrike likes to default to that to Eastern for us, right? Um go ahead and next slide real quick because I know our next slide's about this too. Now this is a little bit of an eyesore but Aleandro go and point out the differences here that you identified. Yeah. Um well I just want you to focus on like the daytime uh column. So you can see that uh some of the events at first on the first piece of the timeline are like happened on the 20 January 23rd uh 2025 at 7:57 p.m. Um and these logs are all related to the

second piece of the timeline. But you can see that like right after a minute like it goes from 57 to 58 uh which is like um it should be the same time but it's 5 hours behind because it's at two. So this actually happened in exercises and it can also happen in real life. Yeah. So funny enough this exact situation happened to me while I was at Cyber Florida helping out Let me make sure they're not here real quick. Yeah. Okay. I don't see them. Um, we're helping out another SLT with an investigation and they actually sent me CrowdStrike logs as well as some other logs and I'm looking at the investigation. I'm like, something's not

right here. What's going on, guys? And eventually I figured out um, you sent me these logs in UTC and this evidence was that we also collected and did was in UTC. However, your cross track logs are in Eastern time and so it's not lining up. Not only is it lining up, it misses the complete window of investigation that you actually care about. And not only that, as you were probablyware aware, unless you have a lot of money, your default crowd strike telemetry, I think, is about 14 days. And now we are outside the 14 days. Ouchies. Right. We just had some major oops ouchies with that. And that was uh that was a real investigation that we

were helping an SLT with uh SLT out. I'm like, well, sorry. I'll put as a finding to keep time keep same time zone because that's the finding here. Um, but this is good for them to do because you know like typically like where I work in the public sector like we deal in Eastern time predominantly, right? Why do I use UTC? Well, I have a lot of formal forensic training and if you've had any formal forensic training, UTC is like the only real time zone. And if you ever work in a cross uh time zone company or global company, it is a mess trying to keep track of APAC and then but but we need this print server and this even

though this print server is in North America, it was being used by Apac and then oh by the way those time stamps are just off because and it's like trying to re reign in time stamps is awful like especially when you're dealing with multiple time zones. So we just keep to UTC because I think that is a good fair practice and most of these students are going to be going to places that are likely crossing time zones. So that's why we uh we we do this practice here. Um but yeah, that's our second oopsin uh that we learn from. Next slide. All right. Um Alandre, I think you're going to take the learning outcomes, right?

Yeah. Um so some of the learning outcomes that I've seen from like these two obsidants uh is for example like double triple check that we save all the data somewhere safe and it's actually there. Um something else is actually like taking a break and check that everything looks fine because after some time that you're like there for like an hour even just an hour but just looking at those 200 logs all together trying to make sense of them you you just need to take a break and then just come back and make sure that everything is good. Um, another thing of course as I said previously is to not underestimate a task that seems simple and also make

sure that everyone is using the same time zone. Fair and so I'll talk about safe environment and I know we're wrapping up here so I'm going to go through this a little fast. Uh, it's a great place where it's acceptable to make any kind of mistakes, right? So because we're making these mistakes in simulation and we felt this little bit of pain now in a real situation, a real intrusion, hopefully that same mistake will not be made. And if it is, there will be much more pain involved. But I think we all can agree to that. Um, and practicing the same tasks on realistic settings, right? We want to get that muscle, that brain muscle primed because

when we get into an incident, we don't want to constantly have to think about everything that we're doing. We want to be able to, okay, I know I need to go get this. I need to get these logs. These are the tools I need to do to get this. And practicing that in a simulated environment is just going to help that for when you come to the real thing. All right? And then, you know, nothing will replace real life experience. I will say that nothing will replace a real scenario. Nothing will replace a real intrusion. But how can we get to the next best thing? And that is where uh the simulated instant response in my

opinion will get you to more experienced security analysts to where they'll be more prepared and more ready for that real life action, that real life intrusion. Next slide. Go ahead Allesandre real quick. Um yeah, so why is this important? So uh we get uh from the point of view of a student we get to learn like tools like splank and crowd strike and but the most important thing is that we actually learn and develop our own thought process which is very important because uh from the academic perspective you don't really develop anything like that unless you get the actual experience and for example I have some images here or like a template that um Ryan created for

us uh of some specific queries for Splunk. Uh and then as you can see I made my own query after uh tweaking some stuff and like after some having experience with some instant response exercises and all these uh I'm sure I'm going to bring it to my jobs uh that I will have in the future. Uh so it's very useful and these students add to our template right? So we have like a running word doca like spunk queries and now KQL queries because of Azure Sentinel and the students add to them like it's not just me who's doing this. They add to them too. So I get to learn from them as well. Um next slide. All right. So I'll

take this. So creating a shared timeline. This is a team sport. I can't emphasize that enough. So we have our shared timeline and we have a timeline slide for you that's kind of gross but we'll do our best to get through it. Uh but ultimately we do like I said assign one of the students as the incident lead. uh one of our analysts who is actually in the uh crowd, Tim Kercher. I saw him. Raise your hand, Tim. He actually led I think arguably one of the best instant responses that we had. So, if you're looking to hire somebody, he might be someone you want to talk to. Um but he had like in our teams, he had

breakout rooms for proxy, breakout rooms for EDR. He and like he led that pretty phenomenally. Um, but anyways, the student will be the lead, assign out the work tasks, um, follow up, make sure we're doing well, and then we take breaks every so often to, okay, let's let's stop, let's take a break, let's review the timeline, let's see where we're at, and then what do we need to tackle based on what the timeline says. So, we're letting that timeline of evidence dictate to where we need to go. Um, and then, yeah, decide the next best action. And just just show you that I keep learning from the students, uh, it's hard to see, but it says Erica.

This was Erica Dval. She's, uh, she got hired at Rapid 7 this past year. she's alumni of us, but she created some notes on how to dump the memory from CrowdStrike because I was always like, how do I I I couldn't remember the timeline and I was really bad at taking notes. So, she took notes for me and uploaded. I'm like, yes, you're awesome. So, uh this is a great example of that. But it takes all of us to uh to conquer the incident. Next slide. Um yeah, you want to go ahead Allesandre? Yeah. So something else which is really important in every technical field is being able to break down technical concepts in simple words.

And in this specific case of incident response exercise uh is being able to summarize a 4 hours incident into and more than 300 logs into just 10 slides. And for example creating like a diagram like uh these are shown in the picture that you can just see the main key events that happen in the incident. like you you have a get request to this uh binary file that is communicating to these two domains and whatever host machines were infected by it and this is like very easy to visualize and to present it to like also nontechnical people. Yeah. Excellent. I and I know we're wrapping up here. I think we have one more slide but one thing I like to

say also is I make the incident lead responsible for writing a formal report just like you would have in the real world. You know the technical stuff's for show but reports are for the dough as I've heard before right. So they actually have to go through the process of writing executive summary, technical findings conclusions and recommendations. And sometimes we actually present this for QA purposes to our cyber range operators to say, "Hey, this is what we're finding good. This is what we're not finding good." But also to show and teach other seerts uh this too. Next slide. Uh development. I've talked about this enough and I know we're at time, so I'm going to go ahead

and just let you take a quick look at this, but ultimately we're looking to develop to make these guys stronger, better, ascend to that next level of Super Saiyan, and we get that through the hyperbolic time chamber training. Um, next slide. Um, that's just more of the same on that. Now, go ahead and go to the next slide, Aleandro. Uh, see if you can zoom in. I know this is an eyesore, but I really want to get to this. This is the crux of it. As Alessandro said, we do about four hours and we had about 300 logs of in our timeline. That's 300 logs compared to tens and hundreds of thousands of records that we filter and go through,

right? So, we're finding the things that are most crucial and we're putting them into a timeline of activity. And we're just looking at a sample here, but we can basically see at the top that we have a a cmd exe spinning up schedule task.exes exe. And we're seeing essentially a schedule task being created. And that schedule task is essentially uh called TN uh launchd. And then it's uh tying to a malicious binary. All right. Um and then if you follow that, we can actually see several instances of uh that schedule task uh being spawned off. So it's nice when we have CrowdStrike seeing it. What's nice when we have Windows log seeing it. That's just creating a stronger

activity. Um, and then we can actually see what looks like uh this HS.exe spawning uh or or executing on the system. Uh followed by now we're seeing post actions from that compromised asset to this uh blah blah blah blah.Z domain. It's a little bit hard to see, but there are slides. Um and you can you can definitely review those. Um further down the further down the line, uh we can also confirm HSE.exe is running followed by task list.exe. exe system info.exe and then it's eventually we see a PowerShell command that's doing a get include uh recurse on app data folders and um uh I'll just tell you it's basically doing reconnaissance at this point if you haven't figured it out. Uh

but it's basically identifying uh what what processes are running what's the system information what files can identify and then it's actually trying to xfill that content out of the system and that's just a sampling of the timeline. And if you actually look at our whole timeline, you would see the initial process of fishing email or driveby download to binary running to as to initial um access of the command and control etc etc down the line. So this is what the students are working on. I think this is our last slide, right Allesandro? Yeah. So go ahead and go to the next one and we do have some references. Yeah, that's just uh back doors and breaches uh for your reference

if you want to know how to play that. And then next slides are just about cyber Florida, but I can answer those questions on the side if you have any questions for us. That's it. Any questions from the group one?

Correct.

That would never happen. That would never happen. So to answer that question, we actually did have some incidents where like I didn't get a like I knew it they fired the attack, but I'm like why didn't I get any crowdstrike alerts? So, we had to basically operate without CrowdStrike telling us what it was and it was actually some filtering issues on their side. Uh, that what that's why it didn't alert us or like pop up to the dashboards or anything. Yes. Question.

So, I'll tell you my experience back when I worked at JPMC. Um, now that I'm not there, I could probably talk a little more freely about it. Uh, so that's a problem you're going to have anywhere, right? So, as with anything, you have to ask the question, is training a priority for the organization? Right? And I think initially you'll probably say there'll be a lot of answers. Well, you know, we got these other projects, these other priorities. I promise you because this happened when you get into a real incident, a real scenario and nobody knows what the heck they're doing and you're like, why does nobody know what the heck they're doing? All right. Well,

now we need to maybe incorporate some activities periodically, not all the time, right? But hey, on this day, we're going to allocate this day or this half day or on Fridays for two hours, we're going to do these sorts of activities. And when I was at the bank, um I was actually tasked with working at they use Simspace um to actually develop training to help our forensic analysts on certain activities. So that's what I did. They at least saw it important enough to at least start that activity and and train because there were gaps in that. But like you just can't have people constantly working without any professional development because they won't grow. Any other questions? Yes sir.

So in the in the IR process a lot of times if you're dealing with the organization um you're dealing with collecting targeted images and acquisitions. If you're actually going to go and actually seize the box, we have talked about chain of custody. We have done some investigations where chain of custody would and should be required. Um, but in terms of incident response, that's typically not done unless you're dealing with like an insider that you think is going to go criminal and things like that. Like even at the bank, like we would just collect stuff remotely and have it. But if we needed to actually obtain the asset, which very few times we actually did, we

would either go and seize it ourselves or we would have it white glove delivered, which is a very expensive process to get us so we could maintain the integrity of that asset. Typically for the instant response, no. But do we talk about it when we do other forensic investigations? Yes. I saw another hand up. Yes.

that one. Oh, yeah, sure. No problem. Any other questions? Thank you for coming by. I hope this was entertaining and fun. You guys have a great rest of your day at Bside Stampout 2025.