BSdides LV 2022 - Wednesday - Ground Floor
Show transcript [en]
[Music] foreign
[Music]
[Music] foreign [Music]
[Music] foreign [Music]
[Music]
thank you [Music]
foreign
[Music]
[Music] foreign
[Music]
foreign [Music]
foreign
[Music]
[Music] foreign
[Music]
[Music]
[Music] foreign
[Music]
[Music]
thank you
[Music] because
foreign
foreign [Music]
[Music]
foreign
[Music]
foreign [Music]
[Music]
foreign [Music]
[Music]
foreign
foreign
[Music]
foreign [Music] [Music]
foreign
[Music]
[Music] thank you
[Music]
[Music]
foreign [Music]
[Music]
[Music] foreign
[Music]
[Music]
foreign [Music]
[Music] foreign
foreign
[Music]
[Music] foreign
foreign
[Music]
[Music] foreign [Music]
[Music] foreign [Music] foreign [Music]
[Music] foreign [Music]
[Music]
[Music] foreign [Music]
[Music] foreign
[Music]
[Music]
foreign
[Music]
foreign
[Music] foreign [Music]
[Music]
foreign
[Music]
[Music] foreign [Music]
[Music]
foreign [Music]
[Music]
[Music]
foreign
foreign
foreign
foreign [Music]
[Music]
foreign
[Music]
foreign [Music]
[Music]
foreign [Music]
[Music]
foreign
foreign
thank you [Music]
foreign
[Music] thank you [Music]
foreign
[Music]
[Music]
foreign [Music]
[Music]
[Music]
foreign [Music]
[Music]
foreign [Music] foreign
[Music]
foreign
[Music] foreign
[Music]
thank you
foreign [Music]
[Music]
[Music]
foreign
[Music] foreign
[Music]
[Music] foreign [Music]
[Music]
foreign
[Music]
[Music] foreign
[Music]
[Music] foreign
[Music]
[Music]
foreign
[Music] foreign [Music]
[Music]
foreign [Music]
[Music]
[Music] foreign [Music]
[Music]
foreign [Music]
[Music]
thank you
[Music] thank you
foreign
[Music]
foreign [Music]
[Music] foreign [Music]
thank you
[Music]
[Music] foreign
[Music]
[Music]
foreign
[Music]
foreign
foreign [Music] [Music]
thank you
[Music]
[Music] thank you foreign [Music]
[Music]
[Music]
[Music] foreign
[Music]
[Music]
foreign
[Music] foreign [Music]
[Music]
foreign
foreign
[Music]
foreign
foreign [Music]
[Music]
[Music]
foreign
[Music] foreign [Music]
[Music] foreign [Music]
[Music]
[Music] foreign
[Music]
foreign
[Music]
[Music]
foreign
[Music]
[Music] okay
thank you [Music]
foreign [Music]
foreign
[Music]
[Music]
foreign [Music]
[Music] [Music] foreign
[Music]
[Music]
[Music] foreign
foreign
[Music]
foreign
foreign
[Music] foreign [Music]
[Music] foreign
[Music] foreign [Music]
[Music]
[Music]
thank you [Music]
[Music]
thank you [Music] thank you [Music]
thank you [Music]
foreign [Music]
[Music] [Music]
foreign [Music]
[Music]
[Music] foreign
[Music]
[Music] foreign [Music]
[Music] foreign
foreign
[Music]
thank you
foreign [Music]
[Music]
[Music]
foreign
[Music] foreign [Music]
foreign
[Music]
[Music] thank you [Music]
[Music] foreign [Music]
foreign
[Music]
[Music] foreign
[Music]
[Music]
foreign
[Music] foreign [Music]
[Music]
foreign
[Music]
[Music] foreign
[Music]
[Music]
foreign [Music]
[Music]
foreign
[Music]
foreign
[Music] foreign
[Music] foreign
[Music] foreign
[Music] foreign [Music]
[Music]
[Music]
thank you [Music]
[Music]
foreign
foreign [Music] [Music]
foreign
[Music]
[Music]
foreign [Music]
[Music]
[Music]
foreign [Music]
[Music]
[Music]
thank you [Music] foreign [Music]
[Music]
[Music]
foreign
foreign
[Music]
[Music]
foreign
foreign
[Music]
[Music]
foreign
[Music]
foreign [Music]
[Music] foreign [Music]
[Music]
foreign
[Music]
thank you [Music]
foreign
[Music]
[Music] okay
[Music]
foreign [Music]
foreign
[Music]
foreign [Music]
[Music]
[Music] foreign [Music]
[Music]
thank you [Music]
foreign
[Music]
foreign
foreign
foreign
[Music]
[Music] foreign [Music]
[Music]
foreign
[Music]
[Music] foreign
[Music]
[Music]
foreign
[Music]
thank you [Music] thank you [Music]
thank you [Music]
foreign [Music]
[Music]
[Music]
foreign [Music]
[Music]
[Music]
thank you
[Music]
foreign [Music] foreign [Music]
[Music] foreign
foreign
[Music]
[Music]
thank you
foreign
[Music]
[Music]
foreign
foreign [Music]
foreign
[Music]
[Music] thank you [Music]
[Music] foreign [Music]
foreign
[Music]
[Music] foreign
[Music]
[Music]
foreign
[Music]
thank you [Music]
foreign [Music]
[Music]
[Music]
[Music]
thank you [Music]
foreign [Music]
[Music]
thank you
[Music]
foreign foreign [Music]
foreign
[Music]
[Music] foreign [Music]
foreign
[Music]
[Music] foreign
[Music]
[Music]
foreign
[Music]
foreign
foreign [Music] [Music]
thank you
foreign [Music]
[Music]
[Music]
[Music] thank you
[Music]
foreign [Music]
[Music] foreign [Music]
[Music] foreign [Music]
foreign
[Music]
[Music]
thank you
foreign
[Music]
[Music]
[Music]
foreign
[Music] foreign [Music]
thank you foreign
[Music]
foreign
[Music]
foreign [Music] [Music]
thank you [Music]
foreign [Music]
[Music]
[Music]
foreign [Music]
[Music]
[Music]
thank you [Music] foreign [Music]
[Music]
thank you all right good morning everyone thank you for coming for the first talk good good morning everyone [Music] yeah there we go oh God I hate when people do that all right so uh I have I I have my little talk I Gotta Give real quick so I'll do this real quick before we introduce uh first good morning everyone I did that all right good morning everyone and welcome to b-sides Las Vegas this is the uh uh the ground floor look up doing so many of these right so welcome to the ground floor track um this talk is I got an alert now what so uh and of course it's given by uh Kellen Benson
we're very pleased to have you here thank you so much for coming and speaking uh a few announcements before we begin uh sponsors of course we need to thank our sponsors the uh they are the uh the companies and the people who make that happen we'd like to thank our Diamond sponsors LastPass and Palo Alto networks of course and our gold sponsors blue cat Google and Intel also uh absolutely vital to making all this come down and and working so along with the support staff and the people who have been vital in making the actual b-sides possible so as a reminder uh cell phones these talks are being streamed live and they are being recorded you do not need
to record you do not need to take pictures so please silence your phone as a reminder the policy for taking pictures is if you were to even take a picture you're supposed to make sure you have everyone's permission if they're in the background of the picture right we're very sensitive to making sure we're taking the privacy of people very serious especially within the community we're within so I do ask if you have a question please step up to the mic use the mic it's not so much for us but because it's being recorded that makes sure that question gets on the recording so people who see it later can hear your question as well uh of course Kelly if
you always repeat the question before you answer it that's always nice too so um as a reminder I just talked about that for the policy uh with that let's get started so I'll pass it off to you thank you so much awesome let me here we go okay cool thank you all for being here I really appreciate it um welcome to a talk called I got an alert now what uh before we get started just want to introduce myself very quickly my name is Kellen Benson my pronouns are they them I work at Red Canary as a senior incident Handler um where I put out fires for people it's pretty fun um and there's a little bit about myself
I'm a pop punk Enthusiast if there's karaoke later this week you might see me doing some Paramore so uh look out for that but that's not why you're here you're here to learn about I got an alert now what sorry I want to go back to that if you want to reach out to me Twitter's the best place to do it I have LinkedIn but I don't check it so just find me on Twitter um so I got an alert now what I had some friends that I was talking to and they one of them was a newer person who was like Hey I when I get an alert what should I do and my first response
was well you investigate it and then I realized that's a horrible answer um I did the thing that senior people do where they're like I'm gonna shortcut everything that I know and just give you the short answer which isn't very helpful to new people so I went through this process of what actually do I do when I get an alert what's my methodology how do I investigate this thing um and I think that's really helpful to do as people with experience in the industry because not only are you sharing information to someone you get to check yourself for gaps and think like oh did I miss a step or is there something I could do better here
um so that's really what I did here that's what this talk is about to walk through what my methodology is and some people um along the way who help me and it's not really going to jump into tools a whole lot we might touch on some things lightly but it very much is a methodology based thing so let's Dive In we're going to touch on we have an alert um what do you do when you get an alert so we'll we'll dig into that a little bit we need to figure out what the alert is actually telling us just because you have a blinky red light doesn't mean you know what to do with it so we need to
understand why the Blinky red light is on and what we need to look at to understand that blinking red light what is interesting about the data that we have in that alert as well as is it even a threat like do we care and if it is a threat what do we do next and then we're going to do a little bit of what if because the world's not perfect and sometimes we can't answer all of our questions with the data we have so here's our alert we have a high severity alert that says Excel launch a suspicious program and we see the command line here that we see okay we have excel.exe and suburb budget 22 xlsx
probably very safe very fine budget spreadsheets are normal um but that that's something we need to take into account here is like what is this alert telling us and do we care on top of that um sorry we don't have a ton of data here sometimes when we get an alert we only get like bits and pieces of data and we really have to dig in um and we don't get to necessarily see the logic behind that alert and we have to come together come to some some conclusions um based on what we have so let's try to understand what this alert is telling us and if it's actually true we need to understand did Excel launch a
program if it launched a program is that program actually suspicious or is it normal do we care and yeah so let's just dig into that all right so if we had a data point if we had like an EDR we had system on we had a Sim something like that we have logs of some kind we might be able to grab a processed tree for what happened here this really depends on your data we're just going to assume we have good data life is good everything's perfect um and because of that we have a full process stream we can see Excel launched reg svr 32 or register 32 we can fight about how you want to pronounce that later it's
totally up to you um but it launched red serve 32 and then off of that we have a module load of EF HJ Dot dll um we'll get to that but just take note that's our process tree um we have this Excel file that opens and led to register 32 opening which we should probably grab the hash of that Excel file for investigating we have the name of that file these are important little things just to take note of as we investigate um and then we'll want to dig into register of 32. but we need to understand like I said we're trying to understand is this suspicious or is this normal in order to make a determination if something is
suspicious or normal we need to know what it actually does um you know and so the way to do that to the Google machine or Bing if you're into that um so register of 32 was launched what is it what it was what is its purpose um a quick search of this you're gonna land at a Microsoft dock of what regserv32 is and it registers dll files as command components in the registry okay if we think back a little bit we saw a module load of EF HJ Dot dll so we're registering that as a command in the registry is that normal I don't know yet we're gonna find out um I see some head shaking in the audience
which is fun um so you're already steps ahead of me um but we don't necessarily know if that's normal or suspicious I mean could that dll be normal EF HJ could be an abbreviation for something an acronym I don't know or maybe it's just very bad um so we're gonna grab the hash of that we likely have the hash of that when the file was written or when the module was loaded uh depending upon your data source you can grab it from there um so let's move forward here so these are our points of interest we have an Excel file with a hash we have we have a dll with a hash and then we have regserv but
that maybe it's interesting maybe it's not these two things probably the most interesting bits of information we have so far what do we do with that is it the important next question um sorry um but first wait if you're looking at a process tree you might see something like this where you're like hey register of 32 didn't just load that module it did all these other things that's kind of interesting pause don't go down the rabbit hole too early it's really easy to get lost while you're investigating when you're trying to triage things and deal with the alert initially and end up like I'm gonna go dig into j-e-o-o-n.exe been there done that and then I lost
track of what I was originally doing and it's been like 30 45 minutes and I haven't actually triaged my alert and made an important decision which well it was fun and maybe I found something neat my alert's still sitting there and we haven't done anything so don't go down the rabbit hole too early at this point what I would probably do is just take note of those names take note of the hashes if you have them and put it off to the side because what we really need to dig into is that initial piece of the alert we know Excel program dll with that we can probably come to a pretty good conclusion so let's let's keep going
let's keep going with those artifacts obligatory pyramid of pain if you're not familiar with the Pyramid of pain we're going to talk about it now um so what we have are artifacts or potential indicators of compromise what is an indicator of compromise I defined it as a thing we can observe in our data that allows us to identify a threat at the moment or going forward in our detections so there's some difficulties that come with getting indicators of compromise some are very easy biohashes IP addresses and domain names if you've ever gotten a report from a three-letter agency or read an Intel report from the internet you probably have those things to go throw into your sim dig into that data look
for it in your environment those are very easy to find and search for um and then detect on in the future but those things can change they rotate quickly and honestly you're not always gonna be able to find those things because attackers change what they're doing so we look for network and host artifacts we look for tools and we look for tactics techniques and procedures those become more difficult to identify but also then write detections on and create alerts on if you are a person who writes detections you probably know that writing behavioral detections is a pain um I see nods so yeah so that is though that's what an indicator of compromise is and we're going to leverage those
to hopefully come to some conclusions about what we're looking into what I would do is I'd go virus total what else are we gonna do we're also going to put a hash but wait I'm gonna take this moment to jump on a little bit of a soapbox here about virustotal um because I've made this mistake and I've seen other people make this mistake and maybe you don't even know it's a mistake and you don't know because you don't know what you don't know um don't submit your files to virus total um because they can end up public facing and that can expose your data your company's data to the internet um as we all know the internet is a very
safe place and it's a great place to store your data right um on top of that depending upon your threat model you might tip off threat actors that you have their tool you've detected it and now they're going to burn that and then switch it up and your indicators of compromise are no longer good that's a little bit secondary but it's still something to keep in mind the main thing is don't upload your files because you don't want your company's data on the internet um if you don't know if you have a policy or procedure around that at the place you work you now have homework you're welcome I've given you an action item your managers will be thrilled
um so take that back find out if you do have a Palestine procedure around that if you don't make wood it's easy share it with people let them know what can happen if you do that um all right I'm gonna get off my soapbox now um about that and we're gonna keep moving so is it a threat maybe maybe not we have these hashes we're going to search them um let's move into that we're going to start with register 32 though because there's a potential hang up here when we search this on virus total we get the results back that hey this was distributed by Microsoft it's signed by Microsoft no vendors to detect this as malicious
so we're good to go right everything's fine no not at all um you can leverage clean binaries to do evil um it happens all the time you've probably heard about a good friend Powershell and all the terrible and wonderful things Powershell can do but when you search that also clean hash signed by Microsoft good binary right maybe not um a little bit of story time I've been at places where the alert was suspicious Powershell okay cool is Powershell suspicious or is it what Powershell did the analysis of that was Powershell clean hash close alert that didn't turn out well um there was I don't remember what it escalated to but it was not good and it
went everywhere so bear that in mind just because you have a clean hash doesn't mean the thing you have is safe um it's like a chainsaw chainsaws are good for cutting up things are they safe maybe depends on how you use it that analogy I'm sorry we're gonna move on all right now let's look at the hashes of our other files and virus total it's evil very clearly a mermaid man has let us know this um unfortunately virus total doesn't show you Mermaid Man when you use it I'm sorry um but when we searched the hash of our Excel file we have 29 vendors that detected this thing as bad and then with our dll we have 56. the
denominator here changes I don't know why some vendors are in some lists and some aren't um talk to virus total not me I apologize
Okay so what have we done so far let's recap we have got an alert we understood our alert we understood that Excel launched a program that's suspicious based on the hashes we have we now know we have suspicious hashes which we can call our indicators of compromise and we can move forward with that information to ask more questions and get more data to help us make an informed decision um the next thing I would do is who is the threat and I'm not saying is this wizard lizard spider is this a nation state like when I say who I mean what is our malware family what but what are we dealing with um we're not going to talk about
attribution if you want to talk about attribution find me later at a bar so we can get information about who our threat is typically from our alert and if we can't get it from our alert we can probably grab it from virustotal if our hashes are being detected um because when you search a hash in virusotal you're going to get the names of detections from all these vendors that say Hey this is bad um in this case we see if you can see it on screen says emotat it's highlighted it was all over the page I only could give you a little bit of that but it was everywhere um so we can say with some decent
confidence we're probably dealing with emotep um if we don't know what that is we're going to pretend we don't and we're going to go to the Google machine but first I want to talk about some other sources that you can go to be on virustotal if virus total doesn't have results highly recommend checking out malware Bazaar or triage um you can search hashes there as well and they tag their malware so you can see like oh is it emotec does it lead to something else it's great malware Bazaar is free triage requires a free account so you just have to sign up but it gives you some really good information about the binaries that you're searching as well
great resource for Learning and trying to figure out what you may have um so with this I would feel comfortable saying this is emotat um and dig in from there who's that Pokemon emotep all right now we have a Who and the reason I delineate from who and what is are who gets us to the what and the what is what is the threat and when I say what is the threat we need to know what is the behavior what are we looking for what does emotec do what's its MO when it lands in your environment what are its goals and when you can understand those goals you can understand impact which is what we as analysts are here to do is
understand the impact of these threats to our environment and deal with that and that will help us to triage and escalate appropriately so is emotech going to move laterally is it gonna dump credentials is it going to just change a background to something silly who knows let's let's dig further we're doing a lot of digging should I got a shovel that's what I should have requested was a shovel um okay so when you're searching about you're gonna get a bunch of results uh I'm gonna shamelessly plug red canaries threat detection report because it's a good resource even before I worked there loved it um really good information but if you search emote you're going to get
information um and in the threat detection report we see some important information I've highlighted or folded what I think are the most important bits to answer our question here which is emote head is a downloader or a dropper of other malware so with that we know hey more malware which could be fun right um and its goal is to steal user data and banking credentials now it says banking credentials it's probably all your credentials and you're gonna have a bad day um because I have some feelings that emotec isn't gonna say well that doesn't say JP Morgan in the cred file they're gonna take everything um so now you have the ability to understand impact of what your threat is
I do want to highlight one more bit here that it's disseminated through email links or attachments that's important to note for cleanup later um just bear that in mind
um so if threat then what um if we have a threat which I think at this point with the data that we've collected with the questions that we've asked the data points that we've seen we can probably arrive in a place that we probably have a threat on our computer we have Excel suspicious program virus total lights up on the Excel file and the dll and then if we remember back we have all those fun processes that came out of the register 32 that we took note of um we'll get to those but just keep that in mind um so with that my first step here is contain I don't want that endpoint on my
Network anymore I don't want it to talk to anything something bad is happening um you can do that through whatever tools you have if you have an EDR hit the isolate button if you have Network tooling just get it off your network um so with that and then we move to eradicate win well pause we're gonna go back to contain because if we're going to contain we also need to ban those binaries in our environment without it with whatever tooling you have available um that's a little bit more difficult of a conversation if you have application block listing cool you can do it that way if you have that power through your EDR great if you have to push a really
weird GPO do that I mean you know you got to do what you got to do to get the threat contained in your environment um and we're going to take action on that Excel file and that dll and then probably everything else that came off of that register 32 because we don't know what it did it looked sus so we should probably just deal with that now and ask questions later um so with that we've contained that endpoint but is this other places in our Network is it on other endpoints if it was delivered through email which we saw earlier do people typically only fish one person or is it your entire organization do you have email addresses facing on
your website are they pushed from your people to their LinkedIn profiles you're probably getting a lot of phishing emails with some delicious emotat attached um so I would scope I would search for those hashes of that we have see if they're on other endpoints and repeat our contain phase we need to isolate those endpoints and then we need to eradicate our threat now with eradication and Recovery there comes an interesting question of do we go through each of these endpoints and clean up everything manually do we delete all these Excel files do we delete all these binaries do we look for you know was there a registry key that changed was there a scheduled task that
was put into this device that's pretty tedious um you could do that if you wanted to do a full analysis of what happens I would say one box maybe to do that if you want to do the full investigation but I'm more of a re-image type of person when it comes to those situations because you could miss something uh story time uh when hold on we'll think about how to tell this story I've seen situations where people have manually cleaned up malware they thought they killed processes they deleted files they banned binaries few hours later hey your box is reinfected with the same thing how'd that happen I did the thing I cleaned it up
well you missed persistence ah okay so we deleted the scheduled task a few hours later hey your box is infected again what happened well there was more persistence there was a registered key there was this or that and like it's a problem you play whack-a-mole and it's a waste of time um I'm more of a reimaged person especially if something jumps into memory re-image that's scary um if you want to have an argument about reimage versus manual cleanup I gladly have that one too that sounds like fun um but just keep that in mind as you do clean up reimaging is usually a bit more efficient and faster great we got an alert we investigated it sorry
did the thing again we got indicators of compromise we search those indicators of compromise on the interwebs through virustotal to understand what they are and what we were dealing with we got information that our threat was emotat we dug into what emoted actually is we evaluated what it could do in our environment and then we dealt with it great but sometimes you don't get all that information sometimes you don't have the full picture what if you don't what are the things that you can do to get more information and better understand what's happening we go down the rabbit hole of course um so the first question I would ask if I were in a situation where maybe those hashes
didn't return malicious um or I just didn't fully see something that made me my Spider Sense tingle and go evil um I would start with the process tree and I'd worked my way from where that Excel happened down the process tree which is where we see all those other things that came in and we'd investigate those the same way that we investigated the initial thing we would ask the question of what is this file what is it doing is it malicious can we make a determination on that and go through that um and did you know that this song would come into your investigation where did it come from where did it go [Music] um which is a great question and now
you're going to remember it like that and every time you ask that question you're gonna hear it is the lyric and you're welcome um so asking how did this get here how did that Excel file land on the device was it can I see it coming from a browser download can I see it coming from email um and where did it go from there which we jump back to the process tree a little bit but don't get caught up in processed trees all the time because things can break from the process tree and there's other things that can happen around that there have been times where I've seen data where I dead end at regsurf
it just I see a mod load and that's it and that's all I got um and then I have to say Okay where did this actually go though and pull in more data look at a timeline and look at like you know five ten minutes around that to see if something else happened and I've been in situations where oh something touched lsas that wasn't supposed to we have credential dumping or a command prompt spawned and it's doing recon those types of things can happen and they cannot be necessarily attached to your process tree so just bear that in mind as you ask the where did it come from where did it go question and don't get tied up in your
process tree if I'm like really stuck and those two things don't give me an answer like I'm not getting is this malicious from virustotal I can't figure out exactly where this came from I'm not sure where it's going either like I need to start asking is this normal is this normal in my environment I usually start with the user and then go out from there um because sometimes it is normal for that user if you checked like a 30-day period if you have 30 days of data um very lucky if you do uh you're you're doing great um but if you don't you can check whatever data you have to see if that happens regularly has it happened over
the last seven days okay maybe not does this happen on the computer holistically um not necessarily relevant for this Excel file thing but generally speaking like does this weird thing happen on this device normally or scoping out to your environment that's an important step too because you might realize oh this is happening across the environment and it's weird we should probably document this and understand why it's happening why we're generating an alert on it and figure out if that alert alert is valid or not and then tune your detectors sorry more homework um the other thing is I want to jump back to the user thing a little bit because I've seen situations where Excel does
some really wonky stuff and it's usually in the finance department and they're usually running macros and they have add-ons and it looks bad like real bad um but if you can understand why that's happening and if it's not bad you can write up some documentation to share with your team to say hey this is normal and you don't have to waste time on that alert anymore and you don't have to be sad and you can look at cat pictures on the internet or something instead all right let's recap what do we do here today we evaluated what an alert was saying we took time to understand what the alert actually said which is important because we can end up on a
totally different Trail if we're not paying attention to our original hypothesis which that's kind of how I view alerts is there a hypothesis about a threat that could live in your environment and you're going to prove or disprove them you're like a scientist sort of as long as you take notes of science um so we evaluated what the alert was saying we were able to identify indicators of compromise with the data we had so we have hashes we had file names with those hashes and file names we are able to Pivot to a search on virustotal and understand oh we are dealing with something evil here great now we need to figure out what exactly
that evil thing was um thanks again virustotal we found out it was emotep and because we knew it was emotep we could dig into that further and understand why that was important what the potential impact of emote on our environment was and with that information we knew what he needed yeah we knew we needed to respond because somebody could have stolen credentials somebody could be dropping more malware um and we need to understand if that's happening as well also rollback to remediate one sec rotate your credentials on the boxes that were impacted I didn't say that but make sure you do that if you've got that happening oh that would have been a bad day good job everybody we did it
um we responded we recovered we also tried to understand what we could do if we didn't have all that data what if what if we don't have all the information what are the questions we can ask we cannot we can dig further into a process tree we can ask where did this specific thing come from where did this specific thing end up on the device and what are the surrounding activity and is that suspicious as well and if it's really not answering we need to understand is this normal um and asking that of for the user the computer and your whole environment uh in about 30 minutes there will be a Blog on the internet that has all this
content and you can go check that out as well if you want some notes uh basically it's my speaker notes put into a Blog format so you can just go grab that it's on red Canary's website um go check that out thank you all so much for coming to this talk um I really appreciate each and every one of you I appreciate everybody who helped me put this talk together all the people who uh listen to me fumble through it uh the first few times and um with that I'll open the floor for questions [Applause] thank you so much that was a great uh talk um I have one question so you mentioned this under the identification phase or
where you ask more questions would it be a good step to go to the user and actually I'll Stand directly where did you get this from is this something you expected as a step to include yeah that's a really good question so the question was in the identification phase would you reach out to the user to get information from them about what they were doing and was this the expected result and I actually think that's a really good step and if you have the ability to do that you could cut down on your ability to triage very quickly um I really like that step some people don't I'm a fan of it when I work with customers I often
say can you reach out to this person just to understand what was happening on their device before this went down like were they on a website did they open an email or were they just doing their normal job because that gives you an understanding as an analyst of what you might be dealing with so yeah I totally think that's valid all right thank you yeah
we gotta fight for the mic jeez it's true battle the the network Giants um question for you is so this is like the most comprehensive how to do an alert I've ever seen now we tend to have tuning processes we have a lot of problems there's lots and lots of alerts like is there any way we can sort of get like a like a like a hack of this like a shorter more streamlined process that's how we do this yeah like so what we did here was a bit extensive and I want to talk to that but there is a point where you streamline this I think the thing that I typically cut out when I'm evaluating something is the
what is the threat because just with experience I'm I have the ability to say oh this is evil this is bad we need to deal with it and but that's not something everybody has so I wanted to highlight that but as you go through this process more and more and more and you see more data and you deal with more things you can cut that piece out because you have an understanding of what you're probably dealing with and it is a threat um so you're really just hitting the what's my alert say what are my indicators of compromise what can I gather from that okay let's respond [Applause] listen thanks very much that was an
awesome talk I really appreciated it um so have you ever run into a dead end where you're investigating it a lot and what did you do question was when I run into a dead end what do I do um cry is typically the response um throw the computer into a volcano no um seriously I what I do is I ask questions of people that are around me if you have if you have co-workers use that resource like ask questions because even if they don't know you're gonna have them ask different questions that maybe you did and they can see something that you didn't and you're leveling up your co-worker and yourself at the same time
great teamwork that's awesome do that um that's my biggest thing is use your team when you hit a dead end um that's usually where I go um mostly because my team's great uh if you don't have that like I would say take a break like if you can like pass it on to somebody come back to it because sometimes you just need that two minutes to get away and then come back and you see something fresh um the other thing is change how you're looking at your data perhaps change how you're searching it maybe you don't look at the hash maybe you look at the command line maybe you look at um I'd say the command line actually is
a great place to go if you're not getting anything on hash because you'll end up in a threat report that's like oh that's the exact command line for this threat I now know what I'm dealing with this is bad or maybe you'll end up in a GitHub repo for like an administrative tool I've been in both situations um so what is this [Laughter] no no I got it I can do it okay yeah
I didn't know long time caller first time listener what are some resources you have for incident responders who are just starting out that's a really good question and what are resources that I have for incident responders that are just starting okay um I would say this is a really good question oh my god um I think the place that I would go is actually somewhere like malware bizarre triage or a site that does like analysis of a file and is going to show you information about a process tree and like really detail what's happening because out of that you can kind of understand what's happening with that binary and get a clear picture of what's happening
there and you'll also get like a fair amount of detail I if I if I could show you I would but just try triage it's really good um and it's a great place to look at binaries and the other thing is play around in a lab environment or search your production data for interesting things like do something on your computer that's normal do something on your computer that's normal and then look in your data to understand why that's normal and understand why what Excel does when it first launches what a word does it what you know any of these things do and what are the nor what does that normal behavior look like because when you
understand normal you then are able to start looking at things and say Spider Sense tingles bad um is it kind of boring to watch Excel open yeah but then you have what that looks like in your data and that's really important um and maybe you spin up a lab and you have it isolated and then you rip malware on it and then you look at the data on that also a great thing to do um and then from the internet response side of like dealing with it um I would I'm trying to think if there's any like free EDR tools out there that you can leverage but if you can't um get in there do an investigation and
do the manual deletion just so you understand where stuff is I said rimage your box but like if you're learning going and manually remediate a box it's tedious it's a pain it kind of sucks but it's good for you eat your vegetables
hi I have a question on the same topic so we decided to wipe out a system but you know in the real world people say hey I have a lot of files how do I back them up and that's a good question because you isolate an end point it means internet connection is not it's killed like how do you do it safely because you know you can do you can go to Google Drive and try to save those files but there's no guarantee those files not corrupted you can plug in USB drives the same logic there so that's yeah I guess my question is how to safely back up important files from a corrupted system that's a really good
question the question was when you have a box that's infected and isolated but that box has important files on it and you need those files back what do you do and make sure that those files are clean um yeah that's tough and it really depends upon the value of those files and your appetite for risk um I think that's a really big thing and I've dealt with this with some customers where they're like it's really important like budgeting spreadsheets and they lived on this person's desktop which that's another conversation to have about where you're storing your important files um but if they are there you're gonna have to end up in a spot where you
get a USB drive you never use again probably plug it into a computer that is isolated and do a scan of those things and see is this safe are these infected in some way shape or form um the other thing you can do is if you have the data you can evaluate where those files even touched within the timeline of the malicious thing happening on your device because if they weren't touched you have no evidence of that in your Telemetry and you scan them you're in a pretty good spot to say hey these things are probably safe um generally speaking now we work in cyber security and the answer is always it depends um but that's where I would
land with that type of thing it it can be tough though and sometimes it's you lost that file I'm sorry um you shouldn't have stored that on your desktop we have Enterprise storage for a reason that's a fun conversation too but we won't go there
so I got a question about like time detection or time to decision so you're trying to get in there you're trying to investigate but time is yeah fleeting and like malware replicates very rapidly so especially if it transmits between different computers and things like that okay so if you find that you have like a standard or or anything like that in companies that where they they say you know need a decision by here or we're just gonna have to decide one way or another yeah and I think it's good to have guidelines it's like the Pirates Code um if you watch Pirates of the Caribbean uh where they are guidelines and you should be able to say like okay I have a
high severity alert that says malicious thing with that I would say your time to decision does need to be shorter you because if you have confidence in your alerts and you do have a high severity malicious thing then your time to triage that and move into the response should be very minimal like I hate putting a number on this but like in previous places that I've been um like a high pry like very critical thing is like take five minutes do the initial triage try to get it into the response Zone like if you see credential theft if you see something that's a precursor to ransomware like get that moving if you're running into like
suspicious encoded Powershell okay you need time to actually like decode your Powershell understand what that script's doing maybe that's 15 to 30 depending upon how complex that is um so it really comes down to what your alerts are your confidence in those alerts and what you actually have to investigate to make a decision I don't think I've seen something where like I'd say like an hour is typically way too long if you're dealing with something that's generating an alert that's you know medium or high suspicious if you're dealing with like low interesting things then like yeah take your time dig into that because they're not necessarily always going to turn into something that's so tough though
it's a dilemma uh we have about 10 minutes left uh just to let you all know if anybody has more questions feel free
um so I want to know when the Kellen merch line is going to be available for pre-order what do you want to know in my slides uh yeah I want to know when the Kellen merch line will be for pre-order and uh I particularly want fronting tie-dye crop tops uh apparently we have oh my God a serious question though a serious question now how often do you find that you personally have to go back to the alert logic and completely or even minorly reconjigger it because you're finding tools or giving you false positives or completely maybe useless information um just kind of interested about that of the balance of alerts versus coming from your tools versus what your sim or what
you've said this is a proper alert um merch line coming soon but the real question is what how often do you find yourself having to tune your detections basically like when are how often are you getting false positives from your alerts and you have to go back to tune your alert logic um all the time all the time you can't a Sim uh detection that sort of thing is not set it and forget it it is like a garden you have to weed your garden you have to water your garden you have to make sure that your plants are growing in the appropriate places I don't Garden so that's as much as I know sorry
um but that's what it is you have to take care of it you have to go and deal with those false positives it's easy to close your alert and say well don't want to deal with that anymore and then the next morning you come back to 10 more and you're like gonna close those don't wanna deal with that anymore you're creating Tech debt for yourself and the people after you that's mean Tech that's bad and you might be missing things if you're not tuning your logic maybe something new happens and you had an incident that spun out of an alert that was okay but there was other behavior in that incident where you could create a good
detection create a new detection then tune your detection that's a lot of work I know some people there's two of you doing your security and tuning your detections and doing everything else you do is hard been there get it but it's important and I would say I'm gonna get on a soapbox here is if you're in that situation communicate the business risk to your management and the people above you of why there's business risk that you can't spend time tuning your detections um so say if we don't have time and people to tune our detections we're going to fall behind and we could be impacted by new threat and that could bring down something in our environment or we could
miss something um anyways that's my business brain going off in that too any other questions if not oop one more what tooling do you use to track your workflow through all of these steps could you get a little bit closer what tooling do you use to track your workflow through all of these steps like when you're going through all of them how do you make the notes and what do you use for that okay um so tooling um I have the very fortunate privilege of digging through EDR data of customers so I have like MD I have Microsoft Defender like crowdstrike or carbon black response which great EDR Telemetry what is EDR telemetry it's kind of glorified event logs in
cismon I don't know if I can say that um but I did and if you have that data use that and then leverage network data and use IP addresses your DNS requests all that sort of information like and try to get that in one place I've been in the shops where I had five different tools to do all those things you make it work um highly recommend that um tools I have a hard time saying like this tool is the best tool where that tool is the better tool once worse just know your tools and then in terms of like the methodology of like take notes take notes about what you saw take notes about what you observed what your
thought process was as you looked at that thing and why you thought that thing and take screenshots if you can um because you're not gonna remember everything you did and you're going to come back 10 minutes later and you're going to have 20 different tabs and you'll be like why do I have those tabs uh take notes um and figure out what works for you too like that's a really big thing like understand how your brain works and the way that you investigate because like I might investigate something different than you investigate something but we're going to come to the same conclusion and if we can both talk through how we did that and how we reach
that conclusion that's really valuable too because I can learn from you you can learn from me sorry I like ranted on that but yeah just whatever tool you have is a good tool learn to use it
so I have a question kind of based off of that I think there's a lot of you you went over a lot of like very keeping track of a lot of data as you're investigating things but I don't necessarily think it all of that has to be communicated like say you're doing a write-up on them alert or you're saying hey this I decided this wasn't bad or I decided this was how do you decide what data is actually important enough to be included in that and what to pass on and what to just accept and say we trust that we looked at this that's a really good question the question was we looked at a lot of data and we had a
lot of data points not all that is necessarily relevant to escalate to somebody who might be remediating if it's not you and how do you determine what's relevant to escalate so somebody can make informed decisions that's a really good question and it comes with practice but for me the things that I would have escalated there in this situation are that Excel file that dll that we had our Command lines and then the fact that we had an additional process tree that spun out of that but not necessary and if I had the hashes that piece and then I would say that's a fairly good piece of information for a tier two person to then go take action on an
endpoint um also like know what endpoint you're dealing with in your user we didn't have that in our alert bad alert we want that data um but that information is important as well um and being able to distill that comes with practice it's more of an art than a science in my opinion because it's easy to overshare but it's easy to undershare when you're communicating that type of information um I typically are on the side of overshare personally and it it's just what I do um but again depends the person
any other questions if not thank you so much I really appreciate the engagement [Applause] and the blog should be live in like 10 minutes now look at that so go steal the notes use it um thank you again so much you all were wonderful
[Music] thank you appreciate it we'll work on we'll work on merch later
[Music]
foreign
foreign
[Music]
[Music] thank you [Music]
thank you [Music]
[Music]
foreign [Music]
[Music]
[Music]
foreign [Music]
[Music]
[Music] foreign
[Music] foreign [Music]
[Music] foreign [Music]
thank you
[Music]
foreign [Music]
good morning I'd like to welcome you all to b-sides uh this specifically is the ground floor track so uh today I have the honor of introducing Uriel who'll be going ahead and talking about malware analysis the red team Edition uh before we do that of course I have a couple of those little announcements to give real quick so first we'd like to start with are thanking our sponsors who are responsible for making all this happen without them we literally couldn't be here doing what we do uh we'd like to especially thank our Diamond sponsors LastPass and Palo Alto networks and our gold sponsors uh Amazon Flex track and Google uh it's their support along with our
other sponsors donors and volunteers that make this event possible uh these talks are being streamed live and they are being recorded and will be posted on YouTube so there is no need for you to take photos or videos of the presentation uh we ask that you try to silence your cell phones keep them in your pockets uh as a reminder our policy as photographs is that you if you do take a photograph you should have permission for anybody who's in the shot we're very sensitive to being photographed uh some uh presenters into our members of the community prefer to stay Anonymous so please remember that as you're taking photos around the conference uh also I'd like to remind
you if we're going to do question and answers afterwards please step up and use the microphone it's not so much for Us in the room or for the presenter but it's being recorded So that those who watch it later can hear your question clearly as it's being asked so all right uh with that I'd like to go ahead and turn it over and say there it is uh thank you so much for being here thank you very much for introducing me so first of all cheers give yourself a round of applause please oh oh come on come on open up so uh first of all cheers foreign
thank you for presenting me uh first words of appreciation uh before we going to start with our presentation I want to thank uh first of all I want to thank God and my mom of course because if not for them I will not be here I want to thank besides for making this amazing uh my conference and of course thank you amazing audience thank you for being here so let's get started so I like my talks and my presentation be less about talks and more about doing and more about practical stuff like you know we'll do a couple of demos um so I'll first introduce what I will show you in the demo so you'll have some
interest the first demo will be about the dark side run somewhere and how you can as a red team learn from it and basically how you can take the same Concepts and kind of implement them in your own malware you develop and another demo which I'll introduce how to implement some of the bypass technique antivirus bypass techniques in your own malware in order to basically evade antivirus and EDR software um and my goal in this presentation this talk is basically for you to open up your mind to be fluid not to be rigid and basically help security grow and become better um so basically let's get started so the stock is not only for retimers
it's for everyone but the the aim is specifically more about red teamers uh but again anyone can learn from it take the concepts take the tools and uh you know implement it so I want you to learn malware analysis now what the heck a red teamer want to know about malware analysis like why any red Timber has its own tools his own C2 his own thing um and eventually you know most of the people that are red teamers are uh you know more about infrastructure penetration testing or uh you know uh active directory and stuff like this but in the end of the day most of the adversaries or the attackers will go for your data
okay they will write around somewhere they will write some info Steelers they will use mostly malware so if you want to become a stronger Redeemer and a better security researcher you need to learn malware analysis and not only for the sake of malware analysis but for the sake of offensive security so a little bit about me my name is I'm the book author of antivirus bypass techniques basically the book presents you 10 practical techniques which you can Implement in your own tools and will help you to bypass basically any antivirus any EDR and of course this book also provides some security recommendations that you can use in order to develop better antivirus and do better detection
engineering and stuff like this I'm also the founder of malware analysisco which is basically a website that gives you all the necessary information you need for malware analysis like tools cheat sheets stuff like this um you also have my courses there uh provided of course like basic malware analysis Advanced reverse engineering and other stuff like malware development basically I like to do model research and other security related topics I'm also the red team Tech leader of the biggest company uh like the biggest beer company in the world called ABN Bev uh just a small disclaimer I'm not talking in behalf of a b and web talking bear for myself but just for you to know
um also YouTuber blogger lecturer uh like one of my biggest passions actually is to uh teach students and give them the knowledge and and given tools to become better Security Professionals and also uh to enter this field so I'm doing this for something like 10 years in Israel around the world also and it's a great honor for me also to stand here so before the show begins what the heck is malware analysis I like to Define things very shortly without all the Wikipedia Mambo drama stuff you know so malware analysis basically is the art of analyzing and research of malicious software behavior and patterns and by researching malicious behaviors and patterns you can learn a lot from
them for the sake of blue teaming like detecting malware extracting indicators of compromise understanding the ttps and all the stuff but also as a rhythmere you can learn a lot from those malware from their techniques from their bypass stuff uh how they obfuscate their strings how they use their malware how they how they actually literally move in your network so it's analyzing malware is like getting into the head of the outer of this malware it's like reading a book you know for me it's like reading a book I think it's even better than Mozart yeah most of people don't really like to reverse engineer assembly and single this mumbo jumbo stuff but for me it's actually an art
so the levels of malware analysis so the basic level is you can basically take a malware sample if it's an executable file Powershell script whatever you can throw it inside a Sandbox which is basically a virtualized environment with all the necessary tools um to analyze the malware and basically you have things like any run or hybrid analysis which are basically you can take a file throw it into either the sandbox and it will give you uh like the overview of what the malware actually and how is executed and all the stuff of course if there is malware with anti-send books and other evasion techniques most likely it will not be executed on the sandbox so it's kind of
limited but it can give you a good overview of what the malware does and also helping you to extract indicators of compromise the next level is called Static analysis is basically taking a file uh extracting strings understand the the building blocks of the file before you actually execute it on your sandbox environment on your virtualized machine okay we have a dynamic analysis which is basically to execute the malware on your virtual machine to see how it behaves use tools like procmon cross Explorer API monitor understand what kind of Windows API functions are executed and all this kind of stuff and of course the most amazing stuff that most people are I think Friday know is reverse engineering it's like God
damn get get into the assembly code or whatever it is and understand how the malware is actually work so it's kind of interesting for example if your malware opens a socket and interact with us with our you know with an external server or C2 how it actually do it okay what kind of Windows API function chooses how can I use it for detection engineering how can I re-implement it in my own malware in my own ratings in my own attacks you know for example whenever you want to interact over HTTP with some external server you can use various Windows API functions you can use this socket you can use uh internet open URL and you have all kind of a
bunch of other APF functions okay so this only a small fraction a small example for you to understand what you can do with the power and knowledge of malware analysis so we talked about microanalysis blah blah blah what it is what is the level of this kind of stuff so first of all cheers ah amazing so what is relatively you know um you will you will ask someone and they'll say you oh it's the it's the knowledge of using or text or or you the use of tools like Metasploit and Cobalt strike and do that and do this and but like what is the essence what is the goal of retrieving like actually kind of stuff first of all right timing
is not about achieving a DA domain admin and you know what let me tell you another thing most of the attackers and most of the malware doesn't even care about getting a domain admin you know why because most of the networks today are very very hard to manage their permissions and their security it's very hard to be a blue Timber it's very hard to like literally Implement security controls in the right way so even a ransomware today can encrypt your document files without having local admin or even a domain admin so what the hell it needs the domain admin so most of times malware or attackers will go for domain admins if it's really necessary for them like
to literally move between forests through other networks or just for the fun of it yeah so red team is about simulating real world threats now what the heck is a real world trade uh doesn't running crack map exec with Mets Floyd with all the Cobalt strike Mumbo job stuff is actually rating um it's the tools it's not the essence of it so the purpose of writing is to provide a real world picture of business related threats like for example if you're a pharmacy business maybe your adversary will not be County or revealed um it will be maybe another Pharmacy company I don't know uh maybe another competitor maybe some Insider threat I don't um and basically you need to Define as a
retimer what is an actual threat to your company that you're trying to rate him [Applause] another purpose is to act like the adversary or the enemy based on accurate trait intelligence an act of trade actors targeting your business so the use of threat intelligence will help you to become more aware of what kind of threat actors and ttps you actually need to simulate in order to give the real picture of the security posture of the company simulate potential threats actors ttps tactics techniques and procedures okay it's like understanding the tactics uh the techniques they're using maybe that the threat actors using mimikats to dump Elsa's maybe they created their own version of mimikats with different API
functions which most of videos doesn't really uh even detect them I don't know um and based on this data like the ttps you can actually go and do your stuff for example if you have threat actors like Conti which they basically a ransomware group I hate them of course they attack a couple of my clients also but you know the world is hard um but if you understand that Conte is actually targeting your business and you understand your ttps you can go and simulate the threats on the company you're attacking of course in a secure manner so always take opsec into your mind when you're doing writing [Applause] and eventually the real goal of red
teams is not to go to Blue Team and say yeah your your you failed or be childish I I win you and and I overcome you and all this kind of you know the real purpose of writing is to help organizations become better that's it and of course helping The Blue Team become better so if you're as a retemer understand the craft the techniques the tools that The Blue Team uses and also if the blue team understands how the rating is operates they can basically help each other so retimer must have an adversarial mindset this is for sure you cannot be a red teamer if you don't know how to think look like your enemy like a
criminal okay like a bad guy but mama always said like told us to stay away from bed guys you know oh so sweet man I like my mom and she has amazing tips for life and she's even right but not in the case already you actually need to think like your enemy and it's kind of hard um like psychologically not only technically so uh to know your enemy you must become your enemy I know it's kind of uh hard to do it but uh if your skills of mimicking others and and to be like a uh to mimic others basically and to understand their psychology it will be much easier for you to do this
now behold the real power of redeeming Cobalt strike no no don't take it seriously like Cobalt strike ball strike is an amazing tool you know it's an amazing tool it has a GUI it has a lot of features bypass evasion modules it does amazing job okay so uh take it in the right perspective but mostly when you talk to Red teamers they are about the tools like writing cobot strike my man you're not ready without Cobalt strike you need to have cobal strike ah I don't agree I don't agree it's just one of the best tools out there I think but it's not what actually defines a redeemer so why not doing things like this
now in the left you have comments right in the right you have either probe uh first of all I like to smoke when I doing reversing kind of cool and you know opens your mind um what for me represents the right picture like Ida is like go and understand your own tools go and understand your own techniques go and understand other adversaries techniques and tools and malware go and research stuff like someone asked me like uh five years ago you know what it's mini cats and they're like uh I I think so Americans and it's like what this and that comment does and it's like whatever dumps the Elsa's memory okay it's like yeah and I asked him you know
what how do you actually understand how it works you know were you asking someone what is something do you know about something so what is the actual level of understanding of this question like for example mimikats what is the level of understanding of this tool how do you actually know how it operates how it acts what kind of functions it uses in what languages it was written and compiled do you or just go grab the tools execute them and you're ready which to be honest most of the time it's working it's like not maybe mimikas but other tools you know but in order for you to become a better retimer or a stronger security professional
go and try to understand how they still operate like go do some wire shark like understand how the network packets are going on like understand how the tool you are using for example mimikats to dump Elsas what's actually doing like opening a handle to the else's process enumerating the pids of the processes receiving get a handle with open with the open process Windows API function use the use of the mini dump function to actually dump the LSS and after you actually understand this functionality you can think maybe there's other functions or Windows API functions that I can use which most likely most ddrs and antiverse will not detect it so it will open up for you a greater
world now don't get me wrong both malware analysis and redeeming are not about the tools One is using but the ability to research and understand Technical and Abstract Concepts eventually so I think you kind of understand why marijuana's need to concern retimers okay but one of the reasons is because the bad guys do they go and learn from each other's malware from each other tools from each other techniques and they actually go and re-implement the same techniques the same tools or the same concept basically in their own malware for example if you heard about the log B3 ransomware also did a video about them um they literally got the code from Dark Side ransomware copied some Snippets
from it for the decryption run the decryption mechanisms and for dynamic API result which we'll talk about it and then literally copy and paste it the same technique the same concept in the wrong malware it's like what the hell I'm like researching the lobby Transformer and it's like 50 percent uh like minimum to 50 uh the code of dark side so if the bad guys go and learn from other malware and other tools why not the red teamers the good guys so trade actors evolve by learning and leveraging the craft and ttps by researching now resembles in the wild this is one of their secrets but uh not anymore I hope this talk will actually open your mind and give you the
necessary tools and necessary mindset to become a better security professional so let's talk about uh quickly about the malware development life cycle like the mdlc I call it it's like sdlc for software development lifecycle but for malware which is kind of the same so you have malware development you developer malware you know the malware need to check in which kind of environment it actually resides whether it's Linux Windows what kind of version what is the MAC address the IP address what so-called information gathering or situational awareness so the malware need to be aware of where is actually it and then based on this information which most likely be sent to an external C2 server the malware received the configuration
or the next stage of execution so part of malware development like any other software development is to test and do QA like go and develop your malware in a secure manner of course for good purposes uh and not for bad purposes don't be a black cat be a whitehead you know um it's kind of cool because for example if you want to learn C and assembly and doing this via developing malware it's much cooler than developing uh the Checkmate or snake game this this my my opinion okay um then you want to implement some malware defense bypass techniques inside your malware like most likely if your malware will drop in one of the computers or networks that
you're targeting there will be some endpoint protections that you want to buy because like DLP edrs antivirus stuff like this then you want to actually do some offline AV ndr testing before you're actually Distributing the malware and then it's one of the things that I want to talk about today is ioc collection and removal like it's it's fine that you actually develop your own malware you use C or python or partial whatever language that you want but f after you compile your malware like with Visual Studio or with C plus or with you know GCC or whatever compiler is out there how the file is actually look like for the EDR for the antiverse for the
malware analyst for the researcher does the strings that you actually used or the variable names or the function names or any other IP addresses that you use in your own malware they actually presented in your malware like in a clear text you need to obfuscate them uh maybe one compiler compiled it in a one-way and another compile compiler compiled your malware in a different way which will be much harder for malware analysts and edrs to detect so if you compile for example C written malware in Visual Studio it will be basically a different output if you um compile it in for example GCC compiler okay so it's basically a research topic for you if you want to kind of test write a
malware try to actually compile it with different compilers and see the difference do some diffs strings op code differentiation uh see how actually the compiled versions are differs from each other okay and then for example if you have some malicious or suspicious ioc presented in your compiled malware you want to kind of like somehow remove it obfuscate encrypt it or whatever and of course uh as a white white attacker or whatever you want to use the same uh like techniques and iocs used there and like work together with the threat intelligence and like try to understand uh how they are actually relevant to other malware in the wild or other thread groups basically you do not have to develop
your malware from zero and this is one of the things that basically I want to say here you can learn from Real World malware samples and incidents for example you can go for the colonial pipeline incident if you remember uh the same incident that actually broke half of the gas stations uh in the United States it was actually the dark side ransomware like a small piece of executable Disturbed and and crashed half of the gas stations in the United States something like this not really exactly on the on the numbers but something like this so what one executable file one ransomware can do to your state not only for a company think about it and how much can be
learned from the this ransomware and I also want to dive with you into this ransomware like I don't have all the time in the world but we'll do something simple there so let's talk about darks that run somewhere second eyed runtime code decryption and dynamic API result now don't get creepy on me please it it's going to be kind of a deep technical stuff like assembly stuff um I'll try to explain it in a like in a simple fashion um and let's begin so let me just start my virtual machine here so what do we have here we just do okay can you say something here like like this maybe yeah so we have the dark side sample yeah yeah
the same the same malware that Disturbed half of the United States um uh gas stations so if I open this file inside PE Bearer for example I want to see the sections and imports like the actual sections you have in the executable files and the imported functions now I just remember uh analyzing malware is like reading a book and each one each book has its own sections like you know it has a title it has the data in the title and so on so you can see here we have two text sections now most of normal files will have one text section the text section actually contains the execution code like the machine code the
assembly code that is need to be actually executed on your CPU on your operating system and then you have another section called text one you know kind of weird now in most files even most malware if you go to the Imports you will see much larger dll entries and for each deal as you see at least five to ten imported functions and you can see where I'm going you have only one dll character 32 dll one of the most basic and mostly used DLS in Windows and have only one function called exit process like what the hell like a normal malware or normal file will uh will include more dlls like socket and you know create file create
process you know like basic stuff and here you have only one function what the hell so if you open up the if you open up the malware inside Ida Pro and go again to Imports you will see exit plus also only present here and basically this is the starting point from where the malware actually begins now this is not the actual function names okay so this is the function names that I call okay so each call here in assembly is basically calling to a function so for example you define a function in C like for example void blah blah and it has two parameters so this what we have so you have here basically two parameters which are basically
encrypted blob you're basically an encrypted Mumble jumbo stuff here and this cold Mamo jumbo stuff will do something then do another something and eventually go and decrypt those encrypted blobs now if I'll do space here I'll say that the actual code here is under the text one section that we saw that which is not very normal and eventually if I go here to like for this call where she show begins where it's actually where the Run Source start to encrypt your computer if you go here it's like gibberish nothing the hell so if I double click you can see here we have the text section so it's kind of occurs to me that The decrypted Blob will be
uh placed in the actual text section so this stuff will be eventually decrypted and be an actual code to analyze so we have uh basically we have to execute the ransomware on my virtual machine so be careful with it um let's begin so basically I go and put a breakpoint here and start the local Windows debugger whoa okay so now it got to the break point so now the actual code is here so we did the whatever stuff it does here then we do f8 to actually execute the function now just for understand what what kind of decryption it uses here if you go double click on this function it calls this this function which I call this decryption
Festival which actually go and call another function and eventually to look soar blah blah so it will basically store the values and in that way decrypt the content okay so let's do f8 I hate this computer okay and another f8 and now we'll need to see something like different here so before I actually go inside this code let me just undefine it in Ida and re-analyze it again because for either it kind it's kind of like uh hard to understand what is going on then so if I select here the code like this I go and do uh like I don't like selected area or c analyze and I hope it will give something sensible
okay here we have some things here let's do here see where future begins amazing do another scene we kind of need to help either to reanalyze the code basically so now I'll go in F7 which is basically go into the function um let me just also Define it as a function like with the p now I defined it as a function and like if now I can do space and I will see everything in a fashionally mannered uh you know execution flow okay so basically I either after the decryption has occurred I I undefined the data that I just kind of like got hard time to analyze reanalyze it again and using the P button in the keyboard basically I
Define it as a function for either to understand that is what is the actually building blocks cheers Okay so eventually it will go and call here two functions this first function the second one so if I go to this function it will do some weird things which will basically eventually will resolve the apis or the functions uh of the of the ransomware so suddenly you will have some functions appear there so let's try to do f8 here called the first function as you can see it it resolved like a lot of dlls and now we've got to actually resolve the functions also so then I do another f8 and now you can see a lot of coal blah
blah something called blob or something the word whatever it's actually a sign that is calling to the dynamically resolved IAT the import address table the actual functions of the DLS which are resolved in runtime okay so if I go into this call or maybe to another call like let you I hate this computer and if I do now for example I'll do this call I'll do F7 it will go actually and call shell32 command line argument so this function didn't appear in the executable file how they actually now appear so the two functions that I presented you here they're actually doing what so-called Dynamic API result now if you're as a red teamer go and re-implement the same
technique or the same concept in a different technique you can evade most of the static engines in most of the idios and antiverse and it's kind of easy to do it yeah so for the sake of uh for our time I'll go back to my slides I actually have a run somewhere running now on my virtual machine and I don't give a hell about it okay okay so as we saw no functions in the import address table we can see that the sections are packed or encrypted okay so you see there is a high entropy most likely if there is higher uh value than 7 it will be most likely encrypted and coded or obfuscated
so first before it goes and to actually resolving the apis it will do the decryption so where did where I showed you the two decryption routines with the source stuff this is where actually the store is made and the text section is actually decrypted and the dynamic API resolve basically it takes a bold functions of load Library which literally loads a dll file in runtime dynamically and then it uses the get proc address the get Block address function is like the the broad like load library and get processes like the brother and sister of uh resolving apis or loading functions if they get proc address function will actually import the specific functions from the dlls
and voila you have the result functions in runtime yep there was some confusion uh as far as double booking the room uh if you are looking for the 12 o'clock reverse engineering a Dos PC FM uh V game from 1994 by Andrew it's now down in The Proving Ground I'm sorry to have interrupted thank you it's fine thank you so we talked about or I showed you actually how it looks like reversing a malware or it's not always like this uh if if the for example malware like agent Tesla or the yashma ransomware which I also have a YouTube video on it you'll see that the malware is actually compiled with c-sharp like dotnet C
sharp I hate oop languages I hate I hate them but they're much easier to uh analyze because you don't need to actually analyze assembly because dopet c-sharp is a high level language uh it's called IL assembly and if you put this executable this compiled executable into tools like d and spy it will literally decompile the code you literally will see the source code of the malware [Applause] um in c plus c or C plus plus compile malware you will not see the actual source code you will see the assembly so either will do for you at this assembly not decompile so it's a different process so now let's do some offensive stuff uh present you two bypass techniques the
first one is called rename obfuscation and the second one is called memory bombing which is my favorite um again they're referred also in my book antivirus vipest techniques I'll be more the Glad if you will read this book so let's take a simple reverse show like the most basic stuff that red teamers and pentas are like reverse shell so you have a C plus plus or CC plus basically code here which defines a main function it receives two to three arguments uh you define the IP that you want to call back or to communicate with and the port of the C2 malicious whatever server okay then it actually calls the function with the host and
port and you can kind of get what what I'm getting it like if you have a function named execute shell in your code what is the probability that antiverse and idiots will uh detect it you know think about it so the function of execute chair basically takes two arguments uh you know the IEP the port creates a socket blah blah blah blah stuff and you know defines the connection of ipv4 in this case AFI net and defines the receive of data the buffer itself like whenever the city server as an attacker you want to execute a comment it needs to send a comment and you need to receive the output in the server side yeah so this
is the thing so now we actually compiled this malware or reversial and now we'll start to look for iocs and again not only like you you literally switch your perspective to a blue teamer now we're acting like a blue teamer this is the real power like you're already writing your malware your whatever stuff you're doing then you put the Blue Team hat and you okay if I'm a blue team how do I detect it what can what kind of ioc's artifacts I can find and rely on in order to create my detections with the hour or whatever detection language it uses so if I do your strings on the executable and I'll use find Str in case sensitive on CMD
like if any stairs like the lame version of grep in Windows okay I hate Windows okay um and you have this CMD that something like where is the dot exit one of the reasons you don't see the dot exit it's because I used uh C plus plus which is a shitty language I like C but I hate C plus plus because it's oop and uh um and also to reverse Engineers C plus plus much harder um and basically um you don't see cmd.x because there is a null terminated string not an alternative like a slash n in each one of the strings for example if you have IP like 192 168.0.1 you'll have in the
strings one and two one six eight and then a new line zero and one what the C plus plus don't ask me okay so if you're writing your memory C plus plus it basically be better for you uh or easier for you to bypass rather than C because C compiled executables are much easier to read you literally see the strings the you know variable names or or whatever you know functions they're defined and it's much easier to understand um okay so what now so now I'll Implement some rename obfuscation so as you can see it's the same code um I changed the function from execute shell like a shell what the hell to run you know generic name run run what
then you have Holston Port which again it can be benign it doesn't have to be malicious you know um you're in in the code before you saw that actually uh music create process with the cmd.exe and it's like cmd.x81 line so I did a simple trick you know I basically defined two Char variables cm and dito Texas and then I concatenated them together using Str cat in in C or C plus whatever as a pointer variable of course and I passed this power pointer variable to create process Windows API create process function and it basically changes things in a compiled version then you validate again the ioc removal yeah where is the CMD dot something yeah
gone got away the second technique that I want to show you is memory bombing I I think that I basically the first that named this technique is memory bombing not something new but presented in my book and it's very it's very basic like the first technique used to bypass the static engine of antivirus and edrs this one is just for the dynamic engine like whenever you execute your malware you're executable the dynamic engine of the antivirus or the EDR will basically allocate a protected virtualized memory like virtual Outlook or something like this it will put the code inside this emulated uh environment or memory in your computer and it will try to execute it now as we all know antivirus and edrs
has one problem basically two problems the fear of false positives yeah and the limitation of resources in your computer because think about it antiverse and leaders are residing the same memory in the same operating system in the same computer so of course they have some limitations now think about it if I allocate enough large member like eight megabytes 20 megas whatever so that diverse like looks in your mouth and said ah man it's too big for me move on and it lit it will let your malware execute because if there is any chance that it will terminate a legitimate process or a legitimate executable big problem and this is the same thing that I'm actually exploiting here the fact that
any antivirus in either has some resource limitation and I'm allocating a big chunk of memory for him to choke out and later malware executed and now for the detection test results
before the obfuscation and memory bombing is made 35 detections after the obfuscation and memory bong has occurred halves down F down like more than 50 it's a lot now think about it if I'll do things a little bit differently and I'll Implement maybe even one or more bypass techniques I will create a fully undetectable malware zero detections yeah this is the reality we're living at this is why you need to become a better security expert this is why you need to understand both sides red team blue team whatever color you want to purple team yellow all these colors it's not about the colors it's about the way you're working the way you're thinking the way
you're approaching stuff so for you know for the sake of ending this uh presentation you know some important tips for success always understand your tools and malware go as deep as possible understand iocs generated in your compile malware the network packets the actually used functions everything that you can go as deep as possible you will be surprised as what you will learn the most learning experience I had in my all offensive blah blah career it's true doing the bloating stuff into the response forensics around us like seeing the other side of things you know what I really appreciate blue timmers because it's a hell of a hard thing to do and always the complaints are against
The Blue Team not directly it's kind of sad but learn our analyst to understand think like a blue teamer and again marunas is not only for Bluetooth it's for red teamers the bed is like any other sdlc process so write your malware in a good military there which I exploited dll hijacking like if I can exploit the same malware the autonomics kilogram to be to resigning your computer so it's kind of you know so if it's very important for you for like opsec uh operational security and everything securely code security code your malware yeah I know it's kind of funny to say to that but yeah be curious passionate and Innovative I don't need to say this
and take some breaks in between don't work like a like a juggernaut take some breaks in between cooperation cooperate don't be a jackass yeah if you want to do a real adversary simulation adversary emulation rating call it whatever you want you need to cooperate with teams like threat intelligence you need to or do the 30 damage for yourself and understand what is the actual value you provide your company based on the actual threats posed on your company and eventually take your report that research simulation you did or the retaining work with the blue team and gain a better security and monitoring again think like fluid water not like a rigid Rock taking operate like a criminal
help security grow and become better if you want to have some more resources some more fun more learning experience you can go into malware analysis code you don't have to take my courses do whatever you want you can learn for yourself but you can use the resources the cheat sheets everything is most of the things are for free basically um my webs is like an index you have everything there whatever you need uh if you have some suggestions so uh things that you think you it I may add to to the website feel free to contact me in my Twitter whatever yeah that's it thank you very much
thank you Monday yes yeah questions questions [Music] cheers uh in terms of uh sandbox analysis uh now that cuckoo's kind of been not updated in a very long time I saw on your site you have a bunch of sandboxes listed but uh mostly could it be close to the microphone yeah okay testing amazing all right on your website uh so you listed a bunch of sandboxes um I used to use cuckoo when I used to do this kind of stuff yeah um I haven't updated that in Forever The Sandbox is you listed I see a lot of them are like paid Solutions is there anything you can still run locally similar to how Kuku used to be where you could step into a
process and interactively you know like make a change like click on a link or something or a button and then go back and get your analysis at the end yeah that's not like uh running the cloud because if I'm dealing with very sensitive data I don't necessarily trust the company that says they'll keep it private I want to keep everything local okay thank you for the question so basically asked about uh you know uh plausible and workable sandbox out there so basically if you go to my website you'll see there's a bunch of uh you know sandbox that you can use maybe there's other symptoms that I'm not aware of you can you basically have two options
you can Implement your own automating sandbox in uh using cuckoo sandbox which is free open source I actually use it in my Army service where I did the research and security stuff uh and it's kind of helped me you know to gain the first visibility and understanding of the malware it actually um also communicates with your virus total and brings you all the detections all this kind of stuff so if you want to do something locally use a very strong server with you know SSD storage and whatever uh you know asean CPU everything um at least 32 gigs of RAM because it takes a lot of resources basically you can Implement cuckoo sandbox it will do
amazing job but take into account there's a lot of troubleshooting and a lot of problems that you need to face like python and Linux and yada a lot of yeah if you want to use a cloud-based sandbox you can use any run which is amazing uh for runners to this Russian base or something I I don't know from what I heard take care good luck don't put any personal information in your malware or something um uh you can use uh you know hybrid analysis um and you know other other Solutions just go to bangladesco you have their uh sandbox links you can use all of them another questions thank you your amazing audience thank you very
much may God bless you all thank you thanks
foreign
[Music]
foreign
[Music]
[Music] thank you [Music]
thank you [Music]
foreign [Music]
[Music]
[Music]
foreign [Music]
[Music]
foreign [Music] foreign [Music]
[Music]
[Music]
foreign
[Music] foreign
[Music]
foreign
foreign
[Music]
[Music]
foreign
[Music]
foreign [Music]
[Music] foreign [Music]
[Music] foreign [Music]
[Music]
thank you [Music]
[Music] foreign
[Music]
[Music] foreign
[Music]
foreign [Music]
foreign
[Music]
[Music] foreign
[Music]
[Music]
[Music] foreign
[Music]
foreign
foreign
[Music]
foreign
foreign
foreign [Music]
[Music]
foreign
[Music]
[Music] thank you
[Music]
foreign [Music]
[Music]
foreign
thank you
[Music]
foreign [Music] foreign [Music]
[Music] thank you
[Music]
foreign [Music]
[Music]
[Music]
foreign [Music]
[Music]
foreign [Music] foreign [Music]
[Music]
foreign
foreign
[Music]
foreign
foreign
[Music]
[Music]
foreign
[Music]
foreign [Music]
[Music] foreign [Music]
[Music] foreign [Music]
[Music]
[Music] foreign
[Music]
[Music] foreign
[Music]
[Music]
foreign
[Music] foreign [Music]
[Music]
foreign [Music]
[Music]
foreign [Music] [Music]
[Music]
foreign
[Music]
[Music]
foreign
[Music]
foreign foreign
[Music] thank you [Music]
foreign
[Music]
[Music]
foreign
[Music]
[Music]
foreign
[Music]
foreign
foreign
foreign [Music] [Music] thank you
[Music] thank you
[Music]
foreign [Music]
[Music]
[Music]
foreign [Music]
[Music]
foreign [Music]
thank you
[Music]
foreign
[Music] foreign
[Music]
foreign
[Music]
foreign [Music]
[Music]
foreign
[Music]
foreign [Music]
[Music] foreign [Music]
[Music]
foreign
[Music]
thank you [Music]
[Music]
foreign
[Music]
[Music] foreign
[Music]
[Music] foreign [Music]
[Music]
foreign [Music]
[Music]
foreign
[Music]
[Music]
foreign [Music]
[Music]
foreign foreign
[Music]
foreign
foreign
foreign
[Music] foreign [Music] thank you
[Music] foreign [Music]
[Music]
foreign
[Music]
foreign
foreign
[Music]
[Music] thank you [Music]
thank you [Music]
foreign [Music]
[Music]
[Music]
foreign [Music]
[Music]
[Music]
foreign
[Music] foreign [Music]
[Music]
foreign
foreign
[Music]
[Music]
foreign
foreign
[Music]
[Music]
foreign
[Music]
foreign [Music]
[Music] thank you [Music] foreign foreign
[Music]
thank you [Music]
[Music]
foreign
[Music]
[Music]
thank you [Music]
foreign [Music]
foreign
[Music]
[Music] foreign
[Music]
[Music]
[Music] foreign
[Music]
foreign [Music]
foreign
[Music]
[Music]
thank you foreign
[Music]
thank you foreign [Music]
[Music]
thank you
[Music] foreign [Music]
[Music]
foreign
[Music]
foreign
[Music]
[Music] thank you [Music]
thank you [Music]
[Music] thank you
[Music]
[Music]
foreign [Music]
[Music]
[Music]
thank you [Music] foreign [Music]
[Music]
foreign
foreign
[Music]
[Music]
foreign
foreign
[Music]
[Music]
foreign
[Music]
foreign [Music]
[Music] foreign [Music]
[Music] foreign [Music]
[Music]
thank you [Music]
[Music] foreign
[Music]
[Music] foreign
[Music]
foreign [Music]
foreign
[Music]
[Music] foreign
[Music] [Music]
[Music] foreign
[Music]
foreign [Music]
foreign
[Music]
foreign
foreign
foreign
[Music]
[Music] foreign [Music]
foreign
[Music]
[Music] foreign
[Music]
[Music] foreign
foreign
[Music]
[Music] thank you [Music]
thank you [Music]
[Music]
foreign [Music]
[Music]
[Music]
[Music]
thank you
all right good afternoon everyone thank you for attending our presentation uh if you're looking for the right room you are looking at parsing the differential problem uh bunsim will be presenting today so uh thank you for attending I have a couple quick announcements before we start uh one is of course we'd like to thank our sponsors who make this possible uh without the help from sponsors such as LastPass and Palo Alto networks we wouldn't be able to do these along with other sponsors uh such as and vincim Flex track and even blue cat it's their support along with others who support and donate including the donors and volunteers who make this event possible with even their time so thank
you for being here and for supporting that uh just remember as a courtesy uh please keep your cell phones off or in like a buzz mode the entire presentation is being recorded there is no reason no reason to uh take pictures of the screen or take video it's all going to be online so there's no problem there it's all being broadcast up to YouTube and we'll be there for future purposes so there's no reason for you to have to worry about that uh also remember please do keep your masks on and above your nose it helps keeps everybody safe and uh one thing we would ask is if you do have questions towards the end of the
talk please use the microphone in the center of the room it's not so much for us we can hear you here it's because it is being recorded that helps us capture that question in the recording so that whoever watches the recording will be able to hear that question as well so please if you have a question towards the end come up and use the microphone in the middle of the room um uh also as a reminder uh if for any reason you do take a picture before or afterwards uh we do have a policy here at b-sides that says you explicitly need permission for anybody who might be in that photo right we're a little
sensitive towards uh capturing the uh photos or images of certain people who would choose to remain more Anonymous so and we do try to respect that so please try not to take any photos where you don't have explicit permission of uh anybody who might be captured within that so um all right with that I'd like to go ahead and give a round warm welcome to Boone Boone go ahead and uh take her away foreign today I'll be talking about passing differential problem so if you have heard about this um good if you have not then let me take you down some memory lane and a bit of Storytelling on why we should care about this and how it not just affects what we
do in cyber security but software Engineers as well because I myself am currently a software engineer so I'm from Singapore and in a day I'm a software engineer and at night I teach at a local University so I'm a part-time lecturer teaching Cloud systems and I play for ctfs.sg cctf team and that's my handle so the key takeaways for today which I'll be delivering is what is passing differential problems some of you might not know what it is right and how does this affect us like why should we care and how can we address it throughout the entire ecosystem so just imagine this that we have a single system that we built freshly like hey I have a super app idea let's build
something right and we all always start with this monolithic architecture everything's locked into one giant service and then we expose different paths and resources right so but as we progress we would think hey the system is getting larger and let's try to implement different subsystems into the ecosystem and that's how we grow right we don't recreate the thing and break it eventually we try to add new new features or new subsystems into the entire ecosystem as we grow we might introduce different systems with different languages so it could be at the very beginning I'm using Django as my primary language or system or framework and then I decided hey I'm I'm I want to be like Fang I want to use go
and then I'm like I want to go dangerous let's use node.js so these are some of the different ways of creating things and part of it is also the motivation to explore the path of least resistance to ghost systems now I work in a super app company right um it's a right healing app similar to Uber and Southeast Asia and what we do is we have a large system that's built in a single language but new businesses and new technologies also like new features are built in using different Technologies because the people they hire to do all this are comfortable and something else like hey um I'm going to build a note you build a car rental
system with node.js and once that proof of concept is done you realize that hey we have gotten knee deep into node.js we shouldn't be refactoring these things right but the main set of features and services they are written in an entirely different language with an entirely different pipeline so one of the motivations to expand with different languages is to explore the path of least resistance not just in a scope of technical work but also business decisions as well so what could go wrong if different systems have recurring variables right so for instance in system one is written in gold and system 2 Etc in Django what could go wrong right so let's take a look at some sample code
so this is something that could be written right for the top part let's go so you have a param that's you got it from this ul.url.query.get so this is the reading directly from the HTTP package and then this gets the value that's tacked to full and in Django you do the same thing as well in Django's way of doing it is to use request.get.get then you get that value of that variable looks the same right if let's say I pass it from the first is on the second system it should be reading the same thing correct but in fact it's not so the first one will read as John and the second one do it as mayor
now if this in mind what if my first layer of system is a check or Security check or Samsung of a firewall right and that got through my firewall and the back end Downstream systems that is reading the value of the variable there's a totally and high different framework and it's reading the for example an SQL injection payload so that bypasses the first layer of check and breaks Universal logic within the system so why does this happen so these two images they are from the official uh packages and Library documentations so as you can see in the first top part of this leg the gold HTTP package defines that using the get function it will return you
the very first the very first variable even though you have multiple currencies it will return the first one but in Django you'll return the second one it's a bit funny because when reading a source code it it kind of feel like it's quite messy but if you take these two together it makes sense so the get item the get function from Django implements the get underscore underscore get item which takes the last occurring variable in the request parameter so that is why when we have this gold will see it as John and Django will see as the second variable which is the escrow injection payload so this is what we call the passing differential problem this was first this
term was first used in the lens SEC language security landsat approach where they describe this as the different interpretation of messages or data streams by com components breaks any assumptions that components adhere to a shared specification so this is nothing new and I'll tell you why 10 years ago in the tangled web when Macau published the book in 2011 it was mentioned in briefly in the second or third um the third chapter about passing differentials and also Orange has mentioned this but in a different form of passing the passing between DNS and also home function where he made a variable about HD3 parameter pollution which stems towards the which stems from the passing differential problem and
also we more recently a gitlab block talking about how passing differential can be used to exploit a foul right or foul read from the RCV 20206833 so this is nothing new and it's been mentioned briefly in many cases and in fact there is and in fact there is available resources online so this is from this table is from payload of all things if some of you know that we bought three I thought that it should be there so I compiled different sources from the information from different sources and I made a comment to it and it it's there really and people have been contributing to it and it's good that we know what are we dealing with
because as a software engineer it's it's quite scary to know that my peers do not know that different systems handle different languages sorry different variables differently so if let's say you look dot net and uh Apache and PHP juice they handle it way differently from say node.js or even go or python Flash and Django so if you have this type of architecture you should look at this as well so that you can make sure that your system conforms to what you required to do
so unlike URL and URI right we have I if ietf IFC 29862 tell us how we should pass DNS as it is there isn't to this day any IFC is telling us how we should handle URL variables and especially in multiple occurrences and this makes it confusing for different Frameworks because let's say you are you are full stack developer working with different languages you'll be like hey what's happening you know and so most of the times it could be this right and this is just the tip of the iceberg it can also be observed in the entire body for example if you see the HTTP post body it can happen it should be haters it can happen and let's not forget that
HTTP haters can be cannot criticalized or uncannaconicalized and moreover I was reading some Django documentations yesterday because I was making dislikes Django would replace your hyphens in your headers with underscores to to match their haters and do a dictionary so I'm not I'm not saying that there's a security impact but it is a interesting implementation that they are doing this because if you see in the other web packages they could be just matching those by just making it lowercase so there is one example of why we should pay attention to what kind of Frameworks we are using so I've been saying about all these problems and how it affects us but what can we do
right of course number one is to be aware what we're using and uh not to assume uniformity across the entire stack and if we are if we have the time and we are committed enough we can create test coverages to include such cases to see that you know if there is any breakage of logic or we can just do it at the API Gateway level and just normalize the normalize the variables and pass it down to the downstream services now today I've talked about web itself right so passing differential permanent web Technologies is something that has been mentioned and spoken in small little places for the past decade what about the cloud so I think this is something that I
would like to leave you guys with as a pondering thing to think about because this is something I'm still working on the differentials within Cloud systems and how multi-cloud affects us and how we should be using it correctly so yeah um thank you any questions [Applause]
so we had a lot of foreign
[Music]
foreign
[Music]
foreign [Music] thank you [Music]
thank you [Music]
[Music] thank you
[Music]
[Music]
foreign [Music]
[Music]
[Music]
thank you [Music] foreign foreign [Music]
[Music]
foreign
foreign
[Music]
[Music]
foreign
foreign [Music]
[Music]
[Music]
foreign [Music] foreign [Music]
[Music] foreign [Music]
[Music]
[Music] foreign [Music]
foreign [Music]
[Music]
foreign [Music]
[Music]
foreign [Music]
foreign [Music]
[Music] foreign
[Music]
[Music] foreign
[Music]
[Music]
[Music]
foreign
[Music]
thank you [Music]
foreign
[Music]
foreign
foreign [Music]
foreign
[Music]
[Music] foreign [Music]
[Music]
thank you [Music]
[Music]
foreign [Music]
[Music]
[Music]
thank you
foreign
[Music]
thank you
[Music] foreign [Music]
[Music] thank you
[Music] thank you foreign [Music]
[Music]
[Music]
[Music] thank you
[Music]
foreign [Music] foreign [Music]
[Music]
foreign
foreign
[Music]
[Music]
foreign
foreign
[Music]
[Music]
[Music]
foreign
[Music] foreign [Music]
foreign
[Music]
[Music] thank you [Music]
[Music] foreign [Music]
[Music]
foreign
[Music]
[Music]
foreign
[Music]
foreign
[Music]
foreign
[Music]
foreign [Music]
[Music]
[Music] foreign [Music]
[Music]
[Music]
foreign
[Music]
foreign [Music] foreign
thank you [Music] thank you
[Music]
foreign
[Music]
foreign [Music]
[Music] foreign [Music]
[Music]
foreign
foreign
[Music]
foreign [Music] foreign [Music]
thank you
[Music]
foreign [Music]
[Music]
[Music]
foreign [Music]
[Music]
foreign [Music]
thank you [Music]
foreign
[Music] foreign
[Music]
foreign
foreign [Music]
[Music]
[Music]
foreign
[Music] foreign
[Music]
[Music] good afternoon everyone thank you for attending this afternoon's presentation uh blue will be presenting today on Tomb Raider automating data recovery and digital forensics so I'm the absolute honor in uh going ahead and starting this presentation of course what presentation would be complete without a few prior announcements such as going ahead and thanking you for being here uh we do like to do take a moment thank our sponsors whom with this would be absolutely impossible to be here with you and present in particular we would like to thank our Diamond sponsors which are LastPass in Palo Alto networks we do have a couple of gold sponsors too that we would like to go ahead and say thank
you to and that includes Amazon Intel and Google today uh it's their support along with other sponsors donors and volunteers that make this event possible so a quick reminder please do keep your masks up during the presentation we are trying to keep everyone safe so we appreciate your support uh cell phones uh go ahead and please make sure your cell phones are not only uh turned into like a vibrate mode but also please try and keep them in your pockets if possible we are recording this going ahead so we'll get we'll get it posted online uh there's really no need for you to go ahead and take photos or videos you'll have the ability to see
all this afterwards uh and just a reminder of blue uh of b-sides uh not Blues sorry b-sides policies we do request that you do not take photographs without explicit permission of everybody in the photo of course we are trying to be sensitive to those who are in unique situations and prefer not to be photographed so please try not to take photos uh here without explicit permission if everybody in the photograph so uh also as a reminder um well I was ahead of myself I got all of that all right so without further Ado I will go ahead and turn it over to blue blue thank you so much for being here and uh have a nice afternoon
thank you very much glad to be here man what what did you I knew I forgot something okay so um please if you go if you're going to go ahead and ask a question there is a microphone in the middle of the room it's not that we can't hear you it's because it's being recorded those who watch the recording afterwards will not be able to hear the question so we do request if you're going to ask a question uh towards the I guess the end of the you know q a session please come up and use the microphone in the middle of the room so we can capture that if we're blue it's always nice if you can
repeat the question when they say it just so that we go ahead and capture it as well so thank you yeah I'll repeat it no problem this is still good for Mike right you guys can hear me back of the room you guys can hear me too okay great uh cool without further further reduce I'm really glad to be here and uh thank you guys for all for coming so I'm gonna be talking about a tool I made today called Tomb Raider much like the game there will be a few jokes about that but overall I hope the naming of it becomes obvious soon enough uh automating data recovery and a little bit of digital
forensics as well so first I'm looking for introduction my name is blue yes like the color if I were green I would die this guy he gets it it's a difficult life pronouns or she her or whatever but uh yeah and that's the problem you get the bad Kermit we don't want to see that otherwise we're good to go so thank you all I like hard drives and data recovery that's the topic of today I started on my own broken hard drives I think many of us have these and uh we just had it laying around and I figured I'll figure out how to do this and uh work from there since then I've started doing all manner of other ones
the uh ones I could find at local hacker spaces in the dump and the trash anything really and uh yeah let's talk about how you do what I do so first we're going to talk about an overview of everything that this tool does we're going to talk about the steps of data recovery and a little bit of the background of that and then walk you through how we're going to automate those tools those steps and improve them with Tomb Raider so with Hardware prep you have a broken hard drive how do you go from that to an interface you can work with afterwards how do you image the drive we'll talk about what that means and how
to do it then how do you possibly recover a file system if you've lost it and if that doesn't work or if it does work how can you also get extra deleted files determine what a random piece of memory is something known as file carbon which I'll go into in a minute and then afterwards some stuff for Tomb Raider Tomb Raider really shines is uh post-processing so handling all of this data that you get from a given hard drive you might have terabytes of data from even a small like 500 gigabyte hard drive how do we deal with that how do we sort it out and actually find out what's hours what's not or if you're dealing
with a random dumpster Drive hard drive how do you figure out what's interesting and what's not how do you figure out what the actual user data and not just Microsoft system files additionally if you're looking for it you can also crypto Salvage and look for if there's any evidence of cryptocurrency on that Hardware so yeah first again why would Lou do this well there's lots and lots of reasons you are already here so I I've kind of succeeded congratulations you're now you're now trapped you you now have to learn about data recovery uh just kidding you can leave if you want but whatever point is I usually don't like asking why because I mean I mean I'm already here I'm
already trying to learn it I digress a lot of us don't want to pay tons of money just when we could do it ourselves we have a lot of broken hard drives I did I didn't want to have to Shell out 500 bucks for them to maybe not even get anything or especially if I have my own private data on there I don't want to just give it to someone and be like hey try and get this the only way to know if you've actually gotten it is to get your data so that is the Privacy concern there's lots of reasons you might also just want to explore like I've done where you find random hard drives that otherwise would
have been disposed entirely and uh figure out what was on them what is the story there additionally if you are interested in digital forensics I am happy to announce that you cannot do forensics if you don't have a hard drive with data on it if you can't get the data off of it how are you going to do forensics on it you can determine oh this hard drive got hit with a sledgehammer and that's about it but if you can't recover the actual data from it it's a bit of a bummer so I consider this a stepping stone additionally better programmer and a better hacker so first what you can do here's what you can do with data recovery
more than 90 of the problems with hard drives with minimal effort you can just solve them really easily from my experience with ones that are literally e-waste thrown away and I have no reason to believe that most of them were I have no reason to believe that they were function functional or rather I should say many of the hard drives I found I found little errors on them that were really easily fixed but I presume people threw them away because they didn't know how to fix them and now with this tool I was just automating it and going through it and finding all this stuff that people didn't realize was still there they thought it was just
oh it's not booting this is happening why is that there are of course the other 10 though five percent which are harder software problems or Hardware itself like if you did hit it with a sledgehammer you might have this problem where you damage the board on top of it or in case of even worse cases you really destroyed it in which case well I can't cover that in this call the other five percent as well unfortunately I only have so much time so ask me about it later or in the Q a if you like that other other five percent the five percent that we cannot really recover we've actually seen some death gun talks
about them if it's properly encrypted if it's completely overridden with zeros and or random data or just new files it's going to be gone if it's blown up and the platters are broken if it's melted into slag well we don't really yet have the ability to do that unless maybe you have a really serious microscope like and crazy resources and patience additionally some stuff with solid state drives I will cover that later if you guys want to ask questions and tell us about it um anyways intentionally destroyed drives this was a hard drive that I found in the San Francisco uh dump and um you'll notice a lot of things about this first off all the pins are really
weirdly bent and the ribbon cables on the left are cut the there's cables that are directly cut without much damage to the rest of it which is a little unusual you would expect If this just got dropped on the stairs it wouldn't really look like this and that's because it I have reason to believe this was intentionally destroyed um and they well they intentionally tried to destroy it quote unquote because in reality all I had to do was find another hard drive with the same model replace the board and then I can read off of it just fine because the underlying thing the underlying data storage device that had all the files was still there
it was a little beaten up they presumably fruit against the wall or something but it was working just fine and on that interesting things this hard drive contained data from the San Francisco water this guy who's on the San Francisco water Transit Authority we had some information on boat requirements and specifics for vessel voyages and all manner of things that had to be ready for additionally random stuff like this presentation which has wingdings in the title I'm not entirely sure and uh making fairies of viable Transit option this guy's article on feet of fairies which is a little bit of a weird name but I'm not going to judge so Steve here feels very strongly about
fairies he would choose fairies one day the day came down to it so good for him um other stuff you might find interesting account info first name last name home addresses phone numbers this is a massive address book it keeps going and these are just random names of somebody who presumably was an accountant or secretary or just a home just a normal person and they left this there thinking it would be gone it was not gone there are a scary amount of data on these things if you're interested in finding stuff like that I don't encourage using it for bad but if you're interested and you just like finding stuff like this like I do
get a shot uh oh yeah psychiatric records of this guy's medical history that was pretty rough that was quite a story I blurted out but yeah that's a thing on a brighter note also found copies of the room by Tommy Wiseau and other wonderful movies and films like that where you can just get for free because why not it's on someone's hard drive if you've already got stuff on your hard drive and someone else got it congratulations now they can get it using this tool you also find glitched images like this which can be uh which can have little stories of their own like I like the soccer guys who are like really upset about their image getting glitched out
and they're trying to argue it back I have no idea what happened with the AOL image on the left or the ducks or whatever that is curious things like this interesting and foreboding ones like this and the terrifying monstrosity on the top left that looks into your soul uh little Darla has a treat for you is what that one in the middle says which is let's continue to something less scary oh God no anyways this guy I have no idea where he came from but he was also on a hard drive his name is Craig and at first I was terrified of Craig because I was like I mean no offense Craig but look at him
um but he actually just has a little spatula and a little Skillet on the left there and I realized it's just making a little brekkie and he's misunderstood and now I love him anyways you can also find Dogecoin of Bitcoin and yeah whatever um slightly less valuable now sorry guys but I have found a few of those and made money but I'm not sure if that or Craig made me happier all this from these intentionally destroyed raps you may have one like this you may have one that's less damaged so with that said let's get back into how to do this how Tomb Raider can automate it how you can find your own Little Treasures how you can recover
your own personal data stuff that is important to you we'll get the brief description of each bed and how it works because I sure didn't know all these steps beforehand and then we'll cover how Tomb Raider automates it we'll cover how we can make that faster optimize things speed things up and walk through to do that I'm going to employ what I call the library analogy and to explain that first imagine you have a big Library this library is your storage device your hard drive it has all of your documents all your downloads all your pictures somewhere in it you have it organized in your own specific way using a catalog an index just like you might go to a library and
look up where is the fiction or non-fiction where are the books by Jules Verne or uh a ground pose poetry you have documents down those desktop pictures media music everything like that somewhere in your library of your hard drive this is your file system and without it it becomes uh really really hard to find stuff there's a lot of books in there so this is how we're going to represent it and with that said talk about deleting files no data cut recovery talk would be complete without deleting files right so Library storage device books the data and our catalog is our file system what happens when you actually delete a file in your file system
the normal way growing the recycle bin empty the recycle bin throw in the trash into the crash whatever whatever yada yada well unfortunately for those who really want it gone that is not going to actually delete anything that is going to remove it from the file system but not off of the hard drive itself it's never deleted it's only overwritten this is as if you remove something from your catalog from the system that stores it and lets you find it but not from the Shelf like that that never happens unless you really really try hard to get it to do that sometimes it's easier but by default that's not the behavior and default behavior is uh what most
people end up using people who I've read these drives from we're using specifically and if it's still there well let's just you know take the bookshelf and handle that later so let's get your device ready for Tomb Raider to run on I usually just plug it in to my desktop I'd recommend not plugging in random drives into your work desktop because that is basically just as bad as plugging in random flash drives into your desktop and we know that's a bad let's not do that uh I usually just use a normal motherboard connection on my desktop to do this you can use a lot of adapters however that work just fine on most computers and you may not be able to recognize it
in your file system itself but once you find the identifier for it which I describe on the GitHub page and at the end of this presentation you can see the link to that page where you can download the software and deploy it and everything it's all documented along with the software it's got usage instructions and even if your file system doesn't recognize it that's how you can find it it works best on Linux I did develop it on Linux so yes sorry maybe future development in the future it's so much easier in Linux I can just download all this stuff and there's only a handful of commands to set it up if I have the time I'd love to deploy it for
other os's but you can also just help me make code requests on the get a repository please anyways Imaging once we've got it plugged in once we've got it recognized using Tomb Raider and the instructions on the Repository we want to image the drive what is that well basically we take the entire Library why would we do that because it's pretty big and we don't really want to just have a big disk that we have to plug in all the time just to read it it might get damaged it might get like it might get damaged it might actually get worse over time the more we mess with it and that's no good at all we wanted to
reduce the same results every time we want to be able to copy it if we can so we don't lose anything else so Tomb Raider uses a tool called safe copy to incrementally get as much data possible from the drive it uses this incrementally makes a bunch of passes through it tries to get as much as possible even from areas that get damaged or corrupted safe job is a wonderful tool so thanks to that it's the only part that requires the physical Drive after this you can unplug it throw it in a blender maybe don't but you know you can anyways now that we've got the entire library in a file on our computer we want to try and get the catalog for
it file system recovery we want to find our way to recognize where our documents are to find out if we still have pictures what are the audio files it's pretty helpful to have our catalog we've likely spent a lot of time organizing it it's not explicitly necessary because the books are all still there we can still take the bookshelves but it's very very helpful and we want to find it if we can so we start with that it yeah metadata we use testdisk to get a set of these files yes this is another lovely tool we make use of a lot of them to get all of the files that it can from a variety of supported file systems and
uh recover those if at all possible because of course if we don't have to go through those bookshelves on our own those bookshelves on our own let's not because we'll see later that we'll see in just a second uh on that topic it's not too fun so try to test this afterwards we do file carbon this is if we did have to go through it manually this is if we don't have a file system and we did take the bookshelves if we do that we want to look through them for stuff that still looks like books we wanna discard the catalog entirely and say screw that I don't trust anything I'm just going to look through all of my
bookshelves even if it looks empty and see what I can find and that's what we call foul carving you are carving out unallocated space you're carving out of this empty space this randomized data Maybe actual meaningful data we use headers Footers there's a lot of different mechanisms to possibly get this data from it
super fun let's go through an example to show how fun it is so jpeg this is uh some of the markers for jpeg from Wikipedia I don't know if you can see this but it's okay if you can't there's just a lot of uh different indicators of uh hexadecimal encodings whatever stuff that represents the start of the image and the end of the image and here's an example image in this example image you can see all sorts of different jumbles of hexadecimal characters and uh if anyone can spot the start of the image in this on the right here let me know and raise your hand I'd love to see it as an exercise yo what's up
crap I I can't hear you okay wow they found it it's in the top left good eye um there's this random octet right here this random set that is the beginning of it obviously it starts with that and then there's another code the way later down on the list that goes right next to that and that's another indicator that this is a jpeg very helpful not very easy to spot we do have to go through all this list of checking for these things and then at the end of the image we have to look around for the code that indicates it's over because we not only have to know if we're carving something out where it starts with to know where
it ends we don't know how big the image is without this so if you look for a go that indicates that and you can find that right there now let's do that for every extension ever let's find Wikipedia Pages if they even exist for stuff like raw files and Dot AAA files and whatever else this is a list of like just the ones that start with A and just the ones that are the very beginning of this tool we'll cover in a minute that'd be great except it wouldn't suck so we instead make use of and use our own tools as well in combination with this to recover all extensions that are supported from that uncataloged space the space
without this tool is awesome it covers 480 file extensions and file families those are like audio music videos archives and as you might expect from somebody that's just looking through random data for anything that looks like important data it's going to make a lot of resulting data if I gave you a ton of specifications even if I gave you random noise if you are going to find stuff that looks like it there's likely that we will find something that looks like a JPEG even in random noise eventually and don't worry about the interface on the right it's uh this stuff is very thin small text let's just emphasize that there's a lot of extensions here
and that's the interface that photorec provides unfortunately a lot of the data that that's produced isn't always meaningful you get a lot you get false positives you get data that is Microsoft system files you get random log files corrupted data so that's not useful also your computers have a storage space sorry about that it this is a lot of data that's getting downloaded from OneDrive you are getting the disc image the file system and now these card files and that's a lot of that's a lot of storage space there are of course options in Tomb Raider so you don't have to worry about any storage space or as little as possible you don't have to make a disk
image you can just read from it directly if you're limited by space you can skip the carving entirely you can just get file system you can literally run it with everything disabled and it'll just say great well I don't know why you did that but great so let's condense everything let's make this a little bit smaller and let the raid finally begin we have our disk image we have our file system well we might not have our file system that's the one we might not have just because it might not be there and we have our carb files first thing we can do we can get rid of the disk image now we already got an image of it and if we
are low on storage space we are going to get rid of the disk image so that we don't have to worry about it because the disk image is the entire size of the drive if you have a terabyte drive and you only have 200 gigabytes on it it's going to be a one terabyte file if you have a library that's full of that's full of books in a library that's empty they're going to still be a library we don't want that so we get rid of it if we can't fit a library in our other Library I guess um and then we have our file system and card files that we want to combine in
summer better a tomb file system which we can raid get it yeah uh and then we can continue with a more efficient representation so we're gonna make a lot of upgrades and changes for this for making it more rateable we're going to start with flattening everything into one directory instead of having this massive tree structure we're going to represent each location of a file with its directories it'll end up with a big file name but this will also represent where it's from without us having to go through every single subfolder in every single folder and so on trying to figure out where something is you'll just get a big list of everything will also sanitize the file names remove
stuff that would break the program because Unicode is God it's a nightmare we'll create an index and uh that will contain the file name and the hash no the hash of the content sorry about that this should help make things a lot easier but why do we need to get a hash of the file content for this index well first a quick overview of hashing hashing is a one-way operation I have a hash function it's going to generate a unique identifier on the right so if I have box on the left is going to produce that weird string of digits on the right I hope you can see this in the back sorry if you can't I'll narrate
um and second we may have something like the red box jumps over the blue dog which would also produce a different operation even if we change a tiny bit of it it is going to produce an entirely different result now I know some of you are very fond of cryptography I'll say hashing is not encryption there please don't kill me anyways hard to find collisions in this function these are the properties of the hashing function there are pretty much two the third one's kind of a repeat whatever I digress we use a hashing function to get identifiers the hashing function has several properties firstly it's hard to find collisions it's hard to find one piece
of data that produces the same output as another piece of data it would be hard very very hard for me to find another sentence that would produce the same output as the red plots jumped over the blue dog I'm sure if I tried very very hard and used a lot of resources on something that's not very secure I could and people have but for our purposes we don't really care about that it's also hard to reverse so if I just give you this random data on the right you can't really go back and especially since these things tend to be very very random or very very different you can't really determine much about the input from the output
or really anything at all so even if I have one modification like a typo on the second line we're going to get an entirely different output why is this relevant at all to what we're doing now and why is that useful it seems like we would want to know the input well this produces a lot of very small and for our purposes unique identifiers for each of our books for each of our files these but these books or files are just data at this point they don't really have file names we only have an extension so how do we name them in a way that we can compare them to each other compare them to stuff we already have
we use hashing and specifically we use md5 which is a fast small and widely supported one we use this to remove duplicates as well as some other functions we'll get to later and well why else will we do this well to do that we'll cover about hash sets on the top right on the top right just for the record I usually keep the small stuff not the stuff that's really small that you might not be able to read it that's because it's not that important don't stress it in the top right there's a bunch of hashes of random files and in the bottom right we have a bunch of different sets of hashes of those of uh
known files because some books are boring some books are Windows systems files some books are log files some of these files are not interesting if you want them you can enable Tomb Raider and it'll get them but most of us don't really care about that if we're trying to just get our own data back we don't want to be saying oh my finally my uh collection of Metallica and it's just that from Windows booting up so all these things you really care about but how does hashing come in that bottom right picture is of what is known as the nist in IRL it is a wonderful online repository with tons of known file to files so these
windows system files they exist on everyone's Windows system there have been lots of operating systems and we know what files they have on them so what do you do you get them all together you collect them and you provide them for anyone to use by doing this you end up with a ton of hashes of known boring files stuff that we don't care about stuff that we can ignore and save us some time defining what we do here we use these sets to considerably reduce the noise get rid of all that Garbo and get us some more important stuff to continue onwards any ideas for some pitfalls for this where this might not exactly succeed
really ideas questions
well earlier we saw a we saw what happened with typos even small modifications can produce entirely different hashes and unfortunately if you're recovering data you're going to end up with a lot of corrupted data so that's going to be entirely different even if there's just one bite flipped and it's not going to be in the hash that anymore so while this has been great it's basically taken this massive library and removed one of the bookshelves still have quite a lot so what do we do from then well we sort it as best we can so that we can comb through it and figure out what we want to figure out based on uh better categories like music
and videos and stuff that might have been on the desktop or other stuff that was created later in the computer's time span if you can we already know how to classify and find what type of extension a file is using file classification using photorec from earlier and we can also create subclasses of those specific using those specific extensions like let's say we have a uh what's a good example you might have a PNG or JPG file and you might have GIFs these are all subclasses of an image these are all lower on that hierarchy so we would want to sort them into different things just so we can more better better and more easily look through them
this is just extension classification we're simply taking the extensions and we can organize them into categories based on what type of extension they are it's nothing crazy it is however very tedious which is why Tomb Raider doesn't we can also use content classification we can open up the books we can look through them and figure out if it's a fiction a non-fiction book if it's a image if it's a piece of media if it is uh random noise and it doesn't actually seem like an image at all or rather a file at all and there's a lot of different ways of doing this for hours we use Lin Magic those who are familiar with Linux will
know this thing called the file command but it's not super relevant point is that there's a lot of methods to determine the data type based on the data of the file determining what type of book this might be based on the pages we use both of them and afterwards we end up with something that is that represents our images our audio our video our document files our program files system files and so on we end up with an organized representation of data that was on the drive from this random massive assortment of images we end up with keep on calling files images with this random assortment of files we end up with something that is our own directory structure our own
better file system it also Flags it for anomaly if the best it can so if you have something that looks like a JPEG at first but then has a bunch of playing text in it well that's a little weird but you're probably looking at all you want is to whitelist the images all you want is to get the images back from this drive Additionally you could probably skip a lot of other steps you could skip the uh file carving if you can find the file system because you don't care about your deleted images you can skip the Imaging process of the drive if you really want and you can skip every single out every other thing too
Tomb Raider allows you to do that and lastly it runs entirely on its own so once you start it running this operation which can take hours or even days to do all of this it will just go automatically I like to just start running uh the night before and then check on it for breakfast and find other uh Craigs or other such content on there and uh be very confused so that is the walkthrough what Tomb Raider does at the moment what is it going to do in the future a lot of this stuff I'm going to add as soon as I possibly can I've been a lot of my time recently focused on this presentation
but first thing I'm going to do is rebuild a better version of the original file system this would be essentially a better Library so instead of one that might be not very well kept a library or hard drive that is very disorganized we can make our home we can get rid of duplicate data we can have a better way of looking for what we want faster a lot of times computers seem to have a very hard time finding files in massive massive sets which frankly doesn't really make sense to me because we can always do other stuff like uh what's it create an index yeah but anyways we can replace duplicates of files with links to those files we can get rid of
these long file paths these things that are under a ton of different folders we can include the files that we otherwise would have had deleted and we can uncompress stuff this is all just lovely nice to have afterwards I'm working on this now it works just fine already promise and these are features that I'm planning on adding back so if you had a system that was damaged or destroyed you could plug and play and it would be exactly like you remembered it rather than just being something for exploration otherwise I'm thinking of adding an entropy classification this is uh entropy is a measure of how random a file or set of data is so a
random number very high entropy but a Shakespeare's Works low entropy this is useful to us because we can compute a number zero to eight that actually measures the entropy and use that number to maybe determine if something is noise or not which is great right that sounds very useful why didn't we use this earlier well there's a bit of a problem with that unfortunately that uh a lot of file formats like encryption encrypted file formats are very good at encrypting they produce things that have very high entropy very high seemingly random data so you might have eight for completely random noise which is actually kind of hard to get because nothing is ever completely random
and you might have 7.99997 for an encrypted file or a compressed file because of course it wants to try and take this long organized data and compress it into a very compact unique representation additionally on another note the file formats we use for images and audio MP3 and jpg are also very good at compressing so they're basically the same so I didn't use this in Tomb Raider because I kept on finding that there was just uh too many false positives but I'm still working on it and I'm hoping I can use it to filter out the signal to noise ratio a little bit better so uh uh yeah that's it everybody that is uh now uh I guess it's time for the QA and
resources we use a ton of software for this for uh Tomb Raider and I special thanks to all of these amazing places for the wonderful software and wonderful data set I got to use to make this thing work and put together a lot of pieces that would have taken me ages to do otherwise additionally uh special thanks to everyone who throws away hard drives without destroying them Scott Moulton noise Bridge the hacker space and lastly lowercroft thank you Tomb Raider uh so yeah I guess I said that was it and then I had some more to say but uh any questions yo can you come up to the microphone so they can hear you if you have like if you have questions
can you guys come up to that microphone and ask I love to answer them what's up so big files with the high entropy are probably compressed I'm sorry can you big files with uh High entropy are probably compressed small files with high entry might be interesting things like SSH keys I agree so they said uh big files might be something that's compressed uh and have high entropy small files might be something like SSH keys or stuff that's important I agree thank you and at the moment that's actually covered by the tools that we already used for classifying so the content classification and extension classification that would look for DOT key or dot pem files works really well and I find those all
the time they're like one of the most common things I find uh along with Shadow files which are fun to test out on another note but yes I agree and uh that's part of the reason I want to implement it so I can use that sort of stuff to shorten things down or at least provide an option saying I for those who want to use it and say really really really crank down on the noise in which case they could just remove all of the uh um files with a higher entropy anyways yeah what's up so uh I first I just want to say that this is awesome this is so cool so much I really appreciate it it's I I
I I really love it and it's been a lot of fun making it so thank you so much and my question is uh when when did you start approaching this as a signal to noise ratio problem and what did you use to kind of get yourself there when did I start treating this as a signal to noise ratio problem and what did I do after that to like work from it okay um basically I would recover data and there would just be so much there would just be tons of tons of files and even on software I knew more or less what it was or what would be there because I would find the file system and I could look and see
what were actually documents or downloads it would just be some office computer uh there would still be like a massive amount of Windows files that would just ruin any of the user data um at that point I was looking through it and I was spending all my time trying to find actual relevant data and stuff that was added by the user not by the computer um that I was like I gotta work on improving this so I spent a lot of time a lot yeah there was a lot of time spent on Tomb Raider just doing that and uh looking at shortening it down making it so that you can actually process it because there are a lot of tools that Tomb
Raider uses these problems are not you know unsolved for recovering data and the problem is that people don't know how to use them and they produce so much noise that they can't really use them and that's why I made this I wanted people I wanted you all to have a tool you can use where you can just plug and play and run it on any sort of Hardware I am so looking forward to using this thanks again for the presentation I really appreciate it man so I looked up your tool before the the conference when the description for the talk was posted looks super great it's like a really awesome wrapper around a lot of other tools that we all use very
frequently and so I'm excited not to have to do it all manually anymore um but I couldn't find any good descriptions on how to use the tool or how to set it up especially where it uses so many different tools you probably got to set it up just right um kind of what's the process you're looking at and creating that kind of documentation so we can actually go and use this absolutely so they brought up a very good point and it is that when they looked up the tool like a few weeks ago a few weeks ago it didn't have as good documentation uh you're totally right it's because I was making these slides and uh I am I have updated the readme
since then to and other documentation to make that better and support more stuff um now when I'm done with this presentation I'm going to be improving that even further and continuing my development on that uh in my free time so you're totally right and I'm sorry I wasn't there earlier but thank you for looking I'm glad you checked it out um I guess I didn't expect anyone to be looking at it much earlier so it caught me off guard but I will uh I'll be improving it a lot more I I've already added some documentation on these steps and some usage and installation instructions my pleasure thank you um any other questions does anyone else
want to add we have a lot of We Still actually have a lot of time I'm a little bit early so if anyone wants to know about uh stuff that wasn't very clear uh any possible future features bring it I'd love to hear about it hiya hi so for some of the maybe more interesting things you found like the diagrams can you speak up sure the like the diagrams you found the architectural diagrams for the uh the Water System have you done any reporting to the owner of that data or thought about how you might do some I haven't been I haven't been able to find the owner of the data also fortunately those diagrams
were they seem to be yeah they weren't they didn't seem to there wasn't any actual compromising data for their systems on there just interesting stuff regarding the technical specifications of how fairies worked and how the San Francisco water Transit Authority ran their system that's a very good question though I have reached out to a lot of owners of these drives um when I found like when I'll find all their data on them and I'll just be like oh wow there's like an entire this is this person's life These are family pictures and so on um oh yeah to clarify the question you asked was have I contacted the water Transit Authority about this data anyways sorry I didn't say that earlier
there have been a lot of people that I've found their hard drive and I have been able to find contact info tried to reaching out um unfortunately usually they don't respond or they or it's out of date information because it's been in the dump for years and I just randomly dug it up but um yeah I also run a side business doing this so I can more like directly help people and help get them get their data back without it being any like problems with other big problems uh what's up do you have a question actually I do have a question I was interested to know only because you're so intimate with the topic uh what is
the current state of backlog in like organizations who are trying to get through these petabytes of data using these kind of tools and are you finding the tools have been effective in trying to help them or it's been a few years since I've looked at the numbers as to the backlog the backlog for oh uh for example uh when uh they're trying to uh government agencies or local law enforcements are trying to go ahead and create a case against somebody using the digital forensics and there's usually these backlogs of years sometimes to try and get a hard drive through for analysis because it's so manual in many state in many ways that's a really good question thank you
um so they asked what does it look like for existing Enterprise implementations of data recovery and digital forensics I can only give an approximate representation because I am do not I have not worked as a digital forensics person this has just been my fun learn how to do data recovery thing however I do I did get familiar with that because my roommate works as does digital forensics and I was like hey what do you think of this and they were like oh that's great and um and uh so I guess my answer take this with a grain of salt um I think they just pay for the time they just spend a lot of time going
through stuff they have to do some stuff in the manual fashion because they have to keep because it's just such a strict legal process so like full disclosure you couldn't just be like uh your honor here's my here's my rated tune arrest that man like that's not really how it works exactly you have to use like the original image you have to use all this stuff but you also have resources and you're still gonna have to look through tons of files and you're still going to have to try and recover all this old data why not save yourself some time and automate the hell out of it and find as much of it as you can immediately using
this that's what I made it for uh not as a first as a very good first step for that and uh final full solution for anyone else so thank you for the question it's very good uh what's up so when the when the hackers all go back home and everyone's in their local municipality yes where do you suggest we look for busted hard drives that's an amazing question oh I love that question okay so when hackers go home okay I have two minutes uh when hackers go home how do they find hard drives great great question so it depends what there are some places where you for the dump specifically you usually have to know people to be
able to like look in the dump and find them I know people uh and made friends with the way friends of people who work at the belt which is great however that's awesome there's tons of ways so I only have so much time so I'll go through some of them eBay tons of stuff you just look for the ones that are like broken or or parts and then you just can buy like a stack of 20 of them that are from laptops for like 50 bucks it's the best um you can go to garage sales you can go to [Music] um old offices they're always throwing away computers um you can go to places that have
E-Waste which is also old offices and other companies and uh if they have public access to their E-Waste which sometimes they actually do uh and it's a little bit annoying if you work at a place that has E-Waste just go there um and then you can find a lot of hard drives and other such things in there if you find any anywhere you might find broken computers broken computers usually have hard drives um so I went to noisebridge special thanks for them which is a hackerspace to translate that just means that it is a big workshop for anyone who wants to build stuff hack stuff and people donate stuff to it all the time they have they
had tons of just broken computers and shells of them and I would just be like all right time to go get some hard drives and they used to have a lot of hard drives in those dead computers now they're all mine and I am uh using them for practicing and for testing this and practicing this um don't worry I'm I'm I'm not I'm not actually taking all of them because people should hack on them and do positive their own stuff with it um so yeah oh cool okay all right I'll take an hour to explain the rest of this question then um so yeah hacker spaces are basically workshops where you can do that and uh
people just donate spare computer parts to those you can go look through those find hard drives garage sales people just throw away old computers because they don't work and it's because the hardware was broken or you know broken um you can go to let's close it uh Craigslist lots of old stuff there Craigslist is like I swear Craigslist is like the black market but it's like not the black market you can find everything there um I wasn't gonna say about garage sales just um you can go through the literal trash of places that are near um corporations that's not very fun though because you can usually find them in a lot more uh convenient places
anywhere where there are people Gathering to do a lot of tech uh there will probably be broken computers somewhere around there and you can just say hey can I try and fix your hard drive and a lot of times it's like what do they have to lose if they just have a broken computer that they've given up on and you could fix it even if you don't fix it you're gonna be totally fine with that um a lot of the time so just uh ask around look for places where there are tons of leftover computers something else I'm thinking of I can't really remember at the moment um
thank you thank you [Applause]
foreign
[Music]
foreign
thank you [Music]
foreign [Music] [Music] thank you
[Music] thank you
[Music]
foreign [Music]
[Music]
[Music]
foreign [Music]
[Music]
[Music]
foreign [Music]
[Music]
thank you [Music]
foreign
[Music] foreign
[Music]
foreign
foreign
[Music]
[Music]
foreign
[Music] foreign
[Music]
[Music] foreign [Music]
[Music] foreign [Music]
[Music]
thank you [Music]
foreign
[Music]
[Music] okay
thank you [Music]
foreign [Music]
foreign
[Music]
thank you
foreign [Music]
[Music] [Music] foreign [Music]
[Music]
[Music]
foreign foreign
[Music]
foreign
foreign [Music]
[Music] thank you [Music]
foreign
[Music]
foreign [Music]
[Music]
foreign [Music]
[Music]
foreign
[Music]
[Music] thank you [Music]
thank you
[Music]
[Music] thank you
[Music]
[Music]
foreign [Music]
[Music]
[Music]
foreign
[Music] foreign [Music] foreign [Music]
[Music]
foreign
foreign [Music]
[Music]
foreign
[Music] foreign
foreign
[Music]
foreign
foreign
[Music]
thank you
foreign
[Music]
foreign [Music] [Music]
foreign
[Music]
[Music]
foreign [Music]
[Music]
[Music]
foreign [Music]
[Music]
[Music]
thank you [Music] foreign [Music]
[Music] foreign
foreign
[Music]
[Music]
foreign
foreign
[Music]
[Music]
foreign
[Music] foreign
[Music]
[Music] foreign [Music]
[Music] foreign [Music]
[Music]
thank you [Music]
[Music] foreign
[Music]
[Music] foreign
[Music]
foreign [Music]
foreign
[Music] foreign
[Music]
[Music]
foreign
[Music]
[Music]
foreign [Music]
foreign
[Music]
thank you
foreign
[Music]
[Music] thank you [Music]
foreign
[Music]
thank you [Music]
thank you
[Music]
foreign [Music]
[Music]
foreign
foreign
[Music]
foreign [Music] [Music] thank you
welcome to the b-sides you are in the ground floor track this talk is zero days should not be a fire drill by Steve and Tony I do have some announcements before we go ahead and get started first and foremost we'd like to thank our sponsors especially our Diamond sponsors LastPass in Palo Alto networks and our gold sponsors Amazon Intel and Google for their support the support of our sponsors along with donors and volunteers is what makes these events even possible cell phones I do want to give you a quick reminder I'm sure you've heard it many times already these talks are being streamed live so as a courtesy to our speakers please make sure your cell phones are silenced so as
to not be disruptive um in the event of questions which I believe we're going to have some questions time after uh we're going to ask you to use the crowd mic there's a mic in the middle again these talks are being streamed so for the sake of the audio on the YouTube we ask that everyone speak into the mic uh as a reminder besides photo policy strictly prohibits taking pictures of anybody without express permission of everyone who's in the frame so as these talks are being streamed you will have access to the content later so if you are concerned and would like to know more about the content you can look at it later and uh unless you are 100 sure
that you have the express permission of everyone in the frame please refrain from taking photos please make sure you have your masks on at all times [Music] that is it so with that I will kick it over to Steve and Tony thank you so much so my name is Steve winterveld I have um responded to both uh data incidents data breached at a loss vulnerability uh announcement zero days in banking and Retail in government and so I want to share some of those lessons with you my co-presenter hello everyone uh so thank you we know you have a lot of choices for our talks uh one thank you for showing up after happy hour uh and also thank you for
coming to uh to our talk this is me um we didn't really do intro sides and then you threw this one together for me so thank you um but uh thank you guys for being here we're gonna talk a little bit about uh zero days uh response to those I've been in infosec since uh the late 90s um and whereas I feel like you've kind of been in the management you know senior track I've been the person who's like always been under this I do work I do work I do sometimes I don't actually do but you know when you're the person you know with your boss breathing down your neck like what's your next move is
is very critical so we're going to talk a little bit about taking the guesswork out of that process um and how to make you more successful dealing with zero days so let's get started and so a big fan of the office uh I'm not gonna lie
a lot three hours watching uh clips of The Office uh if you go to YouTube and you do fire drill in the office I'm not responsible for the next three hours you guys lose but it's awesome and essentially it is saying you know Dwight had a fire drill and nobody did anything so then Dwight had another fire drill and it went the other direction and so we want to say that you know how often are we going to have a vulnerability coming out and what time does it come out Friday at three o'clock so log 4J is going to come out before the holidays on a Friday at three o'clock how much of our Lives should we be given
up and what can we do to kind of turn that around hey just a quick note the the other most awesome episode of The Office is when Dwight cuts a face off of the CPR dummy and then puts it on his face and he's like hello Clarice remember that one and so when we get to questions if you don't have a question about our talk anybody's willing to willing to can try to stop Tony on movies you should be able to do it so we should have some questions all right so let's talk about some examples here's uh kind of a whole Myriad of examples and you'll notice too that these uh arrive in your uh you know
in your um life in a number of different ways right solarwinds this is like supply chain like a product that you're using where's that product installed who's the product owner who's the owner of the server that the products are installed on that's a very different story from uh like a large scale um you know like heart lead or log 4J a protocol related risk those require a whole different group of people to be contacted and to respond uh to this incident and then going down you look at things that are kind of status quo parts of your infrastructure that you're using um you know they're they're obviously you know there was a great talk the
other day about devsecops and how it's failing um is actually in this room um and the funny part is most people always talk about Dev and sex the Ops part uh is all the other uh all the other things that have to happen for Devon sec to work properly because implementation deployment um who's using those systems that's the operational aspect and then obviously stocks net um you know a meltdown Specter these are things that most people aren't really prepared for in general right because these are these are vulnerabilities or you know exploit attempts will be happening against systems at the very deepest root of your of your environment so it obviously requires a different perspective
and you know again this is a team sport so as soon as I hear about what's going on I've got two challenges where is it in my environment and where is it in my critical provider's environment and so you know you need both those process you need to have thought through on you know as soon as I start on my checklist I have a partner in Vendor management who's going through so at Nordstrom we have had and my wife is very mad that I moved away from the Nordstrom discount but the we had 5 000 vendors so then we had a category of which of those vendors had our customer data which of them had credit card data you
know some of them were just they were in our system so when we sold a pair of shoes they sent us another pair of shoes some of them knew who bought it what it was for and so you know that's that's the first one the next is where is it in my environment how many have a lot of confidence in their asset management system you're in trouble how many people have an s-bomb know where all their log 4J is and all that that is awesome working on it nice to see I have this going next to me you all take your mask off and yell you'll have an s-bomb do you know if it's correct maybe it's called us all
your stuff you know and then firmware how many had a process when Specter came out to patch firmware I mean for us that was a new process we had to develop and so I and is iot in your asset management system today or is that just a rogue system not considered to have data on it so it's not in your asset management and so the reason we built these out just like when you're doing data breach I think you know third-party data loss I lost data I lost access to ransomware you know there are classic scenarios you should go through in your planning and we think this is a framework yeah the last Point too
um especially when you're talking about where these things exist in your environment um you we you know he just mentioned about s-bombs right a software bill of materials um there's a lot of especially on the iot side uh the kind of the antithesis of s-bomb is do you have a problem maybe scanning into your environment because of risk of these fragile TCP sacks being knocked over from your asset identification scanning tools right so there's a there's a big play there are you really catching everything that could be at risk here from these types of vulnerabilities and if you're trying to figure out why the Raiders are up there it's because I live in Colorado and this is one of the few safe places I
can put a Raider logo up so while in Vegas I'm going to take advantage of that opportunity
so I think part of the problem today is we think of a zero day as an incident and these are our two classic processes to deal with an incident I'm saying that a zero day is not an incident it is a systemic problem and it belongs in a different box I think it belongs in crisis management it's not something that's going to be finished out in the sock more often than not it's not a quick remediation with our tools again it's a committee it is a large phone call you know when the CEO hears about Lockport J on NPR on his Drive In what's the first phone call you're going to get that day you know and so it's notification
it is having a coordinated response who needs to be notified for what at what level of incident does it um so again a customer is going to hear about log4j and they're going to call the call center and the call center is going to be asked is my data safe from log4j does your call center have any idea what they were just asked so again I don't think this belongs in crisis management because you need a message for people that call in the next person that's going to call when I was a CSO for Nordstrom bank is the feds the OCC they're going to call and say what's a risk to the bank are you guys safe
oh so you have this constant stream which is why I think we should move it out of incident response into crisis management and follow that same thing and so one of the things that is you know which systems impact it by want to cry log 4J pick your pick your zero day of of choice solarwinds what access was it to data Tony was talking about solarwinds who was running it I don't care who was running it what was on it what kind of information was involved is it customer information is it infrastructure information those are different reactions those are different people I have to notify if there's customer information in there now I'm calling privacy they need to be
on the phone call if we lost it I have 72 hours to notify somebody and so you see where I'm going um and then again I can't bang on this drum hard enough I need you to be in lockstep with your vendor management or whoever's going to own all that others and so for log4j what's a reasonable amount of and this is manual I haven't seen any program and if somebody has let me know to automate this where you're calling your key vendors and saying are you secure from log 4J you have my customer data what's your plan and what are they doing they're doing the same thing you're doing trying to figure out if you even had that protocol
anywhere so it's not a quick call and done
no no go ahead uh so so this goes back to what we were talking about before uh there are a lot of different people if you look at the racy chart at the bottom this really depicts who was responsible for getting that ring uh into the Flames uh to destroy it right um but uh you know when as you kind of map out who in your environment needs to be contacted there's some really kind of practical things you can do so one if you've ever done like a tabletop exercise think of something where you're under duress right so uh so we work a lot with um you know large-scale web app attacks or DDOS attacks and you'd be
surprised going through a tabletop exercise uh before we go through the process of developing one you just say hey you're under attack what do you do you're like okay we're gonna call our two contact people in network Ops and and blah blah blah and then they don't answer the phone because it's lunchtime or it's midnight where they live what do you do next right who's the second person to call on the call tree how do you you transfer responsibility at what point do you open up a bridge and you say hey we're going to have a uh a bridge that's open from now until this this attack is remediated to where anyone who needs to know needs to be
notified of what's going on they know this is the bridge to call into right there's a lot of procedures that can be done beforehand to save time because when you're under duress when you're uh you know when you're hard down or if you've failed over to your failover data center just say and God forbid your failover data center is not failing over um you know everyone starts to get red in the face they start to make poor decisions uh and you frankly you have you know somebody that looks like this standing over your shoulder just that was that was not cool uh but you have like true you have your boss looking right over your shoulder and you're just
like I've got to focus I've got to remain calm the more planning you can put into uh this whole procedure here um for Who's involved who owns what who are the secondary contacts all that good stuff that has to be done beforehand um because the longer you wait when when it eventually does happen um you're not going to be able to respond uh in time right you're probably gonna have have a poor poor experience and that's why I love that when you're building things like this and then you do the exercise you do two things one you discover that you know Tony built this and it says create fellowship and there's two people responsible for that
is that legitimate can two people own the process I tend to say a racy can only have one R well Frodo was a key contributor to the ownership in fact some might say he was not the most equipped to take the ring on that yeah you know I think I think you have it anyway um and so exercises are are critical to both make sure everybody knows who owns it how they own it if it's a partnership how they're going to come to resolution and also to set expectations so for those of you who have been involved in some kind of a major incident the first thing you get from the leadership is what's the status and we want an update
every 30 minutes and then you turn to your sock analyst and you say you go do your work and when you have something come tell me and I'll update them every 30 minutes you don't need to come back to me every 30 minutes because guess what how long does it take to troubleshoot something until you figure out what's wrong you know I love those those hypothetical questions when are you going to know how much data was lost um whatever when I can tell it's not happening anymore and so yeah building expectations building the team the role you know it is it is key especially now with the high turnover all of us are experiencing you know how many of us have that that
one guy who left was the historical knowledge of all your firewall policies you know so so these exercises are really important so let's take a little journey back in time uh even further than this let's go back to 2014 when there was an SSL vulnerability called heartblade that was disclosed uh and oh yeah we found out that it had been actively actively exploited in the wild for a previous two years to that right so this was one of my first big experiences when I started off my um uh and to the point of you know who gets notified before anything was fixed we had to make a public press announcement right because everyone knows there's TLS or SSL certs deployed
globally across 350 000 servers uh that are oh yeah in front of big Banks big social media companies big Commerce platforms so we probably have to give some kind of notification what's going on um that that process what what's really interesting about this is we learned a ton of lessons from this uh and we we've uh announced it publicly um but the initial response was our implementation of openssl was not vulnerable to this exploit and we started sharing what our implementation was uh and then like three days later uh researcher came back to us and said hey you guys are still vulnerable here's here's a POC of of the exploits still working so we had like a three or four day
um lack of a head start on going out and re-keying all of our our search globally that's a huge impact and oh yeah when you go through that process you really start to find out man a lot of people need to touch a lot of systems at the same time in order to get this done expediently um so it's like hey can you spell computer congratulations you're on the task force right um so so uh flash forward to the response to log 4J and here we have um I what that's I guess eight eight long years after that and we started uh seeing that all the things that we learned from those initial uh you know
harp lead and then later poodle uh we put into action to have a really proper response for log 4J one the visibility around you know where it exists in your environment that's a real key issue um but more so you know we saw so much exploitation attempts uh against our customer base at one point I think sixty percent of our customers were being hit by exploitation attempts um as we started to kind of weed out and we started to see patches come out then we could start doing a reduction of that and say okay these are vulnerability scanners that are testing for the exploit based on this vulnerability scanner code that it had and we can
start to take that out but this this was a really massive massive problem and so this is where we've kind of diverted from our generic example to an example of a security vendor so how many of you work for a security vendor of some flavor so this is where you're good for you guys never never change protecting yourself and simultaneously looking at your product and how your product can protect your customer and so what we're talking about here is what we were seeing hitting our our thousands of customers on our web application firewall and and this is where you really kind of break out and there's intricacies to this right so like we have an
adversarial resilience team that really looks at how adversaries might and do try to bypass or do try to bypass Akamai platform security so securing the platform means making it continue to run properly but then the other side is anything that's deployed on behalf of a customer uh how are those configurations and that implementation uh secured separately right so again it goes into the the racy chart again who's responsible uh who's going to be brought into a task force um and uh you know if we go to the next slide we'll see that uh so the the red graph here on the right side is uh percentage times 10. so these are six sixty percent at one point sixty percent
of our customers were being hit um by these attacks uh and then you'll notice here as well as new uh exploit code is being released we saw a huge influx in attacks that test out that new exploit code right because anytime new code is released in a zero day there's a race against the clock like when is this going to get patched if I'm going to exploit it the time is now so we see like a fevered pitch here you'll see around the um a few days after uh Christmas time here massive massive uh increase in in attack traffic um now some of this was attributed to uh scanning tools that got the previous patch and then they were testing
obviously um but but again this is the interesting case about this again that I mentioned is 92 percent of the attacks that we saw were being blocked uh far away from where the exploit uh was was actually vulnerable so this is where you come into what what is your model for securing your environment do you have like virtual patching capabilities for your the exploit Mason this but you're buying Time by blocking the visibility of that you know from your uh from the adversaries right so um this again it was a great learning experience for us and one of the key lessons is never let a good crisis go to waste and so I want to talk about the three
aspects we have here the first is protecting our customer um you know on day one you have to get the message out to whoever your partners are whoever your customers are if it's internal I.T if it's if if it's making sure your your third parties are working on it if you're supporting customers if you were a bank managing wealth you're notifying people what you're doing so you know that educating and getting the message out that you're on top of this is critical um I find it interesting that this was a huge chance for people to re-evaluate the risks they'd accepted how many people have a security control in monitor mode only why because we're worried about the
first to be honest I was going to say be honest but yeah no need to do that how many people won't raise your hand no matter what I ask okay good nice psychological warfare I like that so uh you know this is something that that is is fascinating because when this came out the leadership was now asking are we blocking this not security banking can we please put in a blocking rule and so they took advantage of it I I saw some customers change their entire platform to block mode and some customers only block on the log 4J rule um and so there were again this is risk tolerance it's the customer experience security balance out there
but a lot of lessons learned around this is an opportunity to go back to leadership and talk about your your risk appetite which is a an ffiec banking term going back to what's your corporate willing to to say is the right thing to do hey on that point will you talk a little bit about uh you mentioned you know someone's driving to work as a board member they hear Hey the latest you know earth-shattering vulnerability that you're all gonna die because of talk a little bit about the involvement of the board when things get to that level how it changes the scope of uh of how you have to respond to it you know from a notification perspective from now
they want to be updated every 10 minutes just like you know your boss would normally want to be um what's that look like so my experience has been 10 years ago our boards had more luddites on them than technically competent people if you don't know what a Luddite is I'm going to Google that it's worth going to Wikipedia I'm not going to go down that Rabbit Trail but it's it's one of my favorite derogatory terms and so um the the thing we've seen is how many businesses now don't have a critical part of the business that depends on I.T it's fewer and fewer every year and so boards are now talking about the business strategy in terms of Technology
capabilities and so we're seeing more of them educated more of them they've always talked about risk they've never talked about Technologies and so the board doesn't want to know what log4j does they don't want to know what open source is they don't and you are not a technology advisor you're a business partner and if if you go up as a ciso as a technology advisor you're you're not going to knock it out of the park and so you're going to go up there and you're going to talk about risk and on day one you're going to say right now we have risk around customer data in these three places and we've put a a segmentation policy we
have additional monitoring we uh have an immediate patch process once a patch comes out we've implemented a wow for whatever you know this is if it's solar wind it's a product then you know there's a different set of what you have to do or you're going to uninstall it what's the impact of the business again rabbit Trails I won't go down but it's going up and talking about the impacts and the risk to to your business and if you go back 10 years 10 15 years uh how many how many companies didn't rely solely on the internet to do their business um so whereas you might be able to say just say you're in secops uh or you're
in incident response um and there's an infrastructure problem it's so easy to kind of pass that over and say oh that's a that's a um you know a platform issue or that's the cloud people that have to worry about that or that's a uh you know or there's some kind of if there's an internet outage if there's anything that could affect your company making money trust me anyone who could possibly fix it is going to be in scope for you know a resume writing opportunity right so the whole idea here is that as more organizations are relying on the software stack the internet stack to to viably have their business function it changes the conversation it changes
the education of the board a lot of the board now or a lot more technology Savvy worse it versus you know 10 15 years ago they were just kind of pure play business people advising on the next strategic step to take uh so I think that's an interesting change too uh and and we should be cognizant of that when it comes to how we deal uh with our superiors off the chain because you have to you have to understand where they're coming from too and you probably haven't heard of this organization she may want to write this down but there are a lot of Industry eye sacs out there never heard of it and that's his favorite topic um and so
that's another thing that you've got to decide to put in your process when you have these are you going to call in and see if there's an ISAC call FSI Sac Healthcare ISAC hospitality and Retail at one point there were 15 major ice acts I think there's more than that now I think there's like 24 now or something like that it's pretty pervasive but where are you going to go to your your peer group what communities are you going to reach out what external resources are you going to leverage and there's a bunch of them out there DHS is going to put something out there about a week after you need it um but again no offense State DHS no
offense is what I meant to say and so um you know but I will tell you why it's worth going and looking at that is it's a great lessons learned it's a great checklist to see what if you did and what other people did the last thing is expanding our tool kit for each one of these there are different remediation or mitigation techniques out there and so kind of think through that that first set we went through what tools do you have in your tool kit and which where do you have gaps you know we we talk about segmentation we talk about access control we talk about you know if you're in your AP environment API environment what what
are the different tools you have or levers to pull during these incidents
so again I want to talk a little bit about you know what is the next step what should we be doing and um I've really tried to make a case here that we should shift off of thinking about a zero day as an incident a sock owned problem and expand this to be what it needs to be you know a communication exercise a racy with I.T vendor management forensic the pen test team going out and PR even public relations even if you have to give incident response or notification that hey you are looking at this or maybe because of the vulnerability you had to take systems offline for patching all of that has to be in scope
absolutely and so um you know if you don't have a crisis management prop process you can go Leverage then we need to have another discussion but for those of you who do I think this is a great um I'm a huge advocate of nist uh simply because nist is something that you can show the Auditors is the best practice so you know something like um you know moving missed 190 I think is for apis I think we have uh Miss 207 for um Joe trust you know there are a lot of special pubs out there that you should be going pulling verbiage from Best Practices from and crisis management well and don't forget that first incident response cycle I
showed you is from nist as well and that's more of the traditional 853 so go out there build the documentation exercise practices I know you wanted to talk about that as well um and and cover a little bit about the s-bombs and the vendor management there yeah you know we talked earlier about the idea of uh um an s-bomb right so what software exists in your environment uh whether or not it's your responsibility to patch it um I mean years ago I used to use tools like sikunya which are agent-based tools that will install and basically map out all the software that's installed on that system and then compare it to the latest patch update for each of those
different software packages um things like that exist now I know they got bought out they're doing similar work though but the idea here is if something exists in your environment eventually it can come back and bite you because you know who's going to be looking for your you know uh your 20 year old uh you know software version of whatever package you're using on one of your old web servers the bad guys the bad guys are going to be looking for that and it's not just by happenstance it's the path of least resistance I don't have to use a zero day if there's a seven year day out there that hasn't been patched yet because you haven't
gone around to it um so that's that's one of the big pieces and again to the whole point of of walking through this and practicing it uh I can't stress that part enough um pretend like you are going back through heartfully say what is it going to take for us to re-key uh all the um the private certs uh and then redeploy um you know the search to our other machines what's that going to take who's going to be involved just practice that um and without going through a process like that if something like that were to happen again um you're going to be starting from square one and that's what we're trying to avoid here we we do a lot of this
when we work with our customers and our clients just because if something were to happen we don't want to be the ones who are in the way of them getting their problems fixed even if it's nothing to do with us uh so we're we're trying to help kind of Grease the skids so to speak those things working better within our customer base so if something does happen again we're not the we're not the roadblock yeah and so again going back to that first Slide the different categories that you should do exercises around uh and then there are different types of exercise there's a this typical table top where you're you can just sit down with the racy and talk through the
racing and you should have a timeline in steps notification discovery patch release you know you'll have some key milestones in your exercise for each of these and some of these will involve leadership some all of these will be more technical in nature uh so so think about the kind of exercises you want I think they're all beneficial so we put up some resources here one our threat research group has a bunch of great Publications uh I was just talking at uh at the other conference across the road about our gaming state of the Internet Security report so cool findings that we found uh basically if you own a gaming system or if you've been uh during the stay at home order I
like to call it the stay at home in game order because as a gamer it's like you mean I get to be at home all the time and if I'm on a boring call I can be on a oh is this being recorded oops um I can be playing something right so uh also I have ADD so I work better with that kind of distraction in the background um our security blogs uh again a bunch of cool stuff here just around things that we're finding um we have a security researcher named Larry cash dollar he has something like 350 cve says his name we call him the WordPress killer although he has recently kind of branched off from
WordPress um and on YouTube we have some developer videos uh a lot of these are Developer kind of advocacy videos talking about how you guys as security Personnel can interface with your development teams better and again how we try to do that within our organization as well and then the last piece is we are hiring so if you guys I think it'd be cool to work for a multinational Cloud security organization uh we do really cool stuff I've been at documents we hire just about anybody that's that's true I I snuck in um but I've been to the Akamai for nine years uh it's the first time I've ever worked for a vendor uh but I've been in
secops since the late 90s so it's really been eye-opening kind of seeing all the cool stuff that company like like us are doing so that's the uh the blogs a lot of the stuff we talked about for Lockport Joe you'll see in our blogs a lot of the charts Tom Evans great about DDOS or is great about fishing we've got I'm really proud of some of our research all right who's gonna who's gonna stop us Now's the Time we don't have to go there you don't have to stump us but um I if you use the microphone it'll be recorded for the uh for the people at home as well
okay really important one thanks for this this was great up who played the woman's lead role opposite of Roger Moore and the Spy love me uh okay I recently just watched all of those again uh with my wife because we were like dude these are classic call your wife dude no no I I was just saying dude like these are classes okay shut up Steve um Ah that's that's a tough one I I should have we should have put a caveat that said he's better with lines I'll give you any movie that is from the what what no go ahead be here all right dude it's too late for the caveats okay I know right question for you I know one
you can answer yeah yeah so um the pr part you touched on is really essential I I think that when a breach happens um eventually your Tech guys are going to figure it out but if you botched the pr your rep is screwed for a long long time if you're a smaller group and you don't really have people that are specialized in that area how do you handle that that's a great point so I will go back to my first hand experience in 2014 with heart bleed we put together a group which was our PR AR team our platform security Architects some other kind of security evangelist type people and our CSO and we all kind of work
through literally like the wording of what this press release was going to look like uh what we were going to say sometimes you can't say too much exactly uh but sometimes if you don't say enough it's like are you are you giving proper warning to someone else who may need to take care of this right so that's a it's a very fine balance and I think ultimately depending on um what organization you're a part of and what your responsibilities are there's legal responsibilities but also ethical responsibilities I think all that has to be taken into account so it is see I don't know if you have a more specific answer but so I mean um
transparency equals trust and loss of trust is hard to recover from that said I think about two things during like during a breach I have a person assigned who's going to tell me if I'm not following our process because if I don't follow our process and they dis the class action lawsuit discovers that I'm going to be sued for negligence I have my chief legal officer my general counsel hire my forensic people so the forensic evidence is under attorney-client privilege so I mean that's how I'm thinking at the same time when I'm talking to the board I'm saying where we don't have enough to say if we say we lost 3 000 records today I'm we may come back tomorrow and say oh
make that 1.5 million yeah big difference so it doesn't sound transparent to say we're still trying to figure out what's going on not hard to get the the marketing and PR people to sign off on that statement so it's a delicate balance I would lean into a culture of transparency uh and and talk to legal and talk to PR about sharing up front yeah thanks for that I what I'm seeing in general is people weighing cya against yes what is the damage right so that seems dangerous if there's like a framework that really makes it clear this is your legal obligation to disclose this portion versus you know what I'm saying yeah yeah and I think
they've attempted to create Frameworks around um you know one from a vulnerability perspective if you have any risk involved there responsible disclosure and then also breach the disclosure is being handled a lot more um from the compliance level where they're like hey these this is what you have to do if something were to occur as per the the compliance that you've you know opted into that you've uh agreed to right so there's also some psychology around admitting mistakes um the the classic example that I always remember is Warren Buffett every year puts a mistake he made in his annual report because he has so much more credibility by doing that and the one year he didn't
feel like he had a good oops he talked about the UPS from the year before yeah um so they're they're again transparency generally wins uh and it also goes back to how negligent you were uh in the first place thank you thank you thanks for the question so you guys are a huge company very technical serving very technical customers we're serving their very large very technical companies I work at a startup and I've got three Engineers I'm in QA by the way I'm not even a security person thank you for your service I try um are there resource resources on your blogs for how to get the rest of your non-technical team on board with these
ideas with these for lack of a better term war games about like how do we bring everybody into these because I used to work in support so I'm really good at writing effective apologies and emails that don't get us in legal trouble there's a lot of resources outside of just the technical parts that can be looked in here and be really helpful for cya for making the apologies for doing the press releases do you have more resources for how I can get my marketing team interested in this for what happens if HubSpot goes down and you're planning for what happens right yeah yeah um you want to take a several so um there first of all there are great
industry specific podcasts uh and and one of the things I like to do is figure out how like some how somebody likes to learn I prefer to read uh my ADD makes audio books impractical what was that and then um you know other people have different methods so um find out what they like uh Sans has a lot of great examples uh they're great to plagiarize from uh YouTube has a lot of great resources um like I said I plagiarize a lot from nist ISO is a cost so I I don't if you're International you want ISO overness but it's not free um and again belonging to a community there are so many communities out there
beyond the ice acts that uh the more people you can plug into them but um you know like I said for your marketing if this is hilarious so I was I was talking to the marketers and they're like the book influence is great we use it and then I'm talking to my fraud expert me and I were doing a talk together and he's like well all the fraudsters know that the book influences the best book to learn how to be a good frauds and so you know that's everybody yeah yeah that's that's a resource that uh that went right to the top of my reading list well I think the uh the point too about uh joining Community uh
another good piece of advice might be build a community within your organization say hey I really want us to be prepared for what happens if something catastrophic words who happen uh find stakeholders within each different part of the organization and all have lunch together one day and then maybe the next time you say or at the lunch maybe say hey it'd be great for us to have like a monthly meeting just to kind of talk about all the pieces that fit into this um and then maybe from that you guys can start assigning UH responsibilities or areas of expertise even a lot of times people's areas of expertise is not the role that they're currently working in I worked
for years at a company where I was like to do everything person uh and it turns out you know I would rather do less so uh from that perspective not not just less I would rather not have to do everything the hard way um I want to automate myself out of a job kind of situation right so um but I would say build a community within your organization of people who are like-minded that that think there might be a risk if something were to happen and how are you guys going to respond to it so hopefully that helps thank you yeah and if I come up with anything else I'll post it on Twitter thank you so follow me on Twitter that's
not a Shameless plug nobody follows me I don't know why I'm I'm just not interesting really but I will share something if I find it okay awesome thank you so much also my favorite line that no one seems to get I am an excellent driver I'm an excellent driver excellent driver yes well now this is like really putting me on the spot that is 80s is that um Weird Science it's not okay all right what you got it okay I don't want you to stump me and get the one the cheese okay anybody else know it oh yes okay to be honest I probably should have put some caveats at the beginning of that whole this whole talk has been for
naught uh anyway if there's no other questions anybody else have anything else all right thank you guys for being here really appreciate it and uh have a great weekend [Applause]
foreign
[Music]
foreign
[Music]
foreign [Music] [Music] thank you
[Music] thank you
[Music]
foreign [Music]
[Music]
[Music]
[Music] foreign [Music]
foreign [Music]
thank you
[Music]
me ant
[Music]
foreign
[Music]
foreign
foreign [Music]
[Music]
foreign
[Music]
foreign [Music]
[Music] foreign [Music]
[Music] foreign [Music]
[Music]
[Music]
foreign [Music]
[Music] foreign
[Music]
[Music]
foreign
[Music] foreign [Music]
[Music] thank you everybody I just want to make sure everybody can hear me to the back of the room okay awesome all right I'm just gonna stick close to the mic I'm sorry okay it's great to be back it's wonderful to be here at besides Las Vegas after a very long time apart and I hope you are all having a terrific reunion and experience at summer camp 2022. so
Related talks
51:51
32:48
33:48
54:33
31:06
20:51