David Bianco - The Secret Origins of the Pyramid of Pain

right welcome everybody I don't know if I can follow that great oscent presentation sorry I will try not to move around too much so uh welcome I'm a I don't know a last minute emergency replacement for a a hurricane emergency speaker so Phil came Phil Plant America comes up to me he's like tomorrow he's like I hate to do this to you but I need you and I'm like I've been waiting my whole life for you to say this Phil but it turned out he really only wanted me to speak today so I'm here and I'm happy to be here so let me ask you start off by asking a question is anybody in here familiar

with the Pyramid of pain a couple people the rest of you who are not why the heck are you in here the old talk yes yeah I don't know who the old talk was but sorry you're not getting that one um so that's that's okay though we'll talk about that um but before we get into that a little bit about who I am my name is David Bianco um right now I'm a security researcher at uh splunk's surge research team but I have about let's just leave it at over 20 years of experience in incident detection incident response a little bit of cyber threat intelligence mostly about how it relates to using it for incident

detection incident response and threat hunting I was I like to say I was cyber security before there was cyber security because you know there wasn't we didn't even have the name cyber security when I started I started as a system administrator actually at when I was in school at University and just kind of grew into cyber security through from there so even before my 20 plus years of experience in cyber there was a let's say an unnamed number of years as a system administrator of Unix and Linux systems uh I spoke fun fact I spoke at the very first b-sides Augusta uh which I think was like was it 2013 I think it was a

lot different first of all a lot smaller than this we didn't have like four tracks of talks we only had two tracks and we certainly did not have these fancy um badges which I which I love by the way these are my favorite circuit board badge actually we didn't have these fancy badges what we actually had was when you came in we had a red team track and a blue team track when you came in Doug Birch would punch you in the face and which eye was blacked would be your your red team or blue team Track ticket no I'm just joking because Doug would never taste such a nice guy he would never punch any anyone in the face it

was Phil and I'm I'm also let's say a a noted pyramid Aficionado which gets us to my only actual slide for this entire talk the Pyramid of pain the idea that I want to talk to you about is is how I came up with this but first for a little over half of the people in the room probably I should talk about what it is very very briefly if you want the full right up there there's the URL here I promise it's a bitly linked but it's a totally Safe bitly Link this just goes to my blog I'll tell you shortly how I came up with this but basically think of the pyramid as a model that you can use for some of

the different types of uh indicators or CTI data that you might be receiving to help you defend detect things on your networks for for defense and it kind of gives you the relative what I consider anyway the relative value of those different types of common indicator data so I value Things based not on how easy they are for me to use but how much work or how much pain it causes to our threat actors like they have in all of their attacks they have some type of you know indicators that they can't really help but but leave around like if you really want to get into this uh Google lecards exchange principle but the uh the whole

idea is if a threat actor is doing something on your network they are leaving some kind of trace and they can't avoid it our job then is to find those traces notice when those traces are present in our Network and quickly do something about them so that we can interrupt their use of those like maybe in the midst of their tax cycle before they complete their kill chain if we're able to do that reliably especially but that means then if we can find that indicator they can't use it anymore they have to find something to replace what we detected and that's where the pain and the Pyramid of pain comes in because it means that

not all these are as very difficult to replace like on the very bottom down here you'll see that the hash values I marked them as trivial because you know one bit in a giant file you get an entirely different hash value and that that's not including the fuzzy hashes by the way I really could have called this static hash values like sha-1 md5 those kinds of things not things like ssdeep or others but you make a very small change and they have an entirely different hash value sometimes these things are so easy to change a hash sometimes it even changes just by accident like a transmission error flips a bit and it's still a fine

binary and it still works sometimes you will see like a threat actor just changes the the compile date because they just make a fresh compile before they deploy that changes the the hash value entirely so hash values by the way are kind of garbage for detection with a certain specific exceptions to that rule try they're they're low on the bottom they're on the bottom of the pyramid because threat actors don't care if you notice their hash value they'll just come out with a different one domain names and IP addresses the the next two layers are a little bit more it's not so difficult for a threat actor to control a bunch of like let's just

say I need 10 000 IP addresses I'm going to keep all of them in my back pocket and when you find my one IP address I'm using for Tool downloads or command and control I'll change it no problem domain names are pretty much the same although you just you generally have to go through the trouble of either registering it or co-opting it somehow from a legitimate domain owner but still pretty easy especially for some of our threat actors who might be closely associated with their domain registrars for their country next level up we're starting to talk about tool marks or as I call them here the the network or the host artifacts that get left behind how do you

establish your persistence is there a registry key or a certain file on the disk is there a certain way that you construct your command and control channels a user agent that you use a URL pattern that you construct even if it's randomized there might be a regular expression that you you go by that can detect it at that point you can find those and in order to fix it to so that you don't find that anymore they might have to go in and make some changes to their code they might be really small changes like renaming the um the registry key or changing the way they build their URL but they have to do it and then they have to recompile it

probably retest it and redeploy it so it becomes a little bit annoying there if you force them to do that over and over eventually you get some experience with the tool that same tool you might see over and over again and you can maybe develop more robust detection like where's Paul Nelson he was talking about Yara rules earlier today you can maybe make some more robust detection so that no matter what small changes they make to the tool you can still reliably find it that way you've just told them they can't use that tool against you at all and they have to go out and either create a new tool from scratch or maybe acquire a new tool and

try either way train up with it become proficient and that's pretty challenging they could do it once Maybe above that though we call it the ttps the tactics techniques and procedures if I had more space and had I thought of it at the time because TTP was a cool phrase back then I probably would have called it just behaviors because that's really what it is it's just the way that they're trained and the way that they their preferences mix up together to to express their activities on your network or on your systems so you find their behavior no matter what tools they use no matter what artifacts they leave behind you're looking at for them at a very high level

and I will give you some examples of these um shortly but when you come to the part where you're attacking someone's training and you're saying you can't do things the way that you are used to doing them with you can't you have to retrain find a new method that's extremely tough for them especially if you can do that to them multiple times I like to say you're putting them on the pointy top of the pyramid and it's a very uncomfortable place and you've forced them to either retrain and change their behavior which if you've tried to lose weight quit smoking whatever you probably have an idea that that can be hard or you they go away

because they're not willing to change so the Pyramid of pain the whole idea is for you to prioritize your human effort on things that give you the most potential benefit and make your systems most resistant to a a particular threat actor so I keep wanting to change the slide now but I don't actually have any other slides so I'm just going to put that thing down I originally published the pyramid in 2013 the end of 2013. um several employers ago um but you know I kind of came up with this idea over several years prior to that I just didn't know it as a as a pyramid or as a model at first it was just like an

instinct so I started at a giant company I won't name them uh we just followed the oscent presentation so it probably won't be very difficult for you guys to figure out who it was but um they had just started a new um a new security team for all of their businesses worldwide and we had a lot of them and one of the things that we started off with was a I don't know a uh a baked in set of threat actors that were already targeting the company we just hadn't had as much visibility and and response to to combat them as we developed over the next several years but what we did have was we had really good information

sharing with other peers in our industry so we started off with our I don't know the baby steps of our uh intrusion detection our network security monitoring our incident response by taking a look at what the other peers were seeing in their Network they were doing threat Intel sharing before we had ice ax and things like that but this was basically the same almost the same thing as an ISAC we would we would take basically IP addresses domains and hash values from them against the threat actors that we thought were likely to come against us and there was a particular one that we were concerned about that we kept dealing with over and over again

we started essentially even though it didn't exist yet at the bottom of the pyramid and every time we learned something new we put it back into our detection so that we could find it again this particular threat actor had a an implant that they used that they would leave behind to perform command and control on just you know whatever systems they had compromised and it was really interesting it was the core of the implant never really changed but it was a network command and control so they had a bolt-on like a dll that they would change that would change the way it it established its command and control over the internet and it would use a two-way

command and control so typically it would it would set up to um sends data back and if it or sorry it would set up to to receive commands and if it connected successfully to receive a command it would set up another channel to Res to send data back so they had two channels for two-way communication the initial versions of this were HTTP based plain text HTTP and we would see these and we were mostly by the end at first detecting them on hashes or domains or maybe IP addresses that we knew were were bad from our threat intelligence but we collected a few samples from ourselves and from our our peers and I started noticing that there was

you know random these were HTTP based so the URLs had kind of random looking pieces in them but there was a predictable structure and I could create a regular expression that would find them and given enough samples which we had I could tell you like for example this part of the URL ranged between five to eight characters long and it was always composed entirely of say uppercase letters so made a made a regular expression all of a sudden we started to be able to find their implant when it beaconed out and we had at the beginning maybe not quite 24-hour live coverage but we definitely had pager coverage and so we had the ability to like as soon as it beaconed out jump

on find out what was what was up interdict cut them off and so they didn't like that and they were like fine we'll just change the way we build our URLs suckers and okay we temporarily were blinded a little bit but we still had a little bit of that domain and hash and IP address knowledge not to mention the knowledge from our peers so we were able to quickly adapt and find out oh they just changed the way like the format string for the way that they built their their URLs we can make a new regular expression and again started interdicting them in the middle the next thing they were like okay all right you can see the way we make our

c2s very clever guys what if I encrypt it so started seeing SSL yeah that was a little bit of challenge at first except that we got a little bit lucky because um these were not native English speakers I don't know if that was a factor or not but when they've created their own SSL certificates they had a weird capitalization like a one letter that was not should not have been capitalized uh that was capitalized and it was like I can find that so we did and we started to be able to interdict them there now notice we started off with the the Intel at the bottom three layers domains IP addresses and hashes down here

I'm pointing to it like all the people at YouTube can see it uh domains IP addresses and hashes but as soon as we started collecting samples and doing the regular expressions and the the examining the certificates we moved up into this network and host artifacts layer after that guess what they were like okay HTTP you guys are really good at it props to you what if we change to FTP because remember it's just a bolt-on whatever surprise to them was by the time they were fighting us at the network artifacts layer we were fighting them at the tools layer because we had enough experience with this tool that we noticed no matter how they created the connections

the the data flowing over the connections was actually from the back end of the tool and we had identified a heartbeat that we could reliably find so as soon as they put out their new FTP version we were like we found it here it is jump in enter date I say we were at the tools level that was a little bit of a lie we were actually at the behavioral level the ttps because we had the detection there but guess what we also had we also had a lot of experience doing incident response with them while we were busy down here and they were kind of in the short term winning so we knew all about what they liked to

do when they were in the network so we were able to create rules and detections that would find their behavior so one of their key behaviors was they would take all of the data that they were going to steal from around your network put it into one place and a like a data staging area on your network use the rar chive utility break it up into exactly 640 megabyte chunks I never figured out why except if they were going to write it to CD-ROMs or something maybe could have been I mean that was a thing right um and and encrypt it and then they would transfer it usually over SMB but not always so we had a

detection looking for encrypted raw files going from um you know across the internals of our Network because we never used them for legitimate business purposes so while they were still fighting us on the artifacts layer we we had brought the battle to the tools and the and the ttps and there came to be one night I I don't exactly know what time but it was well after midnight let's say two or three a.m because that was probably it where we got alerts from both the the FTP and the behavioral things and we jumped on while they were still copying that data from the staging point to the place that they were going to exfil it from so it was still in our

Network they hadn't escaped with it yet and we were able to jump on in the middle of the night cut off their their transfer the pieces that they had begun to exfiltrate cut those off and all this time with all this experience we had with their Network C2 we actually even had a decoder so we could take the pcap we were capturing from our NSM sensors and run it through the decoder and see what all they were doing including when they were on the systems typing things so we cut off their command and control node and they they fell back to their their backup command and control node and started pinging it it's down oh no okay but nothing else

was down just their one control node so they were thinking they're watching us so the next thing I saw was the maybe the funniest thing I've ever seen in my professional life as they started um they started typing all these commands to see who was logged on to the machine the NT machine that they were on the Windows box like that's how I discovered that the Q user command existed by the way they were looking for are we being monitored on at the host level but they didn't really think about the network level and a minute or two later they logged off they never came back this was a well-funded state-sponsored threat actor that we

chased off our Network essentially even though we hadn't again I hadn't created it yet by putting them on top of the pyramid we chased them up a perfect example of how we chased them up the pyramid over time and they never came back and we knew they never came back because we knew what they did when they wanted to maintain access and we kept seeing that they tried to get that access back but they were not never successful at least for the next few years until I left the company the pyramid I'd never I hadn't written it down at the time but a couple of years later we had we had recently created our own

CTI team to replace my janky CSV file of everything we learned um we created a CTI team and we had a like a global meeting of the security teams coming up to headquarters because each of our business units had their own security teams or many of them did anyway so we had a big stakeholders security team meeting where we covered we intended to cover a lot of topics one of the topics we wanted to cover was our brand new threat intelligence database because we had just recently come up against the our a big major Milestone we had a million indicators in our database wait sorry hold on one million indicators which seemed so great and they were like

so that means you you have deployed these all to detection right and I was like hold up wait stop raise your hand if you know why I was telling them to stop yeah yeah Mike does the security onion folks all know this is better you have a million indicators which ones are good for detection which ones are not big clue is I already said all those hashes not great for detection usually good for threat intelligence and and crawling up uh threat actors on on stalking them on virus total and whatever not so great for detection but I couldn't really figure out how to tell them this and we actually spent a my mind was like in my memory I think it

was like the entire afternoon but it can't quite have been that much um and uh I couldn't get it across why we didn't just like deploy everything to detection so I was sitting in the hotel lobby bar with some of my teammates that night and we were going over to you know the meeting how we felt about it and I was like yeah you know something I just can't figure out how to express this so I had my laptop in my bag next to me and so I just pulled it out and it's yeah I'm drinking a beer in the in the lobby and I was just like I need to show them something so I started

playing out I'm terrible at at Graphics at all so my real only real option was Microsoft smart art and so I started playing around with some ideas and I was like oh yeah pyramid a hierarchy yeah I I have a hierarchy in mind so I'll just put them in the and that is the the secret origin of probably the world's most famous piece of Microsoft smart art is I had I had developed this over a set of few years by actually interacting with threat actors and and seeing that this worked and then at one day I just got frustrated not being able to say how I was doing what we were doing and why and so I created the pyramid and I

brought it into the meeting the next day and and that was it that was immediately they were like okay I got it thanks kind of anticlimactic end to that story but they were like thanks okay next item on the so I I have some stuff here to give out in my last couple of uh minutes some nice prizes from the from the conference sponsors I have uh an intrusion detection honeypots book Chris Sanders wrote this I've read it it's actually a really great book so I recommend it um also I have from from Southward I think right is this lock pick and beginning lock pick how-to book so I'm going to ask you a couple of

questions and the first person who can tell me the answer can choose which one they want and then the second person who can answer my next question is to just get stuck with the leavings first question is this is actually pyramid of pain version two who can tell me the difference between the original pyramid of pain and pyramid of pain version 2.

good answer not correct he says oh you must have had behaviors before ttps did you colors the colors were different the colors were different I mean technically yes but that's not the answer Point have IPS around did I have IPS and and hashes switched around no but good guess it's close to that less items in the pyramid but what was I missing it's not the ttps it was not the network artifacts come on three more at least right yes it was it was the hash values which which one would you prefer all right yeah the first the first version of the pyramid did not have hash values in it and the colors were different because

there wasn't a blue so all right um the next one I don't really know what to to say so I'm gonna say um this is probably gonna really be a disaster who has the best use story of the Pyramid of pain who wants to come up and tell me how they've used the Pyramid of pain and in in a real life nobody oh yes

[Music]

that was a great that was great so for for everybody who didn't hear it that he said he he was trying to convince his his team to to get beyond the atomic indicators my words but I think that's this yeah and into the behaviors and and he used the pyramid in in part of his presentation to show them why you should do that and that's exactly the reason that I eventually published it because I wanted to we found it so helpful I wanted to share and speaking of sharing I'm kind of out of time now but I'm here for the rest of the afternoon if you are interested in the pyramid you want to talk more about

war stories or if you just would like some cool stickers or buttons or whatever uh hit me up outside and I'll be happy to hook you up thank you very much [Applause] oh yeah