MOSE: Using Configuration Management for Evil

thank you all so much for coming out I can't really see any of you there's just a big light but I assume you're there and you're awesome so thanks for coming out and hanging out with me for the next 25 minutes we are going to be talking about Mo's which is a tool to help you use configuration management servers for post exploitation this is the mandatory legal slide that my employers have me put on here essentially any opinions that I express up here are mine and not those of my employer so I've been playing death-metal for several years now I used to tour but now I just play local shows every so often previously I've worked as a sysadmin pen tester and

an SRE and before I started my current role I built and ran the corporate red team at Sandia National Labs and currently I'm a penetration tester on the product security team at Splunk which has me finding issues in the various products that we make teaching secure coding practices to developers and automating vulnerability regression testing and I really enjoy automating things it's one of the pure joys that I have in this world but I have a special place in my heart for automating post exploitation tasks all right so on the agenda for today we're going to start by briefly discussing what configuration management tools are following that we'll get into how attackers can take advantage of them

we will then discuss what Mo's is and what we were hoping to accomplish with its creation and there will be a demo also some awesome dude game this like Def Con New Mexico sticker I don't know who you are but this is awesome thank you alright so configuration management tools they're used for provisioning systems so if you have a system that needs to be part of a cluster or it needs to run certain software or services you would provision it to do so with configuration management tools they can also be used to manage the state of systems in your environment so for example if you want to make sure that the service is always running or you

want to make sure that machines are automatically updated on a regular basis these are some good use cases for configuration management tools these are some of the more popular options on the market today we've got pop it chef salt and ansible and a ton of companies use a variety of the different CM tools that i showed on the previous slide so if you haven't run into them yet trust me you will alright let's talk about what these tools have in common so they all have this concept of idem potency meaning that you can run the code as many times as you want and expect to produce the same result you can also scale infrastructure while

guaranteeing consistency across it and you have the ability to provision machines that are running on a variety of operating systems so you could have a puppet module for example or a chef cookbook or ansible playbook that works on RedHat to say Debian Ubuntu Windows etc you get the idea and they all have a form of Secrets management which we'll get into a little bit later all right so a common question that comes up around this technology is who needs it we've got containers and kubernetes right and so there's a couple of things to consider here first and foremost not everything can be a cloud native ephemeral workload that runs in a container and it's typically not a

trivial effort to migrate older monoliths to container based deployments I mean he'll sometimes it's not even possible and you still have to install configure and manage your kubernetes deployment with something ideally something that's repeatable and can be tested regularly in an automated fashion like configuration management tools all right now that we discuss what these tools can do let's get back to talking about how we can abuse them the great thing about successfully compromising a configuration management server is that it gives an attacker the ability to run any command on every managed system this means that you can do things like introduce backdoors add delete or modify files you could just collect a bunch of shells I mean really the only limitation

is your imagination and so long as you're implanted code isn't changed or removed most of these tools have a built in guarantee that your code will run on a regular interval so persistence is effectively built in for you and another great feature these tools all have secrets management solutions most of the time do the nature of cm tools these secrets are usable across a number of managed systems subsequently stealing them can provide us with ample lateral and vertical escalation paths and so quick story time a while ago I was on an engagement and I compromised a Jenkins box through an open groovy console once I had my hands on the Jenkins SSH key I was able to get into around 15 or so

systems one of them was a puppet master and so from this system I was able to compromise hundreds of servers countless databases I got into the devs chat server which is always entertaining I got in his admin for their code repo it was it was a real good day for me and not as good of a day for the target but with this context in mind let's move on to talking about Mo's and where it fits into all this so Moses is an open-source tool that I came up with in December of 2018 and I released it and presented on it at Def Con 27 in August of 2019 so far we have a small team of core

developers and we're always looking for contributors and testers so if that's within the realm of possibility for you to assist with please help us moses is effectively a translator between a user and a variety of SIEM tools the idea being that we know what commands we want to run on our targets but we don't necessarily know how to implement them for a given cm tool most takes the commands or scripts that you want to run on cm managed systems and translates them into a format that a given cm tool can understand it's written and go aka going I chose going because it has a few really useful features first and foremost it enables us to easily target

a wide variety of operating systems and architectures it also has native concurrency and allows us to generate a binary that can run on a target system without having to worry too much about dependencies I was motivated to create Mo's for one main reason cm tools can be is difficult to exploit as they are powerful one reason for this is that each cm tool has its own learning curve if you check out the docs for any one of these tools you'll quickly see that there is a lot to digest and there are many tools specific idiosyncrasies and gotchas that you need to be aware of before using them they also have significant differences in their workflows syntax and architectures and

since you will likely encounter several different CM tools while on engagements or at different jobs you'll have to deal with this learning curve multiple times over you at the end of the day you can't choose what tools your target uses and so you want to be able to weaponize them all without having to worry about tool specific differences it's also important to keep in mind with great power comes great responsibility you can break a ton of systems while attempting to backdoor a cm server if you're not careful imagine accidentally taking down hundreds of production servers all at the same time if that's not a problem for any pen tester ever you'll be in a world of hurt and if you had to manually

do what mozo tomates for you you'd have to do the following to start you'd have to figure out how the CM server is configured and fair warning a lot of the implementations of these tools that I've seen like uniformity and how they do things apparently best practices are merely a suggestion for a lot of organizations so you'll need to figure out how they've done their deployment before you can get to the good stuff this typically involves a lot of finding and grepping and it can get to be time-consuming and torturous once you've got the lay of the land you need to figure out where your backdoored code needs to go and depending on how the environment is

configured this could be in a number of different places you could have a configuration management server that is supporting a number of different clients for example so if you put your backdoored code in one particular place it could only hit one client if you want to hit everyone you got to account for that and once you've got your backdoored code into place you need to figure out the file that or files that you need to reference a code in and this is the tricky part you make changes to those files in order to hit whatever host you're targeting so salt and ansible for example are Yemma based yeah mo is whitespace sensitive so in a less than

ideal scenario you could put in something that just breaks the amel so you know the the CM server breaks not ideal even worse you have a side effect where you you put invalid yeah Mille but it's not indented where you hoped it would be so essentially you end up doing god-knows-what to which ever systems so that's really dangerous you got to be real care for there so once you go through that you need to figure out where the secrets are stored and how to decrypt them and this method of course varies for each cm tool alright so that sounds pretty painful right fortunately Moses here to help to start Mose automates all the aforementioned steps for you which is already a massive

time saver in addition you know all those awesome post exploitation tools that we know and love Mose makes are real easy to deploy them on mass and for some of the CM servers you can strategically pick the nodes that you want to target for specific tasks so this could be uploading a web shell for all the managed web servers or popping a reverse shell and all the manage laptops and they come online or you could just introduce a payload specifically targeting manage Windows servers and so because Mo's allows the user to clearly go from thought to command without having to go deep in the weeds of tool specific details it helps to prevent someone less experienced with these

technologies from accidentally decimating a whole bunch of systems all once furthermore it empowers you to focus on what you know and not have to get wrapped up in the minutia of learning new technologies in a limited amount of time in order to achieve your goals and you don't need to know tools specific implementation details to run Mo's commands our input the same way for all supported CM tools the code that Mo's uses for payload generation has the same core pieces and directory structure which makes it clear and straightforward to develop and modify as needed so if there are users that do have CM experience you can use modes to customize your attacks further Moe's relies heavily on templating for CM code

generation so advanced users who have experience attacking CM systems can modify these templates to extend on the functionality that most provides out-of-the-box alright so this is the workflow for Moe's essentially again it's written and go so you have to compile it once you compile it you have this Moe's XE and in this particular instance here we are planning to run LS on manage chef agents associated with a chef workstation or server that we've compromised and for people that really like to get in there and tweak knobs we've got a Settings JSON config file so you don't end up with CLI soup so essentially once you specify this command you would run it and what will happen is it'll generate a

payload for you it'll generate a binary and by default it will stand up a web server that you can use to transfer that binary to your target alternatively you can just say that you want to spit it out to the filesystem and you can transfer it however you choose SCP whatever once you get it in place on your target you run it you'll answer a couple questions like do you want to backup all the important files on the system you're about to modify unless you're trying to be super stealthy like this is probably a good idea right so you do that and you're off to the races Moe's will introduce your backdoored logic and try to pillage all

the secrets all right so we're gonna do a demo and in this scenario we're going to leverage Moe's against a compromised chef workstation for both lateral movement and persistence compromising a chef workstation enables us to do things like introduce new cookbooks to the chef server which is the mechanism for managing chef agents we can also modify the existing cookbooks used by agents connected to the chef server and lastly this is where we can find and decrypt secret so needless to say this is the optimal system to compromise in a chef environment it's important to note that we are in a post exploitation phase of this engagement in the scenario here so it's assumed we've already done some of

our due diligence in terms of recon and have some understanding about the assets around the network

there we go

all right so to start we're gonna take a look at a payload that we are going to run against one of the targets this is just a reverse shell through Python pretty textbook so essentially we're going to take this the script and we are going to upload it to the chef server and then we are going to use it against a chef agent or several chef agents depending so this is us generating the payload and then we are on the workstation that we compromised we download the payload and run it and when we run it we see that there's a couple of chef agents that are associated with the chef server chef workstation dev assistant and web server we're going to

opt to just hit the dev system so we're going to pop a reverse shell on the DEP system and so essentially again the rebbe shell dot Sh that showed you earlier that's now on the run list for the dev system and it looks like we found a secret so we'll go ahead and hold on to that for later that could be good and so now typically a chef agents check in with the chef server every 20 minutes I've sped it up here but basically I set up a listener waiting for the agent to connect back and automatically it's connected back to me again if you're doing this in real life that probably take about 20 minutes and

we're just kind of looking around seeing some interesting things like this and we're also going to take a look at the hosts and it looks like there's a Maria DB system conveniently hooked up with this system so we can test out that secret that we found earlier and in case anyone's confused Maria DB is a fork of MySQL so that's why I'm using the MySQL a executable to connect to it that works great we can come back and pillage that later we also noticed there was a webserver so we're going to go ahead and target the web server next and in this case we're going to upload a web shell and then we're going to backdoor the

dub-dub-dub data user with my beautiful said so run the payload and what it's going to do it's going to generate the we're gonna run most it'll generate a payload that we can drop on the compromised chef workstation when we run it we can this time say that we want to target specifically the web server and one thing that's important to keep in mind here is anything that's kind of funky in terms of configurations on the different systems that you compromise chances are that's gonna be the same everywhere because again this is a cm managed environment so for example that permit route login we can assume that's on most of the systems here's the web server

that we're hitting and as you can see there is nothing to see here at the moment but when the agent checks in with the server typically in 20 minutes or just you know immediately because video we get our web shell into place and for the sake of this demo we'll say that this web server is externally facing so this could be a means for us to get back in if I our pops us and kicks us off of the chef workstation or the Deb system that we have a reverse shell - so indeed it looks like we're running nginx and the dub dub dub data users there so that was a pretty decent user to come from -

back door and we'll just check out our said work make sure that we did well with that it looks like that's in place now we've got the dub-dub-dub data user and if we check out the hosts it looks like this systems connected up with the dev assistant the chef's workstation so this could be a good means for us to go again from the outside in so now we can lay the framework for that essentially maybe using something like SSH so we'll go ahead and generate a rogue SSH key and a after this command so and again just to kind of reiterate this because we saw the permit route login again it's a cm managed environment so that's the

setting that's probably in place for all the system so again we can exploit things like that so here is the SSH key gen and we'll go ahead and grab the pub key and then get into place and of course because we have shells to the dev system and to the chef workstation we could just knock those and manually but you and if ir comes in and they remove our key from the authorized keys file we are kind of Sol but if we leverage cm thank you if we lever to cm server then even if they remove it it'll come back in 20 minutes and the same thing applies to our web shell and our reverse shell now

of course with the reverse shell they could just do something like you know blacklist our IP but with the web shell you know they remove it they don't check the CM server which chances are they they won't not my experience anyway they'll all come back so here we are just getting our payload to throw the SSH SSH keys into place for the chef workstation in the dev system and when those systems check in with the chef server then those will both be loaded up in the authorized keys files so let's look at this from the outside and let's pretend that we just we lost our shells and and just got totally kicked off the network so essentially we wait for the

web shell to come back we use it to get a reverse shell so now we've got root on the web server and because we also have those pub keys that are being kicked into the authorized keys files for us we can go ahead and just SSH into them without a password which is great and if it doesn't work the first time that probably means that the agent hasn't checked in with the server yet like here so you wait a couple minutes you try again and everything's probably gonna be dandy for you like that thank you all right now let's get back to

[Music]

all right so I imagine after that riveting demo you all want to give this a shot fortunately I've created test labs for all of the CM systems so you don't have to go through the pain of setting them up and they're all docker based there's one vagrant based one that's one of the chef ones chef does not like to be in containers but we threw some black magic made it work anyway so there are some dr. environments for chef and yes so please check these out and mess around with the tool also like if you use this an engagement before launching this against a whole bunch of systems run in the test environment make sure it works

properly how you expect it to and you can actually do the demo that I just showed previously by using the chef test lab repo and if you watch the DEF CON version of this talk you can do those demos through the puppet tests lab repo so we're currently working on supporting all the tools but we're not quite there yet Moses open source please help us get there with your contributions we'd also love some feedback from testers let us know that the tool is working out for you what we can do better what we've done great salts almost there we're not quite there yet but we're very close like to thank a couple rockstars in my

life al is the other contributor on the project and he's done some amazing work I don't know where you are but you're wonderful thank you and like to thank my wife for supporting me working a lot of extra hours outside of work and making this happen so thank you and I really like to thank the b-side staff for making this an awesome event year after year so thank you guys this is a real treat I love this conference so much so it's cool to be talking here all right so the QR code will take you to the master of servers project if you don't trust it that's fine there's all the links that you need

they're also real quick Splunk is hiring if you want to be my boss the project team is hiring a manager so if you want to be my boss there's my contact info get with me we're also looking for senior in principle level product security engineers well so that's of interest to you there's my contact info and I don't have time for quests Jen's maybe I have a time for one question thank you I that could be interesting we'll have to see how my employer feels about that good question [Music] all right cool thank you guys so much for checking this out [Applause]