BSidesPGH 2024 Track 2 ch3f The Whole is More Dangerous than the Sum of its Parts

not that anyone asked but I have to use the handheld mic because if I use a lav mic my beard gets all up in it and uh makes noises so and every audio guy I've ever worked with said oh we'll just clip it onto your beard now we're not doing that it is my honor to introduce to you Chef to talk about the hole is more dangerous than the some of its parts and anybody that runs any vulnerability exception process probably knows this pretty well so so take it away thank you very much good morning good morning I am Chef also go by Damon small and uh we're going to talk through U an interesting

attack chain that actually happened to me when I was on a job with a very large client I can't talk about who the client was um but I think it's an interesting narrative uh and we'll we'll get into it uh let me start off by saying uh I'm originally from Louisiana I've been living in Houston for the last 30 years and I'm it's great this is my first time at bsides Pittsburgh so I'm great to very happy to be here thank you all for having me and I realize I'm competing against lunch so I will make sure I make it worth your worth your while uh I've got a great employer and a great job and

a great career and my employer uh supports us me and my colleagues and being here and similar uh I well I'm hoping to get another one yeah so um that having been said uh I'm going to make you sit through a short commercial and thanks to my employer so I work for blue Bastion uh blue Bastion is uh uh the where the our parent company is ideal Integrations that's based right here uh in in Pittsburgh a lot of us that work at Blue Bash some of us are here in Pittsburgh and some of us are remote all over the place uh so blue Bastion I'm not going to read that off to you but you know check out our

website we Pro uh provide blue team and uh red team Services uh as well as some advisory services so um thanks for letting me be here um I mentioned the vcso and advisory Services that's the team that I work with mostly uh we've got a variety of services uh that are very interested uh that are kind of holistic in nature that include the business uh as well as the technology stack and happy to talk more about that later um but anyway again shout out to my employer um enough about that let's talk about me my favorite topic uh I've been in it or infosec for about 30 years like I said in the Houston area uh a lot of

healthcare uh uh and oil and gas those are the two largest Industries in Southeast Texas and so I've spent a lot of time working with those Industries as well as a little bit of Aerospace Johnson Space Center is down there in the Houston area uh and of course Consulting um so I've enjoyed that I've sat on both sides of the table right I've been a blue teamer uh I've been a red teamer and uh I've also been very involved in the Defcon Community if anyone's uh familiar with Defcon I've been a speaker there uh I've been a Shepherd just like Dr chick who we just heard which by the way don't ever follow her she's fantastic I really enjoyed

that slide deck so I'm just going to try and keep up with her um so she and I met actually uh working with the wall of sheep and the packet hacking Village and this year I'll be a goon so if you come see us in Vegas and you see a guy that looks like me with a red shirt on it's probably me doing the Goon thing native of Louisiana go Tigers any LSU fans here yeah wrong state for that right okay that's fine I got into Tech uh while studying music at Louisiana state long story but you know I'm still a frustrated musician now I'm computer nerd 30 years in Houston like I said and

yes I'm very happy to be here because hurricane Barrel was a pain in the backside that was not pleasant and I've been through plenty of hurricanes but I'm here so anyway that's me special thanks uh at being working with the Defcon and doing all those things uh well thanks to Blue Bastion and ideal Integrations as I said uh besides Pittsburgh I know how much work goes into putting on a show like this thank you all this is fantastic it's it's great oh and there's one of the organizers now oh no you gonna kick me off already no never first time speaker so instead of making him drink a shot like they do at Defcon Blinky cat

ears and he h there we go and he has to wear them the whole time during his talk okay thank you Cat how do I look is this good all right Q I'm going to go ahead and wear this on our next uh client call and I'll be sure to have my camera on yeah all right uh my wife and editor if any of you are content creators and you create content you have to have a good editor and I happen to marry mine she does a very good job of going through my slide decks making sure everything makes sense she takes out the stupid dad jokes I put them back in that's just how it goes and of course

you y'all are giving me the gift of your time being here today I very much appreciate it so thank youall if you read my abstract you might have seen it basically it's a complex chain of uh a attack chain scenario to demonstrate how own individual vulnerabilities themselves might not be damaging but when you have enough of them chained together uh you can go from these individual exploits that are kind of uninteresting to gradually elevating your privileges and eventually getting the keys to the kingdom and if I do my job well the point I'm trying to drive I'm going to tell you now the the the the exciting ending the lesson industr uh illustrates that managing risk is more complex than

understanding the threat landscape but rather understanding it in the context of the entire Enterprise and that the vcso and advisory services that I work on that's a big part of what we do is understanding those threats and those vulnerabilities in the context next so what this presentation is and and isn't I'm going to talk about a lot of exploits and a lot of vulnerabilities I'm intentionally not going to go into very very low-level detail on how some of these exploits um took place I'm more interested in showing that because of one exploit we got this level of access and then we pivoted and found another vulnerability that we exploited and so on and so on so um there's going to be

some assumptions I'll make but I'll do my best to explain things so it's relevant in that way and also um there there are some steps I'm going to skip over so as as Dr chick said when she was up here uh you know sometimes when you're going through a narrative like this time is compressed uh just a little bit so I'll point that out uh here we are here's the tldr on it this is a recap and I said this is an actual and forgive me folks I tend to Pace a lot um when I'm on stage I'm going to do that I'll do my best to not fall off uh it was an 85 day engagement that took

place in four business weeks so 20 days that we had we had multiple Consultants working on it so we were able to get 85 days of work done effectively in 20 days uh I had on my team I had a blue teamer three red teamers a systems architect and a wireless engineer often times when we engage in things you think that okay this is not going to be successful un unless we get domain admin right that is one goal for sure but all the steps that we took to get there tell an interesting story so if ever if you yourself are a consultant or a red teamer pentester if you don't get da that doesn't mean that the engagement was not

worthwhile right so I I want to just emphasize that point we tend to focus it's easy to focus on individual attacks rather than the systemic issues that led them to exist so as we go through this narrative you'll see sometimes it was uh technical issues vulnerable vulnerable software or misconfigured software uh sometimes human error was involved somebody on the client side made a mistake and we use that to our advantage the result of this engagement when we were done with the readout and everybody was fine our client contact uh said this is going to take us about three years to fix all this stuff that you guys found breaking is fast fixing is slow and I tell my clients all the time my

job is easier than yours because we do the red team we do the gig we drop off the report and we get out meanwhile y'all got to fix all the things so breaking is fast fixing is slow and that's that all right let's jump into it now I realized the text over here is not very easy to read don't worry about it I'm going to describe all that stuff this is actually a screenshot uh well heavily off fiscated as you can see I can't give away the client name nor the name of the company I was working for when all this actually happened because this this was a number of years ago um so we're going to step

through these steps but this I I don't know if anybody else in here is in consulting or um you know has done a tack chain type work like that anybody I I I'm sure there are more complex attack chains that exist but this one was impressive and I still think it is so nine steps as I said so let's let's start talking through them all right the very first thing that we did oh and by the way it's important to point out uh that when we arrived at the client site we were on site with the client it was an assumed physical breach so the client said we're going to assume that somehow you social engineered your

way into here so we were already in the building other than that all we had was ethernet access we took our machines plugged them into a port we we did have an IP stack yeah they gave us that but we didn't have domain join machines we didn't have credentials in their active directory nothing we had layer three access that was it so starting with that we started enumerating we started scanning all the things to see what IP addresses were there and try and understand what network Services were available you know the usual sorts of things pardon

me I do love the sound of my own voice but man you get thirsty when you're talking so we scanned around found some HTTP um Services running and if anybody is a developer in here you might be familiar with the tool called Jenkins as you can see up there if you conf figure Jenkin by default there's a vulnerability that will allow you in that console let's see if my I thought there was a laser on this thing oh there it is uh you can see here there is a place that you can enter commands um to do things unauthenticated we were not logged into Jenkin at all that's just the page you get when you go

to Jenkins um and the listing one here shows the payload now here's one of the thing I'm not going to go through exactly what this does but running that command gave us a reverse shell on the system that was running Jenkins as the local system account so at this point we had pretty much full access through this reverse cell Shell through Jenkins and we could run commands as system so we made a little progress now we've got access to this one box and then we needed to figure out okay now that we've got persistence on this box through a reverse shell what can we do with that so see the arrow there we're moving on

to step two so we continued enumerating uh and we found a file server oh and by the way for those of you I I mentioned that um I Opus skated all the important information uh if you're not aware if you're not a network person that's fine all the IP addresses you're going to see are uh rfc1918 addresses 10. something those are not routable on the internet so it's impossible to use any of those IP addresses to figure out who these hosts are because anybody can use 10 anything so we went after the file server and uh on that file server as we were poking around um we discovered that it was running silence um which is a tool that's very

good at what it does it will prevent uh people like me from doing snarky things like scraping credentials out of system memory uh and and a variety of other things so silance is pretty good at that um but what we discovered on this particular machine on the left uh it had a different policy than all the other Machines That We examined and we're like that's interesting by messing with that file server with the incorrect silance policy uh we were uh able to execute mimic cats if you're not familiar with mimic cats it's a tool that will examine memory of a running server and it will extract hashes that somebody has used to authenticate into that server or workstation or or

whatever uh and because it runs in memory mimic cats runs in memory silance normally detects it and will not allow it to execute but again because this policy just happened to be out of date we were able to execute mimic cats and extract hashes uh from uh this machine now if you saw my buddy Q his presentation and and Corey were on stage talking about some of that stuff and what you can do with hashes so we got a little lucky on this deal had that device been configured properly we wouldn't have been able to to uh to do this however in this industry as many of you know I'm sure sometimes it's better to be lucky than

good so we got lucky uh moving on to the next block this was a little complicated because one exploit it was actually two exploits that resulted in a third so that's why those devices are kind of you know steps four and five U had to happen for us to uh get to step six so let me go into that one okay so we um were able to compromise All Join domain join servers so when we ran mimic cats on that file server we got it wasn't domain admin but it was a valid set of credentials that we could use to access any device that was joined to The Domain in which that account lived so

there's a lot of text up there um but basically by using that one set of credentials that we had we achieved remote code execution on nearly 7,000 servers uh by scanning for uh SMB on Port 445 with mass scan um and if you haven't used Mas scan I mean there's a variety of ways to do this to enumerate networks inmap is a very popular choice uh Mass scan there's other things that are newer tools and I'm drawing a blank on the name but Mass scan will very quickly fling a lot of packets around a network it's worth pointing out at this uh at this time that when we were this this organization was so large uh that they were nervous

they knew that we were going to be doing this so they gave us another uh account they gave us access to their network monitoring system specifically we wanted to watch the CPU utilization of their core routers because as we started flinging packets around we didn't want to interrupt this organization's business right so as we were scanning and flinging packets we were watching the utilization on their routers and that'll be an important detail in a minute so if you can see that payload can you'all even see that laser pointer by the way is it yeah you can okay cool uh we executed this and again I'm not going to go through the details uh but we were able to run that and uh

execute a command with the SMB service running and it uh downloaded mimic cats into memory to avoid any virus and uh we actually the guy he was sitting right next to me who ran this he took the output of that and um he had already stood up a little file server of his own on the client's Network so he would take the output and dump it to that file for later examination and with that we got 1, 151 unique plain text domain credentials so huzzah now we've got a bunch of accounts that we can play with moving on and so now we had more credentials that we had before so again this whole time we're enumerating oh and

when I say it was a big Network when we started off uh they gave us it was five or six Class A subnets that we could scan they were all internal they were all 10. something but if you do the math on that each class A has what 16 million or so IP addresses so there was a massive IP space and I think we were enumerating throughout the entire engagement you know just as we would scan I would take results that I thought were interesting hand it over to my colleagues on the red team and we' try and go so it was it's not like we scanned multiple millions of IP addresses you know it it kept going

on forever so this next one is interesting so now we had valid uh a bunch of valid credentials and we found a system on their network uh called tanium if you're not familiar with tanium it's an inpoint management software so it will sit on your network and it'll keep track of what devices are on your network lots of ways to do that in addition tanium would uh tell you things like which users were actively logged in to each device if there were local accounts on it what the local admin uh group membership was and those sorts of things and you can see by that query that we uh pasted up there that was the query we entered get logged in users

containing what I had to outus skate that from all machines I think that was uh a group that had local administrator access uh on all devices so the reason this is interesting even though we were authenticated to the network at this point with valid credentials that we had taken tanium is a very any type of endpoint management software is very very useful but we the credentials that we had were not necessarily part of the IT team right this was just a normal user within this organization and we saw this and this is weird this should be restricted to only people whose job it is to manage devices as in the IT team everybody in this

large organization has had access to tanium so that was one of those moments you know don't mind if I do we thought it was I don't want to say it's funny but it's funny we used the client's own software to gather more information about soft Targets on their Network so we did a query to say who are all the uh administrative users essentially on this network an tanium did its job and gave us that information so at this point we knew about a bunch of machines on the network we had a bunch of regular user accounts now we had the uh usernames of people that had domain admin um privileges armed with that information and and and by the way

I should back up for a second it's not like we knew that this is how the attack chain was going to come out just every time we would get a little more privileged we would just keep looking and looking for something that we could use so it's so that's why it took 85 days and because every time you get a little step further you've got to keep examining the network and trying to figure out oh here's some new information can I use that to my advantage so um now that we knew who the domain admins were uh we thought okay that's a soft target we need to find the computer that these domain admins use now the

best practice as I'm sure you know when you're doing just normal office work you don't use your privileged account right you only use your privileged account when you're doing administrative duties for your day-to-day stuff you should use your non-privileged account but we knew that if we could find the workstation of a domain admin there was an opportunity for sensitive information being on that device right so that's this step number seven we found a workstation that had been logged into and appeared to have been assigned to a domain admin so we went to see if we could gain access well we already had access to every domain joined machine as I mentioned so then this happened uh one

of my colleagues um using a couple of different tools PS exac and interpreter was able to get a reverse shell on that domain admin's workstation and the screenshot you see there is when he basically did a dir a listing of files that were on the desktop if this hasn't jumped out at you already whoops sorry pushed the wrong button where did I go okay see I tried to get clever those are actual files that were on this user's desktop look at that last one there credentials something something admin do- user. CSV okay what do you think we did with that file yeah we downloaded it in a hurry if you're G to keep creds in plain

text on your machine first of all shame on you but if you are going to do that come on guys don't name it admin user like that was too easy so we downloaded that file discovered that that CSV that spreadsheet had a lot of credentials and this particular organization used a lot of virtualization right so this allowed us to log in their server farm and have our way with it uh which we did and that led us to uh the virtual server Farm as I said so we had the admin credentials we logged into their virtual server Farm a lot of their domain controllers had been virtualized so when we got into the hypervisor we saw all these DCS sitting

around there were multiple of them so um we we shut down one of the domain controllers and since it was a virtual machine we cloned its disc um and attached it to a new VM that we spun up uh for the purpose of mounting that volume from the domain controller which As We Know the domain controller has all the usernames and passwords in in a file um and we were able to use that hypervisor that we spun up to start cracking passwords so I had to I can't show you the command line that we used uh to authenticate and to do all these things but we you know like I said we gained access to a domain controller shut it

down cloned it spun up a new uh virtual machine and use that to um to crack the hashes so this is the second time during this engagement that we use the client's own resources to our benefit the first time was with tanium right we gathered information and now the second time we use their own virtual devices to crack hashes so we didn't have to even bother with using our own Hardware or anything we're just like well we've got access to it why not use it adding a little insult injury so here's uh the the listing of how we um mounted the volume on uh of the domain controller to our new machine and use the packet to um excuse me a python

script to manipulate it further and eventually start getting um hashes for administrators that that that was we already had uh non-privileged user accounts so we were really interested in uh getting further administrative access remember on the domain admin uh user whose account uh excuse me whose workstation we accessed uh it was using his non-privileged account so we didn't yet have administrative access but we are getting close and finally here is what happened Manel still here and I know we're all hungry thank you uh in the very last step uh we were able to log again to um a domain controller using uh some stolen credentials that we had cracked using their own virtual machine um and then as you can see

there we uh connected to it made sure we are on the right device cool H probably should have taken that user out um and here it is here this was we had three business days left on the engagement so this was like a Tuesday or Wednesday and the gig was going to be over on a Friday we did the net user command uh added our added our new user to The Domain and that worked and then at the very bottom here yoink you can see we added ourselves to The Domain admins group um when you finally get domain admin as a consultant consult you know you are working for the client I find it in poor

taste to say nener ner ner we got domain admin however when the client wasn't looking we did have a big group hug because it had been a very long month of a lot of us working together and we finally got there um so yay and there was much rejoicing all right let's pause for a second so I've talked about the level of access we had when we started which is basically just Network layer 3 access uh we slowly got system level on a local box use that to Pivot to another device and found the silence misconfiguration and so on and so on so every step along the way we just had a little bit more

access here if you don't remember anything from this talk uh please remember this there were lots of failures and as I mentioned in the beginning they were not limited to individual exploit exploits I mentioned to you that they gave us access to their network monitoring system so as I was flinging packets all over the network I was very concerned about taking down their core routers we started we got to what would I say oh yeah 20,000 packets per second which that's that's kind of a lot that was enough to raise the overall CPU utilization on their core routers by about 10% that's when I got nervous I thought all right a 10% increase we probably could push it a

little further but I didn't want to and anyway 20K packets a second was pretty good no one on The Blue Team noticed that I was sitting there thinking man surely someone's going to notice wow our every single one of our core routers just spiked by 10% for no reason why is that happening we managed to evade detection and we weren't going low and slow trust me when I say I just told masan like go nuts man um and we did uh configuration management that Jenkins um uh issue that I talked about you can plug you can fix that vulnerability by changing the configuration but if you deploy Jenkins on your Enterprise in the default config that's what's going to

happen to you so there's configur configuration management issues uh the technical control that would have prevented us from SC scraping credentials out of memory uh was deployed it was not deployed consistently uh and and again we admit we got very very lucky by finding that that one device uh policy and procedure at some point along the way we discovered there was an awful lot of password reuse uh that helped us in stealing those credentials uh privileged credentials stored in clear text uh like I said that poor admin who had that file on his desktop just plain as day uh that was against policy and it's probably worth pointing out as well this organization although I can't say who

they are they are very large and otherwise very very sophisticated we were surprised that we found some of these defects uh while we were examining them uh as I mentioned our reconnaissance included using their own management tools those uh the uh tanium product should not have been available to everybody on that Network and it ended up giving us a critical clue that we used uh along the way and monitoring so I mentioned there was three days left on the gig and we created a domain admin account and it was obnoxious I I mean it was very obviously not an authorized domain admin account it was clear as day you could look at it and you know it was

something ridiculous like haha killroy was here or something you know it's like they they would have known but three days later on Friday during the readout that account was still there and it was still active um so we brought that up what could the client have done differently I mean each step along the way and that was in the report I'm sure you're seeing it just as plain as day I mean all those things that I mentioned uh blue team if you're a blue teamer I get it you have a hard job because the mail never stops coming right every single day you've got to protect your information assets but be if you're a blue BL teamer it's not just about

looking at the alerts from in in your sim there are other things that will give you clues about the health of your environment and that CPU utilization of routing and switching devices is one of them configuration management again you know if you're going to let your developers deploy that because uh that first instance that I talked about it wasn't like on a server available uh to um a lot of people it was installed on some workstation that a developer was using if you're going to do that make sure you have policies and procedures to support the appropriate config uh memory scraping again I we we were very very lucky on that um but obviously one device didn't have the

correct uh policy applied and policy and procedure no one likes policy and procedure it's not sexy but as it turns out it's very very necessary uh we they should not have allowed the password reuse and storing uh those credentials in in uh clear text and um yeah so there's a variety of things that the um client could have done interestingly during the readout uh you know so I I was the uh Team lead for that engagement so me and my team were sitting on one side the client sitting on the other and there was this one gentleman uh throughout if you work with clients you know you can read body language right right so I'm sitting

there going through the thing using all the big words and everything this guy is sitting directly across from me just kind of look in and I'm like man this cat does not want to be here and so I got to the part about the domain admin group and was like by the way we checked this morning before we had this readout and that account is still there oh I forgot I had these on see if I can get through TSA tomorrow with these um we said hey you know you might want to have a system that will monitor the membership of your domain admins group yeah we have that like oh uh well it didn't work what what

happened yeah um that system went down a week ago and no one noticed like oh uh all right so I don't know do you have to have a monitoring system for your monitoring systems I mean how far can that thing go you know what I mean but that's you know if you have systems that are critical in monitoring things somebody or somebody's need need to be watching that because they would have caught us pretty quickly had that system been running I have not heard from that gentleman I I thought I was going to get Christmas cards and whatnot no he's not a big fan of my so again as I said on the outset I think

the individual exploits that I talked about are pretty interesting and any one of them could probably a be a longer form discussion but what's really interesting is that each of them in aggregate led to us becoming domain admin on this machine and and like I said they estimated that it was going to take them uh three years to fix all that and again I will close with my little catchphrase there breaking is fast fixing is slow does anyone have any comments or questions or do y' all just want to go to lunch yes yeah your team yeah it was uh me and five or six other people yeah me and six people so that's how we got the 85 days

delivered in 20 days yes uh so this might just be on my mind because a talk I heard this morning when you found the CSV file with the plain text credentials um was your team concerned at all that that might have just at that point been a Honeypot and if so um how do you go forward like do you just go forward and hope that it's not or do you look at that differently if you think that it might be a Honeypot that is a very good question the question well he had the microphone uh yeah were we concerned that it was a Honeypot because we knew we got the host name Andor IP address of that device

from tanium it could have been a honey device anyway it could have been planted in there for us to find it um and it's it's you bring up a good point it was such an obvious and bad file name that it could have tricked us and certainly if we had interacted with that device Honeypot could have alerted somebody that something nefarious was going on we took our chances and grabbed the file and then when we in fact were able to log in to the hypervisor to get exit so that thought did cross our mind and to be fair uh The Rules of Engagement for this client were that if we found something that we thought was egregiously bad we

were supposed to you know ring the bell and tell leadership at the client that we did this I felt awful because we rang the bell on that one we said like look dude you got admins storing stuff like this on their desktop that is bad um he became red in the face and left and I'm like look man as a red teamer I don't or if we're doing uh uh security training the goal is never to get anybody in trouble or get anybody fired but the the manager was enthusiastically pissed uh when we told him that I don't know what happened to this I I hope this person is still working there and still has a job

but anyway great question that thought did cross our mind but we were pretty confident in the data that we had leading up to that anything else all right oh one more yes hi uh so you talked a lot about how you used um internal Management Systems to be able to like act yeah so um just based on that uh is this something that you commonly do are you used uh have you used other like living off the land techniques before um and if so like what are maybe some of the more out there ones that you have used let me make sure I understood you you're asking so since we use the resources against them is

that a common tactic that we use that more just in terms of do you commonly use living off the land tactics yeah oh oh uh yes is the answer however when we're doing that it is still an opportunistic uh activity right we don't know what's out there um and and so um yes living off the land as you say that is a common thing and if we can get access to that we are going to use it hi yes um can you explain your coat thank you Michael uh I can't okay so my wife whose handle is seams she is a seamstress and we've been married uh during Defcon we will hit 13 years of being married but she

specializes in women's clothing early in our relationship I was like honey you got to make some clothes for me back 30 years ago when I was a CIS admin I I worked in a data center where it was a little chilly so my sister got me went to a medical supply store and got me just a regular lab coat which she then tie-dyed for me so I had this tie-dye lab coat and so my wife said well I don't know how to make men's clothing but I have a pattern for a lab coat since you used to wear one so she made this lab coat for me and then I started I mean this is my walking ego wall I

have my first speakers patch from Defcon here I've got my little chef emblem here my three favorite Flags United States Texas and this one I'm told looks like the Philippine flag but it's not it's from a region in southern Louisiana called Acadiana so I just decorate it Michael with things and oh what do I have on the back of it oh yeah my this was my failed blog experiment security kitchen you know chef and all that I did I discovered that making video blogs is an awful lot of work and I stopped doing it so anyway just for decade now I've been wearing this thing and if I get in TR this is our CEO by the way so hi M if

if you hit me up about the dress code I'm going to have to remind you that look for 10 years people recognize me because of this coat so it's you know I I am a fashion statement okay maybe yeah or as my wife would say I've got a face made for radio you know anyway y'all uh it's lunchtime thank you very much for sitting through this I'm je I'll be around all day y'all are awesome enjoy it