GF - Rolling out the C2: A Take on Modern Red Team Infrastructure

uh hey everyone uh Welcome to our talk uh is everyone having a good beights this year hell yeah awesome glad to hear it uh so it's actually George's and mine uh it's our first talk so uh yeah it's a special moment and yeah thanks for being here it's going to be fun so uh with that out of the way just some quick intros uh my name's arv and like I said this is George um I've spent about 5 years in offsec uh four years in Consulting and then I switched over to the red team here at Costco uh which is where I currently work uh yeah like uh my my nerd credentials like uh I just

like any type of World building in science fiction media big soccer for good World building so if you know any nice fonts and stuff um I'm all about it so um and most importantly for this talk I am not a devops person I not a devops person so grain of salt with uh whatever I talk about today so George yeah don't believe a word we say um basically U my name's uh I've been in offs SEC for about 3 years uh that's not to say I haven't had a career before that it just took me about 15 years to get into information security so I was a a convert from audit risk and compliance um and it was a

journey so uh I'm really happy to be here it's definitely I think the highlight of my career uh working in researching doing things like this uh and being able to actually wake up in the morning and and really enjoy going to work so um I've had various it roles been administrator depth uh risk obviously looking at process looking at all kinds of other aspects of the business um and something about me personally I love I love cooking I I can't say I'm a very good cook but I definitely if I like you I'll probably bring over something that you think is tasty or at least make the best attempt to do so um and then uh again caveat uh

I'm not a devack Ops person uh this uh there's there's definitely a lot of stuff that has been uh Googled and researched and uh thank you chat GPT uh definitely helped facilitate this in like a two-month time frame versus a year or so um so yeah yeah big shout out to open AI um so um I hear uh hackers like Jeopardy so I have a fun little trivia question for you guys next and it's related to the name uh can like we work at Costco so that's your hint uh does anyone want to take a stab at why it's called Buck fitti what is the H yeah exactly it's the hot dog and drink combo a staple of

American culture so uh yeah that's kind of where the name comes from and uh yeah like our North Star for developing this was really like one of the quotes by uh one of our former CEOs which was you know if you raise the price of the hot dogs I'll [ __ ] kill you but and and I hope is there anyone who works at Azure here anyone who works at Microsoft anyone no okay well this this quote is for you guys like you know please don't raise the price of the VM like this whole talk depends on it so um so yeah let's uh get into sort of the overview of uh bucket you like why did we make

this tool right and uh just to like sort of uh set the stage a bit uh what do we do as a red team here at Costco right like we have essentially three primary functions right uh the first is red team engagements like spoiler alert and the second is penetration tests and the output of both of those types of engagements get get you know fed into a reporting Pipeline and we deliver reports to the teams that we work with right so that's kind of like our our main like our main goals uh as a red team so when I first joined Costco and I think GE George has only been there about a year or so more than me uh so I

started working there about like nine months ago uh one of the big challenges that we had with our current infrastructure setup was that it was static it was inflexible and it was opaque right so when I say it was static I mean that the infrastructure stack itself didn't lend itself uh a lot to experimenting to iterating to building like new components and adding them in uh inflexible in the sense that we we couldn't just turn it off between engagements like it would it would always stay up and uh you know why would we want to waste money on infrastructure that's constantly running if we're not using it at the time right and opaque uh so people come and people go uh you know

they switched jobs so the people who developed the infrastructure ended up leaving uh Costco so uh what we were left with was this sort of really brittle tool that we couldn't really experiment with but we were super scared if it ever went down because you know you don't want to break something and not know how to bring it back up without the right documentation so those were the three problem like sets that we identified and that we sought to address with developing Buck fitty and and I think these and I think I don't think we're alone in in this kind of issue across the industry right like I think a lot of teams find it difficult

especially in larger more mature organizations to to be kind of uh have a startup mentality and be able to spin up things uh and have a lot of flexibility and not be part of an oversight Board review something just to have like you know a VM spun up with a a public interface so um definitely applies it's not just Costco I just want to say that oh and we don't we're not representing Costco like on this top we just work at Costco just for the lawyers uh yeah uh this is our stuff yeah so yeah we definitely felt a lot like this uh like every time we looked at our infra we were just like how does any of this

stuff work uh so uh we had sort of like the choice between reform or revolution right uh we could either like paper it over with Band-Aid Solutions like incrementally change things around try and figure out what was happening or just re-approach the problem and come up with our own uh you know sort of solution that hopefully wouldn't take up too many Dev Cycles right which is what I'm going to get into next so our goals were uh for it to be cloud-based uh easy to understand uh modular and you know have it is technical debt but we wanted it to be worthwhile technical debt uh so drilling into those uh cloudbase really works for us because uh it allows us to

be very ephemeral with our stack we can spin up and spin down as we see fit and you know uh one of the big advantages of using cloud-based compute is that you only pay for the compute that you actually use right like as opposed to on-prem Solutions uh and uh easy to onboard team members onto uh that's so we made this tool very e very TurnKey so any like you know even if George and I were vaporized tomorrow uh like other team members on our uh like other team members would essentially be able to download the uh the code and just hit terraform apply and like it spins up right no Arcane knowledge required uh modular uh so that

we could experiment like we can plug in plug out different components uh you know just experiment with different tools uh anything that we wanted to uh try out like see if it works for us and then like take out if we don't like it or keep it if we do and uh we initially we're not a big team right like we're like around eight people so technical debt really weighs heavily on us so we we didn't want to take on technical debt uh that wasn't worth while right so we're not trying to avoid technical debt entirely but we do want it to be like a tool that's actually needed that's not just like a hobby project uh you know uh

so it has to be worthwhile like it has to be worth our effort and our time so getting into the design philosophy uh are there any devops people in the room Dev SEC Ops okay cool uh so this is for you uh this is like I am like you know I all of my knowledge about terap which I'm going to get into all of my knowledge about all of this stuff uh definitely like I got it within a two-month time period so I'm sure we did we we made some mistakes as we made this yeah we need help yeah like so come talk to us afterwards if you know or you know shout it out in the question section uh

if you have any doubts or if you see any glaring errors um cool so uh getting into the stack of things right like we're we're this is an Azure Tool uh this and the reason it's Azure is just because like you know our company's going to pay for it uh that's the that's the only reason we definitely like I I'm way more familiar with AWS uh than I am with Azure but uh you know it's this is the cost sync that it goes into so uh that's the reason why we went with Azure and we we're going to get into this uh at the conclusion where we talk about some of the improvements we want to make but we definitely want this

tool to be multicloud like Cloud agnostic we don't want to be locked into a specific uh uh like cloud provider so uh that's definitely in the Horizon uh cheap compute uh yeah we definitely wanted this to be like uh bank for your buck uh we we wanted to use the bare minimum amount of compute necessary to actually accomplish our goals and this is a really great website um I'm not sure if you guys have heard about it but Cloud price.net uh it's kept uh pretty up toate and it's sortable and you have pricing for Azure AWS and gcp so it's a great resource it's uh definitely resource we leveraged in terms of trying to come up with what

kinds of uh you know virtual machines we wanted to use uh during uh like while we were building this Tool uh terraform was really uh sort of foundational to this whole project uh it allowed us to think modularly about how we wanted to develop uh components off the stack and I thought it was a really neat way of approaching the problem where you don't you're not really stuck with a manual way of doing things you don't have to go into like the Azure CLI and like you know set things up uh in a manual I mean you could scripped it out but I feel like terraform handles you know the how parts of interacting with

Azure uh and we can focus on the what like you know what what is the end state that we want right and that was really valuable to us at the sort of development devel developmental velocity that we were going at and uh yeah it's easy to iterate upon uh you can just you know experiment like that that's entirely like that's how Buck was built like we just kept like experimenting with different uh you know terraform config files uh seeing what worked what didn't and yeah it was really it was really useful for us to kind of iterate quickly and I also think it goes to the uh the not the tribal knowledge kind of like alleviating that tribal knowledge

issue because you have infrastructure as code you don't have a bunch of guessing like well this run with two CPUs will this need like multiple discs how much RAM um I need to call you know the guy that just left for Fiji quit the company uh to figure all that out um Tero provides like a kind of a unified kind of source of truth of what it takes from an infrastructure level to actually build up this this platform and the tools that you know this red team needs to continue operations yeah definitely it was it's super readable and anyone can read terraform code it's uh very easy for humans to parse uh so it's definitely useful for you know uh going

back to one of our goals which was we wanted it to be easy for our team members to you know decipher uh so it's it was really handy for that as well like anyone can look at terraform code and like figure out what the gist of it all is um yeah George you want to talk about tail scale yeah so jumping into tail scale I know uh just to give a little bit context take a back step um I know a lot of the the overview of this talk is very tail scale Centric um and when we got the email that our talk was accepted um we were like oh oh crap we have 2 months let's um let's maybe like

do something on top of tail scale so we're not just you know talking about like an ivory Tower this is what we could use tail scale for we actually developed a platform with tail scale being the kind of hinge pinge of the networking and tling aspect of the C2 infrastructure and the other like operational related services that we've built into kind of like this demo uh and we're going to demo like what you can do actually live with uh an overlay Network kind of topology uh used for tunneling uh as part of red team operations or pen testing as you compromise assets and pivot within a network and want to like tunnel out or remain silent um that kind

of tail scale was the foundation and we're going to build up on top of that in this talk so um so on that note tail scale it's a it's a great service uh it's an overlay network uh we looked at a few different competitors uh like nebula from slack uh tail scale seemed a lot more featur rich with specific tools and baked into the client for things like net catting uh once you've established a connection uh file sharing was really straightforward within that wire guard tunnel it's all very encrypted uh I I really from the experience I've had with uh using tail scale as a way to access dropboxes within a network work it it's it seemed

very almost too easy to get in and out of of a an Enterprise environment not saying that Costco was that environment but I would say of the many places I have worked which were fortune 5500 companies I think the the natural aspect of overlay vpns that have coming out recently in the last couple years are really um are really I don't know a lot of ways to avoid that being able to get in and out of almost any network that's routable from the internet that I've seen so um I was excited to start deploying it within um as a as a backbone for other services and I haven't had any problems as of yet so um

yeah moving on from tail scale uh as a VPN service um we also wanted to kind of stick with the modular context of our platform cuz if we did all get vaporized me and arv as The Architects um we work with very competent people and and techn Technology Savvy um folks on our team but they they are amazing at things like developing malware uh they really want to focus on what they love doing on their team and we love doing architecture and solving problems as well as doing pentests and other things but uh we found a niche that that really was a support role with building out tools and and talking to our operators and asking them what what would make

their lives easier and then trying to build that out within Azure or whatever and and Docker seems like it was a perfect method of getting an idea from an operator uh finding a like a proof of concept open source solution to that that someone spun up uh making sure that it's secure enough to implement for operators so we're not piggybacked on by real malicious actors uh and then deploying that quickly without a lot of uh development overhead like on the operating system level of a VM so uh we could pick and choose from a few competitors for a specific service throw that into a platform and have it play nice with other services uh with very

little overhead and that can be easily translatable in the event that somebody does leave the company um to to someone who's technically Savvy knows what the service is doing and can kind of like you know Google the rest of the way there to either spin it up replace it or you know update it as needed so um Docker really helped with this environment uh and it's also integrates really well with terraform and that's pretty much why we chose it open source components in general um we we wanted to stay cheap we did this research on the side it wasn't sponsored by C um so we wanted to stick to the B 50 rule very closely um we got it down to

about 150 a month or so if like you're live no not even that 50 bucks like 50 bucks yeah okay wow I'm I'm glad I didn't open up Myer account it's it's all our up um so yeah about 50 bucks a month to run like a fully functional platform that you can do pen testing off of or red teaming um with a lot of Open Source tools so like zero costs on the open source tooling and amazing Community out there for security especially like the the even the like tail scale op Source their protocol and within a m like a year or so uh there was just really great tools out there that employed that protocol uh and you

can spin up like lookalike copycat tailscale clients and servers uh so you didn't have to go through tails scale.com you could have as many users as you want um if you had the kind of the motivation to configure it all yourself so you definitely get into the weeds with open source uh sometimes there's a lot of uh support in forums and guides and things like that and sometimes there's not and there's not at all or it's completely wrong and you're like I got to reinvent the wheel here but um you know the I was surprised I mean I was really excited to get onto a red team at an Enterprise and have like an open checkbook for things

like Cobalt strike and all the tooling that you know I imagined would make life super easy and and then when I started doing the research and I got to know those tools and started doing the research with the open source Community I found a lot of comparable things to some of the really Advanced tools that we saw um within the Enterprise that had like counterparts that were completely free and and really secure so it was it was a it was a fun two months I learned a lot um and I fully support open source components to cut costs and also stay really safe and secure so um yeah so back to back to my um prior

comment about like we had uh we were coming up with all these Grand schemes as soon as we got the acceptance email for the talk and we were like Let's do let's do like a cloud agnostic terraform Bild that can be like deployed on any cloud service whatsoever that spins up now or like historically and like a bunch of like really cool things and services and and then like after like four weeks of trying to make it all fit together we're just like okay let's get this over um you know no fancy stuff we'll just get the the the operator tool belt of services so that uh you know if you get if a red team gets fired or the

company closes doors they can spin up a consulting firm the next day and and still be you know pretty dangerous so uh that's what we're going to see today yeah and this is a pretty ugly architecture diagram uh but essenti like it's like the broad the broad Strokes of this is uh you know our stack is very simple and its currency state right now uh it's basically one vnet one subnet all of our VMS live inside of it uh there's three different sort of control plane kind of VMS um and then there we're also spinning up like three uh pretty much like vanilla Cali Linux VMS that you can like pentest from um so in terms of what's on each VM

George you want to take yeah so breaking it down just just uh because we were really using kind of barebones uh VMS and trying to trying to keep the operating cost as low as possible we use the we use some very cheap VMS for things that didn't require a lot of compute so like a management VM with uh tail scale slhe heads scale as the open source solution for the tail scale protocol uh keycloak uh and engine X they all they're not really memory intensitive uh compute intensive so we just put them on a super cheap uh VM uh and then uh moved on to the app server which had a lot more of the like

operators are going to be actively losing this this this could have memory spikes um it could suck up a lot of hard drive space resources uh we want this to be super secure maybe you know if if this server gets compromised we don't want it touching like other things so uh you know we we stuck with we had a little bit more money put in to the appvm still really reasonable uh and then we have the dashboarding tool homar um to make everything kind of pretty and and very usable for The Operators uh uh g t as a git repository I'm you know it was it was a tough call between gitlab g t g gitlab's a industry

standard I think it's really good if you're going to get into infos SEC and especially like uh security development malware and things like that um gitlab gitlab is a great tool um and uh but G is is is lightweight so if you're just kind of do Bare Bones um it's not going to have as much um like a performance hit to your server uh it's much lighter weight and it gives you what you need it's also a code repository that that Rivals like giab in gitlab in the in the uh in the services that it does provide um the functionality that does provide uh so for for on the Cheap on your own uh G is a great option and then a couple

things like uh private bin and Ghost Rider private Bin's great just for sending you know files and texts and things it's nice to have a past bin um server up but if you're red teaming and you and you could potentially be like especially white hat like red teaming pentesting uh where you don't want your findings to be publicly disclosed or revealed uh we we we found a solution that in-house that uh so there's the convenience still to The Operators but it's all within the servers and it's all going over a a a client to client like wire guard connection so um we did throw that in there and then Ghost Rider uh one of the one of

the totally overlooked kind of aspects of working for a company and doing things like audit or pen testing is the reporting aspect of it uh so the communication the clear and concise communication of what your findings and the impacts of those findings are to an executive or Senior Management Suite of people um it needs to it needs to have not only good written like communication um it it it doesn't have to be written just well it also has to look good uh and there's actually a Darth of of report writing tools out there like there's very Advanced tools that are super clunky um and super difficult to actually use and just way overdone in a lot of ways um uh from my

audit background like that that it was just a pain to work on a lot of tools uh and then there's like the other tools that are more convenient and more intuitive tend to lack a lot of features and especially formatting of the final report uh with whatever branding and backgrounds and templates you want to use uh from the marketing team uh but uh Ghost Rider was very surprised it's something arv found um and I wish I had known arv earlier in my career because like I could have presented this to a lot of other places I worked at and been like this is this is using open- Source kind of scripting like technologies that you can you can can copy off of like a

lot of forms there's a lot of support for this it's not proprietary kind of file like formats and things like that that it's going to export into um and it's also incredibly secure like I was really surprised um I I just gave up trying to incorporate it uh as a Docker container into our own system I just installed it on the bare metal because even though it was containerized it was it was kind of all wrapped up into its own like binary and um it just was really difficult to pull apart and that's part of its security so you want something if you're going to store your findings on to not be able to be

compromised you know even if the servers um servers broken into and someone's escalated privileges you you want those findings as secure as possible and and Ghost Rider for being I don't know how it's free honestly it's uh it's a great tool I'm probably building up way more than I should but um I was really impressed with it yeah shout out to Spectre Ops for devel and open sourcing it um then and then kind of wraps up the app thing so that that's like a tool belt I think you got reporting uh you have uh development uh repository code repository uh and then you have a dashboard uh I think if if you have a a

team that really is able to to do their own stuff and write their own stuff and you have developers on that team uh and you um are capable group of people that's kind of all you really need to start being dangerous um of course there's other things uh that you could have that that add efficiency or operational convenience um but uh a repo um a file transfer system and a reporting function um that's kind of that's kind of the Crux in my opinion uh so take that with a grain of salt I've led many people astray in my life um and then the last component this is really for red teaming this is red team Centric

uh you have uh C2 servers uh that because it's exposed publicly kind of wanted to keep it on its own thing versus rolling it into app because it is going to be talking with beacons and implants that you have on the network and you don't you kind of want to isolate that from the rest of your operation center it's kind of like a tiered system uh and you also don't want other things inter intervening like the less processes on a C2 server the more lightweight it is and less things potentially going to mess with uh some of some of like the the beacon Technologies or the the way things are are barely held together on some of

these you know uh tools um especially the open source tools they're they're great but you know it could be taken down you know with a feather at sometimes um so so we had metas spit Handler which is basically going to receive like meterpreter shells so if you want to deploy payloads The Interpreter is a great open source tool for developing uh just quickly spinning up like a payload for an x86 Windows architecture uh I can use Ms Venom uh dump out a payload in like a couple seconds uh drop it on uh a Target Network and have it start calling back uh especially if my target was compromis and I install tail scale on it um it it

it's going to it's going to have no problem busting out of whatever natat devices firewalls um as long as as long as that initial client registration worked um you pretty much can have a beacon silently doing whatever its thing is and executing commands like no problem so uh we chose sliver S 2 it's open source it's actually a shout out to arv I used to work for Bishop fog I used to work for the company uh one of the people at the company developed this uh you could use another C2 for sure like uh I'm just more familiar with sliver so we dropped it on here but you could definitely just swap it up with

something that you're more uh familiar with like Mythic or Havoc what have you yeah so that rounds out the basic architecture yeah uh cool so in in terms of like demoing this uh so the PRX that you would want like to run uh this tool yourself you would just like clone this uh uh clone the repo right now we're still sanitizing some of it uh it's going to be up within the week uh an Azure account uh admin privileges on the Azure account and like $50 or whatever you can scr up from your couch yeah good I'm sorry I'm sorry thato is live oh yeah it's live but there's only like I I think there's only like a read me and

like a Wiki on there yeah like I I just got the skeleton out but we'll be putting the code in there it was up for the last minute cool so think we can play the video so we're just going to quickly demo uh terraform spin up and tear down yeah just to show you guys like how fast it it can be uh like or how fast it is I I should say uh to spin up and spin down uh so uh so on the left we have our terraform uh sort of codebase and like the shell and on the right we have the Azure account in which uh we're deploying this into uh so right here I'm just uh hitting apply

and um there's going to be variables that you can um configure like the name of the like the host names and like uh you know uh uh like the what do you call it like the the resource Group names and stuff like you can just customize it however you want and that's the nice thing about terraform is that you can essentially feed it in these variables like whatever you want that that's the part that you can edit to suit your purposes H but the core logic stays the same uh so uh it's super flexible like that you just pass in your own variable file so I think around 215 yeah so it takes about like a couple

of minutes to spin up uh so you can see it's just it just finished over there and uh once we refresh the Azure portal you're going to see all of those resources live and it's ready to go uh you know I'm probably going to click around and yeah there's public IPS and everything so the whole stack is operational at this moment it took us about like 2 minutes and 15 seconds to get it running and at this point I'm also going to run the destroy command which is going to spin everything down and yeah you can probably I yeah and it's it's just going to go through uh the uh the resource Group delete everything and at the end of it uh

you're going to see me yeah like right there it took like what like two minutes and yeah like uh the whole stack is down again right so super easy to spin up and spin down uh really flexible uh you know so if you have an engagement get started up at the beginning of the engagement uh with super little latency and then at the end of your engagement you can just tear it down uh as quickly as you spun it up yeah and and definitely definitely definitely remember to take everything all your reports off the servers before you do like print out the report get into the executives and then tear stuff down because you're not going to get it

back yeah it's lost to the ether um cool so so yeah now we can uh show you guys like uh the live demo we already spun a stack up uh so uh George is going to just walk us through like what what it looks like under the hood so I'm just going to have a moment of silence for the demo Gods no okay no it hasn't broken yet I just warded it off um so so this this what's going to be on the repo publicly is kind of a sandboxed version uh I you know the it was configured to kind of be self enclosed uh initially we didn't even want to open up to the uh to expose anything to the

internet is kind of like a uh an environment that you could play around with in Azure to just get a taste of it and see how you liked it and if you wanted to add stuff to it and there's going to be there's definitely a good amount of configuration you could do to lock it down to get everything self sign inserts enabled to get um you know let's encrypt or CA s inserts um on the uh the domain names that you would buy to to then connect to this platform from external devices or if you want it to be working remotely and attacking like a network um you you just you just this there's a lot to be done on top of it

but what we did uh we just kind of uh bogarted the name Buck 50 on the Azure domain um for West Us 2 uh we uh tweaked this a little bit so that we could access it externally so I'm accessing from a web browser um and the team Buck 50 are all kind of dnsa records that are built into the head scale configuration uh and and you can modify that as you see fit uh but it does take take some kind of reversing or at least looking up and Googling like how to install you know head scale server um what do I need to do if I want my own like personal records for devices and

call outs so um once you get it to that point um make it your own a little bit uh you can be presented with like this dashboard so I set up three different dashboards I have a default dashboard for someone who doesn't even have access this is a public view dashboard for a Noob on your team um you can totally mess with them uh and and uh have them do like a checklist onboarding checklist it's just it's kind of like the gloss of having a dashboard service versus just having a static web page with some buttons for you know your other tools so um we got the orientation page then we have like an operator page this has the

services uh added some bookmarks that are kind of relevant to infos um integrated with slack um this one you can just like open opens up the app Discord uh and then you can access like Ghost Rider G T you know the pacin services we talked about um but you can't really administer anything so I set up a third one where you can actually go in and change the reverse proxy you have uh links to the administrative portals for all the services uh and you can really start customizing like when you add devices to the network are they going to be uh like special case systems or are you going to add like more infrastructure where you

want to mess with enginex and and and have Services route directly to that um uh and and uh now we can start getting into kind of the administration part of the thing so head scale um kudos to head scale uh it took it was definitely a little bit of a learning curve for me um I'm sure not for a lot of people but um getting it set up so that it was as tailes scale.com um like intuitive and usable uh it it took a minute um but we got there uh we got a web UI front end install on the on the head scale command kind of CLI interface it's an additional service you don't really need it if you're

comfortable with CLI you can just SSH into the management server and and start running uh tail scale CLI commands unlike Docker exe um in the container and add devices register things create preo Keys um some of the some of the some of the things I like about this web U wi is just like like graphically like you can if you had a bunch of um if you were trying to like just give a link or or some screenshots to to an operator so that they knew okay you your devices are on this subnet you can route to like the 10 do 16 and 192 on on this part of the network and then like 172 is going

to be another device on another part of the network you can really quickly reference that with the overview um but essentially um head SC scale and and tail scale the client acts as its own DNS server so um the gooey client like uh I was kind of like is this too much is this going to be too much overhead is too much fluff for engineers but um this this client really kind of streamlines a lot of the the command line commands that you would want to use as an operator in my opinion like uh accessing exit nodes so exit nodes is a feature of tail scale where uh since since it's a a distributed VPN architecture there's no

centralized VPN server that you're connecting to every client can itself be a VPN server and every other client you can have multiple exit nodes enabled uh and any other client can then use any device that they wish at any time independently of the rest of your team um to exit node your traffic through that device so basically spontan you can like you can have ephemeral VPN and servers as soon as you get a a Dropbox or a Target compromise and you set up the client on that system that system can then start accepting traffic from the rest of the team uh and Route it through that that Network and uh effectively start pivoting and doing

whatever you want and you can dynamically shut that down at any second um and uh yeah it's just it's it's just very um very convenient and it's it's all over wire guard so it can run silently like you really like there's no there's not going to be any logging of connection attempts to the VPN server like in a traditional VPN once once everything's kind of inh housed on the tailet network connections are only going to be like connection attempts will only going to be visible to the devices that you're either connecting to or from so um even other devices other operator devices they don't even have visibility into what exit Noe I'm going to be using and like why I'm using it

what time of the day I'm using it uh and one of the main motivators for doing the open source head scale build despite the learning curve is that every connection attempt like if you if we're going to use the Enterprise version of tail scale and pay the money and have as many users as we wanted um they're we're still using their coordination server uh which kind of allows the whole natur aspect that makes it really easy to get in and out of networks um but tailscale also is going to have records of those connections so what we did is we bit the bullet and and just kind of built our own coordination server uh in inhouse

tailscale decom completely uh so it's not just like a money thing it was like if you want to be silent and you want nobody to know and you don't want to be part of any derp Network um that other people may compromise in the future just because there's not a proof concept now doesn't mean like eventually there's going to be a big breach um that was a huge motivator for us to get this totally like insourced within Azure and uh basically run completely silent it's like red October basically um so I think I plugged head scale enough and tail scale um moving on to Ghost Rider again a super important like unsung hero of the whole pent test process uh report

writing uh you have access to this uh this environment uh it it you know it looks kind of unlike the first time I open this up I'm like oh 1998 I'm back like hi um but uh like looking into it like the integration with SSO like the groups you can create through keycloak to really pin down um all the access to the different findings the different reports like it it uh I have some familiar with Plex track it it it gave me everything that I could imagine from like most pay for subscription report writing services uh so uh really happy with that again easy to get to from the console like all these are just like

those DNS records oh yeah and that that's a really handy thing about uh our head scale implementation or a head scale implementation because it has its own internal DNS functionality you can you can make these really easy to remember internal sort of URLs so all of our urls are just like you know team. buuck fitty for the dashboard or get. buuck fitty or P.B fitty so it made like it makes the usability off the stack a lot better uh for the average operator on our team yeah yeah one of one of the reasons I I think we might not have our our code up yet is because I'm going through all the comments that I wrote as

I was troubleshooting like configuring stuff and like trying to like you know Easter eggs too many Easter eggs too many funny jokes uh just need to cut this out and sanitize it uh but long story short uh SSO keycloak uh if you want to do something like authentic and really cut down like just your bare minimum you want authentication services and SSO to these Services which I highly recommend you do uh especially in a group environment it just makes collaboration much easier uh and spinning up of of uh user accounts on those Services kind of seamless um with keycloak it was a lot to digest at first but uh um eventually we got there we uh

there's a lot of um capabilities for MFA especially um using devices RSA tokens um but and also setting up user profiles so I like being able to have custom attributes where you have SSH keys can use for later um but yeah SSO and nutshell very important keycloak um highly recommend but good luck finding documentation out there it's it's a little bit tough uh and then last lastly uh not lastly but we got a couple more services um the GitHub the gitlab uh gy uh Service uh definitely pretty important uh adds Wiki adds more collaboration services for your team it's also a secure environment to store code uh without having to get approvals or um you know from from the board of

directors or whatever to to have malware in your environment so um pce spin kind of self-explanatory and that's pretty much [ __ ]

cool oh we do have one last slide to get through uh so just in terms of what's next what's on the horizon uh definitely want to make this multi Cloud uh don't want to get locked into Azure at all uh uh we also wanted to one well and that dovetails very nicely with what we also want to do which is kind of like a python wrapper around the whole thing so it's kind of like a like a wizard experience almost that you can just like go through and uh you know it like it'll dynamically alter the terraform variables for you so you don't have to go in and manually fuss around with them um and one big one is kind of like

parallel stack deployments right now we're limited to one per subscription we definitely want to move to a uh to a place where we can have multiple Stacks running at the same time uh uh because you know like our team is pretty small we we we don't really do multiple red team engagements at a time but I'm sure other uh teams do do multiple pentests or engagements um uh in parallel so we definitely want to implement like a a parallel sort of uh deployment model uh also want to like bake in like known good uh images and then just deploy them on the VMS right now we run a lot of inline commands on the VMS upon startup

and we want to move away from that just clean up the terraform code base a bit more so that's like so that you know you can abstract all of that stuff away into the actual image layer of the VM and then you know just use a registry or something to deploy it um also for the pentest VMS like the Cali Linux VMS we want to implement some sort of VNC client um on their end so that you can just use your browser to access them uh yeah sort of like if if you guys have played around with hack the box or offsec uh the way they have like uh browser based uh VNC Solutions we want

to have that implementation as well and uh this one is real like pie in the sky sort of thing just like a local sort of like uncensored llm uh uh model or containerized model that's running uh uh on maybe a GPU cluster or something that'll help you craft uh exploit code and stuff uh without having to resort to you know chat GPT or you know Sonet like which have a lot of guard rails around trying to generate malicious code so we definitely want to explore that but that's definitely like uh you know version 3.0 or something and it's definitely going to be like an optional uh sort of module uh in in the future stack and that's about it uh so thank

you very much uh like I said it's our first talk so we're so glad we had this chance to talk to you guys uh so uh yeah our our email and Twitter is on there so uh feel free to hit us up anytime with any questions or comments about any of the stuff and like I said the the git repo is also there and uh if you want to bookmark it and like you know revisit it later in the week it should have uh a work a turnkey stack for you so you can just download it and get going with it yeah we we promise we promise uh yeah I don't know if we have

time for questions but yeah maybe one or two okay one or two questions uh yeah how manyu okay did you hear the question you sorry could you repeat that how many engagements have you run this infrastructure with how many engagements have we run this infrastructure with as of now zero uh we just like sort of operationalized it maybe a week or so ago so we're looking to get it going we just demoed it for our team last week so uh this should be the model uh going forward in terms of how we deploy infrastructure for engagements yeah uh you mentioned a lack of documentation on keycloak um and also like like liking open source I'm just wondering like how

you got through that like just a lot of Googling trial and error like what did you do there um well I think to answer that I really have to thank my wife um for letting me stay up till like 5:00 a.m. like many nights in a row just trying to figure stuff out and pushing like the same red button over and over again until something different happened um but yeah no it was it was tough I think a lot of the demos that uh I ran into for setting up things like G and and the other SSO enabled things were using keycloak like 23 22 uh and they recently upgraded like keycloak upgraded to a V2 which I I

could have gone back to 22 but I didn't I did like the additional security of 25 and although was it was definitely a steep learning curve and a lot of I think just magic to be honest um we got through it and uh um but yeah it was I I you know I wouldn't I wouldn't wish that trial on a lot of people I think if you're familiar with devops and and keycloak specifically SN know I think it's it's not going to be that big of an issue but some of the upgrades were difficult to find and there was one or two like command line arguments that I needed to run within the docker that

just made things work finally that I would have never found out unless I was just on the forums until 3:00 a.m. yeah hey folks congratulations and a great talk first talk and I admire and commend Your brave on learning something and within two months just putting out a talk together to share it with the rest of the community thank you I am also very excited about the fact that your folks are talking about red team infrastructure C2 infrastructure and whatnot that's a topic that uh I am in charge of within my team and I have another teammate and you you kind of reminded me of that as well so I feel identified with some of the pains that

you're talking about especially with you know running the same deployment over and over and over trying to figure out what's going on especially when you don't come from a background of building things now I have a question when I saw the um x uh proxy manager are you guys using that as well for through R directors um I think that's I think uh right now it's it's just for the platform services but if uh if there's a need to not route through tail scale I think tail scale a lot of the configuration uh can be done like within the tail scale like head scale config um and then we do have forwarders oh we forgot to talk about the the forwarders

so we had a few forwarders set up that you can have like regionally diverse uh servers accepting connections um right now it's just kind of going over the Raw on this platform but um if there's a need to not just use like a socat forwarder um to send like Mater sessions back to the C2 or Beacon information back to the C2 um then we could probably spin up another engine X or you know caddy probably like a caddy something lightweight on um one of the the forwarding servers themselves and just make it I mean that I feel like that's that's something we would run into if we were doing multiple operations at the same time um with a larger team uh so

it's on the horizon and I think that's probably something we'll have to tackle eventually fair enough uh I'm I was also very excited to see that you guys implemented head scale uh and not nebula because that was a different pain that I looked through and I'm like we don't have appetite to take this level of complexity especially in a small team uh a way that I got around the problem of sending uh data to the hosted control plane in tail scale was just uh SSH tunnels reverse SSH tunnels essentially right between the different infrastructure components so that way you know if you don't have the appetite or the time to set up all the open source side of head

scale though it looks like an interesting tool and last couple questions before I send send it off is u i saw a dashboard that had some of the the DNS names for the different assets that you had right does that dashboard dynamically update as those resources change or is it is it manual um I think it's manual it's it's manual you they homar has the ability to do if frame like you can create like an if frame widget so you could you could link that and that's that's kind of like our theory on the whole guacamole kind of novnc uh webshell aspect is to put it in an if frame so it dynamically updates based on um what you would see if you

went to that link or service itself and you could just like point it to a Wiki right in an iframe and and put it on there and that's dynamically updated you know in real time as the Wiki updates so uh there's a lot of potential like this is just this is just like going to Home Depot and getting like the tool belt with like a hammer and screwdriver kind of thing but like um I was very surprised with hom R like props to homar it's from kind of like the Plex Community um and uh I was I was happily surprised how easy it was to kind of like corrupt it even more uh into like a

kind of hacking tool um with things like iframe widgets and things like that so well fair enough and congratulations again for your talk thank you just want to get his question and this is the last

one hello my my question is about is your Azure sub completely separate from your rest of your org sub um subs or like do you access it from your day-to-day laptop or completely separate um and then how is like if it's not completely separate was there challenges with like Risk and compliance that sort of stuff getting getting your getting your Azure sub to do the bad stuff that they don't normally want you to do in in a cloud environment like that uh yeah so our Azure sub is like we're it's kind of unique for us just because we're the red team so we got to cut through a lot of the red tape that normally exists with Azure Subs uh at

Costco uh so we were able to I I think like currently we are kind of like just we have admin rights like which you they normally don't give to everyone right uh or anyone on like different teams uh so we do have like an isolated Azure subscription uh that we can spin up resources in that you know uh you know it's its own cost center uh but yeah it's not like linked to the broader like Azure infrastructure at the company yeah it's it's kind of isolated does does that answer your question yeah just like controls like antivirus and uh typically they typically do and in our case I think made an exception uh to allow us to because otherwise they would

just keep getting and this was the case like they kept getting uh ping from their uh monitoring software saying like hey like there's malicious code here there's d d d so uh they basically whitelisted us uh uh from like the usual um controls that they apply on everyone else okay just okay um if there's more questions if you guys don't mind taking outside so we can get prepared for the next speaker right awesome thanks everyone thank you [Applause]