L0ss sans C St0 - PAMdemonium! Privileged Access Management

take around the pulse for loss that we're talking about privilege access Madness cheers [Applause] so everyone good microphone's working um so as uh you would have seen it from the slide from the uh program uh I was supposed to have a code co-speaker Cameron Stokes c-sto unfortunately uh he's had a family emergency and cannot be here so I'm gonna have to cover his bits and if that means that I kind of have to hand wave some of the cryptography stuff um if you'll forgive me I'm sure so uh what we're going to be talking about today is uh you know what is a privilege access management system what does it do why a geode clock me might

care about that and want to break into it um some you know specific examples of horrible slash fun things that we might want to do to one some demos and because it's mandatory for this kind of talk a little bit about parent is good actually if you actually do it good um mandatory disclaimer I'm not going to cover much in the way of kind of the cloud side of things um like Microsoft identity manager um and also as the fight says vendors do like to kind of play whack-a-mole um with us on this this sort of thing so if some you know minor detail of how cryptography Works doesn't match up between the exact version of an even

product that you're looking at and what we said it's not our fault you just might be able to do a little bit of reverse engineering so modern Pam systems are kind of an evolution of the Enterprise password Vault system most of the solutions on the market seem to have started out that way um you know a central application that CIS admins can store their passwords in so they don't store them in passwords.txt on the desktop then at some point I guess someone noticed that there was a bunch of increasingly Baroque recommendations coming from security people about how these credentials should be rotated and managed and you should only assign access as it's required and you

should order access and no sys admins are actually doing any of that and so the password Vault vendors started bolting those features on and then they caught the resulting monster a privileged access management system so most of the reasonably well-built pan setups look something like this if you squint hard enough um there's a bit that the users will interact with usually like a web interface hopefully with some MFA on it there's a bit that stores all the secrets usually an off-the-shelf database server with some sort of encryption happening hopefully and then an assortment of other bits for doing stuff like rotating the passwords automatically setting up shortlisted sessions on managed systems temporarily assigning privileged group membership

and then taking it away again once the admins finish their business um sometimes all of this might be a single host but more commonly it's kind of spread across a bunch of them sometimes bits of it are in the cloud other bits are on-prem um but they're kind of idea is that users needing privileged access are forced by Young well-implemented well-configured firewall to go via that front end where their privilege access can be managed that's kind of how it's meant to work but unfortunately reality is a lot less than tidy than that in reality things are rarely tucked neatly behind a firewall and uh most if not all the parent component servers are domain joint

also Mr legitimate key business uh is accessing the Pam system from the same laptop that he does all his web browsing and email and office docs and then his kit installs fortnite cheats [Music] um this reality sets us up the bad guys so that uh if we can compromise one of the component systems or ad itself which is a lot easier than you might hope in most environments we can access the system secrets in the same way that the component systems do um but wait I hear you say it says in that very diagram that the secrets are encrypted and strong encryptions like the one thing this whole industry has going for it that actually works the way it's

supposed to yes and and if you cast your mind back to security 101 you might remember when you were taught that you should never ever store passwords in plain text and you should never store them with a reversible encryption you should store them hashed and that's good advice um but unfortunately it's fundamentally incompatible with Pam systems and and the way they need to work um admins need to be able to pull out plain text passwords for use in situations where the pans session broker can't set up a session for them um and the session broker needs to be able to authenticate to things on behalf of the user so it needs to have the password too

so um we're gonna well I'm gonna talk because it's gonna be weird cam um I'm going to talk a bit about how some common Pam systems store their secrets and what's involved in pulling the map

because he does all my crypto for me so um while psychotic secret server is pretty typical of a lot of sort of Pam and password Vault systems on the market it does have one very very weird feature which is that it uses Pig lightning to obfuscate streams in the web app code I have no idea why um if anyone knows anyone at psychotic I'd love to know I don't know what it achieves but it's funny because it gives you an excuse to say password so um

why why pretty unremarkable uh I mean that like under the hood it's pretty much this in 90 of cases you've got you know a couple of Windows servers that are domain drawing one's a web server one's a database server the web server knows the cryptographic secrets to get to the staff in the database server minus the pig latin is it the same for a bunch of other products so password State manage engine password managing password manager Pro um a couple others that I can't think of right in a second some of them will keep the key material in like web.config some of them split it up between the database and web.config some of them just keep it

in the database next to the secrets that are being encrypted which is recover but that's been really the way a lot of them work this is the slide cam was supposed to do with the actual cryptography but I'll do my best um so the crypto itself is is mostly like pretty insane and and sensible but it does seem to be focused on a threat model where the hacker might own the database and they might not own the front end and also not own the database server or the host that the front end runs on just the database each each row in the database is an entry the entry can have multiple items the entry has its own unique key

and the items are encrypted with that key the entry key is stored in the database and it's encrypted with a master key that is stored in a config file on the the web server that master key is also encrypted with playing scare quotes but it's mostly hand waving because it uses static hard-coded keys to do that um and according to cam that means it might as well not be encrypted at all so it's a hack secret server to get all the secrets out you need uh db.com database.com it's actually cool sorry and encryption.config um and you need connectivity to the MSS URL database database.com contains creds that will let you authenticate to the database um if it's configured to use an HSN then

you'll need to basically own the um the web server entirely and if it is using DP API then you will also pretty much need to open the web server entirely but that's not out of the question um there are a couple of public tools for dumping stuff out of um secret server uh Secret Service secret stealer is a pretty good kind of jumping off point for understanding how to do it it's a little bit out of date in terms of versions this is the result of the kind of guacamole thing I talked about before I understand there is a more recent Motorsport module that'll do it as well I don't know exactly how up-to-date that

is I haven't had to do a secret server in a little while but um thankfully it's dot Mount so you can like for uh libraries into our data plower and just have a look at how stuff works so beyond trust is another example um as I said we aren't going to talk much about Cloud stuff and Beyond trust it's got a bunch of its bits in the cloud um we didn't spend a whole lot of time wailing on it but we couldn't find a way to get into any of the Cloudy bits but unfortunately the on-prem bits have all the same kind of unavoidable issues that everything else does um and the risk of sounding overly

enthusiastic and crazy I would say broadly speaking you're better off with the cloud-hosted versions of most of these products um if only because the vendors are not going to let you do stuff like domain joint the servers and have them sit in the same flat Network as all your workstations um but yeah unfortunately there's some functionality that requires you to have on-prem presence and Beyond trust one of them uh it's got one of the entrance has these things called Beyond trust password safe Cloud resource Brokers that sit on-prem and handle these four jobs here um One Way or Another at some point to do those jobs the resource broker has to have plain text grants to achieve these

um I've only had a chance to look at resource brokers who do the middle two the ones in red and I can confirm that they do have plain to experience at some point and they store them at a pretty easy to retrieve now um so here's an example of how a cloud resource program goes about trying to discover unmanaged passwords and computers and environment first the user goes to the web interface says hey go manage my stuff the web server says sure thing I've got a guide for that he's going to take your network already the cloud thing yells at the resource broker and says go along into staff as administrator using those passwords you

already have and the resource broker does that using the highly privilege credits that installs you know sqlite database um there is some cryptography involved so the file that these things live in is called btconfig.sqlife um the descriptions for each entry are play attacks but the secret values encrypted um the unlike secret server the master key only exists in LSA secret it's called if you're interested Finance master key um so again you'll need to have admin on the um on the box to pull that out usually um but yeah all you should really need is that Central light file and Away you go foreign was for years my kind of a personal great white whale every time I'd have a

gig where the client used it I'd run out of time before I got a chance to really like to chew on it um I'm pleased to say that since then I've had a little chance to kick it around um I'm also quite pleased to say that in terms of preventing us from like yanking everything directly off disk on the um encrypted password Vault it's a lot better than most um that doesn't extend to all the other components but we'll get to those in a second because I vault is pretty good eh um vocal screaming you super hard if you try to install it on a domain joint machine it'll actually tell you that cyber Arc won't provide you with any

support if you try if you do it hardens Windows really good during the install it turns a bunch of the unusual kind of best practice stuff up to up to Max disables a bunch of services that aren't needed that sort of thing if you follow cyberox instructions to Atlanta they'll even have you load the master key for the database off a CD-ROM or an ISO and then like eject it or unmount it after the system is booted so that way the master key is not even on disk so even if I somehow managed to pull a 40s backup of of the Vault server and all the component service I still don't have all the goodies

but um perhaps most impressively The Vault service itself will actually firewall off all the other network ports other than 1851 which is the one port that Vault means to function and yeah just lock them all out so a plus no notes on that that's pretty good unfortunately yeah almost every environment I've seen all the cyberox component servers like the password well web access and the privilege session manager and so on the domain joined and at least some of them will be accessible by The Usual Suspects you know RDP SMB RPC although typical kind of admin interfaces um so they're generally pretty potable once we've um achieved some level of privilege escalation in in ad when these Subs need to interact with

Vault to get their configuration settings and and the juicy passwords um they do it using a cred file so that's a good old-fashioned any format configuration file it's got an encrypted secret that they use to authenticate the vault cyber will encrypt the password in the credit file using what they call verifiers so it's basically a bunch of stuff that gets attacked onto the key derivation function to make it supposed to constrain who can decrypt the value and where um depending on how secure you want to be you can use like one verifier or all of them or any number in between um examples use things like you know the path of the XE that's being used to

decrypt when it's trying to decrypt the file the username the hostname IP address serial number of the hard disk the other times in terms of how well that constrains who can decrypt the thing if it really only constrains you if you're using cyberoxone tools to do it um obviously if you write your own decrypter you need to know those values most of them are not actually all that hard to obtain and there is an option when you're creating the credit file to just include all of those values in the credit file as well which kind of defeats the whole purpose but um yeah if we can get these values and I know it's not cam into writing a

decrypton for it reversing the existing one we can write our own um so there's three different versions of these profiles I've never seen version one in the wild so we're going to pretend it doesn't exist for version two there's um a handful of possible verifiers um and there's already a public tool you can use to encrypt them that works pretty good with URLs up there um for version three there's more possible verifier types and the credit file creation utility will yell at the user if they don't use enough of them um they actually have to pass an argument of creation time that's like you know be horribly vulnerable or something like that um I'm not going to understand Cam's notes

about how the favorite kid derivation thing works um it's pretty similar between the two versions there's a few steps that involve complicated words like clone State and iterate um which but basically version three uses Shopify 12 version 2 uses sharp one um if you're thinking shot one is bad cam assures me that um it doesn't matter none of the broken bits of shell won't actually matter for this key derivation function um and the crypto after the key is derived is pretty much the same for both of them with a little bit of change in Integrity stuff um we wrote a tool to decrypted version three and as far as we can tell the only reason why version three was created was

because someone released a decrypted for version two so we're not going to release ours because we don't want to play whack-a-mole with sub Rock um but cam says that if you are reversing a version three credit file and you want some help you can contact him and he'll give you like some cheat codes and point in the right direction so we've got some credits from this how do we use them so in the same way as the uh decryption verifiers try to constrain who or what can uh decrypt the file by using the path of the program using what um the client apps that cyber app provide for accessing Vault um will tell Vol hey I'm the private GUI

client or hey I'm pacoi.exe and then if the account that you're using isn't meant to be used with that client Paul will say authentication failure for yourself um and similarly most of the accounts used by these component Services have real relatively limited privileges so you need to be a little bit clever and make a client that lives so it took a little bit of horsing around in mixedmode.net assembly Health but we eventually managed to figure out how to write our own cyber art client that can lie to both about what client ID it is authenticate to evolve list secrets and retrieve them in plain text and then for bonus Pro points in the in the process

we figured out password Bob web access server has the thing on it called a Gateway account which is cyber art lingo for it's allowed to impersonate any other Vault user so if you get that One account you can log into Vol and say by the way I want to impersonate um Jimmy admin and congratulations to Jimmy admin again we don't want to play whack them all with cyber up so we won't release it publicly but if you you know find yourself following the same path and you want some cheat codes or a pointer in the right direction hit us up um Arcon is a relatively little-known in Australia at least vendor that has a pound product

this one's kind of a reminder that even security products can have horrible vulnerabilities in them you just need to look to find them and really just look um there were also some decisions that seemed to have been made early in the development progress process that kind of not well thought through but you'll see what I mean in a second so um this product was intended to be used in a web browser with a plugin that would communicate to a local service on the user's machine and this service then talked to the Pam server to do Pam stuff and handle installation of client apps required for privileged sessions or whatever unfortunately the way it does that is by

running a web server just listening for get requests with no authentication and not much in the way of sort of validation so one of the instructions that it will receive is just download and run this actually I told you about and so if you send someone who's got this installed an email and this image tag is in the email they will download and run malware.exe without even knowing about it and then cam has helpfully left me with a demo video of doing just that so we can see pound plugins running on the right uh things can be put in that URL that's got our malicious image tag in it and Bam it downloads and runs whatever

we tell it which is just Bonkers um

the server also had a horrible remote command executions vulnerability um here's a very condensed version of how we found RCA in the Pam server component um the timeline from getting a copy of the service XE to a working exploit was maybe an hour or two um so Step One is open up some of the x's and downspire have a look at the code um see that the server runs something called win holding service which listens on tcp445045 um see that anything that gets shouted at that Port uh it kind of just decrypts it using this hard-coded key that's the same across all the installs um that then gets fed to a very fancy switch statement which decides which

method it should use to handle whatever you shouted at it examples of methods it might use to handle that have fun names like execute Powershell script and run with custom commands using a CMD I'll play Windows Decon password update Windows service user password Etc and then it just runs it

the characters to Magic Replacements that they have there but it's literally just you know cmd.xe C it's just running it and here is just running it so there's the service it's running and here's a yeah

I know I said I picked up like this I was lying um so you can see you know there's uh administrators are Mike any favorite uh and that's the wrong thing this is the right thing moves very hard for everyone should speak so uh this is as much of the code as I had to write um all of this down here is just copy pasted from the library that they provided called winds up client hole that it's probably and so all I'm doing is just telling it the IP telling it what custom command to run and then saying create a user called base loads per and put them in the administrator's group let's go da

program there the contemplated successful let's check okay now we're in trouble no don't worry oh thank God you can say Ned local group administrative is based on its purpose [Applause] all right the cryptography and that was also about it um uh my notes Here literally say can talk briefly about the config file where they wrote the string of ones and zeros as text into the text file and I don't know why they did that this is Pam's notes on uh I think this is a um sunshaft formula for how to just turn there like zero one zero zero one one one zero file into um yeah a cryptographic pay that you could just use to

um different anything in that in the hospital but yeah probably go um so I have an empowerment right in one slide so you know we keep saying it we all keep saying it assume 80s are already home and never ever ever domain join your power servers anything that like actually goes into the database and retrieves stuff or talks to something that does and has that access don't demand join it because our Step One is get da and then after that we're like well I want more stuff where is the more stuff that da doesn't get easy as a power system yeah um use separate virtual infrastructure if you're going to use Virtual infrastructure so don't

you know if um you know if getting da gets feed onto your virtual infrastructure don't have your Pam stuff uh living on that virtual infrastructure whatever cryptographic options they give you to turn up turn the knob as far to the right as you can if they let you use deep API use it if they let you use a TV TPU use it just turn them all as hard as you can um aggressively segmented just nothing touches that Vault nothing touches the front end except on the web Port nothing touches any of it on SNB or RDP or any of that and yeah I know no one although if you use privileged access workstations I'll be super impressed and I'll give you an

A on your pen test report and yeah and yeah if you've been wondering um all the images in this deck are from Dutch Master Hieronymus Bosch's late 1400s painting The Garden of Earthly Delights it's a it's a real trip but yeah um it had a fun depiction of pandemonium in it so we thought it was appropriate um question time do we have any yep three minutes three minutes two and a half two seven um

but in order I mean the pan servers have to or probably have to have creds for A.D because the OS doesn't have to be domain joined and manageable by ad you don't have to give the domain control of the service yeah but yeah um yeah it's it's a matter of kind of keeping that control relationship one way on the back some of the the products you've worked with uh I will integrate if you have that with HSN there's a little bit of research you've done is it against just feeling something or as if I've never seen anyone actually use an HSM on one of these products here's the short answer I recommend you do it it's a good idea

um that said if you do it depending on implementation details but like you still have that kind of fundamental problem where if I can own the um say the web server that people are using to pull passwords out I can still somehow yeah I'm still gonna be able to do whatever that web server can do whether that's tell the HSN to decrypt staff or you know tell the privilege session manager standing up a privilege session you know but hsn's are definitely going to make it harder and that's the whole game

so um in terms of kind of the web side of things um anything that's going to let us yeah hijack a session or create a fake authentication token or whatever yeah that's going to be a weak point um it depends on what your what you're integrating it with and whether that system that's um if it's like adfs well then all of a sudden you know you've got you're back to square one with exactly the same problem you know if my own ID then I own a to get best then I can make my own sample token do whatever I want but yeah um it's complicated um yeah generally this is the other thing is that uh

none of this matters if we own the administrator's workstation hijack is browser session and then just get up in on the thing because we are being here on the animals sorry it's 2023 I should remember cool time for one more just one quick one yes

that must have he ever needed again is that um yeah so um the question was about the cyberock the way they they tell you to kind of only provide it to the system at boot or at the start of the service and then like take it away um it's it's going to like hold it in memory where yeah if we got admin on on the Vault machine while it's live yeah with enough time and and stuffing around you could get it out um and yes the the downside of that as you notice is that if the machine bounces you've got to then put the CD back in the drive or Mount the iso again or

whatever and then um that could have availability Max

a normal home show and so I'm awesome foreign