Layers of Cloud: Azure and the Mishandling of Secrets

um so a little bit about myself uh right now I work primarily in incident response uh and just generally securing things I love focusing on cloud prior to that I've been a pen tester I'm a security engineer I love just learning as much as I can about the industry and working with as much of the industry as I can because the more I cycle around the different niches the better I get at everything I just love it so much um so if you ever want to talk Cloud stuff at all or anything security really I'm a game so today I know on the schedule it says this is a talk about Secrets it's really to talk about secrets and the shared

responsibility model but the secrets are a case study for the shared responsibility model and it's a lot more exciting to talk about passwords than something that's like three words and sounds very businessy acronymy right up here I've got Microsoft's diagram of said model if you look on the right you'll see that on premises we understand our responsibility so clearly right it's obvious everything's on us we have to you know patch manage maintain the perimeter we get full control of everything and it's very comfortable to see those light blue boxes come in for parts that we aren't responsible for anymore with Cloud but we have to remember that this is three columns and these columns actually represent a bunch of different

settings for each service that we'll be using so whereas we think about you know on-premises can we secure this with our firewall okay we control the perimeter that's easy whereas looking at software as a service platform as a service infrastructure is a service there's three different ways to think of those right am I configuring a piece of software am i setting up an individual firewall for a service with is or what what's going on here so we used to have this kind of like very one this is our responsibility and now it's just a mess we're not responsible for everything but what are we responsible for I want to keep that in mind while we

talk about passwords so passwords obviously are a pen tester's favorite thing they're our favorite thing to whine about in security you have an insecure password you put it in a word doc how silly can you be and I got really curious being a former pen tester about what this would look like in Azure one of our favorite things when I was on an internal Network assessment is to go hunting for passwords right you get access to a basic user account what credentials can I find in documents in scripts in open file shares on the network that I should not have access to at all where have people been really silly kind of stupid and just put it wherever it's fastest to

go and what's the equivalent of that for azure so I'm going to talk a bit about Azure fundamentals if anybody's super new to it it's just one slide to kind of dip your toe in you don't need to understand Azure to kind of get the gist of what I want to hit home with this talk so I encourage you to hang on if things feel a little overwhelming and then we're going to talk about correctly storing Secrets incorrectly storing Secrets getting at those secrets in both contexts if you're a pen tester if you're reviewing your own environment security um and then just things that are still concerning to us for cloud misconfigurations we're not going to

have control over uh and then that will kind of take us back into the shared responsibility model so up here basically what I want to touch on is in Azure there's Services Services get resources that's kind of a unit of service that you have either a VM or a uh I don't know what other Azure services to talk about right now but just a thing of service it goes into a resource Group which is kind of an organizational container that goes into a subscription which is another organizational container but also a way to manage billing which can be quite helpful and then that rolls up into management groups which are a great way to manage policy if you want to

implement you know for a whole bunch of subscriptions that people have different ownership of they should all have these settings so that nobody exposes RDP to the internet you can do that with a Management Group pretty well at the bottom this is most relevant to us so the dotted line on top is for the reader role it's supposed to have limited access so you can just see the configuration of different resources but you can't manage them you're not supposed to have access to Secrets within those resources you're not supposed to be able to change those resources contributor and owner roles are the ones that are going to be tinkering around and doing things they're what we think of a little bit

more from a classical pen test perspective as having access to something I can access your VM change your VM look at the stuff in your storage accounts add stuff to your storage accounts change permissions with the owner role maybe add a backdoor account with the owner role you know give granting permissions so the red box is what we would think of as a classically risky set of permissions the purple box can actually get us at some stuff though if people are being silly so we'll come back to that in a little bit in terms of secret storage there's a lot of text on this slide I want it to be more of a reference so let me talk

around some themes generally we want to put our secrets in a place where they're going to be stored securely obviously we want our secrets to be somewhere like this key Vault or a hashicorp vault or a password Vault like one password that's where we want to put our secrets not a document so if there's no other available option in a service you're using you'll want to be putting things in key vault moving on we get into things where Microsoft is managing some of the secrets for us so in a storage account they'll generate keys for you to use and you can have access to those keys if you're a contributor or an owner and use

them to access the Storage storage containers kind of being a place where you just dump data right if you've ever heard of S3 buckets kind of like that for azure so we get access to those keys we have to store them securely on our systems but inside the service they're stored pretty securely then we get into things like service accounts so we have service principal credentials and managed identities service principles we'll see those Secrets once when they're generated it's our responsibility to download them and store them securely but again we're not going to see them in Azure too much and for managed identities we're never going to see that password at all it's fantastic right the best password is no

password in this case and I don't mean that in terms of no authentication it's just that we can use this account for Automation in the background to be performing tasks and automations on our behalf without having to set up the dreaded service account password which is going to be you know welcome one two three secure service or we're going to put our company name or some kind of inside joke that's going to haunt us for 20 years right when pen testers come back and tell us did you know you had you know this exact make and model of car and you're like oh my gosh I remember when we set up that domain in you know 1880

um so those are some places we want to put passwords if it feels confusing and we then look at this and we acknowledge it's not an exhaustive list like wow yeah so things are already more complicated in the cloud than we're used to passwords are not just passwords if somebody does get access to a contributor or an owner level role from a pen test perspective from an attacker perspective there's this fantastic tool called microburst that will pull down all the secrets that you can get your hands on with that this tool really won't work at all if you only have reader access or if you don't have access to a specific subscription or Resource Group but if you do it's going

to get you everything you can get your hands on right so you'll get access to those storage account Keys we just talked about if there's administrative passwords that are stored within Azure container instances or ACI you'll get access to those and you can really use those to Pivot and do some damage the thing is we're making the assumption that we have access for a contributor or an owner right like stuff that we should need we already need privileged access in order to get at these passwords so we come back around to passwords being stored incorrectly what's the equivalent of putting a password and a shared Word document in azure what's the equivalent of putting a

password in a script in Azure and does it actually matter I think I have a lot more words up here but that's what this one boils down to just a note um this is all just based on curiosity being a former pen tester I was looking through documentation in Microsoft's databases uh in blogs about how to set up different Services correctly and just some general observations the first one I want to talk about is run book parameters a run book will allow you to automate the creation of infrastructure infrastructure there's the nerves um and different tasks within azure but it's basically just a script that lives in the cloud that does things for you and what do we put in Scripts

passwords no but it can happen quite frequently especially when we're moving fast we're working with new Services if you think about an I.T department they have a script they want to use that script but in Azure to deploy systems to set things up to set permissions they might just you know Whole Hog throw the Powershell script or the python script up there and run it with a service account once you've put that run book into the cloud the publisher the published source of it will be visible to anybody who has reader permissions so now we've stepped away from talking about contributor and owner permissions privileged access that we would need in order to exploit something and we're talking about reader

access depending on what environment you're in if you are a consultant or a pen tester or if you're looking at your own environment you might not actually have much reader write assignment but in some environments it can be a little more prevalent maybe you're an I.T auditor maybe you're a help desk worker you need to look at Azure infrastructure and in these cases if you're able to get that like very very limited not really privileged access this would be a good place to look you do need to pull down the runbook code to look at it and review it but the top here we've got the runbook view when you're editing and placing it in there

the bottom is the view if I'm coming back at it with a different account with the reader role so just make sure there is a shared resources section for runbooks at this point with a credentials area you can place credentials there and they won't show up in the runbook script again Azure will protect them I'd still recommend trying to lean on key Vault right in this case just because we don't really know what's going on with those attributes in the back end and we might see another reason to question this a little bit coming up but just don't put them in code right passwords in code is a bad idea and now we're in the cloud

the next one I want to talk about is coming back to ACI the Azure container uh instances is that how what it stands for this is the problem with acronyms um but this is basically just a way of running kubernetes at scale and Azure very easily provides a level of management for you and you can place different environment variables that it should run with with containers we want to be able to run them regularly all the time just spinning them up spinning them down making them do the same exact things and to do that you need to pass a lot of environment variables to say configure this service this way configure that service that way run this

script with this thing with a five and obviously another great place where we're going to need to handle access or passwords or automations things like that any environment variables that are not marked as secure laser pointer nice are going to show up in clear text in the script so the resource Json if you look at the graphic view of azure it shows you a lot of different settings visually in a way that's very pleasing to the eye for a lot of resources there'll be this little button that says like look at the resource Json and that's kind of the back end it's like if you open the hood of the car and look into the resource then you're looking at

the the engine the settings all these different things and if you pop open the resource Json in this case you'll see a password if it's not protected um so obviously we want to Mark these as protected so that they are secure so that they are not showing up in the resource Json uh another great place to consider key vault in all cases but here this is how the service again with these specific service settings right you listen to me go through these and I'm like well you want to use keyfault but you want to use this here but for this one you want to use a different setting the cloud is complicated but I guess I

thought that was supposed to be easy right um but another great place where the reader role could access permissions or secrets that you shouldn't have with the ability to just view something the third example I want to talk about is azure logic apps so this is a great example of like no code low code Type Solutions I know we love to bash what new technology brings to the table but it's honestly pretty cool like someone could integrate an API uh with all these little visual building blocks without having to think about code syntax or you know essentially learning a new language in order to describe how to do something this HTTP block right here is one of the

most common foundational blocks and it includes all these useful parameters you can add one of them is a password of course if we're connecting things together we're pulling data from different Services we want to have a username and a password um it's not protected in this service at all you can't mark it as secure it will show up as plain text in the visual code but also the Json that's or not the Json the code that's generated by this visual app as well so it is completely accessible to the reader role as well and this is moving us closer and closer to talking about each service is different each service might be newer than the last one it might be configured

differently than the last one there isn't a way to store a password securely or at least the last time I checked this in Azure logic apps you need to integrate with key Vault you need to get this little visual building block that you can put into this value and then put the password in key Vault and access it that way but you also need to make sure that you tick a box for that block to that says to obfuscate sensitive inputs so that it doesn't log that secret back to the debug logs that Azure logic app keeps basically what I'm saying is that the debug logs are going to show you all the inputs the environment variables blah

blah blah what happened with the execution and if you fetch a secret from Key Vault and then it gets logged to the debug log it's still not great like this is really not a great solution and I don't want to say that Azure is broken by saying any of this it's just that wow shared responsibility is uh still a lot of work so in terms of security concerns around this and how am I doing on time just to check 10. cool um this takes us out of talking about just passwords I want to talk about some misconfigurations and vulnerabilities in Azure that might be relevant to keep in mind when we deal with some of these

things when I talk about vulnerabilities I mean things that weren't supposed to happen that were reported that happened and then were fixed and when I talk about configurations I talk about it's not a bug it's a feature but your business doesn't have the ability to say that anymore Microsoft is saying it for you good things to keep in mind when we're trusting the cloud so the first one I want to talk about is certificate exposure this is a vulnerability found last year and a lot of this will come from netspy they have fantastic research Fantastic Tools so I am relying on it in this case the description of a service principal account those automation accounts we

were talking about before if you looked at the resource Json for that the Json description of it the certificate the private certificate was exposed so this is something we trust to do automation on our behalf we never want to see those credentials again if ever but it was right out there in the properties of the app in the Azure ad application by default most users in your domain are going to have access to the Azure ID portal unless you lock it down I do recommend locking it down um so in this case somebody could have gone in found the basically the password right the private certificate in this case for the automation account if there

were permissions for that account they could have had access to it and done some damage a lot of these vulnerabilities both because Azure is new and because we have to trust Microsoft on it to a certain extent they'll come out and say that there was no evidence of exploitation found so that does depend on what logging they have in place whether things are going to be found but a lot of it is just researchers have found things attackers haven't found it yet they fixed it thank goodness but how long is that going to be the case so this is a screenshot from pre-remediation um this is obviously not the case anymore but when we were talking about

all those little areas where it's like oh if you mark this variable as secret or protected then they'll just blank that out you won't see it in the Json when you look at it this is a case where they should have done that right it is secret it is protected they intended to make it protected but because the back end it still exists in that Json they're just protecting it from the user level View and they forgot to do that another one I want to talk about coming back to Containers an ACI is a container Escape so the cve that we reference here is from 2019 but this was discovered in 2021 where the version of kubernetes

that was running in ACI was vulnerable to a container escape and one of the things that makes ACI cost effective and useful for businesses is that Microsoft is just sharing some infrastructure on the background back end so if you're able to escape the container if you're in your container that's fine you're in your own tenants container lovely day nothing to worry about you can't get at anybody else's data but because we're sharing this infrastructure to save costs I might be right here I might have company A's containers next to me but I'm never going to know it company C might be over here maybe Company B is off on a different set of infrastructure but if I'm able to break

out of my container I might have access to company data that's not my company and nobody likes that so in this case again no evidence of exploitation in the wild was found but the researchers who discovered this would have been able to access data in containers so basically systems company systems that were Running Company apps uh that were not their own not a good day also not a good look that this was found this vulnerability was found in 2019 but the version of ACI deployed at the time hadn't patched it yet in 2021. coming out of the vulnerabilities and into the misconfigurations just configurations uh because this is a feature not a bug if you use Azure ad single sign-on at

all I know we're talking about an Azure product now an active directory not to get too far into active directory but it is possible for usernames to be enumerated and if you don't have MFA set up for users it is possible for those credentials to be brute forced an attacker could Target your users from anywhere on the internet unless you've set up conditional access for it right all these caveats uh for shared responsibility you can set up things like conditional access like smart lockout to make sure that it's not possible to just Brute Force passwords but at the same time if I was running on premises active directory there's no way I would expect someone to be able to

brute force my usernames from off network if I was running a little company Network and someone told me that you know three countries over somebody is trying to see what usernames and passwords work even if I have MFA enabled for my users I'm not comfortable with that at all so coming back around to conclusions um I ran this by a colleague and he said you know really you're just talking about the fact that shared responsibility is your responsibility I don't know if you got that from somewhere but it's a brilliant line and it's really what I want to end with here on premises we know where our permissions stand we know where our access our management stands in the

cloud oh my gosh it is such a mess it is so complicated it's delightful to research and work on but it's also kind of terrifying right as a technologist I love Azure there's so much happening all the time but as a you know practitioner in the workplace it can be really scary sometimes so just make sure you know to come back to passwords to make this a password talk please make sure to educate users especially when you're using a new service on what is secure what is not secure per service if you can get documentation on those things on using key Vault correctly on protecting setting that secure attribute all those things um remember that shared responsibility

aspect even though we're pushing our stuff into the cloud we're getting a lot off our plates with that we still need to be responsible maintainers of the service and just keep aware of what new services are coming out what's getting adopted if a service is newer like logic apps is this security for that adequate for your users is there guidelines you need to give them outside of the obvious to protect things is there policy you can configure with management groups policies so great when you can configure it because you can prevent somebody from doing something silly um and if you are curious about Azure I did just want to mention the learning paths that Microsoft has online are

fantastic they'll spin up sandbox labs for you something that kept me from getting into Azure for a really long time was like I do not want to set up something and get a surprise 200 bill with the sandboxes you will not get that bill it will be on Microsoft you can use it to complete lab activities and it's a fantastic way to learn things without worrying about the costs the credit cards the billing all that nonsense so even if you're not taking the certifications for AZ 900 and 104 I do recommend those labs and at this time I will take any questions

yes

so is azure active directory easier to attack than classic active directory I'd say yes and no they are separate products which is something good to keep in mind they're working towards future parity so things like care broston used to be in Azure ID and now it is I don't know that it can be attacked the same way um I think it is definitely if you rely on Azure active directory you need to make sure you have policies in place to prevent someone from logging in remotely or accessing or brute forcing remotely so I would say if you prevent remote authentication in all the ways that you can for Azure active directory or from you know networks that are not your own

then it's going to be a lot harder to attack um it's easier to expose to the internet than classic active directory but it won't have the same vulnerabilities you're dragging around since the day A.D was born yes

yeah that's a really good one so the questions about finding these issues at scale fixing these issues at scale um there are benchmarks as you mentioned for this kind of thing there's also the ability to set policy with Azure to look for these kinds of certain kinds of things um one thing that I've loved as a technical practitioner not moving towards like Enterprise policy side but there is an Azure resource graph query blade where you can basically run a database query in kql if anyone's familiar with it against all of your Azure infrastructure and that can make identifying some of these things a lot easier out of the scope for today but happy to chat about that more if you're

curious yes

I love that question so the question is can we remove the Json from Azure can we stop people from looking at the Json and Azure not to my knowledge so Azure is meant to be interacted with from a programmer's perspective and a graphical user's perspective Microsoft will protect certain attributes in that Json and you can prevent people from seeing certain attributes based on whether they're a contributor or an owner but you can't really take the Json out of azure and I think that's all the time that I have so thank you so much [Applause]