Infrastructure as RCE: How to abuse Terraform to elevate access

well thank you all for coming out to this talk uh thank you to b-sides NYC uh for having me so today we're talking about infrastructure's remote code execution uh so quick intro I'm Mike McCabe I'm the president of cloud security Partners uh we're a small firm that helps companies with their Cloud migrations Cloud security challenges application security you know pretty much everything in the kind of modern web iot mobile space so we've done a lot of work with clients who are using things like terraform and doing Cloud migrations utilizing infrastructures code and as a security professional I'm super passionate about this because I feel like it's a huge benefit to people in a way to really secure secure their infrastructure I know you guys just care about remote code execution so so what are we talking about do people here know what terraform is or have heard of it or no infrastructure as code some of you probably wrote cloud formation and then cried about that but terraform's a lot better so terraform is infrastructure's code so what does that mean you write code it is interpreted and then you create infrastructure out of that so it's it's codified it's consistent there's a standard API for how you write terraform um it's free and open source the hashicorp the people who created also have terraform Cloud terraform Enterprise so if you like paying for free things you can do that as well so what are some of the benefits of this so I think my favorite term in this space is Click Ops I mean everyone familiar with that basically everyone went to devsecops and now we're going back to click Ops which is basically people clicking in the AWS console to create stuff so this is trying to fix that this is how you centralize your deployments of infrastructure it's really mostly focused on infrastructure not application deployment so it's not that part of the CI CD pipeline you can deploy consistent infrastructure so what we've always struggled with is people deploy something it changes or they deploy it once manually they don't you know commit their code for doing that then they deploy it again and it's different somehow and we have breaking changes and you know every Dev team does it slightly different so you just have a mess in the cloud so things like terraform have the benefit of giving you a codified consistent way of doing it the other great thing that what I was talking about before is you can apply security controls for preventative measures at that level if anyone does appsec you know we've we've always tried to do the SAS model of running tools against our code to try to prevent things like SQL injection or process scripting all those types of things you can do the same thing with infrastructure as code and I'll be talking about some of that as well what are the challenges with infrastructure as code and especially terraform one of the big ones that we see is uh terraform or pipelines are often given really high privileges so if you right now have your pipeline assigned to admin access in AWS you're doing something wrong so if you're if your pipeline can create new users can modify policies you can do everything under the sun you're doing something wrong but that is not very surprising that's that's very typical for what we see for people using you know centralized pipelines to deploy infrastructure or applications um and like the the title of the talk there's multiple ways to execute code with terraform again it's a infrastructure provisioning it's not a you know endpoint provisioning it's not an ansible it's not a chef for puppet it's just meant to provision the infrastructure it's not meant to do and uh you know end system configuration that doesn't mean you can't do it and we'll get into that it's also a really great tool as an attacker or a malicious Insider to gather information about the environment and I'll talk about some of the ways you can bypass some of the controls around terraform as well so what is uh our experience with terraform infrastructure's code so we've worked with a lot of large financials who obviously have a very you know they have a regulatory and compliance regime that means that they have to do things in a very secure very structured way um and they've moved to these types of pipelines where they're deploying things through terraform through uh cloud formation other tools like that so we've helped them build out basically programs to create patterns and security rules and security tools in their pipelines to deploy out infrastructure in a more secure way and the reason why I like infrastructure as code is someone who's you know struggled to help people with appsec for a long time to kind of reduce appsec findings and I've always found SAS tools and code review just to be not something that scales that well on the cloud side you can actually scale this up really well because it is a standard API everyone writes terraform mostly the same way the providers have their own you know API documentation so it is a codified set of set of things you write against therefore it's very easy to check if something is true or false or something is this value or is not this value so actually in some of the environments we've done this we've looked at something you know like a csbm a cloud posture management system and seeing that if they deploy through this through this terraform pipeline if they have all the security controls applied they have zero Cloud findings and if anyone has worked in the cloud environment or has looked at one of the cloud security tools it's all read all the time basically so it's a huge Improvement for some of these places that have a really big cloud footprint so it's a very powerful preventative control and you can map it to your internal and external controls your CIS baselines your whatever it is you want to map these things to it's very easy to do that so um so what does terraform usually look like in a pipeline so this is kind of a typical this is terraform Cloud so this is their SAS platform not just the typical you know terraform running in a Jenkins pipeline or some random devs laptop that you see here and there but um it's very similar to any other pipeline you have a centralized system for source code GitHub unfortunately for some people gitlab I'm just kidding um bit bucket if you just are struggling um I'm just just joking but um so you have your infrastructures code there your terraform committed there and then you have a hook from terraform cloud or whatever it is sem hook to detect changes in your repo that runs the terraform plan and then then eventually apply and then that reaches out to your Cloud infrastructure so that could be AWS that could be Azure that could be on-prem there's a million different things you can use terraform for so that's the general that's a general kind of layout of how people use it either it's seriform cloud or just running internally um so how does terraform actually work so there's kind of a few main pieces to it um when you first when you you write your terraform um you make sure the formatting is good make sure to have your secrets in there because that's how we get stuff done um you run something called terraform plan and that's kind of part of the magic of what terraform does for you is it looks at what you what you wrote and it looks at what you have in your Cloud environment and it basically tells you what are the changes that it's gonna you know what it's going to do is it going to create something is it going to update something is it going to destroy something and that's the kind of plan piece that's the we haven't done anything yet but this is what's going to happen once we do the apply it gives you outputs for what things are going to look like after things are created it creates a whole dependency tree to kind of um show what has to be created before something else has to be created determines unknown values which we'll get into that because that's important from a security perspective um and then once you have that you do a terraform reply if everything looks good and that's where things actually get updated created destroyed whatever it might be so that's when it reaches out to your your cloud apis makes your updates creates your ec2 whatever you're doing um and then what this actually looks like if anyone's never seen terraform before this is a super super simple example but this is an AWS ec2 instance you don't have to give it much in order to get stuff going it's not a very complex language hashicorp did create their own language because everyone has to have their own thing but it's very simple to get started you define a provider you define the resource you're trying to create in this case it's an ec2 you give it a couple different you know pieces of information the image it's going to use the size of the instance and then you run terraform plan and this is what you get out of it which is basically a spec for what's going to be created so you can see the action is create it's an AWS instance which is an easy to all these other somewhat interesting Fields you can see things like security groups they're key names that's SSH key there's a fun one get password data so you can see all these individual fields for the resource that you're creating so that's the plan and then once you do the apply this is actually provisioned or whatever you're doing in your environment and then kind of the final piece of this is terraform state which is uh very important because part of what terraform does for you is not just provision uh infrastructure but also track differences between um what you deployed and what's currently out there so a lot of people will deploy something and then go into the console update it which kind of breaks terraform because the whole idea is to manage it through terraform so you have everything centralized and codified but terraform state is basically that map of what did you deploy what is the current state and then you can update that over time obviously when you run changes that updates they're also doing drift detection soon with terraform and the thing that's most related to Security in this is secrets so any secret that you create in your terraform obviously has to go somewhere it gets sent to the AWS provider and in this case AWS provider and it also ends up in your terraform state so if you're creating an RDS instance and you have a you know default master password if you're creating an SSH key if you're creating you know secret values in SSM or Secrets manager that's all going to end up in your terraform state which um you know if you're if you're running open source and you have this you know on Jenkins or if you have this back to S3 now all your secrets are sitting in whatever you're having your terraform State backed up to uh which is obviously a pretty big issue um and terraform hashicorp I should say the people who create terraform are very well aware of this might be hard to read but this is a GitHub issue on um on the terraform repo it was opened in 2014 uh I took this screenshot a few weeks ago it says commented 20 hours ago and this is an update from a PM saying we're working on this this is coming real soon we we've had you know eight nine years to work on this but real soon we're gonna get this so um and someone said I shouldn't shame the PM so I'm sorry Omar but uh you put yourself out there so obviously it's an issue it's not one that's solved from hashicorp and terraform themselves so it's something you have to solve on your own by storing that terraform state in a very secure place and there's actually systig the endpoint monitoring The Container company they did an investigation where an attacker actually found a terraform State used the secrets in the terraform state to access other systems and pivoted basically so it is a real a real world threat and you have to have one meme per talk and give everyone a second um so that's that's just the before we've created stuff just the kind of terraform infrastructure aside it gets much more complex so terraform is just a language specification it's just you know a tool that takes your inputs and creates some kind of outputs a lot of the magic happens in what are called providers there's AWS providers Azure providers these this list right here is not comprehensive and these are just the ones that Hazard Corp creates there's obviously Community providers people create their own providers and you can do you know if you write a provider to interact with an API you can then use terraform to manage it what that means though is any of these providers are huge they have a huge internal API for how to write code against them I don't remember the size of the cloud the AWS one but it's you know hundreds of thousands of lines so all those apis have interesting edge cases and edge cases and things you can do with them so it's again when we talk about third-party code um you know sourcing from open source we also have to think about on our infrastructure as code side we're taking all this community code and putting it into our environment as well and there could be malicious or they could just be you know dangerous code practices or functions inside these as well so um another fun thing terraform obviously has its own API so if you're using something like terraform cloud or Enterprise it has its own set of apis um the one that terrifies me the most is as I talked about with State it's your it is a representation of what you have in your environment if that's cloud or otherwise uh the thing I always think about is you can do something called a terraform import where you import like current running State into your state and you can manage it through terraform even if you haven't provisioned it through your terraform I just always think if someone could import a production you know all production rds's into their state and their own run terraform destroy the Havoc that you could you could create with that so these apis are obviously pretty privileged you can do things like manage State you can do runs you can do all kinds of things so it's another attack surface within the terraform ecosystem you uh you have to think about um terraform logging this is a fun one this is something we're dealing with a client right now which is terraform obviously has to have credentials into your environment and I'm not sure if people know what that is but basically right there is an STS assume roll session so when terraform assumes a role to work in your environment it has to get credentials back from Amazon to assume that role to that interact with the apis all that is dumped out to the logs if you have verbose vlogging turned on so if you allow developers if you're using something like terraform cloud or Enterprise to set a variable they can dump out the STS role and then have direct access to whatever role you give terraform so that's another area that you have to kind of worry about and this is true for things like Secrets as well you can dump those out to logs um and it's uh another area that you have to lock down and think about when you're building kind of your threat model and your user privilege model for for terraform so with all that said what's that for common for use by competent people only um so that's kind of the lay of the land with terraform um you know I wanted to talk about some more malicious use cases and ways you can abuse um terraform in an environment either malicious Insider or an attacker who gains access to a pipeline or just to a git repo that's hooked up to a pipeline there are dozens these are just kind of techniques that we've we've found and we've helped lockdown but there's dozens and dozens of different ways you can use terraform and either the built-in apis or the provider-based apis um to kind of do different malicious techniques so these are just some of the ones we're going to talk about but there's a lot more so I'm not this is not a complete list of things you can use to abuse um the first one up remote exec so like I said terraform is not meant to be um like a systems provisioning tool they actually have a they have one called Packer which you creates you create images with it you can reference those images in your terraform to build you know ec2 like Linux images things like that or containers but terraform is not meant to be that endpoint management system you take an Ami you reference it in your Z2 config and that's how you move forward with terraform but they do give you a few options um hashicorp very strongly says don't use these but we give you these options which is really handy um so remote exec is basically a way it's like an ansible provisioner where you can run scripts on the endpoint so you boot up an ec2 you run remote exec and you can run whatever you want to as that user on that easy too and this might not be relevant for maybe places that don't have a huge amount of uh kind of security controls when it comes to endpoints if every one of your developers has root on every one of your production instances you don't really care about this but we work with a lot of you know financials where they do care about privilege access management they spend Millions to make sure that developers can't run things randomly on uh on production instances and this kind of bypasses that um so really in this example creating an ec2 instance logging on and then running netcat to get back out so if you're thinking of a malicious use case someone has access to terraform they can't get onto the actual endpoint but they could provision an ec2 run netcat through remote exec and then get remote access back into that production Server Like I said this is basically a way to run any kind of script you want on that endpoint that you set up so it's a great way if you can't get remote access into a production network if you know you'd actually do have privileged access management well well managed and people can't just get access into production this is a great way to bypass all that you could do pretty much whatever your user your user has rights to so if you want to uninstall you know crowdstrike if you want to uninstall whatever kind of endpoint tools if you want to run scripts if you want to get a remote shell back out because no one does egress controls well um you know this is this is your tool to do that and this is built into terraform this isn't something you have to like get a custom provisioner for this is all part of the base API so if you're using terraform right now you can write terraform to do this that's one of the one of the techniques which we usually lock down and I'll show some examples of how we um how we lock this down in terraform pipelines and environments we work working so similar local exec so whereas remote exec is working on the remote instance the ec2 or whatever whatever endpoint that you're creating with your terraform local exec runs where terraform runs so in some environments we've worked in again you give terraform a privileged role to do what it needs to do and then you either do appsec reviews on the terraform you run SAS tools you run different kind of tools that I'll kind of talk about to lock that down but then local exec basically give gives a developer or malicious Insider or whoever it is the ability to run whatever they want to run as that terraform role uh uh so remote exec works on the remote endpoint vocal exec runs wherever you run terraform so again you can run basically whatever you want to run so shell scripts python whatever it is in this case we're curling the STS the metadata endpoint um and then curling that out to a remote endpoint so the idea being if you can run this you can pull out the STS session for the terraform Runner and then send that off to you know whatever host and then you can run that externally outside of the environment to do whatever you want to do um so the problem with you do