Rusty pearls: Postgres RCE on cloud databases

Thank you. Okay, so let's start. Um, I'm T Pelleg. Um, as was mentioned, uh, I'm a cloud security research team lead at Veronus. I don't need the slides to know that. Um, yeah, I like hacking things, breaking things, fixing them, writing music astrophysics >> and I'm Kobe Abrams. Um I work with Tal at Veronis as a cloud security researcher and also working on is research and I have a passion for teaching and specifically teaching cyber security. So what's about to happen? Um first we're going to talk a little bit about the vulnerability that we found just how we run it, how it runs, how it works. And then later on in the presentation, we're going to talk a little bit about

the methodology about how we found this vulnerability and how we looked for it and what we did with it. Um, of course, we'll share with you the different things that we learned along the way and hope you enjoy. If there are three things you want to take away, one, we found a cool RC on Postgress databases and it partially worked in the cloud. Two, you should be looking for vulnerabilities everywhere on all managed services, not only workloads that are more popular today. And three, collaborate before running a remote RC during vulnerability research. Okay, so a little bit of background before we start just on Postgress. Uh PostgreSQL is a relational database. It's open source. It's extendable. So it

has a really great uh platform for writing extensions for it. uh we really like Postgress and because it's so common you can find it on almost any cloud provider out there. So there are two parts to this story. The first is about using Pearl to modify environment variables in Postgress sessions. So Postgress has extensions that add functionality to the database and one type of these extensions are language extensions. Language extensions allow writing custom functions in various programming languages including SQL, C, Rust, Python, and Pearl. Therefore, there are trusted and untrusted languages. So, untrusted languages can access the system environment variables. They can open processes, sockets, access the file system. Thus, they can only be run by

super users. While trusted languages have limited access to the system, regular users that aren't super users can only run trusted functions, so they can't access the underlying system of the database. We were able to break that assumption by using the trusted PL language to modify environment variables, something that only untrusted languages should be able to do. So, how does that work? Well, Pearl has this magic variable, the end of hashmap, and it's used to manipulate environment variables, read and write environment variables. This functionality is embedded deep within Pearl. So, we thought that it wouldn't be overridden by the trusted PL/ Pearl, and we were right. So by by writing a function like like this one uh we can set the path environment

variable of the current Postgress session uh to whatever value. Okay. So now at this point we have a pretty strong uh primitive right we can change environment variables and usually we can elevate that to code execution. Um so what we're going to talk about now is how we actually elevated that into code execution. Uh in order to do that we used another language extension also trusted called plust. And a little bit about rust rust is a compiled language. So when you're creating a function in Postgress for for rust uh you're going to be rust needs to have cargo called in order to compile that package right so cargo is then going to call the the the rust compiler

which is rustc in order to create the binaries and all that is happening when you create uh a function in postgress. So that's good for us because when we execute uh when you execute binaries, you're loading environment variables into that binary. So you can see that Rust tried to remove some of the environment variables that we have, right? They're trying to clean the environment so that cargo can run um cleanly, but they use the denial list. So you can see here the interesting environment variable that we looked at is called uh rusty wrapper. And what rustc wrapper does is basically tells cargo instead of running the rust cyc compiler the rust compiler uh it tells

it to run a different binary that's supposed to wrap uh supposed to wrap rust. Now we can use that to point it to whatever binary we want but at this point um we couldn't run that because it's you can see it's removed from the environment before we run cargo. Now a different environment variable that has the same exact effect uh is is not removed and because it's a deny list we can change it to utilize it to run any binary that we would like. So this environment is cargo build rust c wrapper. Now we can't yet control the parameters that are passed to the binary that we run. So we needed to find a different

way to inject some kind of bash command. So what we found was Rust when it's installed comes with Rust GDB which is a bash script that runs uh a different environment variable called also rust GDB. So how does this look? Cargo when we create a PL/ Rust function is going to run either Rustcy or whatever is in cargo build RC wrapper and we're going to point it at the Rust GDB script and change the Rust GDB parameter to contain whatever code we would like and that's how we can change the parameters and inject the things that we would like to inject. So let's see this in action.

Okay, so first we're going to create the different extensions that we need in order for this to run. So we're going to create the PL rust extension and the PL pearl extension. Uh after that we're going to create a PL pearl function that's going to change the environment variables that we would like to change. You can see that here we're changing the cargo rustc wrapper environment variable to point to rust gdb and we're also going to change the rust gdb parameter so that we can inject uh bash commands and then we're just going to create a pearl function uh a rust function sorry and that rust function doesn't need to do anything special it just needs to be

compiled so when we run the pl pearl function uh it's going to compile the pl rust function with our environment variables And as you'll see in a moment, right, we get some kind of error, but at the end we have our std out. [applause]

Okay, so what's going on? Well, you probably have some questions. How did we find this vulnerability? Why do we choose Pearl and REST? And also we promised to talk about cloud RC. So the idea started after we disclosed a very impactful SQL injection on a database that was hosted on AWS RDS. We wanted to see what impact a real malicious actor could have on a postress database. So when you create a database in RDS, Amazon brings you an admin user that while it can modify the schema and and manage data, it's not a super user. So we wanted to find a privilege escalation to gain super user access or even better try and execute code on the

underlying system. So now we have our objective running shell commands on Postgress as a non-super user. This is especially interesting in the cloud since breaking the barrier of trust between the user and the cloud provider and running in areas where the cloud provider is responsible may potentially allow exfiltrating cloud credentials or even moving laterally between cloud customers. So, um, extensions are a common area to look for vulnerabilities and we chose Pearl. Um, we chose Pearl because it's in the official Postgress repo, so it comes built in with lots of Postgress installations. Um, and that means it's a big it will be a bigger impact. Um, it's from the 80s, so there's a lot of code written in

Pearl. Um, and it's difficult to change all that code. Um, that's before security was really popular, I guess. Um, and Pearl has these built-in magic variables. Uh, and it's open source, so it's easier to look at. So, we have, uh, we ch we that's why we chose to look at Pearl. And we found that we could use these uh magic variables in order to change environment variables. So at this point right in our research we have this primitive where we can change uh the environment variables and usually that's really easy to elevate into code execution. Now the problem that we came into is a difficulty that Postgress actually created. Um Postgress has an API for extensions to create

different uh processes. So if you want to create a process from an extension it's not doing that from your own process. it's going to send that to Postmaster and Postmaster is going to create the process, meaning that the environment variables that you changed, they're not going to be in that new process. So, what we needed to find at this point, what we were really looking for is an extension that an extension that can can run that runs a binary directly without using this API. Um, this is where GitHub, Search, and Kenopilot came to the rescue, right? We're looking for these in open source code and we just looked for, you know, different functions that run binary. So

system or the execv family. We also looked at get, right, to see if maybe one of the extensions is loading environment variables in the middle of the process. Um, and in Rust, we searched for the command uh the command function that creates uh new processes. And that's how we found out that Rust actually compiles its packages uh directly within the extension. Now we promise you a cloud execution, right? So we did briefly run code on uh a Postgress RDS instance in AWS. Um why did we attack our own cloud database? So like Tal said earlier, we were looking to kind of see if we could get some network information. uh see if we could get cross tenant access. We

were just trying to test for different maybe container escape vulnerabilities that may exist. But what we found was a very limited environment. Um there wasn't much that we could do. There weren't very very many binaries that we could run. And this actually created a problem for us because we couldn't run the Rust GDB script uh because it relied on some binaries that didn't exist in the RDS uh the RDS instance. So we had to get a little creative and we end up using uh bash environment variable which is just expanded uh when bash is run to run our code. It's a really cool way to leverage environment variables I think. [snorts] And we also found that the AWS incident

response team is is really really good. They're really fast. Uh so within a very short amount of time from when we started running code uh we had emails, we had LinkedIn messages, we had uh messages from our CEO who got messages from Amazon, we had all all kinds of um interactions from Amazon trying to figure out what exactly we were doing. And they also >> collaborate. >> Yeah. [laughter] >> They they also put our instance um into the storage failed mode just in case we were thread actors, right? I mean our our database had like information that we were researchers but just to be safe. So really really effective and in the time that we were running we weren't

able to really get any kind of impact um which is why uh the the impact of running this vulnerability on AWSRDS was was minimal. Okay. So we can't leave you without some takeaways. Uh so first of all if you're managing a database uh keep it up to date of course um especially at least the minor version um all of the minor versions since postgress 12 have uh fixes uh lease privileges if you don't need people to create functions to create extensions don't let them do that um of course and limit the extensions that can be loaded so there's uh allowed extensions parameter in Postgress. Um that will limit what extensions are allowed and this is a white list allow list in in

this example in this case. Um you have anything else? No. Okay. So something we would also like to to leave for cloud providers is um managed database backends are are kind of prone to these vulnerabilities. This this isn't the first time this has happened. Uh you can read the speckle umbrella story uh from Imr that's really interesting about how he ran code in a similar way on GCP. We also recently found a vulnerability that allow us to run code on Azure Cosmos for PostgreSQL. Um that was also really interesting. We haven't yet published the the information but uh we should soon and should read about it. But the point is that these databases and these services

um have these kinds of vulnerabilities and we need to have some kind of um responsible community research, right? We need to be able to have researchers from the community checking these environments, checking to see if we can do the things that we were trying to do like network access. Um try to see what kind of uh credentials we can find, see if there's any cross tenant access really just in a responsible way of course, right? We can't just have anybody running code in in AWS or in Amazon or uh Microsoft. But I really think it's important for us to have something like this. Finally, for us um for researchers, um first environment variables are fun.

So look for custom environment variables. They might allow you to run code. Um and prepare before running code on sensitive systems like AWS. So collaborate with the cloud provider. Um if you're about to enter cloud managed territory, uh remember to take pictures along the way. We forgot to take pictures of most of the exploit. Um call your database uh with a indicative name. So we called our database Kobe. Um and that helped AWS contact Kobe uh via LinkedIn. Um, he's apparently the only Kobe at Veronus that writes it with a C >> and a Y. >> Um, yeah, and and and that's how they they knew that we were okay. They they contacted Kobe and and asked if our

account was being hacked or if it was us. Uh, and collaborate. So, as we're not too happy about having lots of unexpected alerts appearing just before the weekend. Um and finally set your goals before researching. Um so make an assessment of the potential risks and think what would interest an attacker. This is how we got there. So uh to summarize three things you should remember uh we found a cool RC on Postgress and it even kind of worked in the cloud. Um look for vulnerabilities everywhere not only workloads. Um you might be assuming that something is just static or or but everything in the cloud is made of many microservices. There can be vulnerabilities anywhere. Um and

finally don't exploit on a major cloud service without telling them in advance just before the weekend. >> Thank you. Thank you. [applause]

Thanks for the talk. Um, I was wondering if you looked at the ASM like embedding ASM inside of a ROSS uh function or like your extension because we and call a sys call directly using ASM or is that like banned from from the PL uh PG ROS? So, Rust has both Rust and Pearl have some really interesting mitigations in order to turn them into trusted languages. Um, you can't just run anything from Rust. Uh, we did look to try to see if we could use any system calls or use any kind of functions. Um, but those are are pretty well limited. Uh, Pearl, you know, the function call that we were able to get was set n

through that um through that hashmap. Yeah. Anybody else? We got a minute or two. Y'all not awake still. All right. Give it up for Kobe and Tal. [applause] Thank you.