BG - Redis or Not: Argo CD & GitOps from an Attacker's Perspective

good morning everybody walk around from that thank you for being here today I hope you're over the jet leg by now so you could stick around during the talk I know I am super excited to be here today is it too loud no okay I'm super excited to be here today talking on besides LV and before we dive into the into the technical part of this talk on a personal note I want to share a story with you that illustrates what I think is my job as a security researcher that is my motivation over 100 years ago a revolutionary technology how do we avoid that a revolutionary technology was invented that technology allowed people to move in a faster and more

personalized way than ever before that is the car by the year of 1952 there were over 25 million registered cars on the road and in that same year an Industrial Engineer named John hatrick was driving to church on Sunday morning with his wife and daughter suddenly a deer ran into the road and John to AO to avoid hitting it turned the wheel of his car and the car went into a ditch John and his wife instinctively threw their hand to protect their daughter from getting hit luckily none of the family members got hurt but it did make John realize that that technology lacked a crucial safety feature and in that same year he invented the

Prototype of the airbag a safety measure we take now for granted just as John didn't stop using the car but instead he pointed out its safety flaws and suggested potential Solutions I believe that my job as a security researcher is to identify security failures in emerging Technologies in emerging Technologies and suggest potential Solutions and and today in our talk we'll present how this mind mind State came into our research in GTO on GTO TOs and Argo CD but without further Ado our agenda for today this talk will have four parts first we will explain why we identifi GPS and Argo CD as an emerging Trend emerging technology worth re researching next we'll have to study

these products what is gp's Manifesto how is Argo CD operating and how does it look like from an attacker's perspective next for our main part of this talk we'll share with you the research story of how we managed to find a critical vulnerability in the platform of Argo CD and what are its implications if being exploited by an attacker and lastly we'll share some security take wave for you to take home and Implement in your organizations drawn from our broader research on gitops security now you must be curious who Am I by this point my name is Orin I'm practicing cyber security for seven years now I used to work in care Bros and networking uh research and

today I work as a supply chain security researcher at pyod hey

hey hey hey hey I'm alad I'm also seven years in cyber security field I'm practicing web application security that's my main my main focus and supply chain security I'm love asking Grand stuff and pretty much everything with nip so I'm also a security researcher here at pyod so both together with and that's our team thank you okay so let's start let's start with identifying a trend why we decided to research Argo CD and giops so you know how starting a new research can be like super confusing experience because you have you have all these new technologies you can start and and research and all these lining lights it's like you you don't know what to do

but this time we tried to do it differently the opposite way we analyzed Trends from major kubernetes and clown native conferences by the talks that were held in these conferences and by that try to identify a trend and there was that word we weren't familiar with that kept coming again and again and in it talks from major companies so we've seen G GPS at Adobe and giops at Spotify and even how gitops change our lives by a person working in VMware okay so these are some serious companies talking about GPS but the next question will be do people actually use it like regular people organizations do they use it and apparently 91% of the respondents in a cncf

survey responded that they already use GTO in their organizations and just to illustrate visually how this crazy number looks like and what even crazier is that out of the 9% holding back over 2/3 claim that they will Embrace GitHub to their organization during the year of 2024 now it is August so as for the question do people use it probably yes and the next thing we'll have to do is find the leading gitops tool we have read that gitops is some kind of way to do continuous deployment for cloud native environments okay sounds kind of similar to supply chain security we could we could uh um try and research that and out of the same

survey on the top left corner the most left corner we can see with over 60% embracement rate a tool named Argo CD and when we went to further research what is that tool in their website we've seen that small some small organizations claim that they use Argo maybe you know some of them the last thing we had to do just to to close this identify Trend story as a security researchers we we just had to go to Shodan to se for public um Argo CD applications and we have found over 12,000 Argo CD applications which made us wonder how many private ones are there so gitops Argo CD was it worth researching We Believe yes but let's try and further study what

are they exactly now aad lead thank you wi so like every good research we started with studying studying what is gitops what is Argo CD and basically be the Argo CD gurus learn it from bottom Up Inside Out learn every feature and be the best in that to even think that we can exploit that so for studying what is even gitops gitops uses git as a source of Truth git repository could be GitHub gitlab bit bucket or any other sem it stores there all the configuration files right there meaning all the deployments all the iac's files all everything that we want to deploy to the cloud everything straight into the git repository so you probably think why save everything

inside G repository that leads me to the second point versioning so you push new features to the cloud every day continuously manually maybe new Services new ingresses new load balancers and one day everything crash you're trying to do and think what made that messed up and what to really roll back and what to do but with gitops that's not the problem all you need to do is go one commit backwards and your all infrastructure will be rerolled and back fixed again so in the bottom line the current state inside inside the repository that contains all those configuration files will represent the state in the cloud environment both of them will be synced so to visualize this let's see

how it looks like firstly at the left side we all we have all the configuration files to the cloud environment stored in a g repository then we have a gitops agent the one that sys every time and all the time for the git repository that we configured and checks if there's some new configuration files to think to the cloud then if there's a new commit a new deployment everything every new uh configuration file it will instantly deploy that configuration file to the cloud saving that uh principle of the source of Truth is the git repository so from an attacker's perspective it's way simpler what attacker wants to sit in production environments to sit on the

cloud environments right there is the all sensitive data so what we what attacker needs to do is to break the gitops agent and that leads me to the best and most popular Argo CD is the the most popular gitops agent is the Argo Argo CD is an implementation of gitops meaning it's focuses only on kubernetes so taking this all concept of thinking G repository to some environment in kubernetes meaning taking all the deployments ingresses services and sync them to our cluster you install Argo CD inside your cluster in a separate name space and then you are good to go all your deployments are synced right in your into your cluster it has thousands of stars in

GitHub and it's a graduated cncf project that only made us understand why it's so adopted and on the community so the same slide here only with the Argo CD all the configuration files are under one git repository Argo CD syncs them all and deploys them to the cluster and make sure the cluster is up to dat with the latest changes so that will be our Focus today Argo CD so after understanding a little bit how it looks like and how the how Argo CD and gitops operates let's see how it really looks like we set up Argo CD application with a default configuration and started pushing some buttons and created a new application like we understood before

new application first thing first connect our git repository the one with all our configuration files all the deployments one sitting okay we can see everything is starting to be synced and after the sync is done we can see the kubernetes cluster is synced with the latest gate commit and we have all of our uh deployments all of our infrastructure synced right into Argo by Argo CD so UI is okay and it's nice but we we really care what's happening behind and under the hood right so let's understand what's happening there we have right there a g commit with the first commit and the kubernetes cluster are synced into the same commit then a user comes one day and

submits a new commit can contain any configuration file any any kubernetes configuration file a repository service a new component along the way being notified about the git commit and makes sure to update a ready cash server then a kubernetes controller takes those changes from the red cach server and updates the cluster so at the end of that process we can see that the uh state of our repository is synced to our kubernetes cluster so after understanding all of those what is gitops what is Argo CD why it's so popular let's move on to the exploitation phase a huge component that we saw before is the red cach server the one that holds all our deployments and is

staying up to date with the latest changes going back going to the documentation and to the learning learning phase we trying to investigate a little bit about that specific red cash instance we are going to the documentation and we see something we really really like the next sentence secrets are available to anyone who has access to the radi instance that's nice we love secrets and we love unauthenticated access so together is Magic we tunneled our way and tried to connect to the r instance and he suced it's nice we we are putting inside the secrets part and trying to eat some buttons under the r instance see what it contains what data we can find and we

see very particular manifest keyy going to that key we see that all the deployments from the git repository everything all the secrets all the deployments all the ingresses basically all the configuration files from the git repository are right in there that made us think as a security researchers how we can interfere in that full flow from git repository to kubernetes Cluster can we maybe in inject some deployment straight into the Ready Cash server will it be succeed so that's our main when thought here so poison the red cash server we are building a malicious deployment that will be privileged with all access to the node host of the kubernetes cluster that will be with that will be with the access to the file

system to the network adapters everything and what it will do is only create a reverse shell to a Ser to attacker server meaning we expect you to get a connection to our server with a privileged permissions we poison the red cash server and we wait we wait a couple of minutes and it failed we didn't get any connection it's very sad maybe we'll start another research idea maybe move to another another platform I don't know a couple of minutes of researching that and we find out why our changes and why we didn't get any connection so our changes were rerolled and nothing is been changed in the r instance so we are going back to

that DB and then we see that nothing changed nothing at all and the red is cach server has been synced again with a repository with in that case a guitar repository few minutes go by and we understand we a an attacker inside a cluster in another name space tried to deploy a malicious deployment by injecting a configuration straight into the radi instance that's okay suddenly the repository service got notified about that about the change and what he do is rethink everything from the git repository to the ready instance so everything is back to normal state and we didn't manage to inject our malicious deployment so what now are we leaving that what will we do will we try to use

the regular flow from the G repository Orin will explain what happens next thank you a lot you're passing me the mic in a complicated stage of this talk but what we did try to do is to go back to the application manifest and see if we miss something and this time we've noticed an entry called Cash entry hash and that entry possessed some kind of a string look like a b 64 so we were thinking could it be some kind of a validation mechanism like a check sum for the application Manifest content luckily Argo CD is an open source application so we could go to the source code and look for the function generating that cach entry

hash and in the source code we've found a function called generate cach entry hash that function receives a structure of the application manifest and Returns the string of the new cash entry hash perfect if they went through all the trouble for creating a validation mechanism for the content probably it's going to it's going to use some kind of private signing mechanism right with a private secret this is what we thought but then we found that comment inside the source code and I'll read it they say hash the Json represent ation into a base 64 and coded fnv that was written inside the generate cash entry function I'll get this part we don't need a cryptographic hash

algorithm since this is only for detecting data corruption I swear to God I haven't changed anything so we were super happy because that means that we could recalculate manually the cash entry has in order to sign our own malicious deployment right so this is what we've done so our second try to inject malicious deployment we inject a kubernetes deployment containing a pod with all the capabilities granted meaning privilege B and his whole purpose in life would be to spawn a b shell to our attacker server but this time we signed our own application manifest using the logic from the source code we waited for the changes to take place and this time it worked we managed to succeed and in our

attacker server we could see a new connection from within the client's cluster granting us privilege access to the cluster and from here the sky the limit it's like every attacker's dream come true when we went back to the Argo CD application we've seen a new record was added and when we took a close closer look we could see that a new resource was created named that ATT attacker spoiler this is our malicious deployment and theoretically if you're being worried about um getting detected we could inject our malicious deployment perform the attack and then delete everything so we won't remain any trace and we won't get detected now how does it look like from an attacker's perspective what could have

attacker done with that privilege access so first it could access the file system of all the pods within the class fere all the production pods we could steal their data which means that we could also seal their tokens their kubernetes tokens to perform actions with their privileges on their behalf also an attacker could steal all the kubernetes secrets in the cluster because he's able to deploy whichever resource he desire thus he can mount the secrets next you're probably familiar with this uh application it's wire shark and the sniff you're seeing is a sneff from a production pod being taken from our malicious injected pod because since we've granted Network capabilities to our malicious pod he

could sniff on the network adapter of the host node and read all the production communication and also it could intersect and perform men in the middle attacks to the to the pods in the production environment and if you're not kubernetes fans don't worry we have something for you as well we could use those privileges to steal the I am role of the host node and from here starts our journey of lateral movement inside the client Cloud environment so that attack Vector that vulner ability really allows attacker to have it all to sum it up the attacker's new capabilities would be he could steal all the secrets in the kubernetes cluster he could sniff all the network

communication and intercept it it could deploy whichever kubernetes resource it Desires in the cluster and Escape to the cloud environment now imagine that any pod in the cluster could perform this attack any pod any compromised pod in the production environment could access the ready server and take complete control over the cluster we have disclosed that vulnerability to the Argo CD team and it got a critical score of 9.1 we were super happy it's like it was the the highest score we ever got so we were very content and to recap the exploit part we have learned that any compromised pod in the cluster could access the ready instance of the Argo CD application which resides

within the Argo CD name space it could retrieve the application manifest of the Aro CD steal the secrets if they exist in there and inject whichever malicious kubernetes resource it desires effectively leading to Cluster compromise okay that part was intense um as for the disclosure here it's important to me to give a shout out for the Argo CD um development team to Michael Leonardo and pav for helping us to mitigate this vulnerability and make Alo CD safer for everyone as for security takeaways I have four things for you today the first one is pretty straightforward update the version of your ago application that's easy that's an easy one to a version which is not vulnerable vulnerable to that ATT

attack the next three are going to be General GTH of security takeaways drawn from our broader research on GTH of security and these are as following first deploy your G up tool in a separate cluster since if the client or someone have deployed the Argo CD in a separate cluster a compromised pod in the production environment couldn't have access the ready server the same logic goes for the second uh takeaway the enforced Network policies Network policies in kubernetes are like firewall or IP tables and by enforcing an allowed list that allows access to the gitops resources only to resources that should access them in the first place you could have mitigated that vulnerability easily and it's important to make sure you also

have a cni plugin to enforce those Network policy rules and the the last one is the Golden Rule it's a principal for cloud um environments in general or any environments actually deploy GitHub tool in a least privileged um with the least privileges possible instead of deploying it with the default admin PR permissions it's better to customize it to your privileges that um you actually need in your organization now let's watch a live demo that was recorded before here we can see our attacker server we're listening on Port 50 852 this is like a normal ec2 machine and we'll wait for connection from within the client's cluster and this is the compromised pod it could have been compromised by a

webshell or whatever it is a pod in the production environment and it has low um permissions in the cluster it is not supposed to be able to container Escape neither it will privilege es escalate with inside the cluster and next we will see the Argo CD application and this is the same um structure as we've seen in the present station and right there on the right you could see this B this is where we are again it's a production pod that has been compromised with some low privileges in the cluster perfect now let's try and explain that we have uh written a tool to um to exploit that vulnerability and it has two modes it has the detect mode

which will allow us to detect the Argo CD ready server and the server and it has an exploit mode for detect mode it will try to resolve the fqdn of the Argo CD ready server against the kubernetes DNS server inside the cluster that way you could know the IP address and we could see that the Argo CD applications U version sorry the AR CD version is prone to that attack it's a public API the version next for the exploit the exploit will require two flags first one will be the malicious deployment to inject it's a kubernetes yel um infrastructure we will want to inject and we used um malicious deployment taken from Bad pods public project with WR by Bishop Fox so

thank you very much for that and it contains a pod with all the Privileges that will connect to our attacker server and we will also give it the red IP address when we hit enter we can see that it found one application manifest it injected the malicious deployment and recalculated the cash entry hash and the attack completed successfully and when we go to the Argo CD application web view we can see that a new poll was created that ATT attacker that is our malicious pod and now when watching our attacker server you wouldn't be surprised that we have received a new connection from within the client's cluster that allows us everything on the cluster to

compromise the cluster as a whole and that is with without any privileges in the cluster I'm happy it worked that time the recorded demo was a good idea um that's it for today if you're interested in more technical details about the research or some um more detailed remediation guidelines for that vulnerability please read our blog post on the on the vulnerability and on a personal note I had great time thank you and we are very enthusiastic for supply chain security so if you want to collaborate and you have any ideas or questions please feel free to DM us on LinkedIn and we'll be more than happy to answer and if you have any questions um now it could be a good time we have

time right yeah perfect thank [Music] you if anyone has any questions just raise your hand and I'll walk around with a mic so everyone can hear we'll also stay after the talk if it's like a stressful um position position all right thank you very much much have a enjoy the rest of your day