GF - Code Dependency: Chinese APTs in Software Supply Chain Attacks

it's great to be back it's wonderful to be here at b-sides las vegas after a very long time apart and i hope you are all having a terrific reunion and experience at summer camp 2022. so thank you for your time and to join me here with my talk today a little bit about me i am a strategic threat intelligence analyst a specialist with a bank i won't name the bank and i've been doing that for a few years now i'm also very happy to be a founding member of the diana initiative where we promote diversity and equality and bring new ideas and new people into this realm so that we can all grow better together

today was our kickoff day after a long time away and finally uh an obligatory disclaimer the views expressed here are mine mine alone and not those of my employer okay so in this talk what i'd like to talk to you about are software supply chain attacks what are they and about the growing threat that they pose we're going to take a look at code dependency as a factor and then we're going to talk about how that is compounded by mistakes and misconfigurations then we're going to take a look at a brief history of some major attacks and then the state-sponsored actors who are involved in those attacks who's been watching and what they're leveraging finally we'll wrap up with what we could

be doing better and some suggestions we don't fully appreciate just how interconnected we are and that translates into software code dependencies and it took an event like the solarwinds orion hack in late 2020 to rattle the bars on that cage and to wake us up to what's been going on for some time the reality is that software supply chain attacks aren't anything new they've been around for many years and we've been watching that check engine light for some time but we haven't really addressed the issues so my hope is that the knowledge and examples that i can share here with you today will give you some new insight and awareness so that you can go back and

better secure the projects that you're working on solarwinds was truly unprecedented how many people here had to work got the call to respond to that yep this is the kind of event that we we talk about for decades it's a it's a teaching point a learning experience a case study it's something that we can build and learn from that's the important thing now what you're going to see is that there have been a lot of software supply shenanigans since that point what is it it's an abuse of trust and as we like to say in the security community trust no one or at least trust but verify this involves compromise right at the source it's the insertion of malicious code

into legitimate software and that gets distributed on mass so you have the compromise of one trusted source and that's done by the adversaries because they want to control your distribution system so that they can deliver malicious updates this then creates access points into the networks of those targets customers thousands and thousands of customers as we saw with solar winds but with other attacks similar as we've learned nobody is untouchable and everybody no matter how well defended you think you are is a target as for the impact well we're looking at being incapacitated disruption sabotage if you measure your productivity and your profit by downtime this is for you it's for all of us and if you consult the mitre attack

framework this is referenced as t11954 initial access you want to know what initial access is that's how the hell did they get in all right so software supply chain attacks take time resources and skill to plan that's the purview of state-sponsored threat actors typically russian or chinese-backed groups there's also cyber criminal and state-sponsored groups who are targeting the technology industry why because these companies are relied on by so many organizations and individuals and that is a wide-ranging impact there is a lot of happy hunting in there attacks on tech companies enable third-party compromise of enterprise customers enterprises big enterprises money enterprise is a very juicy target and china has actively targeted tech companies in taiwan with supply chain

attacks why because they see them as a threat now an important disclaimer i'm not a developer or a coder and i'm not even gonna pretend to be what i am is a threat intelligence geek it's my day job and that's how i'm presenting my research for this talk what i do plays a key role because we watch the geopolitics play out then we look back for context historical context because we're looking at patterns of behavior history repeats and actions have consequences this is from ian pratt who is the global head of security for personal systems at hp

there's nothing like getting some historical context so let's review what's on this timeline starting in 2009 we have operation aurora we'll move to 2017 for not pecha and the sea cleaner 2018 we'll talk about shadow pad 2019 there's operation shadowhammer in 2009 operation aurora was a wake-up call the chinese state-sponsored group apt 17 also known as elderwood targeted google and adobe among 34 other significant companies and they went after them for their source code management systems why because they wanted to alter the source code kind of like the goose and the golden egg now when we talk about digital crown jewels we need to be talking about this specifically because this is at the heart of software supply chain

compromise and if you alter the source code well imagine what you could do so that attack was sophisticated it exploited zero-day vulnerabilities in internet explorer and vulnerabilities were also leveraged in a product called perforce and this is source code revision software that was being utilized by google you can just carry that concept right into present day same kind of scenarios exist

so the good news is that this event led google to take some action it implemented zero trust and it tracked lateral movement and it put in place better infrastructure interestingly enough google wasn't involved in the carnage of solar winds and then 2017 there was not petcha is everybody here familiar with that story and with what not patio was okay i will enlighten you this was the work of an elite russian hacking group known as sandworm and it was part of the gru military intelligence services they compromised and they took over software updates for a mandatory accounting package known as emmy doc this was used throughout the ukraine it was tax software similar to i don't know

quicken quickbooks turbo tax but everybody pretty much had to use it to submit their taxes so russia has used ukraine as a cyber petri dish for years they have a rather hostile relationship and russia has control issues so soundworm was detonating logic bombs in government and companies all over ukraine now there was a little family-run firm known as lincols group they were responsible for pushing out those software updates for emmydoc that was their job what sandworm did was they hijacked those update servers and they installed a hidden back door so that they could access all the pcs that we're going to be receiving emmydock

and in june of 2017 sandworm effectively pushed that play button to launch not petcha this was more than just destructive malware this was a cyber weapon lobbed at ukraine and it was designed to spread with breathtaking speed which it did and when you saw it the damage had already been done it leveraged eternal blue and mimikats to get complete control now it was supposed to be a punitive measure against ukraine however it escaped its confines and it spread to major companies globally like the shipping giant maersk the unintended consequences were costly there was a lot of collateral damage and at this point we're all pretty familiar with what the impact looks like to supply chain because of covet

this is a global impact the costs of this one was 10 billion dollars all right shadowhammer january 2019 it's a sophisticated supply chain attack against a company known as asus who here might be using an asus product yes it is big this was by a group apt-17 also known as barium who've been behind a number of software supply chain attacks and they targeted the asus live update utility now according to gartner in 2017 asus was the fifth largest pc vendor that's a huge customer base that is a very juicy target and that live update is a pre-installed utility on most asus computers it's a thing designed for convenience and most end users don't want to have to worry about

updating their bios or their uefi's or their drivers they would welcome this kind of a facility so what happened was they sabotaged the developer tools to be able to get in and run what they wanted on this particular utility they modified an older version of the asus utility and then they signed that malware with two asus digital search then they stored it on official servers how would you even know it was there then they pushed this out to over a million laptops to be able to control more remotely and install more malware

okay nick weaver is a security researcher at the university of california berkeley institute and he shares this you are trusting every vendor whose code is on your machine and then you're trusting every vendor's vendor take that home with you

all right let's talk about our code dependency you may grow that was not funny sorry so the fact is applications increasingly depend on external software to work proprietary code of open source components third-party apis modern applications are just really big it's too much for one developer really to have to do it on their own and software reuse has become the norm popular open source projects are they're used as dependencies and that's a word if you're not a programmer if you're out on the other side you don't realize what that means until you realize what dependencies mean now these are attractive targets to the adversaries because then they can add malicious code to those dependencies and then claim the users downstream

so we need to appreciate the increasing complexity of what we're building in this digital realm of the applications that we rely on and how the functions of just one library rely on a number of other libraries in order to be implemented that's a lot of things to keep track of and this creates complex dependency trees that developers either won't remove or update because of the possible downstream consequences does anybody has anybody ever found themselves in that position

let's do a quick little look at some numbers the average app has about 118 open source libraries an average library is about 2.6 years old your average jeff java app has 50 open source vulnerabilities 99 of organizations have will at least one high risk java license and false positivity rates if you're using these tools for java there's 23 23.net is 13 and node is 69

let's take a look now at four elements of your software supply chain there's what you write that's your custom code that's developed in-house there's what you build with so that's your software development tools there's what you buy off-the-shelf software as a service type products and then there's what you use those are third-party libraries and we need to understand the areas of risk around open source libraries and frameworks which can come from active or inactive libraries or library classes and the age of the library licensing and vulnerabilities i learned a lot about libraries i had no idea how pervasive they are how vulnerable we are because of this use very good to be able to share this with you

this is hard to do if you don't have visibility into which libraries and classes are being used by that software and when you're only relying on static scans from legacy security code analysis tools that are out of date so when you change something it can't capture that change or what breaks this is from a 2021 contrast labs open source security report it's a lot to take in but key points would be one if you are using open source code you need to be paying more attention then you happen to what we've been doing to identify and remediate things it's not comprehensive we're missing stuff and three those legacy security tools and use are not securing us

this is a timeline of software package repositories that have been in supply chain attacks part of my day job is to be tracking all the events all the attacks things that are happening i have seen a steady uptick in events involving software code repositories github npm npm npm pi by it's just tracking like this it's really easy to get code from various projects online and then you incorporate that into other software but the risk some of those open source projects are not maintained well they may even have been abandoned you're putting yourself at risk and you may not even realize it code reuse is a deliberate strategy of scene is a good thing it's a differentiator why

because it's for efficiency and innovation it allows for speed and application development these are things that we want they help us grow however the cost is of being vulnerable to compromised off-the-shelf components some of that repurposed code can come from internal repos but a lot of it doesn't it comes from open source libraries third parties and for an attacker compromising a software supply chain can come through manipulation of the application source code or you can manipulate how they update and distribute things or you can replace compiled releases with a special modified version your targets can range from very specific to broad stuff happens and that brings us to this the continuous integration continuous development pipeline now this

is considered a best practice for devops because it delivers code change frequently and reliably who here is familiar with the concept of agile so ci is a coding philosophy it's the way you do things it's set of practices that help drive development teams to implement small changes incrementally and they check in the code to the version control repositories frequently consistent automated and you can build and package and test your applications thing is you're still using other people's technology and unfortunately snakes mistakes will be made so this brings us to sonar cube is anybody familiar with this product it is widely used okay november 2020 the fbi issues a flash alert warning threat actors have been

actively targeting vulnerable sonar cube influences instances since april of that year they're accessing source code repositories and then they're exfiltrating the proprietary code of us government agencies as well as private organizations a lot of people got hit they found multiple potential computer intrusions that corresponded with leaks associated with that sonar cube configuration vulnerabilities which had been exploited for months before back in april it was in august of 2020 that an unknown threat actor leaked internal data from two of the organizations through a public lifestyle repo tool they scanned online for exposure via a default port 9000 which is the publicly accessible ip address and several of the source code repositories also contained hard coded credentials

which just opened the door to accessing other resources and expansion of that breach and i can tell you based on the research we've i've done from 2021 all the way through 2012 we're increasingly seeing stored things keys secrets hard-coded credentials in source code it shouldn't be there with regard to that sonar cube event sikar zarakai wrote this oops sorry

misconfiguration is constant it shouldn't be a constant but it is

all right let's talk about some major software supply chain attacks from the past and lessons that might not have been learned if that looks busy yes yes it is so the identity theft research center stated that 137 organizations reported they were impacted by supply chain attacks at 27 different third-party vendors that was an increase of 42 in one quarter attackers manipulate software dependencies and development tools so that they can compromise data or systems before they reach the recipient as we've seen they go after source code think of recent attacks think of lapses group for example what did they take they took a lot of source code microsoft said that they had gained access to some of the source code for

exchange and azure and intune okay if you're in an enterprise you're using exchange and you're using azure having that source code gives adversaries the advantage why because they can be the first to look through it find the vulnerabilities and exploit them it's code there are vulnerabilities in there they also go after certificates and certificates are important because if you have stolen code signing certificates you are now giving that malware legitimacy it's going to bypass detection and antivirus and protective measures and then you can send malware payloads as a legitimate source and those certificates are simply a staple of attacks let's talk about attacks there's vendor compromise which is considered among the most sophisticated exploitation of third-party applications

could you find and exploit the vulnerabilities especially zero days and then these bugs can enable that attacker to manipulate software to do things that it was never intended to do stealing credentials for example accessing information that it has no business accessing exploiting open source libraries we know these have vulnerabilities and they can be exploited it's different from software packages because it's harder to update and maintain open source code here however because the source code is available attackers have the opportunity to study it and practice on it case in point apache 2 apache struts 2 framework there have been 13 critical zero day disclosures just in the past five years dependency confusion that's an attack we'll talk about later

and then there's hostile takeover this is an example where somebody will take over in a community a project offered to look after it they'll be handed over the control but with that they can then feed malicious content downstream in 2018 there was an event involving event stream and node.js and it was caught because there were mistakes and the people at node thought it was just bugs fortunately they caught it they were able to flag it down i found this as i was doing my research and i thought it was a great way to share what a plan of attack could look like from the back stabbers knife collection which is a pretty cool name you start by injecting malicious code

into that dependency tree and then you can create a new package with unexpected features like a trojan horse or typo squatting or use after free you can also infect an existing package and then spread some malicious goodness by either injecting it into the source or into the repository system or during the build which was what solarwinds was and then it component the build system and it contained packages downloading that brings us of course to solarwinds so this is a trusted network management and monitoring software that's used by the government fortune 500s microsoft firewall so many people and nobody knew that anything was wrong and it's described as the most significant attack of our time it's

pretty pretty true what's distinctive in it is that degree of stealth that was used the ability of the adversaries to conceal their actions and the length of time it took for discovery this was an operation that took time and precision to do the reconnaissance sophistication to tailor and patience and in my day job what i'm doing constantly is anatomies of attack understanding the adversaries and this stands out from all the attacks we've looked at we spent a lot of time mapping this out and looking back you just would go oh my god they were in there for how many months they targeted and compromised that build environment and the code signing infrastructure for orion that gave them complete control

that's your level of trust there that's just been completely compromised they modified the source code to add a back door and signed it and then they leveraged the existing software release management system and used stolen certificates which let them laterally move through chains of trust who here has heard of dependency confusion this is a wicked cool attack and it's increasing good so security researcher alex person shot a bright light on a very scary possibility he took a hypothesis about a supply chain substitution attack where you have a software installer script that tricks people into pulling a malicious code file from a public repository instead of getting the trusted file with the same name from the safe internal repository

to griva's point he targeted apple microsoft tesla and 32 other companies and executed unauthorized code inside their networks well that worked but within 48 hours of it being publicized a lot of other parties who did not ask permission to do this thought it was a very good idea too jumped on the bandwagon and there were more attacks and i can report we are seeing an increase of these type of attacks from 2021 straight into 2022. and the issue here existing dependency scanners do not detect if a dependency executes malicious code and since these tools are limited to identifying dependencies with known vulnerabilities there was an interesting study done as a result of this at red hunt labs and they

found these results 93 repositories out of the top thousand github organizations are using a package that doesn't even exist on a public package index which can be claimed by an attacker to cause a software supply chain attack of the top thousand organizations that were scanned 212 of them had at least one dependency confusion related misconfiguration in their database why does this matter these are big names that are relied on by many many users so if one of those projects gets tainted there's a high probability that potentially millions of users will be affected xcode spot how many people use ios yes well this is a malicious project that affects the free application development environment and a new malware variant was observed

targeting ios developers in a supply chain attack they abused legitimate development environments from apple the targets were sharing sites and repositories this is common practice it's encouraged it's a belief system this is a total abuse of trust

how many people are familiar with the company code cover and what it does there was a major breach involving code cov 2021 it's an online platform and it's used for hosting code testing reports and statistics and it gives developers developers tools that will help them to quantify how much source code is executed during testing it's supposed to be very safe it's trusted they serve over 29 000 customers globally many of these are enterprise level godaddy atlassian banks procter gamble however the impact spread beyond that to thousands of public development projects as well ansible kubernetes rapid7 twilio even e-commerce platforms so on april 21st of 2021 code cult reported a supply chain attack that had actually occurred back in january that's

a few months lag time that's not going to help anybody and what happened was attackers had leveraged an error in the process that creates code cuffs docker image and this lets them extract credentials which protect the modification of something called the bash uploader script and that's the tool used by customers to send their code coverage reports to the platform and what's in there all kinds of secrets credentials etc so that the attackers could reroute this once they received it and gain that information

and then they sent that information to a server outside there was the cassaya breach again this is important in the way that it was trusted by the users there's over 40 000 of them it's a it solutions developer for managed services providers and enterprise level organizations their product vsa is a unified remote monitoring and management tool for networks and endpoints this has a high level of trust on customer devices sounds familiar doesn't it this has so much trust that the attached client devices will do whatever they are told and that's probably one of the reasons why casey was targeted rival ransomware operators delivered a fake vsa update that sent a ransomware payload from compromised vsa servers to

vsa agents that were running on managed windows devices that malicious update was dubbed the vsa agent hotfix the initial access came from an ex zero day vulnerabilities that existed in cassay's vsa software they were reported they were known but they hadn't been fixed and they were leveraged and the attack was triggered through an authentication bypass vulnerability in that case a vsa web interface so the attackers could circumvent authentication controls they gained an authenticated session and then they uploaded a malicious payload they executed commands using sql injection and they achieved code execution in the process i'm not super technical but we had to analyze how the hell did this happen and what did it mean and there was a lot of

learning and i'm hoping that this will help you too they then attempted to disable ms defender using a signed certificate the trust issue here that can say a hotfix that was leveraged was ready to go thanks to the anti-malware software exclusions that had already been set up

so who's maximizing these opportunities of all the state's sponsored adversaries chinese cyber espionage groups have been and will continue to be the biggest threat to tech they conduct economic espionage and intellectual property theft they are very very good at what they do and there have been repeated warnings again just this year by agencies and governments in the us and the uk about stuff they are finding which is very alarming what's been accessed what's been stolen so technology companies are rich targets just on their own but these groups leverage them to invest supply chains and then to go after those customers don't underestimate the capability or intent of these groups if you want to try mapping apts to

government and military in china i found this work by anastasius pingus and i'll share the link here it's complex but it's interesting all right attacks by china these are some of the most well-known attacks involving chinese threat sponsored actors the russians are formidable but you really don't want chinese adversaries in your network in 2017 there was king slayer this involved the chinese threat after group apt 19 and they went after an administrative software package known as evlog a targeted enterprise sorry enterize operations globally and this included military organizations and defense contractors this is something the chinese care very much about they're highly competitive with the west and they want to gain the upper hand at any

cost these are high value targets it's an impressive client list and it is very similar to solarwinds key point here several years later and we've still not known how many of these customers were and could still be compromised cc cleaner you've probably heard of it it seemed like a harmless product until it wasn't this involved apt 17 axiom group again on the alteration of source code of a product that was widely downloaded and convenient for customers how did they get in there well initial compromise was through an unattended workstation of a ccleaner developer who was connected to the piraform network using a teamviewer remote they reused credentials from previous data breaches we've heard that many times in order to access the

teamviewer account and then they delivered malware using vbscript they developed a malicious version of ccleaner and they used rdp to set up a back door on a second unattended connected computer there they dropped a binary and another malicious payload and went on to claim more and distribute millions of versions shadow pad apt-17 also known as barium they delivered shadow pad malware to infect enterprise networks by a product known as netsurang netsarang specializes in server management and security connectivity software used globally by major organizations where have we heard this before so a module was hidden in a code library it was making suspicious dns requests dns it's always dns and it was sending data back to the

operators who then made the decision whether they wanted to act on it or not

2020 able desktop there's a chinese apt group known as lucky mouse who went after chat software that was being used by the mongolian government and businesses there they hijacked updates in the software supply chain and the attackers didn't need to steal or forge an update signature because able's updates were not signed an operation sign side targeted the vietnamese government looking at their certification authority they compromised that this software was mandatory and widely used throughout vietnam the key point here is the targeted abuse of trust by leveraging a service oligopoly this is believed to be possibly a group known as ta-428 who target east asian countries and also russia to gather intel finally 2020 golden spy this is

intriguing because this is malware embedded in required tax payment software issued to corporations who want to do business in china and the chinese banks require businesses to install the intelligent tax to pay the local taxes it's produced by the golden tax department does it sound familiar a little bit like not petcha perhaps well the malware installed a back door on systems which enabled a remote spread actor to execute windows commands and then they could upload and execute any binary including ransomware it's persistent you can't kick it off the system that malware gave system level privileges and it had that capability to install anything on a system where it's installed and then it was connected to a c2 server

that was distinct from the tax software network infrastructure so where do we go from here there is the new executive order but the picture is far bigger as you can see now package management is an ongoing challenge that needs to be done better registries of record can help us improve reliability in the chain but it's like finding a needle in a haystack if you have the right security telemetry and visibility at the right control points in your organization these are a couple current resources you might want to grab a screenshot if you can there that talk about what solarwinds learned and shared publicly so that you can secure your your enterprise your operations better there were hard lessons

and some takeaways first we need to be paying more attention to the open source software that we're using we need to change what we've been doing and how we've been doing it to identify and remediate risks because we're missing a lot of stuff those legacy security tools they're not securing us there are some new things out there to help us one of them is called six store and one of them is called package hunter but we need to level up okay so i will share this with you this came out 2021 sig store it's from the linux foundation and it seeks to empower software developers to help them securely sign software artifacts it's free for use by all developers and

software providers

and then package hunter this is from get lab and you can use it to detect malicious code and unexpected behavior in your dependencies because there's a lot of malicious stuff out there in the wild it's free and it's open source so let me end with this i love podcasts and one of my go-to's is risky business they had a great segment featuring mark rogers from octa talking about operation aurora if you can gain a position of trust then you can exploit it think about service or non-personal ids where you are working accounts that nobody really scrutinizes but that have enough privilege to leverage and to do e-discovery and to find all the stuff you need to launch your attack

because your chain is only as good as its weakest link and there are more ways to abuse the chain of trust than people realize and with that i say thank you so very much for your time and being here [Applause] okay are there any questions got a quick question on the uh dependency confusion one do you have any more information on that because i think like just off the top of my head like maven you've got there's a like a vendor and then the actual specific library then like the version the very first thing that does when it goes to grab that is search your cache on your computer and i assume all the others work the same way

and everyone knows assuming is a smart thing to do um and so i'm just wondering do you have any idea like any more information on how i've never heard of that particular compromise and how that would work because it would search locally then it goes off to nexus or archiva then if they can't find it there that nexus or archive then goes out to the main public repo so i was just curious well that's the beauty of it and that's why alex berson was asking the questions if you google alex person there's a couple of real there's several really good write-ups about it and and who was impacted and where it's gone from there and they're citing dependency confusion

style attacks increasingly now when it comes to software chain and repositories a b-i-r-s-a-n okay any other questions

okay all right please go have a good drink and a nice dinner and thank you so much for your time [Applause]