
All right, welcome everybody. First I just want to say thank you for attending our talk on supply chain leakage becoming the uh the norm. Um my name's Charles Adams. I'm the founder or one of the CEO of Expel Strike and also co-founder along with uh Teddy. He's our chief information security officer and also a co-founder. Um I'm a eight-year Marine Corps veteran. Um I interned at Google when I was in college, which was kind of cool. Uh everybody knows what Google is, but yeah, and we have uh Teddy. He's did some time at Cisco Systems and a bunch of other cool tech companies. He's a PhD student or can- student at the University of Delaware. Should be
graduating really soon. Very interesting guy and yeah. >> So, about [clears throat] 10 months ago, we were asked uh we were doing a pen testing for a health insurance organization and the CISO asked us, "Hey, I have a concern. I'd like to know if there's any source code out there. Can you do some OSINT? Start with GitHub." And uh right away when we started looking for, you know, seeing if they had any code leakage, we found hundreds of repositories that were false positives, meaning that like uh people were making their own uh you know, programs or APIs, but it's not really uh affiliated with the organization outside of the domain being in there, right? And then we
stumbled upon this awkward gem. A recent senior engineer had just retired. He had been working for the organization for 20 years. And he obviously hadn't been doing dev work towards the end of his career, but decided to when he retired take all his notes that he acquired throughout his entire time and dump them in a public repository. And his notes were so immaculate. There was overwhelming for us. There was thousands of files, most of which were word documents with screenshots containing all of the secrets for the different VPNs, how to connect to them, the two-factor tokens, and his level of access, which senior had admin access for quite a quite a lot of things. Including all the every
single EC2 instance, how they were deployed, what the admin credentials were for that, and the dashboards for Azure, Ontra, and AWS. We tested about eight credentials and after a while we got a little overwhelmed. Had to disclose with the CISO right away. And he was shocked because this had been public for 5 months and he has a team and paid a lot of money for tools that supposed to prevent this type of stuff. And so we That was our first thought like how how's that possible? Day two, we shifted to Postman. And then found a different kind of leak. Seven years prior or back in 2020 um when this health organization was using a supply chain vendor to do some
integration work for all of their core functions providers members patients integrations, and things like that. The IAM access and all seven of the core function API Uh, admin access, both of which were super admin privileges, were all in there including uh, all of the different endpoints and how to use them. All production. And the first thing when we disclosed that with the with the CISO, he said, "Oh, that's from 2020. We rotate our secrets every 90 days. That's not a problem." I said, "No, these are application secrets. These are not human identities. Let me show you." So, I used curl, took took uh, just one of the APIs, and said, "All right, so here I
go. Now, I'm I'm going to authorize my curl application to uh, do what?" I used a patient patient one because we had it in the pen test. We had a a fake uh, pen test patient account, right? And so, I I revealed how I was able to expose all of my patient records for our information. But then, I explained that because this application manages all of the patients, we could change the ID, and then view any patient record. And this has been public for many years. It's just waiting for someone to take PII from this health insurance company, right? So, right away, we were like, "Okay, how is this possible? They're paying for tools." They and they're not
paying a small amount of money. This They're paying a lot of money, and they actually have two other tools that they were shopping around, trying to decide if they wanted to integrate more act attack surface management capabilities. And so, what when we started digging in and looking at some of these tools and and what they were doing, we realized that they were so focused on on reducing false positives, and most of the platforms and tools out there, which we support, are focused on preventing leakage from happening, which is great. Fantastic. But, to reduce false positives, both of them actually will ignore binaries, will ignore archive files, will ignore um, uh, word documents and the few that do actually process word
documents do not deal with images screenshots which older developers love to use for some reason. >> [clears throat] >> So what we found is the the definition of often times when repo contains a leak. Spurned up probably about 12 years ago when some of these big platforms that we know of today we're getting off the ground and they classified them as accidental misconfiguration. You know, the developer on pram submitted committed some code 2:00 in the morning and unintentionally put the environment file out there and so some some access credentials were were leaked right? Well, that makes sense at a time when many organizations hadn't quite yet migrated over to cloud environments and we're still on the fence of like if they
want to go completely all off pram and on pram and many organizations including big ones had more developers on staff. Today that's not the case. Today many organizations choose to use service providers, have migrated now to to cloud environments and do not have on staff their own development. They actually rely on third parties and so ironically what we're finding because we're choosing to look at the open internet space outside the perimeter that with developers who have personal accounts and we are choosing to unpack binaries, documents, machine learning on to do OCR for images. Most of the leaks that we find Uh, I'll give you an example. There was a developer who every couple years obviously worked
for a different company, and that seems to be a common trend with today's developers, and had a directory for every company that he worked for. And in the beginning, he would run tools he he would put all of the projects he was working on and everything he had access to. So, he would run BloodHound, AzureHound, and other pen testing tools and hacker tools to enumerate all of the systems that he had access to, database dumps, and then Excel files, uh, which had different organizations all of the usernames and passwords if he was able to access them. And then you could see, uh, for the next organization, as time went on, his hunting capabilities improved. He got better. He improved his
portfolio. That is the type of leaks that we're finding right now. It's not just a developer accidentally dump uh, you know, committed something affecting their own organization. No, it's a developer that that uh leaked something that's affecting multiple organizations. No longer is it I don't think we should be classifying as an accidental misconfiguration. So, to help us, uh, you know, sort through this stuff, especially for organizations we do work for, uh, we actually had a build our own tool leaning off the tools that exist, but adding the capabilities that, uh, they choose to ignore or don't process. We do support all the tools. I just put that out there. Uh so >> [clears throat] >> as a security organization, most of our,
uh, clients are with the banking sector, uh, credit unions, health insurance, and, uh, corporate legal firms. And so, um, our data does actively reflect that, right? But, if you cover up the those sectors, and we can only fit so many of the industries right? The remaining industries were all things that we found by looking at our uh the industries we work with, and there's a pretty even distribution of all of the industries affected. We've disclosed in the past uh few months almost 200 disclosures involving 300 repositories affecting 1,500 confirmed organizations. I say confirmed because that number is actually higher. Uh it's just a lot There's a lot of uh organizations that uh are in um
acquirement or uh merging and acquisitions, and we just haven't classified them or confirmed, you know, the independent set. Um >> [clears throat] >> but, and 15% of them represent uh Fortune 100 companies, and a few uh global conglomerates, a term I didn't even know until we encountered one. >> [laughter] >> Um and one interesting thing I want to point out, I wasn't going to include this. So, first party, third party, unknown is our biggest category. Unknown because even some of these service providers, these supply chain vendors that are doing the integration works on uh as a third party operating to an organization, net we're seeing an increase in them choosing to contract uh developers on demand. So, not all
service providers now actually have their own developers on staff to do integrations for clients. So, it's getting trickier for us to classify, is this a first party, third party, fourth party, or somewhere down the line? Right? And because of that, it's also very difficult for an organization asking another organization who's contracting another team to have visibility on all of those developers just on what we know the accounts that they have and we don't know majority of their personal accounts that they have. >> All right, on to a couple case studies. So, we all know leaks happen whether they're a first-party leak come from an employer at the organization, contractor at the organization um Early on So, as Teddy said, you know, we
made 200 disclosures in the past few months. It's It's been a really long security research journey for us, but we finally started actually reaching out to companies and telling them So, it's really hard to get a hold of of a a company and to take to gain a listening ear. Right? If If you're a big like Amazon or Google or I don't know, whatever. Uh you can use bug bug crowd or or HackerOne for vulnerability reporting. But if you're smaller like a credit union, you probably don't have an intake for for you know, leaks or vulnerabilities that people find. So, generally our escalation path to the organization is submit a bug bounty vulnerability disclosure or that's
preferred. They have a process for it. But if they don't have that we might have to buy some phone numbers and start calling. So, I you know, I buy phone numbers for this credit union. Uh we found we found So, should go back to what we found. On a on a GitHub repository, we found domain administrator credentials for what looked like was a credit union and it was so bad that I thought it might be a joke or it might be a honeypot or a personal project. But I I should probably tell the credit union anyway. It's been out there for 2 years. They maybe they don't know about it. Maybe they do and they already rotated the
credentials and just haven't been able to take it down. So, I'm calling calling the the some some security personnel at the credit union. No one's answering. Um I So, I call their help desk. Somebody answers the first time and I tell them, "Hey, I found your donate domain admin password out on GitHub. We should probably talk a little bit more." And they say, "Okay, yeah, I'll relay it up the chain of command. But, somebody will call you back." I called them again the next week. Didn't hear anything. I called them again the next week and say, "Okay, I'm going to read your domain administrator password to you. Are you okay with that?" They said, "Yeah, if you have it, read it to me."
Okay. Capital A at exclamation point one. They say, "Okay, stop, stop, stop. Okay, that's the password. All right, okay, okay. Um let me Someone will call you shortly." So, you know, make the disclosure with the or with the with the credit union and then and they take it down and, you know, it's a good day for everybody. Though it was ex- it was it was a you needed to be internal their environment to to use the credential. I mean, what could you do with a 200-person credit union and with their domain domain admin password? You send a couple good fishing emails, you're in. I mean, you already got the keys to the kingdom. Um so, that's a first-party leak, right?
So, everybody knows, you know, companies leak their their data. Sometimes it happens. Big deal, right? Um well, the third parties that have access to your environment will also leak your data. Um because, you know, humans exist in your company, humans exist in their company. So, >> [snorts] >> we deal heavily with credit unions as as as Teddy was saying. It's it's sort of They're They're um high risk tolerant or sorry, uh very low risk tolerance and um it was an industry at the time we were trying to break into. So, we're trying to get friendly with credit unions. And as as part of our security research journey, uh we we we we come across a
lot of source code. Some source code is more, you know, meaningful than others, right? Um it's really bad when you hardcode all of your credentials into your, you know, dot properties or dot env files. And then, you it's all committed to GitHub. So, we're monitoring for a credit union, who was one of our clients. And on the first day, we come across a very large repository. It's a bunch of source code. We triaged it, and in our initial triage, we go up a directory, and we see that there's a different different directory for each one of this company's clients. Each one of credit union. There were 33 credit unions. 23 had production, they were labeled
prod. Really bad day. In each directory was a username and password and a couple client secrets different applications that the vendor owned, but also that each credit union owned. So, there were credit There were usernames and passwords to each core banking platform in each credit union that was externally accessible. Very bad day. We were able to contact the vendor and have it taken down, and you know, each one of those credit unions was contacted, credentials rotated. But, it it happens, and it happens a lot. Um it's sort of unsettling. We, you know, go back to this slide. Um Almost half of these are from uh or I say at least 20 at least that we found,
we know are from third parties. Um and up, you know, about half of which we think are, but, you know, we're not exactly sure cuz attribution is kind of hard when, you know, an alias is Pat back 1234. So. >> Yeah, and it's also kind of challenging, too, when uh the third party decides to contract out the dev work on the dev team or the integration team. Um then it's not truly a third party, it's a fourth party or a fifth party, right? Um When we do these disclosures, uh often times, because of it's not just a secret, it's not just a credit card number, it's not just a merchant. The leak that he was
just speaking to, uh I think there were seven or eight merchant account numbers, you know, banking numbers, the things that handle the payment processing for the bank, for customers, for businesses, right? Uh you know, when we do these disclosures, it's not just a CISO and the SOC engineers that often are in the meeting. They'll bring in usually legal, someone from legal, someone from their asset management, someone from risk management, someone from um uh threat threat intelligence. And what the most common thing we get, uh outside of how did you find us when I pay a lot of money to prevent this from happening, is um where do I put this? Do I put this in
legal right now because I know this is about to go to litigation. I They already know that's going to happen and they know that asset management's going to work with that team, but risk management also needs to do an assessment on it. And then all of the customers that are affected, uh any of those secrets, especially if it's back end and core function stuff, the SOC team needs to deal with and uh just make a decision on whether or not, oh no, all of the MFA two-factor tokens are out there, too. Should we just reset all of them, right? So, what we're finding is that like because for so long in industry, we've got so
accustomed to these tools and dashboards to prevent leakage, which is great, we've allowed the definition of accidental, uh you know, misconfiguration to a leak as a category that organizations, we we have yet to find one that has a a good plan on how to deal with this type of leak. Um So, that's kind of why we're here. Uh >> [clears throat] >> That explains exposure treatment Yeah, you know, the best defense, in my opinion, and I could be opinionated here, is to have a plan. I understand prevention is great security. We support that. We're We're not against it. We're not trying to tell you to change your tools. But, what we are saying is have a plan
for the worst-case scenario you didn't think about could happen. Right? Uh often times when we bring in developers to do integrations of back-end core functions, we often don't even monitor those developers as for their activity because we want them to integrate our stuff in a smooth and timely fashion. Right? What we're not expecting though is and I don't think many organizations realize this, even though it's in the contract, for some reason, developers like to take, especially if they're overseas, they like to take all their code with them just when it's about to be in production. And so, if we're watching a Postman repository, we'll actually see, oh, okay, looks like they're working on eight different integrations.
These two are are in UAT still. This one's in sandbox, and they just started that one. This one's in staging. Uh-oh. This one's in production. You can almost see and just wait a few more days and then, oh, that one's production. Oh, that one's production. And when the developers like to put the production stuff, it's always the cleanest. You know, it's always organized, right? And live coding, a lot of these supply chain vendors that we're finding too are We've seen an increase in live coding stuff. Live coding's great. I'm not against AI. AI organizes everything really nice and neat, including all of the access credentials and everything that they're working on, right? So, we've actually seen an We've
gotten influx of of more of these supply chain leaks with the assistance of live coded uh repositories that developers are choosing to take out of the organization and put away. One of the things that I know and no organization will ever share their security vault or secrets vault with another organization and we don't recommend that. Right? Imagine yourself you're a developer and you're trying to do seven integrations for seven different organizations. You have different access ways to each different environment and you're the you're the manager to manage all of these. Those are the ones that are the most scary to us because they have the highest level of access and they will put all of the access
stuff usually in one word doc. And it's almost like if you find that doc it's game over. But it's like dude, you're you're like the senior management. Why are you doing this? I don't know. We can't I don't know exactly I have ideas, I have theories, but we don't really you know, we can't tell can't tell you how people think and why people do what they do. And why developers feel that it's okay to take code and take it with them. Probably because they have to they're pressured to do so many integrations and what work here might work there. It might work there. Right? So if all you future developers out there, please Oh, okay. And that's
that. All right. >> Two minutes [clears throat] left. Thank you for our talk on the supply chain leakage coming the norm. If you have any questions, please feel free to find us afterward. We've got tons of war stories and funny disclosures. So Thanks. >> [applause]