BSidesSF 2026 - The Heist: Chasing an Advanced Crypto Attacker Across the Multi-cloud (Yotam Meitar)

All right. Hi everybody. Um, so great introduction. I'll give you a little bit more background and then we'll jump in. Uh, my name is Yo Tom. I'm director of incident response at Whiz. Uh, I've been doing incident response for about a decade now and before that I did offensive cyber operations in the military. Um, what I'm going to share with you today is the story of a real incident and real incident response effort um that I managed. Uh I do just want to give one quick disclaimer before jumping in and that is that to avoid identifying the specific victim of uh this attack. I have modified some of the details I have combined with a couple of

other attacks by the same group. So if you're trying to guess who I'm talking about, you can't. It's it's been modified. But everything that I'm showing you is real uh has been seen in the wild. And that includes uh some of the more uh strange elements of this attack uh that you're going to see. So with that we can jump straight in uh and I'll give you a little bit of background about the victim organization in this case. So uh like many similar uh attacks in this case uh the victim was a cryptocurrency exchange rather large one operating a lot of different currencies on their platform and enabling users to trade or sell uh cryptocurrencies. And

at a high level, we're not going to get too much into their architecture, but at a high level, there's a few basic things that I want to walk you through to understand how they work so that we can understand how the attack unfolded. So basically the way things work for this organization when we have uh a customer that wants to make some sort of withdrawal of cryptocurrency whether it's to another cryptocurrency exchange or a different wallet that they have whatever it is they put in a request through the app or the website and then that request goes first to authentication of the customer themselves. So this organization used entra to do this mostly internally but

also for customers. Then this request goes to a dedicated KYC now your customer server within the organization. So this is how they managed things like you know making sure this is the actual person who's making the request. Cryptocurrency exchanges some of you may know this they have these annoying requirements. Give us a picture of you with your uh you know holding today's paper like a ransom thing or uh with writing down what it is you want it to happen. So that's the process that gets approved by this KYC server. Only after that does the transaction gets forwarded to what they call the hot wallet transaction server. So essentially within cryptocurrency exchanges often a hot wallet just means um the

cryptocurrency exchange wallets that are online that can create transactions in an automated way rather than a cold wallet which has its uh signing keys uh in some usually physical safe or something that's not online and that's where bigger sums of money are stored. But these are pretty big too, the hot wallets for these uh bigger transactions and these bigger uh organizations. And then after this server, this specific hot wallet transaction server signs the transaction essentially creates a a transaction that will send funds from the cryptocurrency exchanges wallets to whatever the outbound wallet was that the user requested. After that, there's another level of security here, which is the transaction then gets sent to a third party, in this case, BitGo, um,

which is designed to verify cryptocurrency transactions before they go out. So, we're not going to get too much into the, uh, crypto of this. If it's multi-IG, they do this by simply having separate keys. If it's smart contracts, they just require both signatures. But essentially, what they do here is they use this other service to implement certain rules. Like, if something's going to a wallet that is known to be bad, they can block it. if it's over a certain amount or something else that has been implemented as a rule that's not allowed, it can be blocked on the big go side even if uh it was approved within the victim's network. So, it's another layer of security. And

one important piece of the rules that this specific organization had is any transaction over $1 million uh requires manual authentication. So, it's not just a rule if it's over 1 million only approve it under these circumstances. It actually says any transaction worth over $1 million at the time because you know crypto fluctuates a lot. If we get a transaction like that we need manual approval by a very small group of admins within the organization. They have to log into their account and approve say this transaction is valid from our perspective. So this is the highle structure which eventually leads money to flow outside to customer wallet. So if a customer wants to withdraw, it's a

pretty arduous process from a technical perspective to ensure that these funds don't get stolen or if there's a hack somewhere in the system that attackers can't just make off with these funds. Now this all sounds great on paper. Um this incident response effort from our perspective started when this cryptocurrency organization woke up one morning to discover that an $18 million transaction uh occurred just not to a wallet they know. Doesn't look approved in any way. Um, and obviously this is a big number. They noticed it the next day. They said, "Wait, what's going on here?" And this is what triggered the investigation. So somehow this entire system, which is meant to prevent fraud, and definitely any kind of fraud over um

a million dollars, was bypassed. And this is where the initial triage effort began. So let's and throughout this uh investigative process, I'm going to share with you the current status of what we know because this gets more and more complicated. Let's start with the basic information that we had in initial triage here. First, we know that there is some uh uh compromise of the organization. We know that there's at least $80 million of lost funds and we know that the attack has gone on for at least this one day because the transaction happened the day before. We're going to keep updating these uh stats as we discover more in our investigation. But obviously the first

place that we wanted to go and look is the logging for this third party service that is supposed to block any transaction over a million dollar because this was an $80 million transaction. So when we go and look at those logs um we see uh the the transaction itself. It's not like it somehow didn't go through Bitco and it was approved through the system. We can see that it was approved by a Bitco admin. So one of the users that is actually supposed to be doing the approving for this company. And when we look at the logs themselves, so some of this is obviously redacted, but there's a couple of interesting points here. First of all, we can identify who that

individual specific user was. And we see in this case, it was a real admin that works for the victim organization. Here we're just going to call him victim admin, but it's a real user and that user performs all of their authentication through Entra. So somehow we have to assume that user with their multiffactor authentication, everything they require in order to validate this kind of transaction was compromised. That's one important point of this. The other important point here is that we now know first of all that we have this one entry user. But we can also look and see the source IP for what was going on here. The transaction itself. And what's interesting is that we see that the

malicious transaction, the one that was signed on through this compromised user in Bitcoin was also created from where it's supposed to be created. It was created from this server, the hot wallet transaction server where legitimate transactions are supposed to come from. So essentially already at this stage just based on this initial triage we know that we have at least two points of compromise. One is this admin this approver that approved the transaction even though it wasn't supposed to happen. And the second is something within our infrastructure itself which led this transaction to come from our infrastructure. Somebody actually initiated it from the infrastructure and not just at random uh by getting this access to the approver. So I should say

what often happens in these cases as well is that once a human approver like this gets bypassed um immediately there's fingerpointing within the organization. Senior executives start saying oh maybe this guy's somehow an insider. Maybe he's involved and a big priority of the investigation becomes as quickly as possible figuring that out so that the organization can move forward. In most cases that's not where we land. I have some uh crazy stories I can share separately about cases where we do find insiders but those are relatively rare. However, that is a concern to be addressed here. So our first priority is to look at the approver uh himself, this user. And obviously, as I said, they do

all their authentication through Entra. We go to Entra Logs, right? We want to look at these Azure entrance and figure out what was going on here. And when we look at the logs, we pretty quickly see that it's not hard to figure out what happened because shortly before the malicious transaction was approved, we can see that this user's password was reset. And it looks like it was reset by the help desk, the company's own help desk. This is not just a password. When we look at the logs, we see their multiffactor authentication was reset as well. Now, I'm sure many of you have seen this in the past before, but unfortunately, this is becoming more and

more common that attackers are just leveraging help desks as initial vectors of compromise instead of uh trying something more complicated. And in this case, this was only a couple days after it happened. It wasn't hard to go back and try and find the recording of that conversation. As I said before, we've anonymized things here, so this is a recreation, but all the text is real from what happened in the field. >> Thank you for coming home, Justin. My name is James. How can I help you today? >> Yeah, hi. I lost my phone, so I need help connecting to my account. >> All right, not a problem. I'll be glad to help you with that. May I have your

full name and employee ID, please? >> Sure. >> I don't have my employee ID, but my email at.com. >> Thank you for that. I'll have to verify your identity. Can you provide me with the last four digits of your social security number? >> Yes. >> Awesome. Thank you. So, I have you all verified. So, what I can do is remove the current MFA device from your account so you can log in and set up a new one. Would that be helpful for you? >> Yeah, that would be great. Oh, wait. Can also uh can you also reset my password? It was saved on my phone and I don't have it. I lost the phone. >> Certainly not a problem. Give me one

moment. Okay. I have removed the MFA from your account and the device. So, everything should be good. You can try to log in now with the password summer 2020 with a capital S. >> Okay, thanks. Let me see if it works. Going to try to log in right now. >> Great. Thanks. I'm in. Have a nice day. >> Awesome. You as well. >> Excellent. Wonderful customer service. Um I I heard a lot of laughs here. This is real. Unfortunately, uh no no text was modified here and and we we're seeing these more and more. It's just, you know, we are in this room talking about an attack. We're on the lookout for this. But if you're imagining the

help desk this guy that gets I don't know how many calls a day, it's harder to protect against these things. It's not impossible. We'll talk about that later. But it's harder for them to detect that this is something malicious going on. The fact that the user didn't have their employee ID here, not uncommon, but the fact that they also did have their social security number, very common, right? So, I'm sure we've all gotten a hundred emails by now in our lives saying, "Your social security number was part of this data leak or that data leak." It's just not that hard to get. Uh, and attackers who know what they're doing, especially if it's this user, this admin, likely did the

research beforehand. They came prepared for this call knowing they might have to provide something like that. They probably didn't know about the employee ID or if they did, they just had no way of getting it. But it was enough for them to get access to this user's password and multiffactor authentication, reset them, and we now understand a little bit more about how the attack unfolded. So, from the very end of the attack, we're now unraveling it. Right? So this very large $80 million transaction was caused by this initial fishing of the help desk. We have a little bit more of an understanding, but now there's a lot more left to unravel because as we mentioned before, this was just a little

part of it, right? This just approved the transaction, but the source of that transaction actually came from within our own infrastructure from the actual hot wallet server, the one that's supposed to be signing transactions. So there's a lot more investigating to do. And the first thing we did before even uh diving into our own infrastructure here is look at the actual malicious transactions. And this actually revealed something very interesting. So when we looked at the blockchain to look at this malicious transaction and the way that it evolved, what we saw is that the funds were transferred from the company's the exchanges hot wallet to some random wallet that we've never seen before. Not shocking, right? That's how

these usually work. Attackers aren't stupid. They use new wallets every time. Then immediately all the funds from that wallet were sent to another wallet that we've never seen before. So another hop of just random as far as we're concerned. But then that second wallet was cleared again. All the funds from it were sent to a specific automated exchange. So these automated exchanges or mixer uh mixers that are operating on the blockchain essentially are often used by attackers but also by legitimate people to move money around in a quick way. And here for laundering likely, right? don't want this stolen funds to be followed. So you send it to a mixer. You basically, let's say you send it X

Bitcoin, it shoots out Y Ethereum to a different wallet that you told it to. And you can go and do this again and again and again. Makes it a lot harder to track. And what was interesting about this pattern because it happened so quickly and in an automated way. We said, this is this is kind of cool. Maybe these attackers didn't only do this once. Maybe this thing that they were doing here following this double hop tomixer uh process maybe they did this more than once. So what we did is we went back looking through the blockchain and we looked for any other transactions that came out of the company's hot wallet, their legitimate hot wallet, which followed a similar

pattern. Nothing as big as the $80 million thing that would have been detected. But as we were doing this analysis, very quickly we started finding more and more transactions. Over a period of months, not a period of just this one day, much smaller amounts. It was 40 something transactions for much smaller amount. But when we added them all up together, and obviously we verified with a company, none of these were legitimate. They just didn't go noticed by anybody. Our estimate for how much was stolen now had to go up from 80 million to 107 million instead of over one day over at least a 7-month period. So the understanding of the scope of the attack has expanded dramatically, right?

Obviously the 80 million was the big hit that came at the end and that's what caused uh the organization to figure out something's going on. But this is a bigger story, right? The scope of compromise, the length of compromise and the amount of money stolen, this extra $27 million is not nothing. It's not as big as the 80, but it's significant. So based on this understanding, we now know a little bit more. It's not just that we had this big transaction that came from social engineering, but before that, we had a bunch of much smaller transactions over the scope of 7 months. And so on the bottom of the screen here, we're going to keep having the evolution of

the attack as we're unraveling it before our eyes. So taking another step back, of course, now the focus had to become the AWS environment itself where these malicious transactions originated. So we wanted to look at the AWS environment at this hot wallet server and figure out what was going on. So we looked at the server itself to see if there's any indication of malware or persistence like reversal, anything on this server. We found nothing. We looked at VPC flow logs to see if there's anything indicating malicious communications to it. Something out of the ordinary also couldn't find anything. But what we often see attackers do, especially in the cloud, is utilize cloudnative tools to run these kinds of things even on

dedicated machines. And here once we started looking through cloud trail logs to see what was going on that had to do with this machine, what we found was this. So we can see that the attackers leveraged uh SSM the AWS tool to run commands directly on this machine. So they didn't do anything involving malware here. They just used the native AWS management tool to target this specific EC2 instance which had the signing keys accessible to it and was able to sign. And when we look at the actual execution uh that was done through SSM, we can see that they specifically triggered the script that is there to initiate transactions. valid script, a script that was created by the

organization and managed by the organization for valid transactions, but it was triggered by the attackers presumably with arguments that involved the attackers target wallets and the attackers target amounts rather than legitimate ones created by consumers. But that gives us a little bit more of an understanding. We can now say, okay, all of these minor transactions and the big one as well came through SSM being used to target a specific EC2 instance within the environment. Great. That's a little bit more information. And that also expands our scope of what we know is compromised. Now it's no longer just this one entry user. We now also have at least one uh AWS user which is also compromised as well and that is this

secops user which you can see on the screen. Now this secops user uh which was used to trigger these SSM commands. We asked the organization about it immediately and their response was well we have no idea what this is. This is not a user that we use or have created ourselves. So that immediately makes us think okay well the attackers probably created this user and you know used it how they did that is a good question but that means our focus now turns from the transactions and how they were created which is not really a mystery anymore to the user right on the AWS side we have an AM user which was very very privileged was able to use SSM to run

commands directly on the most sensitive server within this environment the transaction signing server. How was this done? Well, we look back. We try to figure out based on our logging what was going on with this user. And what we see as we look at logging, uh, and I apologize that a lot of this is is redacted, but the interesting parts are not. What we see here is a specific set of events where this user, this secops user was created in the environment and then assigned very high privileges. And these events and again some some time stamps here are blacked out to avoid identifying victims, but these events occurred seven months earlier. So this aligns with our timeline of when some of

these smaller malicious transactions began. This aligns, right? It looks like attackers got this and then they started doing the malicious transactions through this. Now what's obviously another key consideration here is well how did they create this user? What did they use to do that? And when we look at that, a lot of this is blacked out. What we see is that there's a specific user, a different IM user that was used by the organization mainly as a test user for test environments that did have unfortunately the privileges to create these other users and assign them privileges that was used to run all of these commands. So that is to us a key finding because we can now say okay as

far as expanding our scope of compromise what we understand to have happened it's not just one entry user and one AWS user it's at least two AWS users right it's this other one as well so we're updating that part on the screen and very importantly what we also uh can point to at this phase is where these events came from right we want to figure out well this other user this test user which is a valid user not created by attackers this was also compromised how where do these events come from Well, we look at the source IP addresses. And you might notice this is one of the few things on screen that I did not redact at all. Um,

and the reason for that is that these IP addresses are simply Microsoft IP addresses. Uh, they're sometimes used by Azure services. They're sometimes used by other Microsoft services. And importantly for this case, because we talked to the customer about, wait, what is this test user? Where is it usually used from? These are often used by GitHub. So things that happen within GitHub, this is often the IP address range uh which such events come from. Now that's very important to us because when we were talking to the customer, they told us yeah this test user is actually used a lot when we run CI/CD testing and actually it is used in GitHub uh action secrets that the

credentials for this user its access key should be stored within those GitHub action secrets. And because we're seeing what's interesting look it's not just the malicious events the creation that's coming from GitHub ips it's everything right? So legitimate stuff consistently happening from this user from GitHub IPs and the bad stuff happened through there. To us, that's a very strong indication that well that's the way they're operating. So also that's why this probably didn't trigger any alerts at least on the source IP side because you know this is the way it is. This is what this user is for. But to us that says well the attacker likely came from there as well. So again we got a shift

right? We had our our Bitco environment. We had this Entra investigation that we had then we moved to AWS. Now we got to shift again and we're moving to GitHub, right? We want to look at what happened in GitHub and because we already have this theory in mind because we already think well we know this thing was stored in GitHub action secrets. Maybe the attacker got it from there. Now I don't know if everybody here has worked with uh these GitHub secrets before. They operate in different ways. the the idea behind them is actually quite a good one, which is instead of hard- coding credentials into your code that you're running tests with on GitHub, you put

them into this service and then they're not immediately readable back, right? So, it's not like you can click on it and say the content of the credential. And actually, even if you try to run a GitHub action which just tries to print out this credential, like access it and then print it out, it it doesn't let you. It it replaces the characters with other things. Unfortunately, this isn't perfect. For example, if you double B 64 these credentials, you can print that out uh and then just uh get it back out the other end. So there's lots of tricks to get around this and we were seeing attackers target GitHub more and more. In this case, once we started looking at

GitHub actions which accessed this specific secret and what they were doing, what we found was this. So again, some of it's redacted. We're not going to go through the whole thing, but what we have here is a GitHub action, a specific action which can be executed within GitHub. And what it does here is exactly what we saw on the AWS logging side. What it does is create a brand new IM user in AWS later assigning it privileges as well. So this ties very neatly into what we understand to have happened. And this also expands our understanding of the scope of compromise yet again. Right? So we went from AWS IM to the rest of the attack. Right? IM was

used to target SSM and then the EC2 instance and then creating these malicious transactions and then social engineering to improve the big one. Now taking another step back, we understand this also involved GitHub and specifically the targeting of uh GitHub secrets through this GitHub action and we know that at least one GitHub repo must have been compromised as well. Right? So our our little list uh on the right hand of the screen here keeps growing. Uh unfortunately we have to find everything so that we can remediate it properly. Right? So we can't just stop at a certain point. There is another question here which is you know even if I create a GitHub action uh it's

got to be executed right some something I got to put it someplace where it'll be actually executed the problem is this isn't very hard to do because once there is a PR a pull request on something that contains this action it's going to get executed and so we just went looking for that and we found immediately a pull request in this environment which just executed this action now the fact that you can just create a pull request that executes the action and that's That's it. You you've done what you needed to do to escalate privileges here, use these credentials within GitHub and in this case also do crosscloud lateral movement over to the AWS environment.

That is major because that enables attackers to really be very stealthy about the way they do this, right? They can create their own action in a little branch that they created for themselves and then do a PR and then once they're done even delete it, you know, delete the action, delete the PR. In this case, we had it. They didn't do that, but they can do that. We're seeing that in other attacks. So this really complicates the potential negative consequences of these attacks. So we have another step of understanding here. We have this uh specific GitHub repo which has been compromised. This PR that took place and to figure out how they actually executed this PR, we just look at the GitHub logs

themselves and we see where this PR came from and we see that it was executed with a specific personal access token on GitHub. So personal access tokens are often used by developers to access different resources on on GitHub. Some organizations uh in many cases give individual personal access tokens access to the corporate resources. So your own pat from home might get access to something uh that's corporate. A better practice obviously is to separate that. Um but in either case we can see that a specific uh personal access token caused this pull request which we know is malicious. So the GitHub side is starting to gain more and more clarity. But these logging um reveal a few things

here to us that are actually very important. So first of all, we know that a GitHub pat was the way they got into GitHub, which gives us another little item on our list on the right. Right? We now have a GitHub user that was compromised as well. But more importantly, we can see what's going on here as far as where this is coming from. And what we can see Oh, I'm sorry. There we go. Uh and what we can see is that the the specific um PAT the personal access token that was used here belonged to a specific developer working on this uh on this environment for the victim. Again, a legitimate person. We were talking before about fingerpointing

and the way these attacks develop. As we were finding out more and more of this, you can imagine for leadership of the company, this is distressing. It's also distressing that they've lost $107 million. Um, but the more they figure out that this has been going on for a long time under their noses, this is meaningful. And what often happens, and this really depends on the industry, but what often happens is it's easier, especially for senior executives who haven't faced a situation like this before, to say, well, okay, show me the guy who did this to me, right? It's it's a lot easier than show me the source IP address. Um, and and in this case, uh,

well, yeah, and in this case, um, and in this case, the the shift of suspicion just happened again. So if initially they said, "Well, this guy who approved the transaction on Bitco, well, okay, we've got a bigger issue here than just some guy on the side, but maybe this guy now. Maybe it's this guy who's involved, right?" Um, and again, that changes the way the investigation progresses because it's really important to answer these questions quickly. If there is actually some sort of insider, you want to know so that you can take away their access, prevent further damage, prevent them from leaking important information to the press as this is going on, right? There's lots of

risks that are associated with insiders. But if it's not an insider, you need their help, right? Getting information from the users whose credentials were actually compromised is a really important part of investigating. It really speeds things up. So, we want to be able to move quickly. So, it's a fine line often collaborating with lawyers on how we communicate with these people. In this case, the user was asked like, "Okay, well, he wasn't told exactly what was identified here in these loggings in in GitHub, but he was told, listen, we're doing this review. Where do you store uh this personal access token? how how how is it managed? And he his answer was actually pretty good. It wasn't,

well, it's just, you know, on my desktop on my private laptop at home. That wasn't the case. Many times it is wasn't the case here. Instead, he told us, "No, it's in our corporate password manager. It's where it's supposed to be. Um, it's it's stored there." Which again shifted our our investigation focus. Think about how many different uh cloud services we've been through here already, right? We had Bitco Entra. We had AWS in multiple different uh forms. Then we had these GitHub things. Now we're switching over to the password manager, another SAS cloud service which has its own logging. But when we look at that logging which likely uh luckily we did have in place here uh we see a couple of

things. First of all this specific uh pat the one that was compromised wasn't indeed accessed. So we find the logs again a lot of it's redacted as you can obviously see but we can see this is a log for access to that specific personal access token and you can see it on the screen. We can see that the the user that was used to access this is the same victim developer that we had before. So his user must have somehow been used to access this resource uh in a malicious way. When we look a little bit further in these logs in the password manager logs, what we see is well oh and of course now we have to add to our list

that the password manager was compromised as well. Um we see that this session which enabled malicious access to the personal access token was created um from a specific IP address. And what's interesting is it was created through an entra login again. So a legitimate login. It's not like somebody somehow got around our entra being used as an SSO. Uh you know that's just not what's going on here. Somebody logged in again successfully as the victim developer to this uh password manager and used that access to extract the PAT. Used that PAT to go to GitHub. Use GitHub to get credentials that were stored in secrets. Use that to go over to AWS to create malicious transactions

to call the help desk to authorize a much bigger malicious transactions and make it off with a lot of money. So we're unraveling the attack. It's a bit exhausting, but it's it's it's something we got to do all the way through, right? So here again we're saying okay somehow this guy's entra got compromised again but now unfortunately we had a bit more of a complicated question as to answering how this happened. So again just looking at our list here we have now two entry users that we know were compromised. If you recall we know how the first one was compromised right it was a call to the help desk. So immediately our minds go there like

maybe they just did this again seven months earlier with a different user. The problem is uh the organization didn't have entring seven months back and this was not identified or detected at the time. So there was no evidence like there was no reason for the organization to say okay we have to store these logs or we have to store these help desk calls which they didn't store for that long going 7 months back. So we didn't have evidence of help desk calls about this specific user or entra logging showing us exactly where uh these password resets came from. So we were like okay how are we going to figure this out? How are we going to

figure out 7 months ago what caused this specific user which triggered the entire mess? What caused this specific user to be compromised? And when we tried to figure out what the organization had, we did find one thing uh kind of an old-fashioned tool but which was useful for us. So they did have uh an AD auditing tool uh and Entra was uh synced to their uh on-prem uh active directory environment and they did have this AD audit tool which they did have for its retention was much longer uh and therefore we could look at it and figure out when this user's passwords were changed. Unfortunately a big uh downside of this is that this does not contain

source IP addresses for any of these changes. It just tells us password change, password change, password change. And when we look look at the screenshot, I know a lot of a lot of it's redacted, but what's annoying is we're seeing a whole bunch of password changes. And many of them happen in very close sequence. Like within a couple days, we had like five password changes for this user 7 months ago. So it it kind of aligns with what we're thinking. This must have had something to do with the attack. But it's it's really hard to figure out what is going on here. The user got asked. He was like, "Yeah, I remember at some point I may have

changed my password more than once within a couple days, but he said definitely not five times." Like, there's no way I did it five times, which kind of makes sense. Like, why why would he do it five times? Plus, it aligns with something probably malicious here. But this is really hard to like what do you do with this? Like these five password changes hard to figure out because the user then kept working, right? So, how did this happen and the user could keep working and many password changes and the attacker? It's all pretty much a big mess. So we have our user that was targeted here and the way we decided to go about figuring out

um what password tied to what and how this all thing ties together is actually kind of cool. Um so as I mentioned before uh this entra environment was syncing to an on-prem active directory environment. So old-fashioned um active directory environments you know we have the nds.dit DIT file uh locally on the DC within this company's on-prem environment which like thankfully hasn't fully rotated since the seven months ago uh when this happened and we said well that's going to contain a whole bunch of password hashes for all of these users previous passwords at least as far as back as the file goes. So we went and we exported that information for this specific user. Now the problem is of

course these are just hashes and these hashes also aren't timed within the entdate file. So it doesn't just solve our problem. This doesn't say, "Okay, we now understand what happened with each password change." But luckily, we could cross reference this with a couple of other pieces of information. First, we talked to the user. Again, this is really a valuable thing to do. And the user told us, you know, he didn't want this broadly released, but he told us, listen, I basically almost always use the same password pattern. He had like a phrase and then he changes some characters at the end of it and the beginning of it. So, we're like, okay, we can come up with a list of all the

possible hashes for passwords that this user could have done because he didn't remember what his passwords were at the time, but he told us, listen, it's got to be this pattern plus something. So, we came up, it was a list, not too bad, like 10,000 options or something, a little bit less, not not the end of the world for us to create a list of all potential NLM hashes for this user's passwords. So, we can try and identify his passwords. The second thing, of course, we could do is try and identify help desk passwords because, as you heard on that call, they aren't that brilliant. Um, these help desk passwords, they're meant to be simple,

right? Because the way the help desk does it, and the way they should do it is they force you to change your password immediately upon login. So, it's not like this summer 2020 we heard about before stays there forever. They want it to be something simple so the user can log in. So, we again generated a whole bunch of options for what it could be, summer, winter, different years, different all that kind of good stuff for what we could get from the help desk. And then we compared our lists of hashes to this to this list of old hashes for the user. And very quickly we started finding matches. So going and this is of course uh in

chronological order. So the oldest password change that matches one of what we're interested in here is this one. It's when when it says pattern it means it's the user's pattern. Right? So this is one of the user's own passwords. Uh we've identified it. Second right after that we can see that we have a help desk password. A help desk password that was set. 10 right afterward. And we can match these to the five password changes that we knew we had in the AD audit logs, right? That we can do. So we can say, okay, maybe these match, maybe they don't. We try to figure it out as we go along. But because we have all of the

password changes in AD audit and we have all of these uh all of these uh hashes, we can say, okay, this hash matches this password change. This hash matches this password change. Then as we go on, we can see a password that we do not recognize. Not on our list, unfortunately. But then the one that comes after that is again a help desk password and very quickly after that again a user password. So user help desk unknown help desk user. Interesting. Now this pattern tied to the timings that we had from 80 audit logs finally let us understand what probably happened here. So figuring out what happened on these dates, what we figure out is that this

when this thing began, it was the user, the real user that actually went in and changed his own password. He came in and changed his own password and then went home for the end of the day. Then presumably the attackers called in, asked the help desk, we're assuming this is what they did. We don't have recordings from back then. Asked the help desk to change the password, got got a help desk password, then immediately had to reset the password, right? login. So the no match one is the attacker's password, right? So the user changed their own password legitimately. Then the attackers called in, got a help desk password, put in their own password. Next day the user came back in

and couldn't log in. Now unfortunately the user may not have realized that something bad happened here because he had just reset his password the previous day. Maybe he thought he fat fingered it or forgot exactly what he put in. So he didn't freak out that he can't log in. He just called the help desk and reset the password again. So the next help desk that we have up here is the user calling in resetting his password and then setting his own password again when he had to log in. So essentially now we understand what happened, right? The user changed his password. Attackers leverage that to right after that call in change the user password and then

when the user came back the next day they didn't even realize anything was wrong because they changed their password recently. They didn't know maybe they forgot something. They just called into the help desk and changed it again. So this really helps us understand the sequence of events and how this happened there. And this is of course very valuable for our understanding of the investigation because we now know with pretty high certainty that there was social engineering again that started this. The problem is and this feels like a pretty comprehensive story, right? Because from this compromise everything else kind of falls into place and we'll do a quick review of everything in a minute. The problem is there's one question that

this raises which is you know attackers did this thing with the passwords. you know, the user changed and they changed and the user changed again. This can't just be dumb luck, right? There's no way that they just happened to reset the user's password that night after the user changed his own password the same day. That's I mean, what are the odds of that? So, we were trying to figure out is there any evidence that the attackers could have somehow known that this was coming? And then as we were talking to the organization about this, uh we found out a couple things. First of all, we found out that when the user changed their own password that first time, uh

that was actually a mandated password change that was long overdue. there were emails running around the organization for a while telling people a reminder by this date there was a group of people you have to change your password as a part of a larger group not nothing directed at this user. So we're like okay maybe the attackers already had access somewhere else and they saw these emails and that's how they knew when to do this. But then the question became well how could they have access somewhere else? We asked the organization forget this user have you had any other anything indications of compromise bad things that have happened and they were like oh yeah there was

this one thing uh of course help desk again. Um, so they actually told us and when we looked at the date, this was like a few weeks before this user was targeted. There was actually another user targeted and they were successfully compromised through the help desk. Let's just listen to a short snippet of this one. >> My food died and I can't log into my account. Are you able to help with that? >> Yeah, of course. I'll be happy to assist you with that. May I have your full name and employee ID? I don't have my employee ID. Is there another way you can verify then? >> Sound familiar? >> Thanks. >> Yes. If you don't have your employee ID

available, I will need the last four digits of your social security number. >> No problem. One second. >> Yes. >> Thank you, sir. So, I will temporarily disable multiffactor authentication on your account. That way you could log in without your phone. >> Thanks. What about my password? It was auto filled with my phone so I don't have it. >> Okay, I understand sir. One moment please. All right. I have removed the MX Bay device from your account. I will set a temporary password as well and then you will have to set up your own password. Is that okay? >> Yes. Thank you. >> All right. So why did we have this call? Right. This is also from a long time

ago. This is like 8 months ago. But this call, what happened was attackers presumably the same group called in, got the user's password and MFA device changed and then very quickly the user realized that something's wrong like they lost access. So they reported it. There was an internal investigation and the credentials got rotated again and the call got saved because they found it back then eight months ago. They figured, yeah, we had a fishing attack on our help desk. They weren't too concerned about it because it wasn't a very privileged user. They found it very quickly. didn't look like attackers really did a lot in their short uh time frame of compromise. However, what they

did do when we looked at the logging that was saved from back then, we see that what they clearly did is they went into this user's mailbox after gaining access and they searched for words like password and reset. Now, usually attackers do this because they want to delete the email that says your password has been reset after they've done it themselves. We're assuming that's why they did it here as well. But it's very likely and again we don't know if they got lucky or they were actually looking for that kind of stuff that that's how they actually also found out about the planned mass password reset that was happening just a couple weeks afterwards because they were in his email looking

for words like password and reset. So they would have known about this email and then coming back at the right date a couple weeks later when they knew a lot of people were going to reset their passwords waiting until it's late enough in the day that people have presumably done this and then calling in to the help desk. We now understand really how the entire attack evolved. So quickly just to run through this again, they had this initial not very sophisticated help desk call which worked which just gave them this basic information just by reading one user's emails. That information that recon led them to targeting this other user in a much more sophisticated way which led the user not

to figure out that something was even wrong. And through access to this other user which was a more interesting user presumably they did some LinkedIn research or whatever they knew he was a developer. they were able to get his uh access to his password manager and from there his GitHub personal access token which was used to run through all of these steps, right? So used to compromise a GitHub secret by running an action that created a brand new user in AWS. That brand new user in AWS was used to start initiating malicious transactions in small amounts from the transaction saver uh server. They've been doing this for months. Presumably throughout these months, they learned the process for how you get a bigger

transaction going too. And then they went for one big hit. They put in a big transaction presumably after they already called in the help desk to compromise. One of these very few people who could approve it approved it and made off with the money. So we have this pretty impressive attack, I got to say, spanning months, but with a pretty impressive payoff as well. Um, and there's a lot that we can learn from this about specific things that we have to do in prevention and detection response of these kinds of attacks. I'm not going to go into all of that now, but I do want to leave you with three higher level key takeaways. The first is

that cloudspecific attacks really do demand and require cloudspecific defenses. So every single step of the attack that we saw here was cloudnative everything right we had so many different cloud services involved but everything is cloudnative and organizations we heard before about how organization used to try and uh you know take what they used to do on prem and then do it in the cloud to make themselves secure. it doesn't work right. You have to have a specific cloudtailored approach to security to detect and prevent this kind of stuff. And this organization wasn't doing it well enough. A lot of this stuff could have been prevented and detected had they had these tools in place. That's

number one. Number two is speed. So this attack the whole span of the attack was very long, right? Actually not that fast. But every time the attacker was active doing something new, they were able to move very quickly from initial access to GitHub to AWS, from getting to AWS to starting malicious transactions, from getting the last the most important uh help desk call in to actually putting the malicious transaction out. And that means to us that we have uh really little time in many of these cloud attacks to detect and respond to these attacks. And that means that you know if it takes you two days to investigate an attack that took the attackers one day

to perform end to end you've already lost right and we see this all the time and that's why having a single pane of glass with everything and running through these drills and automating as much as we can of response procedures really is a must in the cloud and then finally and this everything boils down to this having centralized visibility in the cloud. So here we had like six different cloud services. Sometimes we have dedicated applications like we had here with the crypto applications. Unless all of this information from all these different places goes into one place, then a we can't create automated detections that are huristic and clever and go through the whole thing and b

during an investigation we're going to be much slower and much less effective at actually investigating. Thank you very much. That's great for yot folks. Uh I don't think people saw the slido, but we will take uh three questions from the room. If anyone has a question, stand up and speak up. >> Yeah, there's somebody on the left. Yeah.

So the question the gist of it is is about improving help desk procedures. Please your time. What do you say? >> Yeah. And and specifically as to whether this company failed by not doing it after the first time. I think there's there's a couple of parts to this. First of all, this company, you know, the first one, the one they knew about was really unprivileged and from their perspective, they found it really fast. So they did tell the help desk be on the lookout for this kind of thing but they didn't do a lot more than that. A lot of organizations would probably do the same there. You could do better but a lot of

organizations would probably do the same. So it wouldn't be too hard on them specifically. Generally speaking this is a huge problem especially scattered spider and you know we've seen so many uh attackers leverage these help desks. What organizations that are trying to do better on this do is sometimes you can just make a decision that for certain users especially the other two users here right not the one they knew about the very privileged users that can do a lot of damage. You just say no help desk for these users. You don't give the help desk the ability to do this. And yes, it can slow things down sometimes because these people now have to call their

manager or some other process for having them personally go through uh a change of their credentials, but I guess for some credentials, there's just no choice. Um, so I think there's definitely a lot of room for improvement, especially now companies have to do better on this help desk stuff because these calls are not unique in any way. We've heard so many of them over the last year or two. >> All right, we got two more in the room. this gentleman over here in the row and I got someone in the middle. That's what we got time for. Go ahead, sir. >> So, um, like I said before, I'm very careful not to reveal too much and so

not to identify the victims. I will say uh and I also mentioned this is actually a combination of of a couple of attacks by by similar groups that I that I put together to avoid identifying the victim, but um they both suffered very similar attacks to everything that went down here. I will say the most common attacker that we're seeing running these kinds of attacks is by far the North Koreans. Um, specifically they've, you know, they've been funding the regime through crypto heists for years. They love doing this stuff. I think my first big North Korean uh crypto heist like this was like almost a decade ago. Uh, and they've been getting better at it, but they've

really been spending time doing this. I think specifically the other attackers are obviously also interested because it's very lucrative. But when you see something that goes on for this long, this many months, it's less likely uh that you know just some ransom group or something would do this because they have a lower uh patience threshold. I also think you know there there's other attacks where the time isn't necessarily on getting the money out but on even getting in. We've seen I once worked an incident where a North Korean group was they established a fake academic identity and they actually created a relationship with a senior engineer at a cryptocurrency exchange talked into them about you know blockchain research for

months months and months before eventually sending him a malicious link which led to stealing of a lot of money. So they really have this patience and that that shows in these these kinds of attacks. >> All right, we got time for one more. I think the gentleman in the back's been standing here patiently. Yes. No.

So well I guess it depends on your configuration. In this case this user um had you know they were using Entra as single sign on and this user had this in his uh personal one one password which he could access through the entra panel. So once they were on his Entra single sign on panel you know with all the tiles where you can access different services once they had access to one password they could just click on a password and see it. Um, there are better configurations, especially if you're using a shared vault here. This wasn't a shared vault. This was just an individual's password or credential. Um, but yeah, of course, there's, like I

said, we're not going to go into this. Every step of this is absolutely preventable, right? We want to do better. That's a lot of what we're focusing on at Wiz Red is preventing this stuff. >> Well, thank you folks. If you didn't get a chance to get your question, I bet Yoton will be there later on during the conference. Whiz has a booth around here some, too. You can find him through that. Let's give Yoton a big round of applause again.