When authn breaks: real world failures

All right, I'm going to get started. You're you're in the uh talk when authentication breaks real world failures. If you're not here for that talk, you're in the wrong track. All right. Uh my name is Maya. I'm the founder of a new company in the enterprise identity space called Oblique. Uh it's an identity directory for managing employee groups and entitlements. Previously I was at Tailscale where I led um product and engineering and um I've basically had a career in enterprise security and product management. I was also at GitHub working on software supply chain security and at um Google cloud working on container security and encryption and encryption key management. Over the years, I've had a chance to work on a

lot of authentication and authorization features like adding pass keys for authentication, building skim into products to let you sync enterprise users across systems and adding more fine grained user roles for authorization. And they've also had a chance to deal with a handful of small incidents around authentication where authentication fails like um what happens when a user logs into their account and sees another user's data. And so today's talk is about authentication. Um how it can fail and what happens when it does. Um I spent a lot of time researching authentication authorization recently and um I wanted to look back at the last 5 years of um failures that we've seen in the wild and see if there's any lessons we can

actually learn from that. If you think about it, authentication is one of the most critical identity controls that we have. It's both the first and last line of defense in some sense. Right? Right? If authentication fails in your environment, it means that attackers can bypass all other controls that you have. And since identity is tied to every other system, the compromise of even a single account in your environment can have really wide reaching impact. So the hope is by understanding some of these common patterns in off failures, we can build more resilient systems. So the agenda for today's talk is just that uh we'll cover authentication failures again all of which happened in the last

5 years in order to understand what happened, how it happened, and how to avoid it in the future. and I'll also briefly cover common implementation failures in authentication protocols that you should avoid. So, let's jump right in. Um, there are so many different kinds of ways that authentication could go wrong. Uh, if you want to go check out CWE287 for some of the ontology that you can use to enumerate um, improper authentication weaknesses. And so, there's a few different kinds of authentication failures we'll consider today. First is credential compromise. These are the most common type of authentication failure with password theft, credential stuffing, fishing, brute force attacks, and social engineering. This could also happen if

your identity provider or your password manager is compromised and someone gets their hands on your credits. Though they're the most common, we also know how to mitigate these kinds of failures. It's by using multiffactor authentication. So, I really kind of want to rename this this whole category. It's not credential compromise. It's you didn't have MFA. Um, so let's just call it what it is. That's the real failure here. we know how to fix that. There are so many of these that I could give an entire talk about just this. Um so I'm only going to limit myself to a couple of examples in the next few minutes. The next category is MFA and recovery weaknesses. So um SIM swapping attacks,

MFA fatigue, vulnerabilities and implementations around tokens or also again social engineering. The next category is session management failures. So session hijacking, cookie theft and cross- sight scripting or curve attacks against authentication. Then authentication pro protocol weaknesses um where OIDC or OOTH is misconfigured or you have jot tokens that are not properly validated. This is also where I'd put like just generally broken off flows which are more common I think than specific implementation failures. And then lastly infrastructure failures including man-in-the-middle attacks and compromises of C certificate authorities. Again, we're going to focus on real world examples from the last 5 years. Mostly look at examples that uh involve user accounts being compromised and mostly focus on tech companies since

that's where, you know, a lot of us work and what we're familiar with. Um these are also the organizations that we should hopefully be able to learn from because they should have some of the basics in place and look a lot more like us. So, let's look at these specific examples. One thing I'll call out is that although it's tempting to make fun of some of these organizations for poor controls, going to try to refrain from doing that since it's by them sharing information about how they got attacked that actually helps all of us learn how we can get better and protect ourselves. So, let's start with um Twitter in July 2020. Apologies to the folks in the

audience who worked on this. Um, Twitter had an internal admin tool that allowed the support team to view and change information about a user's account, including suspending the user or changing the user's recovery email. Internally at Twitter, access to this tool was apparently quite widespread with something like 1500 Twitter employees having access and using it inappropriately to track celebrity locations based on their IP addresses. Knowing that access to such a tool internally was widespread, attackers scraped LinkedIn profiles of Twitter employees who were likely to have access. Then using LinkedIn recruiter and LinkedIn sales navigator got those users phone numbers. They contacted those users likely who who were likely working from home because this was at

the very beginning of the pandemic pretending to be Twitter IT and directed them to a fake VPN login page. They then asked those users for their two-factor authentication codes and users gave them those codes. In this way, attackers got access to the Twitter network as employees with this privileged access. Um, and you'll see this pattern happen a lot in the common in the following attacks. Like this is a very common pattern. The impact was that around 130 Twitter accounts were affected including those of Barack Obama, Elon Musk, Apple, Binance, and about 45 accounts were used to tweet a scam message asking users to send Bitcoin to a specified address. around 320 people did that and sent

about $120,000 worth of Bitcoin before the messages were taken down. This is a pretty classic like social engineering and fishing scheme again which were very popular at the beginning of the pandemic before organizations had the time to necessarily educate their users that this was not the behavior that they should be expecting from their IT teams. Twitter had appropriate security measures here in terms of passwords and and MFA uh but insufficient authorization since there were so many employees who had this access to this tool and it was so accessible once you were already on the network. In June 2021 um a flaw was discovered in Microsoft Azure ID's seamless single sign on. Um seamless SSO works such that

when a user is logged into a device that's already joined to an Azure AD domain, they don't have to log in again. their identity is automatically used to log them into AD. This is implemented using a Kerros authentication challenge. If a user doesn't exist in the AD tenant, then they get an error back. And that error has like a little bit too much information like whether or not that user exists, if they have the wrong password, if they don't have an Azure ID account, or if their account is locked. This provides enough information for attackers to then enumerate accounts in an Azure AD tenant. Uh, and notably, this is possible without generating any kind of sign-in events or getting a rate

limited, which allows an attacker to attempt to brute force these with a password spray attack. Microsoft quickly added logging to this flow for allowing you to detect whether or not this was happening, and it made it possible to disable the feature. As far as we know, this was not exploited. Um, but it just points out that like no one is immune from incorrectly implementing authent. It's really about how you respond, right? Even an organization like Microsoft with a critical authentication product like AD can inadvertently weaken authentication with a poor implementation of a legacy protocol like Ker Ross. Like everyone everyone's subject to this. Another Microsoft issue a few months later in September 2021 um a Taiwanese

researcher discovered several vulnerabilities in Microsoft Exchange as part of a pone to own contest in Vancouver. The Exchange server front end allows direct access to specific URLs in the back end. So you can link directly to a user's inbox via um an explicit loon feature. In order to do this, Exchange normalizes URLs, but you can manipulate that URL. So you can have an unauthenticated attacker giving access to getting access to other internals of the Exchange server. Dubbed proxy shell by chaining together this and two other CVEs that they found, an attacker can obtain remote code execution on several generations of Microsoft Exchange servers entering through an exposed port. Globally, tens of thousands of Exchange servers were vulnerable. Microsoft

released patches, but this was just like too juicy. They started getting exploited in the wild. And about a year later, in 2022, new attacks started being detected that looked a lot like the old attacks that looked a lot like proxy shell. Um, so it turns out the patches that they they released were actually insufficient. Exchange server was still vulnerable to serverside request forgery with two CVEs dubbed proxy not shell with this beautiful logo here. Um and in this case user loon passwords were needed but the second factor authentication was not required. So if you were an authenticated attacker you could still get um remote code execution. The initial issue here was insufficient attention to input validation uh

manipulation and normalization. But in general again this reminds us that uh you should really implement further authorization. You shouldn't just have users trusted after an initial off um the principles of zero trust. Basically, in January 2022, Octa was targeted by the Lapsis hacker group. Um, a lot of the authentication vulnerabilities that we've seen in the past few years are due to Lapsis. Uh, and they definitely have a playbook. So, they look very similar. Um, Lapsis was able to compromise a third party support engineer's laptop, gaining access to Octa's customer support system. Several months later, the attackers posted screenshots of the Octa's of Octa's own Octa instance on Twitter um with access to super user accounts and basically this sent the

industry into complete mayhem for a couple of days. Um in all, it turns out the attacker had access to the employees laptop for 5 days, which included access to 366 customers octenants um allowing potential account taker through password reset requests. But apparently that specific ability they only had for about 25 minutes and there's no evidence that they actually used that. So although the impact ended up being quite limited, Octa's comms and response here was like pretty bad, right? They should have disclosed an issue to their customers way earlier. In their blog post, they called this an unsuccessful attempt. That's not what it sounds like to me to compromise the account. Um, instead what ended up

happening is a lot of the news around this was actually the attacker kind of doing PR. So the attacker went on Twitter and was like, "Look what I have." Um, trying to get credit for the attack. And the attacker was answering journalist questions because Octa wasn't. Um, so notably, Cloudflare was like pretty upset about this. They published their own investigation and out of an abundance of caution, they reset the passwords of all of their employees who had recently performed password reset requests. Again, the main lesson here is more about communication and how to react in terms of an incident because again, incidents will and do happen. Next up, an implementation error in GitLab. When authenticating to GitLab,

you can use your existing um identity provider to authenticate. So, if you have an existing account with, you know, Google or GitHub, you can use that to authenticate to GitLab. Omnioth is the authentication framework that GitLab uses to allow those external Open ID connect providers to to talk to it. In March 2022, um GitLab detected an issue with how it used Omnioth. It turns out that when accounts were registered using an Omnioth provider, GitLab inadvertently also set a hard-coded password for those accounts so that users could log into those those accounts directly um and not depend on the external provider. Um, crucially, these were not actually random passwords. They followed a predictable pattern that an attacker could have

compromised. Um, this was a lucky break. It's not super clear to me how this was discovered, but it sounds like maybe somebody was just walking around the codebase. It doesn't seem like there was any evidence that this this was compromised. Um, meanwhile, next month in April 2022, over at GitHub, um, GitHub discovered that an attacker was abusing OOTH user tokens for Heroku and Travis CI to download organizations data. So, how this works is like GitHub uses OOTH tokens for users to integrate GitHub with other systems, right? So, to connect your CI/CD pipeline to GitHub, you generate an OOTH token for a user in GitHub and then they give it to your build system which accesses the code on

your behalf. Since only Heroku and Travis CI were compromised, what seemingly happened here is that the tokens issued to them only were compromised, probably where they were stored in Heroku and Travis's CI sides, but I couldn't find anything to confirm that. The attacker used the tokens to access the API for a specific user, and then they could enumerate the user's organizations as well as their private repos. They then selectively clone repos that they were interested in um presumably to look for secrets that they could then use in other attacks. They didn't modify any code or packages or gain access to any user account data or credentials, but they did find those secrets that they could use in further

attacks, including the npm team's own AWS API key for npm's infrastructure. Um, from npm zinfra, the attackers downloaded all private npm packages, a package manifest, I should say, and package metadata, 100,000 npm users usernames and hashed passwords, and two organizations private npm packages. If there's a lesson here, it's to review the set of authorized OOTH apps that you have in your environment. Regularly rotate your OOTH um um tokens and ensure that the scopes on on those oath apps are appropriate. Um so that if they are compromised, they don't have as widespread impact in your organization. Um this is a lesson I feel like we've already learned in like the corp side of the house. And so it's interesting to

see the prod side being like, oh yes, we should be doing that too. Um you can also use GitHub apps. You don't have to use OOTH with GitHub anymore. The following month in May 2022, Cisco discovered a potential compromise to its network. The attacker first compromised the account of an employee who had saved their Cisco work credentials to their personal Chrome account. This wasn't sufficient to gain access to the employee account. Um, so the attacker sent the employee a very large number of MFA push notifications and um, this is called MFA fatigue. The user succumbed to this. Um, and they also used um voice fishing or vishing to try to convince the user to to to accept that push

notification. Uh, again pretending to be from the support team. Once they successfully obtained access, they enrolled new MFA devices to the Cisco VPN and then were able to ask escalate their privileges um on the network to admin privileges. The attacker is known as uh UNCC2447 is supposed to be an affiliate of Lapsis, so not Lapsis. Um, but was likely trying to deploy ransomware when this was discovered. um they didn't make it that far. Thankfully, although the attacker managed to get access to internal Cisco systems, none of those were like critical systems used for like product development or like code signing. So, the attack was pretty limited. And then a few months later in September 2022, we see again a very similar

attack. This place taking time at Uber uh and this time being Lapsis. So as with similar attacks on Cisco, Microsoft, Nvidia, and Octa, Lapsis targeted Uber by obtaining VPN credentials of an external contractor either by themselves stealing credentials directly with like info stealing malware or by buying credentials off the dark web, although they had the username and password of the accounts. Again, they were not able to get onto the network without a second factor. And so each lo login attempt that they had sent an MFA prompt to the user. The user, again, overwhelmed with MFA fatigue, was contacted on WhatsApp by supposedly tech support, telling them to accept the prompt, and then they allowed the attacker in. Once on Uber's

VPN, the attacker was able to access employee accounts, including gaining access to Google Workspace and Slack, and a tool that was used to manage invoices internally. I think like we can't put the blame on the users, right? We never put the blame on the users when it came to to or we don't anymore put the blame on the users when it comes to um fishing attempts. And this is exactly the same thing, right? Like users will be users. They will do things like reuse or leak passwords in inadvertent ways. They will accept MFA prompts because it's the middle of the night and they're tired or because they accidentally accept it. Um the main things to think about here

around MFA these MFA fatigue attacks are that we should really be moving to fishing or fishing resistant MFA where possible like security keys or offn and we should also make sure that we like alert on any kind of MFA activities as much as we would any other off activities. So like enrolling new MFA tokens as an example is something that we should be looking for. Next up, Fortnet, a producer of firewalls, discovered an issue in their products in October 2022. Um, this vulnerability allowed remote unauthenticated attackers to bypass authentication and gain access to the admin interface of Fords firewalls and 40 proxy web proxies. Attackers could do this by sending a specially crafted HTTP or HTTPS request.

Fornet released patches to these pretty quickly to remediate these attacks and they released an update a few days later that they were actually aware of an instance where it was being exploited. Uh and where this was seen in the wild, the vulnerability was exploited to connect to the device, download its config file and then add a malicious super user account called Forigate tech support. Um so the attackers were trying to get persistent access. Thankfully this was detected so hopefully this isn't true anymore in the wild today. Um, now let's talk about LastPass. This one's a bit murky. Um, LastPass hasn't released many details, at least especially about the initial part of the attack, which makes it really hard to

figure out exactly what happened. Um, in August 2022, a LastPass engineer's work laptop was somehow compromised and the engineer's credentials were stolen to get access to a dev environment. Um, again, we don't know exactly what happened here, but the attacker did end up, according to LastPass, authenticating with legitimate credentials with MFA. So, it's not clear if this is like session replay. We don't know if they had a username and password, and then they used MFA fatigue. Um, but then from this dev environment, they were able to download source code for LastPass and some system secrets. LastPass didn't really talk about this, thinking that it was over until more things started happening. Um, so several months later in December

2022, LastPass discovered other unusual behavior in its systems, presumably via the compromise of a home computer of a dev devops engineer. I don't know if this is the same person or not and likely by the same threat actor. The attacker was able to capture the employees master password after the employee had off with MFA and then get access to their last pass corporate vault. Um they use these credentials to access and download cloud cloud backups which included API keys, thirdparty integration tokens and also encrypted cloud backups of user password vaults which LastPass cannot decrypt. They also accessed a backup of the LastPass MFA database which contains copies of LastPass authenticator receipts and users MFA backup phone numbers.

Since then, it's clear that the attackers did in fact manage to decrypt some of the users vaults backups, likely due to those users having weak passwords. And amongst other damage, they have managed to steal at least $5 million of Bitcoin from wallets using those credits. Um, CircleCI was alerted to an issue in their environment when one of their customers experienced suspicious activity for a GitHub oath token that was stored in CircleCI. It turns out that an attacker had gained access to a circle CI engineer's laptop using malware on that device and from there was able to steal a valid 2FA backed SSO session. U the malware used cookie theft to gain access to some production

systems. And this particular employee had the ability to generate product access tokens. So the attacker did just that and got access to um be able to steal customer environment variables tokens and keys. CircleCI issued an urgent warning to its customers to rotate any of the secrets that they had stored in CircleCI, including OOTH tokens, project API keys, and environment variables and SSH keys. CircleCI then went a step further and worked with other providers in the industry to invalidate and rotate their customers OAS tokens automatically for users including for GitHub, Bitbucket, and GitLab, and then start periodically rotating them from then on. Um, I think CircleCI like did a really good job here. They had really strong off. They

had two a favor of their employee accounts. They had endpoints uh tooling that didn't detect this malware unfortunately, but then they took really good action to go remediate as many customers as quickly as possible. A few more. Um in October 2023, Octa announced another incident uh where from September 28th to October 17th, a threat actor had unauthorized access to files in Octa's customer support system which affected 134 octa customers. A customer support support employee had signed into their personal account on Chrome and stored their octa work password in Chrome. At some point that account was compromised giving the attacker access to Octa's customer support system. And um this was a system that was used to

that also had the ability to have service account credentials to view and update like customer support tickets. And so in those customer support tickets were a bunch of valid Octa sessions for those customers. Octa didn't initially detect this. One password did because they were affected. Um some of the there was um five customers who were compromised including cloudflare one password and beyond trust and cloudflare was pretty upset again. Um the threat actor also ran and downloaded a report with the names and email addresses of all octa customer support system users i.e the Octus customers, which is a great target for future fishing. And Octa made a bunch of changes as a result of this, including implementing

um new security measures like disabling personal profiles on Chrome Enterprise. In late 2024, Octa again um they released an advisory that there was a vulnerability in how cache keys were generated in their AD delegated authentication system. Instead of using a key derivation function, B-Crypt was used as part of the cache key generation. uh combining the user ID with username and password. B-crypt has limited input. So if you have um it takes up to 72 bytes. So if you had a sufficiently long username, the password was actually not even used as part of cache key generation. Um so users or an attacker would be able to authenticate by providing just a username and a stored cache key, not actually needing

to provide a password. Um also brypt is like a little bit to blame here. Like brypt depending on what language you use it in doesn't throw any errors. Um, but also like Octa is an identity company and should have known what it's doing and not used a function with limitations here and where that was not meant for this purpose. Um, the only real impact here was Octa's reputation. Um, and one last item, uh, Snowflake. In early 2024, some members of the online crime focused chat groups, the comm realized that Snowflake would be a juicy target. Um, there's lots of interesting data from lots of big companies. So, a group dubbed UNC5537, led by a Canadian hacker, purchased

customers Snowflake credentials on the dark web, again, probably obtained via info stealing malware, and began logging into Snowflake accounts that didn't have MFA and stealing that data. Up to 165 Snowflake customers were compromised. Uh, but just stealing the data wasn't the end of it. AT&T customers phone and text message records for 110 million customers were stolen and used by other criminals to figure out who was investigating them in those in their in their um in their crimes. Um they leaked 160,000 Taylor Swift era tour barcodes from Ticket Master. Um dozens of companies were affected including Ne Marcus and Santander and they were extorted in exchange for uh not leaking their data netting the attackers about

$2 million. So, this is just to say like these basic credential stuffing attacks still take place and they still work. Um, these are things that people in this room should know. You should have MFA on all your prod, right? Please do that. Um, so we just talked through a lot of examples. We saw a little bit of everything. There's straight up credential compromise like no MFA at Octa and Snowflake. OOTH tokens stolen at GitHub and an existing session at CircleCI. MFA fatigue affecting employees whose credentials were compromised at Twitter, Cisco, and Uber. And implementation mistakes in a variety of places. I think one of the things that I wasn't expecting to look through this is I was

expecting to find a lot of like basic O implementation things. It's not that, right? Like I just said, it's really basic stuff that we're still dealing with. It's credential stuffing and it's not and having fishable MFA. That's what's getting us right now. Um I'm going to skip this quickly, but if you want to learn more about common authentication protocol failures, there's a great talk I'll link to at the end. So let's see what did we learn from this. If you're implementing O in your system, make sure that you verify common protocol implementations and mistakes. Use the right algorithms and double check anything that is particularly sensitive like or touches cryptography like hashing input input normalization.

uh implement session timeouts, implement other prot protections for authenticating sessions by preventing logging attemp and by preventing and logging attempted brute force attacks and implement more than just basic offer your customers like give them SSO, give them skim, give them MFA. Like these are things people need. Um and like I said, if there's one thing that I have, if you take one thing away from this talk, it's the everyone's been affected by lapsis' playbook. Just roll out strong fishing resistant MFA like hardware tokens or paskis for your users. Um, but if you want to do more, uh, we'll first implement MFA, which you should already have had. Require the use of strong MFA when you do. So that's hardware or web,

uh, hardware tokens or webin. Educate your users on what to do if they experience an attack when someone's actually trying to get access to their second factor. Um, especially new employees. They might not have seen MFA fatigue before. And set up alerts to to to monitor for those MFA failures and resets. You should also regularly review the set of issued and valid OOTH tokens for your users and regularly rotate OAS tokens where they're used in integrations between services and prevent employees from using personal password managers. Sometimes that's as simple as giving them an employer supported password manager instead. Um, pay special attention also to how you're doing off for non-employees, particularly folks in your support teams

or other contractors. And this should also include endpoint management. So like looking for key logging or info stealing malware on their devices. Even the best authentication systems can fail and they will fail. So design for defense and depth. And you should also assume and prepare for compromise, especially how you're going to communicate and react to when those things happen. Authentication is hard. Authentication failures continue to be a primary vector for major security incidents. Our best way to deal with them is to learn from the past and prepare for what's inevitable. Um thank you so much for today's talk. The slides are available online and I'd recommend also checking out these two other talks. One is specifically on

Lapsis' playbook and the other one is specifically on failures in authentication protocols. Thank you.