Don't Trust, Verify! - How I Found a CSRF Bug Hiding in Plain Sight

So without further ado, like to present don't trust verify by Patrick Odarity. Right. Good afternoon everybody. Thank you so much for coming to this talk. I'm really excited to be uh back at Bides after so many years and to share with you uh a story from my last year uh a little bit of bug discovery. Um so I guess to jump in, who am I? I'm Patrick Odardi. Uh, I'm a software security engineer. Um, I've been knocking around startups in the Bay Area for about a decade. Um, I'm currently a member of the Tailscale security team. Um, and away from the computer when I'm not getting sad at the keyboard, I'm often found wandering around San Francisco

with a camera. Uh, this is a rare example of me on the other side of the lens from last year. Um, so what is Guerilla CSRF? The library that we're going to be talking about today. Um, so it's a Go library that is a a HTTP middleware that is intended to wrap your existing HTTP application with a CSRF protection. And we'll get into what CSRF is in a second, but this is an example of what it would look like to to use this middleware in your application. We need it uh we instantiate a copy of it. We feed it a a serverside secret that's going to be used to generate uh secure cryptographic random tokens. We take our

existing application, we wrap it in this middleware and then we expose it to the uh outside world. So what is CSRF? It it stands for cross-sight request forgery. Uh and this is in fact one of the older forms of vulnerability that exists on the web. Um and it's essentially an attack where an attacker tricks a victim and their browser into making requests on their behalf. So it it's helpful to consider the problem from the perspective of an attacker. They know of a website bank.com and a money transfer form and they want to make requests to that form uh to that endpoint impersonating Alice, but they don't have Alice's credentials uh and they don't have Alice's uh cookie

to identify. So their trick is that they uh or their solution is that they trick Alice's browser into making that request on their behalf. And usually this is accomplished by serving Alice, serving the victim a malicious form that they think is pointed somewhere, maybe, you know, uploading a comment on Reddit, but is in fact uh pointing to this money transfer endpoint. Um, and so, oh, did I sorry, excuse me. So, one of the uh the common patterns in defense against this is known as the double submit uh pattern. And it works where the server bank.com for each new visitor. It generates a random secure uh value known only to the server and to uh this user

and it sets that uh value as a token and it also includes that value as a form uh hidden input for every form that is rendered for the user when they visit a page. And then subsequently for every submission, for every post request, it the server requires that the client send a valid and matching uh cookie and form input value. And if either of those two things aren't present or if they uh don't match, then this uh request is rejected. And so this is how the double submit name comes to be. Two uh two places for the same bit of information is relayed. And so if we return to uh our previous example, even if attack.com can serve

Alice a form that points to bank.com, it doesn't have the ability to send uh Alice a token or a a CSRF cookie that will be valid for bank.com. And so this form submission will be sent with only one of these two halves. Even if they're able to submit uh or interpolate the the form value, it won't have a matching cookie. And so the request will be rejected. It sounds all well and good. CSRF uh protection but unfortunately it's not complete. Double submit tokens on their own are not uh enough. There are one thing in particular that we need to consider is what happens if a attacker manages to launch their attack from a origin that has what's known as a

same site uh relationship with our application. And the two ways that they could do this are they could perform a HTTP plain text uh machine in the middle attack against your origin and any cookies they set here will be happily sent to your secure uh destination. It's the inverse behavior that's not allowed or they could attack a related website. So it's pretty common for you know maybe you have bank.com but there's events.bank.com bank.com which which is a little bit more loosey goosey with its uh security and you can launch a cross-sight request forgery attack from there and uh here any cookies that events.bank.com bank.com uh sets on the common site domain that they share between them those will be valid those

will be sent to the onward destination and so many frameworks you know Django Ruby on rails in addition to the double submit token pattern they also have this origin or refer header check uh this is a flowchart that I made of the Django frameworks uh code when I was trying to understand how this all worked and I should note that the double submit pattern is just like the the yellow box in here it's you know there's a lot of other decision points points along the way in an effective CSRF uh middleware. And the Go uh library that I'm going to talk about today, Guerilla CSRF, it is inspired by this implementation. It does something similar. Uh so, you know, every story

starts somewhere and this one starts with an email. Um one of the responsibilities that we have on my team is responding to all the customer questions and interrogatives that we get. I would say that one of the benefits of being uh at a security product company is that the questions are usually pretty high caliber or they can be and sometimes they take a little bit of research and time to dig in and answer fully and this was definitely one of those days. And so the question was asked what would happen if an attacker managed to gain a cross-ite scripting foothold on either our marketing site marketing site or a related domain and would they be able to use this to

perform a cross-sight request forgery attack against login.tailscale.com. This is the control plane, the sensitive application where people log in and administer their tail nets, invite new users, add new machines, all of that stuff. So definitely something that we want to protect and definitely an answer that we want to uh understand. Uh and I I'll put up my hand here and say that my first answer was like flat wrong. Uh you know, I completely just tripped over my shoelaces in responding to this customer. I misremembered how browsers and cookies work entirely. And I gave a very unsatisfying answer, like really uh ate my hat. Um, and rather than do that again and flub through a second round of

documentation, I decided to build a demo to just like see what would happen if you performed this hypothetical cross-sight request forgery attack. What would this uh work like? I'll leave this up if if people want to scan. This demo is live today. You can visit it. And the source code is available uh on GitHub. Um, and I'll walk through just the theory of how it works. So uh this demo application pretends or presents two uh origins. One is the one that we're going to attack and the one is uh the second is one that we're going to launch the attack from. It's a maybe a compromised marketing website something like that. And it's important that they to note

that they share this top level uh site this you know example.est uh domain. And when the victim visits the uh fishing website underneath the hood, it scrapes the target origin for a valid combination CSRF cookie and form value. And then it uses those to render a fishing page back to the user. And it also manages to set a cookie on the common example.est domain that they both share. And importantly, it fixates the cookie uh with a path that exactly matches the form that they're hoping to attack. And this is important uh because it takes advantage of two peculiarities of how browsers will send uh cookies, how they'll choose to order cookies when they're sending to their target

destination. And the first is that they'll choose path specificity uh first over domain specificity. Uh so you know it's perfectly fine here to fixate on uh this this more narrow form path and that will clobber any token that was set by a previous CSRF framework. So a lot of CSRF frameworks, they set their token to have a path of just a bare slash. And they do this such that the or they do this so that the cookie is sent accompanying all requests to the origin. So it matches every request. But this does leave the potential for us to fixate with our own more specific path and then it's sent first. Uh and this is in preference of any domain specificity.

And then the last thing to note is that uh this takes advantage of a weakness in the gorilla library which is that the gorilla library generates tokens that are not bound to any user identity. So the server can't distinguish between a set of values that were issued to Alice to Bob or to Mallerie. And so we don't actually need to guess or know Alice's token. We just need to substitute our own. And so this demo like sets all of that up. But even still, I expected it to be rejected because I expected the same origin policy within Guerilla to kick in. And lo and behold, this is what I saw in response. And I'll admit, I was

pretty stumped. I stepped away and I like really thought that I was the problem for a while. It's not uncommon for the ex the error to exist between, you know, the keyboard and the chair um when I'm involved. Um but after a while, after some digging, I really was convinced that there's something here. And so I I began to dig in like why why is my request passing when the library tells me it should be failing? You know, there are even unit tests here that have been passing since the library was committed that specifically address this use case. So what's up? Um and if we dig in, we can see that Guerilla, you know, it does have this

same origin policy um code that I spoke about. But curiously, it only runs when it considers that the request is being served over uh HTTPS. And it determines this by inspecting uh the URL scheme in the the URL that is parsed uh by the server. And so the question becomes uh where does this data come from? Where does this request URL scheme property get uh specified? Um, and here we can we can look to the Go documentation, which is very helpful. And if you're a Go developer and you're not frequenting the Go documentation, you're missing out. Um, because it changes frequently with updates to the language. And I often find myself forgetting bits of it that are awesome

and that I can use. Um, and one thing to note with Go's standard library is that the HTTP library in Go is used to create both servers and clients. And the HTTP request object in the library is shared between them. And so when you're writing your application, you have to remember which mode you're in. And you have to remember which rules are uh necessary to uphold to make sure that you're effectively holding the request object correctly. And if we scrutinize this documentation, actually all of the information that we need is here. It says uh for server requests the sorry I have to see my own slides the URL is parsed from the URI uh supplied on the

request line and for most requests fields other than path and raw query will be empty and if we click through to the HTTP RFC it gives an example of what this going is going to look like in practice it'll say that most clients are going to give us this uh request target that is just an absolute path that has no uh scheme team or uh host information. And so the question becomes where this where does this data come from? Uh it's not given to us by anybody but yet it is being operated on in the test suite. Unfortunately the answer can be found in a bug in go's standard library and a sort of mishandling of the

standard library within gerilla. Um, so the gorilla test suite uses a standard library uh helper function in Go called HTTP test new request. And this uh function generates request fixtures that are passed through your unit tests for you to make assertions over them. Um, and it takes as its second argument a string that can be either a path or a URL. And if you send it a a full URL, it will parse that and will and it will populate as many of the uh request URL uh strct fields as it can based on what it sees in the URL. The problem is if you do this in your test suite, you will be returned a request object that's like

a Franken request that's unlike any request that you will ever see in production. And so your unit tests will happily pass green. The branches that you want to have coverage for are green. It's great, but none of this is ever going to run in production. And so to recap, the origin checks for this library only run when the request URL scheme is set to HTTPS, but in fact, this is never populated in production. And so this code has been inert and it has like it never actually did anything. So that the library was CSRF. It was just RF. There was no CS. Uh, and so I think this is like a perfect example of something that

friends of mine will hear me harp on about a lot. Uh I love to consume um accident investigation or safety material from related fields. And the Swiss the Swiss cheese accident model as it was popularized by James Rezen of the University of Manchester says that in a human system no one layer of safety is sufficient and we need to compose multiple in succession in the hopes that their mutual uh overlapping uh protective layers will form such that no accident can fall through all of them. But here unfortunately we have you know the opposite mutual deficiencies line up to allow for this bug to happen. We have the lack of identity binding on the uh CSRF uh tokens as they're issued. we

have this subtle behavior in the test uh suite and then also the fact that this behavior only exists in production and that a lot of uh people develop you know with a plain text HTTP uh environment and then they deploy in production and so they don't see this they don't try it and so this lines up to allow for this CSRF vulnerability to exist uh for as long as a decade so uh just in terms of timeline um we discovered this at the end of last year um and we sent a patch upstream to the guerilla project and this code is now available uh in the most recent 173 release. So if you run a Go uh codebase

and you have like other oper uh websites that you operate adjacent to it, I would highly recommend updating uh worthwhile. Um so that that's gerilla but you know taking a step back uh and you know beyond fixing one bug in one library um this CSRF problem is a pain and it's kind of plagued us for 20 years 25 years is it possible that web browser standards have evolved to make this suck less mercifully the answer is yes uh and the uh the answer comes in this uh more recent technology called fetch metadata request headers uh and these are a set of headers that browsers now send accompanying all requests to uh what they consider to be trustworthy origin.

So anything served over HTTPS and they explicitly signal to the server what the client believes to be the context of the request that's being made whether it's cross-sight whether the data is going to be interpreted as a document or a script um what have you. And so previously where we had to infer all of this, the clients now just tell us and we can make these confident decisions about whether the request should be served just from these headers alone. Uh so this is an example of what that would look like um when site.ample uh includes a same origin fetch request to fu.json. And so the client here can understand that site.ample example and the requested resource have the same

origin relationship and explicitly tell us and we can accept that that's all great but conversely if we had a malicious uh origin if we had evil that example try to include that same resource the client is uh can discern that the two origins have a cross-ite relationship they don't have any relationship to each other and will tell us and the server can simply reject us um I am in love with the simplicity of this solution uh where previously we had to infer this and there was a lot of uh like at a distance it's now just explicit um and this is how it looks in practice. So you know it's a much simpler thing to

implement. There's no server side credential to manage in terms of uh token generation. Um if you want to go even further and block get requests to prevent uh leaking of uh resources cross origin or you know prevent cache timing attacks, you can do that too very simply. Um yeah and the there's no potential for this to drift if for example your web application finds itself sometimes behind a reverse proxy that changes the uh origin from the perspective of the client but the app doesn't know that. Uh sometimes you can have CSRF false positives. Uh this is a current pain in my life uh because we ship a client that sometimes people embed in other contexts. And so this is

just much simpler. The client will tell you what it believes its relationship to the requested resources and you can take that at face value. Um so these were made available in uh or first introduced in 2019 but they've become broadly available in all standard browser uh revisions um as of 2023. So they're now generally available um we are currently running some instrumentation to evaluate whether we can replace double submit cookie protections with uh this middleware instead at tail scale and uh it's looking imminent. we see like 99.98% compliance, like a real like rounding error of non-compliant, mostly old Safari, Grumble, uh, releases that are catching up. Um, but yeah, it it's the new technology and we're we're going

to use it. So, um, what can we take away from all of this? Um, the first is, you know, unit tests are just the beginning. We should really kick the tires of our code. Um, you know, it can be helpful here to create small experiments that, you know, demonstrate the integrity of the security controls that rely on that that we rely on. Um, so that when we we poke them, we don't find out that they're paper thin. Um, you know, we should validate the assumptions on the differences between our test and development and uh, production environments. Here, there was only one bit of information that differed, but it was crucially the bit of information that mattered to make this middleware

effective. Um, again, you know, like we should really just do the experiments and empirically measure rather than make assumptions. Uh, these new fetch metadata request headers are awesome and we should all use them. But they're also, you know, representative of the fact that HTTP, the OASP top 10. These are not concepts that are, you know, set in concrete. They evolve over time. I'm actually a little bit disappointed that this technology was available for, you know, some so many years. I didn't know about it. I watched these headers go by in various logs and I just like I wonder who they are for. Um but they're really cool and so I'm going to take on myself

to try and be a little bit more uh you know up to date on the professional standards as they are. Um you know there's there's good stuff in here for us to bring to bear for the benefit of our users. Um and yeah with that that's me. I realize I managed to get through this. This is the fastest run I've done by like many minutes. Um, so yeah, I'm going to turn it over for questions and say thank you so much to uh Bides for having me, for all the staff that puts this on and makes it possible and for you for your time and attention. Yes, thank you Patrick. That really was a lightning round. Uh, you have at least

uh you've got uh 10 minutes now which we could fully put in for Q&A. So, as a reminder, the way we do Q&A, as I mentioned before, is through Slido. That's sli.do do is the fastest way to get to that uh website and the the conference code besides plural SF2025. We are in theater 15 according to the schedule. So just select theater 15 and submit your questions there. I'm going to be reading out the the most popular questions. You can upvote questions if you see them and they are exactly what you want to ask. Um that helps us prioritize things because we have so much time. I'm certain we're going to have hopefully lots of

questions. So here, let's take a look at them. All right, we got several pouring in. Uh, so in Kubernetes, a lot of add-ons are there which are from open source container images from OPN source registry. Um, I want to make certain this is for the right. Does this seem like a relevant question? How do you manage open source container images? I mean, I could speak to that, but I Yeah. Yeah. That that looks pretty strange. So, um, we're going to save that one. Uh how about this one? That seems relevant. What would you would you use this new method in addition to sub double submit? Uh my hope would be to replace double submit with this entirely

because really like the most important bit of double submit is the origin. Um the double submit I think the like cookie portion of it is actually not as useful. I think we kind of combined the cookie method of relaying information and the fact that we can only set cookies for uh same sites with this origin header um checking and in fact I think if we could just do better origin header checking we wouldn't need the cookie thing and it means there's one fewer credentials to manage you can deploy this without configuration so my hope is that it can like entirely replace it because I think some of the uh like even some of the more secure

CSRF APIs um they're easy to make uh brittle by you know generating tokens that are too narrow for the scope that you're authenticating. So I I much prefer this just because it uh gets rid of a whole bunch of usability issues that have uh that I previously didn't consider but that like with the open source uh viewpoints that we have and seeing so many wide deployments they definitely add up to like some amount of pain that could be just completely done away with. So this next one is popular. Uh, tell me what resources you look at for accident investigation hobbying. Oh, um, I read anything that, um, I mean, US Chemical Safety Board YouTube videos. Definitely there's some people

in here who watch those. Those are great. Um, anything, you know, uh, that the US Air Force or the Navy or and various uh, organizations around the world will release because if you look at about if you think about it, they're just, you know, they're they're human computer interface problems. It's just that they're giant, you know, billion dollar um hunking pieces of metal that have computers in them and ours are just little laptops. Um but they're exactly the same thing of human psychology of like the computers uh conditioning us to think that they're always reliable and then you know they have catastrophic failures and those are you those patterns exist both in our domain but

also in these domains and seeing how they make themselves resilient to these like catastrophic failures is uh is highly educational. Do you think this can get into the Go standard library? There seem to be a lot of important checks like this and DNS rebinding that keeps getting longer and longer for people trying to use Go. I would love to see this in the Go standard library. Yeah, I think this is a missing stair and in the fact that you could like write an Go application, use all the Go standard library stuff and then have this as a missing uh thing. So yeah, there is some uh there is some hope. I would uh like to see it in the

Go standard library. I think there was previously a proposal to take the there is an X uh the extended go standard library the X slpace has a XSRF token uh API for generating the cryptographic tokens but it doesn't have a HTTP middleware for integrating it into your app. There was previously a proposal to bring that into the standard uh library but it kind of stalled out for a lot of reasons to um do with all of the usability concerns that I spoke about here. I think this solution because it is just that much simpler to implement has a better chance of being the thing that lands in the Go standard library. So I have my fingers crossed but um I'm

not a committer to go which is can't make it happen. Uh and then just to clarify you haven't shown any issue with the go standard library itself right this is for a particular framework called gorilla. So there's two bugs here. One is the go standard library HTTP HTTP test new request uh helper that had uh confusing behavior that should be clarified because effectively there's no way to easily use it to generate test objects that are like what you are going to experience in production. So it's effectively setting you up for failure. So there is an open issue in the Go uh tracker to fix that and make it possible to not fall into this trap of writing sort of tests that

don't actually represent production behavior. And then separately there was the bug that it enabled in Guerilla and that has been solved. Yeah, we still have five minutes here so uh keep piling in these in we're doing really good. Um do you have any ideas about how we should in general learn about these types of new security tech like these headers? Good place to go or place to start reading? Yeah, apart from reading MDN every now and again, I am looking for good answers to this myself. uh like I I'm going to write about this publicly in blog and I'm going to contribute my bit to you know the amount of information out there. Uh I do follow

newsletters as well. Um and that's like for I try and keep my ear to the ground and see what's coming up. But yeah, I don't I haven't one good answer for where this new information comes from. Um so this one is multiart. So the first really kind of depends on whether you have looked at the docs for HTTP test. Have you looked at uh I have looked uh not in the last week but I have read them. All right. Uh so it it asks do you think the ergonomics of that function are surprising and that similar bugs might hide in other code bases that use it to test prod HTTP implementations? I would not be surprised. I did write a um

a go vet uh that looked for the specific uh behavior in um guerilla that was problematic which was making a a direct comparison against properties in the URL um that you couldn't guarantee were in uh that were set in production. And so I um have a go vet that you can uh I'll actually I can add it to the appendix and I'll uh link it out to people. You can run this over your codebase to see if you're doing the problematic behavior. Um, I tried to run a GitHub search to identify projects and there there seem to be others. So, I'm I'm pretty confident that there are other projects that have either um emulated Guerilla uh their implementation of CSRF

or have made other sort of uh false steps with, you know, using request properties that aren't actually there in production. Yeah, I there's probably more research to be done here. So, I would encourage it. In that vein, do you think there's anything that Go should do differently to prevent these kinds of issues in the future? Quite open-ended. Uh, I mean, my scorchiest take is that HTTP requests like created this because it's dual purpose. You know, dual purpose APIs can induce misuse. And I think my ideal world, you know, having come from some uh languages with more aggressive type systems, I would like to bring, you know, type system design, static analysis, other things to bear to

make this misuse, misuse resistant. I think, uh, you know, if you make something possible, uh, you know, over time, chances are people are going to do that thing. And so I view this sort of like as a harm reduction API design. Like if you make it possible, people are going to miswire it. So just don't make it possible. So, we're out of questions on Slido, but I know sometimes people are shy typing and paradoxically great at speaking out loud. So, as one of the privileges to extend you here in theater 14, is anybody want to ask a question? Just make it short. Raise your hand. I'll I'll try to listen to it. I'm going to

repeat it for the stream and then we'll have it. But if not, go ahead.

Not USF. All right. So, there was just a suggestion of the Canada BC Safety Board, which is alike if you've ever seen the US Chemical Safety Board's videos, uh, is another great accident investigation uh, tool resource. Is that your Yes. Okay, cool. Safety SOS rise up. All right. Well, uh, Patrick has volunteered to, well, as many speakers do, um, a really amazing resource. You can talk to him up on, uh, in floor 4 in the vendor area. He promises also to stay for all the various parties. And so, really like to thank you, Patrick. And, uh, thank you all for attending. Besides here, you know, we have we have one last one. Okay, you ready for start round?

What suggestions do you have for a beginner to start identifying security bugs? Uh, I mean just like keep asking questions and really like there's a humility to um just like letting yourself be ignorant about something and you know experience it for the first time. I don't have any credentials in this. There's no reason that I can be here and nobody else can. So like just giving yourself permission to be silly about something is uh you know opening opening the door to having a fresh perspective on