BSidesSF 2025 - One Search To Rule Them All: Threat Modelling AI Search (Kane Narraway)

All right, everybody. We're going to go ahead and get started here. Uh, introducing Kane Narrowway. He's gonna give a talk called One Search to Rule Them All: Threat Modeling, AI Search. Take it away, Kane. Sure. Sorry I didn't have the, you know, the dramatic Lord of the Rings music to kick this one off. Um, so, hi everyone. Thanks for coming. Um, I'm Kane from Canva and I'm here to talk about threat modeling, AI search. Uh, I'm actually based in New Zealand normally, so if there's any billionaires watching that need me to threat model their bunker, uh, you know where to find me. Uh, I didn't include nearly enough Lord of the Rings references in this and so

I've just included some pictures of myself in Hobbiton and various places. So, uh, and this this wasn't sponsored by New Zealand Tourism. Um, so getting on with the talk, uh, many of the AI security talks that I find go into securing AI platforms. So things like Claude, Open AI, it's often a big focus on the LLM OASP top 10. And this advice is good for people who are building AI platforms, but it's often not good for people who are using the downstream tools. So there's all of these tools and this was from 2023. Um, you know, and if you think about it now, it's basically every tool in existence like all of them have AI built in in some

way. In this talk, however, I'm just going to go into this little slice here. So just enterprise search and my hope is that by doing that um I can kind of give you some some tools, tips, techniques that are applicable across sort of the wider market. So all of these things and more. And so we'll be talking about sort of understanding AI search tools to begin with. We'll do a deep dive on Gleam specifically and then we'll dive a little bit into MCP and we'll see kind of a little bit why they're related. So first we need to understand what AI search is. Um so the old search very standard uh you you go to all of your

things you do your search because who hasn't spent 30 minutes trying to find a document they last opened a year ago. And so the promise of AI search is that you just go to one place you can do your do your search and you get back your response. Simple. Of course, security people often think of this and it's just the increase of risk surface area. So rather than needing to compromise three tools, now I only need to compromise the session cookie for one. And so there's a lot of these tools. Um I've only listed half of them here. Probably since I made these slides a month ago, there's probably like double this amount. And I've probably

missed a lot as well. Like I think Slack was the most recent one maybe a couple weeks ago. And so yeah, like I say, at at Canva, our IT team and our security team decided to roll out Glean, but in previous roles, I've also used Guru uh and I spent a little bit of time looking at Atlassian and rovo in my personal time. So really, all of the stuff that I'm going to be talking about is applicable to all of the above. And so how does it look to an end user? Well, this is our sort of Canva world, we call it. This is our little internet, I guess. And so we built uh sort of a front end to glean into that.

So you can see someone's gone out, they've searched where is my onetoone template and it's given them like a like a confluence uh response back. And so that's what the user sees, but this is what actually happens beneath the surface, which is a little bit more complicated. So you saw number one and number six really uh you didn't kind of see number two, three, four, five. And so really what it's doing is it's going out, it's making a bunch of API calls, it's returning the responses and indexing them in a vector database and then it's using whatever LLM provider you select in order to sort of summarize that text um give it back to you in a way that sort of sounds natural

and provide it back in in whatever UI you're using. And so everyone has their own preferences when it comes to threat modeling. you know, there's tons of different ways to cut it. Um, this is sort of my generic way of thinking about how I do it, which is when I'm looking at sort of an AI system or any SAS system really. Uh, you've got your integrations. So, you've got your integration risk, which is what things am I connecting? How are we integrating them? Uh, you know, what service accounts am I creating? Are they in scope for socks and sock 2? All of that stuff. We've got third party and privacy risk. So, you know, do I trust the

vendor? Does the vendor train on my data? All of that stuff. We've got access and authorization risks. So, can I can Kane only access files that I should be able to? Is there any way around that perhaps? Uh, how do I interact with Glean? Do I do it through a phone? Do I do it through a laptop? That's going to be an important one for later. And finally, we have standard SAS risks. You know, where like, you know, who has admin access? Um, where are the logs going? Can we rotate access tokens? That kind of stuff. So, I would like to talk about all of these, but I'm going to skip these two just solely because um

they don't change that much really. Um like they're very similar to any other SAS tool that you'll be looking at. And so really, I'm just going to be focusing on the beginning and the end. So, I'm going to be starting with integrations. And the first thing I'm going to talk about is search fidelity, which is that not every data store is um sort of the same same level of quality. Connecting things like Slack can have side effects. And so when we rolled this out at Canva, we had a number of people who would go out and try to subvert glean in fun ways. Uh so there's going to be a lot of these during the talk.

I'm going to include them in each of the dividing sections. And uh they they get pretty weird. I'm gonna say that now. This is a very tame one. So, um, back on the topic of integrations, these are the three things that I tend to think about. So, it's, um, you know, the first one is, can I connect it? Like, is it in scope for compliance? Can I connect a Feder app? That kind of thing. The second is, should I connect it? Like, is it useful to people? Um, and the third is if I'm going to connect it, I'm going to need to investigate sort of the weird individual aspects of each. So, thinking of the first point,

these are kind of the things that I tend to think about. So, you know, is it supported? Is it in scope for my compliance? What data classifications is it? That kind of stuff. So, it's kind of the more more GRC stuff that you want to think about. There's also varying levels of integration. So an interesting one is Slack. So Slack uh you have a grid uh and then under that grid you might have different company workspaces, got private channels, public channels, private DMs. An interesting one is that you can you can connect up things like Glean to uh collect any of these. And so you know do I care about public channels? Probably not. Like it's

public data. Do I care about my private DMs being indexed? A lot of people say yes. You know, like there might be DEI channels. It might be, you know, me posting about my boss or something. Uh there's like various different things that I might not want potentially indexed. And so we're going to look into orth and some of the issues there. And apparently CTO now has a very different meaning.

So earlier I was explaining uh what these tools are and you know what you might be thinking like why can't I just do this with uh MCP or AI platforms today and the answer is yes you probably could but um access control tends to be the big piece like what these tools provide today is really access control and a UI like that's really the the core component of what they do and so when I said I wanted to investigate documentation. I'll be taking you through a bit of an example. So I used to work at Atassian and I kind of know where the skeletons are hidden in their ecosystem. So one of the things is that ecosystem apps in

Atassian are installed instancewide. So you cannot lock them down to a specific space, specific area, specific person, that kind of thing. So every time Glean does a search, what it's essentially doing is, you know, going to a confluence page and then checking that I can access that doc at the point of time. And so you're building or on top of or off. And you might be see you might see where this is going. One extra thing to think about is you're connecting all of these things with API keys. So you definitely want to make sure that uh you're protecting them well, you're logging who's logging into them, that kind of thing. like it makes the non-human identity problem just 10

times worse. And so here's a specific example. Uh I was basically I saw this this text at the bottom on Glean's bug bounty and I I was a little bit confused. So I decided to look into it. And what we found is that uh essentially there was a case where we were running Glean uh sort of in a cloudm model like hosted by ourselves. we weren't using the cloud version. So, we sort of assumed that all our API keys on our infrastructure, nice and safe, can't be accessed by anyone else. But no, uh that wasn't actually the case. Um the Atlassian API specifically had a weird caveat where they had been put on a proxy that Glean sort of connected to

our infrastructure. If any if there's any Glean customers here, that's actually been fixed. So, don't worry about this anymore. That's something we worked through with them. But there's like little weird things like this where you might assume you are secure because you're running your own setup, but actually you're not because there's weird infrastructure connections you need to make. Additionally, we found a few small handful of bugs in obscure scenarios. So, um, different SAS apps have different APIs and they might have different authorization APIs. So like a great example is that uh we found that if we if we archived a page, moved it, deleted it, and then restored it, the access permissions were wiped. And so in

those cases, Glean would be able to find it. But it wasn't really a game problem. It was more of just a weird confluence problem. And so there's a high likelihood of all issues. Like building orth is hard. Building or on top of orth is even harder. and building or on top of orth when the APIs change every day is probably one of the hardest things you can do. So going a little bit more into the zero trust space, there's an interesting way that you can kind of mess up your own setup. So this is something that a lot of enterprise security teams might do. Um you've got Confluence here which is protected and you need to use a

managed device to access it. Whereas Slack you can access it from your phone. Now what happens if you put glean on phones and you connect at confluence in you've just kind of bypassed all your stuff. So um it's very easy to break your own zero trust setups with this stuff. So you generally want to put glean in the highest security tier possible and then you can kind of negate most of those issues. So do consider what you connect and where you allow usage from. I put this in. I I don't really think it's a huge risk. Um really this this whole model increases discoverability, right? So it makes things easier to find. Now, if an attacker compromised

your systems, they could probably find it anyway. It just makes life a bit easier. So I would recommend you at least do some sort of keyword searching when you do a roll out to find, you know, stuff that might have been open from 10 years ago that you've kind of never locked down. Um so next we're going to talk about controls. And I'm not sure what wasp milk is, so don't ask me. Uh, so in terms of high level controls, you really only have three. You can run your own. Uh, you can run it on prem, one of those. Um, you can decide on what apps you want to connect and what level of connection you want.

And then you can decide where you want to enable access from. And that's like the the biggest things you should think about because that's just reducing your risk. Um and tactical controls are pretty limited. There's no like button to enable encryption. There's not like, you know, the secure button. So you can do some things like um detect API key usage from outside of your infrastructure and things like that. But I would say, you know, most people aren't going to do them unless you've got a lot of time and a lot of high fidelity signals that you can you can alert on. And um if anyone needs me to hook you up with James uh for some wasp milk,

I can always do that afterwards apparently. So um you might be wondering uh why talk about MCP in a sort of glean and enterprise search talk and you'll we'll get to that. like you'll find they're pretty similar. So, I'll give you the quick what is MCP? Um, which is it's basically a layer in front of your APIs that sort of turns LLM prompts into specific actions. So, you know, you say get me all my Canva designs, save them as JPEGs on my desktop. It can then go out, talk to the APIs, get them, do some file system actions, and then return them. And so it's all based on your API tokens and what what access you give it

basically. And this is kind of what it looks like to an end user. Um so there's a prompt which if expanded shows what actions it will be taken. You can do things like yolo mode where it just goes and does it and there's no actions. That's something I would recommend you uh stop your developers doing if you can. Um, but one of the things with MCP is that at the point of talking, it doesn't have authorization. So with that previous design, you needed sort of an agent on every single laptop in your company. And so you might roll them out with J or in tune or whatever you're using to manage it. But uh I find

that more commonly people are just going out downloading stuff. And so it's no different to just downloading random apps off the internet. Now what Cloudfare and a couple of others have done is build a sort of encapsulation layer around MCP where you can host a proper server. And so in that previous slide I kind of had server in quotes because it's running on the workstation. Um but with this sort of new design it is an actual server. You can either have like one server connecting to all your different SAS apps or you can have one server per each SAS app depending on what you want to do and what your risk threshold is and that kind of

stuff. And so what this does is it lets us vet um sort of our MCP server and add things like centralized logging monitoring. Make sure we're using one good version of it rather than a thousand different versions downloaded from the internet. And so there there are some pros and cons of both. Like I'm not going to say that like you should definitely centralize like the the bonus of the local design of course is that you can use local file system actions which you probably don't want to do on a server. So you do get increased flexibility. It lets you, you know, save CSVs, do stuff on your desktop, that kind of stuff. Um whereas with the server design, you do

get better security through centralization. um like you you don't have the problem of all these package manager problems that we're seeing with it and it does give you increased uh ease of use like it just makes it easy for people to use if you say just go here and you can do the thing like generally when it comes to security the pave path is you know the preferred design because we're making people's lives easier and so what I what I recommend when it comes to sort of threat modeling MCP and how to secure it is use logging to detect what pe what what MCP servers people are using today like in your company then security review them

provide like a paved path for those deploy them to workstations to make it easier and so then it's out on people's laptops you've maybe got an agent that can do all this stuff for them and so maybe they just interact with one agent we have one called Otter at Canva so people just interact with that rather than all this stuff and then finally you can provide upstream support and what I mean by that is you can sort of host your single server that connects to it and you know your life is easy but maybe you can't do that if you need the file system stuff we were talking about and you'll find it's a similar

pattern to enterprise search right like a lot of it is going out to different SAS services with this layer in the middle and connecting to it and responding to results like it's it's the same exact pattern I suspect like a lot of the enterprise search vendors are probably adopting this ASAP um because it's probably an easier way to do what they were doing already. Uh and I definitely think that you know you you could build your own personal enterprise search in a couple hours with cursor without all that authorization stuff for everyone. And so this is kind of how I frame it with the sort of server side design. You have the integrations as standard. You have the access layer

again, same threats, the zero trust threat, all the same stuff, absolutely no different. And of course, you do have the sort of privacy risks that we talked about sort of encapsulated over all of this. Um, so really a summary is that um when we look at glean um we'll kind of just go into that extra stuff, but um I did think this was an interesting one.

Um, so yeah, here's my three highle things that I think you can do to take away from this talk. So really, I think that every app is an AI app. Now, like if you are threat modeling SAS apps, you don't really think about it as an AI app and a non-AI app. You think about it of like what is it integrating to? Where is it being accessed from? If you think about it from this, you can kind of negate most of the risk just from these things. And yes, there's going to be specific threats. Like if you saw the talk before this where we were talking about things like yolo mode and stuff like that. Yes, that's very unique to

this one thing, but like I say, most of them are still generic enough across the across the whole stack. The next is just increasing connectivity just increases risk surface area. That's just a thing that you need to accept, I guess, if you're going to do something like this. Um, like things like auth data classifications, that stuff can help, but it's not going to save you. Like the risk is still there even if you do really good and all those things. And finally, I didn't really talk about this just due to time, but uh really just secure your service accounts because all of this is powered by service accounts, which you want to make sure are minimally scoped, that you have

uh good control over that kind of stuff. Like a lot of service accounts use what is functionally 1 FA like if a service account token is out on the internet it can be used unless you have API allow listing or oh sorry IP allow listing or something like that. So I did want to thank thank all the people at Canva who made this talk possible like it wasn't just me. Um this was like a mix of our IT engineering and security teams to build a lot of this stuff. Uh and I want to give a big thanks to the teams at Atlassian and Glean. Like I I've obviously talked a lot about the different kind of issues

we found, but I don't want people to think negatively of it because I think uh a lot of these issues are just like it's difficult to work with or it's difficult to work with upstream providers. Like if anyone here is from like Fanta or Drada and stuff like I'm sure they have the same issue where they're connecting in with lots of APIs, lots of different providers across the stack, that kind of thing, right? And so, um, they've been awesome to work with and actually like get each other to talk and fix some of the issues. Uh, I did have one last prompt to wrap us up, uh, as well. So, uh, I don't know if anyone else suffers from

this, but I definitely do.

Um, and I think I'm on to the questions. A little bit faster than anticipated. Cool. So, there's uh one on the slido. I can just read it out loud for you. Uh, any suggestions for monitoring endpoint MCP use detections, eg identifying ones that need review? Yeah. Um, you know, you use whatever preference you have. like you can use your EDR or your all or your MDM tools to like look for this stuff across the fleet. Um I don't think there's anything really good that I've seen like in terms of detection packs to find this stuff. I think we've been doing a lot of it manually. Um I guess that's an interesting one if anyone else has seen

that stuff or if anyone builds a detection pack for that kind of stuff. I know a lot of people use OSQuery to do searches and stuff across the fleet as well. So, I don't think it would be too hard to use something like that to find it, categorize it, see what people are using. I think, like I said, really the key is making sure that you don't have like 12 different Jira MCPs that are hosted from legit providers, dodgy ones, random guy on GitHub, that kind of stuff, you know. And I can uh run around for any live questions if there are any. There was one up there.

Uh so how does Glean tackle the problem of hallucinations? Does it provide references like perplexity to for the users to verify it's coming from the right sources? Yeah. Um I can go back a fair bit. Um I think you can see it in the UI here. Yeah. like you can kind of see it at the bottom where it will have um links to the documentation. We found that it's been pretty good most of the time, but it will like, you know, you you can ask like how long is um parental leave at the company and it'll get parental leave confused with like I don't know something else like sick leave or something and so it'll be like

yeah parental leave is like 2 years or something and everyone's like oh cool great. Uh but you obviously need to go and check the documentation first and that's kind of why I said like these tools are basically just access control and a UI and the UI is an important part too as well then uh is it something uh similar to what that particular uh SASbased product have a search engine as well right at last will have so basically glean is the search engine for that SASbased product. How is it compared to that particular SASbased product search engine? I I really hate Confluence search, so it's a bad question for me. But uh um yeah, it

really depends on the tooling, right? Um like I find that some some are better than others. Um I find at least in our case, Glean was better than Confluence directly. Um but it depends, you know, from my end slide there.

Anything else? Going once. Going twice. All right. Great. Thank you, Ken. That was wonderful. Thanks so much. Easy. Thanks a lot.