BSidesSF 2025 - The Hidden Access Paths to Smaugs Cavern (Ben Arent)

Hello everybody and welcome to Bsides. Uh it is my pleasure to introduce to you Ben Erant giving his talk on the hidden pathways to smog's cafe.

Cheers. Thank you. Hi everybody. Thank you for coming here today. Um it's great to present here in this uh cinema. I'd like to thank Bides for inviting me. So today's talk is the hidden pathway to smog's cabin. And I'm not going to touch on much dragon law, but just to set the foundation, Smorg was the dragon in JR Tolken's Lord of the Rings. And if you've read the books or seen the movie, he's the, you know, the dragon that loves treasure. And so he's commonly found in this sort of mountain of treasure. The horde, gold, jewels, other precious items. And I think Smorg and the Horde is a bit like some of the developers and SRRES I've worked with.

Once they get access to something, they don't give it up. But there's some things that are a little bit more precious than just the gold and the uh the jewels. And that's the Arkan stone. The Arkan stone within Lord of the Rings. This is an object of desire both for dwarves and smorg. Everyone else is super pumped about the uh Arkan stone. And I put the uh these Bitcoin cufflinks in my talk. I think uh this kind of represents public private keys which are more desirable than sort of standard public private keys and some things are more like the Arkan stone and I'll be using this reference kind of throughout my talk and you know since we're in this

great cinema and I would like to say that all of these characters and names are real some things could be changed um this will be mostly PG and uh I'm from England so there'll be no closed captioning but I'll do my best to talk American English so you can all understand. And so um this is myself. My name is Ben Arrant. I'm based in Oakland, California, uh up there in the hills. I work at a company called Teleport, which is a infrastructure access platform. Um but I'm going to go back and start this talk back when I first arrived here about 13 years ago. um in about 2011 and so in 2011 I arrived sort of um to that bang in San

Francisco and I'll sort of touch on this as I go through and so one thing I'm going to talk about is pathways it's also how these pathways can also result in backdoors and I have sort of two examples of lesser traveled pathways and potential backd doors that I've sort of experienced also going back to my background I come from a product slash um slight hacky developer And so I've normally circumvented whatever controls most security people have had in place. And um this is sort of part of my experience. And one thing I like to think about pathways is in architecture there's this idea of desire paths. A desire path is basically people will find the quickest way from A to B

regardless if the architect wants to or not. And so you see this in the physical world, but you also see this in uh development teams, people coding, you know, hackers, people will just find the quickest way uh and the desire path. And so going to 2011, I land here. This is the stroke building on 9inth and Mina. Um here's a much younger version of me. And when I first moved here, we acquired a few small startups. One called Exceptional, which is a exception tracking tool. We acquired its competitor airreak. And also like say like uh exception tracking tools are an interesting SAS product because people install a bit of code into their application to capture exceptions when

the system crashes. And if you misconfigure this, you can also send all of your secrets to that third party exception tracking tool. And we'd commonly see people send their like mail gun keys, stripe keys, even AWS credentials into our tool, which was a huge nightmare for us because we had to like then report to them that people are sending all of their sort of secrets to us. Uh I was around Reddus to go for a period of time. Uh this was a Reddus as a service product. Uh one thing that's also interesting about Reddus to go at this time uh Salvatore who was the main author didn't believe in that Reddus should be made available on the

internet. So he didn't implement TLS until I think Reddus 6. And so we had like a reverse proxy in front of it. We would also see our customers put their Reddis keys in GitHub and various places all the time. And so one pathway uh to kind of get your job done is the admin panel. And for anyone who has uh worked on any SAS app, this is sort of a common hidden view from the end users. You build this um slashadmin dashboard. It might be in a separate app, but it's a collection of accounts users subscriptions billing. You have your account information, sort of where they are, how active they are, how much they're paying you. But you

often have this too, which is like login as user. And this is some form of impersonation that you might use for um debugging, inspecting, figuring out what's happening to their account. And you know, since this was 13 years ago, I put a poll up and I was like, hey, are people still logging into people's accounts on their behalf? All my SAS friends. And you know, still 60% of people are logging into uh accounts of their customers. Luckily, with a bit more auditing, um but also with some limited logging. And I'll come to the 22% which is probably the better way to go which I'll sort of close for remediation is the most secure way for

this pathway is don't provide any access to your customer accounts and thus a malicious insider or another user who hacks your system they can't get access to um those system accounts. And another one uh we saw was you know using production secrets locally. Um this can be for a range of things from um your application crashes, you need to like break CI/CD, you might use take your production secrets and use those to push your code uh for a break glass fix. You might use a production uh email key to send emails on behalf of something else. This is sort of a pathway to kind of get your job done. But uh I think right now there's a talk from Travelhog.

We know that these sort of credentials and secrets uh when they're locally um are ripe for hackers to get access to and become a back door into your system. And so endo act one you know it's kind like I made this pathway to get my job done but I've also made the back door. And so for act two I'm going to talk about understanding uh user behavior. Um, for this meme, you know, we have the uh happy path, the great castle, which is sort of the good path and then you have like the sort of bad evil path. Which pathway do you take? And this is a story that I am reciting from a CISO I talked to earlier this

year. He had somebody join their organization. You know, he had slightly broader roles and duties than he should have, but you know, it's kind of fine. He needed to like code, develop, build this application. And so he starts going down this path and he's like, "Okay, I can push code. I can do my job." But he realizes that he can edit background jobs, which you know, you can normally edit background jobs, but this is kind of where he changes his pathway from like the good pathway to the evil pathway. And he found that he could edit the recipient field in the uh scheduled reward jobs. And then finally he had sort of excfiltrated almost 250k from

this organization just through a basically a salami attack on editing these background jobs. And I think the thing that was interesting about this is this organization had nearly every security tool you could but it was finance that finally spotted it because when they reconciled their books he had hadn't changed which campaign it was. So they were paying on a 2023 campaign when it was from 2024. And this was sort of the beginning of their investigation. And I think this highlights a pathway that um you can get access to something which can also uh potentially go wrong. And you might think, hey, isn't this the plot of the 1999 hit classic office space? And yes, it is. And I looked it

up. It's 25 years since office space was uh created. And we're sort of still seeing this slam attack in the wild. And I think this sort of highlights that um some of the pathways can also open up back doors. And so um coming next to sort of monitoring of access, this was a uh restroom actually in San Francisco. Um which do people need access to the restroom all the time in a coffee shop? I would say yes. But in this coffee shop, you have to have an app to use this restroom. Um did this person need access to the background jobs all the time? Probably not. Um you know, was he doing off hours or were some other

patterns that you could have recognized? And then lastly, is there such a thing as too much logging? Probably if you're logging your restroom, yes. But I think when I talked to the CISO about this case of the person excfiltrating all of this money from the organization that it was really hard for them to go through all of their seam and figure out what was happening. And I think this kind of comes back to the concept of the arkin stone that there's one thing that's very sensitive that they could have done a better job of sort of monitoring. And so um one thing I've talked to a lot of people I work with is sort of

visualizing the graph of access and change over time. Especially with infrastructure getting increasingly complex, you know, someone might join an organization, you on board them, you add them to an octa group. Hey, you're the surre. But you don't necessarily know the full pattern of access from, hey, you've joined an octa group, you get access to IM role, the IM role has an instance profile, which can also have a level database to something that you don't know. And there's also these sort of semi- invisible access chains that you may not think about. And um also seeing how does this change over time. And so um another thing about understanding user behavior is uh behavioral analytics trying to see hey

what's the normal pattern path of access that people normally follow people doing something out of the ordinary. Um I've talked to lots of people um on this topic about what time of day. Um it seems like most malicious insiders do things sort of after hours on the weekends changing uh dockets. I think um a very simple one is like when running DB queries is the scope appropriate. There's performance reasons you might want to do this but there's also um potential PII um and then I think auditing what people are doing for their database queries and then probably limiting their scopes. Uh how often does somebody use access? Um, I've also worked a lot with Cto's who configure all of the access

and they're basically the one person that has root access to everything but they never use it. They're also a prime target. You know, they're like on their about page. So, um, also removing access for people who don't use it at all. And I think lastly, um, for behavioral analytics understanding is what happens during one-offs. Um, most of uh most people I work with also talk about uh break glass procedures that they might do once a year or tabletop exercises that might trigger off a whole bunch of alerts that go outside of their normal scope and um just seeing what happens monitor and then using that rest of the year. Hey, when somebody has this break

glass procedure or someone's doing a one-off, what are they doing? Maybe building tooling in place. Let's say you're doing uh database migrations or if you are um doing something that outside of the ordinary. So ended act two understanding uh access behavior and it comes to act three which is remediation. Um I'm going to shout out to Oakland. This is Jack London Square. So across the bay if anyone uh everyone want to take the ferry across it's a lovely part of Oakland to go visit. And so I think first thing is understanding the scope of the problem. Obviously there's a range of things within your infrastructure admin dashboards as I start at the beginning there's infrastructure there's clouds

there's users or different permissions and then I think identifying hey what is your arkin stone within your organization in the case of the uh attack I showed you that could have been the reward jobs or payments um but this can be a range of things it can be um various keys and setting those up and sort of working backwards to sort of make the problem more manageable um I think next up is also setting up non-negotiable controls and security invariance. Security invariance is something that's always true to keep a system secure. I think one of the most basics one is you know mandatory hardware MFA if you give everyone on your infrastructure team UBI keys. It

fixes a whole slew of fishing attacks. Um basically stops them from happening. Um zero access to user accounts. Going back to my original story of the uh admin dash dashboard to provide access. Do you really need to provide access to your users? I know uh last year we saw the octa hack which a third party had support position. How else can you build tooling for support to provide access to their account without providing literal access to their account? Um if you're on AWS, you know that IM can get complicated. Um SCP policies are a great layer that you can add in addition to what IM already provides. Um it's great for some network segregation. um and a a good thing to

investigate and then I think automating what uh these invariants saying hey we want to say that this is always true one another example would be don't let people create uh users in AWS uh just use um IM identity center just make sure that you they can't create them and if it does alert and report on it and then I think there's always blind spots in the system um that stops the pathways um identifying what um sort of build systems that stop those pathways. So in the case of like the support setup, build a tooling that provides the same thing but doesn't provide direct access to their accounts. Um coming up on to monitor, I think monitoring what people are doing

on those systems is also important. um understanding their behavior. This can be sort of tricky especially as organizations gets much bigger but what's the standard pattern of flow? Who's accessing which infrastructure when? What is their behavior? Um do people go through pathways? Uh are people touching systems which are outside of your standard monitoring? You know I think um doing red team and tabletop exercises are a great way to understand hey I've bought this very expensive monitoring tool but I don't capture all this in my tool. um thinking of other tooling that you can get in place to monitor those. And so that's end of act three for uh remediation. Um and I'm cut through this pretty quick,

but I will I'm on the happy pathway now. And I think just to close, I'd say, you know, make the secure path the desire path. No matter what you kind of put in place, people will if your security tooling is tricky, most engineers will normally find a way to circumvent it. Um but if you make the secure path the happy path and desire path um people will go through it and I think also it's good time to review is the standard way of doing things um like deploy basic security invariant um you know only add hardware MFA set up SCP policies try to limit what these possible pathways and back doors could be so I'd like to say thank you everyone

for coming here today I also host a podcast called access control I talk to uh leaders in their field about um keeping systems secure and what's the best policies and practical tips. Um you can just search access control on any podcast apps. And I'll also be hosting an identity security happy hour this Wednesday um from 7 to 11. Uh you can scan it here uh dub.shmog and um you know a lot of these kind of cover in identity security. Um, this is sort of a lot of the research we're doing at Teleport, but I want to kind of keep it open and sort of a discussion. And so for that, I'd sort of open up to sort of Q&A and um, some

questions. [Applause]

Any questions in Oh, yeah. Have you been training any models in the modelification? Yeah, that's a great question. Yeah, we have started to um Ben, let me interrupt you real quick. Uh we are streaming this, so when we do have questions, please repeat them in the mic. Uh please continue. Uh the question was, have we trained any models using AI to sort of detect and learn on these things? And yes, we have started to uh do this. I think it's an interesting kind of like big data problem of having all of especially like cloud trail cloud cloud trail has like 17,000 different event types and I think that kind of goes back to like the

arkinstone concept there's probably uh a few things that are more critical um one common one is like KMS keys HSM people taking encryption keys um is the thing that people seem most worried about and kind of working backwards but I think it kind of depends upon people's organizations and would like to like fine-tune the model and ingestion uh to solve that. Any other questions? So, while we wait for questions, I'll remind everybody we we do take questions via Slide View. We are currently in theater 7. Uh so, feel free to submit your your questions there. Um and if anyone has questions directly, feel free to ask them. We'll just repeat them. I see one over there in the third row

here. I'll bring the mic over for you.

So a variety of products have ghost users for employees to see and visualize as the customer is using. It's like a clear valuable use case. Um totally empathize with the security perspective. I was wondering, you know, if we're talking about happy paths or desired paths, how can you solve that use case, that workflow with a more secure alternative rather than, you know, just killing core workflows that people need to get their job done. Yeah, I think um on that spectrum of sort of monitoring there's also a whole range of like you know I'm in product management product management tools like full story which can like record people's sessions and see what they're doing but I think also when you

investigate these tools they are tools for like blare like um hiding secrets and I think especially if you're building tools for developers for security you probably have a different posture than other tooling um and I think it also becomes difficult for um account recovery too. I think especially if you have like a non-technical user base, people aren't going to necessarily print their recovery keys and then if they're fully looked out of their account like what's your account procedure and I think that's just um saying hey how much information do you have like what's the risk and the exposure and um kind of going backwards on it. Uh Ben, we got a couple in from Slido.

Uh the first one is from Seth. Uh he asks, "What is the secure alternative to ghost users for simulating user experiences?" Oh, yeah. Same question. I'll just move on to the next one. Uh one from anonymous uh requester. Uh how do you find Arkinston? Uh I know that for the IT department in my org, we're not thinking about our financial systems at all. We're thinking about systems that are customerf facing. Yeah, it's a great question. Um you know, we call them like Arkinstones or crown jewels. I think in the case that I showed of like the background job that someone edited for a reward payment for like 250k, it's not obvious that that, you know, this is

kind like a true salami attack over time. Um, you know, I think that kind of goes back to like permissions. Do you always need like write permissions? Can you just have read permissions? Can you give people like minimal access and then build on top of it? I think another way of identifying it is figuring out things that people rarely pull in. I know uh Facebook I think has a very sophisticated tool for figuring if you look at like your ex-girlfriend's profile and uh they go very deep on like the kernel level inspection because anyone can access their database and for them that's kind of like the ark and stone is like hey employees shouldn't be

able to access um user accounts and so I think it's just kind of what the organization is and what's the biggest risk patterns. All right, we had a couple more questions come in. Um, uh, let's see. Uh, next question, anonymous. Is there a more secure alternative to using a traditional SAS admin panel for customer management? Uh, yeah, there's a a range of tools. Uh, I mean, people kind of have the homegrown, they build it themselves. Um, depending on the programming language, it can also um differ. I know like Ruby, Django have specific ones. And I think they've got better over the last 15 years for authentication. But I think it's also good to have if you have these admin

panels is do you have strong second factor? Do you have strong recovery? Do you have good logging? Um and I think this is the range of tools out there. You probably look on uh online or GitHub. Um I got more of a comment than a question. Maybe you can respond to it uh from another anonymous person. Uh my suggestion for ghost users is to build a customer authorization tool that requires explicit customer permission for session impersonation or access to PII say by change uh challenging UX uh but doable. I didn't have much more of a comment on that one. Sounds like a fun project. All right. Well, we uh have a few more minutes. We have the room until uh

quarter till if anyone has any other questions. I'll keep looking at Slido. Um, but again, thank you to Ben for the wonderful talk and uh teaching us a little more about security. Cheers. Thank you. [Applause]