Heard you liked access, so we built Access to manage your access for Access

hello besides uh it is my pleasure to introduce to you uh Mr Peter Collins and uh Alysa Gant uh they're going to be speaking than again all right all right cool all right so yeah Welcome to our talk uh titled heard you liked access so we built access to manage your access for access it was yeah that's hard to do um it's mostly uh we're going to be talking about our path to building internal uh new access control system at Discord um for employees so let's get started with some introductions uh I'm Peter Collins Senior staff uh security engineer at Discord I work on cloud security and identity and access management on the platform security team my focus is on

writing software to pave paths that make the secure way the easy way I strive uh as a senior staff security engineer to build a security culture that is transparent scalable empathetic and positive and hopefully you'll see how we integrate that security culture in our talk today um I'm Alisa I'm also a security engineer at disord I work with Peter on the platform engineering team security engineering team um so far I've mostly worked on Mal prevention and access management um and my goals through work are just to help make security intuitive and computers and applications safer for users um so as just like a quick overview of our talk um we're going to discuss kind of our path to deciding to

build a new Access Control control system go over some of the design that went into it um then highlight a lot of the main features and then discuss the tech stack and the impact so for starters why are access controls needed in the first place so according to Verizon's 2023 data breach investigations report 74% of all breaches involve the human element within that privileged misuse and stolen user credentials are two of the primary threats as part of business and security maturity this is something that we need to address and set up plans and policies to control access to sensitive data and resources one way of doing this is by making sure that users only have access

to the specific data resources and applications that they need to um complete a required task cool so now that we've talked a little bit about why internal access controls are necessary let's talk about some of the goals we had for internal Access Control at Discord um what are our really guiding principles um so the first we had is we want this system to be intuitive um so what do we mean by that we want it to be easy to use understandable uh so that we can show current and historical permissions and allow for people to easily uh request additional access uh we want this system to be usable by all employees not just a technical Bunch like it or security

engineering or generally engineering um so because of that we want a web user interface that people can easily look up and intuitively find what they need uh with access control permissions and not necessarily um use a command line utility or requiring poll requests via some giops flow the next goal we had for our internal access control system was we want it to be transparent and discoverable um essentially we believe that employees solving their own permissions uh issues safely increases business velocity and Security in other words we believe unblocking people to do their jobs quickly enables the business to move faster um so let's break that down a little bit what we mean by transparent discoverable um so instead

of entirely relying on a centralized team like it or security to solve certain questions uh individual employees can now answer for themselves what permissions do I currently have what permissions does my teammate have that I don't did I recently lose access to something and what was it and how do I request additional access and not just for an uh individual employees we want to solve uh these similar questions for uh team leads and managers so such as how do I give access to a new team member and how do I give temporary access to an individual for a cross functional effort on my team temporarily um yeah next goal we had um is we want this system to be uh

centralized we want a One-Stop shop uh to replace all outof band access request mechanisms whether they were historically uh it help Des requests or requests in various help chat channels or random emails all these different mechanisms are really hard to audit later and determine why somebody was granted access to something and also uh determine exactly you know what Grant what determine if a user or a team actually needs that access um we also wanted this uh system to integrate well with our identity provider for single sign on and in our case identity provider is OCTA um we want this intuitively to work with uh OCTA as a way that people can both when they want

to log into a single application they go to OCTA and if they need to be authorized for that application they go to this uh One-Stop shop um the next thing with uh centralization here is we want it to become more useful with increased usage and essentially this means is Network effects as people more uh use this uh functionality more and more um they will go to it in the future and it'll be a self- enforcing uh utility for that the last goal we had for the system is we want to be secure so what do we mean by it security um we want it to be leas privilege and audible um so in terms of lease privilege we want to be

able to provide a top level role-based Access Control mechanism and uh in other words we want to provide common workflows with a standard set of permissions and we'll talk a little bit more on what this means in a bit um but we also want to be able to enforce uh time-bounded expiring access to be able to WR size permissions so that unused permissions will be automatically removed uh later on uh as we want this to be audible we want to be able to audit both current and historical permissions to easily determine if permissions are recently lost if they're expiring soon and that sort of thing and then also Discord like many businesses uh and companies out

there have uh compliance goals that we want to achieve and so we want the system to be able to achieve and exceed those compliance objectives to make our Auditors happy so as Peter mentioned we're interested in role based access control so just as a quick overview of what that is role-based access control is the idea that permissions are managed based on users with shared job functions as opposed to team or any other criteria this makes it a lot more clear why someone has access to certain resources it also makes it so ramping up on a new project onboarding team transfers anything like that is much easier so as just a quick example of this we have a

very simplified team with a manager a software engineer and a data scientist and on the far right we have a set of resources like email a Google group for managers GitHub engineering documentation stuff like that so in the middle we have it set up for different job functions so all employees should have access to email whereas managers should have access to the manag to Google group and Engineers should have access to GitHub and Engineering documentation so as an example the manager is given the role all employees and manager which gives the manager access to the appropriate resources versus the engineer is given access to all employees and Engineering which does the same thing without providing without

granting access to too many things so Peter also mentioned that we're interested in top level arback a lot of services offer the idea of have the idea of roles integrated um but those don't propagate across services so we wanted to be able to um do it at the level of the identity provider so that we can set up roles and manage them in one place and then they'll be used everywhere Um this can be achieved generically with nested groups and group rules um but that can complicate group management a lot it also um decreases transparency because if you have a nested group and a nested group and a nested group you don't really know who's

in it anymore and it's really hard to manage in that way okay so now that we talked a little B about uh The Guiding principles and the goals we had for this system uh why did we decide to build a new tool instead of you know buy existing one or use some open- Source uh option so we took a look at the field and we did find quite a few uh uh SAS and open source sols that were available um but we found some deficiencies and actually some boxes they did check for us so first off uh some were compatible and well uh well inrad with OCTA our identity provider and this is pretty important because we

weren't planning to move between identity providers um some were compatible with other uh services and identity providers out there like ldap or other directory Services um such as you know entra idid or something like that um but we weren't interested in those uh most were compatible with Enterprise chat Solutions um not going to name any names but we don't really use them at Discord we use Discord for Discord to make Discord so uh they weren't really options for them us but a lot of them did allow custom integration so we could have made it work um for those at least um the next one is uh few did really provide a view for employees to

solve their own access issues they really uh didn't provide a Way Beyond administrator panel to see you know what permissions I currently have as a random employee at this company um along those lines uh few really did uh support ro-based access control at a top level feature which meant a lot of manual managing of users into individual groups for different applications and there was no way to uh send role-based Access Control uh at the top level with our identity provider in a way that was intuitive um a lot of them actually surprisingly did support some form of time bounded expiring access which most identity providers don't provide this internally but a lot of these out of-

band tools did actually support that so it's green for that reason um really you know good job on that part which is great um however on the other side uh few supported delegating administration of groups in an opinion opinionated way that was intuitive for people to use a lot of them were really focused on some sort of centralized it or security team managing permissions and weren't able to delegate those permissions out easily to other uh team team leads or app owners um and also some with that supported uh routing access requests to the delegated administrator um but a lot of them weren't done in an opinion way uh opinionated or standardized way that would been an intuitive setup or made

sense for regular employees so why did we build it ourselves well a lot of the options we found were compelling none of them really checked all the goals we were highly Valu especially top level arbac uh transparency discoverability for individual employees we really wanted to speed up the company by providing that transparency uh an opinionated standardized way to delegate Administration and access request routing that would be intuitive for employees and team leads and managers to understand and we did want some sort of integration with Discord for notifications but we did think we could build that ourselves with many of these Solutions so now that we've discussed our path to deciding to build a new tool

um we're just going to get into some of the foundational features so for starters we have users users are employees of the company um here you can see an example from access which is what we named the app um so on the far right you can see who the user reports to some details like their title but more importantly you can see what the user owns and what they're a member of these are all going to be groups so we have three types of groups in Access the the first type is like a vanilla Standalone group this is the initial group type that's used in a migration to use the access app it has a

little bit less utility um after the initial migration the second type is RO groups so if you imagine that arback diagram from before this is the central column um these are uh Ro groups can be members or owners of other groups um and they represent a job function and so then the final group type is app groups app groups are like kind of that mapping to the far right resource column of the arback diagram they can be tied to certain permissions within an app so apps are essentially just a set of role groups or a set of app groups app group apps can be um tied to any thirdparty stas application first-party service essentially anything OCTA

compatible um which means that you can manage a lot of things all in one place all right so now that we've talked about some of the foundational elements of access let talk about some of the unique features and how they accomplished our goal so uh first up we have direct user assignments versus Ro assignments we've talked a little bit about role based access control but when I go into a little bit more depth here um so essentially you can both directly assign users or roles to a group um in Access by default uh you can do it directly so in this example on the right you have uh the Phoebe Discord character who is directly assigned to the group

fund Squad and gets the permissions of the group fund Squad whereas via role-based Access Control uh the group characters on the left wus and Clyde are part of the role uh Discord characters and so they because they are members of the Discord characters role they are automatically given uh membership to the group fund Squad themselves um use direct assignment when you're migrating to axis when you first start out without any roles um but as you start using access more and more more role-based access control um the direct assignments hopefully will taper off in their usage next feature uh hopefully relatively straightforward uh access requests we wanted to be able to provide employees a way to both uh request a access to any

group for a specific duration whether they want to be an owner or member and then also provide a reason uh next feature is uh group and app ownership um so essentially we want to be able to delegate owners uh who can then manage uh each group or app what what do we be my manage uh they can manage the membership or ownership of that uh group or application they can manage the title or the name or the description of that application but I think most importantly they can approve or deny access requests and this really is the the main piece of it they can uh lead to faster resolution times because we're not uh centralized on a

single it or security team approving these requests and for security this helps reduce the risk of confused deputies so what is a confused Deputy well it's an administrator with little context about what permission grants when approving uh or for denying an access requests which often leads to a lot of back and forth trying to determine if a user actually needs that permission or who that user is um and eventually this fatigues the administrator and they rubber stamp most uh access requests uh if they've seen them before or if they're a common thing and they don't actually know the underlying permissions they're granting so we want the opposite we want an individual with most context about a

specific group uh they should be able to decide who gets permission uh uh and what is a uh in this example we we want team lead to be able to approve Ro requests and uh service owners to uh for application requests yeah the next feature uh very straightforward um time bounded access we can grant access for a limited amount of time and after that set amount of time a user is automatically removed from a group um this is useful for periodic access reviews uh for compliance requirements uh so let's say you have to review membership of a group every 90 days for certain sensitive groups at your company um well you can do that implicitly by setting the

membership of that group to 90 days and then the individual in that group can either request access additionally um or the owner can uh renew in a bulk fashion all the uh membership of those groups or else implicitly uh that uh membership will be revoked um this also does reduce the likelihood of accumulating unused permissions um so as seen in this uh di diagram below uh on the left without access uh expiring Access Control in a access control system you have uh over time an employee will gain permissions whether they get promoted or switch teams or are in a cross functional ort uh effort temporarily um whereas in a system with expiration um those unused

permissions are automatically removed after 90 days or maybe a year uh depending on what you set for the expiration of that role or other uh cross functional effort um so they're really right siiz to their job as they uh work at that company so the next feature is what we call audit views they'll show historical permissions um for users groups or roles um and to help with our goal of transparency and discover uh discoverability they're available for anyone in the company to be able to view they're useful for troubleshooting historical changes to access and for recordkeeping and anything like that so here's an example of a user audit view in this example the user is named lock

um and you can see that you can sort you can filter you can go through but it displays all of this users access history throughout the time that they were integrated into the app um you can see any sort of membership justification that was provided you can see access duration and you can see who added them and Who removed them the next feature is is group and app taging so theoretically this can just be used to label similar groups and just have a label on them but more interestingly I think is that you can apply constraints to them so these constraints can be an owner a membership time owner or membership time limit they can enforce requiring justification for

Access and for more sensitive groups you can make it so that owners can't manage their own access so here's an example of a tag page so as you can see in the first section you can see all the constraints that are currently applied in this case requiring membership justification and a 90-day membership time limit and then below that you can see all the apps and the groups that are Tagged so an important thing to note here is that any apps that are tagged since apps are just collections of app groups um that tag will propagate down to all the app groups within the app that said individual groups can also be Tagged so in some cases you might not want to tag

every single app Group within an app you can just go through and tag them individually you can also tag roles um so the next piece is expiring access in bulk renewal so without before we had this piece um you'd have to go through every individual group and user page and scroll through the list and try to figure out what's expiring and that was just really inefficient it caused a lot of people to lose access um because they didn't know that anything was expiring to begin with so we created a page that shows all this all in one place across the company that anyone can view um it can also be filtered so that owners can see all the access for groups

that they own and individuals can see all of their own expiring access for individuals um they can create a new access request directly from that page um just to try to ease the use of the app um so here's an example of an expiring access page for owners um so owners can go through and they can it really highlights when a group is incorrectly owned and then otherwise they can go through and filter and select which um users that they'd like to renew access for and then they can do that all at once um so here um an owner can go through and select all the access they want to renew in one place um and if any constraints are applied to

groups like some sort of time limit required reason it'll be displayed and enforced here the last feature we want to talk about is notifications um there's several types that we have implemented um and these examples are from Discord because we use Discord for Discord for everything um but it's the notification system is set up via plug-in so it could be made compatible with email or any other chat solution so the first type um will alert owners of access requests made to groups that they own after decision is made the individual who requested access will be notified of the decision and then we also have expiring access for owner ERS and individuals so owners will be

notified when individuals or roles are losing access to groups they own and individuals be notified of their own um expiring access so this uh notification system generally just helps with adoption of the app and avoids business disruptions due to Lost access all right so now that we've talked about a few of the features in Access let's talk a little bit about how it works uh under the hood so yeah we have uh essentially it's a python flask app with a react typescripts material UI backend um it's a single page front end app with a rest API and a relational database in production we use postgress and in development we use sqlite um you could probably use a different database

if you wanted but that's what we chose um in Access uh group changes are synced uh on demand and periodically with uh with OCTA um on demand for most actions such as adding or removing a member from a group but periodically for um as a crown job uh for expiring access so if a user lost access in the last 15 minutes and we run this job every 15 minutes uh we should remove that access from OCTA but we also run it periodically to sync the state from axis to OCTA so that uh if there are any rate limits or API errors that we run across they're automatically uh resolved and overwritten uh in case people you know

lost access or something like that um we did as mentioned we do have a notification system um sorry a plug-in system for custom integration such as notifications so uh we provide a templated interface so you can use any python uh code that you want to write uh for talking to different chat providers or SMS or email and then also we do provide a plug-in interface uh that's new for conditionally approving or denying uh access requests automatically based on any python logic you want so you could call out to another service or you could you know hardcode a static group or role or something that you want to automatically approve um or for a specific amount of time um and this this

whole system is packaged in a Docker container and can be deployed anywhere where Docker containers are deployed we deploy ours on kuties but you could also use Docker compose or something uh serverless so we've been running axos internally at Discord for the past year and uh what was really the impact for us um well it's been pretty successful um user access reviews take hours and not days there's significantly less back and forth and cat hurting via spreadsheets um access requests are now centralized instead of a whole bunch of different outof band uh ways to communicate and request access uh from it or from various engineering teams it's now all streamlined through one system um and

this has also led to less rubber stamping and Confused Deputy problems because now access requests are routed to the most appropriate individuals such as team leads for roles or um application groups for uh service or to service owners for application groups um we found that this helped us uh reduce highly privileged access permissions because for uh for sensitive critical groups um we were able to easily audit why people had access to those and then set a time limit uh if we didn't want standing access for those so that they could request in the future um yeah and that's how helped us be more secure so this brings us kind of to the whole reason for having this talk besides just

showing off uh what we've been working on access is open sourced um so here we include the links for the GitHub so you can see the repo um we have a server set up on Discord um so you can submit feedback ask questions anything like that um and we also published a blog post a while back um and that features screen recordings so you can also see the app like in Action a little bit so thank you so much Let's do let's get on the back wonderful talk from our speakers uh we have a Q&A taken through slide do uh you can get to the Q&A uh question prompt at bid sf.org Q&A I'm going to be reading off ones

that have already been submitted first question uh how do you efficiently manage ownership of groups and apps within the company for all tool and or changes yeah so um role-based access control is typically how manage org changes um we can easily split or merge different roles together um but we did want this to be flexible so we don't uh rely on uh per setting permissions directly on the name of a role group we set it on the app groups themselves and so we try to orient around that for different uh or changes I think this sl's distracting okay next question uh how do you handle apps that do not have native samel or skin features available

yeah so that was one thing we didn't really talk about too much in the in the talk we do rely on OCTA for syncing all group and member permissions to Upstream applications access doesn't do that uh built in um so yeah there there is a there is a gap there with uh sample or sorry skim support for most applications a lot of the critical apps out there today support skim for syncing group membership but that is one thing we're working on is uh supporting it for future applications via you know contract negotiations or something like that um but also maybe building in ourselves as a plug-in feature into access uh when features don't support saml or oidc I mean ideally it's not an

SSO tax that we have to pay for but um that is typically a contract negotiation thing that we want for an Enterprise solution that we're using at the company next question from Tom uh for the role assignment to a group does the role map to an OCTA entity or is that entity in Access only yeah so all groups in Access are maap to OCTA um the the idea of roles and app groups and everything is an abstraction that's kind of one level up within access um and that helps like if you're in a role and you're added to group that all the members of the role will get added to the group but um yeah

everything Maps talk to groups all right next question from Gabe cool tool the name access makes it almost impossible to search for it by name are you considering any alternative names we definitely gotten that feedback before um not this time but all right uh next question from Frenchie do you accept PRS and if yes how extensively do they need to be tested well Frenchie's already given a PR and yeah we we do accept PR I mean definitely hit us up in the Discord uh server if you want to chat about you know submitting uh any contributions but uh Frenchies PR is hard to test so I I don't know when we're going to merge it all right uh next question any plans

to consider other attributes Beyond roles for Access approvals group membership or expirations on call status device State geolocation Etc haven't really thought about that sorry but like this is it's an open source project it's very extensible and we do plan to build more plugin interfaces for other extensibility as well as built-in features so definitely open to it is it feasible to uh extend it to support idps other than OCTA I would say yeah um access is not a you know it's not an identity provider is more of a UI for a group directory service um so there is actually a very easy way you could um the the integration with OCTA is actually just a

single python class and you could rewrite that to be a different uh group directory service if you wanted to such as you know enter ID or ldap or something else um we do uh if that is like a goal that somebody has we could probably build that as a plug-in interface um so it's easier to swap that that in and out and not just use OCTA uh last question do you use a different security posture from this app when compared to other apps um assuming the app is being breached it would be super bad hey I might have the follow-up question on that one we talk in the back if you're here but uh I don't know what

uh exactly is posture the details of that but it's fine so that's fair all right that's all the time we have for Q&A uh our speakers will be available upstairs in the mezzanine if you guys want to follow up with them um please give them a round of applause for this amazing talk