Slack App Security: Securing Your Workspaces From a Bot Uprising

alright folks the next stalk is slack AB security thank you so much Kellyanne for coming hello all right we're on thank you everyone welcome to slack app security defending your workspaces from a bot uprising oh no did my computer just fall asleep cool security hold on let's try this once more with feeling hello everyone welcome to slack app security defending your workspaces from a BOTS uprising clicker does not work ok a quick note when I joined slack I was really excited to dig into the problem space of app directory security as app ii generis we live for a complicated problem with lots of moving parts gum in the gears and grayscale ethical quandary x' it's kind of our

bread and butter um Nikki Brandt who I'll talk about in a minute she and I started having lots of conversations with peers from other companies and we found that lots of us are struggling with similar issues so we put together a talk at apps at Kali to kind of illuminate that path guide the journey and go deep with other apps X I'm happy to say that we have a little more clarity about where we want to go but I'm bound by some legal restrictions that mean I can't commit to any specific path in our talk but I'll share some thoughts so I'm Kelly I'm a previous pen tester I'm currently a product security engineer at slack

I'm a former eco pirate I lived on sea Shepherd chips for four years if you know what that is somebody does I've battled poachers in Antarctica you can see in the photo you can tell him in Antarctica because I'm wearing a funny hat in a Mustang suit one of my proudest life accomplishments was crafting a media strategy that forced the New Zealand Prime Minister to hold a press conference denouncing the Japanese whaling fleet he looked completely miserable to be do not so just how her all clear I once made the head of a nation-state do something he didn't want to that's a gigantic power flex and I know that I've peaked in life this is Nikki Brandt

I do not her she wrote this talk with me she's a product security TLM at slack she used to program mainframes and she is a total badass she can't be here today because she's currently eating gelato in Rome I think she's on the back of a Vespa scooter at the Trevi Fountain like she's living her best life right now our agenda today is what the slack platform is what the slack app directory is what it is and isn't like and how that impacts the security of apps in the directory what we have tried in terms of securing those apps and the lessons we've learned and what we could try in the future so what is the slack platform

why does slack have apps isn't slack an app let's start simple how many of you use slack it's a lot of hands how many of you use apps in your slack workspaces so that is almost all of you slacks mission is to make people's working life simpler more pleasant and more productive so while everyone at your company might be using slack is the primary tool connecting all your people and projects the tools that you use as an individual to get your job done can vary greatly from person to person so for some of us were in github all day for others it's Zendesk for some of us it's JIRA and some it's Salesforce communication is the core way that our

work is effective so whether you're an engineer or designer an intern or an exec probably most of us are in the security space here we're using slack to accomplish these grand tasks that we've set upon through communication but typically our work is fragmented and siloed across all of that software services apps browser windows with a million tabs open email chains that might go on forever and there are thousands of tools that enable us to do our job on average companies have about 1,400 cloud services running within their org so that's where the slack platform comes in streamline and centralize that work and we have a really easy to build with set of API s and lots of developer guides so

anyone can build apps and integrations into slack show of hands how many of you have built a slack app yeah so a few of you it's fun it's easy the platform takes high frequency low effort work that happens dozens of times a day so things like checking your calendar to see where a conference room is security alerts maybe the output of security tools like if a PR is failing a smoke test and pulls them into slack where the team's already working together so there's less fragmentation logins lookups and the platform is one of the things that makes slack an awesome place to get work done they're not trying to do the job of all these other products

but we're trying to improve your experience with them by giving them space within your flow of communication to meet you where you're working so slack becomes like the command center tailored to your unique work day in this analogy you can think of slack as this cool tool belt or the slack platform specifically as a tool belt where you have your hammer loops and your wrench pockets and carabiner rings that you hang stuff on everything that you use to get your work done can be plugged into slack so what is the app directory it's where you get your apps if the slack platform is your tool belt the app directory is the hardware store filled floor-to-ceiling with your favorite

tools it's where you can discover tools that you didn't even know existed that may solve a super specific problem that you had no idea was experienced by other people fun fact 94% of slack pork teams are actively using apps so again a lot of you are using them some of you are writing them almost every major sass company integrates with slack we've got Dropbox we've got Google Drive we've got github on top of that our customers also rely on what we call slack first apps which are built around slack's functionality so for example pulley or simple pull sorry Polly it's a parrot it's these things that are meant to take polls in a slack channel and many of

these apps were built by smaller startups or even individual developers and this is the beauty of the platform any developer could potentially build an app that gets used by some of the largest companies every app in the directory is vetted by slack to a certain extent we have a C team specifically dedicated to this work shout out to Amy Jake and Marcelo if you've ever submitted an app to the directory you've probably spoken to one of them they're lovely people and they're focused on making sure that all of the apps in our app directory rule we're optimized to make sure that we have the highest quality apps in the directory so part of why we're able to

have that focus on quality is that we don't make money off the app directory we don't charge for apps or directly profit so we're not incentivized to accept crummy or malicious apps we're interested not in bragging about the quantity of apps in our directory but the quality of them a lot of times the absent directory are the result of collaboration between us and developers which we love and we help people write apps we provide SDKs and tons of documentation so one point of clarity the scope of this talk is only on the distributed apps in the app directory we're not talking about internal integrations which for many of you is the kind of app that someone in your

company wrote and that are only used at your company so what is the app directory comparable to the most obvious thing that people think of as mobile stores for example Google Play or the iOS store spoiler it's not it's a similar model in that third-party developers are writing and shipping apps to slack but there are key differences because they're such monoliths Google and Apple can place much more stringent requirements on devs and they've had a longer life span time is on their side here there's been more time to build automated tooling and to be perfectly honest more money to pay for that engineering time to build that tooling we don't review on new versions and I'll

talk about that in a minute we're also similar in some ways to Chrome extensions but our situation is not quite as bad Chrome extensions can change at any time slack apps can also change at any time but we often know when they do one thing that's more complicated than securing Chrome extensions is that those tend to do one thing and that thing is an isolated thing so maybe you have an extension that modifies headers and requests and the scope of that risk is that extension but slack ups almost always tie back to a web application that's hosted somewhere else so for example if you use something like the Dropbox app there's a certain limited amount of Dropbox functionality

that exists in slack and then there's a huge web app or service backing it if you think about the total risk to our customers if we say to install an app that you need to consider not only the risk in the slack app but the larger attack surface of the backing web application finally the closest relative to the slack app directory is the HipChat Store and the big difference here is directory as opposed to store and that lets us treat security a little differently so from talking to some of our peers at other companies and I will not out any of you I promise I've learned that this can be to our benefit because we don't make money off of our

app directory we aren't beholdin to contracts that limit the way some of our peers can or cannot revoke apps from their directory so what makes it different to recap a few points that make our app directory just think and a more complex problem from a security standpoint we don't get source code from developers well we can generally understand what the app appears to be doing we can never actually see what's under the hood to see how it's doing it and any app sect person will tell you that that's a key difference it's really difficult to make assurance without seeing a source we don't necessarily see minor version changes so the exception is that the if the app changes scopes

that it requests on installation then it has to undergo re review so similar to the Android app when you install a slack up it's going to ask the user for specific Scopes like can I look at your channel history send diems as you that kind of thing if the set of scopes on an app changes or if they want to add some kind of a new functionality that will increase those scopes but that triggers a rear view through our process but otherwise we don't see changes there's often an extensive backing app behind the slack app the slack up is just the tip of the iceberg and there's a very wide range of app developers which we

love so again this could be a household name intact with a really mature security posture but this could also be one person in a mythical garage building a next big thing and this is what makes us really different than the modern iOS store so let the record reflect that this slide is entirely my opinion and not explicitly the view of my employer cool so we need to take a moment to talk about risk ownership and this is what creates the sticky ethical question that Nicky and I have been wrestling with since joining slack so like any other software that people would choose to use there's risk and using a slack out but whose risk is

it so some security practitioners may be tempted to point to the third party's writing the app and from a legal perspective they would be right slack doesn't really own the risk here and many slack customers and slack ourselves have a process and a panel of cross pillar decision makers to determine what we assess the risk to be for each app installed on our workspace I'm part of this committee at slack it's it's a great deal of work and so one could say that the responsibility is on the administrators of a slack workspace but what are people's assumptions about how much they consider presents in the app directory a testament to its security we would be pretty bad people if we dismiss

this notion that people's assumptions are their own problem if they backfire and to be quite frank one of the reasons I'm proud to work for slack is that we are a thoughtful company that takes ethical considerations like this one to heart so a quick recap of the security implications we can't see what's going on with apps we'd guess based on the behavior we don't have source the app could be doing all kinds of things on the connected back end of The Associated web app and it could change at any time and we might not know about it and so while it may not be a legal liability it's an ethical liability we want to be

good people and offer slack users as much protection from the evils of the world as possible okay all right yeah deep breath slack apps are fun they are fun I'm going to talk about how we're gonna fix this problem so what have we been doing to ensure that apps are secure the current test process as I mentioned earlier we have a dedicated team who does this apps are installed on test workspaces and the testers do a number of things including functionality testing like making sure the app actually works we don't want to put broken apps in the directory then they make sure that a scope that's requested is actually used so we never want to accept an app that

asks for a scope and doesn't actually use it scopes are check to make sure they follow the principle of least privilege so for example like does this app need to have read access to all of your DMS to properly perform its promised tasks if not it doesn't need that scope we perform automated scanning of the app using two in-house tools we're looking for low-hanging fruit so these are tests that are really easily automated new and exciting our QA testers will also be using burp suite I found out that actually half of them have already used this tool before so they're already familiar and we're going to work on training them up to do a little bit of

security review with burp suite as well and then we have a small selection of apps that are manually tested by prod sack these are especially the apps that have really powerful functionality like DLP apps and then we've used some internal metrics to determine a subset of riskiest apps in our directory this was mostly based on scope but a few other factors like the number of installations and we've been having those tested by third-party pen testers so why isn't this enough as we mentioned slack apps are almost always backed by another app so we may look at a slack app that looks totally fine but the web app itself could be problematic if we only look at the slack app we can't

really say whether or not this is a good thing to use but how on earth could we look at every single backing web application like the Internet is huge and full of them of course if we did do full testing the apps change anyway both the slack app itself which we have limited visibility into and the backing web app which we have no visibility into automated scanning is less straightforward than you might hope we've had two separate very smart engineers right tools for this shout out to Fikri and ryan and we've learned it's actually really hard to automate it's not that it's hard to scan for vulns because we've bundled other open-source tools packages like sequel map into our

own sting sweets but the most challenging part of automation automation is installation so because we don't have a standardized installation flow it's really dependent on each app developer and this is the largest piece of overhead and what makes this process a lot harder to automate you also see a very little relevant traffic from the client end of testing so you can't see what's happening between the backing web app to the slack app it's very unidirectional and frankly we don't have enough time to do this on our team we have five product security engineers working out a wide range of security concerns at slack like the app directory is one piece of the large tapestry of

work that we do a few of our new people who are joining out in the audience we're so excited to have you but we are still hiring hit me up so what else can we do we have a lot of ideas of what we could do and I'll go more into each of them and why none of them is actually just great on its own as a standalone tactic so first we could pay consultants I'm a former pen tester I believe in pen testing big fan we could ask des for certs a certification is an indicator of maturity we could host apps ourselves in our infrastructure like Heroku but first slack we could put some extra work on

our risk compliance team to perform vendor reviews but they don't have that many people either and they would probably not be excited about us dumping a pile of work on them bug bounty programs I love our security researchers we know you by name we love you we could just give up uh-huh the spoiler is that none of these works in isolation except giving up but don't worry we're never gonna give you up so what can we try in the future ok pen testing so we can't do pen testing of every app ourselves but we can pay other people to do it but this opens other questions like who's paying for this is the responsibility on

us or the app developers asking app developers to foot the bill would be really punitive against our smaller third-party friends who write great apps but are maybe just two developers working out of their living room so can we absorb that cost and is it sustainable also pen tests are a snapshot of a moment in time if we shell out for them the app could change the next day and we're back to square one can we require people to repeat that testing on a regular schedule or after major changes and if so how do we enforce it and just what does a test include anyway is it just the slack app itself or the slack app in the web app

and this is that tricky philosophical question like are we slack okay with having an app in the directory that maybe isn't problematic in and of itself but might encourage or require users to use a highly insecure web app that's completely out of our control so certifications is anyone from compliance or governance in the audience okay okay um do sorts mean anything so I asked this because the process is really hard and as a pen tester like I said we we don't really feel comfortable unless we've seen the source code and can actually verify what's happening the certification process is really strange if you come from that angle so you know someone comes and asks the developer

team questions and then the auditor like just believes them when they answer and no one's there to look at the code and verify that what they say is happening is actually happening and that they've done at the proper way like are the security features behaving as expected are they actually doing what they think they're doing and I say this not to disrespect developers this is all really hard security is hard and it's easy to mess up we have trouble trusting certs because even the best devs make mistakes so okay even if we suspend that like which certs are meaningful to us and aren't these also a snapshot of a moment in time and how do we check

recertification and finally certs cost money a lot of money and again that's really punitive to smaller developers hosting services we could host apps be the heroku of slack ups and this would give us some benefits like visibility into when they're updated but it's a massive overhead for slack massive expensive computing resources and perhaps we would be moving the risk too far into our own arena compliance focus vendor review so we have this great risk and compliance from it slack who already ask questions when we use other SAS products can we leverage them to perform something like this so similar to a compliance vendor review for web app we would leverage this existing team to ask questions of the

app developers but what questions should they ask and do we just take their word on it bug money this is a cool one and I like this idea a lot if devs already have a bug bounty or a vulnerability disclosure program could they include their slack app as part of that bug bounty again though this requires them to trust slack's platform enough to take responsibility for issues that they have created and not point the finger at us which again the tricky question of all of this is who the risk belongs to maybe we could include slack apps in our bug bounty that might get expensive there is some precedent for this and something interesting that's been happening with

hacker ones Google Play Store program where Google and the devs split the liability of bugs found in the Google Play Store so that's an interesting thing to think about what about a combined risk score so this was inspired by the DMV in New Jersey tech can learn a lot from the DMV in New Jersey they have a system to prove your identity that's based on points so to prove your identity address you have to provide documents that total a certain number of points according to their ID verification program and different documents that you bring have different points so your birth certificate your passport like those carry a lot of weight but you can also bring school IDs

bank statements utility bills and the program allows for some flexibility in what's provided to demonstrate verification and in aggregate a lot of small indicators can prove truth so if we were to borrow from each of the previous ideas and combine them into an aggregate score over time we would have riskier apps and less risky apps the spectrum looks slightly more nuanced than good and bad or safe and scary but the problem here is usability so how do we make this legible and accessible to less security savvy users if they see a grade that's like a c-plus they might be freaked out and not use an app even though it's something that you know doesn't actually

do anything scary or risky or has you know very tight scopes but maybe this is a good thing that will enforce better practices but again the critical thing here is that it'll hurt small dev shops sometimes the simplest solution is the best one we had a lot of conversations with security teams at other companies about what they would find meaningful and when we presented the idea of an aggregate risk score at app set cally the funny thing was that people loved the idea of an aggregate risk risk score they loved it but nearly everyone we spoke to said that they would want to see the specific factors that created that score everyone wanted to see the

details and forming the conclusion so the strongest case that's been made thus far is to arm our customers with knowledge to make informed decisions so we're currently considering a more elegant solution that was staring us in the face the whole time to offer relevant security and privacy information in the same space about info or an app normally lives most likely the app directory page itself but there are other places in the flow where there's dialogue and we could tell people things like links to the policy pages of the company like what's their privacy policy what are their data archival and removal policies pentest information so what are the dates of the last penetration test is an executive summary available to

potential customers if they want to request it is there a bug bounty program or a vulnerability disclosure program available for the web app does it cover the slack out do they have any certifications are they HIPAA compliant again the important thing here is to give the information that people need to make a decision the factors that one's considering when installing a gift finder are different than those when installing a document sharing application we want to process for our users that is as specific as their use cases in conclusion there's no magic bullet or any one-size-fits-all solution I think it's a really exciting time that we have where all these different applications have this fascinating interplay but we

as tech companies are sailing into some new waters and we don't quite have the safety guidelines fully articulated yet but we're not alone we're not alone in this problem and lots of us are working really hard on it and trying new things the general consensus is that people want to see more information and so we're thinking about ways to convey relevant security privacy and compliance information about apps to our customers and when we are ready to announce our solution to the sticky tangled web of app directory security you will all be the first to know at least those of you that follow slack HQ on Twitter and follow the link to the blog post where

we announce what we're going to do and that is all thank you besides SF Thank You slack security engineering team you're all wonderful and other apps Tech folks who've taken the time to swap ideas and commiserate about this problem space and you lovely audience thank you [Applause] Thank You Kelly we do have a few minutes if there are any questions I can bring the mic to you because we do have some folks in the other room hi good afternoon um two questions the one I'm really interested in is what's the failure rate like what's the rejection rate for debt problems that those three people find and I don't know maybe the second one was did you talk about requiring people

to submit the source code we have and the I'll answer the second one first because that's easier very few people want to give us their source code especially for their backing web applications like those are keys to the kingdom as far as failure rate I couldn't give you actual numbers on that but I do know that those people who do the QA testing on apps in our app directory they are pretty stringent and reject a good number of them and they have a lot of factors for them as well so things like they have to have a point of contact for the app and if they write to that contact email address and no one

responds they will not put an app in the app directory awesome I think we're good yeah feel free to come talk to me afterwards if you have any questions or want to talk or commiserate or talk about your app directory with a friend da all good thank you so much [Applause]