BSidesSF 2019 - Vendor Security: Where Our Data Goes We Follow

good afternoon everyone welcome to vendor security talk I wanted to make a quick announcement there will be a movie show hackers movie showing at 7:00 p.m. so just go on to be size SF org slash hackers to find out more information without further ado I'd like to invite Justin for to kick off the talk alright everybody thank you for coming today's talk we're gonna be talking about vendor security and basically it's in the title there where our data goes we follow so what vendor security is is it focuses on including the security maturity as part of the vendor vetting process at a company with the goal of minimum minimizing the likelihood of a breach or

an impact to your company so the idea is anywhere the data goes you follow and we have a lovely panel here so we're actually going to be going down and everybody could do a quick intro yourself your company and what vendor Security's like at your company hi my name is Wendy Sidonie I manage the vendor security program at Netflix we are a fairly large company but I am a lone wolf on my team we have a counterpart in the studio side but he does mostly post-production vendors anything else is my responsibility and I drink a lot my name is Rachel black I'm a senior manager of applications security at one medical I do a lot of things but vendor

security is one of the programs that I help run we don't have a ton of vendors right now in our organization so our team is also fairly small there's two of us right now dedicated to vendor security but as we see that spend increase I fully see our staff increasing as well hey I'm NATO miss acuity engineer at Google so the vendor security team doesn't exist in isolation there's a team of about 25 engineers who do assessments including vendor security Android teams I was asked to say something controversial wargames is a better movie than hackers my name is Kyle toner I'm director of enterprise security at Salesforce my team focuses on vendor / security about 50% of the

time and including me there's 11 of us and we have a lot of vendors and we bring a lot of security diligence to them and I'm Vivian Postell I'm a security engineer at slack there were previously two of us that were kind of part-time on vendor security and we just hired a full-time vendor security person so that's very exciting our program is only a little over a year old so we're really growing it right now so this question is of course a first question being why invest in vendor security and I'm looking at you Kyle open us up yes so I have a very big team we focus a lot on vendor security I think in this sort of cloud security age

it's very very easy for large quantities of data to move from provider to provider very quickly with things like Oh F and so when we look at our vendor security program we're very focused as on making sure that like this the very sensitive data that we hold is protected wherever it goes and making sure it doesn't end up in some places where there's maybe less security maturity and therefore more likelihood that it could end up in the public sphere is there with vendors getting a little bit into that topic there are many many vendors that can have tons and tons of data what about the freemium versions and the the ones that anybody can sign up for

how do you manage a monitor that data freemium is a definitely a tricky thing to do if you don't you know manage traffic at your company if you don't manage traffic then it's sort of left up to the individuals to sign up for whatever they care to the way I try and manage it is by controlling the sort of centers of high data quality so if there's an environment like Salesforce or Google Drive or something like that we make sure to have an OAuth whitelist and make sure that it's very difficult for the data to leave that Center without our involvement Wendie I'd like to hear from you too so when it comes to when it comes to how

Netflix approaches this it's typically a very different model than the rest of corporate world can you talk a little about about how you handle that and your approach yes Netflix is very different we don't have a traditional procurement team and every individual in Netflix Netflix is free to onboard whatever they want we really try to drive home to the business owner that it's their responsibility to protect the data from Netflix and from from hackers and third parties the only you know so becomes a freemium you know it's the wild wild west but we do use Google Drive for most of our sensitive information and and for a lot of our documents and so that's where we

draw the line and I've worked with Rachel on this because she's struggling with the same project is that's where if someone is going to be using an application or a freemium app that has access to Google Drive that's when we draw the line but traditionally we don't say no so you can use whatever you want as long as you're you know it's your responsibility that the person doesn't put the responsibility on us it's for them so they sign off on that risk yes okay so then the the question for the panel is how do you go about evaluating a vendors security posture how do you look at their security maturity as company so at Google we kind of have

questionnaires where we measure the security program by vendor security questionnaires following that we kind of asked for a third-party pen test report I think a pen test report is like one of the best litmus tests of how the security of an organization is because someone's going in actually trying to wreck your environment how do you you know sustain that what is the result of it have you addressed it we also do perform pen tests ourselves is required for the application and that's kind of how we approach it in a similar fashion at Salesforce we also do a lot of pen testing and my sort of belief behind that is if you look at the cybersecurity insurance industry the

way they gauge maturity of a company is by doing their own pen tests and if those people have a very big financial incentive to measure security maturity are doing it by pen testing that's kind of along the same line so we pen test I would say 90% of the vendors that sort of meet the criteria and then about 10% will look at a third-party pen test report I wish we Penn tested every vendor we don't have that kind of people power but yeah we do similar with a questionnaire as well as looking at any sort of certifications third-party pen tests super important and then oftentimes hopping on a call and just having conversations with them about

answers in their questionnaire we've actually been overhauling our questionnaire because we found that just having like a do you do X dou u dou Y it's not as informative as we'd like it to be so we're changing it to almost more of a dialogue of like describe to us how you implement you know encryption across your organization what are the steps that you have in place and being able to have a conversation with them about kind of like their security philosophy definitely tells us a lot more about how they think about and value security and then I'll say for us we also follow a similar pentesting model I think probably the more unique thing is because

right now we only have two people dedicated to vendor security we can't test everything so whenever we're looking at a vendor we try and do an initial triage of are they a low-risk vendor are they a high risk offender and so a lot about what it comes down to is what type of data do they have access to what type of systems potentially sensitive systems are they integrating with and kind of using that as our litmus test of do we need to do the additional diligence of a penetration test which is a pretty big hit on our resources but kind of like the other panelists have said we we do find that to be one of the best ways to really get

a good sense of the security maturity of the organization I personally have tested a lot of vendors that on paper they look like they had a really strong and mature organization but as soon as you start testing them within the first 15 minutes you can find every single OS top ten Under the Sun so it is really a good way we find at least to kind of get that sense of the organization so when it comes to working with and throughout the organization anybody today can sign up for any vendor for the most part and start putting in any data you name it they could do it manually they could figure out ways around it so how is it

that you build vendor security into other parts of the organization so you know when things like this are coming out when projects are coming down the line when people are looking into new solutions how do you do that I think it all comes down to the relationships that you have with the business one of our key goals when we were kind of revamping what we were doing in vendor security was making sure that we had allies essentially with every single department across the organization so legal and finance or big partners mostly because a majority of the vendors are paid solutions so getting them to kind of be on your side and understand where you're coming from from a security risk

perspective they can help feed things into the process but honestly one of the biggest groups that I found the most beneficial to have a relationship with is our executive assistants they spread across the entire organization and being able to know them and kind of have a strong relationship they generally have a good kind of pulse on what's happening the organization and they've been really vital and kind of servicing things to us that we didn't know know about otherwise so go talk to all your e is there awesome people and they can really help you so from Mayan I would say it really depends I think from an organization which is as big as Google we really want everything to be

in a pipeline so procurement is one of your best bets I agree freemium doesn't quite fit into it but procurement catches most things so we actually tie into the onboarding process for most of these or the contracting process so when they kick that off most cases it'll actually feed into a pipeline or require that to proceed the other way to do it honestly is to tie in awareness into the standard information security training we do annually across the organization that's one of the other ways to might actually have outreach one final thing I think a lot of these teams who are onboarding vendors aren't very technical you look at your like benefits team and they're

looking at different healthcare companies in the space they don't know what's good software and what's not good software so if your security team gets involved and brings both the security opinion but also just a usability opinion and says hey this doesn't look very mature it doesn't look like it meets kind of like today's standards they love that opinion and they're more likely to partner with you when they're getting sort of value for themselves in addition to the security value that we bring so when it comes to the hurdles of getting there and getting that that vendor security I guess to par with your organization what are some of the hurdles that are in the way do you have

any specific hurdles a good example would be maybe if an exec your CEO said you are going to work with this vendor what do you do in that case I'd like to address that because I have lots of hurdles the hurdle that I face is the freedom of responsibility culture is amazing it allows us to get our work done without having to go through process to say but one of the hurdles is having the business owner understand the importance of vendor security they don't have to come to me if they don't want to but it's that partnered with a lot of the engineering teams just want to get things done so I often find myself reviewing a vendor

after something has already been influenced implemented but I'm just grateful they asked me so you know I will review them at any point but I think that is a big hurdle for me is getting people to realize the importance of start here and so I do a lot of road shows and and jumping into team meetings and all hands and just say here's where you need to start but it seems to be successful so far but we have a long way to go because a lot of people just want to just get things done I think the one of the things that we've done particularly well one medical is we view our vendor security team not as kind of

a sledgehammer that's coming in to say no you can't do this it's insecure stop we kind of view ourselves as more of an advisory department while I see our main role in the organization is obviously secure data secure our systems we're also there to make the business succeed and so the more that we can really partnership with our business partners explain to them the risks that they're potentially taking on with the vendor choices that they're picking the more that they trust us and the more that we really kind of help them to make the right decisions so there will always be the cases where the CEO says I really want to use this funder but it kind of

comes down to making sure that there's that strong education and awareness there's that they understand the the risks that they're taking on for the business so I think probably one of the biggest hurdles that we run into is that people don't realize that the vendor security review process takes time they roll up and they're like hey can we use this vendor you'll answer by the end of the day right it's like no there's actually a process that we have to go through there's a lot of things that we have to do and so we run into pushback on the amount of time it's gonna take and the biggest tool that we can use when there's people that are like we

have to use this specific software the best thing we can do to make sure that it's going to live up to our standards is lean on them of we're not going to sign a contract until you know you get up to snuff in XYZ ways so if people rush through and they go into the contract before completing the review it kind of ties our hands but that's that's one that often works I also see a lot that we you know we get involved with a vendor and they're trying to do the right thing but there is a serious talent gap in our industry there just aren't enough security engineers to go around and so some of

these companies that are sort of transitioning to a cloud model they they can't find the resources to do things right and they end up outsourcing to a company that also can't find the resources to do things right but will pretend they can and the the security maturity just reflects that outcome like it says it's a trash fire and that's a really unfortunate situation because these people are really trying to do that's the right thing but you have to kind of push back and say like your product just is not ready just a quick note to add for the time lines I think time lines is definitely one of the things that we have a lot of pain points

with you have to surface the time lines to the customer or like the organization basically and a lot of the time is actually spent in collecting the vent the artifacts from the vendor so if we can like surface it in a dashboard where we say this amount of time is actually just of vendors comps and this amount is actually the assessment that helps like mitigate some of the tension I want you to keep the mic because I am really curious on how how does Google scale vendor security it's quite a massive wish we scaled like our search we don't we actually right now the main things we try to do is automate automate automate we try to find like the patterns and

then we try to like basically automate the 80% which can be done through processes or scripts and then we try to tackle that 20% which is interesting on high risk so we have a lot of tool kits including VSA cue plug for VSA cue it is an open source app right now that's how we collect the questionnaires and we actually have a remediation Porter which is automatically populated and we can talk to the vendors through that so that helps us and we have an automated script which actually does triage for us as an initial line and then we have a sister team of junior security engineers who do the first line and make sure everything

is ready for our team to actually jump in and perform the assessment so I think it's like division of labor and making sure that automation so it's easy nothing is ever easy I think we've been doing it for a while I wish it was easier about Salesforce so I have I have a well resourced team like Google as well to do this work but you can only pump so many bodies into this kind of thing at some point you have to find efficiency I think automation is key but also looking at the process and finding ways to streamline things like your contract process or the way you negotiate the security testing agreements like a lot of the time that

we waste in our process is negotiating the ability to security test other companies companies that are willing to you know let us security tests with minimal fuss about it like that process goes so fast so I really appreciate when other companies out there are looking what their customers want from them from security diligence and making it happen proactively so so then in scaling and prioritizing that now for companies that don't have billions and billions and billions and billions in revenue I would like to hear from Vivian slack actually selects doing pretty well maybe I shouldn't mention but really how do you scale at slack you said the team's kind of net new and you have a lot going on

you talked a little bit about scaling your team well that's what we're trying to figure out right now we're kind of at a cusp point of how do we scale this because you know we get a lot of requests for new vendors and so what we're really looking at right now is how can we remove some of that overhead how can we try to find ways to automate things try to find ways to streamline I'm building some internal tools for us right now to kind of kick off the vendor request process through a slack bot so that we're just getting you know the more we can keep everything within like our on ecosystem and have the tracking and

have that information the better but yeah I mean a lot of it right now is still very much just people hours and we're trying to figure out because that isn't scalable so if you if you have these programs and you don't necessarily have the time do you do you outsource some of that work to third parties or pentest firms or anything like that so when we were first implementing the vendor security program and had you know like a backlog of legacy vendors we did contract with a third party to go through a number of our reviews to help get us through that massive backlog anybody else so again our vendor spend is fairly low and our

staff is as well so we haven't necessarily kind of looked into the outsourcing space though I would say philosophically I'm somewhat opposed to it I feel like at the end of the day let's imagine worst case scenario one of our vendors gets breached and either the data we've given to them is you know out there in the open or someone is able to use that access to get into our systems I need to be able to kind of stand there and say we trusted them we put them through this due diligence process you know that if I've completely outsource that to another firm it's harder for me to kind of stand there and say that to a

customer that's beating down our door so at least for now we plan to keep that internal but knowing there are obviously those challenges with scale as our organization grows but one of the things that we have done you know as I've kind of revamped our vendor security program to kind of help with that so we've made it very data-driven so I track every single request that comes through we look at all of the different features of them what type of data they need access to what type of systems all of that is tracked in a super easy possible way so as we start to see kind of those numbers go up I can make a

really strong case to my management of we need more resources to kind of make these reviews happen faster so we've done that very early on in the program just knowing having worked at other organizations with much larger vendor problems that it's something that we need to build that case for so do it early if you can I do work at a company that does have billions of dollars but there is only me and how I'm able to stay afloat well sometimes it's like you know I Love Lucy episode where she's trying to shove all the chocolate in her mouth and that's at times I feel that way but I've also learned that being adopted by the legal

team and some of the teams on the studio side that they put they put questions into place to some of their surveys and so then that way I have I have an extended team that kicks off reviews to me that I have an internal tool that grabs those but it's having multiple teams around that are kind of an extension to myself that's the best I can scale at this point and we are growing so fast so at some point the team might grow but as of now it's it's just one and do what you can we don't pen test either so if the review process is very pragmatic it's just look at the important things and

get people on their way so how has the program changed since its inception for Netflix it originally well before my time I can't speak to that except for the person that handed it off and it was it was binary it was does this vendor process any sensitive information yes or no if yes then we go through review if no we send them on their way as we are growing and maturing and we're looking at risk a little deeper and so now we're starting to measure some of those those numbers and and and get a little bit more granular on what sensitive information you know it's a talent information is it customer information is it credit card information because

the binary way of doing it doesn't work anymore now that we've got privacy involved in legal and it's like it's more of like an ecosystem of all of us working together and not just yes or no so it's involved it's come a long way yeah I mean so you know in terms of the history of what we've been doing for vendor security it originally was fairly ad-hoc of you know different reviews that were coming through and so when I kind of came on board I tried to streamline a lot of that by really tying into the procurement process and kind of building those relationships and with the other business partners in part kind of that revamp we initially were

only focusing on net new vendors that was the easiest way for us to kind of see the new contracts that were coming through and really have kind of the strong diligence put in place for that and so over time we started adding in new and new types of vendors so adding in existing ones that were coming up for contract renewal and then looking at cases like the freemium ones where we're not paying for the solution but trying to identify the right way for us to kind of tie into those processes that's our change so I think I only touched on this automation we also have a sister team another thing we started doing is looking at third-party pentas reports

then one thing I will say which is more like an anecdotal story might not really help anyone is we used to have something called crushinator it used to crush your hopes and dreams on mondays you would pick some VSS and assign it to you and you showed up on Monday based on how much free time you had we have since moved away from that and we have multiple lines of triage and we feature it in a newsletter and then people get like the freedom of choice to pick it we found that to work a lot better I think the sort of growth of the program that I think about is more at a macro level where I remember when I first got

told like oh you're gonna you're gonna start pen testing these vendors like we have this new system it's gonna route them to you all right whatever like that sounds okay and then over the years as I have been on the receiving end of this massive influx of third-party outsourcing to all different kinds of software and staffing firms or whatever the intelligence that it's given me about what my company is doing in a massive scale like I know what benefits is doing I know what legal is doing I know what they're all doing and that that knowledge I've been able to translate and sort of bubble up to my see so and it lets us make more

intelligent decisions about what we're going to do when we prioritize security so I think you know I used to think this work was not very sexy and I thought it was you know like who wants to touch do vendor compliance or whatever but now with the the level of access that we have and sort of how we use it I've loved this role so much so let's let's figure out how we can have some of the folks here leave with some of the lessons that you've learned over the years especially going through all the vendor management channels what is something that folks can walk away from what's what is a lesson learned that you wish you knew in the beginning

evaluating vendors that you can you can take and walk out of here with I think I'm gonna emphasize what others have said which is really focusing on building those relationships I think I don't know if I'm just super lucky at SLAC but people at my company generally want to do the right thing when it comes to security they bring us their questions they bring us their vendor requests and so continuing to really reinforce that to thank people to build relationships to be really transparent and communicative you know I feel like we get a lot more traction that way and it's made us a lot more successful I'm gonna steal a word you just use which is

transparency so for me what I've seen over time it's very difficult to measure the likelihood of a breach and a vendor but what I've seen is that vendors that are more transparent with us as a security team tend to be far more mature so if they're gonna let us pen test if they're gonna show us you know their internal processes their pen test reports whatever they have or they have a bug bounty like more likely they're going to be you know much better on average than the companies that hold all this stuff very close and say all we never share this way it could be very wrong that's because they're hiding things usually I think the most

important lesson I would say is it's true for most parts of security be empathetic saying this sucks doesn't help anyone I think learn to articulate risk so people understand why they need to do something the other thing I would say is like fine PS which have high volume of V SAS or vendor security assessments of vendors and find like partners we have something called tech license they actually work with like big pas kind of benefits and they understand the industry they collect the artifacts and they kind of drive the vendor security and we come and provide the security expertise where we do the reviews and they can help drive it plus one to everything everyone said so far but

probably the biggest thing is if you do not have a vendor security program please invest in one I promise when you start looking at all the different places where your data is going it can get scary so I encourage you all to go talk to your CISOs and invest in it soon and to quickly add to what Vivian said the communication side whether you're on the vendor side or you're on the side that's reviewing if you can communicate even if your vendor security or your security at your company is not exactly the way that you think that Netflix or whomever wants it to be it's when those teams those vendors come to you and say

here and they're honest about it and then you're able to work together it's like I'll said if they're hiding it and they don't show it to you then you're less likely to be like hey let's work together that's when you just say this is not a good vendor communication alright and that's our talk for today so thank you everybody for joining and it feel free right on time [Applause]