PG - A Day in the Life of a Product Security Incident Response Manager - Tyler Townes

Uh, so, thanks for coming to my talk. The topic is PSIRT, a day in the life of an incident response manager. We'll talk a little bit today about how software vendors receive vulnerability reports and how we manage risk on behalf of our customers. So, a quick agenda. Uh, I'll give you a little bit of background on myself and then we'll dive into defining what a PSIRT is and some of the major responsibilities that a PSIRT has. Uh, we'll talk about some real vulnerability reports. We'll talk a little bit about coordinated disclosure and we'll give a little bit of advice on building a PSIRT if that's something you're interested in doing. And we'll close out with Q&A.

So, background on myself. I've spent the last 4 years working at BlackBerry product security. Uh, I spent most of that time with the incident response team as a program manager. I've done a few different things during my time there. I've done a lot of risk and vulnerability management for our operating systems. Spent quite a bit of time running our anti-malware programs and making sure that our app store is free of malware and free of privacy infringing apps. And today I spend most of my days making sure that we can ship monthly security updates for our Android devices. So, PSIRT stands for product security incident response team. This is the team that intakes all of the vulnerability

reports from people external to a company and acts as the overall coordinator within the software vendor to make sure things are getting fixed. We do a lot of risk and vulnerability management. Um, we do a lot of communicating of risk to our customers. So, although the our business is rooted in a technical nature, vulnerabilities, we employ a pretty wide array of positions on our team. So, on top of security engineers, we also employ program managers like myself, technical writers, operation specialists, communication specialists, and all of these people help to make sure that we're able to fix and and ship up uh, software updates and to communicate risk properly to our customers. So, it's a little bit high-level, what

do we actually do? Day in day out we take we triage vulnerability reports, we assess them for impact and for risk. The way we get these vulnerability reports is through what we call secure@, the secure@ inbox, and this is our public interface for how anyone can submit a report to us. It's a public email address. And your company may have secure@ or security@ or vulnerabilities@ yourcompany.com, something similar. So, we receive a lot of different types of reports. We receive a lot of end user reports, normal normal people. We receive a lot of enterprise customer reports. And when I say enterprise customer, this is someone who has like a very large deployment of your product. It's also where researchers submit

vulnerabilities for coordinated disclosure. And it's where we centralize a lot of our open source intelligence. So, of of the end user reports, there are a a lot of different buckets that you could categorize these into. I think the one we see the most is the disgruntled users. And these are users who are just using your product and they encounter some kind of functional error and they immediately assume that it's a security problem and they reach out to you to let you know that they're not too happy. It's pretty rare that these are actually security related. It's a it's a misconception or I think in a lot of cases they're just looking for any public email address they can get their

hands on to let you know let your company know that there's something not right. So, these reports have a really high false positive ratio. It's not impossible that something from this bucket could turn into a confirmed vulnerability report. It's just highly unlikely. So, has anyone heard of the Brad Pitt attack? So, we had a user not too long ago who emailed us and let us know that she was browsing Facebook through one of our devices and she followed Brad Pitt's page on Facebook and shortly thereafter Brad started messaging her and they started chatting and they got close and they entered into some kind of relationship. And she shared a lot of personal detail with him including her full name, her

home address. But at point she thought everything was okay. And shortly thereafter Brad asked her to put up some bank accounts in her name so that his business associates could deposit money that was owed to Brad into her accounts. So, at this point she said, "Okay, there's something wrong." So, she reached out to us and she asked if we could confirm whether or not that was Brad Pitt. So, obviously we can't do that. Uh, we did redirect her to Facebook and said they'd probably be the best people to ask, but not entirely sure they could help in the situation either. But I really like this user report because this was just a normal person who found themselves in a little bit of

trouble and realized they had risk on the table and reached out for help. And that's great. But it hurts cuz we were not able to help her and I think it's interesting that users aren't really clear on where to go for help with their problems even if they realize they have one. And sometimes even if they do know where to go, help's not necessarily available. So, I don't entirely have a solution to this problem, but I think it's a really interesting problem and I I hope that maybe as a community we can solve that at some point.

So, not all end user reports are false positives. There's definitely an opportunity to get a confirmed vulnerability report out of this bucket. Had a user not look too long ago who emailed in and claimed that he had figured out how to bypass one of our authentication methods. So, we followed up with him and asked him for a list of steps to reproduce the issue and he provided it. We confirmed the vulnerability. Work with product development, got a fix, got an updated software build out and everyone's happy. He reported a vulnerability, very quickly got a fix on his device. And for us we got to fix that vulnerability for all of our customers, so it's a great situation.

And this is really the type of report that we'd like to optimize for. This is the ideal report. So, there are a few different ways you can go about trying to optimize to get more of this. I think the most meaningful way is wherever you post your vulnerability disclosure or vulnerability reporting process, include a minimum set of information that you require in order to investigate a report. However, no matter how good of a job you do in this regard, you're still going to get a lot of a lot of noise. So, we also deal with vulnerabilities submitted by researchers. So, this would be coordinated disclosure. For anyone not familiar with the terminology, coordinated disclosure are industry best practices for how a

software vendor and a researcher should engage together in order to fix vulnerabilities and protect customers. So, the PSIRT is the intake point for these reports and acts as the overall coordinator throughout the process. So, we will receive the vulnerability report, work with the researcher to reproduce the issue, and subsequently work with product development, get a fix, get an updated software build, test to make sure that build is actually fixed, and assess the situation for risk and determine whether or not an advisory should be published. So, these are awesome reports to work on because these these are a very low false positive ratio. So, there's just a lot of a lot of genuine vulnerabilities that come through these reports.

So, we also deal with a lot of open source issues. Open source vulnerabilities, open source risk management. And I'll make a note here that not all PSIRT teams are responsible for this. I know some of our partner teams in industry they have moved this responsibility to their general product security team or they've moved it to product development or in a lot of instances they don't do it. So, I think ignoring the open source problem is definitely not the right solution. There's a lot of there's a lot of struggles associated with working with open source open source vulnerabilities. Uh, if you're working on a very large software product, there's definitely some open source in there. I think

there's only a couple companies that I know of that don't use open source. So, I think it's pretty prevalent in industry. And if you're working on a really large product with a lot of software developers, there's a good chance that you have duplicate copies of libraries in your product. And if you're integrating components from other vendors into your product and then say take that product and put it on top of someone else's operating system, you could very well end up with 12 copies of a library. You could ship 12 copies of OpenSSL or libpng or whatever it is that you're using. I think one of the problems, especially when you're taking binary components from vendors, is you don't exactly know

what you're getting. So, it's especially hard to manage that risk when you don't even know what's in your shipping build. So, even if you do know what's in your product and what open source libraries you're shipping, you still need to determine how do you you know, how do you figure out threat intel? Where do you find those vulnerabilities? So, there are free and paid resources on how to track all of that. I think if you use open source in your product, but you're not necessarily managing that risk, definitely start with the free stuff. It is excellent. There's a ton of information available. Obviously, starting with the open source projects' websites, which you utilize, is a great start. A lot of the projects

do a very excellent job of documenting CVEs, vulnerabilities, fixed versions, where to find patches for all of these issues? And if you're just getting started and you need to go back in time to determine you know, what have I missed over the last few years in terms of open source vulnerabilities? CVE Details is an amazing website, highly recommended. It's probably the first place you want to go when you're starting on this journey. They've got data going back to the '90s on vulnerabilities, so it's a it's a great source to start with. There's also some paid services available. They're pretty expensive, but they do a great job of aggregating all this data and making it a little more

consumable. So, in the long run, if you're able to get that budget together, I I'd recommend it, but they are they're pretty pricey.

So, on top of receiving vulnerability reports, we also attend a lot of conferences. We do that for a few reasons. One of the main reasons is coming here to spend time with the research community. So, my experience, it's always an excellent opportunity to come and spend time and and meet people in person who you've gone through the coordinated disclosure process with. It's great putting a face to the name. It's also great getting to talk to people who are on the front lines, so there's there's no one closer to what's happening in vulnerability research than researchers. So, it's good opportunity for us to see what's new, see what's hot in the research community and make sure that we're using that

information in order to secure our products and just generally make them more robust. So, in the same vein, we attend a lot of presentations while we're here. And the bulk of the presentations that we'll attend while we're here are presentations that are discussing flaws in industry standards, industry protocols that we implement. And these talks will be given at an abstract level or they will be targeting one implementation of that protocol. And we'll go and we'll learn and we'll take that knowledge back home and and figure out did we implement this the same way? Are we vulnerable to this or something similar? How can we use this to make our products better, more secure?

There's also a chance that a presentation will be given that directly targets your products. And as an incident response incident responder, those are a little bit stressful. They're great experience, but definitely stressful. And I've I've had some personal experience here. Just thinking back to last year at Black Hat and Def Con, we had some great and some not so great experience in this regard. So, on the good side, we had a researcher who reached out to us about a month before Black Hat last year and said, "Hey, I've got accepted to Black Hat. I'm going to be giving a presentation on one of your talks or one of your products. Here is my vulnerability report."

Here is my vulnerability report. Uh

All right. All right, thank you all for your flexibility in dealing with uh the entertainment of the day. So, we're going to see if we can get this kicked back up. Um depending upon how long this takes to finish up, we might end up pushing all of the questions out there or you might just end up being done. Yeah, go ahead. All right, so I'm pretty sure I was talking about uh a researcher disclosing to us ahead of Black Hat last year, so I will start off there. So, about a month before Black Hat last year, we had a researcher who submitted a vulnerability report to us, let us know he was going to be speaking at the

conference and and wanted to work with us. So, we spent some time with him to understand any changes that we needed to make in our products. We worked with we worked on some proactive communications for our customers because once presentations like that go public, it stirs up a lot of inquiry. "Hey, are we vulnerable? Are we at risk?" And we came here to Vegas. We got to sit down with the researcher the night before, have some drinks, strengthen that relationship. And then we subsequently went to his talk. So, it was a really good example of coordinated disclosure. Everyone was happy. One of the interesting things I think is not too obvious is that yes, presentations will stir up a lot of

customer inquiry, but they'll also stir up future customers' inquiry. So, thinking about that presentation, that's a year old now, we're still getting inquiries about it. So, we'll have new customers who are coming to implement that product and during their their testing phase, they'll go to Google your product security, your product vulnerability. And some of the first things that will come up are the YouTube video, the slide deck, the white paper, whatever the collateral is. And they'll run to you and ask, "Am I vulnerable? Is this going to be vulnerable if I put it into prod?" So, those are really easy answers. No, we fixed that a year ago, a year and a half ago, two years ago.

But, I find it really interesting what a long tail those inquiries have. And then on the negative side of presentations that directly impact you, we had a not fantastic experience last year where we were here in Vegas and we got about a two-day heads-up that, "Hey, we're going to publicly disclose full technical details of four vulnerabilities that impact you." It's not a lot of time to respond, but we did some underground incident response. We figured out we had three of them three of the issues fixed already. We just had to deal with that last one. So, we worked with the teams back home and got a fix in place and an updated software build out.

So, no customers were exploited, which is good. I mean, that's the ultimate goal of what we do. So, all's well that ends well, but we definitely would have preferred a little bit more than two days to to fix that issue. Just want to reflect a little bit on why coordinated disclosure fails. I think there are many different reasons why that could happen. Speaking from my own experience and the experience of my team, there have been times where we have not done a good job communicating, which is kind of funny because that's why people like myself are employed. However, there have been times where we haven't shared enough information with researchers working with us or we've

we've shared too much information. There have been times when we haven't been clear on expectations around fixed timelines or advisory timelines or we've slipped on those timelines and haven't communicated that properly. There's a lot of different ways that some strain and stress can be put on the process. And I think one of the things that hurts PSIRTs is that our struggles are not obvious and we we never come out here and and talk about them. And this this move towards 90-day disclosure timelines is really interesting, but it's actually really challenging in the event that you're shipping something like an embedded product. So, if you're shipping through a carrier or some kind of reseller or a regulatory

body and there needs to be some approval process before you can release that to customers, 90 days is really tight and especially in the regulatory space, 90 days could actually be impossible. And I think it's really up to the PSIRT to if you need more time, ask for more time, but you have to be ready to back it up with facts. And one of the struggles one of the challenges of of working at a big company, a software vendor or any company, sharing information with external parties is is a difficult thing to uh for a lot of people to wrap their their head around, but given the position that the PSIRT is in, it's pretty important that you share as

much as possible. So, I found a lot of success myself being as open and as honest as possible with researchers we're working with to let them know about some of the constraints we're facing and I've I've found a lot of success in coming to like a mutually agreeable timeline. So, that would be my advice, as open and and honest as possible as you can be. Some quick advice on starting a PSIRT if it's something you're interested in doing. Set up that public interface for people to report vulnerabilities to you. Post your process publicly for people to see. And it sounds obvious, but if you're going to open up an inbox like that, you have to answer the emails that come

through it. I've had a lot of researcher friends in industry tell me their experiences of submitting vulnerability reports only to never hear back or to hear back a year later, and that's not acceptable. So, set yourself up a service level agreement and pick whatever time frame works for you, whether it be one day, three day, or five days. Post that publicly and then stick to it. And then internally within your company, make sure that your team is set up in such a way that structurally, you're the only team responsible for answering customer inquiry about vulnerabilities. Seeing as you're the team that is doing all the intake, all the risk assessment, you're in the best position to answer

the questions. Don't let PR do it. Don't let marketing do it. It's the PSIRT's responsibility. And that's it for my talk. So, I just wanted to thank Tom for his mentorship throughout the process. Thank you to my team back home for helping me get here today and thank you to B-Sides for having me on this really cool track. So, if we have a minute and a half for questions? We have a minute and a half for questions. Do we have a question? Hey, uh thanks for the great talk. Uh I just wanted to know what your experiences have been like in internal politics in a PSIRT. Like, in a big company, how do you handle that and

where does your alliance lie like with the researchers or internally in those sorts of things? So, there's always there's always challenges and struggles. I think that I'm in a particularly privileged situation relative to some other software companies where we put a lot of emphasis on security. So, we have a lot of say in what needs to be fixed in a release. So, we actually have a vehicle which we call an SRR. It's a software readiness review where we have the ability to to block software releases if there's something so heinous that we can't live with it not being fixed. Uh but there's there's always vulnerabilities that are submitted on a rolling basis and there's always

releases going out. So, trying to time that all is always difficult. So, politics do play into it, but I I think it in in my experience it's uh a winning battle for for myself anyway, but I know a lot of other companies do struggle quite a bit with getting the mind share for for security fixes as opposed to features. Okay, thank you all. We are close enough to the end of this that uh because we had some shenanigans in the middle, we'll go ahead and continue the Q&A outside if anyone would like to. Um other than that, thank you all for showing up.