BSidesSF 2025 - Plays Incident Response (Maya Kaczorowski, Whitney Merrill)

All right, folks. You are in the IMAX theater and we're just about to get started for Besides Plays Incident Response with Whitna and Maya. So, I'm going to turn it over to Whitney. Whitney Maya. Maya to kick us off here. Uh, take it away, Maya. First, a big round of applause for both Maya and Whitney. [Applause] Amazing. Thank you so much. Besides San Francisco, welcome to our presentation. Today we'll be talking about incident response. There's been an incident. There's been an incident. Okay. Uh time is of the essence. We have to address this now. Hi everyone. Immediately we're going to ask you to follow these instructions. We need your help. Um, and although we're very

excited, please remember this is a learning exercise. As CISA RIP would say, there is no hidden agenda and there are no trick questions. The resources and written materials provided are the basis for discussion. We will be using the emojis on Canva and you can add comments, but don't put anything in there that you wouldn't want recorded. Be excellent, please, to your fellow attendees. So, um, let me introduce myself. My name is Whitney. Um, I am a lawyer. Um, I work for a tech company. Um, but I am not your lawyer. If I were your lawyer, um, I'd be billing you about $1600 each for this 30 minute session. Um, I serve as a privacy counsel who works on

incident response continually, um, day after day, incident after incident. And this is my co-presenter. I'm Maya. I'm a product manager. I've had a background in uh security product management. Again, I'm not your product manager. Uh I'm the founder of a new company called Oblique, which is an identity directory. Um so that's who we are. But who are who are you guys? Oh, you guys have forgotten. That's so silly. Okay. You're the TL of the signup flow. Uh also the TL for growth at a a company we all work at together. It's called Hyper Finance. It's a B2B public fintech company. So we sell financial services to other businesses. And because you work in growth, you love money. You just like

money. Um, all right. Uh, shall we get started? Uh, trick question. Uh, you have no choice. The game is already started. So, what's going on? It's Friday. After lunch, you've had some cookies, free lunch. You know, it's a tech company. I don't know about you, but I feel a little bit sluggish. Um, a data analyst hits you up at your desk and they're seeing signups drop off. They reach out to you as the TL of the growth team. The signups are slightly lower, but not that much lower, and you're just not sure why. They ask you, "Did you push any changes to product?" What would you do? Please use the emoji reactions. Do you roll back

the most recent change? Dig deeper into the data, perform a cohort analysis, wait and see, maybe it will all go away. Or, oh, look at that. We We got a lot of thumbs up here. People are looking for dig into the data. Oh, wait. No, maybe. It looks like you're going to roll back the most recent change, it seems. Okay, let's roll it back. Uh h I don't know. Nothing seems to have changed. The signups are still lower than before. And I mean, it's Friday afternoon as they always are. But still, I'm out for coffee. It's Friday. Come on. It's a tech company. I'm just listening to a podcast on my walk. It's very calm. Customer support calls me,

which like never happens. Uh I own the identity flow, so they reached out to me. It sounds like they're seeing an increase in tickets for manual verification of new accounts. So, there might be something broken in our know your customer, the KYC flow that we have here at Hyper Finance. Um, it sounds like something's not working. They want me to take a look. So, until then, they're just going to keep using the manual verification process, which is limiting how many new orgs can sign up. The Yeah, I mean, I guess I'll take a look like as soon as I get back on my desk, right? Like my coffee is way more important. Um, so I leisurely make my

way back to my desk. I'm going to look at some of the tickets that customers report to send over on Slack. Sure. Uh, there's lots of people complaining, so I'll test it myself. So, let me just try to sign up for a new account. Oh, that's weird. I filled in my info and scanned my ID, but then I get an error. Like, the form won't submit. Let me try again. What? What is it actually scanning? That's weird. I scan my ID, but then the scan that's shown on my application isn't mine. It says that my name is Eric Chang. My application doesn't work because the ID being submitted isn't mine. It's not my name and

photo. What should you do? We really, really need to sign up new users to meet our revenue numbers. Don't forget that. So, do you reach out to the vendor who provides the service immediately shut down the service and revert to only manual verification? But what about growth? Reach out to your friend who's on the security team and let them know that you're seeing something weird. Um, let's see. I'm hearing a lot of Okay, so I do reach out to my friend on on Slack who's on the security team, but they're actually at Bides SF, so they're not online right now. Okay, I immediately shutting down the service seems a bit aggressive. So, I'm going to reach out to the vendor. I

email the vendor and um it's also the end of the day on Friday, so like most of my colleagues have gone home. It's still a bit weird. I'm still intrigued. So, I'm going to keep, you know, give it a couple more hours of digging into logs. Good news. I get pinged on Slack. Uh, we heard back from the vendor. They emailed the billing address that we had. So, they emailed the person who owns that that service who's on finance. So, finance reached out to security and then security reached out to me as a to me to you as the TL for the system. Um, the vendor says that something's not been working on their their end and they're

investigating. And by the way, our contract is up for renewal in 10 months. Um, security is asking if you know what might be going on or if you're affected. You explain the weirdness that you've been seeing and that you we've been dealing with. Again, what do we do? Do you wait until Monday morning until you can talk to the services billing owner? Email the vendor back, declare an incident. Um, and if or if you haven't already, shut down the service. Uh, I'm seeing declare an incident or shut down the service, which are going to do. It's an incident. We're going to declare an incident. We have an incident. Everyone freak out. Amazing. Amazing. Okay. Okay. So, what have we learned so

far? Um, an incident is a disruption to a service that could result in the loss of confidentiality, availability, and integrity. This could look like a potential data loss, ransomware, or uh some other unexplained event. Anyone can declare an incident. No worries, no harm. And we also learned that you should have a single place where you can report and declare incidents. And like just a single Slack channel is fine. Also train everyone in your organization on security incidents, not just the security team, so that other people know what to do when it happens. And set up a security at email so that vendors can also contact your organization. Often the only contact they have is the

whoever actually bought the service. So we're going to move on to level two, uh, investigation and containment. So we have an incident respond to. Okay, congratulations. You have all been upgraded to the incident commander role. Um the vendor is now in contact with us. Um they have reached out again. Here's what we know from the vendor. They were compromised. They confirmed what we saw. A user user data was being leaked from one user to another. They don't know when it started. They don't know how they were compromised and they don't know when it will be fixed. It doesn't seem like they know a whole lot except that there our contract is up for renewal in 10 months.

So, you go to retry rep um you try to repro the bug yourself and see what the issue is. First of all, I'm annoyed. I already reprod the bug. I don't know why you think as you as an engineer need to go repro the bug, too. Like, come on. I know how to use my own product. But anyway, so we know what the issue is now. Multiple people have

reprod Oh, no. No music. Do you Oh, yeah. Music. Do you loop in counsel? If you haven't yet, shut down the service. Ask an engine on your team to go see what they can find in logs. Email the vendor back asking when the issue is going to be fixed. I'm hearing people literally say all of the above out loud. Um, well, okay. I think I think we need a lawyer. And since we're here at Besides San Francisco, we all know what to do. If there's something wrong, you call and phone. No, no, no, no. Not that kind of lawyer. Hey, Winnie, you're you're privacy counsel here at Hyperfanets. Can you help us? Of course. Happy to

help. Um, can I want to talk to you about this like really weird thing that we've been seeing? Uh, sure. What's going on? Okay, so when somebody tries to sign up for the product and they scan their ID, they get somebody else's data, like Eric Chang's data. Oh, no. Have you called an incident? Oh, yeah. I haven't Sorry, I forgot to tell you. There's an incident. Okay, so let's put this under attorney client privilege. I want to be looped in on absolutely everything and I'll be keeping a list of everyone who knows about this incident. Not only are we a public company, but it's an open trading window and every new person who gets li looped in must be approved by

council. I'll be thinking about reporting obligations that are time-sensitive, thinking about how this might turn into litigation and what documents might be turned over as a result of litigation. Since we are um a public company, we have additional obligations about reporting for cyber security. I'm also thinking about whether or not I need to bring in law enforcement, notify our insurance broker, notify and talk to our comm's team, and loop in other members of the legal team to support. That's cool. You'll be doing a lot of thinking. I like it. Um should we should I do anything? Like do you want me to like pull logs? Yes, we we really should have logs. Do do we know what kind of logs we

need? Uh all the logs. Um but I will get back to you about specific things that I need. Great. Um, should I like email the vendor? Yeah, it sounds like you've already been in contact with the vendor, but let's keep chatting with them and please loop me into any communications with them. Okay. And should I shut down the service? You know, um, based on our previous documentation, I really think we should talk to executives before we take an action like that. Okay. Anything else you need from me? Um, no. Just keep sending me updates. Um, can you update me in the next two hours? Yeah, that sounds good. Okay. So, I think what we agreed to to sum up, the plan is that

I'm going to loop you in on any future coms and incident updates. I'm gonna check the logs to figure out what happened and you're gonna tell me what I need and we can figure out exactly who's affected. Uh, and I'll email back the vendor to see if they have a timeline on the fix. Okay. So, Whitney did email me. Dear Maya attorney clientprivileged confidential. I need you to find out how many people are affected, where are they located, timeline of how long this has been going happen happening for, and who was had access to the data. Was it temporary or contained? Please let me know your findings. Yours in law, Whitney. Amazing. We all know I'm just going to

give that to an engineer. Um, all right. So, while we're pulling those logs and waiting on the email back from the vendor, um, I go catch up with you, the the TL, the incident commander. So, I'm in charge of incident comms now. Uh, and this incident is under attorney client privilege. I'm just updating you on the fact that I talked to Winnie. You really think it's time to take the service down? So, I reach out to Winnie to chat. Hey, hey, Winnie. I think we need to shut down the service. But it's also the end of the quarter. I agree with you. Um, I don't know who can make the decision to shut down the

service. Let's ask our incident commander. So, what do you do? Do you make the decision yourself? Make the decision in consultation with counsel? Reach out to your boss or set up a meeting with relevant execs and stakeholders to brief and make a decision. I don't know why you all like meeting so much. Um, but it sounds like you want to make a decision with council. That that sounds like a plan. Making decisions during an incident can vary from company to company, but there are best practices that no matter the culture, you should be following, including making sure you have a rapid or racy model for roles and decision-making during an incident. Align on these before an incident

happens so everyone knows who should be doing what by when. Trust your gut. If something doesn't seem right, speak up. Also, think about decisions that are easily reversible versus those that are not. For easily reversible decisions, bias towards moving quickly. Prepare your teams. Run tabletops at all levels of the company, hands- on keyboard all the way up to board of directors. And finally, learn from incidents and tabletops uh and documentation that helps you respond next time. All right. So, we shut down the service uh and you we all catch up with council. We discovered that about 20,000 customers received a copy of a driver's license or national ID from another individual. We're a global company here

at Hyper Finance because we can make money everywhere around the world. Uh but most of these issues are in the US and the issue has been going on and off for probably weeks now. The vendor still hasn't fixed the issue. I really wonder what their contract says about this, but thank goodness you shut down the service. But the service shutdown hasn't been communicated to anyone. So c customer support is reaching out and they're like, "Hey, we have like way more tickets. I don't know what you did and the press knows that you're potentially affected because your company is listed on the vendor's website." And so they start reaching out. What do you do? Do you post

something in the general Slack channel? Draft coms for your exec to send out to other execs? Write an update to a short list of looped users that you want to update on the issue? Or just don't do anything. This is sensitive. Sh, keep it quiet.

Some of this is cultural. Um, if an internal comm's team exists, align with them. Also align with council on what can be said. If the press ever reaches out to you directly, point them to your comm's team or designated representative. Yeah, the worry here is having a press leak. Um, it's also internal documentation. So, what do we learn here? Um well, first of all, does it feel like you have way way too many things to do? Like, yes, this is what incident response is. You're always overwhelmed. Um so the the the first lesson here is to delegate. As an incident commander, it's your job to assign work that needs to get done. And when you require someone to do something

like to stop their current work, you have to tell them it's for an incident. You actually have to say, "Hey, this is an incident." That doesn't mean that you have to tell someone what the incident is who's not read in to what this is, just that there is an incident and you're taking this engineer to go work on something. Be careful how you talk about attorney client privileged or sensitive sensitive incidents internally at your company. In in-person communications are the way to go. Don't use words like breach unless a council instructs you to do so. Breach is a legal determination. When in doubt, talk live. Make sure you how to know how to actually reach important members of your

team. Get their cell numbers. Establish call trees. Be aware of the impact of knowledge internally on commu on employees specifically as it relates to material information about the company. Again, I mentioned it was an open trading window and you're a public company. You may no longer be allowed to trade. Um could it be insider trading? Are you in an open trading window? Other issues, litigation concerns, etc. Council will be thinking about so many various legal obligations that are time-sensitive and how this might turn into litigation. The goal here is for them to protect you and the company. Um, if you're worried about the public narrative around the incident, they know, just know that like council is uh worried about how

companies are defined in their response. Um, set up regular stakeholder comms, provide updates like daily stand-ups or comms. You can always cancel a meeting if scheduled. Make sure you provide updates and to execs and stakeholders. Establish a communication cadence and stick to it. Reliable communications establish trust and help prevent micromanagement and fear, uncertainty, and doubt. Always communicate when the next communication is coming. But our work here isn't done. We've shut down the service. Um, but we now have to recover from this issue. So, nobody knew is going to be affected given this this service was shut down. But it's costing us money to have it shut down, right? Nobody else will be able to sign up for the product. Hashtag

growth. No. Uh, so we need a fix. And again, the vendor still hasn't told us when the fix is coming. Come on, vendor. Um, what do you do? Do you reread your vendor contract to see what you can do? Look into alternative vendors, look into building your own solution, or wait for the vendor to do something. I'm seeing a mix of alternative vendors and read our contract, I think, are the most popular. Cool. Well, it doesn't really matter. Good news. the vendor emerges from the gutter and announces they have a fix already. Uh, finally. So, we're good to go here. We have to deploy the fix as soon as we can. Like, we're going to do it

right away. Okay, you ready? Like three, two, one, like take off. It's deployed. Good. Great. Um, you know what we haven't done? We haven't told our users. So, what if anything? What are you going to do? What are we going to tell our users? If anything, do you tell them nothing? Do you confer with council on legal obligations? Do you even if not legally required, just say something anyways? Um well, you're mostly saying the last two. And since Whitney's here, I'll just ask her like Whitney when which we do. Yeah. So, um as your counsel but not your actual counsel, uh communicate, uh especially when customers already know about the incident or the incident is public. Your

legal obligations are only the first decision point. And even if not legally required, you should ask your customer. Ask yourself if your customer would benefit from clear, transparent, and apologetic communications. Even if not legally required, work with counsel and a great comm's expert, and they can help you land this correctly. Nice. So, we did what Whitney says. We communicate this to our users. And you know, that wasn't so hard, was it? I mean, other than the entire instant response. So, what did we learn from remediation? Um, first consider planning what's I've heard called like cyber resiliency, which is what happens if one of your vendors uh that one of the vendors that you depend on is

compromised or down. Like what's your backup plan? You can't necessarily have this for every single vendor in your in your environment, but like know what you're going to do in that situation. In this case, we were still able to do manual verifications and onboard people much more slowly. Um, but it wasn't an ideal situation. So again, it might be a theme you've heard before, but looping council uh they are going to think about the regulations that we have to comply with, including notification to regulators, any data protection authorities, materiality analysis to file with the SEC if you're a public company in the United States. Um they'll think about the customers, right? Are the customers seeing or thinking something weird?

Should be we how should we be thinking about this with a comms team? Should the comms be reactive, proactive? And what does communication with law enforcement ultimately look like if needed? In this case, it looks like a weird bug from the vendor. I don't think we need to sick the FBI on our vendor. Um, but we're still not done. We We still have to do a postmortem. That's one of the most important parts of doing incident response. So, um, now what went well and what went poorly? You can use the comments in the app. Again, don't put anything on here that you wouldn't actually want at biz. Please be nice while I figure out how to sway them.

Lots of emojis. But can you Did we allow you to Oh, okay. Legal advice. Look at this. Recovery went well. Oh, no. No. Oh, no. Incident, incident, incident, incident. Uhoh. How'd that happen? I don't know again. Well, see, look. Incident. I think just hit play. I think you ladies decided to yo dog everyone here. As in yo dog. We got an incident. So, I don't have an incident. There we go. There we go.

Short the stock. I love the advice. Chat, we're hooked. Okay, cool. Cool. Love it. Love it. Um, well, now I lost my presenter mode. Okay, we're going to just move forward. See what happens. This is you. Congratulations. You all did a fantastic job. You were amazing incident response uh commanders, and we really appreciate your participation. Um some of the takeaways from our incident response today are a little game. Um the first one is be vigilant, right? Anyone can report an incident. Um things that look suspicious in your internal applications, make sure you react to those appropriately, react to messages, react to reports, etc. Uh prepare. Take the time to figure out what your incident response plan is, how

you're going to actually act in certain situations. If you have the ability to like do tabletops or something like that, that can also potentially be useful. Um, any like written down policies or like pre-written comms can be super useful as well to have ahead of time. And work with council. I mean, every council at every company is going to be slightly different, but partner with them on establishing an incident response program. They should be your partner in in in an incident and make sure that you're not alone in figuring out what you need to do next. Um, even if it's chasing down the vendor. Yeah. And also communicate, right? Communicate internally to um people at the company

of what's going on appropriately. Right. the right level of information about the incident. Communicate to execs and other stakeholders. Communicate with your council. Communicate externally to your customers, vendors, whoever it happens to be um in in this situation. But the real lesson here is incident response is everyone's responsibility. Um if you are called into an incident and you're not the incident commander, you should make sure that you're participating in helping reestablish or fix the problem at hand. Sparkle emoji. Okay. Um, here's a couple of resources that could be helpful if you're thinking about incident response at your organization. The first one is, um, some example policies. So, I said earlier, you know, you should prepare by having some

policies. Here's what some other organizations have put out. Um, some other things you can do are tabletop exercises. Here's a couple of example tabletop exercises from CISA and from a no starch book that you can use to prepare. And then the last uh, line there is a link to these slides in case you want to use these uh, with your team. Um, I think that's it. Yeah, that's it. Thank you. Uh, we can play again. [Applause] And then I believe the slido is what's being used for questions. So, I will kill this. I think the slido is being used for questions. Although we have no questions in the slido. I think everyone has just been blown away by what an

amazing exercise we've gone through. So, first of all, once again, a round of applause for our presenters, Whitney and Maya. Thank you guys so much. I will say we do have a little time. We'll say I one burning question from the audience. I see a hand up with the scarf. So, he says someone else is there first. I can't see you. You're in the light. Please speak clearly and loudly. All right, it's you with the scarf. You're up.

All right. So, we're gonna close this out by saying if you guys have any questions for our great presenters, find them up during the reception. Thank you so much. This ends our program day. I have a few announcements and awesome. Thank you guys. Round of applause for you for being an excellent audience here too and participating in our incident response. All right. So, since we are the last talk of the day, I uh definitely want to tell you we have a happy hour starting at 5:30. So, you get a little time to uh check a bag, visit a vendor booth, go get uh things stamped for tomorrow. After the happy hour, there's a party this evening, also

sponsored by Whiz that starts at 6:30. There'll be orders, not a full and hearty dinner, but there will also be food and drinks at that. Tomorrow we are back at 10:00 a.m. with a keynote from the amazing Wendy Nurther. And you definitely want to see her. Finally, we need to all be out of this theater [Music] by very soon here 5:30. So that means everybody uh you don't have to stay here. You can't stay here. You don't have to go home, but you can't stay here. I fumbled that one. With that, this ends our program day. Thank you so much everyone. And I've been John D, your MC and ring master. Enjoy the rest of Bides.