BSidesSF 2018 - Rapid-Fire Stories of Creative Solutions to InfoSec Problems (Katie Ledoux)

[Music]

so I definitely put my title in the notes in case I forgot who I was which has happened so everyone's good and hungover and deeply emotionally physically exhausted right yeah that's so good well I am you heard my introduction Kayla do internal security team are up at 7:00 and I am very excited to be here but I'm also so scared and I'm scared for a couple of reasons first of all the medication that I take before speaking at an event like this does have some side effects and I'm also scared because there's no other tracks happening right now you're all stuck here with me and that means I have to talk about something that's relevant to all of you

I can't rely on my mastery of dankmemes alone to keep all of you engaged I also need a dank theme I settled on a topic that I think is relevant to everyone and that is how we fix stuff I know I'm speaking to a group of people that likes to hack all the things but we talk about how to break things a lot and I think that we can afford to spend a little time talking about solution building and problem solving and here's why this topic has been top of mind for me especially lately about two-and-a-half years ago when I started at rapid7 as an analyst on the security governance team my job largely consisted of

reactive work hour security governance function was new so there wasn't a lot of structure in place yet for example we were always buried under requests for information about our security program and my team didn't have an organized way to respond to those requests which left us in a constant state of running around putting out tiny fires and we weren't putting those tiny fires out with the hose it felt a lot more like we were running around with buckets that were half filled with water and we were spilling a lot of the water on our way between the tiny fires it was messy today fortunately my job is very different I spend much less time reacting to little fires we still get a

lot of inbound inquiries about our security program of course but now we have more efficient processes for triaging these requests and we have more documentation to address frequently asked questions and that gives us more time to work on fun things like testing and improving our fullness closure processes and building our own security awareness training program in-house meaningful projects that actually improve my company's security posture so not only did we figure out how to put out little fires more efficiently we're also proactively spending time making our environment less flammable and that feels awesome managing little fires without losing sight of long-term goals is an issue that basically everyone who has a job needs to deal with but it's particularly

challenging for us in InfoSec because so much of our work is by nature reactive in time-sensitive if you get a suspicious behavior alert because one of your users logged in from China 15 minutes after they logged in from Boston you're not going to let that take a backseat while you clean up some of the beautiful animations in your security awareness training module hmm oh no oh my god spoiler situation that's fine it's great everything's fine I'm cool my skin doesn't feel like it's on fire everything's chill so how do we do it what inspires enables and empowers us to put big systemic efficient solutions in place because we know it's not easy so I reflected on my own experience and

I talked to a whole bunch of other people in info about how they fix stuff I looked for patterns in these stories and I'm gonna break my very unscientific findings into two categories first individual factors can you see my mouse doing crazy stuff but then why isn't it here okay so we're that's that's Josh we'll get to him later oh heck yeah you rock so we're breaking into two categories individual factors characteristics of or actions taken by the individuals involved in building the solutions who were they what did they do and environmental factors stuff that was going on around them and hopefully by the end of this we'll know more about what we can do as individuals to solve

problems and we'll know how to build environments where problem solvers thrive an individual factor I saw again and again in these stories which really surprised me was the ability to define the problem it might sound really obvious but we often fail to clearly articulate the problem we want to solve before we jump into action Josh find him who hired me at rapid7 is in the Executive MBA program at Sloan and when I told him I was doing a presentation on fixing stuff he shared a story with me that he learned in one of his very fancy business classes in the late 1990s Don Keefer wanted to implement the Toyota Production system oh yeah I know they told me not to touch

it when it goes away no it's good I like it like this this is good he he hired one of the foremost experts on the system when he arrived Don immediately started asking questions like when do we start there we go what kind of result should I expect and how much is it going to cost me but the expert wouldn't answer those questions instead he responded repeatedly with one of his own mr. Keefer what problem are you trying to solve Don was perplexed he was ready to spend money why wouldn't this guy tell him how to get started the expert was trying to get dawn to formulate a clear problem statement our brains are prone to leaping straight

from a solution straight from a situation into a solution without pausing to to find a problem and that is great when the car in front of us stops suddenly and we have to slam on the brakes it is not so great when we start pouring resources into something that isn't working exactly how we would like without pausing to actually figure out what's wrong and by the way when we are running on autopilot our brains rely on past experiences to figure out what to do next so that's gonna bias us towards the status quo in a way from innovative solutions a good problem statement doesn't try to diagnose the problem let's look at this problem statement we

need to implement a bug bounty program that might be true but it's not a good problem statement why do you need to do that what is the actual problem that needs to be addressed we don't have enough information about vulnerabilities in our products is a slightly better problem statement sure a bug bounty program probably is a part of that solution but by using this problem statement we left ourselves room to explore other important elements for example it invites us to consider whether we've invested in being responsive to researchers and remediating their findings quickly regardless of whether you have a bug bounty program in place prompt and effective communication will make it much more likely that disclosers will

have a positive experience working with you and they'll work with you again in the future and maybe we could also benefit for more regular pen testing again I am NOT saying your initial diagnosis was wrong but a better problem statement invites you to consider all of your options another individual factor that showed up again and again was the ability to stack small victories when I ask people what inspired and abled or empowered them to fix the thing pretty much everyone cited previous accomplishments Marie Byrne was responsible for training hundreds of human rights activists on security basics super casual very casual life and he was finding the lessons weren't really sticking with them they weren't able to recall the information

at the critical moments when they really needed to use it so he and his team built a free open-source tool with lessons and checklists that they could access from their phone and it covers everything from sending secure email to crossing a border after peppering him with a few other questions I got back to the matter at hand and asked what inspired enabled or empowered him to build this before he left college he helped build Vieira de an NGO that used hidden cameras to film and expose human rights abuses again like super casual life super chill in addition to teaching him a lot about security fundamentals this experience gave him insight into how to build a project in an

organization that he would use later on when he thought back even further he recalled problems he had solved as an Army reservist and while volunteering at the MIT Media Lab as a teenager the observation that it's useful to practice our problem-solving skills with increasingly complicated challenges is not exactly groundbreaking but I know it was a good reminder for me when we're trying to fix something in our organization or our industry there's nothing wrong with starting small not only does every improvement count but these experiences are also teaching us valuable lessons that we will use when we're tackling larger issues down the road while Rory was accumulating skills and experience from different projects we can also use this approach of

stacking small victories when we're working towards a single end goal take Bob for example Bob is not his real name because we're going to get into some gnarly deets about his environment in his previous job Bob felt that some of the solutions he was working towards fell apart because he had a very clear vision of what he wanted the end state to look like but he failed to break those projects down into small steps and little wins that he could stack on top of each other like Legos now he's having a lot more success in this area he shared his approach to access management as an example at his organization there hypervisors are their crown jewels and way too many people had

access to them he knew he had to fix this and he had a pretty good idea of what the end state should look like but instead of starting with that final vision he broke it down his first mini project was to require users to request access to the hypervisors no one even reviewed or approved these access requests the user just had to initiate the request by slack and they were in it was a small change but this just-in-time provisioning did improve their security posture because circumventing it would definitely require more sophistication on the part of a would-be attacker then he took groups of users and added some more restrictions so this just-in-time provisioning process would only work on the

hypervisor as they access regularly if for whatever reason those users ended up needing access to other hypervisors no big deal they would just follow the access request process they were already used to only this time it actually required approval by the service owner of course the benefit to doing this with groups of users instead of all the users is that he could see how it went and make any necessary adjustments once this is running smoothly their next step will be requiring the user requesting access to enter a business need which will be reviewed by the security team this is data driven change management that introduces friction slowly and makes the organization a little safer with each

step you don't need to fully master the principle of least privilege across your entire organization in one fell swoop and based on Bob's previous experiences if he had tried to do it all at once he probably never would have gotten anything done and we know this works so crowd exercise if you super hungover morning crowd exercise if you are still primarily using the waterfall method for development or general project planning please make some noise that's that's what I thought that was I started with an easy one for you so now if you're using agile please make a sound I kind of feel like I could led you in a direction there and I regret what I did

but you get the point okay now if you are using agile but sometimes you just put a giant story in your sprint that says something like fix access management without actually breaking it down into pieces and testing each one and adjusting your approach when necessary hang your head in shame and I am right there with you in my last sprint I had a massive story called fix the tickets with no description because I am a garbage person this is a good reminder that we really need to apply these agile principles to solve these big problems we need to break it down into pieces change our approach when necessary and stack those small victories another individual factor that

kept popping up was the ability to leverage diversity of thought let's talk about the catch-up principle you may have heard this one before I don't know about you guys but in Boston we keep our ketchup in the fridge it turns out if you're British you're likely to keep your ketchup in the cupboard so let's say we've got some hot french fries in front of us and it's time to get the ketchup an international team has a much better chance of finding the ketchup in a timely manner but now let's say they've searched high and low fridge and cupboard and it looks like we're just fresh out of ketchup so what are you gonna use instead well if you're a

fridge person you might use mayonnaise or mustard because you're seeing them right there in the fridge if alternatively you're a cupboard person you're more likely to see other ingredients that might work like malt vinegar or my personal favorite old bay I know Old Bay is delicious on fries because I went to Villanova which is near Philly home of chicky and Pete's crab fries world really good at basketball but I digress go cats what I'm getting at is the more diverse backgrounds we leverage the more associations we get and the more paths we have towards solving a heart problem we're a relatively young industry which means this room is full of people who have had past lives before they got to

InfoSec the knowledge we gained in those past lives makes us better problem solvers and the same goes for the knowledge we accumulate through our interests in hobbies outside of InfoSec Rebecca brown who leads our threat intelligence program at rapid7 was spending a ton of time going through link after link of open source intelligence which is sometimes referred to as news and a lot of that time was spent reading articles and reports that were biased or straight up wrong obviously that information isn't very actionable so it's not a good use of her time she knew she needed a way to proactively weed out that crappy content so she could focus on actionable information and she drew on more than

her threat intelligence experience to solve this problem in her spare time Rebecca is casually racking up graduate degrees and she saw that she could leverage some academic principles here specifically she looked at leadership trait analysis and integrative complexity leadership trait analysis is a technique for analyzing what political leaders say in order to assess their leadership style it literally gives us a manual for taking their words and turning those into a profile of a leader that includes factors like what motivates them or how they might react to a new piece of information obviously that is very actionable information integrative complexities similarly looks at the structure of the content instead of just the individual words used to

create a profile of the subject Rebecca used these frameworks to create a sentiment analysis process to identify content she could ignore her automated tools scans the text and identifies articles that show bias demonstrate low confidence in the assessments being made or have a lot of opinions or conjecture in them that way she doesn't have to waste your time reading that rubbish all of her code is up on github if this sounds like something that could be useful to you so check it out and let's take a moment well everyone's taking a picture of this sweet link to remind ourselves that our experience in other domains is relevant and useful again our industry is only about 30 years old the

best practices that we're following were not made up that long ago if you think you've identified a better way to do things you don't have to stick to the status quo just because you don't have a CISSP and yes I am one of those people who attack certifications because she's insecure about not having them and once I get my CISSP I will never stop talking about it and it will be on my resume in the top section where other people's names are and I will get a tattooed on my face anyway Rebecca's story is great but what about us normal people who don't have expertise in a dozen different fields well I would consider my thought myself

a thought leader in these areas it's not immediately clear to me how I would draw on them to complete a project like Rebecca's well there is good news for people like us it turns out we don't have to be the individual with the expertise from other domains to solve in post type problems we just have to listen to them let it delight Nazira cartilage a senior manager of product security at Dell told me that two of the stars on her team have backgrounds in QA and operations obviously for their team to be successful they need to build effective processes that involve coordinated cross-functional work these two team members are exceptional performers in these areas and that makes a lot of

sense to me I am positive that someone with a few years of business analyst experience could come into my team and clean up a whole bunch of our processes for us I should also mention that her her younger sister is on my team and we definitely did not anticipate she has a master's degree her undergraduate degree is in accounting and her ability to function really well with auditors and our security governance activities it I definitely it's not something I expected before she started her job and it's incredibly helpful we can also go beyond just listening to people with different backgrounds to help clean up our processes and solve our problems we can also amplify their voices Colin Morgan

is the director of R&D and product security at Johnson & Johnson product security went from being Colin side project to a full program in 2016 and conveniently in just a few weeks later J Radcliffe who was a researcher at rapid7 disclosed a vulnerability in the OneTouch paying insulin pump system after discovering that an attacker could potentially trigger unauthorized insulin injections again Colin's product security program was brand new so for basically anyone outside of Colin's team responsible disclosure was a totally foreign concept and understandably a lot of people initially saw J as an adversary gen Ellis our VP of community and public affairs had a lot to add to the conversation around the best way to

disclose it was a very complicated situation in an ideal world you could spend some time strategically reaching out to individual users without telling the whole internet but unfortunately as soon as one person tweets a picture of the notification they received in the mail it might as well be written in the sky from his position of power at Johnson & Johnson Colin was able to bring J and Jen's perspectives and expertise into other areas of his organization and help everyone understand that they were all working towards the same goal of patient safety rapid7 does a lot of security research but internally I've heard Johnson & Johnson sighted more times than I can count as the gold standard for Volm

disclosure partners responsive solution-oriented effective really what more could you want and that's all because Colin was able to effectively amplify the input of people out his organization with very different perspectives internally so those were some key themes I noticed about who was solving the problems and what they were doing individually to put those solutions in place but I also notice notice some patterns in what was going on around those people maybe you're thinking I just need more resources more time more human capital more money and then I could solve these problems and that may be true but I also know that my team very easily could have poured all of the additional resources we've gotten

over the past two and a half years into those little fires without ever tackling the big projects that we're very proud of and we definitely never would have run out of work to do so more resources alone won't solve our problems we have to be strategic about how we direct them first let's consider why do a lot of organizational structures push us to focus on little problems instead of big problems one of my friends has a job that's very similar to mine but his company is about twice the size so I pick his brain a lot because it feels like a little glimpse into my future as I mentioned earlier it is completely reasonable that both his customers and

my customers need to know what sort of controls we have in place to protect their data and it's on us to address their questions and concerns and we love doing that in his talk yesterday poor no doopy did such a good job describing how these types of enquiries make us a better company they force us to think about risks we hadn't considered and they push us to prioritize projects that are important to our customers but he has warned me that as his company grew and particularly as they started selling into industries with more red tape fielding these sort of inquiries from customers and prospects about his company's security program became increasingly challenging he kept ending

up on long calls with the customers risk management departments going over and over some minut detail about his security program maybe this process is more difficult for him because his company doesn't make security solutions the person in risk management he's talking to is probably 15 silos over from the person who's using the product and that may lead to some miscommunications or maybe I'm just so much better at my job than he is it's impossible to be sure but I'm worried about eventually getting into these situations because in the long term after he's already taken three calls with that customer and they're requesting a fourth to revisit something that he genuinely believes is a non-issue I would argue that it's better for him

to spend his time building a state-of-the-art security program that speaks for itself but it's easier for him to justify getting back on the phone there's more immediate pressure for him to choose that option from sales and from other members of the organization who have eyes on that customer relationship and it's easier to put together a business case for taking that call than it is for his long-term projects when it comes to fielding that individual customer requests it's easier to articulate the problem and anticipate the return on that time investment even though taking the call might be a misallocation of resources so whether it's responding to customers or incidents or service tickets I know that everyone in this room deals with some

version of this problem when environmental forces are pushing us towards the little fires how do we make time to fix the big stuff a lot of people I spoke with were able to solve these problems with some crafty metrics one approach is to proactively set expectations and boundaries about how much time you will spend on operational work for example that same friend is currently experimenting with giving each customer a set amount of time with the security team to address their questions I'm going to let him try and fail with this approach a few times until he comes up with a really solid solution and then I'm gonna put it in place at my company and I'm gonna look like a genius and I'm

gonna take all the credit another way we can use metrics to drive resources towards meaningful solutions is by establishing KPIs or key performance indicators around our projects Nazira cartilage who I mentioned before in product security at doe share how they use one KPI to ensure the problem she's worried about is getting adequate attention and that's on-time remediation rate for vulnerabilities the individual product teams are responsible for reporting on that KPI and that puts the onus on them to come up with an explanation or a resolution if that KPI dips that creates an environment where resources are automatically moving towards the problems that she cares about when there's an issue once again super excited to just brazenly steal

that approach implemented at my organization take all the credit okay now let's talk about my friend Alex he's on the incident response team at Atlassian and this is our signature pose alex built an amazing web app and i timed this talk this morning and i just don't have time to tell you about it but it's so good and i'm gonna like tweet about it and he's so smart and great and you're gonna love it you're gonna be like wow alex you're so smart how did you build that I guess follow him on Twitter he's so good at memes so good at memes I probably could have told you about it in the time I just spent doing that when I

when the point is when I asked him what empowered him to put that awesome solution in place he mentioned that the idea for this solution was hatched during 20% oh I lied it was hatched during a hackathon and he finished building out the project during 20% time which is time that Atlassian employees have to pursue projects they're passionate about there are a lot of organizations that set aside time specifically for solving big problems like this the post-it note was created by a 3m employee using his 15% time to try out something new which I would like to point out is less time than Alex had to work on his pet projects but he still hasn't been able to develop the most

iconic office-supply ever that's kind of embarrassing for him hackathons are another interesting way to set aside time to work on building solutions and I have mixed feelings about them a lot of them go overnight that feels really unnecessary for me I'm a very tired person I get tested for mono every time I go to the doctor because medical professionals don't believe that a human being could possibly need this much sleep to exist so this format is an ideal for me but it's probably a bigger problem that we often and I would dare to say usually lose momentum on our hackathon projects as soon as the hackathon ends in Alex's case he used the hackathon time to tackle a very real

business need and he was able to continue pursuing this endeavor with 20% time that's the ideal state for this that's how we want to see these things work a lot of times I think that these projects die out because we didn't document why they were a problem worth tackling to begin with I think it would be interesting if there were some forcing functions that made people connect their project to a really good problem statement like the ones we talked about earlier I did discuss this predicament with Rebecca Brown and we decided we're going to host a documentation a thon where we document the business justification to make various changes and no one will show up

and we'll have all the snacks to ourselves so joke's on you another environmental factor I saw again and again was that these problem solvers were in a place where challenging the status quo was encouraged now that's a very nice blurb that you might see in a company's mission statement but what does it really mean I keep referring back to the projects my team has been working on lately and one of those is our new security awareness training program my team made a list of everything we wished everyone at rapid7 knew about how they can help protect our organization and we turned it into a script the approval process consisted of us sending it to the chief product

officer and the head of pupil strategy and there was no pushback on the concept honestly they just pointed out a few places where we had kind of typed gibberish words that didn't make any sense and they were like you might want to make these not gibberish we're like Oh true oh yeah let's make those into words so we cleaned up the script with their feedback we recorded it and we pushed it out to everyone and it was absolutely awesome however every time you pause the recording I'm making a stupid face like this and also we tried to do a jump at the end that didn't go great and also I added this last slide and yesterday

after I found out that we were going to be presenting this on an IMAX screen because it's funny I'll just hold it on this for a little while in case anyone wants to tweet it at him yeah no take your time that's really tough to take from the front row that's not a great that's not a great angle yeah someone's gonna have to send that to Keith cuz he's not he's not getting a good shot so if I was in an environment where making wacky homegrown security awareness training videos was risky candidly I wouldn't have done it because it would have been a lot easier to just use a boring module from a third party with

information that isn't really customized for our users and no one would really pay attention to it but I could say that I sent it out to everyone and I could check that compliance box I definitely considered that some people might not love it but I really am in an environment where trying out new things is encouraged so I'm not gonna get in trouble my worst case scenario is we don't get great feedback and we won't do it again but we did get great feedback and we will do it again hopefully today you learned some stuff about how you can fix stuff that you're passionate about and how you can build an environment where the problem solvers in your organization

thrive thank you very very much for having me and I hope you have a great day too of b-sides and all of you are special gems and I really like being around all of you and also all the volunteers are the best things back [Applause]