A Security Champion's Journey: Making Things A Bit More Secure Than Yesterday Every Day - Lisi Hocke

so hi everybody my name is lisy my pronouns are she and her and today I'm going to share with you my journey as a security Champion so you'll hear about all the things that helped me and my development team make things a bit more secure than yesterday every day but also what didn't work and what we shouldn't have done in the first place so welcome to the ride I'll split this in three sections so first one I'll share also a bit of how I unlocked security for myself so you know where I'm coming from the context I'm speaking about and then the main part will be on how do we Advocate Hands-On from inside a development

team finally we also need to extend our reach we're just one person so that's going to wrap up the talk but first things first so where am I coming from um let's let's give you some context so hi again I'm a specialized generalist I've always been a generalist throughout my life actually I just love learning I love Tech and um this really fit well with uh working in testing and quality related roles for 15 years now and I've always been um working on product development teams so we owned a whole product or a big part of it from idea to production all the way through for that to do we need all the skills um on that skill so on that

team so we were always cross functional with all the roles that you need to do that personally I've been embedded on those teams just as well so Hands-On building things together and making them better every day now working in tasting quality means you see everything it's like quite a broad spectrum that you can get that you can involve yourself basically from architecture decisions to building features yourself fixing things operating infrastructure increasing observability and production and all the things and I just love it now how does security come into play here so it started for me in 2017 at a conference an open space conference where someone suggested hey how about we do some a Juice Shop challenges together

vulnerable web application to practice on like really nice thing first time for me to see something like that and I was like oh that's actually a lot closer to what I'm anyway doing all day and just got hooked I just wanted to learn more so I also tried to bring in that kind of security mindset into the teams that I worked with little things like introducing threat modeling and thinking of risks early on basic security testing these kind of things and also continue to learn in my free time uh I love personal learning challenges I usually have an annual theme what I do so things like also diving deeper into security connecting with with the security

communities coming here also last year and well also sharing back with the communities speaking at conferences what I'm doing for some time now now also about security which actually brings me literally here so around 3 years ago I also changed companies and this company well you can imagine like roughly 300 people 20 engineering teams for me it was quite a new things because new tax stack first time that I was on a mobile team I was like okay lots to learn but also it's healthtech so definitely a new domain for me I've never worked in that before and that also comes with regulation so lots of new stuff and I did find out well they had a quite

active security team already getting involved from onboarding sessions onwards and I couldn't believe my luck they had a security Champion program before I just heard about these things that other companies have and now I joined a company that actually had one in place I was like can I can I sign up somewhere until I figured out actually my mobile team that I just joined well they already obviously had an appointed Champion H what can I do can I still involve myself somehow well working in tassi and quality um I learned over the years that sometimes you have to invite yourself in so it just you know join the security Champion slack channnel like can I get

involved here can I engage yes of course I can uh any kind of security related session my teammates are doing oh yeah we're looking at mobile device uh requirements we have a session with the security team can I can I listen in yes of course and I found out I can also contribute so I really got myself involved now what is that security champion in a nutshell basically so you usually it's someone not being part of the security team but embedded in their own team mostly a development team but doesn't have to be they're acting as a communicator between their team and the security team and it goes both ways so they share important news and things to

do with their development team but also the context of what they're currently working on all the challenges that they face back with the security team so we can find Solutions together and they're advocating for security improvements handson in their team which can mean spreading awareness but can also mean really tackling security issues Hands-On fixing bugs maybe introducing tools whatever is helpful so such kind of a program can really help scale the efforts for um the security names teams um scale their knowledge and expertise across teams you always have a first Contact point to reach out towards and you can also have your influence way earlier in the process because these are the people that are right there in the room when

the decisions are made but looking at those core aspects I felt like I already was a security Champion even before being officially appointed and this also showed in My First Security audits so uh I was invited in the into the auditor sessions and I was also introduced as a security Champion I was like okay they see that um I was usually quite nervous whenever it comes to audits like ah scary but I found out it's a great learning opportunity even if you can join the live sessions but you're just joining preparation for an audit like what I found out was taking inventory is actually very crucial know what there is that we need to protect so

what is there to protect all the components that we have in our system what kind of services do we have we usually have an architecture diagram or something that can you know really help us inform that and which of those components are public facing because that's even more crucial to protect think of public API endpoints public buckets public repos whatever you have now how confidential is that kind of information that they provide is it maybe public it's totally fine that's out there but it could also be confidential or it may be top secret and what if these kind of assets would be breached what's the associated business risk that comes with it it's very context specific so it's

good to have that category what kind of security controls do we already have in place and where are the gaps all this really really helps Foster that awareness now time has come that our security champion in the mobile team uh left for another company for new Endeavors and I was like that rule is not free maybe me so it was quite an easy one uh because everybody was already working in that mode anyway everybody was fine my team included so here came the phase where I advocated even more Hands-On now working with our security team and all the policies and policies and the audits well there's there's a bunch of guidelines for every team that's very useful but there was nothing

really specific to our team's contacts especially for mobile so I sat down and wrote a mobile app sex strategy something really specific for us what are our goals where do we want to reach where are we now how do we plan to get to where we want to be and just writing this down and making it explicit it really helped align my team and a security team to what we actually want to achieve and how we can achieve that it also served as a reference whenever possible so when we talked about our security posture with our stakeholders there's a reference document to point towards it helped kep keep us focused on what's most valuable to do next of all

the possible options that we have and it also allowed us to measure our progress how do we uh did we Fair did what we did in the past did it really bring us closer to that goal now writing down all the things does it solve all the issues no of course not but it's a great starting point so we really know what to tackle next our starting point based on that uh strategy well was definitely our tooling we had a lot of Tooling in place lots of things in our pipelines and it was quite noisy so think of for example dependency Checkers yeah when they're like shouting at you all the time this kind of

situation tooling that that keeps telling you what more of what you already know that you're not in a good shape it's not really helpful so that's where we started from so we had situations like this we we had this very very important feature it needs to get out of the door but the pipeline fails because a dependency Checker says oh there's a new vulnerability detected fails the pipeline and we're like but we have this pressure we need to get it out we don't have time right now what can we do oh let's just suppress it let's look at it later yay pipeline is green we ship it everything good right well that thing is solved right now

right and that we look at it later what if the next priority comes and later is basically never nothing is fixed now in this kind of situation it doesn't help to throw more tooling on it it just doesn't help it just makes it worse and helps the team to learn more like to ignore all that kind of feedback because we need to remember that people usually have good reasons why they behave in certain ways and reward systems is one of those big reasons if the incentive is on releasing and not on security guess what happens so instead what can we do we need to clear that kind of noise and that can also mean really removing all

those tools again and just thought one by one because any kind of signal needs to get through and needs to be a like people need to be able to hear it we need to be intentional what kind of tools we really use and also teach people on how do we even respond we get an alert what to do next how to even how to evaluate it what to uh what to do to actually solve the problem so better less than more and helping people relearn speaking of that dependency example because yeah that there was a red thread um in our situation dependencies big big topic so now we know what to do right uh let's

just update them easy well let me ask you a few questions uh you can show your hands if you want to but also just oning your had uh who here Works in a company on a product where you have third party dependencies could be you know paid or open source doesn't matter yeah I'm not alone yeah I guess so keep your arm raised uh if those product dependencies are outdated or at least some of them yeah easy right they're outdated the minute you update them at least in some ecosystems keep your arm raised if some of those are not only outdated but actually deprecated not maintained any longer so you have to remove them yeah okay so we're not alone it's

it's a common scenario quite a sad one too so what to do about it just do it we literally didn't find any way around it it just makes it worse like sitting it out is definitely not an option here we really have to do it and we have to do it often and regularly small batches actually also help reduce the risk as we anyway learned in software development keep it small reducing risk but we have to do it so let's tackle the things where the risk is highest first for us that was our public facing components that where for us was the the highest risk but it could also be the main defend offenders what

whatever and then uh what really helped us was to tackle quick wins first things that you know just upd you can just update it it it's working fine no need to worry because this way you can clear out noise again you can practice and get into the habit of really updating regularly and often but sometimes you have those big things big framework updates oh my gosh you pull one thing and you know everything falls apart yeah what helped here is to write things out what is really changing in those versions what do we need to change because usually you have to adapt your implementation uh towards the framework update and it can be a big big mess so

writing things out to have really very high Clarity helped us to then just tackle one slice after another and reduce the cognitive load that comes with it and once you have an update strategy that works for you in your context cling to it like hell because otherwise you're back to square zero very very quickly and yes we're all humans and there will be stressful times and we will fall back to Old behaviors but let's catch ourselves as quickly as possible because yep not going to help to sit this one out speaking of technical debt like outdated dependencies well one thing that um really resonated with me was was when I heard Tanya dunker say that

technical debt is security debt and I was like that makes so much sense in my world it really resonated heavily and it also provides leverage because fixing technical that is actually also good for security more arguments to get these things prioritized one thing how it can show up is outdated implementations never saw seen this one right but anything outdated can increase our attack surface yeah could be old features still left in but nobody's using them anymore deprecated API endpoint versions uh feature toggles that are not cleaned up or any kind of config or Secrets especially if you inherited a long running Legacy system and you know people on your team like never worked with certain parts of that

system quite a common scenario as well yeah it's often quite scary to touch this part that nobody knows of like is it actually even still in use well how can we even see that there's a lot of uncertainty usually but it doesn't help we need to clean up and we should make clean up non-negotiable we definitely did that wrong in the beginning we had migrations from one service to a complete new one and we didn't clean up for 2 years yeah that didn't help because it stayed in the backlog it stayed in our heads it stayed in the implementation so also pen tests and buck bounties and vulnerable um like coordinated disclosures and stuff they reported on

that outdated part of the implementation so just going through all those things and gauging like what is it really about oh no it's about this SDK that we should have removed like long time ago it's a waste of time so nowadays cleanup is non-negotiable as soon as we can do it thinking of Backward Compatible but as soon as we can do it we're cleaning up now speaking of things that are that we leave in for too long but there's also some things we that just should have never made it in in the first place so letting implementation in that's insecure by Design that's that's quite another sad part and it's so hard to get it out again if at all and sometimes you

know up front already like a this is this is a bad decision like no let's not make it in maybe you can't veto hard enough sometimes we just learn about it in hindsight and it's already in there but if it happened because it might we're human get it out as quickly as possible again right away because with every pausing day it's going to be more and more acceptable that this is in there and people forget about it or what was bad about it in the first place even worse people start building on it so we developed new features on it our business departments like marketing is using that kind of feature our users of using that feature and building their

processes around it and so on so let's prevent bad things um making it in how to do that well think of those risks early on we can do thread modeling or use malicious user stories or personas you know anything that works in your context whatever works literally to think of these risks early on and then we can we have to act on the identified risks because otherwise it's the same situation again it doesn't stop with risk analysis though but even for that we need to educate otherwise people just don't know what to even think about so educate educate educate what kind of risks are there why is this a problem what back could happen and it shouldn't stop with analysis it's

also about secure coding basically security testing knowing how things could show up on production and what we can alert on speaking of production so it's anyway good to know what's out there and how your system is actually behaving so observability is a big thing also for security let's observe frequently and regularly and use that information to make our next well informed decisions here's well constant attempts of uh attacking our systems can be observed left every day probably for everybody well we've seen a lot so we should also have ways to get alerted on things that we already know is happening which could be those kind of malicious uh activity like probing our system like H what kind of

TX St do you have here are you a PHP server is there WordPress implemented here or whatever people try everything right but it could also be potential leaks that we caused ourselves that we shouldn't like we don't want to leak personal identifiable information in our logs right we know this is a scenario it can easily happen let's let's test for that and alert on that early on so we hopefully even catch it before production we need to have approaches to keep the bad actors out yes make it harder for them to get in so basically not leaving the door wide open but keep it a bit more shut that could be web application firewalls and rate limiting and these kind of

things and yes it's not everything but it's a layer of defense and it's a good place to start for people to also see what's actually caught by that and what not we need to also learn from external feedback like pent tests or bug Bounty results or any kind of feedback that uh comes in because this can really show us what people try try what kind of attack vectors they use um to really get into a system and then we can also we need to use that input obviously to improve our system and last but not least while the the world is continuing to turn there's new things found every day new exploits new vulnerabilities new tools as well

and techniques let's pay attention to that and see how it fits to our contacts in our case that was like oh polyfill I take over H that might hit us or the cocoa pots Supply CH chain attack as well this year things like that we need to check how does it apply to us and respond very quickly lastly we need to also talk about prioritization or the problem of later quite a common observed situation everybody agrees this is a security Improvement we really want to do it's really important and then we all agree to do it later and there's lots of reasons for that it could be lack of knowledge or skill or it's just like

we're not really certain how to tackle it maybe something else is just easier right now or we're so overwhelmed we just don't have headp space for this thing but having topics overrule security constantly that that's a big problem like this we do that when things are calmer I mean that's basically never and sometimes there are more important topics indeed in my team that where like we saw constant fires all the things like things that would really block us from releasing yes we need to tackle that first but not everything is a fire there's a lot of things that are negotiable because what happens if we let all those security improvements that we want to do rot in our

backlog well we implicitly accept the risk that comes with it and that's what we need to communicate very very clearly right now this is implicit in the backlog yeah we're still going to do it in 2 years I don't know instead let's register those risks explicitly and really document what's the problem how would it impact us how probable is it actually in the business context and then we can also decide together with our stakeholders what to do with this kind of risk do we need to mitigate it it yes well then let's do it right away because it's really important maybe we can remove that risk by maybe changing that implementation removing the outdated implementations whatever

refer it or we can also accept the risk because it's not too bad things can also be accepted it's just like we don't fix every bug because some are actually acceptable we can also accept certain security risk but we need to decide this together and if we do let's close those backlog toss because otherwise it's again just noise now time has passed I've been the security Champion now for quite a while uh we've tried a lot of things in my teams and some yeah worked some didn't really but looking back I found a few things that you know made the difference now as a security champion on the team I learned to really Foster relationship

on the one hand well it's been invaluable to build up that really good relationship with our security team uh I also really personally enjoy working with them because I really appreciate their very collaborative mindset very pragmatic like truly working together with the teams listening to their contacts and being in there together like not condescending or gatekeep at all but quite open and appreciative so huge shout out to them but I can't do everything on my own either in my team although I I mean I'm just one person but I might also be on vacation at times sick leave speaking at conferences I'm not always there so it's important that I take my team with me

just as well and also connect them with the security team so whenever I'm away they also know each other already and know whom to reach out to when I'm off also this means I we're not missing out on their expertise because I don't know everything either that I mean there's a reason we work in teams right we have all the knowledge together I'm not creating a bottleneck this way because bottlenecks will always slow things down not really effective and the good thing as well is I instantly have allies on my security topics they know more about it I'll have ad like people to Advocate with me together in the team and as my team is now very aware we

can then also experiment together to figure out how we can tackle all those things how we can prioritize for example when do we do this what we uh tried and what really worked well with for us was using 20% time this was a general approach for us uh it was a non-negotiable time that we already had you know like apart from the road map and my teammates used it for example to drive refactoring topics or cleaning up or observability tooling the noise there which by the way both also help security again I focused on security improvements we all had a win with that but not everybody has 20% time or the chance to do that so you really have to figure out

what works in your context try things out small things that don't hurt that where you can learn very very quickly and then do it now I had this great collaboration with my uh security team my teamer mates were on board great but sometimes I felt quite lonely because like where's my fellow security Champions can't we connect and learn together more so engaging those security Champions was also quite key for us to share more in our share Channel sharing blog post sharing about conferences like this one uh asking questions how did you solve x um whatever is working but also bringing people together at the same time we had weekly Security reviews uh thinking of the like checking the last

week's news um trying new tools that had been recommended which really uh engaged interesting conversations together and also biweekly Hands-On sessions where security interested people can just bring their challenges to the table and we work on them Hands-On together but there's also more spaces to learn with because maybe you don't even have a security team or champions in your company but there's a lot of communities out here like here literally to connect with so connecting outside the boundaries of a company can be really really valuable there's a lot of like-minded people out there to learn with and grow together with because we're all figuring this out but why not you know keep learning with those allies

and learning and figuring things out together again I'm just one person and I can make things more secure for my team but we need everyone we need my whole team I need my security team I need my fellow Champions but I also need the support of management and business to give us the space and time to do that kind of valuable work security Champions they will have very different backgrounds different skill sets some might be very specialized some might be generalists like me and all of that is a good thing because there's strength in diversity given we Foster inclusion and Equity because we can actually all learn so much from each other for that to happen

we need to actively include folks make it safe make security a welcoming space where people are really feeling oh I can see myself there I can learn is with everybody knowing more about security and learning more we're all better in it and we're stronger together when it comes to my development team and me well we tried a lot of things some things worked some didn't some yeah we shouldn't have done that in the first place but what worked what worked was really to invest in the security culture on that team level among Champions across the company level to really raise the awareness and finding those allies and improving things step by step and continuously what didn't work well not

using our veto strong enough like not to prevent those insecure implementations making it in to not suppress that alert right sometimes really to stay up there and gatekeep when basic security controls are missing and well what we definitely shouldn't consider it's not doing anything at all it's not working like keeping those open security topics in our backlog let them Rod there nope that's not an option we don't always know what to do but that's where the experimentation comes then we'll have to figure it out we can always experiment to see what's going to work but did I find this magic recipe is everything great now no what I found is that the journey is the

answer what we need to do is to keep going keep raising awareness keep taking action and keep learning from the outcome so we can do better opting for many small steps continuously small steps already bring value for everybody not overp polishing some stuff small steps delivering you instantly make that constant progress so you can also get timely feedback and you can very quickly do cause Corrections mitigating risk through small changes always a great thing and we have a lot more wins we can celebrate this way because also small wins you make a lot of them they accumulate optimism and enthusiasm go a long way because in the end it's about always being a bit better than

yesterday so we're at the end of this talk if you take only one thing with you let it be exactly this let's make things a big more secure than yesterday every day well that's my journey so far as a security Champion I hope you can take a bit or with you wherever you are whatever context you have I also like to gather resources that help me on my journey so you will find them uh behind this link and in case you have any questions I can answer right now because now is lunch time um you you can still find me around in this conference uh or on social media and most active on mustan you can also find further links

on my website and with that like to say thank you