My CI/CD Pipeline Contains All Security Tools Available! Now What...? - Jasmin Mair

[Applause] thank you very much I'm very excited to be here and talk to you about yeah what we can do to make products more secure instead of just adding tools um so thank you very much foran for introducing me I'm yine Meer I'm an Italian living in Germany since a couple of years now um I'm passionate about secure development about security itself but also about agile methods to implement security or to implement uh product my in my past I was working as a consultant as a security consultant working with different customers and clients around Europe to make their products more secure and their development life cycle more secure now I'm a product security manager at Leica

Microsystems and I'm working there with all development teams to make sure that the products are secure by Design so what are we going to see today um we'll be looking into the Temptation everyone had as of buying more tools but then also taking a deeper look at what teams are really struggling with and we'll try to figure out way out and to start implementing Tools in a responsible Manner and I'll also have a small uh practical example that I'll be showing you ready to go who has heard of any data breaches lately yay every body hears about them um everyone is confronted with companies bringing out reports talking about how companies get breached how many incident

happen how do they happen and where and everyone's confronted with news news of data breaches companies ransomware being um yeah disrupting companies and who of you wants to be associated with such news in the headlines none of you good I mean there could have been somebody what are people thinking about um how can we solve this everyone has the same thought we want to become more secure we want to to stop the breach we don't want it to happen and the first the Temptation that they have is let's buy some more tools um we have development teams working in a death setups or a waterfall approach that doesn't doesn't matter much um what tools can we buy

well actually one can start very early in the process and Define and Implement and buy some tools to secure their yeah their planning process do secure architecture security enablement one can buy tools to secure the software development when the developers actually write the code static application security analysis software composition analysis um and all types of tools that you can have there then we can Implement some tools in the CI pipeline when we build the the the the the application scan for the containers for some vulnerabilities and so on and so forth we can continue and Implement tools when we start testing the application and as we go down the process we can buy tools to um to scan

our infrastructure to scan what's happening in the in the operations and to monitor them throughout so I hope you hate this slide as much as I do um and if I forgot any tools it's not that I support any of these or all of them uh but it should just overwhelm you just as it does with me it gives me some anxiety and let's think about what are the development teams confronted on a daily basis they have their pipelines with existing tools and everything we buy just adds on top add complexity on top of it I had had a customer when I was a consultant that came by and said Jasmine please help me um I have so many Tools

in my cicd pipeline but still at the end at every pentest the pentesters find a ton of findings and they were not figuring out what was happening and then we took a deeper look and I we just saw that the tools were there they were scanning no one was actually looking at the results the results weren weren't being included in any of the work the developers did and while digging a little bit deeper we just saw or I saw and figured out that the struggle was somewhere else who who can relate to this image developers and security Engineers Security Professionals um are fighting on a daily basis not because they want to but because they often times have

conflicting um responsibilities they are measured by other things um for example developers as a Le also mentioned fast development bringing the features fast on the market while security people oftentimes have a little bit of another thought on their on their minds Security Professionals can and are oftentimes perceived as a blocker to the development life cycle we're raising the red red C all the time no you can't do this no you can't do that and blocking everything um around but then what makes it even worse is the this tick in the Box approach to security do you have a SAS tool implemented yes who cares if nobody looks at the results tools are helping but they

aren't adding any value if they're not implemented in the right way while develop oh

sorry I hear some laughter who's guilty of having an security requirements Excel spreadsheet somewhere don't be shy every one of us has if the work doesn't happen where developers actually work that works not existent it's going to stay somewhere and be um yeah nice spread sheet somewhere and as Ali also mentioned before you can't buy def zups what can we do instead instead of adding more tools right away we can start with implementing a security culture which means Security Experts embedded into the development team we want to clarify the metrics so what's the what are we achieving what are we trying to achieve together um so not measuring the Security Professionals by how many vulnerabilities are in the system or a

zero vulnerability policy right away but instead aligning the teams and teaching them also to work together a very crucial part on it is aligning on the security priorities if we don't know what we're building if we don't know what risk appetite an application or a system has how can we Define reasonable and sensible security measures to to prevent what could go wrong developers have been developing since like more than the last 5 years and they have been doing it in their own way without taking security into consideration right away which means we need to Foster this security mindset there's a lot of focus on enabling developers to become more secure to write better code to to write

secure code and so on and so forth but it doesn't stop there we need the awareness also on a management level product managers product owners need to be enable to understand what's happening in the world and how they can can work with it this is why establishing a security Champion program can also be very reasonable and and helpful to then also train the developers and last but not least embed Security in each step of the software development life cycle and then based on the priorities based on the risk appetite of an application one can then start working and act accordingly what can teams do in order to get started teams can um do plenty of

activities that do not require a tool and there are some that I'd like to highlight um for example application profiling how can we Define good security measures if if we don't know what we want to protect are we protecting pii data are we protecting Health Data or are we protecting public data there are different measures for each one of these areas another area that I really like is threat modeling threat modeling is helping um reframe the problem and helping developers see and try to put ourselves put themselves into the shoes of an attacker what could go wrong what could I do deliciously with an application and then we can also start with software composition analysis very

early on try to find out what are the ingredients what I'm working with um I've read a um or I've heard of a very worrying statistic I wasn't able to find the the actual source of it so I need to double check that one that was mentioning that in 2019 round about 60% of the application code of specific applications was based on third parties so more than half of the code can come from third parties we should definitely take a deeper look into the security of those as well once one decides what one can do um in each step then there can be a time to start buying tools but let's not start with all at

the same time when we when we Implement a tool um I've seen it often times that that teams just buy 10 different tools maybe 10 is a little bit of an exaggeration and Implement them all at the same time and then we exactly end up in the situation where nobody is looking at any of the results if we Define and decide to implement a tool we should set a clear scope so a tool is just as good as the scope and the goal it it it follows and needs to to to cover then we need to start enabling the developers to manage the findings where are the findings where can we find them um what are the

priorities and then we are able to scan the existing codebase scanning the existing codebase can be a very very painful step because the results can often times be really crushing and then we can work with these uh in scans fine-tune the scans um root out false positives maybe add in some false negatives if we find them and then comes the most important thing and it's create meaningful quality Gates I'll be going into the details on how to create meaningful quality gates in in a second and then we need to make the work visible we need to bring the work the security challenges that are in the code to the developers we can't expect devel developers to open up three different

Tools in three different systems and interpret and be able to to manage all of that then we need to help them prioritize the findings we will see it in a moment when we scan we do this first scan we find a ton of findings how should a developer be able to prioritize what's the next step he needs to do and in this we as Security Professionals are key to help to enable them to prioritize and to visualize the work so when I Was preparing this this talk I thought um of adding in an example and I was trying to find an example from my past um as a consultant but I wasn't able to share any any

details of any of the customers I've had and so I thought to myself well let's build a tool let's build a pipeline um let's do something with it it isn't going to take much long too long oh boy was I wrong um what did I do I did decide to implement a static application security testing tool to the OS Juice Shop who of you knows the OS Juice Shop nice um for those who might not know it it's a vulnerable web application that is developed by the OAS foundation in order to exercise and and yeah test out security vulnerabilities and so on I worked the OS Juice Shop and added it into GitHub where I had my my repository so

you will be able to check that one out if you want and then I decided to start with a cicd pipeline with GitHub actions why did I use GitHub actions GitHub actions is um or GitHub has some very nice security features that can be used um for open source projects for free is why I went for this technology decision and then I started to take a look and by default you can run software composition analysis in analyzing what are the packages that are inside of the the application and for that I use dependabot in the second step I used linting um I used um es lint to check for yeah if I'm able to find any um any

challenges there how the code quality of the product and the project is in that perspective and last but not least and we'll be focusing on that um in the in the next slides I implemented three different static application security tools um why did I exactly choose these because all of them are freely available for um open source and open projects so was the only reason for that I started with running a manual scan and then when I run the manual scan this was the output and as you can see through three different tools one code base very different

results oh okay okay um just let me very I think there's an outdated version of my slides but anyways doesn't matter um when I started then I thought okay um let me Implement a cicd pipeline um who of you knows what a c CD pipeline is okay very good so a cicd pipeline is just an if this than that change if this if I um submit a code change then start start some scans if the scans are successful then continue down the chain until the last step should be then deploy the application managing the findings is a hard thing and when we usually have developers being confronted by themselves with these results in their cicd pipelines they're just looking at

them and saying oh oh boy 200 findings oh no I'm not even going to look at them so what can developers do um we can help them prioritize the findings based on severity we can for example state that we start with the most critical ones look at based also on what they're building um working with them to assess the um yeah the criticality what they can do um one one type or one way of clustering them would be for example by vulnerability type for example we can start with um hardcoded credentials you've mentioned it also in your talk what what are where are the hardcoded credentials where are the passwords to my system in my source code um or for

example SQL injections and then um with very big applications developers could or could be worrying about where to get started and we can cluster them by um finding out if there are some affected modules that we need to prioritize first so if the there's microservices one can really start with one microservice at a time and so on and so forth so how many injection flaws were there found so 1 found 7 116 so this really shows us that we as security profession need to help the developers make sense out of these findings how many are there what are they what do we need to do and are there maybe false negatives or most importantly false positives because if

we start implementing in our pipeline an automated way to filter out or to find such results it is very bad if there are false negatives so flaws that are not really flaws because they they are F they are not real in the application that would mean that the developer is always taking a look at something that's not applicable for him or her moreover the findings and the outcome of tools is and can be really complex it's not very developer friendly so I've uh before this um presentation I have submitted or I've talked to some developer friends of mine and if they are senior enough they are good at also interpreting and finding out what they

what they went wrong or where they did something wrong if they are more Junior they often times don't really understand what's what's what's going on there so working in different uis um as we Implement different tools in the pipelines is also adding to the fatigue of the developers and what they are doing in general to to in their work what can we do to fix all this so how can we Implement a tool in a responsible way and this is in an approach that I have used with multiple teams in the past and it was very successful but it was also quite long and tedious we selected one vulnerability type for example injection and figured really on that one

vulnerability we AED the developers to understand where they went wrong we gave I gave the developers working examples of how it works in their programming language how it's working in there and how they need to do or what they need to do in order to fix this so raising the awareness was very very crucial but also enabling and training the developers and then we started to fix the vulnerabilities that was the first point where we really really implemented the tool and the quality gate so um if you're familiar with cicd pipelines you know that you can build in a quality gate and set some kind of threshold for example if you find another hardcoded credential stop don't continue with the

build don't deploy it give it back to the developer and tell him what they did wrong um and that was the point where we started introducing that they started to fix the vulnerabilities and after that we created this quality gate in order to make sure that every new line of code that was submitted was definitely not including that vulnerability and then we start you start from scratch you go to the next level so you fixed all the hardcoded credentials activated the quality gate for the hardcoded credentials then you go on with SQL injections injections overall cross-site scripting and so on and so forth and while this process is and can be long for companies we know

that enabling and embedding a security culture into a development team that is already working can be a long run and you do this um in order to in the end go on and have more secure products while working together between the development and the security teams while I was doing this I have to be honest um I thought well the cicd pipeline is going to be fast it's it's not going to be a big deal nevertheless I encountered so many issues with giab actions it was so so painful um for example while I was doing it I had um I had challenges with the secrets in order to make the the the results available to the different

platforms and I know I should not put Secret public in GitHub nevertheless it would have been much much easier to just hardcode the credential inside the code and push it there instead of going the way I did it with environment variables it cost me 30 plus runs for each tool that I implemented in order to try to make it the wrong the right way this this really shows that the tools need to be intuitive not only for Security Professionals but also for developments Developers for operations people for people that are um regularly using them and implementing them into their pipelines because that is the point where the real value of the tool comes out the tools will not be helpful just

by themselves in scanning and just staying there in a bucket and not being used anymore thank you very much

let's try again okay all right so thank you Jasmine um does anyone have any

questions thank you jasine for the presentation uh one challenge we have with tools also in the context of G Hub is we have more than 400 repositories at this point and how how to measure our coverage like how many of these repositories we have like for example like security scanning enabled or thinking about quality Gates how to measure coverage the ones who are integrated with GitHub like secret skin we can see coverage right but for the tools we integrate it's it's difficult to measure do do you have any advice on that have you ever done something like that to measure across the whole company your coverage I I think the approach of having it across the whole company is is

a very good one but we you will not be able to do it from the start so the responsibility of securing the products needs to be also given to the product managers the product owners we need to enable them and a key point in the in implementing such tools is for example the loop back into jira Azure devops and so on and so forth where we then bring the results from all these different tools into one point um and from there we can for example also create dashboards that span across multiple projects to cover and see from a management perspective how the teams are performing and how well they are performing but it's for sure a very

challenging task as every tool has its own dashboard um its own findings its own coverage metrics and merging that together is is definitely a challenge so what I've done in the past was really integrating them into for example jira um having automatic tickets created but only after um we set up the reasonable quality Gates and we try to fix the the the false positive rate and then from there we took it and centralized the management in jira with dashboards I hope that the answer is your question

okay uh yeah thank you for the interesting talk um one question from my side so you mentioned that I mean it's not only about the tools and having all those findings and this is perfectly valid um in multiple projects where I worked um the teams were actually struggling with the res the necessary knooow to to have like a security engineer in the team and to do these kind of things I mean you named one example to just to prioritize the whole this big list of findings of an initial scan and this is what I saw is the the biggest issue for for teams and it's also not so easy to get people with security knowhow in those teams so do

you maybe have some ideas um how the teams could build up the necessary knowledge to actually do this by themselves so within an application normal application developer Team without having the the luck or to to just get a security engineer into those teams to do these kind of things so thank you very much yeah it's it's a very interesting question and I think that it's about really raising this awareness and training um I see security Champions security enablers security Engineers as a crucial point in starting out a security program uh because developers have done it the way they've done it many years before and this focus on security is definitely newer um and they need some help with um training

prioritizing and so on and so forth what I've seen very beneficial is for example the topic of threat modeling guiding the teams in finding out what could go wrong or really trying to impersonate a threat actor and what they could do and really try to think a little bit outside of their box and need to develop this feature um and everyone is maybe for example using abuse stories abuse abuse stories um to have yeah to shift this mindset a little bit step by step but definitely it's it's a long journey um and step by step one needs to Ena

them any other

questions okay then um let's give Jasmine once again a round of applause thank you very much