DevSecOps On A Budget: Building A Secure Dev Pipeline Without Breaking The Bank

So I also work at Sonatype. So we get lots of projects in and being able to rapidly deploy a a DevSecOps environment is really important um in order to uh take it through on our short-term projects. Um also work as a security lead there. So um in sort of like a bit of off time with projects, I sort of head up the security for this company. Um and yeah, my LinkedIn profile's here if you want to uh chat. Um cool. So today we're going to start off with an introduction to what DevSecOps is. Uh we're going to take a look into the commercial solutions and then look into how we can make it cheaper with uh using

open source software instead. And then we're going to analyze the tradeoffs between using this open source software and the commercial products. So firstly, sort of give a bit bit of a background of like why we want to implement DevSecOps. Um part of it is to get towards the shift left, um which is implementing security measures earlier on in software development life cycle. The idea being that that the earlier we check the security, the cheaper it is and cheaper it is to fix and therefore harder it is to exploit and it's more intact. Uh so a bit of an example on this is I worked on a client project where we didn't really do much security checking

while we were making it and we sort of got to the end of the project, we handed it off and they went and did a penetration test and they found a few issues and then they had to rehire the team to actually come fix them and that's because we checked for it so late, ended up them spending more money in order to fix the issues. Uh another example is I'm sure some of us have all seen a pull request a bit like this. And what do we do? We panic. You're going to have to uh rewrite your Git history, you're going to rotate your keys and it takes a bit of time, you know, if

you know what you're doing, maybe it takes like an hour, but you know, um it's still time wasted. Whereas if we have something set up like GitLeaks, we're able to find this earlier before it even gets pushed to a remote version control system. So it will be blocked when you make the commit and therefore you can spend more and less time fixing it and it's a lot cheaper. So next we're coming on to what is DevSecOps and DevSecOps is creating a shared responsibility for security by embedding it directly into the software development life cycle. Um so yeah, here we have a yeah, the life cycle of uh DevOps. So planning, the coding, building testing releasing deploying

operating and monitoring. So in order to uh accomplish this uh embedding of security, we want to focus on adding automated security testing each stage of the develop life cycle. This will lead to security being an integral part of the development process whilst also reducing the overhead for developers writing the code. This makes it easier to spot and to fix any security flaws that are introduced. And as we're aiming for the shift left in our security testing, I'm also going to be focusing on these four steps as part of this uh presentation. Now, we obviously don't want to be writing all of this automation ourselves and in most cases these are solved problems that there are companies out

there that uh sell out-of-the-box solutions for it. Unfortunately, these can get quite expensive. Um for example, Snyk here, they charge $75 per user per month and that's if you want the uh full package. And for a 10-developer startup, this is already going to cost $9,000 per year. To some companies this might not sound like too much, but having worked with a lot of startups um in my consulting role, uh they uh quite often have quite limited budgets and some of them will just not want to spend this on security when they're already dropping a lot of money on other SaaS products for the whole develop cycle. Um so the solution to this problem is to

use open source. Uh I promise this is the last fun pool GIF in here. Um So yeah, open source is a huge part of the security community um of both homegrown community-led products and open source tools created and supported by larger companies. Um so being able to harness them in to build our DevSecOps systems is really important. Cool. So jumping into the first stage of the development life cycle, uh we start with planning. So this is a stage where a lot of the big decisions are made on the technical strategy for the rest of the process. Making a mistake here can have big ripple effects to remaining stages, but unfortunately it's also the least automatable.

Um so the general strategy here is to analyze what threats there are and build up a plan on how you're going to mitigate them later on in the process. So a lot of frameworks we can use here such as STRIDE, which is a model for identifying threats and looks at each of the potential types of attacks that could occur and help you think in categories about which are relevant and important to your system. Uh other tools that we can use are some from OWASP projects such as the threat modeling cheat sheet. Um this will give you guidance into how you can generate threat models and then you can use a tool like Threat Dragon, which uh helps

you actually generate the diagram and keep up to date. Uh so we have an example of that on the left here. Um and also what's really important at this stage is trying to encourage developer involvement as much as you can um cuz the developers are going to be the ones writing the code. They need to know what threats they're looking to protect against and showing them the importance of doing this uh will uh increase their buy-in as well. So next we get on to the coding stage. Um so this is before anything's pushed up to remote, it's uh the developer just coding on their local local machine. And here we want to introduce tools and

encourage workflows that help developers avoid creating commits and pull requests that violate secure coding guidelines. Um so this stage will both involve educating the developers on coding as well as providing automated tools um that can be directly integrated into development environments. And this will help them catch vulnerabilities as early as possible before it's even pushed up to version control. So examples of things we can do here is uh secret scanning as I showed earlier with GitLeaks to prevent any sort of secret keys being pushed up and also doing static analysis security testing in pre-push hooks and in your IDE like this. Um this blocks vulnerable code from getting pushed to remote. Um so examples of tools we could use

here are GitLeaks as mentioned, but also something like Semgrep open source, which you can hook into your IDE to spot these vulnerabilities. There's an example of a vulnerability being shown directly into your editor. Um so you can hover over it, see what it is, get some advice on how to fix it and hopefully the devs see that. So moving on to the build phase. Um so this phase occurs when the code has already been committed and pushed to repository. Um so you'll often want to run these checks on PR open to alert the team to any new security issues before they merge into main or in staging. Um so here we can use lots of automated

tools to look for common vulnerabilities. Some of them quite similar to what we have in the previous step, but actually enforcing it on the pipeline level so they can't sort of get around it by just dash dash no verify, um which is the favorite of many people, I'm sure. Um so yeah, we have a developer pushing up to uh version control system. Here we have GitHub. We can run secret scanning, run static analysis security testing and then run software composition analysis as well, which is what we'll get on to. Um so I've mentioned static analysis security testing, but what actually is this? So this is a white box testing method to help you find vulnerabilities

in source code without like executing any of your code. Um so yeah, this is a really good tool to use um to sort of catch the low-hanging fruit of vulnerabilities, um but it also comes with a quite a few challenges. Um if anyone's used it, they know it's just fills up with uh false positives and lots of warnings about things that you may have already mitigated against or or decided so I can accept it a risk and you want to make sure this isn't blocking lots of pull requests cuz otherwise your developers will try and find a way around it. So ensuring you keep um ignores and everything up to date and manage well is really important

here. Um so again, another a tool we can use for this uh which I have up here is using Semgrep open source. Um they have paid plans, but you can run it locally with the open source for free and therefore put it in a GitHub action. Um and it will also just run on your PR's opening. Um the downside to this is we don't get access to like dashboards and things like that. It will just be sort of like text in your in GitHub. So it's not quite as nice to use, but we do get the benefit of being free. Um yeah. So next we have some software composition analysis and this is an important part of managing third-party

dependencies and open source components. Um it helps you track and analyze uh components being brought into a project uh including their direct and indirect dependencies. Uh they can then analyze potential vulnerabilities and exploits within them. Um so I've got a bit of an example of why this is important for a project that I've worked on. Um so we were building a Node.js app where we had multiple different packages um like our own packages we made and they had to uh inherit each other's uh peer dependencies. So here for example, we have a UI package and an auth package. Um what happened was we changed the name of the auth package to account. So when this happened, we also forgot to

update the peer dependencies. So now whenever we ran yarn, it uh decided that it was going to look into NPM and try and find a package that matched auth. And this ended up with us installing this auth package which has been downloaded 1,500 times and hasn't been updated for 11 years and this was uh yeah, got to our staging environment before we caught this. So having good software composition analysis would have helped us catch this earlier. Um so how do we actually do this, especially in open source? Um we can use something like CycloneDX to generate the S-bomb and then upload it to a tool like Dependency-Track to get like a nice dashboard um like this.

So the downsides of this is we have to pre-install an instance of Dependency-Track to set it up, uh but it does give us a very like polished application with a nice dashboard and everything like that. Um another tool we could use if you want to get it up and running a bit more uh like a bit quicker uh um is something like Dep Scan. You can just run this as a GitHub action in command line and I found it really easy to set up. It's kind of just set it and just leave it and it kind of keeps checking for you. Um so yeah, it really handles it nice uh it really handles it nicely, but it won't give you a nice

interface like this. It will more just be like warnings in your GitHub PR. Cool. So now we're on to the testing stage, which is the last stage we're going to look at in this talk. Um so this is targeting the build artifact once it's been deployed to its uh staging or testing environment. And the aim here is to simulate real threats and attacks that that could occur in a production environment um from real malicious users before they're able to be exploited on production. Um so the downsides of this is it's a lot longer to run but and more computationally heavy. Um so we may not want to run it always on a purpose basis. I've often set it up with a

running sort of overnight um something like that where it doesn't run as often but still alerts you quite often. Um and it's often best practice to try and set up a separate environment for it so you're not like constantly hitting your staging environment with lots of requests and like slowing it down and things like that. Um and yeah, there's a pre-built actions that you can use if you're using GitHub for example. I'm sure there are other ones like a lot of other CI tools. Um so here's an example of some alerts that you can get uh running uh an OWASP scan uh which is a a tool I'd recommend for from open source stuff. Um yeah, so it gives you some alerts and

you're able to have a look and fix these. Um so taking a look at some of these open source tools, um we obviously have a lot of big savings here. Um but there are some missing pieces that we don't get by taking this approach. Um so we're not going to get out of the box combined dashboards where all your tools are together in one place and you get all the outputs together. Um you're going to have to like sort of hunt around your different, you know, your PRs or your like your local environment to see these different reports. Um we also aren't managing a single dependency anymore. We're going to have like quite a few different bits and

pieces floating around but you're going to have to manage. You're not just installing or like purchasing one product from one vendor and using that. Um so you have to take that into account. And the last thing we're missing is dedicated support. Um so if your open source open source secure uh sorry, open source community is very good at providing this support. So I'm sure if you went on Slack channels or GitHub issues you'd get help when you need it, but it's not a guaranteed thing. You're not paying people for it. So it might not be as uh like guaranteed basically. Um yeah, so finally just to put what we have together um so uh so taking through our first four

stages, we have our team generating the tech plan and generating a threat model. We have a developer um writing code, doing local static analysis, security testing, and secret scanning. They then push to remote where we do more secret scanning, more SAST, and then combine it with SCA. And then we're able to build and deploy to staging and there we can run dynamic testing uh on the staging environment. Um yeah, so thank you for listening. Um please reach out to me if you have any feedback or questions. Uh but if we have time for questions now as well, that would be good. Um yeah, there's a link to my LinkedIn here. Um the QR code takes you there as well if you trust it.

But yeah. THANK YOU.

ANY QUESTIONS? OH, THANKS FOR THE UH TALK, VERY GOOD TALK. Um question regarding uh S-bombs. So uh S-bombs in in concept are a fantastic thing for both, you know, the dev teams and the security teams, but it seems like in reality, at least at the moment, they're not really widely used. Um why do you think that is and uh do you see that changing anytime soon? Um I think like partly it might be just to do to the fact that there are other tools that kind of give you similar things without taking as much time to set up. Like people are on GitHub, you just have like Dependabot will just give you alerts and maybe a lot of teams

might think that's just enough or just running NPM audit, that kind of thing. Um but yeah, there's I'm sure there's multiple other reasons. I'm not too sure yet. Hi. Have you ever done any analysis on the the cost difference between running a an out-of-the-box solution and the extra time involved in people's resources having to piece these other systems together? Cuz you sometimes find that by the time that you've taken all that into account, the out-of-the-box system not only is slightly better, but it's also can be cheaper. Yeah, I think that's definitely a very good concern and I think it probably depends on like how much your company's scaling. Like if you're you know, the higher number of

developers you have, the more that these out-of-the-box solutions are going to cost cuz they're often on per developer basis. But at the same time, it might be worth the cost when you're a bigger company to make sure you get these like rock solid tools instead. But um yeah, I think it really I've never done analysis on it, but it I guess it really depends on your organization.

Um do you do the threat modeling throughout the whole process of development or is it once at the start? And do you have any kind preferences for threat modeling methods? Um so yeah, so it definitely should be done sort of throughout the process um pretty much all the time, especially when you get new features in. You definitely need to do threat modeling there and then adjust it as your product uh evolves. Whether it always happens like that uh who knows? Um depends on your team. Um but yeah, in terms of processes, um I'm definitely still working on that. Um trying like I'm currently in a company that doesn't have like a great security posture like a really mature security

posture. So we're just still trying to like work it out from that. But yeah.

I dare you. Uh so you mentioned uh the AWS API key leaked uh on a GitHub uh PR. What's your preferred method for password management within applications? Um so yeah, so it's usually using the sort of like built-in tools. So when I'm actually using passwords in an application, I'd probably use like something like SSM if I'm deploying on AWS. Um for the actual API key on AWS, you're probably looking at using um I think it's like GitHub secret store or something um that kind of thing. I've also had projects where we uh sort of encrypted the dot env and then actually pushed that onto GitHub and then we can sort of share the password through like password managers.

Um I don't know if that's the best approach but it's something we've done before. Um yeah. Okay. Cool. Now we'll go That's the last one. In the same example, you have uh shown a screenshot in which the the com- the push to the repository was rejected. So I assume that was done via pre-commit hooks, right? Um yeah, so the GitLeaks so you can run that on remote um to sort of detect it when it's get pushed up, but you can also have a pre-commit hook that runs it and it will check your commits so you're pushing, yeah. Okay. In that example, uh um GitLeaks was configured at GitHub before uh it actually merges the push, right?

Um yeah, so it detects it and blocks it before you actually push it up, yeah. So it was done on the developer machine. Yeah, on the developer machine, sorry, yeah. Um you can kind of get it in both cases, but yeah, I'd recommend having it on pre-commit and probably also in your GitHub. So yeah. Thank you. Cool. Good to see you again. So thank you again. Yeah, thank you.