Beyond Coverage and Capability: Shining Light on the Real Application Security Problem

[Applause] Hey everyone, how are you all going today? >> Yeah. Yeah, good. I'm not a clever man. I'm very thankful that [ __ ] and Dolls gave me some panadol cuz I had way too many old fashions last night. So, and afternoon slots, I got saved cuz this morning I went and met a lot of my team members down near the lake. I thought to myself like, yeah, I I could absolutely live here. But then, um, I realized I'm living in the bushy part of the town, so probably not. So, that's kind of broken my love story. Anyway, my name is Cole Cornford. I'm uh Australia's application security guy I guess at this point because I do a lot

of annoying things on LinkedIn. So I um run Galar Cyber software security company. So if you need help with random software security stuff, hit me up. Maybe I can. We have a lot of different types of customers around the world. I've had some really weird engagements and also a lot of really fun ones. I love being a dad. Uh my family was overseas in China for a month and they came back last week. So I get to spend a lot of time with my kids and the first thing they said to me is, "Dad, you've gotten fat." Which is which is really great. It's just exactly what I want to hear from my two little ones, you know? So, um I'm

really thankful to have the opportunity to be presenting at Bides Perf. And like that's one of the great things about going to community conferences is it's about the community. So yeah, let's make more friends over here. Now, on to software security stuff. I'm going to promise you all something that we're going to explain why coverage and capability are flawed ways of measuring your software security programs. We're going to outline a new framework that I've called the five eyes for some reason. Explain why I'm including each of the eyes in there and provide metrics to using it. And then get some Q&A from you all. Okay. Now, I'm usually a pretty happy person. I've got at least three people in the

audience that can attest to that. I got three people put their hands up. Three people who think I'm a happy person. I got one, one, two, three. Yep, there we go. Good. And but I I've been pretty angry at the at the um the coverage and capability narrative. I I think it's false. And I was thinking about it when I was doing a webinar at towards the start of the year where I was talking about what are the trends that I'm expecting over the next couple of years to be and I was thinking oh I was looking at the slide deck and thinking abst products are failing our developers and we were seeing a lot of this um where

it's talking about we have different types of products that have evolved over time to solve different types of issues in software security like we have um patternbased static analysis tools that'll look for like hard-coded regular expressions or AWS API keys and so on. Eventually moving more towards um data flow or taint analysis based static analysis and but we know high assurance is usually manual interrogation of source code or doing penetration testing. But then the industry realized that you know reading every single line of code is very expensive. So we've got to go find other ways to find vulnerabilities. And so we started introducing lots of capabilities. Now we have composition analysis to look at all the different dependencies that

we have and the vulnerabilities associated with them. We have containers to look at like container security and infrastructure as code scanning. We have dynamic scanning to look at uh what's happening like you know simulating basically a penetration test and all these other acronyms up the top that I can't remember what they stand for. So, so I got my bird brain on and I was thinking hm wait coverage like what why are we why are we doing this? Like is it is it a good thing to have like 30% of our our applications have a waff in front of them? Is is it is it better to have 70% of our apps or 100% of our applications have a

WAF in front of it? It's a I don't think that that's a very good thing. The same as like is this the way to measure an application security program is a number of capabilities that we we're introducing. I I thought it was about reducing risk. I mean it's it's easy to measure. You can you know time scale stuff and say hey we have more apps now covered by these different types of controls. we we do more programming languages um so on so forth. If the line goes up, the executive gets happy. If it goes down, they get sad. Um it's just horrendously ineffective. So like let's let's think about coverage. Is this good coverage?

>> Yeah. Why? Why is it good coverage? Perth is coming. >> Yes, go Perf. It's it's where the people are. Um there's not that many people who are living in like central Australia as opposed to on the east and the west coast, right? Like I think that if you're focusing on introducing controls to get 100% coverage across things, then it means that you're spending a lot of extra effort on things that don't matter right? Here's a common question. You need a W. Who needs a W? You need a W. Do you need a W? You need a wife? >> Yeah. Okay, sure. Yeah, it's great. We'll just set our entire environment on fire. It's fine,

you know, and just burn some money or pay. It's good for Cloudflare. Like their stock is up 150%. But then like if you go back to those metrics I was talking about um what would be a little bit more appropriate is probably having a W in front of your API gateway and then having that um talk about different resources behind the scenes. Well, does that mean you have 100% coverage of all your internal applications for a W? Maybe no capabilities. So this is from a product called Iikido and it's an application security posture management product. Look at all the things. Wow, it does so many things. Oh my gosh, that's that's just assuming ASPM. You know, any

individual one of those capabilities is challenging to get right. Here's a bunch of different static analysis tools. And I'll say a few things that I've come across in my days about using and trying to run out and roll do static analysis. This product doesn't support Java 22. This team uses Poetry, not Pippi. Sorry, can't use that scanner. This product decided to delete my integrations because they got acquired by a bigger business. This business unit is on GitHub and everyone else is on prem TM TFS. I can't swap tools without breaking all my legacy systems. I got invited to play golf with the sales rep. It's just very heavy lifting with a lot of overheads, you know. So I I worry

about every time we introduce an additional capability, you create it maintenance effort for every single one of those, you often have overlaps, a lot of duplication of work, a lot of change management, a lot of process. So I was thinking this is basically the same as the thing I was talking about before, coverage and capability. you know, it's more coverage isn't necessarily an improvement in security and more capabilities. Is that justified for the cost that you're putting into it? So, I went from having a bit of a love story to saying that it's trouble, you know? So, what could work instead? What's a different way of doing things? And I was thinking, oh, like, yeah, we need

something that's cool. Like the ASD top four, the application security director at top four, the um Essential 8 top 10, Nifty9 software serious 7. And then I landed on Oh, that's it. Five eyes. I got it's just that's it. It's so easy. It's brilliant. And then I was like, okay, well, I got to make an acronym from this. So, we'll go with five eyes. I wanted to be risk oriented because I see so many people who just misunderstand what risk is like is an open door a problem. >> Situational. >> Yeah, entirely situational. You know, it turns out that you probably want people to come in and listen to my talk. I also wanted to make sure that we

didn't focus very heavily on just purely the technology part of running an appsec program because as often as software engineers which is the traditional pathway for most people to move in into software security they tend to focus on the software aspect and not on the other parts and voila this is what I came up with the five eyes you can inform understand like the threats your current state and what your business goals are. You iterate because you can't stop learning and you need to keep improving. Need to innovate because every business is unique and you can build unique solutions for them. Integrate because tech doesn't play nice with each other and influence because sometimes you need

a stick and sometimes you need a carrot. Yeah, I know. Like why didn't I think about it? There's six six up there but there's five eyes. Okay, so let's start with the first one which is innovate. Who works at an enterprise? >> Is this calendar familiar? >> Where's the managers and directors in the room? Just just love this. You you have two things that happens when your calendar ends up like this, which it invariably does. this is the only time I get work done late at night or well I'm just going to go put my hard hat on and start firefighting constantly because I got no capacity to actually think about things. Everyone agree with that statement. Who

are the late nighters? And who are the firefighters? Firefighters, hands up. Yeah, late nighters. I feel bad for you two. You shouldn't be late nighters. You worked for me. Main thing is that if you're either having to use all your outside of work hours to think about work, that's that's not so great. And if you're firefighting constantly at work, that's also terrible. You got no time to think about things. And that means that you're not necessarily doing the right things. You're just doing things right. One of the things back in the day that Google did was this idea of 20% time where they'd say, "Hey, one day a week, as long as you build spend it building

something, maybe you can innovate and come up with something cool." And they did. They had Google News, they had Google AdSense, Gmail, and Google Maps came all out about 20% time just telling smart people to go off and figure things out for themselves. And then they became an enterprise and killed it. But I still like the concept. I like the idea of like having some, you know, capacity walled off cuz that those success stories come from having some capacity available to actually innovate about stuff, right? Like one example I have is go SDL. So that came about because of um Slack. So Slack had um issues with getting engineers to be doing a lot of just like self-service

about understanding what requirements they had. So they built this product called Go SDL in that innovative time to help get them to start doing like secure by design checks and self-s serve at a local company that people have heard of called Seek um built this product. Listo very similar but it was done independently to um to Slack OF's SAM came about because um a lot of people at I think it was synopsis were just had a benched time when they were doing consulting and they figured oh why don't I have a way to like do security assessments in a structured manner right I'm not saying that every week you need to have people innovate because the

beatings will continue until innovation improves you But it it's important to explicitly call out that you need thinking space integrations. So we have a lot of SAS things, a lot of clouds like green cloud, yellow cloud, pink cloud, pink cloud's my favorite there. Uh we have a lot of legacy. So it's very hard to integrate all of those different things together. You've all got the lines and the the drawing. How do you connect them? How do you make them work? Let's give an example. Who's worked with GitHub before? Oh, yeah. Good. Oh, I love it all. Yeah. I once did this and no one in the room put their hands up and I was like,

"You're all software engineers." Anyway, the general situation I see is if everything is in GitHub, you'd say, "Well, of course, I'm going to use GitHub advanced security because then I it's in a single ecosystem. I don't need to worry about it. It just works, right?" And then your development agency gets acquired or merges with another business and suddenly that other business has GitLab and Bitbucket and GitHub and then you start asking questions to yourself like which of these is the actual source of truth? Do I migrate things from GitHub to GitLab or GitLab to GitHub or just stuff it all and go to SVN? How do you cover the gap? Because if you have GitHub advanced security working in

the GitHub environment, what what's what's happening for all the GitLub GitLab code? You know, what's the preferred security products or how are you covering this? What happens if there's duplication across across all of these things? Over time, businesses, you know, get bigger. they get additional technologies, they aggregate and they grow. And so you need to be prepared to manage the integration challenges that are going to keep occurring over and over as you as that happens, right? And if you don't do your preventative maintenance, then your car is going to break down. >> And I've heard people say, "Oh, what about just ASPM products that just do everything or ecosystems that do everything?" That's great. We're just

going to buy another company that uses a different ASPM product. Oh, so we got two APMs now. You need to consolidate, you need maintenance, you need support, you need integration work, influence. Now, I I like to think that developers are a bit like dolphins. They're clearly intelligent. We just suck at communicating with them.

We need to convince people to do software security. It's It's not so easy. The most common way to see it, sticks and carrots. See See, with the stick, you put a door up, you block and frustrate people, and they're going to try to find ways to get around the door. With the carrot, it's more of a doormat. You can say, you can just have people walk all over you and like what security you actually introducing to the business if you don't actually meaningfully change behavior of people, right? So, it needs a bit of be a bit of a healthy balance right? So, anyone know the marshmallow experiment they did like 30, 40 years ago with kids?

They basically gave a bunch of kids a marshmallow and said, "If you don't eat the marshmallow in 15 minutes, you get another marshmallow." And then they re-evaluated those kids later in life and saw that the ones who could wait and had the selfd disccipline to just chill and get the two marshmallows ended up becoming lawyers and doctors and more high demand professions and doing more successful in life because they had that inner self you know um discipline. So I think that yeah that is totally bogus and we need to give developers marshmallows immediately. I don't have time to wait because they're going to go move on. They're going to get new jobs. Like the average

tenure for a developer is like 1.1 years. So if you're trying to spend like a year building relationship with them and getting them to think about the right behaviors and so on, they've already moved to the next job. So you need to do things that change behavior immediately. So you can look at all of the positive nice beautiful things to do like give rewards for good stuff or tell people that they're really smart and handsome. Um, look at all the the vision for software security in the world. Give them recognition. Give them swag. Or you can go the other way and go full authoritarian and say you I'm disappointed. Create friction. Make it hard for them

to do their jobs. Put contracts that say don't be dumb. Don't introduce security vulnerabilities. Have objectives. Have accountability. You need to have a balance between carrots and sticks. Otherwise, when we're going out and marketing and educating our workforce and trying to get them to do the things that we want them to do, they're not going to listen to us, right? And we need to do that. Even our JavaScript developers, you know, and don't just stick it in a policy. No one's reading it. Never. They won't. Iteration. Now, businesses change over time, and so do the risks that they have.

What kind of uh issue is this? I'd say >> and do we have any like um I don't know exploitation developers in the audience that might be experts at this. >> Is it a user trade? >> I don't know. You're the researcher. You tell me. Anyway, 30 20 30 years ago, this was like the most common type of security vulnerability that people were talking about was like buffer overflows. And like nowadays, like people are starting to think of concepts like prompt leakage or goal hijacking and so on. So like the types of attacks and things that we need to be protecting against and the technologies that we're protecting are fundamentally changed, right? And so if

we're not iterating on our programs, we're not preparing and changing what we're doing for that. So yeah, our appsec activities need to change with the technology that we're doing we're using. And the other thing about this iteration is it needs to be on a set cadence. I think quarterly is best. I think doing it faster than that if your team allows you to do that like because some companies are really good at doing things in short cycles like sprints, two week sprints, month sprints, that's great, too. But you want to be trying to make sure you do it on a consistent cadence. And going back to inform, I think that it's important to go out and inform our

peers in the broader industry about the work that we do and what types of challenges that we're having within software security, but also more broadly cyber. I love going to conferences and just talking about this kind of stuff, about the things that I see. Um, here's me being annoying with my friend Toby. So, so go make sure you go out there and tell your friends what types of challenges you're encountering and try to see if they can help you out as well, right? Don't just stay in silos. Now, there's a lot of different metrics that you can use to be measuring the effectiveness of each of these eyes, but I pretty much always end up going back

to NPS. Now, it's not easy to measure, but that's the the the hard thing about hard things, right? And building a good appec program is really hard. So, stick to something like this and maybe you'll do all right. Anyway, my promise to you all was that I'd be able to walk through No, no no. Go back to the other promise slide. My promise to you all was that I'd explain why coverage and capability are flaws, outline a new framework about the called the five eyes, explain their inclusion, and provide metrics. And I have done that. So I'm Cole. Thank you very much for having me here today at Bides Perth.

Any questions at all for help? Probably got time for one. Is it? >> Thank you.