Jumpstarting Your Appsec Program

All right. So, can you can you see my screen now, Bryce? Yep, it's there and you sound good. Excellent. You're not presenting. There it is.

Are we good? Good to go. All right, great. So, thanks everyone for joining and letting us talk here about jump-starting your appsec program. Jacob and I worked together for a long time at Adobe. Jacob started out as a developer on the analytics program. Or the analytics product in Adobe's digital experience BU. He worked there for years and he started to see some needs in security and so he started the application security program there. I joined his team and spun up our SDLC. And security champions and did a lot of acquisition security there over the years. We've done a lot of jump-starting or kick-starting appsec programs kind of with different cultures and our company cultures and different

technology stacks and things like that. Um So, what is app security? It is kind of a specialized security field. It can mean a number of different things. It takes different forms at different companies. So, you might be threat modeling, pen testing, doing some SAST or DAST things. Or even delivering components of the product itself. With varying levels of kind of success and engagement. Um It could be any one of those things or a combination of those things or those things plus plus. Um These are the kind of the main things that you'll see in the app security space, but the application of these things in your security's culture as well as with your security's your um company's needs in

mind is really important. So, we've taken a stab at defining it a little bit more broadly. Um We've defined it here as helping your technology team write defensible code and release trustworthy software. And the way that the reason we've defined it kind of broadly and intentionally vaguely here is because code and app security is kind of now everywhere. It's it's not just on our engineering teams, it's in our infrastructure teams. Products. And people may be writing different internal microservices to solve little problems and so we're seeing kind of a code explosion or we have seen code explosion. So, we need to have app security in the places where it matters. So, depending on how your company

operates and what your company's goals are, you do product security or application security kind of by any means necessary. And so the way that you approach application security is going to be a really a custom fit depending on how your organization works. Thanks Julian. So, that that definition's kind of broad, but that's that's on purpose because of that that custom fit you have to have for your product security program. And the way you start is what what are your goals? What risks are you trying to reduce? I mean these are all the like classic questions, right? Like where is your sensitive data? How's it accessed? What's your attack surface? What is your attacker look like? All

those classic questions. Those are super important, foundational to your program, but I'm actually going to say you got to go a little bit further and actually talk about your company's goals. You what is your company's brand? What does it look like? What brand are you trying to achieve? How does security impact that brand? For example, if you're in the you know medical industry, um then maybe your brand's really around this compliance and and HIPAA and your security program should reflect that and should focus on that and that formality. That's different than if you're a startup that's looking to get acquired, right? If you're a startup that's getting acquired, you might not have a ton of resources, you might not have a

lot of money or people, but you still need to tell a compelling story to someone who's looking to acquire you. And so your security program should take that into account, but how do you going to tell that story? How are you going to sit down in front of somebody who's looking to acquire you and make a really compelling narrative about about how you've you've done security. What about your customers? What do they expect? How do you build trust with them? If they're a bank, you know, we've all worked with or maybe not all of us, but someone who's worked with like someone like a bank, they've uh they might have like really steep kind of compliance requirements. They want to

see a like a stack of formal papers to show you're doing security. And if you've worked with someone like this, you you might realize that a lot of what they ask for doesn't actually help your risk, but it's important to present it to them, right? And it's important that you do it well because your job is not just to reduce risk, your job is to help your company be successful and achieve its goals. And I think this is super critical. This is why we started here. I think this is so important you should share this with stakeholders like often and from the very beginning. If you sit down with a stakeholder and you tell them like, "Look, my mission

and building up this appsec program is not just to reduce risk, but actually help the company be successful, help you as a stakeholder be successful." In turn, they're going to help you achieve your goals. So, I would I would share this message and and make this like the DNA of your program. All right. So, you've started to figure out what your company's trying to achieve and and trying to figure out like how your customers view you. It's going to be really easy to fall into the first big pitfall. And that is I think all of us know or you'll find out at some point, it's so easy to spend a lot of money on security and it's just

as easy to get hardly any return for that money spent. There are so many tools out there and there and there's a lot of great ones. You know, all these scanners and aggregators and reporting tools, but if you're not enabling the right way cuz you you haven't done your homework yet, you're just going to throw a ton of money and get all kinds of interesting glowy dashboards, but you're not actually going to accomplish your goals. So, I would say instead of starting with with spending money on the tools, which will come, I think you need instead start with doing your homework, which is effectively getting in their sandbox. Like obviously you need to get in there

and you need to understand like where is the technology? Where does it sit? What's the technology stack? But I'm talking about more than that. You really need to understand how this how your product is built. Like how does your company make money with this product? Um if it's software or whatever it is. And I'm not just saying like what is the deployment pipeline for your application? I'm talking about like how is the backlog built and prioritized? How are things communicated? Right? Where how does things flow? What are the different people involved? Uh it's going to be super critical for you to figure out how to integrate with that. And beyond that, you also need to really

understand, you know, your customers and how they expect to interact with your customer with with your product. How do you communicate with your customers? If it comes to a breach or a you know, some sort of report that you have to get out, you need to know ahead of time how you're going to work with those customers, how you get those messages out, how they expect to you know, have a dialogue with your company. And finally, I'm going to emphasize this again, what are your stakeholders' goals? You'll probably find in any dev organization, there's probably somebody, maybe it's like a VP of product management or something like that, who who like really is a linchpin for the whole process.

Like maybe this person like holds the roadmap or the timeline with with an iron fist, right? And if you want to get resources, you've got to go through that person. So, it's important for you, just as important as you know, knowing your risks and you know, getting the details for the latest vulnerabilities, you need to understand how to work with that person. You know, are they driven by data? Um You need to provide them good options and good and good reports so they can make smart decisions. Or are they somebody who's more narrative driven and you have to sit down and tell them a story about security about how it's going to help them accomplish their

goals. You have to figure that out because if you don't, these folk can make or break your program. All right. So, you figured out what your goals are. You went through the whole system from from the beginning to the end. You know exactly how the product works. You know exactly how it all flows. All the details. The next thing you're going to do is figure out how do you measure and define success. I think we've all been part of a program at some point or something, a project, where we work really hard and and we ship it out and it's out there and it's doing a bunch of stuff, but we have no idea if

it's successful and doing what we want because we never took the time to figure out how to measure success. This is this is very true in security. In fact, I would say in security more than most any other place, cannot afford to know what success is and how to measure it. So, define those metrics, define those KPIs. Figure that stuff out. I would also say just as important as making sure you have a KPI, is making sure that you don't over rotate on it. If you're spending all your resources on driving something to three nines when two is sufficient, then that's money you could have spent somewhere else. And so be really judicious about finding

just the right measurable goals and driving towards them. So, the next thing so after you figure out your goals and your KPIs, next thing you're going to do is you're going to figure out how to integrate with your devs. So, it's going to be really tempting and we've all done this. I certainly have done this. It's going to be really tempting to build this beautiful security biosphere. It's got all these tools and reports and integrations and logins and you're going to want to like go and grab you know, your dev engineering manager by the collar and drag them over to your biosphere, make them log in and view all your beautiful reports. In my experience, this does not work or

at least it's a lot harder to be successful. Cuz the reality is every dev team, product team, they have a way that works for them. They have a way to be successful. They have a flow. They have aggressive timelines and they have a lot of pressures. And the last thing they need is to be dragged out of their world and they have to go into yours in order to get their security work done. So, you need to figure out like how do I if I if I analyze that CI/CD pipeline, how do I get in there to add my automated scanning checks? Or how do I get in there to make sure that my gateways are in place? You know, there's

how do I how is there backlog prioritization shaped? How do I make sure that I'm in that scrum of scrums so that I can, you know, inject my stories in there so that that they'll get done in the right way rather than making come to me. This is this is super critical. It takes a lot more work up front. It's many times harder to do, but it'll pay off dividends in the future. It's been our experience at Adobe. So, I would say after you've done those things, the next thing comes and this might be the most important thing on this slide is establish establish those relationships first. Right? So, go to all those key stakeholders that you've found, sit down

with them, talk about what you're trying to accomplish, how you're going to help them be successful, what the program's going to look like, get their feedback so that they have a sense of ownership of this thing, know that you're in it together to make this thing work. I think we've all run into, I know I have, that grumpy program manager that's been doing program management for two decades, you know, worked at Novell before, you know, whatever company he's at now, and has had nothing but problems with security, nothing but friction. And you know, has this natural inclination to build walls and to separate and push you away. If you go to the very at the very

beginning and sit down with with people like that, some of these important stakeholders, and talk about how you're going to make them successful and how your thing that you're trying to do is also their thing, they're going to be they're going to open up and they're going to be your your your strongest advocates and allies. We found this time and time again at Adobe where like some of the the hardest people to to like kind of crack that that outer shell and get them to appreciate the security work or the biggest advocates once you do. So, okay. You've done all that, you've established your relationships, you've got your integration points, you have metrics and goals all lined up. Okay, so

now it's finally time to go buy all the fancy tools and scanners and reports and and processes and AI and ML and all the the fun stuff. So, so now it's time to go.

So, Julia might be muted. Sorry, muted. Just one second there, Jacob. Uh one of the things that we need to make sure that we strongly consider when rolling out a program is that uh culture eats strategy for breakfast. We've all heard uh the best-laid plans uh often go awry. So, um if you're marching in there with the world's best plan, uh something that you truly believe is going to change the face of security at your company, uh it may fall flat if you don't take this into consideration. Uh there's a lot of reasons that something can fail, but one of the top reasons that I've seen um is a lack of cultural fit. Uh not to say

the lack of cultural fit of an individual, but the lack of cultural fit of a program inside of the organization uh that you're trying to introduce. So, if you're in an organization that invests heavily in developer productivity uh and is strongly oriented towards speed of release uh and you want to pull everything to a dead stop to do a 3-day security review, uh that's a lack of cultural fit that's not going to be aligned with your company's goals. Uh similarly, uh if you're in a risk-averse organization that wants everything that documented formally uh and sign-off at every level, you're going to need to meet that criteria with your program as well. Uh application or product security has

to be a strong cultural fit in your engineering organization uh because it needs to be an integral part of the engineering program rather than some icing on top. Uh so, in that organization that I mentioned where developer productivity is paramount, uh you may be able to find some low-overhead, high-leverage solution by partnering with the teams that provide that. Um so, make sure to do an assessment of your engineering culture and find where you can inject secure defaults or good choices that have the biggest impact. Um and and a note on building relationships with your tech team. I think a lot of times we think of the tech team as engineers, but uh in reality, the tech

team is so much more than just engineering. Uh if you want to release software uh at scale, you often have a team that looks something like engineering and product management and program management and all of these people kind of in their own uh lanes and realms of influence and and talents are able to release this uh important software. So, um uh a lot of times they have different responsibilities. I'm going to talk to an example organization, but I think definitely find out how the engineering team works with the other technology teams to release software. Uh in my example, uh we'll talk through some kind of prototypical things that I've seen. So, uh engineering is responsible for uh

technical implementation details, technology frameworks, architecture, and design. Uh you might talk to them about how to fix an individual security vulnerability or uh if they're evaluating a new technology or framework that they want to use, uh you can add value to that by going and doing some groundwork to find out how that changes the security posture or to recommend kind of the secure choices that they can make with that framework. Uh but you also want to get to know your product management team because these are the people who set the who know about the business and customer priorities and often set those things. Uh they kind of create the future state of products and they are the ones who

kind of set those long-term visions and road maps. Uh and they also have a good idea of the revenue uh that their products are bringing in, which products are the ones that are um invested in the most because they're bringing in the most money or which ones are kind of up and coming. And so, you can kind of get in early if you get to know the product managers. Uh you might want to talk to product management about ways that you can help them sell security so that you can get a piece of their road map. Or if you're trying to push um an important security initiative that's going to take a lot of time from

engineering, uh what you want to do with product management is negotiate that space on the road map. Um a lot of times you can get the engineering buy-in if this is the right choice, but product management has to also say this is the right choice to do at this time. Um one of the things that that we've been successful in and or previously been successful in helping with product management is uh helping to put together um artifacts for uh PMs or or sales people to uh point to our security uh posture. So, it helps them sell and then we can uh when they see how that helps the product process go faster, um we can get a piece

of their road map to push some of the important security stuff. And then finally, program management. Program management keeps the trains running on time. They're the work uh execution arm. Uh they know about deadlines, upcoming, current status, and a lot of cross-functional initiatives. Um if you want a uh to introduce a new practice or a new consistent program into uh your engineering organization, program managers are uh a good place to start. They're the ones um who you can talk to about consistently reviewing the vulnerability dashboards that you put together or time resolution on things um and find out from them if those things are getting done. Um so, definitely make sure to work with all uh different people on your tech

team um and also at all levels. So, uh engineering ICs, you might need to work with them on like I mentioned resolving a vulnerability, but if you need to get if you're trying to fundamentally change the face of of security at your company, you need to certainly get uh buy-in from like the VP director level, the people who are uh making the big business decisions, and and get buy-in from those levels as well. Um So uh which stakeholders can help you scale your security program the best? Um we had some success scaling uh security at Adobe using security champions. We started with security champions in the engineering organization. And uh security champions has kind of been this

buzzword uh in the security culture circles for a while. Uh the idea here is that you can recruit people internally to be force multipliers for your security program. It works in some security cultures and and we successfully rolled it out at Adobe. We had uh appropriate buy-in there. So, if you want to be successful, uh there's kind of uh four things that I would recommend uh for a successful security champions program. And uh those things are to train your security champions. So, there should be a common baseline uh security understanding amongst the security champions. Um so, make sure that they're technically trained as well as informing them of security's goals and sharing that context with them so that they know what

the future is bringing uh and they can kind of bring that back to their teams as well as bring uh to you, the security team, what their product goals are and what their pressures are and what's kind of coming down the pipe for them as well. Um and another important thing that we did was when we uh designated people as security champions, we uh went to the top levels and got the buy-in at the top level and had that thing had uh VPs and directors kind of nominate security champions on their teams. And so, it was something that was an initiative or a directive from uh the engineers' own management team to go and do this thing um as well, which really

helped with uh making this program uh take hold. Um and then also have some immediate wins for getting people on board as security champions. Maybe um have them resolve some security issues or push a security initiative across the finish line. We really found that this helped us scale in addition to automating as much as we could around kind of environment discovery and and reporting as well. Um and security champions can be an intensive job if it's a if it's a full-time thing, but another option is if you don't have buying or budget for this type of thing, see if you can get the budget for like a once a month pizza lunch and learn where

you can talk to dev teams about security and become familiar with what they're working on. Um you've probably seen this if you've been in security for a while. A lot of the stuff that we find out about are or talk to talk with security or engineering teams about is things that like they saw us passing in the hallway and they were like, "Oh, lightbulb went off. I really should have talked to you about this thing, but you know, now that I'm seeing you, let's bring it up." So, make sure your security team is visible. Stuff is moving so quickly in engineering organizations that you want to make sure that you are top of mind for them

as much as you can be because there are a lot of competing priorities going on in engineering. Um Great. So, at the end of the day, your security program or your appsec program should be enabling engineering to make their own security decisions. Uh slide, Jacob. Thanks. Um so, we talked about environment observations. So, find out what's going on in your environment, where you're working, what's your tech stack, what's coming down the pipe, what are the things that you can observe factually about the environment and then put your security lens on those things. Are those things Are Are those behaviors good? Are they bad? Are they things that we want to change? And this part I think

I want to point out here as well. Definitely make sure you talk to your stakeholders about this cuz I think we a lot of times come in um or I've seen myself make the mistake of coming into a conversation saying the world is on fire. This is a horrible thing. I can't believe it. And the engineering team will say, "No, that's you know, that we do this and this is working as designed because of this reason or you can't possibly expect us to turn on a dime and fix that thing and here's all of the impacts." Right? So, so definitely talk with your stakeholders about what it is that they're trying to achieve so you can enable their security

story as well. Um and then once you kind of figure out what those things are that need to change, make sure that you plug those things into the existing engineering tools and workflows. As Jacob mentioned, asking someone to step outside of their world to go find out what their security story is is one of the ways that you're going to hurt the productivity of your organization. So, plug into their processes and leverage those. Often times they figured out some pretty efficient ways to work. Um cuz they've got to get a lot of work done. Um and then finally combine those things, the security lens and the things you're measuring about the environment and the status of those

things from the engineering tools and workflows and build out your fact-based but opinionated security dashboard. So, you want to show people where they need to improve, but you want that stuff to be immediately actionable. So, if you're saying you've got things that are way past due, you want to be able to point them to the Jira ticket or the pull request or the issue that you're seeing and how long it's been open and have that thing drive down security risk kind of in an immediate way. Um another thing about setting KPIs uh is that that these things are kind of they're difficult to do, right? So, hard to get it right the first time. Don't be afraid to iterate on this

stuff. I've definitely had to do that several times. I think the first set of vulnerability management KPIs that I tried to roll out were way too aggressive when I was thinking about it from security point of view was like, you know, we should we should just fix things as quickly as we possibly find them. Um But but you know, as we kind of mentioned, that might not be the right choice for the company. And if we're constantly interrupting engineering work to fix security issues that could wait till next week or be mitigated short-term and then you know, fixed in the long-term or or things like that, then we're not adding the appropriate value to our or to our

um our company and our business. So, make sure that you're iterating to make sure that that is driving the right behavior and also helping to achieve the business goals of your company. Talking fast. All right. Uh so, your jump-started program. So, you've done the discovery and met your stakeholders and set your goals and enrolled this thing out. Um We've These are kind This is going to be your checklist for how to roll these uh things out in your application security program. Um but appsec programs are custom fit. We only had a few minutes here to to kind of talk about how to roll out an application security program. So, we gave you a couple examples of the many

ways that you can scale program, but you know, we could chat about this all day. Um how to build that out given I think a company's culture. We do want to make sure that we've got time for questions, but you know, we'd love to chat appsec talk talk shop about this stuff. So, feel free to reach out over email and we're happy to chat with you.

So, Judy, I don't I don't know if you can see if there's any questions. I think I'm looking in the chat and I don't see any. So, there's a Q&A window. I think someone just threw in a question. Um Yeah. Let's see. It says, "A little bit off topic, so if you don't want to spend time answering this, I won't be offended." Um "What can I do to better prepare for a career in application security? Is it feasible to find an entry-level appsec job or where should I start?" So, Judy, you want me to take the the first crack at that? Sure. Yeah, I can I can add a response too. So, I think actually it's a pretty

exciting time to be starting in application security. As we all know, security has evolved over the past few decades pretty rapidly. And with the modern-day application stack, um a lot of the the investment of a company is in the app itself, right? Cuz a lot of you get a lot of leverage out of your your cloud infrastructure and all these fancy pipelines that you can plug in. There's still a lot of room for, you know, operational security and compliance and things like that, but they but the space that's growing really fast in my opinion is that product security. So, first of all, great question. That's exactly a fantastic place to be. Um if I were you, I think one of the

things that makes you can make you uniquely valuable in this space is if you have a really good understanding of how the dev process works and how software works in addition to security. That blend is is actually kind of rare. A lot of times people come from all kinds of backgrounds into security and we do not always come from engineering backgrounds. So, if you can cross those two worlds, I think that's super valuable. So, I would spend some time not only understanding the security side, but also understanding the dev side and getting some of your maybe computer science chops in place. After that, there's actually really good programs coming out coming online at different schools,

but there's also opportunities to go and do things like intern for companies like Adobe or just get your feet wet with some of these other things. But more than that, just reach out to some people, you know, create that network and and that line, you know, feel free to email me and we can have a we can have a chat about some of the things I've seen that have successfully started careers in this space. Yeah, I would say plus one definitely. I think starting to learn programming and and computer science like Jacob was saying, but also there's a lot of really great resources on the internet so that I'm not just repeating Jacob's answer. OWASP has

great resources. There's like the OWASP top 10, but they also just came out with the ASVS, which is a bunch of application security controls and I think level one of those things I'm probably saying this all wrong, but they can be automated. Um so, I think think about those things and what are kind of the common pitfalls that you might have in app security, but also yeah, a lot of lot of resources on the internet both for proactive or I guess red teamish and blue teamish type things. Um offensive defensive. So, learning how to do pen testing I think is a great way to get into kind of application security and seeing how applications can

behave and misbehave, but understanding I think the underlying infrastructure is going to be important, too. So, computer science as well. I see the questions now, so I can read the the next one here. Um So, I got a question from Mark. What do you both think about implementing secure defaults in the paved road for dev teams? Uh So, I'll take a first crack at this one. So for I've seen this be very successful. I think that it again is important that you're able to meet your dev teams where they're at. I think paved roads work where you've got developers kind of all working um already in paved road if that makes sense. If your company invests heavily in developer

productivity and has those resources, those are definitely places where you should integrate and provide security secure defaults and I think it's going more and more in that direction and coming out with coming up with what should an architecture look like, what should security look like and helping to build those things in from the ground up. I think that's a really successful model. Um, but I don't think it probably is going to work at places with a lot of legacy uh, software processes and code and things like that. So the only thing I would add to that is that I I think that approach resonates with developers. I think we're seeing this another like self-organized teams is it's another way

of like putting together like teams. They want the ability to have the control to make smart decisions locally, but they also want like that really easy route. And they want to have security too, right? Not only do they want that in their software stack or in their libraries, but they want that with their security choices too. So I think um, I think if that's your approach, it's going to resonate with the teams and you're going to be successful.

Do you want me to ask question, Jude? Sure. Um, do you have recommendations on putting code release gates on CAST to DAST built into the CI/CD pipeline? I prefer to gate things before release to prevent vulnerable software changes from going out, but it causes problems with current culture at times. Oh, that's such a good question. That's such like a hot topic, too. Right? Like and it like it like pushes everyone's buttons. So here's the deal. It depends This is it's a sad answer, but it's a true one. It depends on your company. Right? So some companies some teams require a release pipeline that has to be able to get out the door within 15 minutes. And

I don't know how robust your scanning is, but that might get in the way of that. So you may have to instead of having it be a gate, it has to be a parallel operation like and then you just have to be really good at rolling back. But for some other teams like if it's like a hard control and this is a promise that's been made to customers that you're scanning before release, then you've got to get in there, right? And you've got to have that gate. So I guess the answer is it depends on what your company needs. And I think you have to be flexible and I think it has to be in partnership with

the dev teams to really figure out if that's the right fit. Certainly if you can get away with it and it's not causing too much friction, then it it probably adds some value there. Yeah, I want I want to say plus one and I want to say that if you're going to do that, you have to be very careful about those things. Choose the things that should always be true or always be false and test for those things. Definitely don't put something super noisy in there.

All right. I think we've answered our questions. Thanks, everyone. It's been great. Thanks everybody.