A Cheat Code for Security Programs

All right, let's get started. A Chico for security programs. Greetings and salutations. My name is Oshan Marshall. I code, I teach, I hack. I'm a product security engineer at Google Cloud. And the topic of today, of course, is product security. And before we like dive into the nitty-gritty, we really need to answer a pretty basic question. And that is why when engineers talk about security often we're just like jumping in into apps or plat or some specific vulnerability or some cool control but often product security is overlooked and there are some fundamental reasons for that. After all, integrating security engineering into the product life cycle is pretty difficult. And why go through all that trouble when you can do vulnerability

research, pen testing, security operations, and patching all with talking to as few product managers as possible. Um, we so is the important question. Why are we doing this? What problem are we trying to solve? And the answer to that question, we need a more holistic view. Focusing solely on the application miss is all of the threats and all of the risk at lower levels of the stack and then spreading your likewise spreading yourself too thin over an entire platform. Often you forget some specificity and some nuance. Product is the answer. you product is where you get the priorities from businesses get involved in software to present something useful to the user and so this talk is for anyone in a

software development or security role whether you're an engineer product manager or security professional I want to share with you some perspective about bringing security into your life so we'll start by defining product security means versus the other paradigms like apps SE and platac and then we'll focus in on the core concepts of shifting left right up and down. By the end of this talk you should be able to pitch a product security program within your organization. So appsc is the tip of the iceberg. It focuses on the security of the applications itself. But that's not really sufficient. This overlooks the underlying problems in the platform and a vulnerability in the platform regardless of how secure the application

on top of it is still exposes the user to risk. So apps is the most visible part of all security work. Beneath the tip of the iceberg lies a vast ecosystem and complex ecosystem. It's product security, but it actually includes everything from the OS it's running on to the databases to the network configurations and to the hardware. So product security is also platform security because a perfectly secure application on a vulnerable platform is still a weak application. And therefore a comprehensive product security approach means that you're looking beyond just the application. You're embracing a holistic view. And when you do this, of course, you make products more secure. Oo. Now, platform security. Product is the purpose of the platform.

And so, when we talk about platform security, it's really tempting to view this as a vast disjointed collection of assets. some servers here, some services there, each with its own health checks and uptime. This is technically correct, but the perspective is too broad to actually be useful. Why? Because these systems don't exist for their own sake. The they're part of a greater ecology. Again, delivering a product. And that's where we need to refine our thinking. There's a popular principle like treat your servers like cattle not pets. Now, this is great for building resilient systems, but it's not helpful to say the farm has livestock. Like, you know the song like, "Oh, Macdonald had a farm, E-I-E-I-O." And on that farm he

had vertebrates. E I No. Um it's not helpful to know that Old McDonald has vertebrates. To keep the farm functional, you have to know the difference between chickens and dairy cows. Each have different needs. Each h have present different vulnerabilities. each have different risk and value rewards depending on the farm's operation and adopting a product ccentric view helps us do exactly that. It forces us to ask the right questions. Which of all of these systems point processes sensitive user data? Which of all of these systems have key pathways that if compromised would have a direct impact on the user's trust? By focusing on the product, you can prioritize our security efforts to the apply the most vigorous controls on to

the most important parts of the platform. And we need to make these ideas concrete. So, we'll turn over to this old standby classic cheat code. You got to memorize. Thumbs twitching a bit to play the game today. I will we'll simplify it be because some of y'all don't know the directional buttons. So, we'll be focusing in on all of the directions here. repeat for just the four shift left, shift right, shift up, and shift down. Now, shift left is an idea we've heard over and over again. It's powerful concept means moving security into earlier and earlier phases of the development life cycle and from being an afterthought into a core consideration. But cloud development is complex. You need more

than just one direction. it playing field is multi-dimensional. We need to look at the whole entire board. So today we'll talk about shifting left, but we'll also talk about shifting right, continuing the security work or through production through active monitoring, robust incident response and continuous testing of live systems. We'll also shift up making a security an organizational priority ensuring that security is part of the strategic conversation and not a just a technical one. and then we'll shift down which is building security into the very foundations of the stack which is all the way down to the metal. So let's first talk with shifting left. Now again it means lots of things to lots of people but today we're focusing

in on integrating it into the product life cycle. So for the purposes of our discussion there are three foundational building blocks. Product is what the user gets. Straightforward. Systems are assets that the product uses. They're closely tied to the product. So in cloud, these could be things like a kubernetes cluster. It could be a a bucket. And if you're an older enterprise and you have production systems, all those weird quirks about your production environment are also just assets, things that the product can use. Now the final part is teams and obviously teams and people are very important right because software still needs people. Yes, we agree. Yes. And so the dreams and visions of product

leadership as well as the realities of being a software engineer frontline engineer is really important when you're trying to integrate security into that life cycle. And every company does it a bit differently but in general you can think of the product life cycle into four key phases. Phase one is just design. It usually means means knowing something about what the users want or need. Phase two is experimentation. It may be one stage or several stages. But the goal from experimentation is just to get some feedback, some live feedback and quality assurance. Phase three is the implementation or build and phase four is the release of the product. Now there's always something that there's always new feedback, new things that

have to be integrated into the product after the launch that can be put in in the 2.0 no version or just the next version and where that's where you can start again with guess what design and again this is just a generalization but it's getting us to start to think about critical queries and critical questions how and where do we embed our actual security capabilities into this life cycle so of course the first thing is to make sure you partner at the very beginning and we could talk about the tools that will help you build that partnership. On the far left, of course, this is uh threat modeling. Threat modeling is a way you can find

weaknesses in the architecture before a single line of code is written, but it also can be used reactively for systems that are already built. You can enhance your threat modeling through targeted threat intelligence. Threat intel shows you how real world adversaries are targeting products like yours. And with that information, you can design proactive defenses against the most likely threats. And then moving into the build phase, this is what the security reviews come in. They work in parallel with development. The key here is not to think of them as a like final exam at the end. Instead, think of them as a continuous partnership, a chance to collaborate and make sure that the design was actually followed

and built through securely and correctly and to look for additional vulnerabilities in the code itself. And then when you find things, then you need to think about remediation and you need to think systemically. How can we eliminate entire classes of vulnerabilities rather than playing whack-a-ole each and every time? This could mean be building better frameworks. This could mean be building better libraries or tools that make it easier for all the developers to write secure code by default. Now we shift right and this is where I remind you that product security just doesn't stop at launch. So there's proactive defense, continuous monitoring and rapid response in the live environment. So big picture and thinking at scale, a

proactive defense of response is different than a traditional one. It's all about securing the product which is really about keeping the users safe. There are a few ways of thinking about keeping product security constant. as we go through them and we talk about these approaches, what these mean for your teams and how you set them up and how they can collaborate and how individual engineers can actually see their work. First is constant vulnerability research and red teaming. So a standard red team will focus on trying to compromise the platform but a product centric red team does a little bit more. So questions that a product aware red team would say is can I abuse this feature to get free

credits or can I manipulate this API to get uh someone else's data? Their work requires deep product specific knowledge to uncover risks in a generic platform scan with MISK. And the same pro and the same principle is applied to product aare or monitoring and incident response. Standard monitoring will send alerts on system level events like high CPU's usage. But productaware monitoring asks the really important question which is why is this service account suddenly trying to export a 100,000 records at 3:00 a.m. always 3:00 a.m. and why am I being paid for it? My goodness. Incident response isn't about isolating a single machine. It's about understanding the immediate impact and the product and the user responding for that.

And one great approach across the industry is partnering with external researchers. When you look, you can look once a researcher submits a bug or presents to use a risk, you can look more to the tech, not only the technical flaw, but you can look through the lens of the product to its users. That's how we can assess real world risk. That is how we can get precise decisions about how to prioritize that fix, the staff it needs, and other resource considerations. And there's a more immediate way of thinking about shift right too. And this is one of the things that you think about first thing that pretty much everyone really learns to do where or we

actually file tickets with the reports. So on the left you have your normal security activity. Um, it doesn't matter what the person behind the keyboard's doing. It could be a pen test. It could be a design review. It could it could be code audit. It could be firing up a fuzzer. Does not matter. If the only thing at the end of it is just a report, that's terrible because the report is then a PDF that gets chucked over the fence and then trickles down the org chart. Sometimes that process takes six months to the year depending on how big the enterprise is. And that's not great. It's just not. On the right, we have a better approach and

a better way of thinking about it. Making it consumable by both executives and engineers. File a bug with the report. Make it actionable so that executives and engineers can work on it and take action. Engineers speak in the language of bugs and pull requests, not reports and documents. And of course, um, you still have to write the report. I can't I can't I can't save you from writing the report, but it's important to point the out the flaws in the engineering context as well. You can go further faster if you shift right and the key takeaways for engineers to consume and act on now hacker in the boardroom. Let's talk about shifting up. So we talked about

shifting left and shifting right. The next part of the cheat code is shifting up. And this is the concept of escalation making it absolutely critical for a healthy security culture. Escalation is a normal healthy security process. I'll say this again. Escalation is a normal healthy part of the security process. It's not it shouldn't be seen as a failure or some disruption into the business as usual activity. It is not about getting someone in trouble. Escalating is just a formal mechanism for getting the right people involved.

All right. And that links up with the second point. Security benefits from m from active management support. So a grassroots security effort where passionate individuals just work day and night is just not sustainable. Good security does not rely on heroics. And so shifting up means escaping the myth that leadership doesn't have to be involved. It you need to put them in a position to do what the leaders and what the managers do best. Prioritize resources, make difficult tradeoffs between features and security work and unblock cross team dependencies. The active support that you can get with management can be a gamecher. And what I'm saying is is that think of escalating as a way of just addressing the complex issues.

When you escalate a security issue, you bring leaders with the scope and authority to align the resources across your organization and to allocate the resources to tackle that problem systemically. So shifting up is about treating security as a strategic issue. It involves getting the right people working on the right priorities. And that's how we move from a state of constant noise and panic to an overwhelm to a respected product security program. Finally, we shift down and now pointing out that product security is a full stack engineering discipline. Circling back on the earlier iceberg discussion when we focus on the product we have to consider every layer of technology that the product is built on. So in the cloud

that means the security spans from the initial user request the gRPC call HTTP request all the way down to the silicon that it runs on. And when I mean by shifting down it's extending our security focus to the entire technology stack. So in the blue and the upper upper parts that is the land of the application the land of OAS top 10 and where many teams focus a lot of their efforts because it's the most visible part but then we need to move down to the operating system and the kernel there like a single vulnerability like a container escape can undermine every control in the layer above it. And then we must shift all the way down to the

firmware and hardware. And this is the lowest layer where the most dangerous threats lie. So imagine this hypothetical scenario or to me or me. An attacker who can compromise the firmware can install a persistent boot kit that survives reboots and is nearly impossible to detect. And now to tie this all back to the beginning, all the layers mean shifting down are also a part of shifting left. So when you're doing your threat assessment, when you're doing your risk assessments and your threat model, your threat intel, all of it has to take in account the entire stack that and from everything from the kernel to third party hardware to the application that sits on top. And of

course, we can go into each of them and like really dive into how the risk assessment piles on, but that's a really deep dive that we'll save for another talk. And now that I've given you like the directional buttons at least for the cheat code, I've shared you some with you some approaches that you can think about when you build your security programs. But if you remember one thing from today's presentation, boring security is beautiful. And now that sounds counterintuitive, but it's not about making security uninteresting. It's about preventing the heroics, the last minute firefighting, the moving toward a system that's resilient by default. And when you make your security pro program boring, it becomes more beautiful.

automate security checks and a and tasks. Human-led reviews are essential for a security program, but they do not scale for thousands of code changes that happen in the product world every day. Sometimes, depending on where you're at, thousands of product changes every hour. You can by automating some of the security checks for the common vulnerabilities, you can make the platform do the heavy lifting for you. This provides constant immediate feedback and frees up your security engineers to focus harder on the more strategic challenges than the routine tasks. Then you integrate security seamlessly into developer workflows. Automation is only effective. It is easy to use. You cannot expect developers to constantly const switch between a separate

dashboard to run a complex tool. Instead, meet them where they already are. Build security checks directly into the code repositories, their CI/CD pipelines, their idees. When security feels like just another helpful llinter, it actually gets used. Make sure the secure path is the path of re resistance. Also be select be selective with which controls that you implement. Like I've had talk in a previous life and when I was consulting I've talked with people who said hey we want to make sure we do a code scan a static analysis that takes 10 minutes and a CI/CD pipeline where efficiency was measured in seconds. there there's a there's a place for that static analysis code scan. It's not

immediately before every code push. Finally, we need to make security a shared responsibility because it is shared fate. Google talks a lot about shared fate and I'll oversimplify all of that just to say when security is automated and when it is integrated it stops being the exclusive domain of a small security team it becomes a natural part of everyone's job when everyone has a stake in the outcome and is empowered to contribute security becomes from boring to beautifully effective part of your engineering culture And so we've been talking about cheat code for security. It's not a it's not a shortcut, but it's a fundamental shift in perspective. We get stuck on the apps side, which is just the tip of the

iceberg, but the real risks lie deeper. The cheat code is part of a four-part framework. And first, of course, shift left. We need to think about security from the very beginning of the development life cycle. Think of threat modeling during design. Why? Because it's cheaper and easier to fix loss early. Then we shift right. Security doesn't stop at launch. We need proactive monitoring and incident response because the threat landscape is always changing. Security is a continuous loop. Third, we shift up. We need to elevate the conversation. Security is a business risk, not a technical run. This means buyin from leadership and making security a strategic priority. And then finally we shift down. You must secure the entire

stack. A holistic application from a vulnerable a secure application on a vulnerable platform is still at risk. So we need a holistic view of the technology. So shift left, right, up and down. And together we create a security program that's more integrated and more predictable. Beautiful and boring. Boring means it's working. It's a well-or machine and not a constant fire drill. And that's the call to action. Take this framework, ask the right questions. Are ask are we shifting in the right directions with this activity. This will help you build a security program that is both secure and resilient. Thank you for Thank you for your time. >> [applause]

>> Do you find any risks? >> Uh the question was do I find any risk frameworks particularly useful for the uh for the for this approach and uh one I will recommend is rapid risk assessment framework. So go like RA Rocks, it's open source, it's publicly available. Um, if you're just like getting started and you need to quickly identify quickly identify risk, that's a good place to start. And if you have compliance, the great things about that particular framework in general is that if you have u a compliance component, they're easy plugins based on that. Excellent question. Next one. leadership >> and so I'll surface this and what the question was is like what how do you get

leadership buy in on this approach and the the simplest answer is product like product is the answer and what you then what business case that you can make in your organization is like hey we have a list of products. We have a comprehensive inventory of these products. These products are the most revenue impacting. These products are depends like the users really love and really crave and if something happens for these products, it it hits the bottom line harder than for these other products. Doesn't mean you neglect those other products. It just means you focus intens you switch and focus intensity. >> All right. Anything else?

So question was when you're shifting down, how do you keep from squirreling and rabbit hole and like um like good scoping from the beginning of the engagement? That's the answer. When you're focusing on the threat model for pro for product for product foo for example foo the product team for foo has explicit and granular controls for certain of their assets than others. So as you're as you're look there's an easy way of doing this. is like you're for the asset and the systembased part of the product security review. You're looking for misconfigurations. If the risk is inherent to if it's a subproduct and the risk is inherent to that subproduct, that risk belongs more in the threat model for a product bar

than it does for product FU. Does that make sense? All right. >> No.

>> I guess the resources to shift in which direction >> I I challenge that. Um the question was is like if you're a smaller company, do you only have resources to shift in one direction? And I do challenge that. But uh and I challenge that assumption like if you are a oneperson shop shifting up and down is within yourself. Like if you're in a twoperson startup with like a businessled founder and a technical founder escalation procedures are easy. You're sitting in you're sitting in an apartment in the Bay Area eating ramen staring probably at the same monitor together. So you just nudge and it's like hey I think this is a problem. You talk to your co-founders like hey I

think this is a problem. Um I don't think we I don't think we could raise funding this way. Like so it doesn't matter how big or how small you can shift in all the directions.

requ

is there does there need to be like quantitative financial analysis for every escalation and the answer is it depends and I'm going to steal from my old consulting job and say it it depends At certain levels of escalation, it will be more efficient if you can provide that information and get that that quantitative an analysis. But you can also just speak to it just from technical hazard and technical risk of systems like like when it doesn't always have to tie into a dollar amount. And if your escalation procedures in your organization are structured enough escalation level zero, you just talk, you just talk shop and you do the technical as it's going up and as it's

getting prepared for higher and higher audience collaborators will come into the it and will make that quantitative analysis easier and if that is what is needed to really make the case for that escalation. Excellent

question.

down. So the question was is like dispute resolution between security and product like is a critical really a critical and you can stand firm on the technical parts of your assessment like does every finding mean that the entire business goes down? No. It does not it does not make business easier. Why would why would we continue to operate in this? Why would we continue to operate with this with this anchor around our like with this ball and chain around our neck? Like it doesn't make sense. This is preventable risk and this is preventable risk. And most of the time when you're highlighting it, it is um business is impossible to remove all risk from business. So

really assuming that the person that you're actor with is holistically motivated. You can have the discussion of risk levels and like okay crit versus high versus medium all of that. But is is moving forward uh an acceptable amount of risk with it with this whether it's this product launch whether it's a new feature all of that make sense. Yeah.

Going once. All right.