Scaling Automated Code Analysis Across an Enterprise

um application security process and we had a need to to scale it across the entire company to really reach all the hundreds of of teams that are building products for adob and I want to tell you the story but before I get into the details of the story let me introduce myself I'm Flor nerding I'm a principal security architect at Adobe and I've been um by now 10 years with the company 10 years ago I joined the company as a software engineer and by heart I'm still a software engineer and and I'm very empathetic to to the developers and Engineers building all products five years ago I joined the security organization as a security researcher

and I did um Security reviews with teams across the end well probably 80% of the company and two years later so three years ago I started to to build project Kodiak that's the logo the Big Bear there wherever you see that that's Kodiak that's our automated code analyzis platform so I was initially the the designer the lead engineer I became an architect 100 of all the work to the team and nowadays I do um security strategy I take care of all the things that are shift left that happened before our code gets released to production or otherwise shared with with our customer or products fun fact I've got a block most uh liked articles there

most often few it is a German bread starle recipe I bake my own on bread I'm originally from Germany and with that let's get started with um what is automated code analyzis anyway so I understand on automatic code and analis a few things we want to look for secrets and source code and I brought here a snippet and I've asked CAD GPT to generate a a small script that um talks to an API here and um jgpt had the great suggestion to put the secret directly into the snippet so it literally said replace this value here and I put a random value in there oh don't do that people accidentally commit these secrets or sometimes on purpose because they

they don't know know better to to source code and once they realize that the wrong thing to do they do another commit to remove it but it's still in the history so we really need for to do the um to look for these things also common thingers you did do a get commit all um and then you have the your environment fils if they haven't been in the ignor also in your history so we want to look for things like that so that we tackle this security risk the next kind of um security issue we want to tackle is um vulnerabilities in third party libraries and that's the XK kcd comic I guess most of you know it it

symbolizes that we are standing on the shoulders of giants we make use of so many components across the the entire open source ecosystem in our own products that a vulnerability like in XZ which happened a couple weeks ago it's really important if you have a back door and say where are you actually using that could you answer that question quickly how hard was that especially software composition analyzis in the CN C++ space is hard because there are no package managers for Ruby package python package Java it's all kind of easy but this is a challenge so we wanted to create an inventory of our third party components and use this inventory to figure out where are we vulnerable to

these um uh where we have vulnerability in these dependencies the third aspect is static application security testing fast for short sometimes I also call it simply static code analysis so we want to look for vulnerable patterns in our source code that might be simp simple structural things or data flows data flows are very good to capture injection issues where code and data are mixed so these are all first party vulnerabilities here I've got another example minimal um Snippets that implements an API end point and we have a source of t data this uh username variable that gets filled with attacker Control Data or can be filled with attack Control Data and then unfortunately we just

concate this attacker Control Data with a SQL command with a query that gets later on execut in in a sync without doing proper output en coding that means this application or the snippet is vulnerable to SQL injection so we want to find things like this in an automated fashion too there are a whole lot more things that that you can do but these are the three things we've implemented today there's infrastructure as code we could look at that too you can look at uh API definitions and learn a whole lot about how your program how your software products actually work and how they interact with their ecosystem but that is all future talk so let's focus just

on these three things the key challenge at Adobe is um that I've tried to capture in in this beautiful image that I've generated we have hundreds of products we have desktop products mobile products we have web applications web services we have a web server and a bunch of other stuff so they are all different very diverse text Texs we have 12 programming languages and these 12 programming languages only make up 80% of our code if I want to go to 90 95% are a whole lot programming languages so basically anything other than the net ecosystem is widely represented across Adobe including some very exotic programming languages like Alexia closure and so on which makes Security reviews and static analyzis of

course very hard even if we look at only one programming languages often we have multiple Frameworks so it's all different if you work in a big Enterprise you know Enterprises acquire other companies and for the Acquisitions the standard operation as well let's make more money integrated into the products but not necessarily on a technical level things get fully integrated so this is the natural state everything evolves to more complexity more and more complexity we even have multiple source code Management Systems not only from a single vendor but multiple vendors and we have multiple installations from multiple vendors luckily we have one mayor installation which covers around about 90% of all source code but there's a lot of complexity here which needs to

be tackled by an automated solution finally we have a whole lot of uh the developers and many many repositories we have over 100,000 repositories not all of them active but it's kind of hard to know um what is still being used and what what not and we have more than 15,000 developers or engineers in general working on on our various products in total these Engineers create around about 30,000 code changes pull requests comits something like like that per working day that we need to scan and keep up so that's the challenge let's talk about the design of our automatic code analysis platform how we rolled it out and what kind of impact we had on the

company and the first part is really design principles when I reviewed what worked and especially what didn't work at other companies and at adobi before we built uh Kodiak I learned that a great developer experience is a make or break of any such system and this can can be shown in uh or one one key aspect is uh you want to really really deeply integrated into the developer workflow you don't want that the developers and Engineers come to security you want to go to them they don't have time to deal with security as as much they if you go into their workflows they'll just do the right thing because you let them know Well there might be a security risk here

there's a concern here please tackle that we needed to provide timely relevant and actionable feedback and if you compare that line to JC's Keynotes mission of of the Marines this timely is a very very same same thing relevant same thing actionable same it's all the same we we want to provide a lot of value to to the engineers who are working on building our products because security doesn't earn money to Adobe it's just the necessity of doing business and of course we want to shift left that means we want to give feedback as early as as possible well not literally as early as possible but still early in the in the um in the workflow

in the life cycle through which code flows all these tools that exist out there to do code analyzes B that secrets in Source Cod detection vulnerable dependency or first party flaws like um um data injection issues the tools are all very noisy some of these tools generate only 20% signal and 80% noise so you have to be very careful how you balance it what are your expectations to Engineers should they fix everything should should they look at all these issues or only tackle the most important ones and finally instead of having uh Engineers work with five six seven eight tools and the P request we wanted a single tool integrated MERS always ask me why did I build my own platform and

that is the reason I can't give many different tools that have different ways of prioritizing things and there's no clear way to to say well I should be fixing this first versus that other thing so we have one single pan of glass in into findings in source code so the overall goal of this project is a pragmatic risk reduction and that means we are not going to zero risk that's not the goal it's never the goal in security otherwise well switched off that's more more secure but that doesn't earn US money so we really want to be very pragmatic about it this diagram is an example how of simplified how Engineers write code they write code they test it locally

they commit and push it into their Version Control System they create a pull request and then they ask their co-workers for feedback and then they merge it and then they somehow deploy it to production and that's out of scope for this diagram and we had the idea and what I've learned by looking at other tools and tools that were successful and exactly in this gather feedback phase we should integrate our tool and that is where Kodiak gets notified by the source code management system that a pull request was created then we analyze it and give automated feedback and it looks pretty much exactly as if a human had reviewed this code so we just aned the specific

lines where we have a concern and we add a few additional commands if you find something and that is timely simply because the pull request was just created it's fresh in the engineer's mind and if you then receive within minutes feedback about what you've just written you're also very receptive to actually act on ons this the thing is actionable because we give very good guidance on what to do about these fundings we explain it and I'll I'll go in in the next slide well in two slides into more detail of of that and it's relevant because we only give or mostly give feedback about these things that are have been changed in this pull request so the idea is not

while dump the pull request full with all the findings you can find on on this Branch but only these things that matter in that context and you have to be somewhat quick most teams have pull request Builders so you have plenty of times that that are processes that make sure that the um code doesn't break any expectations that the test still pass and especially in compiled languages that takes quite a while so we said in addition to anything TimeWise that that is um necessary to to test um the PO request by the engineering teams themselves we have five minutes on top of that by default so for very large and complex project it might take take longer but that's the

number we generally want to add this platform enables a very small group of people to write security as code these are simply um rules for our tools captured in our in our configuration files to scale security efforts a good across a good chunk of the company so we can't always reach the entire company because well a rule is probably programming language specific or at least ecosystem specific specific for software composition analyis CC vulnerable libraries but this is a huge leverage point with very little effort we can reach basically all the developers especially for efforts that need to spend the entire company because we have um a an um some objectives to May well for example fix an a broken API

and make a and deploy the more secure alternative this tool is what helps us do that and for for risk prioritization we feed data and and adversary intelligence into Kodiak and I will talk later on about um how to think about adversities and what does truly matter to fix because there are so many issues in source code you can't fix them all feedback loops that's really the key design element of any um SE security process how do we get Engineers to do the right things and the best feedback loop is of course preventing sta but that's very challenging given the complexity of adobe with the many different text I can't build a pave pass for every

framework out there in the world doesn't work so we have this inline feedback on pool requests where we are giving feedback only on files that have been changed and this is really important and we learned that early on because we made a mistake in the first version of kodc we just dumped all findings into the pull requests and of course that did work well luckily we started only with a very limited set of capab of things that we could detect so it wasn't worse we caught it before we had thousand of findings in the Prest and nowadays we only make very few exceptions if we have something that is especially cretive we break this rule but still the um um the

inline feedback is for net new code or any code that gets changed and these things are prioritized and let's now talk about the next one you see there's there's a gap in there there's something else that's technically more important but this is the next thing that we built we built a custom dashboard so that developers could view all findings on any branch and any branch is really important because we have many products that have many long lift branches especially if you have a desktop product then you have um a long-term support version and you have multiple patch levels and all these different versions of a product are affected by different security issues over time only some things have been

patched and we want to enable teams to really have an easy way to to look up well what kind of mner abilities matter in each of these releases that are currently being supported the biggest challenge with that is we're breaking the rule of going to the engineers and and developers instead we asking them to come to us and it's not yet working well and we are and that is the next next thing um that I want to talk about we've started to drive security campaigns using metric so we are picking a a key concern for example secrets and Source codes or fixing a broken a API and um we drive down exactly this uh this risk by

measuring how widely this risk is deployed across it Adobe and then we drive people to the dashboard wi this campaign this is still work work in progress so we are still learning how how this works really well but I believe this is a powerful way to get people into the dashboard out of their workflow and somewhere else and the final aspect of this is enforcement the three feedback loops at the top it's nuding I'm just asking the developers please take a look this is a security risk but I'm not forcing them to to do anything about it and sometime some teams have the challenge that they face a lot of pressure to deliver products on their road map new features

critical deployments and they might not have the time to work on on security because they optimize with a very short-term focus at least at Adobe e a culture is largely if there's a ticket work will get done reliably that is so we reserve ticketing for these rare cases where we know there's a very critical issue in there and only then will we file a ticket and we are currently testing this with a few uh better teams how that works and then we want to scale it across the entire company you'll see there there is not yet blocking the pull request in there that's something that we'll do in the future maybe but blocking the pull

request means that our platform has to be always available because we can never um be the reason that a critical system orage in production can't be fixed so we need to be very Mindful and empathetic about the engineers who work in this situation I have an on call problem and need to fix it with Kodiak we decided to to build only the parts that are custom to adob that means the integration into our various source code Management Systems the integration into the workflow and have the capability to customize tools but we are not building our own scanning tools so it was really important how do we select scan tools I don't want to tell you which tools we use because your

situation is certainly different but I can tell you the um method how we chose the tools so the uh surprisingly the most important aspect it turned out was how easy it is to roll out as uh code analyzis tool especially static code analyzis tools for the data flow analysis often need um build system integration and since I have so many different products which all have different build pipelines it means I have to ask every single team please roll out this tool and integrate it and keep it up to date and you know the only thing they get out of it shortterm is well more security work that's not compelling so we decided to procure tools and choose tools both

commercial and open source that are really easy to roll out that only need source code access nothing else many vendors will will tell you their tool is has better scanning capabilities because it either uses a build artifact or is integrated into the build system as a compiler basically but it doesn't matter if you can't roll it out the value here is getting security findings and if I get 90 or 95% of the quality of a tool that's built integrated with source code only good enough for me the one exception here is probably C and C++ so there I I believe this source code analyzis will be limited but it's still a good first step let let's let's use it

and then move on to to the next things there are vendors out there that tell you they can find all find flaws with a 100% true positive rate and zero false negatives with the overest Benchmark that is a a test suit for static code analysis tools that has many injection flaws in in it but I don't want to find everything I don't care about that I want to find things that that matter so what we did was very simple during the proof of concept phase of these tools we used an older version of our codebase and scan that we knew what kind of security issues were in there not all of them but some of them

of of course so we wanted to see if the tools could either find these or could at least find them with a little bit of f tuning and that is find important vulnerabilities these tools uh these tools output is primarily zent to Engineers you can't expect Engineers to be a security expert that's your job that's my job to be a security expert and writing secure code is really hard for me too and for an engineer who doesn't have this death and breath of security expertise like we do it's even harder so we want really friendly output there should be a proper explanation what is the concern why is it a problem and how can you fix it if we don't have

this output well then the tool isn't useful what should they do with with a finding well there's an ssrf if you attach that to a finding well it's not helpful you really need the details what to do about it fast enough on the previous slides I I mentioned we want to give feedback within five minutes plus build time most tools can can do that if the code base is really really big yes you might have to configure it so that it doesn't run all the rules but usually that's not as much as a problem as I had thought so initially at least with more modern tools and finally customizable if you a big enough company you have lots of

codes that is very specific to your own company you're creating probably your own Frameworks your own libraries and you want to detect issues specifically with that so you need to have the capability to write content for the scanning tools just as if you were the wender or of of such a tool and not all tools are customizable like that so that is also important but this aspect is least important so you will start to to roll it out then you'll learn findings and you'll find unit and eventually you'll get to this customizable aspect now let's talk about the scope so what kind of repositories should we actually care about of course we want to look at all repositories but that's

actually hard to know once you're in Enterprise because usually you don't know everything that's going on there so we can only tackle known source code Management systems and at Adobe we have a ground rule codes that goes to production must live in an organization owned repository and not be associated with a specific user or employee account and then on these repositories we run all tools and we have onboarded round about 98% of these repositories so that is the primary scope the secondary scope are exactly these user owned repositories that I just mentioned but we are mostly worried about secrets in in these because if you have um a user owned repository that is public or in internal then well anyone

can can look it and might be able to find a secret in there but because this code is not allowed to go to production we don't need software composition analyzis or um static code analyzis in there although most of these user owned repositories are just for of the organization on repus so that's a future and I think acceptable sorry that's an acceptable tradeoff and this is still in the future of Kodak we have not yet built this part the more challenging aspect are the known unknowns so we know stuff that we don't know Shadow it both somebody is running a source code management system or just uh system that hosts a few repositories somewhere where you don't

know about because oh it's only used by a few hundred employees on the scale of 10,000 or adob be around about 40,000 employees that's hard to find some TS you can be smart and talk to the um to your um sock team and ask them oh we have EDI everywhere are there processes that look like uh git is running there on a on a server well then you might be able to find things but still hard hard to find and then there are also um repositories hosted on public platforms like get upom get G lab bit bucket and and so on and sometimes strange accidents happen that employees push um source code with secrets and sometimes

without to these public repositories in their own space and it's in a public rep they realize a mistake and then they undo it but the secret has still been leaked and you know there are systems that monitor all public activity especially on G upom so these secrets leag there are dangerous too but for Kodiak this is out of scope and instead the IDE is to rely on Perimeter monitoring for this aspect so now that we've defined the scope how did we actually roll out Kodiak and what was the process we started by with a very limited scope drink your own champagne basically we started in the security organization and onboarded the um the git repositories of projects that

we've maintained ourselves onto Kodiak to see if our tool did the right things if if it wasn't um annoying to to our own engineers and we got a lot of valuable feedback once we were happy with this integration we started the roll out to Friendly engineering teams and we relied on the contacts of of of my my my team and and myself and a bunch of other people to ask as these engineering teams would you be okay if we integrate into your pull requests and you will get feedback and it looks like this we had screenshots and stuff like that and we told them you can help us make this better so that we can learn very early

on before we roll it out across the entire company and to this day we have sessions with key engineering teams that tell us what we need to do better with Kodiak and that is very important we want to listen to them we can't of course Implement all their suggestions that's there's too too much to do and we have limited bth but we really can make an effort to fix the most um the the biggest concerns the engineering steams have with our integration into their workflows because we truly are in their workflow then we started rolling it out across all Adobe on the slide deck I'm saying well we did first the non-big repositories we didn't know that first

we really rolled it out to everyone step by step but we realized there are very big repositories out there that are multiple gigabytes in size for some of you that might sound small but our system has a complete copy of all source code internally and we need to shuffle this around to get it scanned and it just didn't work well enough initially so we initially removed the big repositories and most repositories are small couple hundred megabytes of of code so that was not a problem today we support all repositories across Adobe all these 98 plus per. um for repositories that are especially big we offer uh gracefully degraded service that means instead of scanning every

pull request we might only scan once a day or once per hour something like that the order in which you roll out the scanning tools was important too we started with secrets and source code and we had a small trick to make this um this better we said we are only going to inform Engineers about secrets that we can actively validate that means we had a 100 True positive rate and we built a whole lot of trust with the engineering teams by saying kodc will initially and initially was really important to stress that show findings that we know are true positives you won't see anything else yeah yes so we had um a vulnerability scanner

that that uh looks for secrets and and and source code so it inside the source could like on the yes and the the um having a secret in Source good is a vulnerability too just just imagine your Source good get gets leaked yes and if you have API keys so a couple days ago this vendor sense um had its API key stolen we don't yet know exactly how but s made an illusion to well maybe why a source code so it's really dangerous we have to be very careful that we don't put secrets in source code also if you share your source code with many employers inside your organization inside inside your company or Enterprise that's good for

developer productivity all Engineers have access to all source code but if you have one engineer system comp compromised that adversary now has access to all source code and your G repositories or other code repositories are a pivot point to get in the rest of your systems and that's a catastrophe that's why this is so important and we'll talk about this prioritization later but that is also the reason why it is so important to start with secrets and the trick here is look for only things that are actively validated and then you get more Buy in easier from the engineering teams

no um there are open source tools for examp example truffle Haw truffle Haw is one open source tool there's another open source tool that's called git leaks so that's a command line tool that you um run either in the direct in the root directory of of your um of your checked out gri repository or you point it to a git server downloads all the history scans all files and it looks for reg specific regular expressions and sometimes also entry based um well things that look very random random strings and then raises them and a tool like truffle Haw is able to act travle Haw like the thing that looks for the the P the pig that look

for for truffle so it's a pretty cool cool name um that they can um because many secrets are have a prefix or suffix they can look um they can then take the secret and talk to the service provider that um has issued the the secret and then they know if it's valid or not and they can give my platform this information too and we say Well only tell us about secrets that are actively validate there are lots of trade-offs so you eventually need to tackle all these other secrets too but that's the the basic idea okay this uh deck uh the next thing the deck talks about is software composition analyzer and that is what I would with

what I know today recommend you to focus on but in practice we started with static application security testing but let's talk about scar first so scar allows you to create an inventory of your uh especially open source dependencies and it is generally easy for an adversary to attack a vulnerable dependency than look for a first party vulnerability in your Cod and that's why I believe this is so important also your the customers of your companies are probably going to ask you are you impacted by the X back door were you impacted by lock for Shell all these big internet wi vulnerabilities always B down do you know what's actually in your products so that's important then we did

sest we're still working on on that repository settings it's strictly speaking not automated code analyzes but you can also assess the security stance of the git repositories for example is Branch protection enabled can anyone push to the main branch is code review required how many re reviewers are necessary to get cod in production and finally eventually we want to add infrastructure as as code and there are a bunch of other things we could tackle in the future and this is the outcome of what we have achieved in in the calendar year 2023 so we got across the entire company across the languages that we we suffer support and the uh frame Frameworks and um package managers we

support we got more than 300,000 code fixes out there and now imagine we have about 15,000 Engineers that round about um 20 fixes per year per engineer that's a whole lot of engineering p and we got that by ning only so we didn't ask anyone or tell anyone you have to do this they did this just because we gave them inline feedback this is a very powerful tool but the key challenge with that is and that leads me to the next topic um did they fix things that truly matter have we asked them to spend this time on things that actually made the business more secure or was it just a waste of engineering effort and we slowed them

down I keep telling my team the kodc team that our platform is a weapon to to minimize developer productivity that's a thing it's a weapon so we need to be very mindful how we prioritize security issues so that we don't slow them down the engineers are earning our paychecks I'm not making any money for the company the engineers who build the products who help ship great new features to our customers and make it compelling these are the people who earn our incomes so I have to really make sure that they work on the right things and don't spend too much time on security so this is a very intricate balance and I can't I can never give a

perfect answer but by talking about risk prioritization I can come up with a model what is important to me now let's talk about adversaries and this is highly highly simplified because we have today limited time and I could talk about ours just in on this topic so they are security researchers back bounty hunters and if you want join our bu wary program too and you can earn earn some money the guys over there in the booth can even give you a a voucher so that you can get a little bit more cash if you find something in our AI related systems and these security researchers have very different motivations maybe they want to break into the industry because they are not

yet in security they are in engineering or something else related they want to build skills they want to make money and lots of other things but the key thing with the security researcher categorization is they all friendly they will not lead to a zero day ex exposure some might tell you after the 90day as well we are going to make this public and still force you to do something about it but at least no zero day leaks and from a strategic strategic perspective I few these security research especially in the form of responsible disclosure in a bu boundy program as a way to identify gaps in a security program so you get feedback about how well your various security

systems are dealing the next adversary kind are e criminals and these are often or maybe even almost always financially motivated it's a business to them so they want to make money out of the misfortune of other people so to have a business you need something that's repeatable and scalable if you look at um the lock bit ransomware and ransomware is generally a fantastic example for E criminals they look for vulnerabilities that they can exploit at scale and unfortunately our schools hospitals and many smaller uh companies get targeted because the these basic vulnerabilities if you don't have a dedicated security team are hard to fix so the strategy for these kinds of criminals is our adversaries is we've

want to focus our efforts on things that are easy to exploit or at least uh widely deployed vulnerabilities first that means something like software composition analyzis a vulnerable third party dependen is a better Target for an e criminal than finding a SQL injection that's um deeply hidden within your application the next group of adversities are nation states and the general idea is they have operations with targeted outcomes that might be pretty much anything might be Espionage um destruction um to prepare for future conflict so that's a lot here that that's going going on generally this is hard to predict I'm also lumping in to technically not nation states individuals who um just want to cause

harm and just cause this destruction because they are hard to predict too and the only strategy that works is defense and depth so there's not a whole lot we can do prioritization voice but have defense and depth so with with this this basic model we can prioritize things we are going through these different areas first secrets and source code so the easiest to find secrets of are of course secrets that are in public or internally public repositories anything that's widely shared so let's focus on that first then we want to focus on easy to abuse credentials and that especially means cloud provider credentials if you leak an AWS access key with the secret on get upom within

one minute that key will usually be found and uh people will try to see if it's valid so you have one minute to abuse so you really need a preventive strategy here then anything that is active and long lift the focus is on Long lift because if you have a tokens that's only valid for 24 hours well maybe it's not worth actually roking so that's a trade-off you have to make and then everything else and in the past we have asked Engineers to remove no sorry to not only revoke these secrets but also remove from the sourc good history but nowadays believ that's a waste of effort well if the secrets no longer valid what harm is it to have it

in the history rewriting G history is really expensive you have to Halt the world rewrite the history to remove it then everybody has to think again don't do that please that's not empathetic to the engineers that earn your money that doesn't make any sense software composition analyzis so many breaches start with out of date systems and that's one of the key reasons and well credentials is even worse I forgot to mention that so which cves are most likely to be exploited well cves that are already exploited in the world and Sissa has a list it's called the CF the known exploited vulnerabilities catalog it's a bit between 1,000 and 2,000 cves you really should patch these

things also it's a brand problem a PR problem if you get owned for a cve that's widely known since a long time then everybody else will conclude your patching strategy at your company isn't working so it's kind of important to to patch that stuff first then the next thing I we are patching is CVS where an exploit is publicly available because if that's available it that is an indicator that exploitation in the wild might may happen it's not guaranteed so and this is not a strict priority order but that's the best way how I could visualize it so you need to wait these different things in different Fashions likely to be exploited you know in my experience maybe

2% probably less than 4% of all CV somehow impact the applications I'm dealing with so any cve and some people call it a curriculum bet enhancer because it's cool to have found the are kind of a denial of service against the entire software industry because everybody now has to patch that thing and look where is it deployed and release uh patch nodes and and so on it's a whole lot of work but most CBS never see exploitation in the world even if an some in in some case even if a public exploit is available because the place where the C the CV could be exploit is so deep inside our systems that first other defense and DEP measures need need to be

um um exploit and then you can only exploit it and first the organization that is also maintaining the CVSs scoring system is maintaining another scoring system and that would be another talk of its own it's called epss the exploit prediction scoring system or this score is a value between zero and one one means in the next 30 days it's extremely likely for this cve to be exploited and zero it's very very unlikely and for almost all cve the number is very close to zero so theyve trained a machine learning model on uh on that compares past data where we knew which CVS have been exploited based on the just the textual description with lots of metadata off CV and this thing

is actually pretty good so and it's a way to reduce the effort you spend on patching to things that probably matter so you could just uh the standard strategy that most companies use this p any cve with a CVS score of seven and higher so all the highs basically but unfortunately we see exploitation in the world even of cve with scores of three something and five so you want really to patch all threes and fives for security your teams don't have time for that and that is everything else but unfortunately we have customer expectations so Adobe sells products also into heavily regulated Industries especially the financial service industry and for our on Prem products that might be a desktop application or a

web application that is run on the customer's own Hardware they run their own software composition analyzes tools across our products to figure out what kind of vulnerabilities are in our third party dependencies they tell us sometimes please fix everything so here's now a a very important split we have um security work that follows this priority list and then unblocking sales on the other and that's just cost of doing business that's not cost of security and I'm very vocal about this distinction because I believe it will get worse soon the US government and in Europe too and other countries worldwide there um there is an increased focus on software security so and this will manifest itself in the need to publish software

bill of materials which is exactly this inventory list to our customers government and otherwise and they are asked to to I expect them to ask to fix things according to the CVSs score and not to this more uh this um more risk Focus strategy based on these customer expectations I it's not an urgent priority me for me to filter out unreachable cves so the security industry invests a lot of effort into this area but I'm not yet fully convinced especially if you have to share an ASO well they might not even care if it's reachable or not not just fix it please and in the future we want to use conal data yeah

so the question is um how is effectively how much do I trust reachability analyz and how do I tell customers if this is truly the thing how do I balance a human looking at it versus a machine telling me we haven't yet implemented this simply because I while I believe it would be certainly helpful to have the context it's not really important because too C many C customers are expecting us to fix everything anyway so in the future we want to do this and I believe we'll just have to rely on on the vendor and I'm speaking already to various vendors and tell them I need detailed information why you believe this is reachable or not and don't tell

me a a binary thing it's reachable or it's not reachable tell me you're absolutely sure it's reachable tell me you're absolutely sure it's not reachable and then give me an in between value for confidence but that's for us in the future yeah it's it's much much simpler the idea is um let's say you use a you have a crypto library and the RSA functionality is broken if you don't use the RSA functionality what's the harm of the CV in there so that's the the level of granularity possessed I haven't yet figured it out so the basic process of exploring in the world ands on still works but if you have figured this out please talk to to me I would like to

learn from from you the key thing I'm thinking about is this is not only CVS common vulnerability exposure there's common vulnerability enumeration so SQL injection cross descripting and so on how can I map this to adversary techniques tactics and procedures and finally we want to have unified risk prioritization so we are telling our Engineers please fix um explor in theal stuff first then fix your critical secrets and then likely to be exploited vulnerable dependencies than everything else everything else is a little bit more complex but um you will have to find your own uh weighted prioritization scheme here and now the to slowly close the presentation I want to talk about the meta feedback loop so how are we

thinking about the whole system and how do we want to improve it by going through the OA Loop the uh um that was mentioned in the initial um the first session today you can think of Shifting left not only in the terms of is it early in the development process like writing code or late like being deployed but also in terms of six buckets of risk one bucket is it is exploited you have an incident it's not yet found you found it externally bya Buck Bounty program for example found it manually if you had a human look at things that might be a pentest and a redim and you might argue a redim is found externally but it doesn't matter

that much here I guess then you have uh Security reviews and a bunch of of other things font manually font automatically that's what Kodiak is is doing for us we have scar sest but dust fuzzing and all the other things you can automate are in there but the coolest thing would be really prevented so secure by Design secure by default and pave path but as I said at adbi this is really complex to do at scale because everything's different so the key idea is think about fixing root causes prevention so just detecting secrets in source code that and asking developers to fix that is fixing a symptom there's a reason why secrets are in source code why it is hard to not

have them in source code there's a reason why people have SQL injection it's not because they don't know about the reason is because they've not been taught to use a demon arm an object relational mapping system so fix root causes if you can finally all these things give you feedback anything that is found on the right side hasn't been caught by something further left so you have program gaps and I mentioned the bu boundy program as a key source of information here merge that with what you know about adversaries your uh business strategy very important and then you have a security strategy and invest in these different aspects so the key takeaway is I believe

every one in sec has to focus on great develop experience and talk to them make sure they you really understand their needs the feedback loop design is crucial and if you only want to take two things away regarding fixing what truly matters consider using epss for software composition analyzes and incest try to Target your root causes instead of fixing symptoms thank you

any questions

yeah yeah so the question is uh why we chose to build our own system instead of using a commercial vend solution that um basically does all of this in an automated fashion for you so it's a make or or buy question really the key challenge I had was that um the Wenders who had the best sess tools didn't have the best of the composition analysis tools so I already had to work with multiple Wenders and secret scanning is even a another problem we also figured out that um the majority of the work is actually integrating any wendor based tool into our complex internal landscape so focusing on these things that are customed to Adobe and

abstracting the security tools away from the company enables us to in over many many years have the ability to even switch out security tools because we abstract them away from the company the rest of the company doesn't really know which tools we are using they sometimes ask but they they don't know so um I can focus on things that are specific to Adobe the other reason is tools like Kodiak didn't exist back then that could do this merging of multiple different scanning tools and there are cool side side effects I didn't have time today to talk about the architecture but we have a code search engine too just imagine imagine having all of your company source code at your

fingertips and you can enter any regular expression and give an answer within seconds it's fantastic for a security team if you have an incident we the index is yet is not yet kept up to date we didn't have time it's more proof of concept but it's a fantastic capability so having the ability to really customize things is powerful but it depends on scale so if you're a small company well just buy something if you're really big Enterprise tens of thousands of Engineers oh then probably custom better answer

yeah yeah we can't in short business logic errors um are not something that I've seen static analyzis tools do well with static analyzis tools can figure out a a a pattern in in source code for example you have a print statement in in a for Loop that's something sess tool can find easily or you have a data flow from an variable in your API to through various transformations to a function that executes a sequence statement um but business logic access control is super hot so maybe in the future we get AI to do that we'll have to

wait uh fuzzing is is not in scope for this but we have other projects that um fuss our uh CN C++ code yeah it's closer to our Des team yeah

yeah we basically rescan and anyone can wi the dashboard um issue a request to rescore their ESO yeah and it's also about finding CVS just because you scanned one week ago doesn't mean there um well if there's a new CV in after that scan well you have to scan again to to see it yeah I've got time for one more question

oh