Rahul Raghavan: The Clutter That's Choking AppSec

welcome to the first session of the uh means i'm hoping everything goes well as far as the presentation is concerned it's probably my first friday the 13th presentation that i've done um but welcome to the top that's talking ab sec um the full version of it which is actually called the clutter that's choking app second how to fix it but it's just that the title was too long that's joking upset a little bit about myself uh before we move in so i'm rahul i'm one of the co-founders um at v45 we are an application security company some of the app at night are listed here a huge fan of appsec and uh devsecops automation um

well pertaining to software security uh threat modeling um automation and so on so we're gonna talk a little bit about some of these things uh here today usually when we do the sessions and this is probably my um 20th time that i'm presenting this i first had a version of this presentation um earlier this year and ever since we've done probably 15 or 20 times and every time i do this one of the things to get a sense of is in terms of uh really the composition in the room in terms of how many of us are in apsec and how deeply i don't see a uh you know a a way here for us to kind of get a

a show of hands but i'm assuming that all of you here are seeing all the other uh do with application security uh with that in perspective let's move south um with really um the the product engineering um uh state of uh affairs today right so that's happening from a product engineering perspective uh most of it relates to the change from just the delivery of product deployments you kind of move waterfall to a file and then there are people who stuck in between the two which essentially is agile fall or agile fail as they call it it's probably an urban dictionary term but most of them are seeing a lot more shift in terms of go to market with their products

um and and obviously that comes with what i like to call accelerated deploy additional catalysts for deployments of products i'm talking things like chef puppet ansible the whole devops movement has kind of which engineering um kind of moves today and we're also seeing a lot more in terms of an adoption of uh microservices and um in terms of being products being architected by themselves so a huge shift not just in terms of the way products are are in terms of how the products are architected and the product the way products are built the team structures and things like that and and to a large extent um itself you see a lot more in terms of um

the speed of testing by itself so so that's some a huge change in terms of of product engineering from the past years from an application delivery perspective right so i kind of see to i probably want to divide the whole application delivery um um types of customers if you will right you first have um typically i'd like to call it the difference between the east and the western seaboards of north america right let's take the example of the eastern steeples you're essentially talking about large banks and insurances uh who probably have a large number of applica but many times in a year so you're probably talking about 3000 4000 applications in some sessions in some organizations

but you kind of see them them kind of happening once or twice a year and then you cut to the other coast which essentially is the typical bay area silicon valley kind of companies are two applications but then that's incredible in terms of the iterations that they have uh that they do you mean we've all heard the story of amazon in seconds but this was way back when amazon had just adopted aws and and we're talking almost 10 years later and you can only imagine this will actually deploy so from an application delivery perspective you're kind of keeping up with this with this huge deployment either happen in terms of scale of applications or the scale of iterations you're kind

of seeing both of these amazing uh views of the world from that companies in between right you you typically have uh uh the small and medium segment between both of these worlds purely from from from a delivery perspective right they have a massive number of applications in terms of their portfolio have few of them that have more in terms of iterations from that and unless and until there's been somebody who has um uh who's who's really been under understood over the last 10 years so has heard of devops um and the devsecops were and and if there are fans of gartner in this room i apologize in advance but it's just that the current view of the

cop which is the infinite circle of gartner is slightly confusing to a lot of us and it has been for me personally um is go ahead and change the way we'd like to look at devsecops from that perspective right so what you see on the top are the typical software engineering uh phases are on our stage during the plan build test release deploy operating monitor and what we've tried to do is on the bottom we've essentially tried to map out the typical phase modeling static analysis dynamic analysis um security and infrastructure as code and attack attack so devsecops in my view of the world is really trying to retrofit the security phases of its respective

uh blocks if you will in product engineering and that's really what the what the what the color code here is for example threat modeling is something that starts off with plan it used to be just something that used to be done at the design phase but as you keep moving forward with things like threat mode modeling move just from the plan phase and it kind of move moving into the whole code aspect of things as well now static code platforms which is largely core its way through bill and a little bit of test as well so you just kind of extrapolate that that statement in terms of how vague engineering are really trying to kind of

expand themselves into their respective positions in product engineering and this color code here has changed see the orange probably move more towards the right as we keep shifting thread modeling left um and the other things would kind of retrofit just kind of put this uh description uh or this representation of of of depth tech ops and devops uh uh just to throw an idea out there um you know um put this idea that i kind of be interested to see in chat if anybody have any opinions or or feedback in terms of what you think um textual uh or really from a diagrammatic perspective of what dev set cuts could mean uh from that perspective so where does

application security kind of uh feature in this current scenario uh thankfully for everybody today security is no longer just an end of the chain activity it's almost become like a piecemeal kind of engagement broken down into smaller chuck you know the devops and the devsecops movement has kind of picked up and businesses across the plain uh want to move faster they want to go to market faster and dev setups therefore amongst other things on the screen that you see here has kind of led to increase in tooling test management build management deploy post deploys etcetera and and security too thankfully has moved from a traditional um a gatekeeper function um to a point today where it kind of

beds different pods if you will um but all this all but all of this obviously means that we're using more tools than what we were using when we were a silo um this also means that um integrations between tools uh from one unit to another also is quite significant from that perspective kind of its numbers and these numbered intention uh from reports of 24 2016 or 2017 because i want to just kind of you know i want to make us appreciate the kind of scale that we're kind of talking about just one second yeah in terms of just the numbers here right so you're kind of seeing a mature devs a cop shops generating close to 8 000

vulnerabilities but which is great in terms of finding issues how these issues have been remediated you're kind of looking at close to 30 days or 35 days at an average for somebody to patch an application but i'm going to keep this just want to throw this number out there because it's just something that for security professionals and for product engineering professionals just kind of ponder about from a remediation perspective as much as we've done uh from finding issues in the in right so and and this is where i want to kind of kind of create a segue to today's stock is in terms of understanding what is it that stops product the fixed bugs as much as we're able to

find bugs early right so so that's really where you're kind of going um the development and deployment architecture uh in the recent times has also kind of increased um significantly right today you have company to us than what you have probably a decade ago uh right and and and more competent in context with more tools and more tools means more scanning and obviously all of this um is going to lead uh to an information overload right and and and that's really one of the biggest aspects of what uh application security is dealing today is in terms of all of these great tools times the number of components times the number of iterations uh and and that's going to lead to a

so let's see how that data kind of plays into um application security engineering as we see today obviously with information secure information overload there are some challenges and we look at some of them in more details but but just as a pointer some serious constraints in terms of availability in terms of time spent and triaging issues sorting issues um and we'll see this later where tools have some major inconsistencies in terms of the way they represent data in terms of how they kind of lay that information and finally you have a you have a lot of contains in terms of just managing those kind of vulnerabilities bandwidth um you know it still remains to be the single largest

constraint with absolute teams in the current scenario style they're hiring more developers they're hiring more agile project managers more operational folks more crowd engineers but almost nobody from a security perspective i mean from both sides right uh in terms of do they know what they need off a security engineer or do they know what they need of an application security an infrastructure network security engineer right so so um and something that you've kind of noticed in a lot of forums like these or when there used to be a time when you used to do meetups and things like that you always have um you always have um you know headhunters saying that we've got openings for security

engineers um but then realize in terms of what they wanted those engineers to do in the first place as far as as far as the profile was concerned right um which is i mean uh and and and they get to work on some of these initiatives that that you've talked about uh but there's another side's a story that kind of goes mostly unnoticed our typical security engineers time is spent in position and i like to take the example of this fictitious character called jason and let's say it's jason the tester for example he works as a lead security engineer in let's say a digital uh let's say in a financial technology company right it's a fintech firm

um conversation here uh it comes in into this work and let's kind of understand what jason's day um day in the life of jesus from the case and security perspective uh really the role and probability would include them doing security assessments which is great a pen tester or a security engineer supposed to do security assistance complaints there help um developers in terms of remediation great no problem there as well and then if you're a financial technology company you probably need to help your ciso or your head of compliance with compliance audits okay still and then you've got security reports uh you've got both tools which is still okay perform threat modeling automate those tools in ci cd validates

fixes with that you've helped them the validate fixes once but then they might not work you come back again you do traditional you review those vulnerabilities respond to customer security almost no time left for um security research which in my opinion is a very very integral part of what an african security engineer is supposed to do which is security research because you're living in a very very dynamic era as far as have success is concerned and obviously you need to kind of be able to update yourself from that perspective now on top of uh if there's somebody's going to come and say that they need to really continuously jason is not happy and this is kind of

interesting that today is the 13th of friday and and jason got a because um and and there's there's there's really a difference in terms of what an applicant and what he or she is currently doing in terms of available bandwidth among these issues presentation on just one thing which is that of managing vulnerabilities right and this really brings us to the core of today's topic the clutter in appsec which is a clutter caused by that of managing vulnerabilities and that an application security program is only as efficient um as its vulnerability management process right having all these amazing tools and platforms teams and processes if software vulnerabilities cannot be efficiently generated if they can't is shared remediated and probably

finally sometimes not even be able to prevent them from regressing back upstream the single most significant reasons that prevent um product and security teams uh from being able to tap between the ideal and the practical of their stick ups um and and and for them to be able to kind of increase what we like to call this product through let's let's kind of dig um into this um okay so i see that there is lag um chris um i'm just let me know if you guys if everybody else is uh feeling this like i said i can kind of swap out my uh video if it's still if it's still continuing to be so i'm

just going to give it a couple of more minutes to see if that's better um anyway so let's dig a little actually do this it's kind of been it's kind of essential that um that we understand vulnerability right so before we kind of talk about how do you manage vulnerabilities and things like that it's very essential that you kind of talk about how do you what

no i know this is the b sides chapter conference and there are a lot of accomplished security engineers here but trust me um the number of times either in an interview or in a security report that i've seen people get really confused with these three terminologies of cwe cve and cvss there's often an analogy that i'd compare this problem with i think the cbe the cwe and the cdss the brothers famous one if you know what i mean right um no one really gets the others movies right the first time around and they always kind of get them confused in terms of who they are or what they do and for people who are interested the one the

far left is uh daniel baldwin um thank you i mean check him out in case if you're interested if anybody can um kind of recollect what movies he's been anyways moving uh so at the risk of uh some of the security experts really grinding your teeth here a quick 101 and trust me this is important because when we talk about managing vulnerability scale data point in time um so um um at the risk of some of you really kind of grinding your teeth i'm just going to quickly go over this cwe uh um mission is essentially a kind of system for categorizing software vulnerabilities right so it has over 60 odd categories um including cases for

buffer overflow sql index race conditions and so on and so forth and it's essentially a bucket that you would kind of put a particular um vulnerability within so in case is that in terms of particular vulnerability you take that and you kind of assign a cwe value to it the cvss is a common vulnerability scoring system and this is essentially um a free and open system an open standard really for kind of assessing what the severity of a vulnerability is it kind of attempts to assign severity scores for every kind of vulnerability that you find and this is where you would probably have people who've worked in security or not you can know like this as

calculator and what the impact of that vulnerability is to that particular scope by getting a value of what that severity is now that's the same now the c the cbe is the common vulnerabilities and ex of standardized names for vulnerabilities so which means that every time there's a new vulnerability that comes in you'll talk pve database and and cv essentially aims to kind of uh uh um uh of of publicly listed exploits the publicly listed vulnerability so that's really the cve so you've got cwe which is a categorization you've got cdn assessing civility or vulnerabilities and then you've got the cde which is essentially a standardization name now with this in mind it's going to move

um forward into in terms of tools now tools are great they make our lives easy they're able to kind of find flaws in ways and approach that manual pentesters might not and they also kind of generate the data required to find these kind of issues but tools have inconsistencies still alike um and they have what kind of intense they kind of they have inconsistent inconsistencies in the way that they produce this data and means and they have different verb edges for those names which means that they would probably call a blind sql injection a time-based equilibrium therefore confusing the audience in some ways and most uh tools kind of rely only on um for example right so some tools might

give you the vulnerability uh information with the cwe and some of them might not so that kind of creates some other kind of kind of manage results coming from two tools uh even today uh very popular open source tools like bandit and draken there's no cwe so how do you kind of backtrack vulnerabilities from these tools and kind of tie them back to a normal a common nomination so that some of these tools don't give any taxonomy so you have no idea is these twos talk about let's say not against the sand star 25 or are you kind of benchmarking them against noah so what are you kind of baselining these wonders in so uh some of them don't really give

you those taxonomies right and then you always have the problem of false positives uh which we'll also talk about in a little so tools by themselves have a lot of inconsistencies and that kind of adds one step further to the existing clothing that we kind of talk let's say you're running this fast sc and a desk for example and let's say you find equal injection there's a good possibility that each of these three tools are going to cause equilibrium for example a zap or a burp might call it blind sql injection a fast might call it a time equal injection or something else and believe me when i say this because we've literally tried doing this for almost a year then we

were kind of moving hard into right every combination in the book we try to map these vulnerabilities based on string matches we try doing this based on leveraging distances really still has some level of inconsistencies and and what is kind of made worse is when you kind of throw in a vulnerability scanner in the mix which does not give a cve right which by no means like we've seen in the examples earlier gives you any information on what the actual vulnerabilities are right perspective just to show you what some of these inconsistencies are are is is a snapshot of a vulnerability of an intentional vulnerable application so no upper the top is from kinetics and one at the bottom is from check

marks we're essentially trying to compare a dash and a flash vulnerability both of them are complicated about the same vulnerability which is that of cwe 179 um whereas acunetix calls it as vulnerable javascripty ability by check marks is called as reflected xss completely two different names for the same vulnerability so apps two dash platforms same vulnerability cwe 693 appscan calls it insecure cross lifting if you complete that on the screen it's the one with the box in there and then um burp calls it protection mechanism failure which has nothing to do with cross scripting again consistently let's take two fast platforms both of them are static platforms find set bugs better code cwe89 which is the same vulnerability

sql injection but different names now sometimes you don't even get a cwe you know this is again cw 89 which zap gives you um for sql but nothing from its uh fast counterparts is also an open source uh os project called the node.js another os i'm not really sure but it is an open source project called the node.js scan and it does not even see when this is run in scale such tools especially as part of nike bills or daily bills there's just so much issues that are generated that when you what you see on here obviously is a very very simple pipeline involving a couple of tools that you use run to run against our own ctf application

but the idea really here is for us to have look at the results not just in terms of the type of issues but also in terms of a format in which these in which the results are kind of generated uh for you to be able to kind of appreciate the issue what issue at hand here right some of the other clutter contributions false policies the art of false positives everybody talks about false positives with two uh with tools and one of the things that i want to lay out very i mean actually it's a personal opinion of mine i'm sure a lot of y'all share this is if people ask someone if there is a tool out there

that can eliminate false positives my best abilities my best knowledge there isn't a tool that can eliminate false positive because you've still not got to a point there where from an application security perspective you could probably eliminate false restricted tools security platform but for an application security platform completely eliminating false positives to advocate zero is not possible uh but what manage false positives from those perspective right uh for those who are interested in those who kind of used tools like burp um you will know that birth has something called a scanner confidence score and and and that's a score that burp uses to kind of give you a asset where particular vulnerability is actually one

and and that's that's the extent to which burp goes and a lot of other tools don't even give you that right next part is managing false power big contributor from a club the other one is in terms of prioritization or security a lot of us rely on the severity scores given by the tools themselves but the fact of the matter is the tools don't understand your application's context the same tool can be run against the banking application and of a retail application it's going to look at the same way the prioritization of the stability of the vulnerability is oftentimes a product of the score text in which that vulnerability is found so it's very very essential that aztec engineers

and product team kind of realized that severity kind of means that that bad year between uh application teams finding issues and development teams fixing fixing those issues in the same speed and more so incorrect prioritization incorrect severity scores kind of create a lot of turmoil in terms of the way security bugs are either from remediation um a lot of us depend on the remediations given by tools now to a certain extent um i i until recently i thought that like uh check marks uh buying off code bash for example was the right step in this direction because you kind of had kind of give out a better remediation strategy which is very localized to that

particular issue that's found but oftentimes patients don't happen or they don't happen quite well or they don't uh there's no source for finding radiation so it's it's very important that given remediation strategies that are alternate to the primary one as well because oftentimes you'll see that the remediation that's given is either insufficient uh so in those cases you probably have the developers trying to go fix issues that don't work quite often and then you kind of come back and say hey this is my work right how do we declutter now we talked about cut up contributors but now we're gonna let's kind of see what's really the solution for this one this is really two things right i'd like

to call something is find early fix early and this is more philosophical than just a technology problem because it's very that any youtube any new process any new person that we get into cart engineering today kind of picks a fix in one of these two buckets is that introduction of that new element whether it's the people process or technology is it going to help security engineers find bugs early is it going to help application developers fix that bug early and it's very important that we kind of maintain the balance between find early and clearly able to kind of have some method in the madness to be able to kind of fix these clutters from that we kind of just

i'm just going to throw three potential solutions here and the first one is that about i want to touch upon is vulnerability correlation is probably what a lot of organizations are using today now going back to our previous discussion let's say you have a string of tools and ci that runs on them means that you're going to get results across all of these tools now correlation platforms essentially help security engineers triage having an application security engineers to actually say okay this results from that description here's something else from fast and both of them came and before actually go ahead and protect the report the correlation platforms help you do that so essentially uh like a nice

cancellation device you remember the time uh from let's say toronto vancouver for example and you can look you look at a look at the if you open up expedia and see how many flights are there same time and you're like interesting seven flights that take off at the same time and land at the same time but you can go dig a little deeper you'll actually see that there's just but code shared across five airlines uh but the actual uh flights are just two so it's very similar from from a vulnerability perspective you probably your ability is coming from tools but essentially if you kind of correlate then you'll probably zero down on 25 or 30 different unique

vulnerabilities especially if you're running one of these tools in scale so correlation platforms really do that they help you deduplicate vulnerabilities um the view uh um which vulnerabilities uh they help you declutter false positives to a certain extent as well so correlation platforms really kind of give you that uh charter to a certain degree um the other thing is security as code um i'm going to spend a couple of minutes here because this is probably the most controversial slide in all of my presentations um now we over the last five over the last decade if you will um security engineers have always gone to developers and say hey we're doing this they think quite more proactive

secure code you need to be able to remediate fixes faster you need to be able to write secure code being this constant bashing of developers need to understand security and it's a time that we said sturdy needs to get code and i say security needs to get code too for the following reasons let's look at the practical reasons we're talking about application security we're talking about need to scale application security and in my definition anything that's automation client word for automation is code so code times n is automation so how are we going to how are we going to be able to automate abstract domain code at its granular level um you're going to be able to automate

code you're going to be able to it's going not be too ability to look under the hood of open source scanners like zap or or whatever it is and be able to kind of automate them with your icd platform yes there are plugins available but if you really understand code so that's another benefit of security engineers being able to understand code from at the last optical level um we've all the siloed approach of developers and security they've always been nemesis from their perspective it's kind of soft it's not as bad as it was earlier but you still can really kind of working as much as you want it but because one of the one of one of the causes could be the

fact that development communities start their language if you will is code they always are talking about what's the natives and the greatest happening in terms of programming languages technology facts and so on and so forth so if engineers are also to talk same language it's almost that you get into that tribe of developers uh and then and therefore probably the empathy uh a mutual entity between security and developers because once a developer understands that the security person understands the way the conversation goes is going to be much different and you're going to get a lot more buying from product engineering than earlier right security engineers really start picking up code because uh that's the way the industry is going as well um

i can speak for myself close to 27 security engineers and all of them are at some point in time and for others it's been easier for us to teach saturday to a developer that teach a lot of one learning and learning more um so that's just us uh but just kind of putting that out there i know a lot of them have had alternate opinions to this maybe we could discuss that as well offline but that's a very very important aspect of cloud with code you can do give me some very interesting things like be able to automate exploit scripts which means that business logic flaws are no longer dependent on visual but it's actually dependent on a

script that runs on behalf so that's a great thing uh build a potential because you have functional regression suites that create uh regression scripts for every iteration ensuring that the same one the same functional vulnerability does not really come across in a new version very similar analogy possible with security as well uh and focal permeation and if you understand code you'll be able to kind of really come up with alternate remediation strategies once the development team gives you a push back to the primary remediation this thing doesn't work okay what can be the alternative that's an easier conversation to have um if there's that whole adoption of code and the last one is that of really

integrating security and engineering from both uh uh perspective uh and from um and from a philosophical perspective again because we've all talked about integrating security and engineering uh we work in total but at the ground party teams are still working in pdfs right everybody's still working in periods irrespective of what you could have there's still a pdf on the table that some team in the organization who says i'll only agree there's a pdf but engineering teams are still working in tickets right so which means that there's a need for product engineering teams and security engineering teams to really what i like to call transform defects of transforms vulnerabilities as defects because you kind of have these two

different views of the world the security teams look at vulnerabilities whereas the engineering teams look at defects and both of them have to be the same so the problem is you're kind of talking in two different languages so integration today is either a spreadsheet or a pdf and both of them don't really kind of work uh frame of things because when we talk about integration and when we're actually called in to create an intent in different projects the biggest issue that we see is that almost six places vulnerability management per se is done using worksheet is probably the most common business intelligence tool that you're going to see there are a vulnerability correlation that you

can see there or we encounter the dreaded pdf uh shifter and pdf compliant audio triple sided pdf which is very very difficult to track and manage in terms of workflows so it's important technology uh whether they kind of onboarding new tools from a security engineering perspective all when you're onboarding new platforms from a from product engineering perspectives it see if these can integrated and most of the tools today thankfully have apis um that cannot help you integrate with these tools but it's very very critical that all components within product engineering perspective are able to indeed kind of collaborate within themselves from a seamless perspective um as well uh some interesting integration uh points here cicd a lot of them are

doing this tools integrated ci cds bug trackers with tools and therefore all of them talk to each other which is great uh tools talk to bug trackers as well which means most of them today any tool worth it sold today has a two-way sync with platforms like jira which means that you can just kind of just push vulnerabilities directly at zero tickets and people just you know the developers just pick on them and start working on it um in some other companies we've seen bug trackers actually being integrated as part of the slack channel which means that every time there's a bug created there's a slack channel ticket that's created directly to that individual so it's much more faster

and you kind of seen um a lot of more dashboards right so you kind of see a vulnerability correlation platform integrate with the bug tracker and all of both of these together would probably integrate with third party uh with the third dashboard let's say for senior management uh or executive dashboard or something like that so a lot of possibilities for my integration okay so in summary i'd like to leave you with this thought uh from a core vulnerability management standpoint i think should be beyond just traffic light indicators right of high medium or low um because because the sheer number of issues that are right so when there's and in my in my opinion there's actual value

in studying what i'd like to call the metadata vulnerabilities right these are important but what if there was an opportunity for us to to kind of look beyond those traffic light indicators right for example with information that you might have some vulnerabilities what kind of patterns can we kind of look for what if we see that these correlation platforms from telling you that there are more issues in in design probabilities kind of have a root cause uh to design so that your modeling efforts as opposed to investing in in dynamic analysis platforms let's say that you're finding more issues from a fast perspective you need to probably address that with developers as opposed to you know uh building on um you know

third-party libraries for example so i think it's important i always say this the last decade of 20 uh 2010 to 2020 is is that um time slot in product engineering that's all i'd like to say right because we now enjoy a position in prime time private engineering where people listen to abstech there's a lot of importance given drops like right so and and it's left to us in terms of what we want to do in the next couple of years where are we going to take the discussion of are we going to address the cultural challenges of implementing app second scale are we going to talk about more tools or are we going to talk about

management for example or is going to be training so so with that in in in fact or um being part of today's talk here um is there any questions um take them i'm just gonna quickly look into the comments here if there is any questions thank you thank you for those who found um the session useful okay i'm just gonna kind of um probably stop shipping and then wait for any questions but any of you all might have

thank you again what's your advice on legacy garbage like old school sequesters that folks know they need to fix the door in the hot spot uh i don't know because uh what was the last time we worked on a c clusters i no we we worked on c plus plus uh but it's always some kind of refined form of cpu of this right so it's not um um but but yeah i think um practically speaking i think it's it all goes back to uh conceptual understanding of security because i always say that product engineering and application security are not too far apart right if someone knows the art of building applications the art of breaking applications is not

too far if you really kind of look at them at the same point in time when you kind of give some space then they kind of diverge into much different ways right so um yeah maybe that's that's one way to do that thank you so much and have a great rest of the conference as well