How to Start and Mature an AppSec Program

all right hi everyone um okay it's visible horrible okay all right well uh my name is niveda murthy you can call me nivi nivi that's the short form of it all about me i'm an associate principal consultant at synopsis um i've been in this industry for 13 years now so it's not like i've just come out of you know college and talking about the whole abstract program i have some experience talking to different organizations trying to set up their abstract program working through it and now i've been focused on devsicops for the past few years at least as part of my job i have to script a lot which is my favorite part and to create

out-of-box solutions um i've done a couple of speaking engagements over the past few years and which includes quite a few b-sides um in other locations i've also been i'm actually one of the organizers for boston security meetup so whenever you're in boston please to come and check out our meetup uh you know third thursday of the month uh we try to get some interesting talks we are now hybrid as well so if you are here and you want to give a presentation in boston feel free to do so um [Music] and finally your proud dog mom this is the first time my dog has come over here has been on a plane so he's enjoying it

out here um but let's dive into the talk now

so what do we have on our agenda today um we first start with the inventory itself application inventory to be precise if you are to start an application security program you need to know what you are protecting you can't just go in blind and start something out there another important aspect for any program is metrics um you probably heard this several times security is a cost center an organization does not make any money through security uh it can intangibly prevent losses so how does security show return of investment while waiting for that once in a while security attack to happen next is technology uh the tools that help us get this done the tool that actually helps run an

appsec program and then there is governance policies and standards that give teeth to this program and that can establish that this has leadership approval and then at last we talk about quick wins we are in 2022 and just about everyone here has short attention span including i mean i i can say i have a short attention span um it is the age of instant gratification so to convince your sponsors or rather your leadership your program needs some quick wins so what is an application inventory in simple terms it's a data set of all the applications that the organization uses now if you need to start your appsec program you need to know what you are protecting

what applications does your organization develop what applications do you buy uh from others where are they hosted where what does it handle out there what data does it handle an application inventory helps you determine easily in case of security notifications um you know that in case of like who could be potentially impacted and application inventory helps tie back the responsibility of security vulnerabilities to the right team and then the application leader itself an inventory health security team plan and organize appsec initiatives uh if you had to find all the applications you know being hosted on a server that is vulnerable you can use the inventory for the same um sorry [Music] um and then you can use the same thing

to you know figure out who to be contacted asap you didn't have to go through dig through and check out like do you know who owns this application do you know who owns the server do you know who owns this particular section or network you know run a network scan and find out the ip address associated who raised the ticket associated to this ip address you don't have to do that and inventory gives you direct access to all of that contact information if you're a large organization an application inventory also helps segregate the risk by division or the verticals that you have in your organization so what do you ex what data should you expect in an application inventory um

let's start with the basics business criticality externally facing data confidentiality and technical and business contacts rtos dr procedures etc and then number of users whether it's developed internally and or externally like do you have you know we have it procured from someone else or is this something developed internally by your own developers with this information one can determine how critical the application is to the organization example an organization would rather run pentest on externally facing applications only since you know printers are expensive and externally facing applications are at a higher risk compared to your internal ones um exercise vulnerabilities have a higher impact on externally facing applications versus internal applications application metadata can be used to determine the

actual severity of an application versus like instead of the one determined by a scanning tool so you can start the inventory with the basic level of metadata out here then you can also you know enable it to trigger processes uh that are part of an app onboarding process uh a mature application inventory uh and process can and database can also notify when an application is due for audits uh such as bcp and dr checks uh and even security scans you can actually set this up quite a few organizations have already this you know in their inventory out there um in the latest trends uh organizations are also trying to include a bomb or bill of materials uh just to show off

hands how many of you have heard of this term bom a bomb or bill of materials so you know what it is a little bit you could say we've been hearing about this through lock4j attacks in case if you have heard um but essentially you can also include this as part of your inventory so you have a much bet accurate view of what the organization actually controls or has in its you know network out there what is it storing out there um does that mean this is a one-time activity um once all the data is captured uh should it be left out like that like any other data it becomes stale over time and people come and go applications also

change they change state they change activities some applications just get decommissioned and it takes time how many of you have heard this term oh the application is actually getting decommissioned and you've heard this thing happening for five years at least i've heard it for five years um and as such there needs to be a process to audit all of this information and ensure the data in the inventory is accurate so that your processes don't get stopped out there um a typical way to do this is to trigger a review every one or two years and some organizations have done it based on criticality so if it's a critical application they review it every year if it's

maybe a medium application then every three years and so on so forth so what tools would you use to set this up if you are at the beginner level i would just say go with a simple excel sheet honestly the best way to start it out is an excerpt because it's under your control you don't have to pay for it it's already installed and you have it ready if you go and try and buy a tool you have to go through procurement you have to grow through legal you have to go through leadership to get the buy-in for that so start off with an excel if you have to um and then post it on something like

you know on a storage service or like a sharepoint excel maybe you know so others can contribute to it make it available to others so that you can keep a track um and it's just not one person maintaining that excel you have quite a few teams maintaining that excel at the same time while it's tedious and manual work you can at least start it off and more importantly you have a good view of your application portfolio you have something in hand ready in terms of actual tools there is servicenow compass remedy and fresh service that i bought from my own interactions with different organizations but there are quite a few tools in the market choose wisely what

works for you so after all of what i've said here let's have an opinion poll uh we have quite a few people out here do you think a product company with just a few products in their portfolio needs an inventory let's say yes okay yes quite a few good hands okay can one of you just let me know why and what do you consider few products yeah go ahead right but i have an example of a company which had just two products does this company need an inventory yes okay why

yeah good that's great so that's why we need an inventory because even if it's one product even if it's one product it's not exactly one small product is end-to-end services being provided through that single product out there and when it comes to development you are going through several end points to finally deliver that product by deliver a functionality of that product so yes you still need an inventory all right next topic what is metric um actually if you wanted to try it out if you run a google search right now at least this work for me for the word metric just the word metric the first result that actually pops up is a canadian rock band it is not the

meaning it's not the definition of metric it is actually a canadian rock band which was there but to most serious conversations out here um it is essentially a system of measurement in terms of business it's an intangible way of showing results um what would you consider a good security metric uh let's discuss the obvious ones anything that talks about vulnerability summaries it's a metric note that you should also be ready with a conclusion or implications of a metric examples of metrics include total number of vulnerabilities by severity or new vulnerabilities found every month by severity um the same numbers can be filtered down to an organization level or to a division level or subdivision level uh then next

date your next point here should be what is the pain point that we are trying to uh highlight here with this metric okay uh or what successes are we trying to showcase out here uh with this with vulnerabilities by severity uh you are trying to point out the risk level of an organization but then you could also be then um sorry if a majority of the vulnerabilities are of medium or medium severity then does that mean the organization is doing slightly better than let's say other organizations out there in the uh you know in your industry or by basically not having so many higher critical vulnerabilities or could it be that because of your team's bandwidth

you're only focusing on critical and high vulnerabilities you're actually just triaging critical and high vulnerabilities you're not even looking at medium vulnerabilities that way you have actually reduced the actual uh true positives i mean these are true positive numbers you never looked at medium and there could be the case that those medium numbers that you see are actually false positives out there that could also be the case because these numbers are coming from scanning tools and we all know when you run automated scanning tools they have false positives so it's not necessary that this number accurately reflects unless and until you go into the details on how this one how are the metrics captured another example of a metric and trying

to explain the context behind it is think about scan times okay when we talk about scan times we talk about average kind times out there as a metric but would it be fair to club scan times for let's say all apps under one average uh when you say think about apps that are like 500k llc when i say loc it's line of code um versus microservices which are really small when you run any sort of security scan on it the microservice will obviously be done within few minutes not even few minutes i would say under one a minute at times but with a fine like a big app like this one which i was referring to 500k it may

take some time try to average out something which takes you know 10 to 15 minutes versus one minute what is the average that you'll get are you are we saying there that the average is actually five to six minutes to run a scan it's not fair to clog these applications together um it doesn't sometimes it could probably take a little bit higher and then trying to average them out doesn't help out here so again presenting the data correctly is actually important and you know this actually helps understand the context behind what you're trying to showcase out here um as you can figure out with examples uh we discussed contextualization is very important far too many achievements are

actually you know lost in translation when you showcase uh this in with the help of metrics sometimes presenting the information correctly is as important as getting the information as well using a pie chart instead of a bar diagram or to using the right colors while presenting you know the right information or presenting good or bad outcomes is important these are all important tools that you you need to learn i mean you need to pick up while presenting metrics to leadership uh deciding when an overall summary is better over granular data um or ultimately metrics carry your story they are the story of how you have been working to secure your organization um i can share a lot more but i've spoken in

depth about this topic at besides nova the vulnerability tell you how to dig in so you can just google it out if you have to uh but at the same time our next uh talk in this room actually is being given on metrics by christine she's over here so if you still want to learn about metrics stay over here another critical factor for metrics is accuracy efforts should be made to get the most accurate information while presenting metrics accuracy can only be possible if you collect the data regularly and in the correct manner as well most teams don't face these challenges in collecting data it is only how frequently you can get that spare team spalter out there

if it was collected manually you don't want to do this daily but if it was being collected automatically you can obviously you know pull the data regularly like almost on an hourly basis if required um next thing is distribution itself um what is the benefit of sending a weekly report when you cannot show your success on a weekly basis but the same success can be shown on a monthly basis so think about that do does your vulnerability count actually change on a weekly basis i don't think so unless you are you know sending one code to production on a you know weekly basis on an average teams have a 15 day life cycle where they have code changes

generally going either every 15 days or every month over here it would make much more sense to distribute your reports monthly on a monthly basis or maybe quarterly basis to actually show that you have made improvement well compared to the last report that we had that's how you show improvement over here so how do metrics deliver we spoke a lot about metrics and how to show it but who's actually watching this your leadership the ones the same leadership that who asks you why should they invest in your security program um why should they at least keep you giving the same budget uh if not more okay so that's how metrics help you metrics actually shows them that

security team is not a liability um the the the work the team is doing is actually helping the secure the organization it helps shows the intangible ways the organization is saving on security incidents by actually preventing them it also highlights the progress made by security teams to improve the overall risk the organization currently has and if you push your developers to undergo secure code training on a regular basis then you should show the results over here in your metrics itself that this investment helped reduce the number of vulnerabilities over time um in in the organization itself next one is technology one of the main pillars for setting up an appsec program is technology itself this is where most of your money goes um

as a greenfield client and at least from a consulting perspective when you say greenfield it's like someone who does not have any abstech program or has maybe a little bit of it they're not exactly they don't have a full-blown abstract program over there uh in my opinion you just start you should just start with open source tools okay there are several open source tools out there um in the market in in general people offer that uh so that you can at least understand how things work that is important before you go and jump in and start buying tools from the market you need to know what are you buying what decision should you make when when

running a poc what works for you what doesn't work for you um like if you haven't heard of os by now uh that is open web application security program uh it's basically a non-profit that has helped a lot in appsec sector uh they have provided tools trainings and the whole assessment methodology over there um and it's one of the driving forces at app say the logos that you actually see right now is just a sample of all the tools in the market it is not the whole thing it's not the whole picture out there um maybe and then just start with a basic scan run a sas scan when i say uh it will a sas scan or a static application

security testing tool it helps identify vulnerabilities in your application in the code itself that you have developed uh there are some decent open source tools in the market for example uh spotbox over there is very good for java then you have breakmen for ruby on rails uh there are some then secure go for golang etc uh get a sense on how these tools work and you know how the results come out get feedback from your developers in the previous session that we had over here they said he hates you i mean they hate security team how many times have you heard like security team is being a big pain out here try and collaborate with them try and

get feedback with them on how these tools are affecting them and what would they like try and get gain their confidence in working with your tools out there um you could also run a secret scanner does anyone here know what is a secret scanner the name itself will give you an indication but does anyone here know what's a secret scanner you want to see

right so just search for you know passwords how many times have you heard development teams actually storing passwords in the code itself i have literally seen quite a few times here in production passwords tokens ssn all of these i mean why is ssn over there i have no idea but sometimes people have stored this information in their code repositories in fact one of the uh clients from what i heard their source code was public facing and the saying passwords for all of their uh servers but it wasn't that code out there so somebody actually found it out it was a major issue ultimately you don't want that to happen even with external you don't want this

to happen so instead of talking to people about trying to teach them about xss sql injection let's go with basics what is a password everyone knows what's a password is it a good habit to store passwords in code no it's not so just run a secret scan and figure it out and just try and close that out um then start introducing like just get into this process of identifying a vulnerability understanding of vulnerability and then fixing that one bitty let your developers get used to certain security initiatives you don't want to go big bank theory on them you just want to go one step at a time they're all babies we need to treat them that way

then introduce more tools that run different kinds of assessments uh example sca or dast now i'm going to say a lot of acronyms i'm going to explain you what are those acronyms as well is rasp and all let's go with sca how many of you have heard of log4j everyone just about everyone okay so from log4jd you heard the term called sca if not it's called source composition analysis log4j came into picture because just about every other application has that as their logging capability it's a free open source library which people can include to start logging whatever the application is doing out there and an sca tool or a source composition analysis tool actually identifies if

your application has that component or not it is basically that its job to find it out then the next job is to after finding that component it will tell you what are the publicly known vulnerabilities in there and guess what as for the latest report that you know checks out you know how people develop eighty percent of anyone's code any application at least eighty percent of it is comprised of open source components so you're actually introducing risk from outside and not by your own developers so running an sca scan is actually important for your organization um we then have das scans or dynamic application security testing that's this basically running this a scan on an endpoint give it a url run

the scan give it credentials so you can run authenticated scans out there some things cannot be found through sas or sc it can only be found through dash cams so try running that and then i asked which is a a little bit more of an intelligent way of running dash cams that's what i would say it's called interactive application security testing so when you're actually running a test or when you're running a regression test out there by a human or by an automated testing tool i is will actually check it and see if this response is vulnerable or not if this is actually going to lead to any vulnerabilities out there so and the other one is ras which works on

the server side similar thing but it has a capability to actually block um any requests coming on production if it thinks it's going to be malicious or not note that is and rasp they don't have any open source tools out there and the market in general hasn't accepted it yet so because it's very difficult to understand difficulty there's a lot of initial investment to get things running to make use of these tools and which is a hindrance for a lot of people when you have to start working with infrastructure teams to get things installed you are going to wait forever at least for me um then after you start running these tools look at the issues look at

what's your you know just uh i would say what's your confidence level in the tools and how how happy have you been in just using these tools sometimes you just hate how to use it maybe there's a better tool in the market which you know makes it simpler it lets you connect everything it works pretty well like you just have to do click click click and it's done or it's like you know give a few parameters at the command line and it's done you can run it automatically if you want to so think in those factors if you had to scale up if you had to improve the quality of the issues look at tools in the market which

offer similar capabilities and then to run a poc and then choose one um the you know the other aspect of running assessments is that you don't need to start scanning you know all of your applications choose the applications wisely you can go in phases you start with the critical ones which are important you need to make sure they're secure out there and then you go down to uh you know hi high severity applications or rather high criticality applications medium criticality and so on so forth just make sure you're settled with one tool before moving on to the next section out here next is governance somewhere in the beginning i wrote without rules there would be an anarchy

i'm sure some people may disagree out here i do not want to go into a whole philosophical discussion about this um but coming back to this topic in order for a security program to be effective uh there needs to be some policies and standards set to make sure that teams just don't go and you know ballistic in terms of developing and throwing things in development how many of you have seen cases where developers actually have access to push things to production have you seen in these cases yes i've seen few hands here i personally saw this happen like just recently and i'm still shocked this happens yet people give developers access to more code to

production um but you need to have some rules you cannot just allow anyone any uh you know to go and things deploy things in production have you heard of segregation of duties this is a term that's coming up you one person develops the other person tests your third person deploys it in production and a fourth person gives approval for the same after reviewing everything is being done correctly um similar way policies need to be established what what is expected in terms of security okay policies also need to be scoped correctly and aligned to the laws and regulations of the land um you know some companies have to comply with let's say pci standards some companies have to comply with phi

standards policies help to make sure that they're complying to the same they then typical examples of a policy could include all internet facing applications have to go through a dash can or a pen test at least and they have to be done every one like every year once a year at least or all critical vulnerabilities have to be closed out in let's say 60 days some teams have actually seven days but i'm just giving you you know a nice standard out there uh 60 days of discovery and as an organization matures you can actually go to you know reduce the same timeline like i said seven days some things actually can possibly do that if

they find a critical vulnerability they just have seven days to fix that vulnerabilities because they have a process in place to go fix it deploy it quickly test it out quickly and move it to production review your policies and standards every two or three years you cannot keep the same policy that was there for 10 years back we need changes are happening changes happen every year you need to keep up with times the policies and standards also need to keep up the times and then metrics can be created to check compliance to this policy you can also set up some automated notifications that tell you know teams that you are actually going to violate this policy in let's say x days or when

they have actually violated this policy say that you violated this policy this is the reason you need to go and check it out here okay this is going to be interesting how long have you lasted how long have you lasted before you checked your phone i'm pretty sure some i did during the calls i mean during the sessions itself but how long have you lasted two minutes four minutes wow anyone more than 10 minutes wow okay there's someone over there wow but no one okay all right all right after literally being thrown you know into acquiring a digital life we all of us and like i said at least i have short attention span i am

at the mercy of my mobile phone see all meetings all zoom meetings i have definitely seen one it doesn't matter it's like there's nothing important coming in there's nothing major coming in there's no major emergency i'm not part of the sock team i'm not part of the incident response team where i should be getting alerts out there if somebody's going to attack me no my biggest notification would probably be an acute dog post coming up that's it um but yes we have short attention span so unless it is something really interesting and you know that i'm seeing and i'm seeing good results coming out i won't stick to that topic whatever is happening out in front of me and

probably jump on to something else that will you know give me instant gratification you could say same goes for leadership they would rather want to see results immediately or quickly at least before they invest in the next big thing especially when they're already investing in something into heavy like a security program security programs cost a lot for them and they need to see results for that and not just security it's just about any program but more for security because that's not an roi this it's not a product which will ultimately make millions for them it is something that's just going to take money it's not but and maybe make sure that you're fine with it

um so that's why our next topic comes in quick wins quick wins a quick win is an improvement that is visible immediate benefit and can be delivered quickly after the project begins the quick win does not have to be profound and you know or have a long-term impact to your organization but it needs to be something that stakeholders agree is a good thing um examples or quick wins include password protection just like we discussed exclude any passwords or any sensitive information from your code and this is pretty easy to implement it's very easy to find it's very easy to implement just have a possible wallet or a special place where people can just pick up their passwords and connect it

to your application um another example is making sure developers have to go through secure code training how many of you have gone through mandatory trainings as soon as you have joined an organization pretty much those who haven't gone through you're lucky seriously i've had to go through at least 90 trainings in one for one organization which was horrible but anyways coming back to the topic secure code training helps developers understand how to write code securely as simple as that it may not be an immediate result like people will write absolutely amazing code it won't stop them going to you know stack overflow and picking up copies of really bad code and including it in their application

but at least it stops them from you know straight away pasting anything is this good code this is not what i was taught maybe i have to validate input something like that um making them go through this training is a quick win why because within 90 days you can actually you know proudly say your your developers actually are trained in secure code and then you can also drive out some uh infrastructure changes insecure conflict not everything has to be tested like for 30 days and then move to production some of these changes just require a quick fix and can be moved to you know within the 90 day cycle if you had to so it's

possible to you know get some in uh insecure conflicts closed up and then while you may have been running scans manually at the beginning automated scans help a lot can anyone say what's the benefits of running an automated scan

yep go to other things short attention span i can go and you know switch on to another task yeah in short it actually helps efficiency drives efficiency makes your developers happy because you are not taking time you are there is no delay as such in terms of running these scans you can scale up pretty fast i can give you uh an example from where i work um it took us like within i was able to actually run scans for 700 applications with by setting up automated scans over there within a month 700 applications where i was able to onboard all of them by after setting up automated i'm part of desktop stream so i have to set up

these pipelines that run automated scans um okay any other improvement any other example of quick wins anyone do you know any example of a quick win that could help say security team is the best something on that all right quick wins are a precursor to long-term goals it's something okay anyways okay they are the initial set of results um that offer you hope and vision into your long-term results out there it also builds trust for the security team across the organization especially the leadership team to know that the security program is running well and it can confidently take on other long-term initiatives if they have to given the right support you need that support from

leadership out there and everyone likes success it shines attention on the team that has actually done it and everyone wants to be part of the success of that success out there so the developer community will actually want to be part of the initiatives that you are taking in because they've seen that you have these wins coming up and like you're getting all that attention out here well that is just a summary of what you can probably do to start an apsec program um there's a lot more things i've not covered up infrastructure anything out here you may have seen i've not talked about cloud security or anything because that's a whole big discussion out there but if you want to talk about

it we can always talk here uh this was just a basic intro but feel free to connect with me on linkedin and if there are any questions i will take it out now thank [Applause]

yes uh shift left that's a big uh keyword that's been running across and people sometimes think what is it all about and yes we have a lot of tools and you're right like how do we trust we don't trust them honestly i don't trust developers developers don't trust us um we but we do like to offer them tools saying use this try and find the basic of basic vulnerabilities out there the basic stuff do not include passwords or validate your inputs those tools actually try to find that out as well and they highlight it for them and it's actually way more easier to show them there itself while they're running the code then tell them three days later

that this is there in your code out there um what we do is there are very few tools in the market that could actually offer that and it it's supposed to be the right fit a lot of the plugins that we call it id plug-ins that actually run scans on your code level while you're typing it can only work on eclipse and visual studio and that was some versions they are not equipped to run on let's say spring suit or any of the you know ids that gen developers generally use out there so the other offer is as soon as they come with your code it's very easy to do that you type you commit your code people do

a lot of comments you can trigger scans over there it's called hooks that we call it and give you an offer where we check for specific vulnerabilities we don't check for everything we just check for specific vulnerabilities and like say this is what it but it's a slow introduction you cannot go straight away jump into it like if they have no exposure to any kind of security scan so if they are already used to finding vulnerabilities you're used to a tool after you you know push it to uh the master branch or whatever sorry the release branch then then you start introducing these tools if you have to okay any other questions

how would you introduce api security that's a good question um i would start with simple training how to write apis in the first place so the question is how do you introduce security for people who develop apis is did i cover that correctly um start with basics how to write an api what what are the good practices sometimes people don't know what are the standard the you know development practices while writing an api let's say you know make sure it limits to certain values records are limited it's not showing the whole database out there when somebody queries just simple standard practices then uh there are certain tools that can only do api scans not all of the tools

out there can do it use that look at ones which are focused on api scans and they're not like all the ballpark figure out there that's how you would probably introduce it any other questions going once going twice alright thank you everyone thank you