BSidesNcl 2021 Automating Application Security Ayush Priya

who's going to talk to us about automating application security so a regular speaker on the circuit for the last 12 months so if you want to take it away there you go sure thank you hey everyone so we'll be talking about automating application security i am ayush uh i work as a cloud security engineer at cloudell and uh i've spoken at a few other places uh gray hat beer con 2 global cybersex summit and i've been recently volunteering my time at the wasp branches will guess lead so what lies ahead right we'll talk a brief a little bit about what is application security just so that we are on the same page right uh we'll talk about what has changed

because application security has been there like since forever uh we'd want to look at how to utilize these changes for obviously automating abstract for our for the presentation we will take a look at a demonstration of how easy it is to actually like bring up something and like automate a sim process one by one as we go uh towards it then we'll take a few questions about uh that might pop up into your head like why change what already works uh what can and can we or can't be automate and uh and lastly we just have some concluding remarks right so let's begin with what is appsec uh this is the wikipedia definition which is uh fairly detailed so

application security is essentially any measure taken to improve the security of an application right by figuring out one of these processes so finding fixing or preventing security security vulnerabilities so finding fixing preventing all can be considered a process and because there could be different places where we could implement these processes so different stages are have different kinds of application security measures that we can take right so that's why application security could be in design development deployment upgrade maintenance and so many more stages in the sdlc or software development life cycle right so in short any preventive measure that we develop either by actually taking a framework that already exists let's say for a cis benchmark or something like

that right so creating a preventive measure from existing knowledge or because we faced an issue and then we found a solution for it we create a measure for it and these could be included in included in the different stages of sdlc again software development life cycle so uh it could be as early as the design stage or it could be as late as the deployment stage right where typically what pen testing the pen testing stuff happens and then you figure out issues and then it goes back to fixing those issues so it's obviously a crucial aspect of security but because without like application without code we don't really have a lot left to protect right and

secure and uh for me at least it's eventually the goal is to maintain the cia triad which we all learned at some point right in our infosec jobs and academics which is the confidentiality integrity and availability uh triangle right examples of abstract the traditional processes uh there are a bunch of them let's talk about static application security testing or sst for short static application is essentially looking for known patterns of vulnerable code snippets in your code base right so if you're familiar with python uh there's a function called eval right which if you pass a string which contains python code it will execute it for you right so us having this code could potentially lead

to let's say for example rc right uh because we're giving the end user supposedly or potentially the ability to put a string which contains python code and hence can be executed right so writing is simple script with the regex pattern that contains the keyword eval the brackets and a delimiter to just check for anything in between right would allow us to check all our files whether they have that keyword or not and we could we have a simple sst scanner at our hands that specifically scans for one particular scenario so not that useful but it's still i mean it works right for this particular scenario so that's ssd then we have dst which is dynamic

application secure testing and a bunch of other processes so what has changed as we said i mean application security has been there for a really long time obviously virtually as long as there has been applications to protect right so what has changed i mean since uh infosec is not the new world so we have decades mult various decades worth of experiences from different researchers engineers and security folks consolidated right so we now have consolidated these these huge databases nvd and other places where we have these uh bugs and issues classified categorized and stored right for us to consume what uh this mass consolidation of these issues allow us allows us to do is to classify these right in different forms

and different basis for classifications one the simplest one could be severity right so you could figure out if a bug typically is a high severity or a critical severity bug or it's a low or an informational security verb of a severity bug right so we another way to classify things is like class of bugs which is like classifying them together so we have self reflected and stored excesses all of these three follow under the category of excesses right so that's another form of classification but uh just classifying them is i mean not the end goal what classification allows us to do further is clearly create standard processes right so let's take the most simple case of a

vanilla my mysql injection right where whoever has started out would have added that single code at the end of a parameter and pressed enter on the address bar hopefully looking for a sql error right that shows up on the screen instead of like actual content on the page so that was that's one of the the most vanilla cases now so if we have a classification which we do for my seek my sql injections or sql injections in general right so uh we could create processes around these uh things that we do or these steps that we take to test for sql injections right so the simplest one is again the adding the code at the end but

what the standardization has allowed us to do is to create tools and frameworks that essentially make our jobs easier so sql map is the example for if or the tool that you would use if you are looking for sql injections because it does so much of the things that you would have to do uh manually and which would only just add to the grunt work right because i mean uh it's it's something that you have to do but uh you it doesn't really require intellect on a human's and right so essentially waste of time for us to do this grunt work uh where we when we could have spent our time uh figuring out other security

issues that actually required human intellect we'll talk about that later as well we have sneak as well which does this for us it directly connects with our github repository and makes our life so much easier and then obviously we now have the advent of devsecops so it's partially this shiny new thing that everyone wants to try right but uh it's not that bad of a thing right if people want to give it a try and then they realize the power that it gives them right automating a few processes and then helping or creating documentation for it let's say for the other teams to use the same thing and this is like an exponential thing that happens eventually that if

you solve one problem and then you figure out that you're solving a class of problems and and then suddenly all problems are automated right i mean this is the best case scenario obviously not everything can be automated 100 but it does take away a lot of the work uh that was essentially could be classified as grunt work right and we are free to do much more clever and cooler things right so how do we utilize these changes so we have figured out there are changes and we have processes around to do things you know so the this is like let's say my checklist for figuring out uh how do you go about automating a task

that you have in mind right so first understand the context of a scenario when i say context i say in specifically for security terms because just having a thing that you consider security issue by the definition might not necessarily be the case when you put in it into context of the company or the business requirements of the company right i'm sure we might all of us must have had seen that security issues being pushed uh the fixes for these issues being pushed back because we want a new feature to be released or the business requires us to do something else for the for the timing something like that right so it the context obviously defines uh

whether this classifies a security issue or not for the particular context we're talking about then we create a standard process for it right we figure out the steps that would that we'd ideally need uh to figure out if this is a security issue or not right so let's say again the same sql injection scenario we could add a quote at the end of an address with the parameter press enter on the address bar and then it would work right and obviously to test this process whether it works or not we'll have to do this at least a few times manually so once we have done that what we can do is automate now this could be using some

code it could be that scripts whatever allows us to carry out these steps right so if you want to do this exactly perhaps bringing selenium and other browser automation things to like do it actually in the browser and i mean obviously that's not the best way to do this in this particular case but that is also a thing that we people can look at right so we automate it and then we move on to the next scenario i and i want to stress especially on the fact that we do not want to automate everything at one go because of quite a few reasons but most importantly that it it tends to be fragile the first

time you do it right uh it would require eventual gain of trust by knowing that your automation can be trusted right it's robust enough and you have cleared or fixed the kinks that uh typically come in with uh different kinds of automation that you might have put in place right there could be random edge cases there could be cases that you did not think about and hence uh automating everything in at once would probably introduce more problems and it would solve so it's more it's a better way to go at a case-by-case scenario and automate with the things that you think are genuinely just stoiled and perhaps the ones that you can automate to the complete uh com

like the whole of its extent right so let's quickly look at a demonstration about how we can auto like how easy it is actually to like automate some stuff

hopefully it loads quickly

perhaps you can come back later to it oh okay

okay if i go back it works i'm not sure why it doesn't work uh we can edit this in the recording for you it's starting to work okay fine it works now cool so uh this is github actions fine and we'll take a look at uh creating a few workflows uh how work workflows are essentially just files that you would write your steps in like the automation steps in for github actions to take up something and do something for you right so this is one workflow it's essentially installing a tool called git secrets it maybe looks a little bit complicated at the first work but it's essentially just installing the tool called git secrets which is a scanner that looks for

secrets embedded in code right uh then we run the scan on our repository and then we upload a report to the artifactory present in github itself so as simple as that like it's it's barely 30 40 lines and you're done with the task okay so we'll take a look at another one

now this one is even simpler to add because you don't even need to write this workflow it's a it's a pre-built action that you can just like click a few buttons and then it will get added to your uh repository right but again if you can see it it does a bunch of things it scan for uh stuff in your repository and then approach the report again back to the artifactory so we'll just skip ahead

now this is the actions tab where you actually see the logs or essentially how what happened when the workflows that you created ram right so we'll take a look at our shift left analysis thing first and perform shift scan so it ran essentially and it figured out a bunch of things it did a bunch of checks and here's your summary right we found like one dependency with critical severity we there's a bunch of scan uh that happened secrets audit failed dependencies can fail obviously and then a few tests that we passed right and this report also got added to our artifactory now the cool thing about github secrets is that we have this tab security tab that gets added like for

with a few actions and you can do security management of these issues here itself so you don't have to maintain another tool like let's say issue tracker uh defect dojo or something like that you can maintain it from here itself so coming back to actions we could go to our git secrets scan and again build build is just the jobs name then we go to the scan where we actually ran it so it ran and then we upload the report and again we don't need external storage for this report again it gets stored in github itself so if you go to the same page same workflows page and just scroll down you see that there's a github report right and we

could just download this report as any other artifact i think by default it stays there for 90 days uh but uh has like you have you have the ability to like extend or delete depending upon the plan that you have with data right so with 30 lines i was able to like add two kinds of scan to my code repository and depending upon the trigger like it could be on every commit or a pull request or a merge depending on what event you want to trigger it on uh github actions is extend extremely extensive to like use and it resides right right beside your repository right so again why would you want to change something

that perhaps already works so it reduces human toil human oil is the same anything that essentially boils up to grunt work right it reduces chances for human errors and biases so uh for some reason i could have a bias towards bugs that i think or may have may not consider the bug right that could be a bias for me or let's say i have an excel sheet of endpoints to test right and i just missed one row because uh excels and what i mean it could happen right and i'm assuming i'm doing all of this by hand so human errors are like they do happen i mean no matter how perfect you are how perfectly can do

your job or how focused you are uh they tend to creep into your everyday work at some point or the other so human errors end up happening the fun thing about computer is computers is like if you have the right instructions to give them they don't make errors right i mean for the most part so yeah if you would want if you automate then you are essentially reducing the human errors in biases because they will treat everything uh the way they've been told uh to treat the the bugs and everything right it makes the system robust because again you're taking away errors uh from the system that reside on the humans or the people that are actually doing

the testing in different kinds of processes for application security right so it makes the system robust it enables trust for further automation because obviously if the automation that you created prior to something it works then you are more confident in creating a like something that is more complex right not every automation can be as simple as adding that quote at the end and making a get request right so depending upon what you're trying to automate uh there's a lot of things that can and can i mean go wrong so uh if doing it eventually again as a case a case on case basis uh doing it slowly one process at a time or one context at a time it enables you to

trust your automation yourself and your team to trust automation itself so that when it does mark something you know that it's actually an issue not not something so broad that every other thing is a false positive right and so forth and most importantly it frees up the costliest resource that a security team has the engineer itself right so when you give the engineer more time on his hand instead of having to do menial tasks they are able to actually figure out bugs or spend time in finding bugs that require human ingenuinity right things that would actually require the engineer to use his brain and not just a series of checks that on a on a pdf or a checklist right

so think of race condition box which uh perhaps computers are not that really great right now to find or changing different bugs to get bugs together right so a computer could find let's say a file upload bug it could find that rc exists separately but it might not be able to figure out what if if i could combine the two to create a third bug or something even greater of greater consequences and stuff like that right so obviously freeing up the engineer and giving more time on his hands allows them to find more quality bugs and essentially improves the security in general right because again these are the processes that he can now document and then we could con convert them into

automation again uh so as a chip away thing like we try to chop down the problem into smaller problems take away as many as things as we can put it into automation and then eventually document and continue the process till we i mean kind of run out of things but that doesn't really happen i mean not not very soon if you start doing it right so what can i automate right uh security testing to an extent obviously i'm not saying that we can take away pentesters or we could just automate everything and sit back and relax but we could do away with a good deal of things uh at least the low hanging fruits we have so many tools we have so

many frameworks uh especially if you look at the cloud environment people already have pre-built solution if you find some of them costly you might want to take a look at github there's a whole bunch of tests scans and workflows that are pre-built for you you don't even have to like write those yaml files right you could just click a few buttons and it gets added to your repository even the github free tier planner gives you a really decent amount of uh compute usage time and you can spend a lot of time that customizing optimizing stuff and perhaps just testing for things that you want to do and you may not want to do testing for every commit that you

push you would want to perhaps do it for every let's say release or on a release tag for your repository or your application right so different kinds of scan all happening at once at parallel saves up a lot saves us all a lot of time right then we have reporting and alerting of issues so once you do find an issue you have to report it right so just having a simple place or let's say a lambda function in the aws that sends an email on a critical bug that or for example an rc that it finds to all stakeholders the dev team perhaps the secretary testing team obviously and other teams that are relevant to this

case so all of those emails at once also saves us i mean eventually you might not think of it right away but uh the time that saves gets like as a cumulative thing is is it's a lot actually so alerting of issues and not just i mean there are complexities here as well because you would want to figure out what you uh consider as a critical bug and are you giving it the right definition your automation the right definition to figure out if this is something worthwhile to be alerted right now and is it not and it can wait perhaps and then uh it could just stay as an issue in your jira or wherever you want to put your issues

right so bullet ability management we just saw that uh github allows you to have that tab itself right so you could either get a security issues for a repository right in the repository but having automation makes it more easier to manage right so uh and plus it's more standard and filterable because you have a i mean a class of things that you have would perhaps have a same tag or same issue and then you could add meta information based on that you could filter out stuff we might not always be that disciplined or again the human errors might come in and we might forget attack for an issue and then it that issue for some reason would

obviously become the one that we're looking for and we are unable to find it obviously we we will at the end of it but again it's automation is not supposed to replace us for the most part it's up it's supposed to like help us do our work better because it's taking away the error prone processes of grunt work away from us and we all i mean maybe it's it's my opinion but like we tend to be more focused on things that we don't know about and we're working on them right rather than the tasks that we have been doing for all this time right and then we inherently become a little i know uh perhaps

a little over confident that hey this this is something that i've been doing for years and years on it and i would probably not miss a thing and that's where the errors start coming in right so again things that we have been doing all this time and we have processes for we know a standard way to do things we could push them to automation so that automation can take that way and we can focus on doing new things cooler things and perhaps bigger things right and figuring out new zero days or figuring out new defenses uh writing new code for software development and whatnot so what can't i automate obviously we have we know like we can't automate

every single thing that we have on our hands so what we cannot automate the simplest answer is unstructured processes right because automation is essentially a steps or a sequence that a computer would carry for you but if you can't define those steps or if those steps are too uh dependent on meta information right like if this then do that if this then do do that and these if and else's become a little too laddersome or uh i mean there are too many of them it i'm probably creating the automation for them it's more tedious than actually doing that uh them by hand right so unstructured processes in general are a little difficult to automate i'm not

saying you can't automate them but perhaps for the most part it will be way more difficult than uh seemingly more structured process so again uh coming to testing for solid ticket class of bugs when x is sophisticated i'm not particularly saying that it's supposed to be this uh always an rca like code execution or you take taking a point our server or you taking down an organization i mean any bug that a scanner can't find right and you could come up with simple scenarios that do this as well but essentially any bug that would require human ingenuinity to solve for that we talked about race condition bugs and uh things that a human can perhaps

change issues together and create a third thing out of them or a more impactful bug out of them that's something that we can't really automate right that's that's where the human brain comes in but uh just think of why even partial automation in this case is helpful because if i can find those individual issues uh with the scanner so the engineer can actually just look at those two box and spend less time and just to collate them and create the third bug right instead of having to look those bugs sequentially and then going on to chaining them together to create the third kind so there's that anything that requires manual analysis so a bunch of the scans that we do

obviously will have uh the false positive thing and like false negatives also so uh this would require manual analysis right so for example there's a process secret scanning where you find secrets in your code which we just take a look at in the demonstration it will well back that secret scanner is not again a hundred percent full proof right we could we could uh with a whole very high percentage of certainty like perhaps can figure out a aws access key because it conforms to a certain pattern but if you just think of random strings for example a randomly generated api token you would want to err on the side of caution and pick up seemingly random

things right and once you do that you will actually have to check it manually whether it's actually a secret or not because i mean it might not be it might not be in use also right it could be part of an older code that existed uh the thing about git is because if you have added a secret anytime ever like you've committed a secret it'll always be there even if even if you like delete it in the later versions of your code right so it could your scanner could be picking up an older secret and hence not particularly the best of uh results that you might get and this is an eventual progress a process so you

would add these false positives perhaps to a list to filter against and your results get better but you can't always be certain that these are the only kinds of secret that will show up in your code base right and false negatives also now this is the more difficult part because what the scanner did not pick up you'll have to go through yourself and for this you need like contextual knowledge of the code base you're testing whether this actually has api tokens and none of them showed up so does the scanner actually pick them up or is it considering them uh not not relevant or perhaps it's not matching the signatures of secrets that we've given it right a bunch of these things

so these processes that require manual analysis are the ones that end up uh being the ones that are more tedious towards or that towards creating automation because obviously i mean analysis to an extent is fine but at the end if the human has to look at it then it's not completely automated even if it's like just before sending the email that you have to send right so uh again all of these cases partial automation is still the possibility and for the most part you should still look forward to automating these parts of it right so scanning for secrets even though it might require manual analysis at the end you will a lot of time you

should get this manually and especially if you have to go through the entire gate history right because the secret might have been there at some point the perhaps the developer freaked out and then deleted that thing and fixed it in a later bug and on the surface of it on the latest commit you might not find it but obviously you can't really check all the commits from whenever you're setting up the baseline for secret scanning same goes for like other things right even a sql injection at this point could be really sophisticated we have race conditions equal injections as well but just knowing that a sql injection exists can help us uh speed up the process of

this uh the whole pen testing process and then we could figure out what i can do with the sequel injection instead of figuring out if the sql injection exists or not right so i mean uh that's that's the gist of it approximately so we'll conclude with a few things that we went through this in this talk uh security aspects dealing with applications right so anything that deals with improving the security of an application is application security consolidation and structuring of data tools and frameworks so consolidation of all the knowledge that we have gained over the years about application security and different kinds of issues vulnerabilities and misconfigurations that we have dealt with consolidating them creating

structure out of them classifying them and creating processes out of them right has enabled us to create tools and frameworks to make our lives easier essentially just to automate processes we could use tools to take up a bulk of the job uh and again examples being a skill a sql map snake and a bunch of other things we have then we talked about how we want to go uh ahead and create automated processes right so we took up a few steps we would think of a context we would create a process that we want to like which according to us would identify the security issue right we want to test it a few times manually

so that we know that the process actually works and it's not just based on assumptions and then we can think about automating it right the automation could be in a bunch of things like it at this point uh it would be python code or any other code for that matter actually and then bash scripts and if you're a little more uh into it it could be ego policies and you can test it with oppa or open policy agent right and things like that so several benefits are there we discussed about them uh robustness is one of them that your system becomes robust simply because you're reducing errors right and biases so we're taking away the human biases

and errors that could be there and we are simply creating processes that give the engineer more time free ups more time for the engineer to actually concentrate on his work and do more work that requires human ingenuity right and lastly we talked about automation what we can automate and then its limitations right about the things that we cannot automate but even in those circumstances we figured out that there are cases where we could do partial automation and by even partial automation could be worthwhile to take a look at instead of like dropping the idea for automation completely for a process that lets it is just unstructured so that was it thank you for your time

and i'll take any questions that you guys might have for me hey yosh thanks very much um i've got a question for you do you think there's a risk of people discovering what has been automated to do the security checks and then work their exploits around that would you think there's enough of the human eye been shown over and shown over the the automation of the results of it that they'd be able to that you'd still have that humanoid element to be able to take on that same the time of getting the tests looked at but like you and i could still pick up actually someone's actually trying to get through our automated checks right

so uh as i told we can't really like automate the whole pen testing process right so we uh i mean we can't expect like tools away tools to take away all of our work and uh i typically am focusing on i or i was wanted like bringing for the low hanging fruits right the things that we would consider meaning we i mean but would still have to do on our part right so figuring out an sql injection we could do things about it later once we have identified but the process of identification is at this point uh pretty much documented and again sql map is a great tool to like get started with it right you just give

it an end point it does a bunch of things for you but i mean even if with automation in place if bugs do come through we still have to have apprentice process but uh having a report that is pre-created for you you might save time on not doing again like let's say the task of figuring out if an issue exists and you focus on what you could do with the issues that have already been found right and based on the application you also know what the tool actually does so as a like a proper protester would also look at the fact that okay this is something that the tool would have missed anyway anyways right so if the attacker could

look at it from that perspective so so can like a pen tester so that's that's how it depends as i said the the talks are all recorded and we'll get them out as soon as we can and chat just for you speaking well we'll get something sorted along the way all right cool thanks very much man and the best look for your next talk oh thank