DevSecOps – The Good, the Bad & the Ugly

let's introduce our our next speaker i want to invite nevadita murthy um to speak on devsecops the good the bad and the ugly uh in the i.t world an i.t security space in particular is filled with buzzwords like devops secops devsecops cicd and so on everyone wonders where security comes into the picture why should security fit into the process and how can it or should it fit in at all without slowing time to market in this presentation the vidita will introduce the audience to the world of devsecops navigating attendees through the basics and key terminology next we'll move into an exploration around the value offered by devsecops and the potential challenges in its implementation and common misgivings

when it comes to expectations versus reality um with the real world implementation examples she's she will enable the audience in in understanding around the understanding advantages and disadvantages of implementing devsecops and import more importantly the urgent need for its implementation in today's software-driven world so without further ado i want to introduce novadita thank you so much for for coming up hi everyone thank you thanks for the invite yo okay um let me share my screen y'all

okay

um okay [Music]

yes

okay i guess everyone can see you um yep we can see your screen we can hear you you're good to go thank you okay thank you um hi everyone uh my name is nabadat murthy i'm a senior security consultant at synopsis based out of boston massachusetts um i started in security due to movies and sitcoms that inspired me starting the security operations where i worked in network incident and uh security operations and application security as well i moved to valverde uh triaging sas pentest scrs risk assessments and finally into desktops i've been working on delsacops implementation for past three years now and while i'm not working i like to travel out and be outdoors yeah 2020 didn't allow me to go places

that i wanted to go but i discovered a little bit more about massachusetts and uh while i'm traveling i like to you know take pictures and especially over landscapes and i'm also a voracious leader i'm hitting my target of 24 books for this year um enough about me um so what am i going to talk about today it's all about devsecops i would like to first introduce to the whole world of their cops and then explain the need and then we move on to the good bad and ugly parts of it um as you can see by the name it's an amalgamation of development activities operation support and security checks it's a cultural mindset altogether

it requires coordination and sync between teams um involved in the sdlc process um this energy and synergy between teams is obviously helped by automation um desktops is not a quick fix or a temporary solution it is a long-term implementation you play to gain long-term goals in ensuring an organization is able to you know achieve and maintain secure sdlc practices it requires development teams to follow a standard stlc process waterfall or agile and ensure they don't break the process well one is actually able to easily detect if they do do that um while not going too much into details about you know about it uh developers ensure that they maintain versions of their code and follow up your review process before

the code is moved to production or like not even just production but through various environments segregation of duties is a must to avoid um one person or one team to gain control and have um one thing how things are updated in the code or environment this means a separate team is responsible for development and then a different team is uh responsible for testing then are separating for deployment etc operation teams is responsible for you know supporting the entire development process which includes maintaining and updating the operating environment defining and implementing our you know deployment process um while logging every detail of the same security team is responsible for identifying any vulnerabilities being introduced through this process

and ensuring it doesn't reach production if it does reach production there's a clear trailer why it was moved over there dev circles or even devops insists on traceability um so that it's easier to identify what change resulted in you know what problem um or an incident or a security threat out there while changes in code is traditionally what starts the process uh desecure starts way before that and requires other activities to complement um its process design reviews and post-production monitoring for that matter um tesla cops is a lot of moving parts like the different wheels that run behind the clock to ensure it's actually effect it reflects accurate time um jessica's as mentioned earlier is

development security and operations each stage in sdlc requires some kind of security review but these reviews don't end at the end of stlc security checks are done before and even after that because devsicops is not a process but a cultural mindset that needs to be encouraged and you know adopted hence every activity related to development even if it's not actual development must work hand in hand with security and operations you start off with training developers in writing secure code you surround development teams with secure secure development standards and frameworks to work on and then you have security champions supporting their teams to write in secure development efforts out there champions are chosen because they are

part of their development team they are actually developers but then they also have a keen interest in security so they do know what comes through the mind of a developer this way they can evangelize about security within their teams um what follows are several security reviews along the path that you see over here with starting with risk analysis architectural reviews threat modeling which come into picture during the design stage and planning stage this is followed by secure code reviews either manually or using various ide plugins yes there are a lot of plugins out there in the market you know which complement the sas scans um and help you run a quick scan locally and then you have sas

sca das and i asked scans you don't have to do all of them but pick and choose what is applicable for your environment um that are done during coding and testing stages these can be repeated at every stage of development such as at build and then test or release as applicable and you can do it several times then we have penetration testing done on like post production but not actually on the production environment you don't want to risk that um to identify any vulnerabilities found using previous like not found actually using previous methods you also continuously monitor your application and like a production environment for threats using various tools such as rasp um that is runtime

application security um and uh inviting external researchers to test your critical applications by offering a bug bounty pro uh you know program out there um all these offers various methods of testing your application for resilience against security desicops insist the integration of all of these along with functional testing methodologies um to test your software's resilience against bugs and uh vulnerabilities um we talked a lot about developmental activities uh but deficios is uh development activities but deficios is not just limited to that the third part of this whole thing as you can see there's dev but then there's also ops out here that is ops or operations you should not be forgetting about that all operational activities need to be

reviewed one can now build up infrastructure using code um just yesterday i was uh i attended the workshop for ansible which was really interesting uh please do check out the recording if you can um but it what it obviously showed and we see out there is like now you can stand up servers or uh configure your environment using code itself um this makes it easier to stand up and like stand on uh different nodes or servers out here in your audio environment easily and but much more important consistently we now have various options available to run our software and application applications the best way to observe and watch an application work and identify security issues in its

truest form is while it is running in its environment so it is important you also need to run checks um and identify for security issues within the environment itself some of these issues could directly impact the application and hence it is important to secure the environment where it's hosted so the operations part of it is also involved in devsecops um you apply the same steps that we did on the downside apply it on the upside um and make sure that what you are standing up what you are running what you are coding to run out there is monitor and you know test it out effectively um so does this mean can it be done using

one tool absolutely not while there are a lot of tools that offer mixed services there is no one-stop-shop tool that does everything for you um you have tools that help you run the whole process and tools dedicated to identify vulnerabilities using different methodologies one thing i observed is that a lot of companies that have exclusively offered sas tools earlier are now jumping into the bandwagon of uh having sas tools added to sorry sca tools added to the capabilities uh and vice versa as well um a new thing that i observed uh is that scam tools have also started offering sas capabilities uh within their repos because they have the unique uh advantage of having the code present

over there and there itself you can now run scans they obviously can now run scans uh when developers check in their code like a as a post comment or a pre-commit check over there or like through web hooks or any of that sort um and vice versa like a lot of them like you know uh sc8 tools are now started doing sas services as well you have tools from open source as well as proprietary uh methods out there it's not all open source or not all property note that no one tool fits all environments and in some cases no one tool fits for the company as well um in my opinion it's not how it

should be but then it's very rare it could happen it's possible i think in such cases organizations should run a bunch of pocs again and see what fits in their environment best um i think there'll be a difference of opinion among different implementals and security folks in general which i would really love to discuss in discard or even in the chat here um but yeah the question was can we have multiple sas tools in the same or like multiple dash or like the same method but different tools in the same and should we be running it in the same environment or not um desktop tools are not restricted to just doing scans but also to do with other activities such as

reporting tools defect tracking or management of it environment building tools etc and please know that it is not restricted to just tools that you're seeing on the screen right over here even a simple thing such as scripts uh like shell script powershell python are for various capabilities i have personally used a lot of partial scripts too because that's what uh worked on a lot of windows environment and for some reason python wasn't allowed so i learned powershell and uh was able to do a lot of work uh especially in terms of identifying new vulnerabilities using the with the help of powershell and api calls there's a large world out there so choose by sleep um so why is everyone talking about

it i don't know about all of you out here but when i see a whole desk ops implementation come alive and watch as change a change triggers a job which in turn triggers a scan which in turn gets the results and evaluates it and triages it or maybe like you know identify new variabilities and based on that um break up process uh like a breaker deployment if it fails certain checks um it's something magical that i feel in my own personal opinion uh watching a lot of work being done in minutes or sometimes even in seconds is mind-blowing desiccops definitely helps team scale up big time we all know in general security teams have always dealt with a resource

crunch but that doesn't stop bad actors to take advantage of any vulnerability present in your environment with more and more applications moving away from a monolithic strict structure to a more multifaceted multi-pronged agile method security teams need tools to buck up and catch up and make sure they have control over what is being moved out here what changes have been done plus there are only 24 hours in a day of which eight hours you need to sleep and hope i'm hoping you all get to get enough sleep out there um that desk ops lets you crunch a lot of work in really short amount of time speeding things up cutting down delays and scalability are

the biggest advantages of desktops implementation with global teams across different time zones and now work from home for that matter organizations are now increasingly seeing the need for processes and frameworks that help get teams together and reduce dependencies and helping teams achieve goals through the same process why is everyone still talking about it what i found in several organizations is that they have different teams working as independent verticals or businesses having their own independent silos uh how does one work in such a fragmented environment how does one manage to get some consistency while still getting uh by still allowing for leaving into each team or silo which requires for their because their processes demand it how do you get teams working

on a single goal that they have all agreed upon um without but they are unable to change the entire development process can we do a gradual change instead of a full-blown change in implementation um deficit office is able to accommodate all of these and more no matter at what stage the organization is in terms of uh security maturity or how fragmented it is um or centralized the organization is desicorps lets you initiate and implement security activities and lets you adapt to different functional um teams out there so you may have often heard this phrase security is not one person's responsibility it is everyone's responsibility um tessa cost makes everyone participate in the process and practices this um very same rule

developers app managers ops team security team reviewers testers all are involved in this one of the biggest processes i'll say one of the biggest pins of desktops is traceability um since processes are interconnected and automated it is extremely easy to determine what change resulted in uh you know what issue or problem in terms of ipl uh you know in terms of ideal tickets which change ticket is connected to which problem uh ticket or incident ticket this level of traceability is also makes it make easier to make people accountable and responsible for uh you know for the task with additional monitoring it induces developers to be more careful uh while writing code and writing secure code for that matter which prevents the

pipeline from breaking in the current process um in the current project that i am working on right now uh an email gets sent out at the end of the job to the entire development team um this job basically what it does it pulls the code runs the scan identifies new issues and sends it sends an email you know if it has if it has new issues out there in high of higher critical severity um it notifies that but there's this one small section out there not small but uh pretty important section where it lists down the developer's name along with the changes and when was the change com checked in um which triggered the scan

which resulted in these vulnerabilities now this is possible because i'm able to pull because it is all automated and it's pulling from the this scm tool i'm able to pick up all the check in the comment details since the last build and push that info into that email as well now you don't want the entire world to know if you're a good like you're a bad developer out there or you're bad at writing secure code right so it kind of subtly tells them right better code and pushes developers to you know be accountable and work on writing secure code yes you saw this right there cops can probably give you the moon but it may

take some time to give you pluto um time and again i've seen some client expectations out there which it's just so out there that the first order comes into my mind is why what were you thinking when you said this um but then next thing is like my mind starts turning on how can i get uh make this possible that's what i think about yes devsecops can make a lot of these far-fetched expectations come into reality but it requires time and a lot of uh other factors to work before it actually is reality unlike other methods a full density implementation takes time and patience is also required so what are the good things about testicles the first and obvious answer

is automation automatically trigger bills scans uh deployment evaluations approvals etc automation works for all now since these stars are automated uh security team can focus on other important activities rather than just the operations of it all as an example with 700 apps in scope um uh you know and counting like increasing it's not just limited to 700 apps it's increase increasing by the day it would have been difficult for a security team of four uh to be able to monitor it regularly and and all of the releases and run the scans and figure out you know if approvals have to be provided or not and again approves to be manual uh the time it would take the team to do

this would be would have been really long and i've seen that case where it it's taken a few days for some time you know because the team is so burdened with you know all of these approvals spending in their queue um however with automation the team can actually balance out the work a bit uh recently the team request received a request um to onboard and scan 30 micro services hear this out 30 micro services two days before production do not ask me what happened why did they come up two days before production but i'm pretty sure every security team member who's worked in the abstract process out here has seen this issue come time and again

where the development team comes to you at the last moment for a sign off um with the automation team sorry with the automation piece out here we were able to complete all of this in just two hours yes digest that to us 30 new micro services onboarded on two tools it was first on board it was completely new so we had to get this added to different application uh different tools set up the jobs to run the scans run the scans evaluate it and then triage the scan results all of this we were able to do within two hours now that is the power of automation earlier i mentioned global teams being the norm and now work from home

we have also the other challenge where we have team members distanced out there within the same city for that matter um desiccals helps to create a framework which reduces dependencies to do tasks and activities a lot of organizations have teams spread across different time zones like asia uk ema um in my client's case it has usually been like the security team is over here in u.s u.s out there but majority of the developers are in india and china um the security team does not have the budget uh not of capacity or the bandwidth to support all time zones desicorps helps them to support developers in india or and china by monitoring code being checked into

late esc and early morning est um we are also able to create portals uh you know which through which these developers could run on-demand scans this is like for the small code or like their own branches that they like feature branches that you're working on and they could run these scans for on-prem tools um another big advantage was that uh that i really like is standardization instead of it being a chaotic world out there um there is a more it makes standardization makes it much more easier to weigh through every step of the whole process with standardization it is easier to scale up and the process and makes updates to it also pretty uh manageable

anything more yes there is of course um aspiration speeding things up agility another pillar of desert cops fast-paced world needs fast-paced solutions desticops offers that with manual steps cut out and dependencies cut out the whole process start to finish gets done faster and earlier in the early example 20 micro sorry 30 micro services were done in two hours this wouldn't have been possible earlier that is what that is the speed that teams are looking for that that is the sp speed that organizations are looking for um with less than a week available in the whole sdse cycle for the whole sdsc cycle out there security uh gets very less time to run its activities a lot of security tools have today have

improved in terms of the time taking to scan and provide enhanced capabilities to customize your scan um and select the checks as applicable uh to run you know them and like cut down on the scan time even further if you want to um speed speed speed again it's extremely important to tell psychopaths traceability with all the things connected one can connect a change request to a problem ticket incident ticket out there to give you an example now it's not the best of the examples i would say but uh i would have to see if something would have seen as a trend happening out here uh in terms of transability um in one case the organization had dealt

with a data breach due to a vulnerability found in one of their applications and another case an external researcher notified the organization of a vulnerability now you would wonder wait tessacops is all about securing your code how is finding help a vulnerability helping your case but hear me out now that the i like obviously i can't divulge any of this details like actual vulnerability or the client uh in the first case the organization was doing regular scans and testing the vulnerability could have been found out easily through that stop and testing however it did have and it had did have some dessicops implementation as well however this modernity determined that their choice of tools was not good uh for

their organization and pushed them to re-evaluate the tools and methods being used to test the security choosing the right security tools that work for you um and for your organization it's it's absolutely necessary um we now have reporting tools that that's what i mentioned earlier it's not just about running scans or running security steps it's also the part that comes after reporting we now have reporting those in the market that help you identify if a validity was found across different tools uh someone mentioned this as defense in depth so for example an sql injection was found through a sas tool a test and even a pen test all such there are reporting tools which help you figure that out

deduplication and consolidation is what is the key factor out there okay so if it's all good is there anything bad about it um well we haven't reached utopia yet with their cops because there are so many pieces involved it is not necessary it will work as one big happy group uh every organization has to deal with prioritizing their activities and does it cost may not be everyone's priority even though they understand the you know advantages of it uh in and this uh i've seen in two cases wherever um we are not able to integrate to the devops process yet because um we are dependent on them to make certain changes um and uh and change the script as well

because you're integrating the tool over there now the devops team is not able to they don't have the capacity or the bandwidth to provide us time and you know work with us to do all of these things uh they do understand the uh you know advantage of such a process it cuts down on the approvals as such but then they also are i've seen like devops teams are not exactly um ready i would say this is in my opinion it could have to be different out there but um not exactly ready to implement security in their process the reason being and now security requires another team to um make sure everything is working fine it is not under devops team control to

you know now the scan servers or the scan part they they cannot control it if anything breaks down in that process they will have to connect with other teams in devops the the devops team has full control on how the environment is running with security added to the process they find it difficult to keep a control on that and that's one of the biggest challenges and that's one of the biggest concerns which is rightfully so um that they have in integrating and that's the dependency that we're talking about not only that they're also dependent on the application teams um not everyone is the same you have apps and you have apps that use legacy versions

time and again you will find teams uh supporting legacy apps because there have been no plan to transition them yet um for example while a majority of the applications may be developed using visual studio 2019 um there are still some using 2012 um or for that matter sharepoint uh apps using the old framework um i've had several cases where the development team itself has fine you know uh said like we are scared to do any making any changes because we do not know uh if we deployed if it break or not it's so old but they have to support it because it is running it is important and there right now there is no plan to transition

it to a new version out there um even now we do find some security tools may not integrate that easily and automatically with other tools so it's not that everyone everything is like you have the tools out there which will work properly or which will connect properly which is working fine uh you have to create a layer of abstraction uh so as to make it you make use of it in the whole automation process in the second process for that matter um a good example of it would be burp uh until now did not have a ci plug-in and it was not easy at all to uh it was very difficult to integrate it into an automated process

but that is possible as of today and i just recently saw it on the website right there is the ci plugin available um then we have those legacy apps we've all heard of those apps that organizations continue use it's critical to functionality but it was written ages ago and there can be no changes done to it or if like no one actually wants to do any changes on it assigning resources to automate such apps uh does not make sense or in some cases it's partially part of the devsecops process but then you know like you may get the wrap being evaluated either you can run the scan but since there are no changes happening it

you would not want to set up web hooks or pole monitoring for changes that will never happen uh but will utilize scm or the ci server resources because even like a poll polling done is a job done a job is running out here and it's checking continuously so it's taking some amount of resource from the server trying to assign uh it to such legacy apps where there are no changes happening um in the long run uh does not help especially when you want to make sure um you know the environment is running efficiently out here um while evaluating an automated process you also need to check what the roi is of getting it done it's still bad uh you need to take baby

steps with uh dessicops it is not a quick fix at all it requires patience and tenacity to implement it all any desiccops implementations take a minimum of a year for sure anything less than that is probably incomplete um dessicops involves a lot of planning and designing first before you actually start setting up the whole solution you need to first identify the gaps in your current process followed by uh and see like what tools would be um required and how do they support the process you need to uh you know implement it then comes coordinating with different teams uh to get their buy-in and then um followed by all teams implementing the required changes this does not happen overnight um

if you are able to make changes uh if you actually were going to make some changes to your process it affects all the applications involved all um you know whatever is following the process out there i heard someone say uh we will go and make this as a big bang change out there um if all of your applications are being scanned in um in this environment using a common set of libraries which is the gen general standard in the seconds you use a common shared script um set of scripts out there to run all of your jobs um unless you put in a specific condition and change any change in this library would impact all of your applications

getting a new application onboarded to this process may take long not necessarily always all the time but it may take long depending on certain factors i find that and dotnet applications usually take a lot more time to be onboarded because you need to ensure they build correctly now i would say 30 percent of my time has been spent on resolving issues arising out of bills because visual studio tends to you know work differently from uh and like hide a lot of build errors and provides um dependency on run time but the same cannot be said said for ms build you cannot when you're running an automated uh build step uh you don't use visual studio for

dotnet applications you use them as well countless of times of encounter a scenario where the app team says uh like has built the application using visual studio and checked in but then since our automated process uses an uh ms build um it the command line that is it breaks due to variety of reasons incorrect dependencies or uh missing dependencies or incorrect directory structure um sorry um and then the ci tool itself is not up to market times a lot of organizations that start using the start the ci process start using it with jenkins because it's free and open source and uh and it's pretty popular as well but you have other tools ci tools in the market as now uh however

the past year i've literally been zonked out due to the number of bugs uh in jenkins and its plugins um as a result of which i had to implement several dirty work around two tasks which i wanted to get done um but i could not do it because of the bugs in there um a lot of plug-ins on jenkins are no longer being maintained or supported that does not mean it's completely bad it is very useful it is definitely helpful um jenkins still does a lot of work out there and other but then other tools also have its own limitations um i personally have worked on jenkins i've just you know badly used a little bit of

team city and uh gitlab yeah but i can't comment on that because i'm not i'm not extensively used it as i have done with jenkins out here finally the first part developers themselves can be deviant little tricksters sorry if there are any out here in the audience but there's a very very very small fraction of it uh there have been times when i've had to escalate it um asking them why they haven't notified of a change or sometimes they actually skip processes uh but we are able to catch them out um or like in the earlier cases where i mentioned two days before broad goal live on you know a new application service coming out there um

it has to be given on you know just because everything is fast doesn't mean it happens in quick seconds out there we also like especially for a new application we still need time there we um there's still a human involved in this whole breast pressure you know process to get things done so we still need time a bit and sometimes developers make it a little bit difficult out here um the other thing is that tools do not have a level of maturity to do everything that is needed every tool has its limitation on what it can and cannot do especially in an automated process though that area is declining um to give you an example

jenkins cannot probably allow conditional parameterization what do i mean by that is let's say depending on what you select in one drop down the uh the values in the other drop down chain it's not currently possible in jenkins as of now maybe there are some other plugins which are for workarounds but not the actual uh requirement and finally the biggest headache that any security team has to deal with false positives um i have this one app uh which is part of the whole automation process which always throws up 350 plus critical vulnerabilities all of them being possible every time i've tried it's all being false positives even with a tiny change in you know um let's say the connection

parameters i've yet to determine what's going on wrong out there why does it keep coming up and it's just one this one particular app it doesn't happen with all apps but i don't think anyone has come up with a decent solution of how to deal with false positives like you know we have workarounds we do have like don't check for certain things or like um by default mark these as um a false positive or something else that's all uh which you have to take the second part i would say be careful with that uh but if you have any good ideas i would definitely like to discuss that in the discord oh or in this chat

well then do we still need this yes and yes we still need this you still need their cops it really makes work way more simple and streamlined um hope this presentation actually you know uh gives you uh like your you know benefit of doubt over there and helps you understand the knitting routines of whole deficit implementations um there's still a lot more to talk about in terms of opsec right if you observe i was more i was talking more on the uh development part uh but then that's a whole new topic that's a whole other discussion out there and that's something that i'm also learning as of now i'm still picking up a lot more things on that

part um but yeah uh that's about it for my presentation and if you want to connect with me here's my linkedin and twitter i'm still trying to use it a little bit let me know if you have any questions