DevSecOps Culture - Ali Yazdani

hello all and uh let me to introduce myself in my syntax uh I'm a security engineer with over 10 years experience in application Security in different uh industry sectors from Telco fintech and right now in AI powered uh company from 2016 till today hope till future I'm o contributor on uh project like mstg mobile security testing guide and right now from 2020 I'm leading project depops guideline uh under o organization and right now I'm senior depops engineer at scalby why I wear this t-shirt this hoodie today so if you want to follow me or the project you can scan this QR code and uh find more detail about that traditional view of uh software development was uh in traditional

software development security measure where the in the right side of the development when product was ready to push to the production on or in pre-production we had some security test if you were lucky you get informed as a security engineer you were you get informed that application is ready to test in pre-production and if you are not enough lucky so the applica you have to meet the application in uh when it was up and running in the in the production environment and with uh Advent uh devops this culture of security test and this approach doesn't work so they didn't work so they they are starting to uh bringing security as a part of uh part

of development process and that was that devops appears and and uh the things that devops aiming is fill the gap between Dev development security and operation but with which which uh um things by promoting a culture of collaboration shared responsibility and continuous Improvement as you can see here we have oops as you can here we shift security checks to the left side of the pipeline development pipeline by adding different security test like SAS es and the software component alysis D and other other part that we can have I just put some example here I know that we can add more or less here and based on this uh boom curve you can see the time of uh error Discovery based on

based on the phase that we we find the vulnerability or error the cost and effort to fix will be reduced so this is this is one of the one of the things that we are following in devops but it's not even just about the technology from a technology point of view we put we add some Security checks into the pipeline development but from a team perspective also we experience some changes like in the traditional development we had solution delivery team that they just focusing to push and enforce product to the to the operation that time the product was like a hot potato they want to pass it to the operation team and then operation team

should be deal with the product vulnerability quality error and things like that because operation team and security team was out of this game but by devops uh operation teams involved as a part of this team so we call it devops team and right now security is also involved as a part of this so all of them should be SE together for one approach that we can see in the in the next slide but it's not enough because I can see a lot of companies a lot of LinkedIn post a lot of tweets it was tweet but I don't know what is right now X that uh people people talking about okay we Implement some SAS D and things like

that and okay why we need penetration test why we need B Bounty VDP and things like that and I can't say while we are while we are shifting security to the security test to the left side of the pipeline development most potential attack surface that we had is the right side because this is the this is the place that every person able to see our products if if you have some git repo private git repo so they cannot access to the code but when we have a application which is web application or mobile application everybody can meet it that and find some vulnerability so the right side is still important part and also I can see a lot of non-security

experts says okay we have S why we need d we have pentest why we need bug Bounty or we have bug Bounty why we need pentest and as security expert all of us sure that each of them has a different different uh uh reason because they can cover but they cannot replace each other to summarize the previous slides let's review some wrong facts devop engineer de Ops engineer is devops engineer plus security engineer so company hired one and get two so it's a cost saving from HR perspective but not true I can't say because because when I joined the company as a senior Dev I say okay can you fix this problem in your kubernetes and I said okay said I

can not all of it but a little bit of that but I'm not a devops engineer sorry for that and Def secops engineer is a security engineer which is able to work as a part of Def secops team and the second fact second wrong fact is by implementing some tools we have devops devops which is wrong okay tools is not enough and we we will see why and the third one is since devop says security is responsibility for all of them let's kill all of of security engineer consultant or especially or lay off them it's a cost-saving time so but it's also Rong because that we reviewed that security engineer should be part of de de seop's team and uh so we need them

fortunately or unfortunately and first uh by shifting security test to the left we have a full secure product but in the previous slide we saw that a lot of vulnerability can came on the desk after pushing a product to the production after passing all of the security test in the pipeline after a good session of threat modeling secure design uh I don't know 1 million different checks that we had but pillars of devops a lot of people right now talking about devops but most of the most of the conversation is around tools and Technologies if you are going to the LinkedIn and open your LinkedIn unfortunately I'm follow the hash devops in my LinkedIn and every day

I can see some academy uh propose some courses by getting by getting involved in these courses you get uh you will be depops from Zero to Hero but it's frun because from a from a real perspective of Def cops and from a real world oops I just spoil my uh based on uh I just want to refer to this quote from Bruce Schnider that is really fit and inside insightful regarding this situation that a lot of people thinking okay def Sops is implementing some tools and B Schneider says if you think technology can solve your security problems then you don't understand the problems and you don't understand the technology and based on this by reference of this code we can say we

have three main pillars for depops he and process tools and technology and governance we have to we have to implement all of them and we have to uh consider all of them to have a full depops in environment people and process this is the most important part because it's enable other component to be work if we want to implement tools we have a we need someone which has a and some process which are really good working good and have a good approach of depops and then at the beginning it it's really a fact that at the beginning of implementing devop since we had to take care about the right side and in another side we have to think about the left

side so we have more workload for security but the good news is please don't uh complain about that because the life will be back to the normal and you have a really smooth stand up at the morning after two or three months of uh hard workking traditionally development was responsible for fast delivery security was responsible for application security and operation was responsible for stability of the software but def aovs try to dis drro the silos and says we are following this goal delivering secure and stable software quickly this means that all of these people in this this different uh team that we had in the past have a shared equal shared responsibility and based on

their expertise they are able to contribute on this approach and second part of people and process is process now we have a shared responsibility model based on the previous slide so we have to update our processes development process all of the policies maybe that we have to Define them based on this this uh promotion clear communication uh transparent development and active collaboration because people will sorry process will help people to stay involved if I'm feeling that I'm involved in the process I will I will be participate on that so one of the key Point here is communication that we should every time try to try to to inform people and keep them involved in the devops process and some topics that

we can hear here um is like a shaping a security Champion team which is from a different department like Marketing sales because as you know marketing is just try to chase uh engineering team to provide some solution and uh sales wants to sell that solution so we have to we have to implement a security Champion team and make sure that all of us in the same page from a providing secure product perspective also we can we can have some training for developers and uh operation team and help them to understanding to and get the more involved in uh security mindset the second pillar was technology okay to make the process more practical we need technologies and tools and uh

one of the things that uh we have to we have to consider when while we want to implement some tools automation is key so if we have some solution if we have two solution and one one of them has more uh give us more uh automation that that is a better choice for us and Tool will uh help us to reducing effort for the test increasing accuracy and repeating the tests and here is a picture that I steal from uh oops guideline project that actually I draw this one that you can see we have two uh two different uh pipeline here one one is uh infrastructure pipeline because we have eok infrastructure as a code

scanning here and some test here here for infrastructure and here is a application Pipeline and you can see different uh um stages steps of security tests that we have here and some tools candidate here a lot of vendor fighting with me why you didn't put our logo here and I said okay make a revision I will consider that one it's good and here is an example it's really it's one of uh real slide that I'm using while I'm trying to uh talking with the development team because I need to give them some insights and one of the approach that I'm taking is uh providing some real word example because I believe that as a security

expert uh you have to able to translate a technical problem to a non-technical language and also techn translate a non-technical problem to a technical language for the team that you are working with them so here is why SEC secret scanning is important and what we have to do with secrets and here is some uh screenshot from hacker one which is a bu Bounty platform and you can see uh what is what is this logo I forgot that a Snapchat Snapchat paid $50,000 for username password of J for artifactory which is exposed over GitHub by implementing a secret scanning maybe they they could to save this amount of money but while as a security engineer in the company you fight for

th000 EUR budget for implementing solution you always will be rejected and here Starbucks paid for 4,000 you can you can't see it but I saw it it's $4,000 $4,000 for jum Cloud API which is exposed in a public G repository that they had so and a lot of example that we can use for different um stage of devops that def seops tools that we will Implement like Des why we need uh software component analysis why need to need we need s why we need infrastructure as a code scanning and a lot but as I said at the beginning uh today we don't want to talk about technology more because I know that we are facing with a lot of information and

a lot of AD advertisement about def cop's tools and Technologies and governance which is a third part of the uh devops having a system of governance enable us to keep the track of our process identify area of improvement and also visualize the outcome to compare them with the uh expectation that we had in the beginning of the process are you going are we going in the right direction or should be changed and some topics that we can cover here is some Audits and checks um security standard that we can see in the next slide that why it's important because we have to consider security standard and best practices in software development and design and uh policy as code which is

really important right now and by implementing a good policy as code you can uh reduce amount of uh policy violation um and a lot of uh things that we have a lot of mass that will be happen in the in the application uh environment for example one of the things that we are doing is since in the current company we decided it's one year ago that we decided to moving our infrastructure from old and Legacy infrastructure which was up and running on the cloud to a new one so we completely create a new AWS account and then one of the policy that we established I provide a name convention policy because in the old AWS environment when I login first time

I got surprised a lot of kubernetes cluster main test test Main and something like that I I I never could trust to which one is which one is and I'm not never enough Brave to touch them without talking with developers and operation team but when we are starting to create to migrate the services I said okay this is a time for name convention I provide name convention policy which is maybe not my job but I did that and because I'm really get it down person and I never care is it security area or not I saw a problem and we have to fix that and then we implement the policy as code solution to even fail the

changes if the name convention is not matched with the syntax that we defined it's a it's a long journey but the longest story short that it was really helpful right now we have a environment that if a new Joiner join the company able to understand okay this is a staging account and this cluster is a staging cluster or test cluster in the staging enironment and data protection visualization is really important because if I back here a lot of people think about that okay if we have a secret scanning s and d and other container scanning in the pipeline so we have everything but the things that we need here is Central vulnerability management we we need a

place to uh collect and centraly all of the reports and results from the scanner and make sure that uh everything is is going good never trust developers or operation this is an endless war between us and them just kidding and challenges and mitigation that we have overcoming to cultural resistance when we are starting to implement devops one of the things is cultural resistance that we definitely face with that and we have to prioritize active collaboration and communication between stakeholders to make sure that all of us in the same page regarding software security regarding security and what we want to do and one of the things that I do that I did is using this type

of slide I know it's a little bit uh a strange so for some for some teams but it's it's really insightful for them because they all of us knows about hacker one Buck crowd and platform like this uh but um as a developer or operation person maybe they they didn't inform about that and uh we need to show them some real cases and this is really good time to show uh show benefit of devops like fast time to Market improving security and reducing the vulnerability because when a product which is pushing to the production environment without Security checks it's definitely vulnerable right now by a lot of security test still application in the production is vulnerable but that

without them definitely is vulnerable and seamless tool integration continuous monitoring and improve of security tooling maybe in the past you implement a s solution but right now according to the changing Tech stack or development stack that you have the current tool is not the current SAS tool is not uh working well for you so uh change that I listen the podcast regarding uh startups and he said a lot of startups fail because instead of uh failing love with the problem they are failing love with the technology so please don't fail in love with the technology you have to you have to consider the problem and try to fix that okay I really like this tools

and it's provided fancy dashboard but maybe not working for you and addressing compliance and regulation try to understand according to the market that your product and application will is working which regulation and standard should be considered and try to uh consider them as a part of the part of the development pipeline development life cycle that you have if you are in charge to participate in application design um session so you have to you have to think about uh this regulation and try to translate them from a non- technical language which is a standard or regulation language to a technical language that developer team and operation team can understand them and in conclusion devops is a devop

journey is a long long-term investment so be patient and try to uh shows uh changes yeah they showed me yellow card so the second one will be the the second is uh the good implementation makes it cost reduction activity as well as you can see here based on ND report uh we have different different phase of uh development and it shows us which um according to the cost which uh which uh stage can be reduced the cost if we catch a issue or security vulnerability or error in architecture phase it's one time cost for us instead of 30 time cost for us so shift to the left shifting to the left catching issue as fast as possible this

is this is a things that we have to follow by shifting everything to the left but not whole thing that we have to we can do we can do we need we still need to take care about the right side of the pipeline as well we need pentest bu Bounty or whatever according to the budget that allows us [Applause] okay do we have any questions from the audience oh we have two okay and they're very close to the speaker is that okay okay super um maybe less of a question more of a side note that I have experienced in the past couple of months as well is that not only security is um a very

popular topic to to look at from the death secops pipeline in itself but also the topic of privacy as with most of the companies nowadays right they all see oh shoot where do we have uh personal data lying around in all of our systems so as much as you do the shift left with privacy aspects as well right where do we need to anonymize data where or mcade it some at some other way um that also helps on the Privacy aspects actually this is really good question and you can see here a topic that I put under governance is uh data protection when we provide these topics for o depops guideline project with Milan which is my co- contributor on

that project um we we uh recognize that data protection is really important and a lot of companies did didn't care about that in the in the pipeline development after after pushing the product to the uh to the production we get a security incident that okay some um pii data are in the logs or things like that it's really depends on application design to to find the point that we have to be able to implement uh data protection checks it really depends on the product and it's really depends on the design that we have but the thing is if we consider the triangle of uh security and put Security in one corner uh is of use in another one and

features in another one it is really basic uh security concept and we put a ball in the in the triangle and try to move it to the security so we uh go far from those those one and it's really depends on the market it's really depends on the business it really depends on the product that okay which security position is really good and good balance for this product or

businesses all right we had one more question but unfortunately we are out of time so please give Ali a big round of applause thank you so much Alie