DevOps Application Security Teams for the Rest of Us

besides DC would like to thank all of our sponsors and a special thank you to all of our speakers volunteers and organizers for making 2018 a success thanks for coming this is my my very first besides speaking engagement not only my first speaking engagement besides about actually my first time at b-sides I'm aware of b-sides and I've seen the videos online and attended a lot of other hacker conferences like Def Con and that sort of thing but I've never been the b-side so actually I really am very excited to be here and to be speaking it's really fun I'm actually loved the the whole concept of b-sides and the one of the principles right even

though I've not been engaged deeply with them we do have a besides Nashville which I'm from Nashville and which I'm beginning to get engaged with but it we haven't done a lot I haven't done a lot with it yet but what I love about the philosophy is it's all about expanding the conversation and participation right they say there's no wallflowers there's no though no observers but we're all our participants in this conversation and I love that because that's and it is relevant somewhat to what I'm talking about that in the security industry and the one of the things I love about the security industry is it's very open open-door policy right we are we want to

share we community we don't look at ourselves by and large security professionals right as belonging to a company or whatever I happen to work it doesn't say it up here but you'll see in a minute excuse me at assure Ian but my work doesn't belong to Austrian now assure you may think that but but but really we we see ourselves as a part of the entire security industry and our work being relevant to not only that the whole industry to our nation perhaps or community and to the whole world right I mean we are sharing across nations we are sharing all around the globe and we we are constantly trying to improve the security across the board it's not it's

not this competitive profit driven market only right it is of course there's that aspect of it we've got vendors out there to like sponsor these events and help us help us to do things but but in general the industry is extremely open and I just love that about that and and so and besides it's particularly embodies that spirit so so I just love that and I'm really thankful to be here and it's not my talk I'll get into this in a minute in a moment but isn't so much a hacker talk I know besides origins is from the hacker community but um I guess just before I get started I wanted to say one thing to you guys that is it is is

tangential to my my discussion but it's just that what you guys do assuming your white hats and I know that the vast majority of their white hats are gray hats at least what you do is incredibly important and every single person in this room has it has a role to play in this conversation and you don't have to be a leader in the industry and you don't have to be the most awesome researcher in the industry I mean we've got both ends of that spectrum there but whatever your role is it's incredibly important and we we all go through times where we're thinking like in the right job in the right line of business am i

why is it that not moving forward in the whale I wanted it to move forward I just want to encourage you that you have a role you have an individual role to play in like my faith tradition we use the image of a body right and the community is like a body and every part of that body is significant parts of that body all have different roles but every part of that body is incredibly important to the functioning of the whole and that's the way all of us are whatever your role is you know if find it well your purpose is find it seize it and and do it to the best of your ability diligently with integrity and

you have a significant you will have a significant impact not only on whatever it may be I mean I'd be on a global scale you're gonna have significant impact and so I just want to encourage you guys with that that you're in an incredibly important industry what you do all the time matters I mean there are jobs that may feel like sometimes they don't matter what you're doing is incredibly important you don't have to be defending a country from a hostile nation state to be doing something that's important you are protecting people's privacy you're protecting people's resources you're giving people peace of mind which is a super super important to the quality of people's life what you do is really in

12 inside your on the defense side you're on the build side break side whatever it is what you're doing is really important so I just wanted to start with with that and let you guys know to carry on and do the work with with passion and with diligence and you've got you got your the you know your the warriors and defenders in a sense of the world in our day and age so so with that hopefully some of you are naturally brave friend naturally brave fans if not I'm sorry for you it's if you've only seen it once and didn't appreciate it watch it again I didn't appreciate it the first time I saw it second time I saw it it was

thought it was really funny I began to appreciate the humor but of course there's this lion in there that's a classic that's who is this in cannot see all right this girl that he's interested in but who am i why am he I hear and why am i speaking is what I just wanted to kick off with whoops someone hacked hacked my presentation probably Casey Rossini I do like smiling smiling is my favorite thing but that that was the picture areas the my background is has been primarily as a developer so that's a lot of times the case with app sec folks a lot of times everybody's different but a lot of in general folks

with development backgrounds do very well in application security and that's my background so I was and it actually is more in line with my temperament and my love and what I enjoy is building things and designing things and analyzing complicated problems and then seeing them come come to life in code solutions so I did that for many many years worked for many companies in the industry of all sorts and shapes but I was always interested in security that was actually my concentration in my undergraduate degree but I was operating in the field as a developer developer with a security focus eventually I turned to the dark side because of this personality flaw I was just always

asking questions always check checking assumptions always questioning right and that's that's the attribute that really makes a great security security professional whether it's defense offense right asking questions challenging assumptions recognizing what the assumptions are and then unearthing unearthing them and then challenging them are they correct what happens when you remove those assumption so I just have that kind of personality and noise sometimes my want my wife and head I just ask too many questions and not satisfied until I feel like I thoroughly understand something so I got into the security industry I worked for Intuit financial services business unit which is now they've been sold off but they used to do a lot of online banking at

2000 banks that they provided mobile and online banking to that folks didn't even realize that about into it but I was that leading an Arab SEC team leading the Citrix ass business unit all of those go to products and then and now I'm at assure Ian it's a pretty large but privately held company most people haven't heard about but it is it's behind a lot of stuff that you may not realize is it's white labeled most of the time and I'm also instructor for the SANS Institute if you guys are familiar with sands and their training I actually teach and I'm a contributor to the content of their new SEC secure DevOps course the dev 540 which is

secure DevOps and cloud application security and then I teach their Java and and then I also teach the web app and testing which is the 542 class so that's me enough of me oh this is my real purpose for living as my family my wife my three daughters and my son so I do like travel which is why I had that picture up there of me before the old city but loved it loved traveling so what's this talk about I'd love to walk around but he says I'll get feedback if I walk over here so I can't do any somersaults or anything but the the if you're looking for the sexy stuff a new exploit this is the wrong talk sorry

maybe just stick with me anyway but no new zero days I'm not going to publish a new open source tool for you although I'd love to do that that's going to do something for you know black ops I'm going to want to talk about building and defending so those are the defenders talk by and large as you probably guessed from the title so oh push the wrong button that's a problem with me so what I really want to talk about is how to secure a tell ops environment right and for a lot of folks it's becoming less so today but but it's still very very widespread amongst the security industry that they think that's that's

an oxymoron right secure DevOps you cannot have a secure DevOps environment well I full disclosure right about four years ago I drank the devops kool-aid and I've been a religious devotee of the movement ever since so it's um I haven't really looked back but it's so that's where I'm coming from I wasn't initially there but about four years ago that's that's I made that kind of switch and when you really get in deep into what DevOps is doing what the philosophy of DevOps is and if you actually embrace the philosophy and you are consistent in implementing that philosophy it actually improves security the security environment so the problem is that most organizations have like this half hazard kind of DevOps sort of

implementation like yeah we want to be DevOps because like developers and I know I was the same way we all want to be free from all those restrictions we all want to do our own thing we don't want to be independent we don't want to document anything we want to we don't want to have any any controls we don't want any roadblocks we just want to move as fast as we want use whatever tools we want pull in any open source we want and and deploy whenever we want right and so DevOps goes yes this is what we've been looking for let's push this as the new cool thing so it's um yeah for first it was agile right and

that kind of got us halfway there and now it's DevOps oh but actually there's a lot more to DevOps than that so it's not it's not just this liberation of developers right there's a dev ops right the ops folks want stability you've always wanted stability quality and stability the dev folks have always wanted speed freedom flexibility new stuff new shiny things bringing those two together into a single unit that owns the whole product from cradle to grave right is the whole power of DevOps that that organizational unit the DevOps team you want to call that the product team owns the product head to tail cradle-to-grave and is responsible for that product they can't push anything

off on a different order and that ownership aspect is really the key that's the that's the whole thing if you don't actually take ownership of that product in every aspect of that product including security although we don't have the second there yet we just have DevOps then you're not really living living DevOps you're not really embracing the philosophy you're just using it as an excuse to be um irresponsible in a lot of ways so so so clicked on that a little early but the key thing to making DevOps secure is this idea of a embed it doesn't it's not rocket science I'm gonna think okay I've heard all this before the embedded application security engineer in every

one of these teams every one of these DevOps teams so highly competent trained application security engineer and every team so you say oh yeah it's security champions right we know that security champions the concepts been around for I don't know more than a decade right it's uh I pushed it it into it I've used it before and it's the idea right think we're going to take all these liaisons out there these deputies if you will in in the teams deputize them as the security liaison as a security representative they're an engineer of some sort or whatever might be a product manager might be anyone on the team but they'll be deputized as our as our

representatives and they become our security champion we give them some training we get together once a month or something like that you know we inspire one another we might send us to some conferences or something like that but I'm not talking about that it's very similar I mean it's just a little twist on that but it's not security champions ice deliberately didn't call our team Austrian security champions I call them security mavens just sort of be cool so it's done I'm not the first to use that term but maven is a corner store an expert right digital used it years ago but but I called them security mavens instead of security champions I want to

distance myself from what everybody thought I know what a security champion is and so what we want to do is not have liaisons that have some training we want to actually have full-blown such apps tech experts in these teams so they don't have to be the world's greatest app SEC expert but they need to be someone we can really consider hey you're an application security expert I would like them to be ideally know everything I know and in some cases and some of my folks they actually have we've been doing this for about two-and-a-half years now they've exceeded me they're off talking at all kinds of conferences they're contributing all kinds of tools to the

open-source security community and they become bona fide applications security globally recognized experts a couple of them so it that's awesome that's exactly what I want that's that's what I would love to do I don't feel threatened by it I'll get into that in a minute it's like like they're gonna take my job away but they are dedicated to their product teams right they work in their product teams and they are part of that product team they're a natural part of that product team we're not like sticking them in a new product team right but they are we're being we're investing in them even more in my company's case than we invest in training are our central security

organization staff so we're we're pouring the money into training these these OPSEC experts they're in the team and I can go into a lot more why this why this works really well and but but you know it's I'll go into that that's actually probably like the last next to 20 minutes of this talk here gonna be going into the details well how do you do that because what I'm saying it sounds like it's an awful lot of money right they have to be putting an application security expert in every single team so and that's why I the title of this talk is actually dev ops application security teams for the rest of us because it is would be extremely

expensive if you went out and hired all these people and that's what companies like Google and Netflix and Microsoft do they have tons of money to pour into their security programs tons of money they're pouring into their application security programs application security has traditionally been kind of the smallest piece of the security budget unfortunately over the years but they pouring a lot of money into that they're hiring an army of applications security experts that's the easy way quote-unquote to do this right but we most of us have no way we're going to be able to afford that you know you know how expensive tired of just one app sec person is and this way you got to keep

in mind and this is what you need to keep in the front of the conversation when you're selling this idea to the company into the management to the executives just one app sec expert it's in a cost yet you know when you count all the overhead when you count all that goes into it quite a lot of money right at least two hundred thousand bucks right maybe more because it's you know their salary is only like half of half of the cost right maybe 60 percent or something like that seventy percent so it's gonna be at least 200 grand and what if you've got like my company we have over 70 current journey teams churning teams is a Spotify model of

agile right you can think of it as a scrum team or you can think of it as a to pizza team right which is like Amazon's model but we've got over 70 of those and we're expecting to have probably over a hundred within the next six months so how could you possibly hire a hundred or even if you had them represent two teams 50 apps SEC experts ridiculously expensive right so that's what this whole thing is about how do you do this and do it in a cost-effective way but it's not just about money it's actually a lot more to it than that if this this whole thing is in fact I think it's better than hiring those apps

experts for a lot of reasons to do it internally by training people so before we go into that a few a few potential alternatives of how do we do app SEC well in a DevOps world right and these are alternatives that hopefully most of you have given up a while ago but I see them still I see security folks pushing this stuff still and it just doesn't work so there's a security through edict right and this is still a big thing I got I got people I know that still say this stuff hey we Ryder wider policy we write a procedure this is the security procedure these are the requirements they have to follow them right I mean they just have

to follow them those fools why aren't they doing what we've established and we need to hold them accountable for that I think most of you know have you been around for a while writing a security policy has very little value except you provided to your auditors right when they come in and say hey do you have a security policy bla bla bla bla bla yeah here it is okay no one has ever read it so if the person that wrote it but but we have one and and they generally don't have a lot of value except covering you right when you say hey you have to have policy it's true you have to have them but the fact that

you've just written if policy and you said what people need to do doesn't mean they're going to do it in fact most of the time they don't nobody reads them and even if you explain it to them if it turns out it's impractical or it doesn't work in their situation or whatever the case may be they ignore it usually then they just rip risk management right they'll take the consequences if in fact something something goes wrong so security through edik never works but still many many central seaso organizations try to implement their security this way what it was lost get down

that's perfect timing actually want to know

in the sound Kamal

[Music]

okay so with that in mind yes indeed very sucky [Music] but he is very sucky right it's just it's not sucky it's very second it just doesn't work right you need the policies tour we need those ideally in a devops world of policy get in go into all these they can implement in your pipeline pipeline your CI city pipeline actually becomes in fact the the policy guarantee right that actually enforces your policy in which you can demonstrate to an auditor that you actually follow your policy I don't you get every way back there okay the that that's in a demo for that point all in automation automation the CIC pipeline is critical right to operate in this

secure manner in the DevOps environment but it is it's just a piece of the pie more important than back in the nap a less critical is actually this Wisconsin of having invented security knowledge and achievement because asked the way tell us works gotta be a self-contained team that owns every aspect that product having a home security not you don't have security

hello hey okay now I can save my vocal cords okay so I gotta make sure I finish early so we can at least have a few minutes for questions if you have them okay security through tooling we've all seen this I think there's a talk tomorrow about dev sec ops and I don't know the gentleman I don't know anything about his company I don't know anything about about him so I'm not saying one thing either way but he does talk about this issue of tooling I saw from the abstract and about how and you guys have seen this right we think if we just get the right tools in place it'll kind of security will happen right and

we've got the central organization and sort of driving that and monitoring those tools we just need to get the tools in place list it and I think most of you probably know No hey that is not going to work yes that one is also very sucky tools are critical but by themselves without that structure in place the human structure in place an organizational structure in place to actually implement them they're almost useless I mean they could actually be worse than useless in some cases because you've got stuff running and you're not doing anything about it there's actually an element where you can actually make yourself more liable right if you've got the knowledge but you're at your finger test and you're

not actually managing that knowledge properly so I mean we've seen this with laughs in the old days of when I was at Intuit we bought some laughs Simon pervo laughs nothing against them they were really good good tools but we recommend it against getting them they bought them anyway that put them in place mostly for compliance purposes to kind of cover us what we were fixing all our cross-site scripting and all that kind of stuff for a PCI perspective but we were just getting this continuous feed of information that required a pretty pretty experienced expert to look at them and go hey is this a real issue is this a real issue is this a real issue

that for about six months we go we don't have a step we don't have a full-time staff person to watch this stuff and it just sort of slowly faded away and and within like nine months they were just sitting there in bypass mode right non-blocking mode just doing nothing but gathering a bunch of statistics that we never looked at so that's just one example actually a my true story no nothing against them they had a very very very small team when we came there they had no app sect team the company I currently worked in when I got there I found out they had fortify they had bought a bunch of licenses of fortify they've been paying for it every year

for like three years and in every minute stalled right so so they didn't have anyone that knew what to do with it and even if they did installed it and you say hey guys you got to start using this static analysis tool the other problem is they'll start using it and they don't know what to do with the results because they've got 5,000 issues and it's just they're overwhelmed with noise right so you've got to manage the tools properly which takes a lot of care and feeding and therefore you need the organizational structure in place for that so security through gating this is what I still hear a lot we just need the gates in place let the DevOps guys do

what they want but then we need the gates in place before they release right they're going to have to go through this check checklist or whatever it is with security whatever your gate process is it's I would say this is suckie suckie but not very sucky because it is it is a step in the right direction but you need to move away from the whole gate you probably know if you've been around the dev ops folks much they hate the word even gate right no gates guardrails like netflix says right guardrails but not gates give us the guardrails of where we can run and we can run freely and we can move on our own speed don't put gates there because

if you put a gate minute you put a gate in the way of the product teams suddenly you've completely impeded their progress you've removed their autonomy and autonomy is is key for for DevOps loss the autonomy of the teams you've removed it they're now going to the central organization for approval again and they are going to um they're gonna buck on that man big time so but you know it may be a stage that you go through to get there it just realized it's it's it's out of step with DevOps so security through consulting so it's um this is like this is different flavors with us right you bring in the big consulting company and they got to fix everything

for you not so much even talking about that although that happens a lot you can do it internally you can say hey we've got all these security folks right already they're not apps SEC but they know security and so we're going to actually have them be our apps SEC embedded experts in the product teams right but of course they're not really embedded they're in the central organization but they're going to kind of be the consultants the coaches they're gonna drop in occasionally to the teams they're gonna say hey guys if you thought about privacy are you aware of GDP are or you you know did you you aware of our crypto standards or you guys encrypting everything

da-da-da-da-da-da-da-da using TLS right hopefully um they'll do stuff like that and then they like woo parachute out or whatever plane comes by like an Batman pulls them right out of there so it's up it's that and then they drop in again maybe later so they're not really an integral part of the team they're just this consultant that drops in there's a place for that but that doesn't solve the problem because the application security world as folks are moving very quickly making decisions literally probably every day maybe every few days right they have releases sometimes multiple times a day if you're doing continuous delivery I mean continuous deployment not saying you have to do that obviously in a

DevOps world but a lot of companies do and so it's not sufficient to have an outside source for a number of reasons one they're just not there enough they're not present at those decisions that are being made but secondly it has a all of these have a problem fundamental problem still there they're there they're not embracing the devil's philosophy of the autonomous team that contains everything they need to move forward you still have them dependent on this outside team you could say well so what maybe we're just not pure DevOps right or whatever well what it ends up doing is they still do not take ownership of security if you do it this way if someone else is always

responsible for the security of your system then you will always say well security security we had their representative they said it was good to go they said we were approved they approved our release right you might know there's stuff in there that is not cool but security approved it it's not and it's not even like they're deliberately kind of trying to hide under that it's just they don't own it I mean just think of legal right your legal organization you all have legal approved this contract with this vendor that's their job I'm not gonna read through that thing I don't need to worry about whether it's got some clause in there it's gonna hurt the company that's legals job and that's

the same thing that happens if you continue to have that security ownership outside of those product development team once they own it and I'm not staying like you don't have a central security organization get there a minute but once they own it then they realize hey we have to answer for this and they generally will embrace it they're not they're not developers aren't really fools like all that to hear guys think we are and I'm a security guy now right I mean I've been for ten years so I'm like if you're on the security side I'm with you I want these systems to be secure you're on the development side I'm not a developer anymore except by

hobby but but I know the process and I sympathize with the process so security through consulting doesn't work either so it's just an sucky because again it's a step in the right direction and then actually you may need something like that because your app second that person may not understand all of the privacy issues or compliance issues they may be an engineer they're dealing with design issues technical issues classic apps SEC issues not the non apps X security issues so you still may those consultants around for those but though tend to be more constant standardized kind of approaches to things the application security issues are changing constantly and so you need that was embedded team so I forgot to mention on

the first slide you may have been wondering why is Spock on there picking that guy's nose he's not picking his nose hopefully it's but if you're young you may not remember you know the classic Star Star Trek I mean ever seen it but you probably know who Leonard Nimoy is at least but what's going on there is the mind meld right the Vulcan mind meld and that concept is a metaphor I've used for for what we're doing at my company right now by the way I don't have anything to sell from the company it just happened to be I work for them I'm not here representing them I don't even know if they know I'm here but it's but

it's I probably shouldn't say it but it's but uh you know our communications department is probably gonna get mad at me when they realize I didn't ask for approval but it's um but it's nevertheless this is the model metaphor I've used the Vulcan mind-meld I say Deb's sec but you could say def sec ops its melding the mind of security and the mind of developers into one mind right it's we want to get it's really transforming in both directions it's not just we're getting developers to start thinking about security that's what we want to do as security folks but we also need security to start thinking like developers so we need to bring the minds

together the security folks in order to adapt to a DevOps environment which is with us to stay if you're still hoping it's a fad and it's going away I don't know what to tell you but I don't it's not a fad I mean it is working extremely well for full product development teams and for companies profits and that sort of stuff so it's going to stick around they maybe have a new evolution but it's not like we're going backwards to old waterfall okay if you're embracing waterfall and you think oh it's just a matter of time let's wait it out we're going to go back they're going to sanity we'll find the return and we'll be back waterfall

it isn't going to happen so embrace the chaos and then let's try to bring some order and security into that chaos and melding the mind the development and security is one way of doing that so so all of those solutions I went through that we're sucky are very sucky right one other problems is one of the issues the main issues is they just don't scale very well if you've got 70 or a hundred teams how are you going to do any one of those mechanisms I was talking about and I gave various reasons why in such an environment it just doesn't happen you have to have distributed security model you need to embrace the idea of a

distributed security mind model it doesn't mean you don't have a central function still but when it comes to product and application security 90% if not more of those decisions being made need to be made on those in those individual teams and you won't how you get there tomorrow it's not like okay we're just moving from here to here flip the switch and now we're doing it no it's gonna be a process but you need to have a goal in mind you need to have a you're aiming for and be approaching it in the right way if you're gonna ever get there so I've switched as you notice to kind of the Star Trek motif and theme

now because of the mind meld so all that other stuff was sort of introduction and I got like five minutes to give you the whole talk no so it's on product security and I've kind of mentioned some of this so I won't just be repetitive but the scalability issue they move so fast in a DevOps world extremely fast you've heard the statistics right in Amazon's case I can't remember what the number is it was like it's absurd how many changes are being made on a daily basis and in a continuous deployment I mean it's absurd look at the statistics you would be amazed and shocked right and it's you cannot scale and you cannot move quickly with anything other than

having the expert there making the decisions with people all the time just like they do how are we going to do this how we going to design this what are we going to do how we're going to scale how are we going to do that you need someone there all the time with them and that person has to have the competence to make those decisions you just don't JEP you ties someone to make those decision that doesn't know it so you're gonna have to train them hire them or train them right and the whole last thing there about the complexity is why apps SEC is different in some ways it just changes all the time you can't

simply state a policy a requirement and say this is the way it's going to function the solution just like writing software is that it Fred what's his name Fred Brooks wrote no silver bullet we all like to say no silver bullet right that's from Fred Brooks right the one of the designers or of the 360 architecture many years ago taught at University of North Carolina Chapel Hill where was that going with those silver bullet don't remember anyway it is there is oh it was the issue with no silver bullet is software is inherently complex it's thought stuff it is inherently complex and you cannot reduce the essence of that complexity you can only reduce the accidents to use

philosophical terms essence an accident so you can only reduce the accidents of the complexity you cannot reduce the essence of the complexity in trying to reduce the essence of the complexity through something is the Silver Bullet approach is going to slay the werewolf of complexity not going to happen same thing with application security the same problem right so the ownership ownership challenge is what I was talking about they need to own it they need to own it because that complexity is in the team they need to be responsible for that for the security of that system that are developing and it works very well in the classic DevOps now is classic right been around for a

few DevOps model we just haven't injected security into that across the board we've done it some places but not widely some companies are obviously Netflix is one of the leaders in this field but there are others guy here from Capital One I just think I think folk just before here cap embraced it ing as a bracelet a lot of companies have embraced it that are highly regulated as well as the high tech startup companies so okay culture clash this is another challenge but I'm going to slip right through this there's the classic cultural issues between security folks and development folks that you can overcome by embedding these folks in there right and I've already mentioned this a number of times

but it's what dev SEC Ops is all about building that expertise into the teams or hiring the expertise hiring is too expensive build it in and but this last line here is critical critical is they you don't just simply say they own it you need to make sure they know what they're doing okay and it is a process to get there but you've got to release relinquish the control it's very scary to the security guys relinquish the control conceptually at least we are going to push this out we were going to push this into these teams we're going to train them we're going to opponent quote empower them we're going to watch them for a while maybe here's make sure

they're doing things well but we'll continue to step back and they'll own it more and more more and believe me I mean I've seen it it's not pie in the sky thinking they will rise to the challenge if they're empowered if they're trained if they know what they're doing they love security securely its security is a sexy awesome thing these developers want to learn more about security generally they just don't and they haven't been exposed to it before it's not been in their purview so so what did we do and I got to wrap this up really quick in like three minutes how do you do such a thing it sounds great in theory but how do you

actually do it we have done it it's like wildly successful beyond what I thought even in my secret hopes that it I had this theory this is going to work right because conceptually it should it's kind of like security champions we're going to taking it to a next level but it works what has worked better than I expected and we're not like some small liens you know okay we were just you know we don't have a very complex environment no this company's been around for over twenty years they have tons of legacy systems that are crap right riddled with security issues and then we have the new guys offshore something not offshore like typical offshore use but in

companies we're acquiring around the world that are moving at the speed of light completely different model completely different toolset then our legacy folks and we've got all sorts of stuff in the mix so we have a very challenging environment distributed all over the world I end up having these these mavens are distributed all over the world of these teams so it's it's difficult for me to manage to lead it but it is working extremely well so but one of the first things you had to do is you have to sell the idea you have to understand the philosophy and sell the idea and one of the key things that executives understand is money right and

when you realize this I'll just give you our numbers it's roughly we've spend roughly training and travel sound like a lot of money went for a moment but when you realize we have trade we've got like 40 something experts now they're there in process some of them are really strong some of our or not as strong but they're in process and we phase it in we're roughly spending about two hundred and two hundred fifty thousand a year on the program I don't know that that will sustain forever we'll build up a body of folks large enough at some point that we don't need to keep doing that but that's training and travel because you've got

to do stuff to unite them as a guild you need it you build camaraderie amongst the team they're independent in their own teams but there's this guild model where they're sharing the information they're learning as well inspiring one another cross pollinating they become this group that is driving security on self self propelled in the product development organization so it's pretty awesome but it's you think of 250,000 a year and like I said how much is it gonna cost just to hire one person right this is extremely cost effective there's a lot of other things to sell it as well some key things fund it from the security organization don't just again say the edict we're going to do this

and now you guys all have to pay for it product a moment let's go screw you you know that's that's ridiculous you know you're gonna dictate to me what I need to do and then you're going to not fund it fund it it'll actually be extremely positive experience then they'll say hey you're giving us something you're giving you're investing in our people you're giving us some payback for actually starting to embrace security fund it it's not that much money or how much money you spend on Splunk this year it's ridiculous right it's uh it's uh this this is peanuts in comparison right and and if you're doing it on a very small if you're a small company will work to

you just do it on a smaller scale we're gonna we've got a pretty large mass of folks we got about two to three thousand developers and are in the organization so we have scaling this we're scaling it out but we're right now at the point where we've only got about less than 50 of these experts but it's growing very nicely so don't skimp don't say okay we're going to just build our own little training modules and kind of do that you know because that we can do for very very little money we're sending them to expensive training classes the same stuff you guys want to go to if you're a security folk and conference is they go

to DEFCON they and they love it and they get inspired they go to things like this I think there's actually some of our guys here somewhere hey and you know good to see you so it's um it is it fund it well one of the key things is I'll step through this really quickly sorry for this I've got like how much time do I got just a couple more minutes make it appealing security is fun make it a fun experience to be a part we've got people clamoring to become part of this or this this team now they wanted to be on it it's growing and growing and growing I'm getting them from all over

the place including the infrastructure folks it's the cloud governance folks want to be a part it's cool it's that's they really like the program they're thrilled with it and they and the hope even HR is thrilled with it because it is actually helping with retention it's helping with job satisfaction motivating people we're investing in them and they're going this company really invest in us and they let us do all this fun stuff you know so it's we have like team-building we go off and do fun things together and it's not really that expensive when you look at it from a the big picture deal okay so I step through some of this stuff a big gotcha is here you do need

to give them authority if they're gonna own security they have to have the authority it's not all or nothing though you will get there gradually but you do have to relinquish that that thing okay really quick why should I trust a Vulcan right why should a developer trust that you're gonna be good for them well it's awesome for developers it's a total win thing there's no downside to this right they're getting trained the development management this is what you need to sell it to we started out selling it to the president of my company he embraced it then I sold it to his directs then I sold it to the directs of those are those VP's and the directors all more or

less maybe 90% embraced it and then we sold it to the to the the folks in the trenches and we had them apply do not appoint them that's one of the pitfalls to avoid do not appoint them or have them appointed because then they just become like that liaison that figurehead of the security folks you want them to won't be motivated and interested to learn security so you need to have them apply and initially you won't have one on every team because you won't have folks applying from every team that's okay so it's on it will grow organically and you have safety nets in place as you're moving in this direction okay so it's really a big

win for secure development and what the way you sell it to them is we're gonna give you freedom we're gonna remove those gates you're gonna be able to make the decisions yourself and they will love that you say but you're gonna have to embrace this training process and this this guild process so for security it's really where I got most of resistance and I'm in security I report to RC so and he's supportive of it but I have had some resistance in security and because they feel like they don't trust them and they're not going to do a good job or we're losing control and you know as a security guy I can say we're all

kind of control freaks right we want to I think I'm running out of time so it doesn't mean you're going to not have the safety net you're still going to provide the leadership the vision role and the oversight and then but you are and the real goal if you're in security should be to secure your systems not to protect yourself right protect the resources in the products your and their uh that you are over overseeing and this will get you there faster than trying to do it those other ways so why is it good for the company I've already mentioned some of this our company loves it HR thinks this is the greatest program we have

right it's because it and we we have open environments as well we do our own constants is we're having one in Nashville next year we're hosting it and we're opening up we're gonna let who knows maybe we can start our own a little Derby con there but whatever use ik city con or something like that so it's and we and so it enhances our company's reputation is giving back into the community because we don't just train our people but we open up stuff for other people to come in and be a part of as well it's it's really cool there's so much stuff you could do with this so it's um you just need to realize

I've got to empower these guys I got to release some control and I need to be willing to invest some money in it you need to sell it to through your leaders okay we've I already said this we had a huge huge increase teams that had no interest in security on us honestly I'm not exaggerating they just didn't care and they are now like hey there's they're driving stuff themselves and then they're managing their managers of our security meetings are getting involved and their directors are getting involved and they all go and hate who else can we get involved in this program this is awesome and they really care about security now and that they're making the

decisions you can see that in their products that I don't have insight to sometimes until after it happens like oh you guys did that awesome because that's that's what we would have told you to do had we been present there now you guys know how to do it on your own so it's just worked incredibly well we've had some some teams that were completely rebellious in a sense if you will against security and against central control those guys have now like most embraced this program and one of their guys is is the this land of global emerging global apps tech expert I was telling you about so he loves it it's like found a new found a new career

for himself even though he's still in development he's still writing code and he's still part of that team that he loves hoping this is like the ultimate right well one of the well no not the ultimate but I say one of the really nice frostings on the cake there is you will develop this hide line of the security experts we've got our slack channel where we all close it off so no one can see what we're saying right and that we all talk amongst ourselves and we are constantly communicating with one another I'm just the I'm just the the leader right but it's it's it we've got all these folks in this team now back and

forth what do you think about this hey one one just went to a loss opposite USA right several of them went there one came back said hey I had two talks they were saying contradictory things I want to submit it to the hi of mine what do you guys think and it generates all of this engaging conversation and research and all that so it's it's cool okay I'm shutting down it's not the Silver Bullet you're going to get resistance the individuals must be motivated and if they're not you've got to remove them from the program gracefully you have to have strong management support especially of the mavens they can't be said hey you're a maven but you can't

think about security we've got too much for you to do if you want to avoid the individualist who'd really not interested in being part of this security conversation globally because they do need to share they needed to cooperate you will still have to care for the gift of strategic vision and you still have to keep momentum or will fizzle out you know this distributed organization needs something to pull it together that's why we do group events together we pull people from all over the world and do things together we provide this unification kind of framework for them and just some essentials already mentioned some of this so I will not I will not go through

that again the next gen right that's not really next gen but it's the latest the greatest Star Trek this main thing is this cascading effects what we're trying to do is now create a multi-tiered thing where these folks are now training other people right we trained them and they're training other people and so it's beginning to cascade down and it's gonna grow hopefully geometrically right so that's it thank you so much for being here I appreciate it [Applause] do I have I don't know if I have time for questions but if do I have any time I think there's a top coming in no okay I got to get out guys if you do have

questions I'm happy to talk you can figure out what my personal gmail email address is there as well you can see the pattern so it's communicate with me through email I'm not a big Twitter guy but I do respond if you send me a message and I don't post the ton occasionally but I definitely with me any one of those ways I'd be happy to connect I'm sure I have more to learn from you than you probably have to learn from me but I'd love to connect with you guys so by all means connect with me thank you so much sorry for running late