How We Got Here: Integrating DiD Into DevOps

[Music] all right I guess we'll go ahead and get started here I want to give everybody a little time to get settled with the delays today but welcome this is remembering how we got here we're going to be doing some talking about integrating defense-in-depth into kind of you know these new-school DevOps types called type cultures that we see growing you know very rapidly in the marketplace and you know the general perception seems to be leaving security in the dust but we're going to talk about some cultural ways to get you know kind of the core of what we do as security professionals back into the defense or into the DevOps culture cool so my name is Matt Krieger I'm the

principal security architect for a company called sports engine out of Minneapolis Minnesota this is casino what everybody my name is casinos while the director of network architecture applied trust and we're headquartered out of Boulder Colorado but we have a local presence here in Salt Lake as well yeah and so to see man I actually worked together when I was out applied trust a couple years ago and him and I had these long conversations you know on a regular basis just about the differences in what we find ourselves doing now I work for a very cloud-first company that has really embraced DevOps culture you know we ship code 30 40 times a day we don't own you

know a single server that our customers will ever see everything we do is in Amazon meanwhile cosima I worked on several projects that were you know very enterprise very municipal we have worked in a lot of hospitals together you know big data center projects and we we've kind of seen this divergence over time of we've got old-school security professionals that are very engrained in defense-in-depth and making sure that we are you know doing everything that we were taught you know in our cissp classes and then moving over to companies like sports engine where you know like that we're very very DevOps oriented we're very agile we're putting everything in Amazon we're like we dependent on them

and what are the consequences of that and how do we communicate the need for this type of security that was really built around the idea that you can lock the doors to the data center into you know these more agile environments yeah absolutely in despite working primarily in the enterprise space of we definitely work with a number of clients and are moving into that either starting in cloud-first environments or somewhere around this hybrid migration path where they're moving from their current on-premise infrastructure and in implementing and incorporating off-premise infrastructure in some cases making that wholesale migration out to the cloud and so even though those technology stacks may be somewhat similar and different those practices

across those different environments certainly are diverging and as we've had these discussions we keep coming back to there's got to be some set of common themes and inactions that we as technical folks and security professionals can start to incorporate into our practices and not have these new workflows like DevOps you know kind of leave some of those those needs behind yeah one of the most important things you know like we threw at the bottom of the slide there you know what we want to talk about today are not going to be you know specific technical implementations that everybody can go back and say hey we're going to you know configure our proxies in this way and

this is how we're going to achieve defense-in-depth in this environment but it's really to look at the culture of how security is being ingrained in your organization and think critically about your environment and find ways to get the security practices that we've had in the industry for quite a while into this this type of environment so as we start to move along if there are any particular topics that aren't very clear feel free to raise your hand and we'll certainly clarify at that point but if you have we'll have some time hopefully here at the end for questions so let's go ahead and get started cool all right so as Matt alluded to we're here to talk about incorporating

those security practices into new were close like DevOps so really what are we here to talk about all these cloud platforms continue to change and we have so many different abstract concepts and these services and capabilities that are presumably released on a daily sometimes hourly basis and that is great for the development world and trying to rapidly deploy products but that rapid movement can certainly create new challenges to security which is constantly trying to keep up with those advances so as those as those concepts continue to change what we have there is a constant change in the marketplace around how those services are referred to what do they even mean between different providers and different

platforms and so with all that ambiguity how do you begin to actually change your information security approach and adapt to those challenges when depend on your organization maybe your strategy isn't a security first strategy it may be more growth oriented it may be partnership or you know sales oriented so how do we get security into that into that that mindset and ultimately there's a lot of folks in the room and we exist at different areas of the service act so we're in that service stack does your organization focus we were talking about that traditional enterprise in addition to that move to the cloud but regardless of what infrastructure you're living on this full application development world is changing rapidly

so if you are an active first shop you're probably not investing a whole lot in infrastructure traditional infrastructure or possibly even operational security so where your existing is going to determine certainly what some of your capabilities are and in okaygo it will determine some of your capabilities and ultimately impact your ability to operate continuously you know we used to talk about operating continuously in terms of high availability making sure systems were highly available and now with many of our services that the end users are depending on the expectation is that service is always available so it's not just highly available it's constantly available so one of the kind of key concepts that I want to touch on is that

you know the entire organization has a basic set of needs and a lot of times these align a lot more than we would really think you know developers really love to throw around the concept of a Minimum Viable Product you know there's some there's some version of this that you know I can go to a console and I can type in some data and I can get a formatted chunk of data back and therefore my job building that module is the absolute minimum thing I can do to say that this feature is going to be ready to ship you know and then you put some UX on top of that you want to make sure the end-users can use it you know

because you know we're not giving people console access to you and nobody wants to do that and then sales and product are going to go what is the minimum lovable product what is something that we can actually put in the marketplace and have people excited to use meanwhile we're sitting over here in security going holy crap don't me over with this minimum crap so the key to remember is there are common goals throughout the organization and we're all trying to achieve it in different ways because I guarantee you that no one in sales is saying I want to put something out in the marketplace it's going to leak all of our customer data to a you know to a

third party and get us in the news because I'm never going to sell the thing again the UX people don't want to have to deal with the fact that hey we had a breach and now we need to go back and put alerts in and change everybody's passwords and and deal with you know changing our workflows you know in risk in an incident response capacity and development those guys is if there's ever a breach if there's ever you know vulnerabilities found in the code those guys are gonna have to go back and fix it anyway so if we just kind of look at this as a common set of goals across the entire organization it's pretty easy to

see where the where all of these goals align from a security perspective there's um you know certain signs of trouble that have noticed throughout organizations that kind of can show you that you're you're on a rough path to creating those common goals if you have some sort of change approval board that only meets on a periodic basis maybe it's weekly maybe it's monthly and your developers are coming to you and saying hey we've got we've got this idea we really want to move forward with it and you say okay get in line we'll get to you next week people are going to start leapfrogging your approval board and you're going to end up with changes

that you're going to have to go back and change later in the day or once it's deployed your architecture team needs to be part of your security discussions it's so many times I see architecture teams go off in the corner and they say hey we've got this great idea we're going to implement these data models in this way and then when they bring security security says oh yeah you know we don't we don't want that data to be available you know across the board in that manner so let's just let's let's back off on that and then suddenly you end up with more contention where again people are going to start Enda rounding the process to get the product done so

that the customers can buy it and everybody can make money policies policies are something that are a necessary evil in this in this environment however your policy committee should not be something that sits off to the side and says all right what's the ideal scenario that is going to keep everything you know completely buttoned up in the way we want to use it because again if your policies are not attainable nobody is going to end up following them there's a couple other you know teams on here that I'm not going to go through every single one of them but one of the key you know one of the key ideas here is to as a security

department and a security organization in general because maybe you don't even have a security department maybe you've got one security guy and a whole bunch of a whole bunch of recruits in other departments that are really you know kind of backing you up from a security front but it's important that your security organization is able to operate in more of an interrupt fashion instead of you know pulling every couple of weeks or month to say hey where we at what's our current list of what's our current backlog that we need to address at this time so when we're talking about defense in depth what are we really talking about well in that traditional space we've got three general categories

that we typically work with so physical controls actually physically preventing somebody from getting to a secure quote secure or interesting resource we have lots of technical controls that exist all at many layers of the OSI stack and I think kind of the important thing to know here on the last bullet is application data controls hello there we go all right sorry so the thought that you have application data controls seems fairly straightforward when you think about say controls like authentication but as Matt was talking about those data models themselves how is the data stored managed manipulated transmitted you know that is a that is a control or set of controls that have to be evaluated

uniquely potentially per application and based on the frameworks that you're using and then finally administrative controls back to the concept of policies if all I have is a set of policies that tell me what to turn on maybe I'm not sure what I shouldn't be what I should be actively turning off and what the rationale be has behind that I think we've all probably been caught by some sort of a software or a platform upgrade where feature that you previously had deployed in a well-known state changed another version and you had to go back and revisit you know some of those foundational requirements and assumptions to make sure that your requirements were actually mapped into

that new platform so as we step away from that that traditional enterprise or physical space well what is what do things look like well in the physical space we were very concerned about resources so allocation provisioning how would scale those resources and certainly space to to house those resources and that commonly associated with was associated with some sort of capex in the organization having to capitalize those assets depreciate them and oftentimes perhaps even finding out that an asset you know usable life time was not as long as you actually need it from an employment perspective so that would tend to throw your capex planning into a bit of a tailspin conversely as we start to move into this virtual space

whether this is on-premise virtual or off to cloud type of a provider we're now starting to talk more about infrastructure of platforms software connectivity you know actual services that were likely subscribing to and what's really can be those are a shift effects so I can be in theory much more nimble about purchasing switching services because if I can keep that epic but excuse me that context excuse me effects expenditure roughly flat nobody's probably questioning my budget very much but if I am trying to acquire another half a million dollar disc array that wasn't on my refresh cycle it's probably going to be a much heavier heavier lift to try and pull off organizationally so continuing down that

path you know as we move from owned infrastructure to infrastructures infrastructure as a service we're changing our mindset from being all about the gear you know what does it take to build out that gear pull it off to the loading dock unbox it integrate it into the environment have a life cycle to it and then ultimately dispose of those assets whereas in the infrastructures of service space it's all about the trust in the service that you're subscribing to or that you're contracting on so the thought is hopefully your contracts are nice and tight and and address the the needs that you're trying to achieve from that service but all the nuts and bolts of building that environment are not

physically running cables and plugging and drives anymore necessarily it's all about the configuration management and deploying those services in the manner that is needed from a lifecycle management perspective you're probably not talking about retiring machines it's more a narrative cycle of continuing to operate that environment and instead of treating those virtual machines or those application instances is these special very unique items that take a lot of care and feeding probably have lots of virtual machines watch about lots of application servers and you spend your time more hurting them as opposed to care and feeding them and really what it boils down to is data management right if I ingest data and doing something with that I'm deriving some value

provide providing platform service whatever and then disposing of that data or doing something else with it so how I process those bits is a little bit different than just pulling it all drying out and sticking it in a in a destruction chipper so let's let's talk about trust a little bit here so really what levels of trust do you have right so whether we're in a physical or a virtual environment you still have multiple implementations of the OSI layer taking place so that's unique code that's running at every one of those layers is that code all provided by the same vendor or is it all even running at the same version level are the service delivery teams across

those different layers are they in sync so that you can rapidly deploy an Orchestrator costo service layers and you know really what is having trust and that service organization worth to you not to dig in too much on some of the the recent cloud outages that probably folks have experienced but I think that illustrates the point that we place a lot of trust as not only information security professionals but operational folks in these services that we depend on from a day-to-day basis so what are we actually going to do to enforce that trust and allow those services to work for us in the way that we need them to and you know when it comes down to it

nothing's new here right I used to have to trust my data center team to be able to deploy networks servers whatever those components were in a manner that I needed for them to work well now somebody is doing that in a cloud environment and I just need to make sure that I can interface with them and provision things in a manner that I need so nothing's really new here what does devops really have to do with the intersection between these worlds yeah so where it really comes down to is DevOps the DevOps culture that is coming into infrastructure management comes from this kind of shortened feedback loop you know it's Christine was saying you know it used to be you would order

servers someone would unbox them you'd have an integration team go put them in your datacenter make sure you had all your power and cooling in place then your server team would come and spin up the blades your network team would make sure that you had the right drops and everything and there was a you know there was a longer period of time to really get all your ducks in a row and now when people have ideas and development teams can turn on new services you know it's a matter of you know minutes or hours to turn on an entirely new environment so we're still left with the same kind of defense-in-depth needs from your physical technical and

administrative controls but the there's some changes you know kind of at either end of that your physical controls are very very different you know if you're running in an Amazon or Nagar or you know any cloud provider you no longer have keys to the data center you no longer have and you know security guard sitting there making sure someone doesn't go in and take the hard drives and walk off with them if you really want a wall around your data you know we're thinking in terms of encryption so suddenly our technical controls are moving up into our physical controls and we're starting to kind of double dip on some of our codes and some of our

control layers to make sure that we're getting the defense-in-depth we need and so you know the reason why encryption there in physical and technical controls is that you know we want to sort looking at ways to we're going to be encrypting file systems and then we're also using application level encryption to take care of some of the most sensitive data that's on those individual drives and at the end of the day if we need to toss it if we need to toss a drive we delete the key that data is useless and then you know at the end of the day we're going to have our administrative controls that still tell us all right this is all the

you got to turn on and don't turn any of it off so moving that forward into you you know what we're calling defense and DevOps is you know as the culture shift continues to move down the line you know we're getting further and further away from the idea that there's actually servers behind what we're doing you know who really gives a crap about physical controls anymore you know we've got people writing code turning on servers testing their code making sure that they can get it you know out on the internet and deploy it as quick as possible so ways that we can manage that is to give people infrastructure as code we can start to take the infrastructure

management portion of it that used to be a very onerous process you know involving multiple teams multiple people multiple days and put it into kind of our same software development lifecycle and make it part of product development you know included so we're including infrastructure and code you know in our application and software development lifecycle so that all of these controls that we've put around the actual product we're making can also control the infrastructure at this point our technical controls are everything on the host really remain the same again don't forget that their servers under there so we always need you know all of our ACLs all of our firewalls ideas IPS encryption antivirus you know depending

on you know like your environment and everything but we can also add some additional technical controls on top of that so we're we had technical controls really living at you know layers 2 through 7 you know in our traditional defense-in-depth model we can take a lot of that and turn it into just layer 7 controls and you know that's where we're adding in things like secure coding libraries automated testing all these things that really live at the top of the OSI model but give us the defense-in-depth that were used to from working our way down through the data center your administrative controls with this agility do take on a little bit you know more of an important role in your

life in your application lifecycle you know where as you're bringing in multiple teams when turning on a data center environment you're change plans are pretty important in that you've got a lot of people that need to know what to do it the necessity you really put those in place however now we've had the ability for developers to go and turn on a server put some code out there run some stuff we need to make sure that our change plans are templated in a way that really enforce the type of security and checks that we want to see as security professionals there's always going to be policies involved you know seriously don't put API keys and source control

this is something that like literally we have to tell every single person you know when they start secure coding because it's the easiest place to put these things however because of the way that we're moving code around and we're shipping features on a you know very rapid basis it's easy for these things to start to get missed if they're hiding in you know large you know five six thousand line pull requests so with those policies it's really important to start putting things like peer review on top of those it's you know you can train your entire staff but if you're ever relying on one person to remember everything that they learned in training at the time that they're

creating a change set you're going to they're going to miss something so we're going to start adding behavioral defense-in-depth on top of all these things so we have multiple people that were trained maybe at the same time maybe at different times but they have different brains in their head they have different memory for different areas of things and one thing that somebody might forget their peer is more likely to catch and then the final thing is we really need to understand trust boundaries in our environment so when we build our technique our technical controls we're looking at ways to manage data flow between trust environments and I'll just our together a really quick example here of you know one of the ways

that you know we've seen these things these types of things change super super basic scenario you've got an application server that needs to make a request to a third party service that goes out over the internet and therefore ends up traversing an untrusted Network your firewall rules can be used to create a trusted path through a mixed environment you know if if you're blacklisting all outbound traffic and making sure that your whitelisting known good services which you know has been widely proven to be the most effective way to catch you know malicious traffic then we're going to be saying okay we know where this remote service is we're going to give our servers access to it and create that

trusted path the firewall is your delineation between your trust or not trust in an untrusted space that creates a path of trust we know our application code is calling a third-party service we know that it's calling a third-party service that we trust because we put it we put code review in place our deploy processes have you know Quality Assurance checkups we maybe we had security checks right into our deploy process host-based intrusion detection make sure that we don't have unknown changes happening to that codebase once it's deployed we know where the trusted service exists because our service provider gave us you know and ID or a list of IP addresses where their servers live on the Internet and

we use DNS servers that we assume are going to return that that hostname or I'm sorry that IP address then just to make sure none of that gets screwed up we're going to add a few rules on our firewall to make sure that okay we've got these you know this list of IP addresses that our application servers are allowed to talk to therefore you know if we break any any portion of that chain we're going to end up you know in a failure state where we're not leaking data things just aren't working and that's a you know it's a better state eventually once the connections are is made all the way through we do a TLS

handshake with the remote server like I said keep encryption on everything and make sure that we have a trusted CA that's sign it you know that's vouching for that remote device now as we start to see even bigger companies make moves away from the data center environment where we have known IP addresses and that's a viable thing for them to hand out you know parts of this path of trust are going to start to break down a key example of this would be if anybody here is PCI environment that works with authorize.net about a year year and a half ago I see my QSI laughing about this right now a year year and a half

ago they decided that they're going to put their entire environment behind act my CDN which sounds awesome for availability optimize you know a trusted provider however Akamai has you know just crapload of IP addresses and they don't maintain that list publicly so if you want to ask my website they would they had a frequently asked questions page that said yeah you just really need to open up 443 to the entire Internet because that's what we use now and there there was actually like a specific point in that FAQ that said hey PCI says I have to like whitelist outbound host and I said yeah but this is a business reason to do it because business and so

suddenly our entire path of trust turns from you know multiple layers of security in multiple places that we have checks and balances into a place that we don't know where this traffic is going to end up going we know the hostname but you know there's plenty of there's plenty of DNS base a pass there's plenty of routing basic tasks that could reroute that traffic okay we've got a TLS handshake at the other end how many times and last year year and a half have we seen at we seen keys leaked or certificates issued to untrusted parties you know in the name of big companies these things happen on a fairly regular basis and if you know there's really no way to catch

it at this point if those certificates are compromised conversely if something happens on our server and our services start calling out to untrusted services sure we're allowing outbound access to the entire Internet but you know that's because we kind of need to for this one thing suddenly if somebody goes out and buys a $60 GoDaddy certificate and sets up you know a service on excuse me and sets up a service that's just there to siphoned data our server is going to trust it because guess what they got no signs sign cert from a root CA and we're calling out to them our firewall allowing it because it's report 443 and Akamai says that we need to do that so

what does that do to our to our trusted path the way that we can address that is start to think about how we're going to pull apart our data and pull apart our request path and build trust at additional places inside the path so suddenly let's what's at a DMZ into our mix we're going to start pushing all this stuff through a forward proxy and you know inspect traffic at that point the only issue with that is now our firewall is a blacklist instead of a whitelist we're no longer saying here's a handful of services that you're allowed to go talk to you we're saying here's some stuff we know is bad nobody can go talk to that

no matter what but everything else working oh wow until one of those IPS that we didn't know was bad suddenly turns bad and we're talking to it so that takes and moves our DMZ into this untrusted zone which means that anything on our for word proxy could end up could end up in the same compromised position of sending data of places we don't know to protect our application server using the old defense-in-depth method you know let's just know route the internet from our trusted subnets so no matter what happens if somebody goes in configure their proxy properly their application is going to fail they're going to go back figure out what's wrong go back and

fix it however at this forward proxy we need to do something to make sure that we're connecting to the host that we think we are one thing I should mention there is that in this environment we don't necessarily want to be decrypting our traffic in the forward proxy because again that's that server is allowed to contact any host on the internet on port for work on part 4 4 3 so if anything happens there we have any sort of exploit that is going to start siphoning information off so the next ways that we can add some more trust into that path is again application code that doesn't change that's going to be the exact same

thing we're talking about before knowing where the trusted service exists we still only know the hostname but suddenly we're not going to care anymore because at that forward proxy we can intercept TLS connect and use sni to verify that we're looking at for the correct hostname itself so if we're trying to get to authorize net instead of authorized fake net you know if that ends up getting shut down at the proxy TLS connection fails someone's going to go back look at the code and say hey we put a typo in there we need to correct this or if we're adding a new service let's go get it through the change process to whitelist an additional

hostname because we're only looking at s and I in that DMZ we're not actually decrypting any of the traffic and so our actual are our barrier still exists all the way through outside of our trusted Network and then we're still relying on the certificate handshake on the remote server but what we're doing is we're adding a few more checks and balances along the way to make sure that if any one of them break we're going to have a loss availability instead of a loss of data so what and then the other the other thing I want to add about that is with those configuration changes we can really follow a score development lifecycle because you know proxy configs

are represented in text code is represented in text we can get developers in there and actually make them cognizant of how data flows out of the network and they can be start to integrate those things into their code for requests that security wants to see so can we agree on the definition of DevOps yet it seems like we've thrown a lot of things out there really nobody's going to agree on a definition of DevOps and I feel like anybody working the double off space if you try to define it somebody else is going to stand up and tell you you're full of but there's four main points that we really want to you know convey in this

talk and that is empathy cross-training visibility and agility we want to concentrate on empathizing with our users they want to be able to get to the data that they're paying for our developers want to be able to provide the services that they're supposed to be providing and we want to just start to unite these we want to start to unite these different disciplines across the organization through cross-training so that you know our security teams which are always incredibly strapped for time can start to use the horsepower that's available throughout can you turn them up but as we've been talking through this one concept that really keeps coming up is the that you know DevOps and in that development cycle

we're needing to be so very agile can often breed a lot of data or or compromises to that general security process and what you find is that operational security folks tend to be have this mindset that they're playing this role of having to literally clean up what the development group has done so that they can go ahead and implement the organizational policies or requirements to protect the platform that you're trying to provide and really where we're trying to get to is you know this this nice nirvana state where both of those teams are actively working together and defense-in-depth is not necessarily limited to that infrastructure layer or secure coding practices but we're as a team

developers infrastructure folks operational security folks we have understanding continuum that spans those different understanding areas and we're equally applying those controls and those policies and those best practices in a coherent manner as opposed to one team being the quote safety net and having to you know do a lot of heavy lifting just to ensure that things are the way that you want them to be and one thing to add there is that you know this is not something where security is just trying to avoid picking up the crap development really likes to be able to push their changes out there see the fruits of their labor and never once stay here hey we ship this thing

everybody that's awesome new features live let's go out and grab beers and then have secure to go well guys hold on no no let's let's sit back and actually draw this back a little bit and rethink how you're doing it or even worse having to withdraw something exactly so what our security folks scared of DevOps wise well first of all as InfoSec professionals trained really not to trust anybody right we are there to question the activities that are taking place and and use that that justification to ensure that things are the way that we want them to be conversely DevOps really relies on this culture of trust and agility to accelerate that deployment process and

there's an inherent trust in that agility to provide that so if we don't have a common understanding around what that agility is or what we're trying to achieve with that agility then you've got to disconnect even on the the DevOps side of things so iterate starts to sound like well we didn't really get it right to begin with like maybe if we had sat down and thought about it a little bit longer and a little bit harder we would have had a more complete package or offering to put out and you know the reality is that iteration gives you small changes to make those course corrections as a is to having to hit major forks in the

road not getting it right sounds like a breach from an information security perspective sounds like we didn't think about it before we actually put something out there which breaches are a lot of work and unfortunately our reality of today's environment and when it all comes down to it I do like beer more than work right so what scares me as an information security professional about DevOps is that I'm going to be spending more of my time trying to incorporate those security practices into my environment then doing more pleasant activities and so kind of what I alluded to earlier is that you know there's a flip side to this also and this is you know kind of stuff to talk about the the

empathy portion of DevOps I we mentioned you know DevOps folks tend to be really annoyed with security in the same way that secure deals like they're cleaning up after DevOps you know again and it's the exact same point in select professionals are trained not to trust people and if you're working in a DevOps first organization and you're on a development or an Operations team that is you know saying hey developers you know we really trust you guys to make quality code and we're just going to go ahead and ship it as long as it passes our our automated tests and you know you put a lot of effort into that culture to have security come back and say yeah but

I'm not really sure you guys are doing it right you know just it just feels going to Shady um you know so really being able to ship fast is core to producing good products and having happy development and operations to use you know and there seems to be this this mindset you know out there that if the security folks spend enough time slowing down the process you know really over analyzing everything you know they have this idea that they're going to get it absolutely right the first time and they won't you know there's always going to be iteration there's always going to be you know improvements that can be made and so if the development team sees it

as banne I have to sit here and do a ton of work to get ready for security to finally sign off on this thing only for them to realize oh we missed something a week later and I'm going to go back we do it anyway why don't we just like kind of keep the whole process moving and iterate faster on those changes because we'll be able to shore up our shop a lot quicker if we can get to that point and DevOps guys also like beer a lot more than work it's a pretty common thing again back to the common goals that we have as an organization as long as we can find those you know we can

everybody's working towards the same end there's a brake pedal analogy that came from a guy used to work with he actually still works with them actually but security is like brakes on a cover they slowed you down so you can make it around the track faster without crashing you know it's a great concept if there is no security guess what somebody's going to go out there we're going to leak a bunch of data and then the company is going to go bust because nobody trusts us anymore and our reach insurance tin kicking as we weren't you know we're being totally negligent and let's go find something else to do but there's also a flip side to that so

there's a happy path scenario that this guy is driving instructor I'll let him kind of talk to it actually pretty happy when Matt put this all these car analogies together because I could play off them much easier the fight here with the happy path scenario is that the brakes are there to serve a purpose right slowing down or stopping seems like one of those purposes but really what we're trying to achieve here is using those brakes as a tool so that you can get around fast going around the track perhaps as opposed to just flying off the corner because you were going too quickly however how it usually winds up working is like the old high school

driving instructor car with a brake pedal in the passenger seat where as soon as that person starts to feel like you're going a little bit off course they slam the brakes and stop you and say you know here's we're not going to do that right and what we start to find is that you know really the best learning happens when you're in an empowered environment as opposed to a environment that is focused on slapping your hand for doing something bad and you know really it's better to learn allow somebody to learn those controls in a safe manner than to basically put them on a busy highway and say well there's the gas pedal there's the very

cuddle and there's a steering wheel go for it you're trying to go that direction you'll figure it out and you should get there relatively safe ideally how it should work is in all these systems and balance right my my development my security so that I can pull these these systems together have them work in tandem and be able to go nice and quick again in this case set a new lap record apparently this just happened last week he was just telling me this yesterday I had no idea what it was when I first saw the slides but somebody broke the record at the Nurburgring so that's cool we were doing slides over beers so really

the world of story here is break uses more than binary write breaks don't necessarily have to be on and off and neither is a security implementation it can be analog in terms of applying it differently in different areas but specifically in the areas that you need to provide the protection that you're trying to seek so you know again continuing this conversation through into the DevOps culture is you know we've been talking about empathy you know a lot in you know some of the subtext of these different areas you know as we threw out there there's a common goal at the end of the day to be able to go home sit down drink a beer

and not have to worry about work and I think everybody has that goal to some extent you know some people do like to go home and work but at the same time when you want to turn off you want to turn off and all of us have the goal of producing good products keeping our data safe and being able to feel accomplished in your job on call rotation when something breaks right that pager alert goes off and you get that dread of oh no so we can build those systems and then build out the trust amongst our teams and not only we're providing a higher quality product and environment hopefully we're minimizing that impact of folks lives and can be more

actionable when issues take place because we have various checks and balances we can go back to those known break points there's no circuit breaker typed approaches and be able to systematically analyze a problem and come to a quick resolution so we touched on cross-training quite a bit earlier you know really empower the different folks in the organization to back up what security is trying to do security should never sit there in the corner and just dictate it's it's something that if again you probably don't have time to do that anyway you have you know too many meetings community policies to review too many you know incidents to respond to you know that kind of thing

cross-training your organization getting recruits in your development and operations team to really be your boots on the ground and your eyes and ears because these guys once they know what to look for and you know kind of key concern areas they will bring more to you than you would think automate as much as you can you know there's there's this kind of concept that you know that I've heard in you know four different security professionals that yeah well the coding portion of it I leave that to the developers however if you automate your job people will use it developers love automating things if you automate security they will use it and be visible you know everybody nobody likes to break

the build you know in a staging environment so if your security controls and tests are visible to the entire organization people are going to be more cognizant of what they're doing and their impact on the security posture of the organization as a whole so really as security folks why should we embrace DevOps it's just trusting putting that trust in people and it feels a lot better when you have mutual trust within your teams as opposed to teams butting heads or these silos of expertise where folks retreat to their technical corners and are unable to have you know across platform or cross environment conversations and things turn into shouting matches or arguments so building that trust amongst that

organization just generally feels better right it's not nearly as contentious folks the devs who trust the security team are more likely to support those security goals to Matt's point about you know it would be great to always have a dedicated security team of folks that are focusing on on your needs but you know you have to recruit and actively engage other folks outside of that core security team because the fact of the matter is they have some security responsibilities as well and if we can do that in a cohesive manner they're likely going to support those goals especially Lincoln tool and automate those those those needs so that it's not a manual effort every time I need to

make a particular change or go through a particular review process when we're thinking security first improvements are part of your daily work somebody is more likely to raise our hand and say hey this doesn't quite look right maybe we should take a second look at this or or review this approach as opposed to that's the other change problem let's just kick it over the fence and we'll see if they figure it out or if they have any particular items that they want to address and really what this all boils down to is reducing that that security risk and that that surface of your environment and and trying to reduce any sort of backlog of trying to

secure that environment trying to bring those security practices to be more agile and support that agile development as opposed to just being that that hard break trying to slow slow down that process so one thing we commonly hear of is well I've got special requirements so I need a dedicated environment or I need a different set of services to meet my needs and really one of the challenges that we start to run into with duplicating these environments is that well now I've got that whole set of software and service tax in another location that have same set of policy quirements same set of protection requirements that I did in my existing environment but now I have two of them

three of them four of them whatever so how you can adjust to supporting those multiple environments you can try and scale up your staff or you can try and scale out your tool sets to help you be more effective and really don't rent organizational structure of your downfall right ultimately duplicate cost to catch up in time I think that moved to beneficial move to virtualization shocked a lot of people's data center operations finding that they had duplicate servers in many cases because they were managed by a different group and a different administrator and nobody really ever talked to those folks you start to see this in the cloud environment as well when all of a sudden

you get a very interesting bill and find out that somebody stood up our huge server and let it run for an entire month and it basically was idle but you were paying for those resources the entire time so again trust your people align your goals and when special things come up work together to identify the best methods for for managing them yeah and so you know one of the key things that we just want to get out of this is security was never really only about governments you know despite the fact that it was implemented that way in a lot of organizations due to lack of resources lack of time or just the way that security really came about in you

know from an incident response first mentality and you know kind of that visceral reaction to go back in and keep that from happening again people would go alright I'm going to sit down I'm going to cope with a set of rules to make sure that this doesn't happen again is really you know not the way that's going to going to work going forward you know to that effect you want make sure you're effecting change in an organization and not implementing failure you know I see this a lot but when I was consulting and I eventually found myself going down this path a lot of you know quite a few times in my current job and I was trying to find a

way to like bring myself back from that cusp of saying you know I'm going to put a policy out there like I'm so frustrated that X is happening and I'm just going to create a policy that says you can't do this anymore but the problem is if you create a policy that breaks the workflow on day one all you've done is implement failure and you're not allowing people to actually succeed so in any time when you're seeing when you see yourself reading a policy or some new rule to address an issue sit back and think about what can you do to improve the workflow can you create new tool sets to actually help people be more secure in

their jobs instead of just telling them that they're failing at being secure so let's eat that elephant one bite at a time right we talked about not being able to effectively cut sit down and do a start to finish deployment without potentially running into gotchas take them in little bites right use that iterative process to your advantage to check those control points and ensure that you're still on target with where you're trying to go and celebrate those small successes right there will be failures along the way but the successes feel a lot better and they certainly feel better than one massive failure because some particular requirement or goal wasn't that and remember your users are very very useful you know if you're

there to help them the best way that you can help you know your users be end users developers operations folks even you know your sales and product folks make sure that they're there to understand you know what your security goals are and make sure your security goals aligned with theirs because then they will help you through the process you know remember tools are a method our method to operate they're not there just to support operations you know like I said earlier build yourself tool sets give your users the way you know the means that they need to get the job done and then they will build really cool stuff again this is Kasim slide with the car

stuff this is basically the same point I just really like the idea of the monkey trying to create a tool and finally give Krim give credit where credit's due the thought here is that if you're spending your time pulling those tools together and training folks on the core goals and the technology of what you're trying to achieve as opposed to teaching them the tool you in theory and a gorilla stack of matches and it will figure out how to light the match as opposed to sitting it in a week-long boot camp and instructing it what the purpose you know when it matches what a match that is in how you strike so so much of tooling today seems

to be very based around that product or the platform that's being deployed and there there was a discussion in the keynote where that changed from a Cisco platform to a Palo Alto platform potentially it throws the the team dynamic into a bit of an uproar because we're not trained on that particular platform well but you know firewalls you know that that core technology take that that core knowledge set and apply it to a different platform and use those tools to to get you where you need to be and I think we're going to kind of wrap up on that note at this point just because we got a little bit late start you know is

there any questions I know we covered a whole lot of ground there you know and there's some pretty you know soft topics that you know can can span a bit and if you don't ask here will be around a conference you know all day tomorrow so you know feel free to stop all this in the hallway but is there anything you guys want to talk about you know in the last minute or two that we got here yeah so one thing that we did in our environment is you know our our software development life cycle lives around the github pull request and we have an open-source toolkit called octo polo that's used to create new branches

create pull requests and really follow our work our workflow all the way from you know the first change in one PR through deployment we've implemented a change change template for our environment so that when our developers create pull requests you know there's already you know a set of criteria that they need to follow to get that signed off by QA security whoever we need to get that push through on to the next phase and deployed out to production the reason we created those templates is that we also have certain applications that exist for instance on our PCI environment so we just use a different template for that and then if applications move we change functionality maybe we take something

out of our you know non CDE move it into our CD we change the template accordingly and then the next time a developer goes to work on that they see a difference and they say oh alright here's some additional checks and balances that I need to I need to follow on this on this change and suddenly we've recruited somebody who you know was there really just trying to you know write some code and implement a new feature we've recruited them onto the security team to make sure that they're following all the criteria without ever having to hold a training or you know take time out of people's day in terms of low-hanging fruit look for those

repeatable tasks for those components in your environment that you touch a lot right if you're not deploying course switches or firewalls in your environment very regularly then trying to automate that process probably isn't going to get you as much as much value as automating an application server spin up and deploy Google well thank you all for your time [Music] [Applause]