BSidesSLC 2015 -- Pragmatic Cloud Security -- Joshua Danielson

all right we'll go ahead and get started here uh thank you for the introduction um a little about myself before we go ahead and dive in head first uh josh danielson i'm a senior security manager within axeway actually has a greater organization focus on b2b uh edi type of software as cloud has kind of exploded um we've opened up our new cloud computing division probably about seven years ago and i've been heading up all things security uh since then so i've been with them for about four years and i was probably the first security dedicated hire since uh since things got started it's been an interesting challenge because we've taken a traditional software company and tried porting him

over to be a true cloud company so it's been a very challenging interesting process we've been going through um i got my started i got my start in dod uh moved to the the academic space did some stuff around there move the ax way and actually i'll be moving over to head all things security out of capital one in their new cloud computing division at the end of uh uh this month so things keep on going as we all know feel free to follow me on twitter as well keep pretty active on there what's the first thing people think of when they think of cloud people think secure right well yeah not so much um despite all the uh all the newer

technologies and all the advancements that have been made in cloud computing it's still not a trusted platform by and large but we all know saying no is a security failure we've all been to experiences in the hall of meetings before that all the infrastructure and all the application all the developer folks will get in one room and by some reason all security people don't get invited i know i haven't i think a lot of you actually have as well there's a very good reason security people hold things up they don't see value-add a lot of times and i think that's one in our case is where we have to do a better job of communicating

what we can actually do giving them more options and actually showing true value add to the organization outsourcing in general one of the first things we uh we discuss in information security whenever we talk about cloud is you can't outsource your critical functions that's insane who they control the data you have no control over it well we do these things every single day right after all we don't have uh money understood in either mattress and despite when no one trusts the big banks we trust the overall governance and the overall structure in which these uh these components work under um cloud computing is still continuing to mature so i'm not way in any way comparing it

to something much more mature as far as the banking industry we have the fdic and all these other kind of checks and balances but it is getting there this is one of my favorite quotes from uh from the uh a cio the formerly with the us government um basically he sums it up in one long sentence amazon web amazon web services google compute all these large platform providers they have experts that can do these things and can focus on these security tasks better than any uh any one of our groups more than likely can so it's kind of more important thing and that's why some of these organizations such as amazon web services can have

true uh enterprise type of platforms uh that we can piggyback off of so i'm gonna preface the rest of the rest of everything i'm saying uh within x-way you we do use amazon web services i'm not going to say they're the best cloud provider or they're the only one if if you're into gartner and all that other kind of marketing stuff they're at the top right corner which is a good thing um so just kind of one more note i don't work with amazon we don't have any part we don't have any other partnership besides we use them i'm not getting paid for them but rest of my conversation will be uh will be a lot of focus around amazon web

services technologies the shared responsibility model this is probably one of the few things that uh throws a lot of security especially a lot of seasoned people off i regularly get into discussions with auditors and a lot of potential customers they bring their security folks in and then we go through traditional audit the the hoops and whatnot and one thing they actually don't understand is what you're responsible for as a customer what the provider is responsible as a provider and what our customers are responsible as a customer so you can take a nice look right here this is one of the things i stole from amazon web services they have a lot of documentation a lot

of white papers some really good stuff if you're really interested in learning some more it's freely available there's hundreds of pages of this stuff you can dive in head first and get a better idea of their security programs and recommendations so we talk about the blue stack over here at the top of what they recommend you do to do as well so this is kind of important distinction we're going to kind of strip away some of the layers of cloud and explain why this is actually more important when we strip away the layers of the cloud stack you can see that kind of blue area amazon web services functions at the very bottom and infrastructure as

a service as you go up you see platform and software as a service these are the areas that people become a lot less comfortable with i'll give you one specific example i know uh work with auditors quite a bit and customers and pci and others of the like they come in and say do you do logging for uh failed log ons well we do at the operating system level at the infrastructure as a platform infrastructure as a service level so we check the box and keep going unbeknownst to them they forgot to ask about any other stuff and these are seasoned people have been doing this stuff for a very very long time they just don't know

so knowing knowing your shared responsibility knowing which uh what cloud solution you're looking at who you're looking to partner with and what you as a as a customer are looking to get into it's really critically important you understand what service you're looking at so know where you're at in the cloud but if you suck now you'll still be pleasantly surprised at the lack of change when you move to cloud you're still responsible for doing the same stuff and although amazon web services may take care of the data center and they may take care of a lot of physical stuff and they may take care of some of the other things uh basically the the the mason dixon line the is

right below the hypervisor which is at the operating system level so you're responsible for within amazon web services for doing the operating the host excuse me the guest operating system so uh red hat windows whatever it may be and they run some bastardized version of zen that they run underneath the end are responsible for doing that same thing on the network side of the house they run all the network hardware you're responsible for doing all stuff right above that so you can still have the capability to network acls and whatnot so instead what we're left with left with is not inherently a secure or unsecured cloud but a platform that can be secure for certain use cases when

used appropriately so a quick rundown of agenda items uh i'm going to go over cloud principles understanding what a cloud actually looks like there's been a lot of misconception as far as what the cloud actually is i have to constantly report back to my boss and tell them all the crap that's coming out that is just complete garbage so all the stuff that's coming out recently around the business side of the house is that cloud is not cheaper in fact it's actually more expensive well when you do this thing called cloud washing you take your traditional software you plop it on an ec2 instance and you call yourself cloud well yeah that's not actually cloud and

you're actually spending more money we'll go into a few more use cases how you can actually take uh take advantage of the cloud principles and you can have a really true cloud solution so that's kind of an important distinction we talk about cloud 1.0 and cloud 2.0 you will never do patching again you will never do vulnerability scanning again people will never ever log in or ssh rdp or otherwise directly into a box i'm talking about admins root otherwise i'm not just talking about users i'm talking about no one ssh 22 is off uh security and devops will kind of talk about how everything kind of merges and blends into a cloud uh dev secops model

and then how do you actually manage this uh scalability when you look at some of the true unicorns in the industry right now if you want to take netflix as an example uh when house of cards came out a few a few weeks ago uh it was rumored that they actually uh they were using one third of the entire internet's bandwidth at that single weekend that's pretty freaking insane how do you manage all these systems that are coming up and down so you have a friday night comes up everyone turns on their house of cards saturday morning around 1am everyone spins it back down they have to stand up uh hundreds of systems thousands of systems that

actually could uh facilitate this so we'll talk about how you actually manage that type of elasticity in these environments and then a little peek about what's coming next

so cloud principles the first one is service based so this is this is commoditization of it we all know what this is this is you plug in your uh your lamp into the wall and electricity kind of works you plug this in you plug in uh your server and then it kind of starts working and you start up your server and starts working automatically so that's kind of service based we all know what that one looks like scalable and elastic so this is one of the things a lot of security people kind of fight with initially and say we can't secure this systems are constantly coming up and coming down how do you actually secure this stuff well i'll

tell you this is actually one of the things that you actually use to our advantage when amazon web services was notified and everything came out about shell shock a few months ago they're actually able to patch shell shock across their entire load balancers within three days when you talk about enterprise and doing any type of patching patching anything in 3d is absolutely ridiculous so this is but so you inherently look at it what's something that looks like a disadvantage and you turn it into an advantage so using elasticity using scalability to your advantage shared to some degree there's going to be something that is always shared i talk about the shared responsibility model earlier you have to understand

where you are in that stack and kind of what you actually have to provide for your customers or what you have to actually manage on your side of the house of things metered by use this just basically means you get a bill at the end of the month kind of like you get your electric or electricity bill or whatever it may be and then you get only pay for what you use and you don't pay for what you don't uh when you're talking to your business people out there i always tell my boss this is cafex versus opex and he gets all excited like we only pay for what we use for and whatnot so the business

people kind of love that term when you talk about meter by use uh use of inner technologies this is really leveraging open apis this is using mobile and what not to take advantage of some of these type of uh characteristics so we'll dive in cloud 1.0 versus cloud 2.0 cloud 1.0 this is the original model t it's good for what it was but it will no longer suffice what does this actually look like so i was giving the example earlier about cloud wash this is so uh the organizations right now that are just taking traditional piece of software they spin up an ec2 instance on amazon web services and they install themselves there and they call

themselves cloud or they come up with a hardened appliance and they uh install into the aws marketplace and say we're in the cloud now yeah not really don't tell some of these vendors out there because i don't want to blow their mind um sev what would several releases annually we know what this looks like whenever there's an update on a database when there's an update of major operating systems and a major update any type of software it's all hands on deck it's a pain in the ass for that whole week everyone's on call you know it's going to be a pain we all know what this looks like it's these huge massive updates that we no one knows what's going to

happen but you know something bad is going to happen and cloud washed i mean this is kind of a marketing term that is coming thankfully starting to become a little bit more used and kind of calling out these people are trying to be cloud but really actually aren't so that's what cloud isn't what is cloud then so icloud say cloud 1.0 and cloud 2.0 cloud 1 is really just porting your stuff over cloud 2.0 is really building your software on top of the platform to take advantage of some of the things that i was describing earlier so this is building your software on top of the provider within amazon web services they have different types of

storage you can actually use so you can use uh local volumes called ebs volumes you can use something that functions a little bit more of a slower sand it's called s3 and they have a really long one really for archival purposes called the glacier if you can kind of tear out your stuff so we actually tier our logs for example you can actually better take advantage of that but you have to be able to custom create your applications on top of this infrastructure has code um as cloud becomes much more popular there will be less and less administrators and there'll be much more type of developer types i could say right now i am not a

developer but i have to spend a lot more time learning json xml and whatnot because guess what my job as administrator and architect isn't going to exist in the same way it does in about five ten years or so you have to learn the code uh from what comes the rule sets when it comes to embedding hardening baselines and whatnot i'm gonna get into the cloud factory of how you embed these things in by default so you no longer have to do patching you no longer have to do vulnerability scanning stream uh streamlined agile processes this is taking advantage of those types of things so you don't have to do patching and whatnot so i don't want to

steal too much of the thunder coming down but i just want to point that out and what does this look like several leases a day so i know i said the example earlier with amazon web services of how they actually um patch shell shock within a few days some of these organizations are known for doing a few hundred updates in a single day oppose a few every single year if you learn to take advantage of those it'll really change the way you can actually manage your environments so what are a few traits of a successful cloud security program some of this we all know is kind of redundant so i'm not going to dive in that know what

framework you're working with have some reference architectures i know i mentioned a little bit of apis um one thing i will kind of touch on is some of the logging some organizations that are truly cloud and want to really take advantage of the average lifetime for a server is 22 days that's kind of around the benchmark that's around the threshold so no longer are servers these uh uh cuddly pets that you keep forever and eventually they die in six or seven years they're things that actually are constantly rotating on a regular basis so how do you actually monitor those how do you actually log those how do you go back when you're trying to find a host

for a system you're trying to look at and it no longer exists because it got it terminated three months ago i'll talk about some of those things coming up ci management so for any people have to do all this type of process or itil and all these other types of terms we all know we have to comply with uh configuration items no longer will you ever have to do change management around your servers fundamentally that completely changes because your is always coming up and down instead your cis become uh your your cloud permission templates your docker images your openstack scripts so that will change and you will no longer have to have a change controller on your

servers ssh is off rdp is off no one's logging into any of these boxes they have no reason to why does someone actually log into a box uh for linux box to do something they go in to fix it uh update a config update some type of parameter you go back and you change that in your cloud formation template and you never have to log into the box there's no reason to you shoot the box in the head you spin up a new one in its place and it gets stood up and it gets remedied in a few instances that's how you actually do things and how that's how things will change in that way

um ci management for the first time ever i came from government i was doing uh auditing kind of analysts work there for a while and i remember one of the biggest things i actually had there's always some jerk in the corner who decided that he didn't want to wait for it so he wanted to put in his noon his own linksys router and whatnot and he had his whole thing going because he had his other personal laptop and whatnot so for the first time ever in cloud you actually know what you actually have in front of you so this is a screenshot that we have in a console you know exactly what's in front of you

that's a luxury i know i never had in previous environments you know exactly who's connecting what within your environment and we kind of take advantage of this with our configuration management tool we use puppet other people use chef and what not puppet worked in our environment we bounce the configuration within puppet against this in real time so anytime anyone stands something else and it's not enrolled within puppet which controls all of our uh logging all of our hardening and whatnot we get a notification in real time if something isn't within scope or anything is being managed by our security teams

so devops what is devops um i'll be honest i've been working in cloud for dedicated for about three years now i still don't really actually know i was doing some item writing for isc square coming out with a new cloud security certification and they had all these people from nisk and iso all arguing with each other and they were arguing if the c in cloud should be uh capitalized or lowercase i was like good god these are supposed to be the industry experts creating all the baselines and all the revisions you see from this yeah that's the state of the industry i'm sorry to say so when all these people and my boss and all these other

people start using these terms in cloud and whatnot start misusing them no one can argue if the if the sky is blue because we all know it is and everyone says that is isn't we know they're crazy if someone says cloud isn't cloud and we're misusing things and kind of you misusing these terms it's because it's not defined yet even people working on it really don't have a good enough idea yet to come up with a single set of terms and whatnot i've seen these people arguing it doesn't make you feel good but devops it's really just the it's a philosophy of how do you actually connect dev uh development with operations um we've we've had a few uh

successful experimentations in this it's really just making everyone own the whole solutions instead of one group pointing for fingers at the other and vice versa and whatnot is having them kind of collaborate much more of a collaborative type of culture and some things we've actually done to make this much more successful we've made some of our developers be on call and fix operational issues we've taken some of our system administrators and we poured them on the uh development side of the house and made them write some code so having them have dual ownership across the board is really kind of critical to building that culture of devops and making sure that you actually build solutions that are much more robust much

more agile

so i talked about this before we're really beginning to streamline and really uh assembly line driven asset management and we're all used to this type of stuff as far as provisioning a new system and it comes down to line and six weeks from now you get it racked and stacked you get it connected and then you can kind of use it and we're like that's that's the assembly line used well then everyone kind of goes in and they custom i'm talking about a hundred percent managed and customized from the get-go so like i said before instead of actually managing servers we don't manage any servers anymore we manage docker images we manage uh if you use

openstack we don't use openstack house but that's another one and then also cloud formation templates those are the things you manage those are the things you need to scrutinize and make sure that people aren't going and messing with your servers no one's logging into them who cares they're constantly coming up and down anyways those are no longer things you have to scrutinize and worry about because you know from the core you know from the very start that they're being being secured by default you begin to rely on tagging logging metadata these are what your configuration items are there are no more snowflakes no longer are these kind of one-off servers that are kind of floating in the corner by themselves and

everyone oh yeah that's that one server that there's a one critical function so don't touch it because everything's going to break this has a heavy reliance on apis this is where we've worked with a lot of vendors so far and we really haven't seen a lot of integration quite yet we've had to kind of build our own um so allowing all these kind of configuration uh uh to be synchronized within we use splunk splunk's probably one of our favorite tools that's probably the only one i'll name because it's one of the few that does a really really good job at doing it but the gaming much more relying on these apis will really be critical to your

success and managing all these systems across the board so i kind of poked at it earlier but we're all used to server hugging everyone gets a box and he likes to cuddle and like to keep it forever they like to name it and they have their ip and they have their host name and these things are really meaningful um some people actually rely on it we also use remember the people who can spit them out and actually remember them off the top of their head and knows every single host within their environment or at least has a good idea of them um they're static systems they never really change they stand up for they stay up

for a few years and they're used to staying there forever we never really moved them they were going to shift them they stay the same they are snowflakes they're kind of sora one-offs each server is different than the other and otherwise we're looking at now is service orientated solutions not server orientated solutions so this is where ips and names become meaningless i don't even i can tell you the ranges but i have absolutely no clue what the bot what our critical boxes and ndip's are and almost no one else in our environment either can uh these systems are ephemeral like i said uh servers are now lasting several days not several years anymore these servers become mechanized by code

so and i'm not a developer but i had to learn a lot more of the development json xml and otherwise to begin integrating all these components together you have to also be able to shoot the other node in the head you have to be able at any moment to get rid of any of your systems any of your servers and know that they can be resilient enough to come back into place um i know i use netflix as a lot of example but they have a tool called a security monkey actually chaos monkey and it goes around shutting off shutting off random servers throughout the entire environment the whole goal is that if you're resilient at any given time when

something bad actually does happen you'll be ready for it because you've already been ready for it so managing scalability i'll give kind of a simplified uh version of this but i think it's still pretty useful uh we call it a blue green deployment we've a few other people actually using this environment too i haven't seen it cut too much fire yet but you can find it if you do a few searches for it basically it's this you have a running stack you have your green stack and it's running all the stuff you currently need so if this is patch management this may be a dated version of openssl a new vulnerability comes out you have to create a new one so you

actually update a new a new stack and all this stuff is being done in real time this is not a whole bunch of server and whole bunch of time this is all being done in real time in cloud formation templates it takes several minutes to mimic a current environment of a few hundred servers this isn't a whole manual process that has to take a scrutiny week or so to get done so you stand up a new environment with the updated version open ssl you swap this through your load bouncers that you have in front that's kind of a prong kind of tried anything looking thing in front and you swap it to the new one and keep in mind your your uh

static content whatever it may be you can see the backend database on the other side in the back end that stays the same but your servers that are just doing the processing those are ephemeral and then you get rid of the old dated version

so kind of quickly touching on this as far as uh security management i know i talked about vulnerability management patch management and whatnot um monitoring a little bit as well but this is how things will begin to change you will never have to patch your systems again because there's no reason to you update the uh base image in amazon they call it a ami an amazon machine image you upload that base image you shoot any type of servers you have in the head and you replace them you backfill them all this happens in a few minutes so what we do we actually have a cloud factory so i want to give one specific example in here um of when we stand up a

new customer environment we have our base ami we have multiple products multiple types of solutions but we have a base ami amazon machine image we have and it rolls down to factory line and in there we're able to embed all of our security baseline we're able to ensure that splunk forwarder is actually monitoring to our indexers properly we're able to ensure we're building out documentation for our customers that need that some have uh some specific requirements that have everything that need to be documented some some governments and whatnot still want to ensure everything's named everything's proper we'll give it to them we have this all created this is already scripted uh as well we have performance testing

and integration testing across the board as well from there we can do our install scripts for some of our customized stuff and at the very end you have a lot of stuff so i'm not just talking about the image i'm not just talking about single server i'm talking about the entire environment i'm saying network acl security groups load balancers all these types of things that you have to create an entire solution on not just a server or an application or a database or whatever this is everything that comes into one single piece if i'm zooming into this that first slide this is the cloud the cloud bakery so you have the base ami it rolls down

the factory bakery floor it turns in these two instances you're able to insert any type of repositories anything that's actually custom for that spin a single solution that then at that point you're able to install the any type of custom applications in our case we have our x-way products we'll be able to install whatever that version whatever that type of software our customer need in that environment and it rolls down and it gets borne into an uh ec2 instance

and then within the cloud formation templates this fits into the bigger picture as i was describing before where not only you have everything sitting in its own subnets and its own dmz its own backend its own frontend type of systems you have its own virtual private cloud as well so each new customer that we provision has its own virtual private cloud and this all comes together you can see the left and right we have different production systems this can be uh uh this can be your dr this can be whatever you want it to be but we want to show together and this is all automated this is all comes together this is not manual

type of uh hands on the keyboard type of work and i touched on this before the best way to avoid failure is to fail constantly and this really i know i talked about security monkey before but this is the ultimate test of resiliency i'm not sure that many people here can say with confidence if you go out and unplug one of your servers that the service is still going to be up and running and you're not going to have too many complaints these are types of things that you should be able to do on a regular basis when you have a true cloud environment because why not if something fails and you know it fails you can know

what that looks like you can script it to automatically become resilient and self-healing

netflix's having gone uh they're really one of the huge proponents in this area and they've open-sourced a lot of stuff they come out with their whole simian army they call it uh probably the biggest and boldest thing they came out with is chaos gorilla so if you're familiar with amazon web services they have certain uh availability zones within each region and they have multiple regions throughout the entire globe so in the united states they have one in north virginia they have another one in california somewhere and they have another one for government up in oregon i think it is so within a single availability zone though you can have a chaos grill shut down an entire data

center for whatever you're running right not everyone else's stuff but you can have to shut down absolutely everything that's running for you this came in really useful a few years ago when uh amazon actually did have this happen and they had an entire availability to get zone go down and every single customer went down in that availability zone except for you guessed it netflix we actually was testing this constantly having redundancy and constantly testing their environment for these types of things a few things for embracing the devops philosophy i encourage everyone to read these aren't security books per se but they're really critical in understanding how you can better manage these types of dynamic environments probably the one i

would say is kind of security is uh anti-fragile from the same talib he describes the property of anti-fragility or there's no really word for being anti-fragile we use words such as robust resilient to describe things that you're resistant to breaking so if i have a steel a steel pot and i drop it it may be resistant but it isn't anti-fragile it doesn't become stronger because i did do that well that's the types of environments we're trying to build that do become stronger because we actually have it undergo these types of stress stress tests and what's next homomorphic encryption this is probably one of the newer things that i think will really start to get a lot of things

taken off in the cloud for a lot of businesses in a lot of those uh countries over in over in europe um i have to deal with these things on a regular basis axel is actually a french-based company and our largest revenue gender generator is actually based out of germany so we have a lot of privacy concerns from a lot of potential customers all the time um even before the snowden stuff about it two years ago there was always patriot act patriotic patriot act i mean all the time we ought to hear about the patriot act and how horrible we were as americans and whatnot and i guess it's kind of true i don't know

um but anyways with the advent of homo homomorphic encryption um these are the types of things that will be and facilitate a lot more trust for cloud so alice hands bob a briefcase alice wants bob count how much money is in the briefcase and bob naturally says give me the key so i can open them count the money alice says no i want you to count it by just from the outside without actually looking at it that's kind of homomorphic encryption be able to process data just looking at the encrypted bits this is no longer theory anymore there's been a few more updates on research i've seen recently that this is already actually happening it's

nowhere near efficient but it's already actually happening so the biggest hurdle's already been done and digestible big data i got a friend of mine who actually runs uh the security team over there at uh microsoft and azure um he doesn't hold too much against me because we run amazon but um one of the things he shared with one of the coolest things they're doing over there within their uh security operation center their sock was actually making big data much more digestible as you're having so much more data than ever before how do you actually respond how do you actually alert to these types of things so this is an actual screenshot of one of the things that they're one of

the panels they actually have in your sock you can actually see and hear events as they're happening in real time so you can see he's kind of the analyst right there is kind of zooming into a subsection within a greater spectrum and you can actually see and hear these types of events happening in real time so with that thank you and that's everything else i'm not sure if there's any questions but uh

no questions all right feel free to come up i'll be around oh no you did not so one of the pieces that uh for actually i had the slide in here too i think last time i did this presentation i ran over a bit more let's talk let's catch up after this because that's probably one of the ones i actually have a dedicated slide for this i ran over last time i actually was giving this talk at cactus con last week and so they kind of scolded me so let's catch up after this and i think that's a very good question of logging is one of the bigger things that we have to do so

anything else

so for instance

so because we actually don't have everything we have everything located initially on the local servers which come up and down we have every we use splunk so we use splunk as our sim all that data is actually still kept in archive within splunk and we have that tear down several type of uh tiers so for hipaa we have seven years for example for those types of customers and we have that in s3 bucket so we still are able to maintain those types of and address those compliance regimes and whatnot but we do it in a little bit of a different fashion so when they come back and look at a server we actually can't go to a specific server that was

actually looking at but we can go back to that overall service and we can show them every single server that was potentially within scope of that which always existed and we can show them the assurance controls we have ensuring that all systems that you just because it doesn't exist anymore that we have change control around the uh templates we have change control and all the logs there within cloud share we can go back and show them this is when the server was stood up this is when the server was uh stopped if they want to go through and look through a few hundred servers have at it i felt working with uh we worked with kpmg for our sock one and

iso 2001 stuff when we worked with a few other organizations uh in the healthcare space and also in the financial sector as soon as you storm the assurance controls and you gotta really get them the warm fuzzies as far as what you're actually doing that actually works well they feel a lot better at it and you can actually just go to splunk and you can show them this is what we've got oh what server do you want to look into we can show you all the servers that were terminated within the past year past month what do you want to see all the servers that were started in the past year past month we can show them all

that and we can show them a few moments not even minutes we can show them like a few seconds it's an easy drop down so once they kind of see we actually know what we're doing and you actually have a good handle of it it usually gives people the assurance they need and all the data is there it's all within our splunk and indexers so it's not actually hiding it's not actually getting terminated the servers are because they're really ephemeral that's just the processing piece but the log data is still there anything else all right thank you