BSidesAugusta 2018 - Martin Holste - An Anatomy Of A Cloud Hack

[Music]

or Indian on five Gnosticism in general I'd say don't do it pick one Wittering on

but generally that guy goes they don't want to do that because they're not really experts in any one cloud so they went ahead and when I go to all clouds now that Walden the security group

Mitchell so one of the the things that we see our developers not knowing that something is a credential so if you've used like as your connection strings they're just kind of long not really opaque string but they don't necessarily look like a key or an API key right away but they completely are they contain the endpoint and all the connection information to connect to it and so that's a real issue and know what you're looking at you might accidentally publish that somewhere you thought it was private it's Pub whatever that's that part as old news would is is still happening though as people are logging in and if you think about I was talking about that top of

the pyramid the ant the analytics that's your one shot at finding this is if that credential sitting out there somewhere on you know it might even be sort of a like a partner like you shared a website with a part not even really public if they find that and they use the credentials and they log in you're now really just looking for a credential theft alright so how are we going to mitigate this so the email is pretty obvious right you try to prevent phishing block any malicious doc can and this is the cloud side by the way the endpoint it means same stuff right trying to just protect the endpoint and all the involved there from the network

perspective I've talked to a lot of companies that are using VDI now so beyond Citrix and that's happening quite a bit and what I'm shocked by are most companies aren't doing things like network security monitoring in front of a VDI so they have a lot less of visibility than they would have they were running that on Prem they just it's not that you can't do it they just basically forget do it and you find places that are running VDI without an endpoint agent at all so they don't have any kind of you know basic protections they're just assumed well if anything happens we'll just wipe the box ok that is a lot easier in VDI and you can still

wipe the box but the credential stolen you can't unwiped the credential not without knowing about it so networks still very relevant but the real the real meat here is the detecting the credential theft and any compromised assets that are already out there and that's generally gonna be through analytics all right so let's look at a couple of specific scenario this means still relevant I love this meme so much but it's kind of I think it kind of wearing it out here most of my internal stuff has this in it got a stop all right so let's look at and both of these are our real-world scenarios but they're sort of an amalgamation of course we don't share

direct customer investigation stuff it's that cool but that we see these are very general incidents and you'll see that they're very applicable to pretty much any scenario so I'm going to try to give you as much information as I can to be helpful but I'm just not too much all right so the first thing there's not an accident that there's day one day two and day three here listed that's only 48 hours so we're seeing a really really short dwell time our last M trends showed closer to like a 90 day dwell time which wasn't really that much different than the year before for your basic compromises for cloud the stuff moves really fast and a large part

of that is that the attacker can leverage all the same things that an admin can because they're basically a rogue admin at that point so think about this if you're red teaming cloud you can set up everything exactly how it's going to be in a customer environment in your own cloud account and know that it's going to be 100% the same to some level you might have to do some discovery to say okay which you know enumerate all subnets well let's think about that if you're on print you do an nmap or something like that to map out the network if you're lucky you can hack into ad that'll tell you you know where everything is but if you have API access

to a cloud you just say where is everything and it gives it to you in a programmatic fashion and suddenly what should have taken days now it took seconds so that's a big reason why attackers can move so much more quickly on cloud and homogeneity of the environments is also a big part of this because once they get this down and attack one place can turn around and reuse almost all of that to any other victim because there is so much that's similar between the two all right so specifically in our scenario here using those methods we talked about earlier we have stolen credentials that come in they perform recon that's on day one on day two they're going to dump the

data through API calls and set up the environment for exfil and this generally speaking the the reason why this doesn't happen all in a single day is that advanced adversaries are switching the keyboard out so they have people specifically that know how to get in in the first place they know how to steal the chills and then they have their cloud specialist they'll come in and say okay I'll take it from here I know this individual cloud provider it's this real bizarro world where you have rogue administrators that have specialties and that's honestly the thing that's slowing the attackers down the most is their resourcing right to figure out who the best person at that time who's on shift

if it's nation state all those kinds of things or you know even if they're just you know crime wear style they're still going to have to have their buddy come in and help them out and something because they're not written everything and so that that the keyboard changing hands is usually where you're gonna see the most delay in what's going on otherwise would be one giant long script and the whole thing would be done in a minute and you know hopefully we're not there too soon because there are still a number of manual things that have to happen but that could you know we could get to that point so there's some specific API calls that will happen to

dump and we've seen a number of these the big one that we see that's kind of interesting because it's nothing particularly malicious but very rare are snapshots so ask yourself how much how many controls you have around like a really important database maybe has financial information in it you'd have really strict Network pulls over it you'd have logins that are really specific for it that are hard to get to you don't have that if you boot up a box and mount the snapshot from that other box on this other box now suddenly it's your box and you don't worry about the network and it's your box you have to worry about the login all those things

go away so cloud lets you sidestep a lot of those things and if you're not paying attention then that can happen now this isn't to say that all cloud is just inherently insecure because their permissions that go with those snapshots right if you do it correctly that no one's supposed to have permission to log it in the first place no one's supposed to have permission to mount that snapshot so it's this little problem but if you're not trained on this kind of stuff than that this will lead to so they mount the snapshot and the minutes later they're dumping all the data all right so that sounds pretty bad what are our opportunities for detection here

it's worth noting here that none of these are gonna be you know oh this is a giant red flag if you see this then I'm over we found them all these are what we call weak signals these are you know if you're hunting these might be good but generally speaking these are not worth an alert to the sock it's only in combination of all these things that you're gonna have an alert so login from a new IP address may not be a big deal maybe somebody move maybe they're at Starbucks you don't you know depending on your policy that maybe you know complete no news at all and you user agent same thing I mean that's the you

know any new browser version will change the user agent right so that happens frequently not a lot there but if you can combine those it starts to get a little bit interesting but we're certainly not bothering the stock yet so maybe between day two and day three this is very enough to go on but again when we look at the timeframe here you're expecting your stock to be able to correlate stuff across the 24 hour time period and now you're just looking at even more weak signals so you have you know API calls that don't happen very often well define very often right totally depends it requires a lot of domain knowledge again can you do this

domain knowledge across every cloud provider probably not I encourage you to focus on one cloud provider and get good at it new kind of at API activity so understanding exactly what calls are and then down to the asset level this requires more things like flow logs between them all right so there's a lot of different things you could do to secure cloud and when you first start out and you don't have anybody do any production workloads you probably feel you guys have all seen this right like a million times so I could just kind of refer to it cuz these pictures are terrible so mr. burns you know walking and they go through like eight different

secure doors and I always feel that way when I'm you know hitting my two-factor authentication for the 90th time that day you know and you know that your secret thing you could like especially with you know sort of not necessarily a bank but you know a car rental company or something with that they go through all this effort to have all these different things and then you know that if you just call the person ask nicely they'll change your password and so it's one of those scenarios in cloud where you have all these things in front so you have the two-factor to get in and then the second anybody tries to actually get some work done there

they're like yeah but easier if I just did this no different than on-prem humans are humans whether they're in the cloud whether they're on Prem that kind of stuff still happens and so the second you guys start trying to actually get work done in the cloud then you end up with a screen door with the stray dog running around next to the keys to the kingdom right all right so this one came in add about three or four incidents over the last well I don't want to go into this some time period that I'm amalgamating here but it was a great example and I had to I had to show you guys so first one I mean come on really

you got a dev server nobody you know it hasn't been patched it hasn't gone through QA or whatever but it's out on the public internet because it's much easier to just do all zeroes right same same thing now normally you'd have on Prem you'd have to go through like ten different tickets to get somebody to open or write because you have these really tried and true processes and cloud often what you have even if it's not shadow IT even if it's like legitimate you know sanctioned activity within IT you'll have and this is actually overall a good thing but we have to have developers I call it democratizing data where because in previous roles and fire I have been an

engineer where I'm just trying to get work done and we want to move quickly and getting ops out of the way is is helpful if you don't screw it up right and so there's generally a lag in process for cloud because you don't have these age-old on-prem scenarios and yet a lot logically you do so if you accidentally open up a port that can be really bad news because really you doing experimental stuff on cloud because no one else give you a box on pram or or whatever all right so that's fail number one number two not understanding the difference between user names and roles it's a lot harder to screw this up now than it was maybe a year ago but we

still see it all the time so developer singing in terms of on-prem said of thinking usernames and passwords and I need a user that has access to this kind of LDAP style like they would an ad it should be all about roles so when a box starts up a call at an instance role is then you've granted permissions to the Box inherently and it's engrained into the box it's not using a credential per se there's and it's even difficult to explain just in a sense there's a lot of nuance in there but basically there's a right way in a wrong way and at no point this should be pretty obvious should a box it doesn't need the privilege have

it right but it's always easier to over privilege and so now this box has an admin role as its booted up and it's 90 the proper role okay so that's double facepalm it's the triple facepalm when now we have a public bucket this is like the one cloud security thing that everybody knows is don't have a public bucket still happens there's a lot of reasons for it often sharing data between partners is the biggest one because you don't exactly know who you're going to email to give a credential to or something like that so it just organically happens and both Amazon and Azure have gone to great lengths to try to notify you and this is

happening and it's still not gonna help because when you set up the public bucket you probably are intentionally setting up a public bucket because there's nothing in there yet you're like I'm gonna drop this one file in there it's gonna be great yeah it doesn't go great but it's an age-old security problem especially bad when the developer thinks well when this script starts up it needs credentials I'll just store the credentials in a config file and I'll put that config file in s3 okay now we have a real problem alright so what will happen is the attacker comes in they notice their server because they can get into this box that's bad that box has way more access than it should

and so happens they've got to get out of this without much of a problem except that the some of the data in the database that they were able to see Cole inject into referred to an s3 bucket so they were able to discover this s3 bucket that they normally wouldn't know about fun fact you can enumerate s3 buckets using DNS so you can pretty much a numerator them as fast as you want because every bucket has a unique DNS entry so if you find out that there's a prefix that a company is using you could start going through there it's another great using reason to use templates for instance CloudFormation templates will automatically append a giant random

string to the end of everything which will completely save your bacon in that scenario because it would take you know thousands of years to be able to do enough DNS queries to find that but if you just go in and you're manually naming stuff then you're gonna end up with you know a very predictable bucket name can be discovered all those things pentesters out there the NS script you know parallelized all that and you can start doing some bucket scanning alright well they didn't have to do that because that was somehow ended that was unfortunate and what was super unfortunate was that even though they didn't get the the database to give them straight credentials the s3 contained a

file that had the credentials and this leads to step five bonus subnet this happens all the time so if you think of why criminals are stealing usernames passwords bank accounts they're trying to take knowledge and convert it into money this skips an entire step why go to all that bother just Bitcoin mine you get the credentials spin up as many boxes as possible if you're nice till you spot instances because they're nice and cheap and you get even more out of it and if you are a pretty big customer that means your cloud customer your Tam is probably gonna call you and say did you mean to spin up 10,000 boxes last night because it seems like a lot you

don't normally do that that's if you're lucky most places don't have an account big enough to have a person who actually wants for this and this is where instance limits really pays off in general because if you're just starting out you probably have a limit of like 100 or so box but if you are a personal developer and you're just trying to get going you're gonna have a bad time at the end of the month when that billing statement comes through because that's gonna be the first time you notice that I think it's like a hundred boxes by by default for a personal account that's that's a lot of money and if you're a medium-sized business it's going to be

way more than 100 boxes and it's gonna be a significant amount of money and this works so the and it's really hard on the cloud provider side I remember Joanna's comment well the cloud shouldn't have any access to my code I don't want them to know about it they aren't and they don't and the reason I know that is they have no idea the difference between an overloaded database and a Bitcoin miner it's basically proof to me that they are not snooping on this stuff because otherwise they would just say oh yeah anytime Bitcoin starts up then we're gonna notify you they don't do that right now they have no idea what's running on your

boxes because that's your business in this case that would be really helpful to know it'd be nice to know that new boxes spun up if you think well I'm gonna prevent that by running an endpoint agent on there well no you're not they're gonna spin up their own agent right they're gonna spin up their own box they're not gonna use the golden image that you created that has all your endpoint stuff on they're not gonna happen so there's really not a whole lot you can do to prevent it other than watching the API calls very carefully and having really good visibility into what's going on alright so how we're going to detect that well some of the basic stuff

hopefully you can alert anytime and any any ackles created it probably has that on purim you can certainly do that with cloud it's really easy same thing when a public bucket is created again at least Amazon I'm pretty sure Asher has a thing for this too will basically put a big red flag on any public bucket but again the developer probably wanted it to be public because they didn't think it was going to be a problem you can have it but your security will know it's a problem so you want to make sure that that information gets you know past the developer to security so that they can do something about it and start enforce policies you're gonna want to

alert on an instance of booking a user channel instead of a role talked about that a little bit it's pretty fine-grained into the sort of iam permissions and all that stuff but it's it's easy to programmatically alert on if you're in a position where you get access to the data right and that was kind of that first part is shadow IT all those things can often make it hard to even get set up to do this and then anytime you have a box that's being manually booted that's a great one to at least take a look at now that happens all the time I mean a developer especially in your sandbox account or something will spin something up take a

look at shut it down is how they do research right and so some of this is just legwork I was I had a thought just like wearing the the crazy cloud jacket wearing the crazy cloud jacket and doing a little bit of a game show called who booted that box and the more I played it out of my head I'm like this is not worth bothering people to come up and because the answer is you can almost never tell who booted a box and the answer is just always no so I called three people and I'd say who go to that box they'd say I don't own it say up you lose again have a seat

that's the show because nobody ever knows who booted the box yet the call a guy or a gal and ask why did you boot that box and you would only do that if it already looked really fishy which is kind of a hard thing to programmatically say but if you think back to that pyramid the hope is that you can automate out all the really basic stuff to get to the point where your sock is actually understanding the development projects that are occurring to that level they know developers or they can at least access gear or what to figure out should person have done that it's really weird what they're doing have a conversation so you get

that sort of dev SEC ops rainbow unicorn stuff going on which is difficult to pull off but it's basically going to be required if you want this to work all right what about hunting so a few basic things fairly self-evident but just calling them out specifically we're there any any new IPS that access this thing again this is soup you're having your developers generally come in through one place there any objects that are coming out us three s3 object logging is actually pretty easy to do it can be quite voluminous so just like Dave Kennedy was saying earlier a lot of places don't want to do it it gets to be too much it's worth it you should definitely turn

it on there might be one or two buckets that are so crazy busy you have to turn it off but those are probably not the buckets that you're worried about generally it's it's these buckets that you reel to know who's downloading what because I bet you if your socks filename come through that said you know admin credentials they would probably ask some questions the developer didn't and then back to the user agents and IPS these are still that for now they're still valuable it's worth looking at there probably will be a day soon that they won't be very helpful but it's at least least worth taking a look at all right so how do we how do we get to this point

where we still have all these problems clouds supposed to be awesome it's not always awesome well that's because of how we get there and I've talked to probably half of the fortune 100 at this point on their their cloud migration and and how to secure that at this point I consider myself a cloud counselor versus you know whatever rank a CTO for cloud and like I normally I have whiskey involved because it's a it's a difficult conversations people think through this well how in the world this is going to work so general or the lift and shift which is there the garage sale moment where they decide you know it's this coming with yes or no and that gives you

a little bit of savings companies you say about 10 percent right there because they're just running less because they decided there were stuff that could consolidate or they wouldn't wouldn't run and this from a security standpoint this is not too bad because you generally just taking literally everything you had on Prime and putting it up the cloud same firewalls same endpoint all that stuff things mostly makes sense kind of a you know just a different name on the data center sort of thing and yes there are some API things you have to worry about it's not too bad but that you have that familiar security posture the part where you get the most benefit as a company

from is when you go and I'm a huge serverless fan for a lot of reasons you could just get more work done faster that way I sometimes an unpopular opinion depending on the room but I've seen it and it containers all that stuff you can go really fast you can do it more securely if you do it right there's a great reason to do it but you have to learn the right way to do it you have to invest if the Train if they understand what you're doing and that's where we run into problems because not all it is it hard for the developers to learn how to do it or I should say hard but they

have to actually learn the other problem is that the security itself is entirely different who here is used in OBS lambda before okay whew there's not really an IP address involved with Lambeth there's not really an operating system involved yes underneath the hood it's Turtles all the way down I know the cloud of somebody else's compute but I'm a business standpoint the cloud is not someone else's computer it is a prompt where I type stuff and it runs I don't care about the underlying stuff that was the whole point so while they're technically is something running under the hood for 99.9 repeating percent of you you don't care about it there as far as you're concerned there

is no operating system as far as you're concerned there is no IP address and that means a lot of the things you count on for security and your day to day when you sit down in your chair in the sock well we have a lot less to go on now now it's all about logins about application logging so now again we had said you might have to call developer you might actually have to sit down with the developers and tell them they need to log more they actually record what that app is doing so there's something to audit because you're not going to have a peek app for what's going on because there's not a server you're not going to

have an endpoint agent that you can run and you need to understand how your your gonna be able to secure that because there's still important critical mission-critical stuff coming through this pipeline it just doesn't have anything to do with all the the tools that you're used to and that's really the big change so there was a great YouTube video about I think it's two years ago at the chaos competing convention in europe this video is amazing so this guy just showed basically this how you can use the cron like persistence we've seen attackers used for years now where a cron job will make sure that you know all their backdoor stuff is still there you can do the same thing with something

like lambda so you can have a lambda run in a schedule and so what you can do is create a lambda that makes an API call to delete itself and then makes another API call that says in 60 seconds recreate myself it's kind of cool it's cute but actually works and so what you have this is a situation where as a admin if you go to the console and say with something funky is going on I want to look at all the lambdas there won't be a lambda there it doesn't exist on the admin console and yet it's running now and it will run basically forever it's actually kind of hard to go and get

you might see it in the law groups or something like that getting back to that audit trail but you're not going to see it in the traditional sense so it's just a great example of sort of the new paradigm that as you go to the cloud you're gonna have to learn it's not hard it's not too bad but it's just a different surface area alright so it said talk about joining again so this one caught my eye it was a forum post and this just general IT admin could not for the life of him keep his HR recruiters laptops from getting owned every single day it was a reimage every morning and he and you think about it

and this happens in the legal world plenty if your job is to take in unsolicited Word docs you are in a really rough spot right now because there's it's so easy to own people using macros and things like that that are you know AV is not catching this stuff you're lucky if if your email is catching it and a lot of time this thing this goes through things like Newton or some other recruiter SAS applications where you don't get direct access for your security tools onto these events and so you have all these unvetted Word docs and Excel spreadsheets and PDFs coming through going directly to laptops in your organization and just completely owning them and so this guy decided that

he's gonna he was that as wit's end so he's going to put them all on you know the cubes OS which is getting a little bit popular now but it's still pretty esoteric for just a general IT admin to know about or care about and that was the solution he was like I got to do something super drastic with this and it so that really caught my eye that this is a big problem that that we have and this is kind of the pattern so you have a CRM that doesn't have any instrumentation on it that you're used to that you've mandated but you don't really have it you know saying this right as security admins you can't tell

them what recruiting software to use they're gonna use it maybe if they get owned enough they'll decide they want to change but there's not really that many to choose from and it's not just recruiting software there's a lot of different kinds of variations on this but that's the general pattern that we see all right so what's the next in cloud threats time here pretty good all right I do want to take questions at the end by the way so make sure we say time so on the confidentiality side the the plastic CIA triangle here data theft that's the one that's in the news all the time like this is obvious right accessibility this is a DDoS problem

Dylan is plenty very understood what about integrity this one I feel like it's forgotten quite a bit so I did a workshop at RSA this year where we did sort of a purple team thing and we came in and we hacked some server list stuff using basically parameter injection nothing too fancy but the sort of capture the flag element was changing a data file and I think the question that I'm posing here is is about integrity so how do you know if something has been changed but not deleted or not you know put on the dark web that's that's the thing that keeps me up at night is how do you know and it's you can expand this

into election results things like that how do we monitor for the integrity is something like that when it's data and we don't really have like a checksum that says well this entire data set should have this checksum that might work well for you know one single researchers file but if we say how do we know that the entire corpus of all the transactions for today in this database are correct I mean you get into blockchain some of those kinds of things maybe that will help from ml standpoint this is where I in the workshop we talked about this in a simple fraud use case and there's a an angle on this from the cloud security as well

so the normally you would do this is you have a big transactional database and you do a nightly CSV export or you know something fancier than a CSV and that becomes your source data and that gets over to the machine learning guys and then they're gonna put it through all the training data and all that stuff now question number one do you have the same security around that you know transactional super important banking database that you do all those CSV files that you just exported and then toad it all over the cloud are you tracking that with the same same rigor you have the same controls over that a lot of times I do but a lot of times you don't

especially derivatives of that file might get sliced and diced and put other places but in any case you have this legitimate source data that you're hoping to protect is turned into training data may go into something fancy like a neural net and its job will say in this case is to detect fraudulent transactions as in it's really weird to transfer a million dollars into a Romanian bank account or something like that alert block that okay make sense well so the capture-the-flag part of the workshop was to basically add a few lines to the CSV that statistically made it normal to transfer a million dollars to Romania and that was actually pretty easy at least conceptually to do so if

you hack that source data suddenly now you've hacked the application itself and so understanding again this is why it goes back to a pelagic how is the application working so if you can get to some small piece of that is that can affect the integrity of the overall thing alright so how are we gonna fight this over alt with visibility this is from last year's Mandy and M trends hopefully you guys are in the M trend here we'll put work into it there's a ton of good stuff in there we put out a lot of stats things like that but my IR guys had had my back from the the cloud practice and they are talking about the

importance of visibility I didn't have to ask him to put it in there that was their own own recommendation having responded to so many incidents in the cloud was that visibility is what its gonna take and they were nice enough to give you a few self tests and I think that these are probably the most important there's probably most important slide the whole thing it's the least exciting but this is probably the most valuable because these are the litmus tests you can bring home and say okay is the answer yes now the for each one of these questions I mean have you operationalized that is in does the sock do this on a daily basis is this normal

so can the organization see what files are being downloaded from a cloud-based storage site it doesn't mean after you know you have an incident can you go pull logs that would tell you that it means if there was something weird coming through there would you know about it before the FBI tells you about it our admin logins traction reviewed the same story these tracked and reviewed by sock you know regularly every day as in would less than 24 hours go by what about unauthorized provisioning and cloud infrastructure the who booted the Box game show starts right up here your sock is the contestant every single day and they're trying to answer that question and that means dev sac ops bringing them

into the fold for new projects early so they understand what's going on from from that standpoint they understand what's normal it's not normal and then what are you getting from the cloud provider themselves that's going to help and this is where the cloud provider specifics are so important so whether you know it's VPC flux from Amazon or Azure network watcher flows it takes a little bit to instrument that and so being able to specialize is very helpful a lot of us would not have that luxury yet to learn both it's it's worth it our you know Google cloud or any of the others you need to do it all right so I break down visibility different domains

pretty pretty straightforward network endpoint and events events is basically the catch-all it's kind of what it should look like when you talk about full monitoring so we have four main types of cloud provider logs full dive and then the next slide and then just making sure that you have a good image for anything that boots in the clouds and this was I get asked about container security a lot the vast majority having built stuff with containers for a long time the vast majority of what you're running in containers the containers need to talk to each other so I've talked about honest startups that have really specific like firewalls between containers because they want like one

container to affect another but the vast majority and whether you're running kubernetes or anything else those pods or whatever are supposed to be talking to each other so that's you're not gonna get very far with the firewall part of it what you need to remember is for if you're running a Linux based container on space host it's going to show up like a normal process for the most part there's some caveats in there but that means that you're you know typical endpoint agent is actually going to provide a bunch of security for your containers because they're running on the host it's very different than a virtual machine situation now the caveat doesn't apply for crossing your operating systems with

containers but same operating systems Linux on Linux especially then you can basically apply your your host based security and get pretty far and a lot of places won't tell you that or you know then they try to make it more complicated but basically do work with the hosts that are running the containers and be in pretty good shape you can still get pretty good Network visibility it won't have all the tags reach you know docker container attached to it but it'll still help so endpoints still critical and then anything you'd get from on-prem to correlate that it's still gonna be really critical so I tell a lot of people to make sure that you're monitoring the choke point if you have a

pretty large cloud implementation you might have paid for a Direct Connect so it's like a dedicated gig link between like your big data center and a cloud provider and that's a great choke point to put network visibility you're gonna want to monitor all that especially the SMB traffic going there and then all that's got to get back to one place because if you think about the last slide we have to answer all those questions they answer them every day as efficiently as possible because we have other things to do alright so the four basic types here and I this might help some of you speak the language a little bit to break it down between Amazon and

Azure so this is kind of the the matrix here for audit and Amazon's couldn't be cloud trail and Azure as tenant activity logs for operational stuff and this is including app logs this is a big one so it's cloud watch and Amazon it's activity logs or OMS and azure and that's a big one so especially with containers by default on a place like Amazon they will all log to cloud watch so as long as you can say I'm getting all my cloud watch into my centralized logging you're good and so when a dev spins up a new container app you're gonna get all those application logs you can have a fighting chance so really important to make sure that

that is the standard for the environment you're not doing anything weird and actually this is my my one pitch for saying do you want to think about something like the native cloud service or like ECS versus kubernetes now you can run both on place like Amazon the security benefit to using something like see s is by default everything will go into cloud watch and so you only have to do that one time if you do the Cooper need easiest route then you have to make sure that every single Dev says they're gonna put their logs in a centralized place which means some of them won't so that's my consideration when you guys are doing this planning think about it

from that standpoint native will be more secure for sure and then lastly and you probably see more announcements from the cloud providers themselves but guard duty and security Center are they're not really equivalent at all but they're sort of in the same bucket those change like month to month so I'm not even to try to tell you you know what the level is but understand that those are the security alerts that you're gonna want to centralize all right so this is part of the overall layout so generally gonna have a few data centers your branch office all that stuff just make sure it's all coming back to one place and then I do want to talk a little bit

about the the sad stuff so this you know this is one of the hardest things that I find when talking to customers cloud means like something to everybody and it's almost never the same thing when I talk about cloud mostly it's from a public cloud provider because the other problems are either you know not interesting or unsolvable but they're still problems so if I as like infrastructures of service providers we're just renting boxes those are still totally cloud you have a lot you know less to worry about there from the perspective of it's truly just a different data center and it might be like just a tiny different API used to boot a box but not really differently

than running VMware and your own data center that kind of stuff but from SAS this is where you know everything changes SAS is a totally new animal and it you know obviously it completely depends on what the application is so there's some big ones they might have the most data in them but the parade of apps that are coming through means that your job is never ever done making sure that if you're using a Kaz be that those logs are being centralized and that are they're reviewable just like the other stuff we talked about that's critical that's a you know it's basically the new firewall at this point right deciding who has access to Salesforce who is and

when they're in Salesforce were they going to be able to do there's a really bad trend that I see right now in the industry we you have to pay more to get additional logging this is one of the most dangerous things I'm seeing right now with Salesforce is something called shield they pay extra to get the verbose logs with Azure it's it's pretty bad too they basically say unless you're going to use our thing which is like half of what you need then you got to pay a whole bunch of money to have third-party looks at it so if you want to dump that stuff at ELQ or something like that it's not as good as if you use their built-in

thing which doesn't do what you want it to do and that's a trend that's very disturbing because you're not getting that full visibility into what's going on and so that depending on you know which cloud you're talking about I think you probably continue to see that because from you know if you're if you're writing a sass and you're selling it to people you want to spend the least amount of time in infrastructure as possible and extra logging might mean more time and infrastructure and what's the immediate benefit so unless we in the security community you know make a lot of noise about that and say well we don't want to recommend anybody using your thing unless it's going to be

secure you have to prove it's secure by having proper visibility with logging then they're probably going to continue to you know give you less and less to look at to the point where you might not even know if you can log in did you know that you have to buy as your premium ad to even have the right to see when people log in yeah yeah I I have some buddies at Microsoft that are equally annoyed by that and you know in some ways if you're a big company you probably buy any three or five anyway so generally that gets bundled in but if you're if you have a development wing and you're doing a lot of sass you don't

want to do that you want to start with the basic stuff because probably to need most of that other stuff in the first place and you just get into the situation where you don't have what you need to get your job done and that's a bad trend so I'm hoping that we could make enough noways to at least get people to give us the basics that we need to get our job done I might want to call the action from all this so yeah VDI is also in there too I talked about that a little bit earlier and that's all all factors of that don't forget about it all right almost time I almost question time here

so final checklist pretty simple fishing fishing fishing mail email email still for everything number one and points and networks are still important even if they're on Prem they can still be used to give you cloud and visibility and all the things so provider event logs files that are being transmitted through stuff that you don't control try to find a way to get control of it or at least get a view into it and then of course any size telemetry you can get all right so we still have a good amount of time here I have a few giveaways so for every question we have nice goodies here for you

your choice so ask questions early because you get the pick of the litter here ooh good choice I know that guy he's awesome

yeah can you repeat the question sorry and we have a challenge of security of keeping up with the changes so we'll have something that that was a preview feature now becomes a standard feature and all of a sudden this thing can happen now and you can log in from it right right thanks okay so the question is how do you deal with the continually changing nature of specifically Azure which so I sound like such a Microsoft bash and I'm trying not to but I had that same experience so Adger is a great example of they're doing amazing things at Microsoft by clarifying the company kudos to them they've done really great but as part of that almost everything

that's not infrastructure as a service on Azure right now is basically preview if they don't tell you it's preview if you get on a tech support call they'll tell you it's preview I think event hub is about the only thing you can like legitimately use and be confident in and then their API is change all the time so but the answer is you know lemonade or whisky whatever your drink of choice is it's not going to get better for a while and part of that's good you want that innovation you want to keep moving but basically Microsoft is is behind enough and they're throwing so much at moving fast that for many years they were good about documentation and things

like that that's not the case right now at least for the next few months and that's really unfortunate because this is a critical time for us to secure things as things are moving quickly this is when errors are gonna happen new developers don't know what they're doing yet new security staff don't know what they're doing yet that's the hardest time and I there's not a good answer you just have to be diligent and beat them up on it with your tan yeah go ahead

okay good question how are you keeping up with the changes on new cloud offerings it is an unwinnable game so basically you try to find what the biggest impact is going to be to your business and sometimes it's a really tough call to kind of Sophie's Choice it a little bit where it's the I only have time to do one of the other but really ask yourself from a sort of a threat model who what is the worst that would happen for this particular thing right and then look at the partitioning there yeah

right so when the upgrades happen how are you gonna find the new stuff Twitter honestly to start with I mean you'll see some security research come out of there but I mean yeah any any bleeding-edge new stuff it's all it's the same game that you would play with anything else it's just in a little faster cadence so you stay on top of it I mean that's that's continuing education is critical as is for any security research

excellent as the question is what about pcap on cloud would you use things like the new virtual app there's probably something like that coming out for Amazon soon to where you have the ability to capture packets there's some value in that but not a whole lot almost everything that happens on cloud is totally HTTP Network flows are going to give you the vast majority of what you're looking for the caveat there is for most cloud providers right now you can't get a solid log of DNS entries without doing some like reading packets to extract that so I would say bare minimum in a report 53 and at least you'll have DNS names but there's not a

lot else you can do so you've at the same time I'm so sorry all right okay how else could a three bucket be used maliciously aside from stealing well there's the integrity part they talked about so if you can alter a file that might be really bad depending on what the file is but the other part is that you have a lot of bandwidth available to you when you're running in an s3 bucket so hosting any malicious files would be super bad is tough to top my head using any kind of steganography to hide really bad stuff would as a drop point you can use oh so in my workshop at RSA how you can use s3 to trigger a

lambda that would have bad parameters in it that would get code to execute so the act of uploading an object to s3 may actually execute code somewhere so just simply writing a file or bossing the amount of logging that comes out of that so back the visibility you can make yourself invisible by Duffing amount of locks coming like we still have we still five minutes good I got in the blue the if I are in the cloud recommendations so a lot of that doesn't change you're talking deify our if you're doing in like getting images is actually a lot easier as snapshots is super straightforward so that's a big boon Oh

interesting so would you recommend downloading it on pram no I would say keep it there because the same read and write ap is that are handling the the actual machine level byte stuff whatever attacker came in and altered that you know the hard drive wherever it is in whatever datacenter had to go through that's API so I think that your bit gonna be better off and certainly operationally from a sec ops perspective gonna be better off keeping that cloud side I don't think that there's a big advantage bringing it down on primal if you have a really specific reason for doing that in fact you can leverage the big data analytics and things like that

to run ten different analyzers on one giant disk image and spin up you know 100 boxes to do that in ten minutes instead of ten hours right so I would say leverage that I don't think you're gonna get anything by downloading it because you're still gonna go back out the same api's so any kind of bytes that would have been altered some way you know I can get the raw harddrive info anyway yeah if it's in it yeah exactly if it's an active incident then yeah suck up still is very important so try to build a separate area for that you can share images and which can get you in trouble if you forget the permissions

and don't set them up correctly but you could certainly share those across account so that you could have an IR account essentially it's very least yep they're in the

okay so you're asking if you got as far as aggregating your data and got that step what's the biggest mistake the sin of omission is the biggest one thinking you got everything and not getting everything because it is a daily challenge that's the number number one thing building in heart beats so that you know when you're dropping that's another big one you may think that what you already set up is working but you need to have a way and someone whose responsibility is to say if this asset is important for us to secure that means we need to verify we're getting that that heartbeat of telemetry so we know if the logging stops there's a lot of

reasons that logging can stop someone might accidentally change permission that allows that to happen the Box made I'm a boot up as a different box all those things can happen I didn't even talk about the ephemeral nature of the cloud but that should be fairly obvious we need when you're talking about DF IR it gets a lot harder because you're talking about a moment in time right so again back to that even the IP addresses for servers versus server list can off often be meaningless in auto-scaling groups where a box may exist for five minutes all right another question go ahead

biggest blind spot in telemetry for the public cloud providers yeah I used to say it was pcap I think they are well on their way to fixing that now and now that by the time they're getting there to fix that I would say that's not even that big a deal anymore because what are you really going to see other than DNS I think that it's really if you're developers are logging things to their app logs you're going to be in good shape but that requires them to do work so really that's the biggest thing that I see missing is you get box booted up box shutdown whatever happens in the middle who notes was a Bitcoin mining

that's on you that's the shared responsibility model like whatever happened with that box when it was running that's on you so making sure that you are going out of your way to as a it's a policy as a security policy and practice make sure that your devs are logging as much as possible so you have some clue what's going on with this box is absolutely critical yeah I think that the network's elem tree is probably the best biggest missing one I would also say the sign-in logs you know whether it's Microsoft or Amazon could have a lot more there and that's why I had on that sort of ecosystem chart that on-prem is a part of this because

coupling your on-prem Logs with your card logs it's usually the way that you get the full story there and all those great questions yeah slide four slash flasher this one yeah yeah that's the one notice how this is just pointing out that all these things play a role wherever wherever those assets are they're still your assets and they're probably going to talk to your cloud stuff and if you're only looking cloud side you're probably only getting half the story just like if you're only looking on frame you're only getting half the story all right Oh last one I can refer you to someone who has it's it's one that I gave my own CEO so yeah there has been stuff published

it's right now very unpopular because almost everyone has no idea what they're doing and so they just say cloud all the things and then hope for the best it's a terrible approach I think it's the monster bleep provably a terrible approach I will see if I can find that for you but yes I would love to write that myself it'll just come off as a angry rant to the editor if I do it right now all right thank you guys so much I'll be in the hallway if you have any other question [Applause]