Sky high IR: IR at Cloud Scale

okay all right we're gonna get started again here our next talk is sky-high I are I are at a cloud scale our next speaker our next speaker is a b-sides Raleigh Durham alumni he's been here a couple times so we appreciate him coming out obviously we wouldn't have asked Novak we didn't think he had a great talk but Aaron is a he's been in cybersecurity for 15 years working bunch of different organizations such as healthcare federal government software energy and defense industries he specializes in managed security and assessments assessment services including managed detection and response strategic and technical assessments and training Aaron's also a graduate student at the SANS Institute and Lancaster thanks yeah I appreciate you guys coming

out to hear my talk today and thanks very much to the besides organizers for having me back count it a privilege to speak to you guys today and so for those of you who don't know me a little bit about myself I am a senior security engineer at RM source a consulting firm here in Raleigh as well as master's student at Sands Technology Institute STI for some some of you may know it I also like to spend time with my family try to put them first and a little bit about myself former army helicopter pilot seven years there also a number of other duties there generally my day-to-day in cybersecurity looks instant response defending networks

assessing networks to find new vulnerabilities as well as writing code to try to automate out routine processes and I'm also into amateur radio as well I do want to let give you guys a little bit of overview of what I want to talk about today so first and foremost here we're going to talk about some strategic tactical and legal implications of conducting ir in public private and hybrid cloud environments we're also going to talk about some industry trends and tools some critical components and crown jewels and how they might be compromised and then we'll we'll have some time at the end for QA I am going to provide a link to my slides the recording as well as some references for

further reading at the end so don't feel like you've got a burn up your battery taking pictures or whatever all the slides and everything are going to be out there for your perusal later and I'll have a have a link to that at the end of the presentation I do have to make a disclaimer about this presentation in light especially in light of some of the things that we've been talking about today I'm gonna present my point of view on some of these issues they don't represent my employer's point of view past present or future and please do not take them as legal advice you need to seek your own legal advice from your attorney or your

company's attorney regarding how these issues might impact you and your company so rule number one I want to make sure to regard I'm going to pull you guys real quick so I can get to know you've gotten to know me a little bit so first and foremost how many individual contributors or worker bees do we have out there guys doing the day-to-day grind either offensively or defensively or both okay good next what I would like to know how many team leaders or managers maybe you would consider yourself to be in that category all right and then finally well not finally but next up I think would be maybe a group of people who doesn't fit

in either one of those categories but how many students do we have in the audience or maybe you're thinking about getting in InfoSec thank you for being here this is one of the best opportunities let's give them a hand the students for being here and then I was gonna ask if there were any feds in the room but we already know the answer to that so we don't have to answer that question all right so no matter who you are I think we expect a lot from the cloud and and that's kind of resulted in some I mean I think some misplaced expectations maybe it used to be easy to be a cloud but it's a lot more difficult

these days to be a cloud because we expect a lot from it right technology has promised us a number of things and we are we are expecting its deliver and the and I think that in tech we feel the pressure of that because as the implementers we we feel the pressure deliver on those promises as well and I think one of the important things that we've got to realize is that sometimes those expectations can come back to bite us be yte I apologize for bad jokes in advance I'm a dad now so according to Google there's not much difference between dad jokes and bad jokes so I've already apologized for my bad jokes but I think you know one of the

expectations that misplaced expectations we have about the cloud is that at the end of the day the cloud is really just someone else's computer and so that creates some problems for instant response right as we start to think about different cloud and there be public private or hybrid cloud environments hybrid being a mixed between private and public cloud environments there are a number of different strategic and tactical obstacles that we have to overcome in order to be able to conduct IR which we've been hearing about today a number of excellent talks about instant response and forensics and in different different focus areas and so I want to talk a little bit more about that from a

broader perspective and kind of help bring some things together maybe from today so toward that we I want to talk about the inspiration for this talk so I actually attended to talk a few months ago that was given by a law enforcement officer and they were talking about some of the challenges they have when they do investigations which I don't think we had time to get into as we were hearing today from our panel about the ins and outs of what an investigation might look like stepping through that entire process that talk and hearing that you know how to it had a focus on on coming into a business environment from an outside perspective and conducting their

legal investigation within the four walls of that business as it were and so today what I want to talk about is forensics but specifically as we're talking about the cloud I want to be clear to say that we're talking about forensics and instant response on virtualized platforms typically legal investigations involve short notice so one of the things I think that was not touched on you know with our panel today that that I want to share from my experience is when you are and suddenly become involved in a legal investigation that you weren't planning on so what this means is that you may receive some short notice that you have to produce some data we're going to get into that in

just a minute but also that each of us has a legal ethical and self-preserving interest to comply with the in its investigations but we also have to keep our customers in interest and best mind and that that could be individual customers or business customers and and so continuing to think on some of the things that we've we've been talking about today as far as that the ethical implications of some of the legislation and policies that we're seeing change in recent days and years so what this is usually you know would come to you in the form of a subpoena and that would be frequently received by you or either by a CEO or someone in your company's legal office

someone who represents your company and then make its way down to the Information Security Department through the chain of command but that would perhaps take a form that would look something like this this is a an example of a subpoena and it spells out some things I just want to point out real quick I'll apologize for turning around because the view on my screen is really small and I want to be able to I want to be able to point out some specific items so I'll try to keep the audio good but as you can see that there's a deadline in the first paragraph this one is redacted but these have this goes towards the point that I was making on

previous slide about short notice this could be as short as 14 days calendar and as well there will be as a point of contact like a special agent whose work in this particular case frequently these are related to ongoing investigations into fraud related to something the government has prior knowledge about so it could be a federal contract it could be insider trading it could be any number of one of number of issues where something has been reported to law enforcement and then now your company because of the services or or either internal or external that you deliver you now have become a part of this investigation investigative process in addition to this there could be a

subpoena to testify before for example a grand jury so this could involve actual appearance like was talked about earlier where you personally as the point of contact for the forensic investigation comin or questioning for the forensic process so due diligence in that forensic process can become a critical thing but in this particular document what the reason I wanted to have it up here was because you have a number of informational elements depicted front-and-center that are related to this case that the government may be looking for in order to substantiate a legal case in and of itself so those things could be for example IP registration history logs addresses upstream and downstream provider information accounts cookie

data it could be raw logs it could be entire virtual machines it could be an entire V host it really can run the gamut and to add to this what if I told you that you might not be able to talk about it individually or corporately so these can be tip can be accompanied by a gag order and this is something that would say you are not allowed to in any way make movements do say anything that would tip off the subject of the investigation or even your customer who might not even be the subject of the investigation that there is an investigation even going on give you a chance to read that real quick so

these can be non time limited historically speaking and tangential of this is the National Security Letter which would say this is an issue of national security and you you know are now under a mom regarding this issue even if you you know never had a security clearance never worked for the government might not even be a citizen in the United States for that matter but as being implicated brought in to the part of this investigation you're now your freedom of speech is restricted so because of this Microsoft decided to sue the Department of Justice in 2016 the end of 2016 because what they had they had encountered was that two-thirds of their 3,000 they had

received over 3,000 gag orders between 2014 and early 2016 which resulted in the case at the end of 2016 where two-thirds of those had no end date whatsoever and they basically went back and said look at some point in time we should be able to tell our customers we had to give your data over to the federal got the federal the Department of Justice for a federal investigation and we want you to know they felt an obligation to do that other companies especially large publics public cloud providers have followed suit Microsoft are sorry Amazon Google cloud etc Rackspace and they're arguing against these section 2705 store communication acts as a violation in the First Amendment so a

freedom of speech to be able to say this is what happened and and we feel that you need to know so because of that in 2017 the Department of Justice changed their policy to the best of my knowledge and from talking to others that has made has stayed that stayed the case and they changed their policy to be them to require that appropriate factual basis for the cases demonstrated before the subpoena and gag order can be issued they have to be limited to one year and that policy is overall barring exceptional circumstances now there's been some speculation that the Department of Justice would recant this policy in an extraordinary circumstance but that hasn't happened yet again this

landscape is changing very rapidly so it's very becomes very difficult to you know determine what might happen in the case of a large maybe in a large-scale terrorist attack that became known in advance or something like that so I want to talk a little bit about I our process for maybe some who are not as familiar with that so standardized process for I are an organization I'm going to go right out of the instant handlers handbook Patrick crawl February 2012 this is available for download on the the sans reading room I'll provide a link to that for you guys as well and so the first step in that as we've been talking about today is to plan ahead to have an

incident response plan one that's tested and is verified and everybody is trained up on how to fill their role as a part of that incident response the second part is to identify every host every Network every database everything that is involved as a part of the incident so this gets into scoping we're going to come back to this in on the next slide containment so containing that incident and then and you know for example if it was a an infection of some sort keeping it from expanding if it it was a persistent threat cordoning that off so the the threat cannot threat actors cannot pivot within the environment to other areas with special care to not tip

off the adversary that you're on to their their actions next is eradication so actually eliminating that and we talked a little bit about that we heard a little bit about that with the panel and as our FBI agent indicated you know a lot of times eradication we think that we have done everything required to remove that threat and later find out that there's some other point of presence or some other issue where they've managed to wiggle their way back in so remediation is the follow-on to that so rebuilding systems patching closing holes closing vulnerabilities etc and then which I think what I think is even more important than potentially planning is the lessons learned phase so we have to

learn to learn from our mistakes learn from history even if it wasn't our fault and to move on with that knowledge our for to better protect the networks that we're we're entrusted with so I talked about scoping the response here so first and foremost determining the scope of the investigation so this could be you know something that's law enforcement driven it could be something that's internal internally driven either by management or legal or finance what-have-you so we got to determine what the scope of that is then determine what systems and data stores are involved in that investigation and finally gather the data needed to conduct our forensics one of the major problems with this is that

tactical issues with this is that when you start talking about the scale of conducting in a response like this and forensics of such this can result in an enormous amount of data so you know say for example you have built this infrastructure in AWS you've built this infrastructure in Azure and you're going to conduct this this investigation you know you you might not be able to just go out there and simply download everything that you need in order to do this this investigation it first and foremost is some of the cost associated with this so as you may know for those of you who don't publicly providers are going to charge you for transfers across the land across the

wide area network the internet and that's typically going to take that take the form of a number of gigabytes per day week or month and that'll be tiered based on what your current consumptions are so for example if we look at Microsoft with Azure you can see that for the first five gigabytes a month there's really no cost within within the zone but for the next five to ten in that month you're gonna pay a number of cents per gigabyte so when we start talking about say for example a a a full scope investigation we're full data and full virtual machine images must be provided to a third party entity and and forensic investigation cannot be conducted in-house that's

going to be conducted by law enforcement agent agency of some some sort you might be looking at something on the order of 14 terabytes okay so your business is going to incur an immediate cost in supporting this investigation that's something that you have to plan for in advance of of an event like this similarly with AWS you're going to you're going to also have data transfer fees and they're on a similar scale surprisingly not surprisingly over 80% of the servers in eight of us are Microsoft servers so you could be on either you know pretty much agnostic of what type of infrastructure you're running and potentially agnostic of what type of industry vertical you're in so what

we're really talking about is honing in on the devices being storage compute and data transmission the data and the destinations involved in in doing this this instant response so one of the best ways to pare this down is to slice this data down to only what we need so for example we can ID exactly what's needed just surgically remove that data and then use compression if possible to reduce the bandwidth across the land so again you know leverage that relationship with law enforcement if it is an external investigation they may not be able to say a lot of things about the case because they're operating on a need-to-know basis and they feel that you don't need to know and it I found

that is best to present the challenges that you're faced with as the responder but also I think general and this is a generally a good rule for career in general present a solution to that problem to them something that's acceptable from a time cost and operational perspective that's going to allow you to be able to meet to produce a win-win situation so discuss those time cost and operational impacts ask them for what you want so one of the things that plays into this is that more investigations or investigations are increasingly becoming it's becoming acceptable to only provide the needed data files emails logs etc due to extreme constraints so you can get the exact requirements that the

investigator may have by at by presenting your solution and saying this is what I'm able to do I'm not able to meet 14 days it's going to take me 14 days just to move the data across the network between backup cycles or whatever it is and then I can have it to you you know by say day teen or a business day 15 or something like that one way of navigating around this in the in-house investigation is to use I our dedicated VP sees for example or virtual environments a whole data center if you will in in the VMware context where a number of systems and accounts are pre-positioned but not consuming resources to use in the case of an

incident response like this so Jonathan poling did a great talk at D for the last two years sans D for conferences there'll be a link to that and my resources page for you guys to take a look at he's specifically talking about Amazon s3 and and how to run an IR in AWS so in order to accomplish slicing data like this if we do our force to move it across a man is to use a tool that can retrieve that data directly from the public cloud environment or hybrid cloud environment and one of the great ways one great way to do that is to leverage Python so you can use a module that understands the

existing structures if there is one depending on the environment and if we need to we can actually do byte wise retrieval of that data and slice down to very granular levels that would look like determining what the data format is what de limiters we can use and then loading that data and saving it off for analysis with additional tools there are other tools out there for example s3 QL which is a really a an additional file system that can be used to move data into almost a virtual file system for analysis it supports things like encryption compression deduplication snapshotting dynamic sizing immutable trees and most importantly supports a number of backends like Google Amazon s3 and many

others so let's talk about some scenarios and supporting architecture we're trying to move a large amount of data ensuring reliability and we need to verify integrity so if I sit down and think about this to me that really sounds like a backup solution backup solutions have their own challenges not the least of which is a forensic Lee sound image so if this type of thing we're leading to an actual case which sometimes you don't know if it is gonna lead to an actual legal case or if you just need to know what happens for the sake of getting back on back on your feet then the backup solution may not be the tool of choice but for Rapid

Response there are a lot of advantages and one of the ways of approaching this is using a backup solution like beam to move data between hot cold or warm environments where you could then conduct analysis from an attackers perspective this would look nothing like nothing more than a routine backup job so that's another advantage of this area is you're probably if you're already using beam in a fashion like this you're less likely to inform an tacker of of your knowledge or of an an instant response that's taking place Vemma offers a backup solution called beam backup free it has a number of really cool features to it one of the best ones is the compression and the

ability to be able to move full VMs even off an array all the guest files and to be able to explore specific items so say for example you know that what you're looking for was exchange-based you're not really looking for everything else you know all of the rest of your you know logs from that server all the rest of the data from that so were you really want to hone in on exchange specifically the view that this product provides viewers specifically explorers specifically to these types of of scopes a part of that is a component called Veeam fast SCP and this is something that they brought from a standalone product into Veeam backup free and it

has a number of good feet features as well and I think not the least of which is down towards the bottom here it doesn't require you to put ESX into an unsupported mode so you'll find that you may find that a lot of investigative techniques or a lot of instant response techniques require you to put ESX into some sort of console mold mode that's not supported it might create this make the system system unstable or something like that and this helps you to avoid that also you're not creating an additional threat surface because no additional services are required Paul Henry did the author of sans 5 sans forensics 5:08 I believe did a great write-up on

this type of technique and I'll provide a link to that and the references for you guys to take a further look at but want to move into some industry trends and some tools specifically related to in cloud forensics and offline forensics forensic analysis so one of the great tools that's out there completely free and and you know some friend pretty big friends of guest experts in our industry have come out and said look if this costs ten thousand dollars it would still be a steal so if you're not if you're not aware of the sans sift workstation as a forensic analysis and malware analysis platform because it can be Co installed with Remnick s-- you

should definitely take a look at that there are a number of supported file systems image types and and tools that are built in let you guys take a look at that real quick here really this is the type of thing that you should have pre position in that VPC environment or in a dedicated data center on your vSphere V host where you can you can spin it up and mount the drives that you need to take a look at right away so this goes back to your planning phase right you're not just planning on paper but you're actually building out these environments to be ready to go because for the eventualities of having a deal with some

sort of incident response so now I want to talk about some critical components or crown jewels and some ways that they might be compromised so attackers are after data where it's stored transmitted and process right so all of those have different peculiarities that allow them to become more or less more or less accessible and so we'll talk about I will talk about some specific areas related to this scenarios without getting into the impact and severity of the actual those actual things themselves so some of you may be it may recall when recall that ransomware a technique of France where especially early on was to disable shadow volume copies and you may recall that disabling shadow volume

copies is one of the primary ways that System Restore and Microsoft and even a lot of backup solutions do incremental backups and restore so one of the first vulnerabilities that we encounter is similar to that and that is of what's referred to as a VMware repair install this is basically installing over the vmware installation partition with a new install remounting all of the virtual machines and data stores and then taking advantage of of what you have ax access to this it you know could be is definitely would definitely be referred to as a hostile acquisition it requires physical access to the box and it's going to seriously mess up your v host including your disconnecting storage

area networks or network attached attached storage and toward this this has actually been done by competitors in customer premise equipment environments where the service provider didn't take adequate steps to lock the case lock the bios disable USB DVD drives locked down the storage Network providing curtain for volumes etc so what do I mean by that I mean someone at the customer premise makes a decision that you're no longer their service provider and invites another service provider in to basically take over everything not exactly serving their company the best nor nor serving themselves the best so again we're talking about where attacks can be perpetrated where data is stored transmitted and processed and some of the challenges you know that can

lead to this are the lack of logging or a failure to protect those logs on our part so first and foremost we've got to turn on logging where it's available and then in many platforms logging can be protected through multi-factor authentication to keep it from being deleted inadvertently say for example with a compromised password or a compromised token and then also there's a number of other things up here you guys can see such as protecting against account lock outs data corruption day loss and each one of these has a nuance for for a public environment or a private environment and a facility for for each one of those one of the the biggest issues that I've heard repeated

over and over again is a lack of object level or object access logging so what changes are being made in a V PC what changes are being made what commits are being made to a docker container you know each object as its accessed and changed should be it should be logged and if objects that shouldn't be changed suddenly are becoming access and changed that ought to alert you earlier today I heard it mentioned that you know unnoticed billing you know it is an issue and the example that I heard is a little bit different than than the example I'd like to pose to you right now that is if you're spending fifteen thousand a month on cloud storage for

example and you incur a bill for an additional fifteen hundred would you actually notice the difference maybe maybe not however in Amazon s3 that's something that you can alert on and additionally you can actually you can log you can actually alert on and you can actually create events based on on billing so things like turning up turning down machines or V PCs etc so overall you need a cloud centric logging and threat management solution that's going to help you to keep a very close eye on what's happening in these environments so I want to talk about integrity quickly so this would be corrupting data stream by altering the data that could be at the origin where that data is such as

persistence on that system deleting log data or deleting metadata or changing metadata it could be an example of that in flight so in transmission that might involve a man in the middle or gratuitous our kind of situation or something like that or finally at the destiny and that would be facilitated by some sort of fishing or drive-by download kind of attack that would give you that would give an attacker access to the destination system and be able to corrupt the data at the destination next up would be availability so disrupting the the job the transfer or the availability of that information via some sort of vulnerability scan those of you who are familiar with vulnerability

scanning on an enterprise scale will know that there is a very distinct possibility that if a vulnerability scan is configured incorrectly it can result in some very severe performance issues and and so precautions have to be taken especially during backup jobs to not produce a negative impact this could be a denying service of an API that's being used to conduct the incident response or even affecting critical infrastructure like knocking over a firewall or router so finally I'll leave you guys with this slide that I promised that has the references for you so I promise this QR code is clean and please go out and take a look at some of the references I provided out there and I think we have a

few minutes for QA so are there any questions no we covered a lot of ground I really appreciate you guys being here today you'll probably find me hanging around the the sans table talking to other students or prospective students please stop by pick my brain if you like and thanks again for being here and listening and I hope you guys enjoy the rest of the day