Picking Up the Pieces: Incident Response For 3rd Party Compromise - Adam Rauf and Conor Osthoff

to introduce our next two speakers we have Adam Ralph and Connor ostrov Adam Ralph is a team manager working in Incident Response in forensics at a large regional healthcare company previously is worked at a managed security services provider solution Airy now NTT security as well as cert Adam holds a master's of science in information security and insurance from Carnegie Mellon University and a bachelor's in communications and a minor in mr. music from University of Pittsburgh currently holds his security plus from Campea DC IH certification from sands and is working on his CISSP in a spare time Adam enjoys playing music in odd time signatures obsessively discussing the nuances of flavors in food and craft beer and whisky and

spending too much money on scheme sales several of these I can relate to Conner ostrov is a senior Incident Response analyst with a strong background in healthcare and finance he possesses a proven track record of responding to compromises of all shapes and sizes he is an avid hiker and autumn their automobile enthusiast and finds enjoyment and difficult challenges both inside the office and outside of it Connor currently holds G AK certified forensic analyst certification and giuk mobile dev analyst device analyst certification as well so with that being said I'll go ahead hand things over to you guys and go ahead get started

good morning Thank You Zack for the introduction so I guess we can actually go right into the talk and skip probably the introduction slide because we already got that information but I'm Adam roof this is Conner ah stuff we're giving our talk here on picking up the pieces which is Incident Response for third-party compromise so this is some of the stuff you guys already know so we're just gonna move on from there the legal mumbo jumbo of course the opinions expressed by us this morning do not necessarily express the views and thoughts of our employer etc etc I think we've all kind of been through this in every presentation definitely familiar so the scope why are you here today it's

probably because you work in security to some capacity you find yourself as an information security analyst and you respond to breaches you work data breaches things of that nature or you are a privacy legal official who is worried about that sort of thing and as I think they are you may handle with data that comes into your organization and you want to secure it as well if you're worried about how individuals working with your data are actually treating your data then this talk is for you as well so we're gonna make this a little bit more interactive right in talking to many of you who work in security in different capacities you guys can just shout out a couple of

things what are some of the first things you do whenever you find out that there's an incident in your network take it off the network anyone else would panic okay I like that what's not bad no do that what was that yeah so what's that call your chain of command that's good yes imperative so we'll talk through some of the things that we've seen is the first party and then kind of talk through what you do as a third party right so a lot of you are probably familiar with the sans pickerel model this is one that a lot of it as a shion's have adopted looking at preparation preparing for any type of incident that comes in the organization

identifying the incident as it comes in containing it eradicating it recovery and most importantly and it's usually skipped over as lessons learned having those after-action reports you probably review your logs you're looking at firewall logs proxy logs AV EDR if you have it system logs DLP mail gateway if you have an IDs or IPs and I think Jay touched on this earlier but training communications so if you need to reach out to your C cert if you have a particular chain of command you're gonna follow in a lot of cases even in our in our environment we usually reach out to privacy because we like to make them aware of any incident in case it has

privacy leanings you might reach out to your legal counsel hree are depending on you know what the type of scenario is you'll probably talk to somebody within the do that disaster recovery space or within corporate resilience and there's probably even more than that in my place of employment we use the pickerel motto in a sort of forked variant so to start things off we have the discover phase that's the point in time where the event is generated the work is a sign of the appropriate parties and the escalation occurs if it is needed in the investigation scoping phase the second ones we have here our event monitoring team spends about an hour's worth of time to triage the event

gather different feelings on the types of data that are associated with it and try to build a case from that if a true positive is I don't know then escalation occurs at that point as well it is logged within the incident response ticketing platform and an escalation to the incident response and forensics team occurs at this phase as well communication with privacy if needed also happens if there is a cyber security incident with a privacy implication we absolutely want to have them involved and lastly for this phase we hold in what we call an IRT meeting so an incident response team meeting kind of gathers all the people from Caesar to get them all together have

everybody on the same page and progress from that point so after we discovered the incident we understand the full scope of the incident we've made sure that we're not missing anything we've began working on containment so we analyze the criteria for containment we start to build our containment plan and then after we determined a plan we have the necessary sign-offs we implement that containment plan once it has been implemented we verify the effectiveness and we make sure that we're not seeing the same types of activity the eradication and mitigation phase which is the next one here we plan and execute our remediation actions we determine all activities we assign all of the appropriate resources to those

activities and the appropriate communications happen here as well the impacts of the eradication of mitigation plan are assessed and the effectiveness is evaluated if we find that we have implemented this plan and we still do not have for mediation then we go ahead and reorganize our plan to keep those gaps in mind afterwards the recovery phase so if the eradication and mitigation phase caused any issues we work diligently to restore normal business operations and that's likewise for the implications of the cyber incident itself we generate an incident report and deliver that report to the necessary individuals and we changed the incident to closure status and lastly here we have the follow-up phase where we identify and review our lessons

learned we begin long term corrective action and kind of keep all of those things in mind moving forward pick those gaps out and make sure that we're not susceptible to them again in the future okay so here are some of the don'ts of a first party incident so if you're the organization that suffers the incident or you happen to be the third party of an organization that you handle first party for one of the first things is you don't ever lie to your customers if you have work to do if you need to figure out what's going on don't tell people that you're going to have everything resolved within a couple of days if you need a week to

figure it out don't trust the assurances you receive from criminals there are many cases where things like ransomware of head organizations and what they go back and say is well if I pay the ransom they'll give me my data back right in a lot of cases that doesn't occur conner we'll talk a little bit more about that in a moment don't promise upfront answers to your customers if you can't deliver so once again if you say that you know we'll have all systems restored within the next 24 hours and it takes twenty-five hours that first hour right afterward your customers are gonna say these guys lied to me and it's going to ruin the faith in your organization

finally don't blame one person for an organizational wide failure if there's an issue that occurs a compromise that occurs within your organization go back and actually take a look at is this a people problem is this a process problem is this a documentation problem is it training is any any of those things because yes we are always going to have a weak link with with humans OSI layer a right but at the end of the day if something else caused it and caused that person to make the mistake try to go back and understand the root cause and that's a big part of the lessons learned piece on the topic of assurances from criminals I think this incident is

familiar to most of us we've seen in 2016 it was not fully reported until the end of 2017 it was a it was a large company by the name of uber and about 57 million records if I'm remembering correctly the implications of this are the threat actor had this data reached out to the organization and essentially said hey I have this what are we gonna do about it so they think through different plans what can we do and they decide that don't turn into a ransomware case they will bribe the attacker will pay you a hundred thousand dollars if you keep quiet and then let us know that we have anything to worry about so they did

such actions they took faith in the assurances that the criminals or the attackers had provided them and this fee was paid seeing as it made its way to headlines its I think it speaks for itself so going on to how to handle a third party incident I think the things on the slide here we've all probably tried it at some point sometimes giving to our own frustration but if your experiences or anything like mine they don't tend to help you very much so we'll go into a few things in detail of how we can respond to these types of problems so the first step is prevention you can't fall for a phishing scam if you don't check your email you can't

have a third party compromised if you don't work with any third parties so let's just stop working with third parties altogether it'd be a good idea right well the world's not like that anymore with the way that people are moving into cloud the way that people are outsourcing some of their work we can't really do that and so a big part of that is trust so we are at this point trusting people we pay to do a job cheaper out of house which means they're going to cut corners in a couple of cases as we get later on to the horror stories is that we've met with many third-party organizations where we ask them for their security folks and they

don't have any it's usually one person who works probably you know 70 80 hours a week doing all of the IT work as well as having to do security so when an incident occurs they are totally burned out and oftentimes they don't have anybody on retainer so that's you know unfortunate and we see that with a lot of organizations another thing to think about is putting as much distance as you can between you and your vendors and third parties so if they have connections into your network to FA or MFA definitely a good idea limited access so principally fleece privilege we see a lot of cases where vendors will have ad access or direct RDP access to devices

they're not segmented they're not on a VLAN so network segmentation tying into that and then the other important part is logging everything within that network we've seen a case where there was a fire alarm system that was compromised and there were no logs of that fire alarm system so those are the kinds of things that you want to be on the lookout for and one other thing to think about is that vendors and third parties will often have multiple clients so in short they care about you just as much as they care about their four hundred other people when responding a third party compromised communication is imperative you need to ensure that all internal members of c-cert understand

their roles so while the media relations team is comfortable working with privacy and legal they might not necessarily have the needs of corporate resiliency in mind and they might not necessarily understand all of the implications of the cyber attacks that they can convey it for a lot of these people who may be spelled out in your cyber incident response plan they understand their roles and responsibilities on paper acting the mountain person is a completely different story usually we find that for a lot of smaller organizations especially whenever there is a breach that needs to be disclosed this is often the first time that any of them are even in the same room talking about it so if you already have one very

large fire it's better to have the fine details and the finer points of communication worked out beforehand to make sure that you're not Fanning those flames in the chaos that you create trying to start your response actions if you have a contract with the third party and I would assume that you do it is definitely a good idea to work with your legal team and find out what you can leverage contractually anything that can give you extra power to make them do what you want which is give you a comfortable feeling that the threat is contained and resolved and explained that you aren't willing to compromise if they still have an active threat if there's still an active breach on their

network take a very firm standpoint and explain the normal business operations when I'll be resumed until that threat is at least under control right so moving on to the next section this is going to get a little bit on the drier side but we actually developed an in-house party compromised checklists that we worked on so Connor John Wolfram is formerly on our team and many others within our Incident Response space came up with with a good practice for us to follow every time we ask third parties for their data whenever a incident has occurred so this is kind of the checklist that we use you guys probably can adopt something very similar if you haven't in your in your incident

response plans but we'll go through these a little bit in depth so the first thing we usually ask is on the left side we start with requested items and on the right side we ask them to identify where they received it from which will include the date of when the when they have the logs from what kind of logs they gave us and what platforms this touches so first we start with the evidence of alert detection so we ask them for you know standard AV EA our hips ids/ips type logs any network forensics artifacts that they can share so that might be NetFlow web logs pcap firewall logs firewall configs email attachments all system change controls and

configurations so this often gets glossed over in in terms of a lot of compromises there are cases where it might be an access permissions issue of something of that sort and if there's a change record that's associated with it we should be able to try to trace a why did it happen and B when did it happen to understand the entire scope and why the dwell time is as large as it is we also look for any endpoint forensics artifacts so this might be log files or evidence of ransomware or malware and so in each of these sections we also ask for any additional notes attachments if they can give the sender and how they

transmitted this information to us whether it's email a delivery or some other method we also ask for evidence of vulnerability scans so if they've done any recent role in really scans and there are third parties and vendors who don't do vulnerability scans on any regular basis but in the case that they do we do ask them for this to identify the scope of the detection and then if they've got IOC scans that they can share with us we look forward to that as well any files that are modified by malware or ransomware or anything else are a classified as they're a portable reach under US law due to those things we ask for the following DLP logs so evidence

that DLP is implemented and it was implemented at the time of the intrusion report evidence of any company data loss to the company that you yourselves work for provide the criteria that was used to perform any DLT searches so that we can review that criteria and determine whether or not there the DLP logs have the integrity that we would expect malware identification we ask for variants type version research data and family so that we can understand all facets of the threat that we're currently facing on their behalf we ask for forensic evidence as registry key changes process hash downloaded files command and control IPs and domains installed locations propagation techniques the exploits of is used and

any other pertinent IOC s so that we can use those things to search for evidence of that malware operating in our environment we also ask that attack vector and root cause be documented in terms of payload delivery so documenting the appropriate evidence of that showing the appropriate logs so that we understand the different phases throughout the malware point of infection and also propagation so document exactly how the infection spreads and how it did spread through their network moving along we touch a little bit more on breach response so asking them to provide evidence that no company data has been modified or exfiltrated each infected host in some cases that might comment is an attestation letter that's the

opportunity for them to either attach that letter to it or speak directly to privacy teams legal teams we also look for evidences of a remediation plan we have a couple of areas for this so one is to maybe at least give us the logs of the IOC scans that they have that would validate that they've wiped and remove the malware from the network we've run into cases where the vendor gave us the assurance that yes we've removed all threats from the network only to be reinfected only weeks later we've also asked for documented mitigation plans so this is to prevent future infection this should be the the problem management or the after-action reports that come out of

this they should be giving us an understanding of why this happened how they're going to prevent this from happening again so we also ask them for things for example if the compromise came because of in on you know an unpatched system we want evidence of what they're patching schedule is and what they're using to patch that particular exploit a vulnerability scan so again if they didn't have vulnerability scans in the past we want them to begin a vulnerability scan program and tell us how often they're going to do it there's a lot of free and open source tools available so it's not like they need to go and pay somebody to do it but if they

can't that's also an opportunity for them to bring somebody in to do it we also ask for evidence that the network protection stops propagation so did they put in a VLAN did they do a layer 3 segmentation have they put any other steps in place to prevent the infection from hitting multiple systems and networks that once again we also look for a modification of protection and detection vectors so that again once again looking for a possibility of reinfection so even if it got compromised by you know and open vulnerability and a pop3 server what are you doing for that pop3 server and also what corrective action plans they're taking so I mentioned earlier that you

never really blame an organization-wide failure on one person you should blame it on what's going on with the process what's going on with the training what was missed so if it is something like fishing okay well do you have a fishing training plan in place if it's something where somebody you know accidentally set access permissions okay well are you retraining that person on how that's done are you putting additional controls in are you putting in other technical controls to prevent that from happening in the future on top of all the things that we've asked for so far we ask that everything be included in an incident report so we can send out that incident report to all

the different members in C cert who may work with the security representatives or the legal representatives of the third party compromise at a different capacity a breach response notification so the report that is provided to the privacy department we have to make sure that that meets all legal requirements and also have our opportunity to feed data into that as well if the type of work that we do with that third party requests such behavior and also to list agents scheduled to work on the audits requested in this document so we have accountability for all of the things that were provided here and if we have later follow-up activities we know exactly who we can take those topics to

so I mentioned earlier about the networking logs at this point if they can provide us with the actual pcaps or any of the actual log files we do ask if they can include that ditto for IDs and IPS logs so if they can give us evidence of the payloads that they've received also for any of the signatures that they use to map this against we requested at this time of course the methodology for receiving this might not be something that's as simple is you know adding it to an email we might ask them to send it via something like a delivery or SFTP patient zero monitoring so again we kind of go back to where did

the actual infection occur and how did it occur can we look at was this tied to a particular account so if we look at the case of a phishing incident did they go back and reset the password for the account did they give the user training are they monitoring that account for more potential activity in case the actor remained persistent within the network what's going on with their email are there any other emails that are coming in are they harvesting credentials is there any strange web activity so looking for things like C - are there other malicious domains that the user is still connecting to maybe unbeknownst to the user are their browser helper objects that are still remaining on the

system a comprehensive review of that system as well as understanding if the endpoint itself is infected what have they done to wipe and reload it are they doing you know a nuke from orbit approach and just seven past the ban on it or are they just going to say we're just gonna reinstall windows and hope for the best we want that evidence for the current AV status we want to understand why the AV might have missed it so if there's a particular compromise with a piece of malware we might say you know what percentage of your hosts are currently loaded with the version of AV that you have how accurate are the definitions and signatures are they

recent do you still have a license for a V we have run into cases where the vendor has had an expired AV contract and so they hadn't had AV updates for about a year is there proof of the ability to detect the future threats on the same signature so if it's a particular piece of malware once again our particular piece of ransomware would your EB catch it the next time that this happens my personal favorite part the horror stories we've all been there the the things that we like to tell our friends after work the different things that we were playing the grates in our mind they're very frustrating at the point in time where you're working

through them but afterwards they usually give you a good laugh so to start this off medical service vendor that we used to of some capacity is breached by ransomware now it's not the vendor that reaches out to us to tell us that piece of information it is actually received by us through different ocean platforms open-source threat intelligence platforms so we work through the data that we get we reach out to the vendor through our privacy and legal team we say it we've read we've received an inclination that you have been breached by ransomware so they say no it didn't happen like but what we heard from a few different sources that did happen public sources public sources yes did it happen

now didn't happen couple days go by we started to see headlines on it it's has a pretty good a bit of know of media attention they still haven't issued any press release at that point in time they they start to fess up they start to tell their clients yes this happened it's all completely under control don't worry about it a few more days go by we find out that it's not under control at all it is actually spread through their entire network it is completely painful and once we finally reach a point where they're willing to be completely open and honest they have nothing left to hide we do find that they had incorrectly addressed the situation from

the starting point so they claimed that the attack had taken place over SMB was actually slow brute-force RDP so I got the infection vector wrong also they claimed that this was a never-before-seen version of criminal ransomware when in reality it was just commoditize Sam Sam second-year hospital chain is breached by ransomware this is a hospital chain that we at Primark work with to a pretty decent capacity and they report it to us we ask them okay what are we dealing with all they say is ransomware oh how did it happen to what capacity that it spread oh we can't tell you that well can we get any IO seized at all no can we have any information can we have

anything no we can't share anything we also expect that you are restore that site to site VPN connectivity that you had removed at the start but sir we can't do that if you still have an active breach on your network regardless of the vector through which the spreads well you're gonna have to we're not gonna provide you any information that was pretty much the extent of that we had a trusted third party that required forensic analysis they had a compromise and so we reached out to them and asked them for the very things that we told you in our compromised checklist they reached back out to us and said we don't have a security team we don't know how

to handle this we need your help and try to figure this out so we said sure we'll take a stab at it we'd be happy to help you out and so we asked them to send logs they were unable to do that they said how about we just send you guys the machines can you guys maybe do the forensics on it and this is a offshore vendor right so we reached back out to them and say okay yep go ahead and send it to us you know make sure that goes through customs make sure that you have it signed for and everything they agreed to mail us two of the machines that were infected one of

them was lost in the mail they did not buy insurance and even the machine that came to us stateside did not require a signature so it could have very well just been intercepted the mail or lost or given up to anybody else we had a call center services vendor that was breached so when they had it we actually found out about it kind of at phase three they told us about a week after they were actually initially infected so at that point we looked at it and said well usually this is kind of phase three of an infection so when did you guys actually get infected and they said oh it happened on Saturday but just that's

what happened and we said no that's when you got hit with the ransomware piece when did you actually get infected oh no it was Saturday that's definitely that and okay well do you guys have a security team that's gonna take a look at it no but we have IT staff okay so you have your IT staff take a look at it so what are you guys doing to prevent this from happening again oh well we already started reimaging the devices okay well did you do any backups did you take a look at you know the forensic evidence oh no no we just wiped and reloaded it we just figured we just try to get back to business as soon as

possible and so through our threat intelligence we had determined that they got hit with a piece of ransomware that was part of stage three or phase three of the initial attack so the initial attack actually happened probably about six days prior started at stage one moved on to stage two which was a dropper and then stage three is where they actually got infected it's anybody in the crowd have a horror story they want to share I be happy to bring my mic down if anybody's feeling confident come on I know we all have one no takers all right we'll move on

okay so I guess we actually wrapped up a little bit on the early side unless anybody does want to indulge us with any of their particular horror stories sure and we figured we'd also leave time up for Q&A yeah there we go so Mike please sure

the best part with these is that these horror stories help us be better so so the horror stories I recall it's been a long long time I do again is first outside a certain or institution was popped and an outside source told us that the institution had been popped and it took and and this was a nation-state actor that had done this and where's the speakers that I don't get feedback okay nation state actor had gotten inside we think and contacted a certain three-letter organization the FBI and the corporation who was going through a merger at the time and it took us about a week to get a non-disclosure agreement in place with the FBI before that we

could come in and get the FBI to do the investigation and by then you know Big Ben by then the matter had kind of disappeared and people didn't really seem to care anymore it said horror enough for you Adam sorta kinda that it makes sense I was trying to keep it you know not a lot of names sure thank you any other takers last chance sure okay you were talking about collecting all the evidence after the breach what are you doing in your contracts to make it so the vendor provides this information because I'm sure a lot of people don't want to share that information it's confidential and then once you get it how are you protecting

that information so that their information doesn't get disclosed so I'll I'll speak to this first and I'll let Connor come in after that sure one of the things that's happened because of all of these horror stories is that we've gone back to our contracting parts of the organization or procurement area and said that you know when we have these cases this is the minimum that we expect from our vendors ditto for understanding for vulnerabilities so as we see vulnerabilities within the organization within their particular organizations we ask them you know how often can we come back to you to make sure that you're doing this that you are patching on a on a reasonable basis I

think the bigger challenge lies in medical device security because there are a lot of folks who probably work in healthcare who understand that the medical device space is a big challenge right now so what we've had to do in those cases where it's been a medical device vulnerability is having to go back to those particular vendors and saying you know hey we know that you know you guys were able to mitigate this particular problem but when this happens again we want to be able to reach out to you and have competent staff that know how to handle this problem and that we want to have something actually built into our contract with you so that if

you breach this we actually will be able to you know get you for a breach of contract there is also the legal strong arm so things at the end of the day it all comes down to money a company that's been breached they're likely spending a whole lot of money trying to get their systems in pre breach state at that point in time it's very convenient for our legal team to mention that we pay you X amount of dollars if you're not willing to work with us on this then I don't think we can provide your business in the future so that's another option as well that answer your question

Oh

yeah so I'll speak to it and I'll let you chime in here too and I think Ken also had something right after this but the question from Lisa was around yes that's what I was going to do was around what do we do with the information that the third party provides to us how do we keep that safe and how do we keep that kept under wraps so I mean at this point when we receive the incident we actually have an incident management platform that we keep our information in and so we added as part of that case and we also add it to the after-action report that is only accessible by members of the incident response team and of our

c-cert there's no one else who has access to that management platform to review that but when we do have items that we can share we do have meetings that will schedule with the various stakeholders to have those conversations to talk about here's where we have you know this the information we received from the vendor in cases where it's something like logs we'll ask them to send it encrypted we have you know 6 to handle our encrypted mail as well as opportunities to do so by a SFTP and a delivery mechanisms to get that information you add to that corner I think you did a great job good and so well so I actually had a question about

one of the horror stories of yours the hospital chain so you kind of trailed off there at the end of the story but it sounded like that discussion about restoring the site site VPN may have climbed the ladder a few steps before you came to some agreement on a yay or nay for that we're just curious what sort of preparation your management chain had and sort of how that debate went and how how much traction you guys had with leadership and how that finally fell out I would say that it was a losing battle from start to finish we did have all the necessary parties on the line and what it came down to at the

end of the day is this is causing an impact of patient care people can't come to our offices and get radiology IT services you need to do this right now and you have to put your best foot forward and believe that we've done the work that we say we've done so it was not a very open and communicative process much further down the road we did finally get all of the information that we were looking for also I I think I believe that the person who was refusing to give us information is no longer with that organization so oh I didn't know that happen all right get it taken care of sure

I work on the network team for a third-party vendor and in the incident that we have it like an oopsie Daisy and we have to provide these things we have many customers data commingled in our logs how do you manage it so that you can extract it out so you're not providing another breach whenever you provide those logs to a specific customer I would say that you guys probably have a challenge on your hands when you have commingled data I would say that in those instances do you have any of the customer data that you have within your organization segregated at all in any capacity it's all commingled [Music] so that I would say is a hard problem

what I would suggest is in those scenarios if you can the best way to do it is to you know be surgical about it and I think that the the hardest part is one of the slides that I mention earlier out don't lie to your customers right so if you are the third party that now has to deal with this breach that you now have to inform your customers about don't tell them that you'll have the logs within a couple of days in fact I think they would appreciate the fact that you say hey you know we're working on it let's have date you know we'll give you a daily status update I think just being more communicative than not

at all we'll probably ass wash a lot of fears that organizations have I mean I'm more willing to work with somebody who says hey I'm working on this and I'll give you an update every couple of days just to let you know how it's going versus I'll let you know whenever we're done or hey I'll have this in 24 hours and they just not respond and not return that information anybody else all right thank you very much everyone I don't remember the only limit is yourself