Structuring Cyber Incident Root-Cause Investigations: A Walk-Through

So good morning everybody. Thanks for having me. Um in this session I'll share um how we did the restructuring of our u incident response and root cause investigation processes. Um walking you through elements used to structure the process tools and the investigation itself. I am Joan and I work for Simmons Health, a med medical device uh medical technology company um based in Germany in Ellen but with a global presence. Um that's where I'm coming from. Um in this session um we'll have four main blocks. First and foremost the background. I think it's important uh that you understand a bit on what's the background that were inserted. um what does the threat landscape look like? Uh what is it doing incident

response for medical devices and a little bit um on our team journey that we are on. Um then we proceed on to the approach. Let's understand what are the environmental constraints in this setting and what needs the stakeholders have. From there on we proceed we proceed to structuring the investigation uh using uh an open- source framework um the FIQ that helps capturing knowledge. From there on there are some takeaways that we would like to share with you. Some of you might be in a similar journey and those should be useful. Thank you. So about the background um what is the situation like out there? Um how does the thread landscape look like in this healthcare sector? Um sad

reality many documented cases of very high bounties being paid to thread actors basically fooling this eagerness to hack and to sell data. That's a very bad thing. Um there's a huge incentive in these things um for threat actors to target the healthcare sector which is a very critical sector for society. Um cyber attacks are considered the major risk for for healthare systems and are very frequently if not every week we have news of disrupted care disrupted hospitals um preventing access to therapy or to diagnosis. So this is a very serious thing. But why do that do trajecctors actually target healthcare? Um first first and foremost healthcare is unfortunately an easy target is easy prey. Um Stefan

mentioned many of the basics uh and this sector unfortunately lacks many of the basic things in cyber security. So there is a large and interconnected attack surface. Um security practices are not the best. Usually teams are also heavily understaffed and suffering under severe pressure. Um we all hear how critical the healthcare situation is not only in Germany but all over the place basically. Second important factor here complexity. Healthcare is an ecosystem that has several legacy systems. Um security in these legacy systems was never built in upfront. Like when you start designing something you think okay let's get it right from the beginning but devices that are living for over 10 years maybe 15 years they were designed with

clinical aspects in mind and lately only with security some sort of add-on. Medical devices will also have will also be operated for very long life cycles 10 15 years. I mentioned from the clinical point of view they are perfectly okay. They deliver good images. They deliver good values for your blood uh uh for your blood uh tests. However um patches patching cadence that is being neglected. Yeah. Um and all in all there is a very strong digitalization push to increase efficiency. Healthcare is expensive. Let's make it cheaper. Let's make it faster. Let's make it more efficient. The third factor again that uh tractors and selling data on criminal markets uh health data is unfortunately on the high

price segment on criminal markets. So if you get if if a thread actor gets health data, chances are that he'll make good money out of it. just for that that that is why there is this uh target this focus on on the healthcare on on obtaining healthcare data. So that's where we are inserted. But as I mentioned, I come from medical technology company and I work on the product security team. Um uh we do uh incident response for our uh products and solutions. That's uh a different major difference between a CSR and a PSR. Uh maybe a quick show of hands here. Who of you are into incident response uh investigation root cause? Okay, so I would say less than the half

of the room, but at least some of you are into this similar setting. But how many of you are also in a product security team? Are there many of you? I see one hand there, two. Okay, awesome. Let's chat. Let's chat. Please come to me. Um so we do we do we do we do incident response for those uh products and solutions. One major difference here is that these assets they do not belong typically they will not belong our company they will belong customer they will belong to the customers. So they will be uh owned and operated by the customers. So we are inserted into this setting and then we come in okay something broke something is wrong.

Let's understand what happened here. Um but why actually do we need to investigate root cause? First and foremost, hospitals need to be back online treating patients. So recovery, you need to understand how the attack took place to make sure recovery is sustainable to make sure actually the measures you take reim is reimaging a system enough. You can only answer that if you see if you know what happened. How how did attackers breached your system. Second aspect also very important regulatory needs. Uh there are some uh regulations concerning patient safety and clinical quality. So whenever there is an there is a cyber security incident there is an investigation started and needs to understand was there actually any risk for patients or

any uh risks for um clinical quality clinical data. That's the second aspect. The third one uh which I think is the one that helps us on the longer term is to enable continuous improvement. So when you understand how your products were breached then you can derive improvements for the security architecture for threat modeling etc. as well as to improve customer care processes to understand what your customers need and whether your process are fulfilling their needs the journey. So we're bringing out the next version of this service and therefore we have a fantastic opportunity to review the process, the toolings and the investigation itself. That's the journey we are jumping over to the approach. Uh what elements we use here to redefine

this process tools and investigation. Uh the very important one the environment. So here is the environment where your forensics process where you are operating your forensic process where you're operating your instant response the environment will provide you with very important constraints that you need to observe otherwise your process will be prone to failure if you don't know if you don't know the context where your process is operating on. Second one, the stakeholders. Stakeholders will have specific needs, questions to be answered uh that you need to observe in order to have a good process. The third aspect here is specialist knowledge threat intelligence uh supporting you understanding okay what are risks in my process are there

things that I am I overseeing and then ensuring the process is complete. um these three elements and then from these elements you start deriving the requirements for process tools and investigations. Uh you understand also what kind of documentation documentation is especially important in my case because I will get to that later on. I am not going to the hospitals and acquiring forensic data someone else is doing on my behalf. So I have to make sure they have a easy life doing that. And the third one is when you know what the what the the constraints are and what the needs are then you can understand okay what do we actually need to answer with your investigation

take one let's talk a little bit about the environment when uh when we when I started working in this setting here I was interviewing the colleagues I was understanding uh from from th sharing groups um and and discussing with people having a dialogue to understand okay what are the fundamental constraints that we need to observe here. And first and foremost, whenever an incident hits and um affects operation on a hospital, the customer care teams and hospital staff will do the best they can, all they can to get systems back up and running to get back to treating patients. So that's the priority. Second aspect here, we discussed this a little bit. Medical devices will have very long life cycle.

So in your investigation you have to deal with legacy operating systems, legacy applications and the third one because priorities on recovery the devices will get reimaged very early in this process when as long as they can start reimaging device reconfiguring to get them back online. That means you only have one shot to collect all the forensic evidence you need to do the analysis in the lab on the right side. What does that mean for us doing incident response on products being operated by customers in these hospital settings? First and foremost, do not never ever get on the way of recovery. If you if you become a burden to them, that's not going to work. They will not perform the

collection that you need to to perform your job. Second one, uh you will have like strange and exotic stuff on the field. Therefore, know your installed base, know your devices on the field, know your capabilities uh of of your collection tools and ensure these tools are compatible with this broader range of device that will be on the field. And the third one is to for you to take as much data as you need but making sure you don't go beyond that. So basically you you have to strike the balance between a broader collection but still keeping the data that you're collecting in manageable volumes to not make collection data collection take too long. Yeah. All right. Just take

some fundamental differences and I'll try to speed up a little bit. Um IT and healthcare incident response have very important differences. uh in the IT area um you have things like telemetry and automation. This is common place. This is everywhere. We're talking about these EDRs and uh what have you detection uh security operation center detection and mon monitoring processes orchestration automation. So there is an abundance of tools for detection and containment on healthcare uh cyber security the incident response for for medical devices is a bit different. First and foremost, there is often no integration. Even if the hospital has uh a security operation center or some sort of security monitoring, not all devices will be sending data to these uh to

these central areas. Um the devices were also often designed with clinical features as a priority and not cyber security feature. I think during the keynote someone mentioned uh Martin mentioned um uh medical devices. No one is buy it's like it just has has to work. No one is going to pay extra features for extra u money for these cyber security features. And so that's that's that's the case. The focus is really on clinical uh aspects. Um therefore you have an scarcity of tools and methods for instant response containment etc. Second aspect here logistics size matters. Smaller medical devices what happens? Oh this device is broken this device is compromised. Someone comes in, takes the device out, puts it in a box,

ships it to the manufacturer, another device gets installed in and then the manufacturer can work on the lab. Um, most of our devices that's not applicable because those device on a MRI scanner or a CT scanner uh I just put a note there 13 tons. So it's a logistic nightmare to install one of those devices in a hospital. Uh so it's not possible to send them back to lab to me to analyze. So there therefore we need to do the collection on the field. Okay. Then we discussed a little bit on the constraints posed by the environment. This slide just uh illustrates a little bit of how do you start deriving requirements for your processes tools and for the

investigation. And I'll be a bit shallow here. Um for instance the first constraint here you're restoring operations uh because the focus of these field teams is not actually performing forensics on your behalf on the behalf of the product security team. The job is to help the customer get systems back online. Therefore keep it short. Second thing long life cycles uh you have to ensure compatibility of your tools. you have to provide alternative methods or even uh enable these teams to if that's necessary to collect a full disk image. Uh and the third one only one shot. That's my favorite constraint because uh that's the most interesting from the from the forensic engineering point of view. You cannot collect it

all. That's too long and you're going to generate too much data. It's not going to be practical. But then you have to prioritize and there are things like it's a shootout for ad adaptive collection. Um some velocity raptor just just mentioned this I think one month ago or a couple of weeks ago. Basically uh a side note because I find this topic especially important for my setting. Uh collectors this forensic collectors still have a static list of device of files to be collected artifacts to be collected. the art the the the adaptive one goes one step beyond this. it has a list but then some of the selected artifacts will be parsed and for instance if there is a schedule task

it's not automatically that a static collector will collect the binaries or the the files referenced on this schedule task the adaptive collection parses these entries and then will collect these related files as well meaning it's a very big advantage if you are on a similar setting as I am just to be very blatant and plative here that's incident response forological medical device. Nowadays, that's the common place. Someone is walking with a USB stick that was hopefully in uh sanitized and formatted before it was used. There is a collector installed here. This collector gets executed. There is a triage package being generated. Um that's when you can and I can start working. Uh so you have to make sure you

provide this plus a little bit of uh harder specs basically reliable stick so that you don't lose data and fast enough so that the right operation will be performed timely. Um moving on to uh stakeholder needs and I need to speed up and I think that's the message now. Yeah. Um so um simplifying this a little bit what personas we have here. Um and here uh I sum I I summarized this uh or simplified this into three main personas. The customers so basically hospitals healthcare deliver organizations responsible for business continuity and for uh patient care. Um and these folks the the ones there is always the kind of IT department in a hospital they are responsible for

installing um upgrades to address security flaws. um medical devices I'm not sure if you all are aware but there is nothing like Windows patch day and then you install all the patches from that does not work like that so the manufacturers they have to produce update uh packages and those updates are deployed on those devices that's how you do vulnerability management for those things the second persona here the regulators um those will define conditions so that you can approve a device on a certain market and then sell the device uh their focus is um patient safety and clinical aspects. Plus the third team, the third uh persona here, the manufacturers R&D. So basically our folks, our engineering

folks, our product development folks responsible for security de architecture of medical device safety and quality and they must deliver um upgrades to address security flaws. I will have to sum uh to shorten things up here. Um just as example of investigation requirements, four main clusters. So basically the scoping the first one containing and isolation the second one is enabling your customers to continue operations although the network has been disrupted. Um the third one being in the investigation towards the integrity of clinical data. Uh and the fourth one also very important one was that actually uh risk potential risk or or actual risks for patient safety or clinical performance. Digital forensics is an enabler to all

of those investigation clusters. Dig digital forensics is how you answer those stakeholders questions. And then we're talking about investigation. We're talking about knowledge. Again, I have to hook on your presentation. Uh Stefan um mentioned so many ideas that you can capture to increase your knowledge base on on digital forensics where to look at things. Um and the FAQ uh is a is an open source framework that I'm starting to adopt to start codifying these ideas and those details. Um this is just think about a catalog. This is just just a catalog with a nice structure. Um this is files are stored in YAML. So it's text format. It's easy to process. It's easy to maintain. Um

some benefits that I see from this deliberate thinking. So you're pushing your teams. You're pushing your analysts in thinking about questions up front and not when the f not not when the situation is on fire, not not when you have to explain something. So upfront. Second thing, repeatability and explanability of your investigations. And the third one, quality and coverage. If you have a road map, you will have less gaps on your investigation. The structure of this thing um scenarios facets questions and approach. Scenarios are really like the higher grouping. So how to start an investigation? If you think about an investigation, if you think about a scenario, you'll have certain angles. So certain facets of this investigation,

that's what they call facets. So basically some sort of intermediary grouping between a scenario and actual forensic questions that you have to answer. So the the the the questions is actually where the focus is here. What are questions and what are approach? So basically questions and how to answer those questions. That's the structure provided by this um um by this framework. I've brought a scenario but I think we have two minutes. Uh right then just very simple and going very quickly over this this is and it's a bit simplified but it's an idea to illustrate. Someone called in hey we need help. Uh there was apparently an infection of a system caused by a hard

disk or infected hard disk. Um then I received the triage package and I start my investigation and these are just like possibilities. Yeah. So you start thinking okay what devices what USB devices were used on the systems. Um and then you be you be a little more specific taking a look on removable storage devices and were the files actually transferred from USB towards the the the main drive of the device. Um was there maybe uh you have maybe metadata to tell if was there malware present on the device things like auto run or hidden folders. Um then you start proceed on to with your with your investigation and some of the hypothesis here. Um were there actually files

executed from this device? Um older devices they were talking about auto run here. Auto run seems to be a thing from the past but not in the healthcare realm. Um, did the user actually click something on the on the USB? There was a nice uh icon there saying please click here and there you have it. Um, and then as you proceed on deepening the investigation then you start might consider okay what are affected changes on the systems changes to users uh to user accounts created files how does the persist persist persistency wise how how is the situation run keys auto start items etc etc. Maybe we could even be so lucky from from parsing MFTt and see okay

there was a huge zip file being created that's where exfiltration likely is yeah all these things you can capture in the structure of the FIQ some of them are also already present in the pro in the project um this is just a print out of the website um and one example here was data excfiltration uh are there this is like the the the scenario the facet Here is are there signs of staging for data excfiltration? And then they have the specific those so-called atomic questions for instance were there files downloaded using a web browser and if you delve into that question they will even go as far as providing an approach with different techniques and different

data source and analytic techniques takeaways and I have to be very very fast but basically three main things. Uh I think Martin on the keynote already mentioned establish a dialogue with your stakeholders with your peers. So basically um yeah so first first the environment so know the environment where your process is inserted into um how do you acquire data is is something like remote executions of your collection tools available. Do you have telemetry available remotely? Um know your systems um is there people working on our behalf? So you have to know okay what are those people's capability how much time do they have to help you know your stakeholders so basically establish the dialogues what are the

objectives if what are their objectives in the investigation what questions they need to be answered from those questions which are the ones that must be answered urgently so as soon as you start going typically around scopy containment for me an example would be I need to tell the customer this and this and this systems were seen attacking our medical device so this would be a very important insight to deliver as early as possible to the customer. Uh while maybe there are others that could be delivered a little later on. So after uh uh after you did their scope and containment um what I call downstream products. So uh secure improvements to the security architecture or threat modeling or even

uh uh further techniques or further aspects of customer care process. um specially knowled uh specially specialist knowledge. Um so the threat landscape is also another important source for your uh collection and analysis requirements. Um what attack techniques are relevant for you. Review your past incidents. You will learn a lot uh when you review your past incidents. Make sure you're capturing artifacts that will enable you to answer questions around those incidents around those past uh attacks. um anti-forensics uh also something to be um to be considered. Um thank you very much.

Thank you for keeping everything on time. I must say as a person who has worked in incident response I appreciate your traffic light protocol clear and that you're talking about opensource frameworks that we can use to actually analyze and learn as defenders um how can we improve so um you actually answered my first question through the presentation of how to start root causing with the USB drive I'm glad that it wasn't with this kits or something like that but >> it could have been you don't Maybe the first uh question I have is what are the most overlooked factors that contribute to recur to recurring security incidents and maybe how can we use the framework um to

>> um address them? >> I think what comes to my mind with this question is really so what are the basics? Yeah, common factors, basics, um things like vulnerabilities, older systems, unpatched systems, uh weak credentials, um those are all very present in a sector like healthcare. So, uh again pointing at my my predecessor here, um you don't need to care about APS or very advanced stuff. Let's solve the the basics first. Let's address the basics first because when we address the basics first, we'll we'll have we'll already have gone a long way. >> Thank you. I love how the presentations all connect and Stefans and Martin's keynote. Looks like we have some recurring patterns to learn from and

bring back. Um any questions from the audience that um we can help answer? We have one question here in the middle. in the middle.

>> Hello, thank you. Yeah, thanks for for the presentation. Very interesting. Uh, one one question comes to my mind. You often mentioned that during your investigation process, you just need to get your data and do your investigation. But how often before the product is actually installed, do you have any interaction with the security teams of the venues where you install the machines to discuss whether there are some tuning you can do to integrate it with their seams or there is any kind of you know security measure they have that could be better integrating with the product you provide and rather than you know working on that when the incident already happened. That that's a very

good question and I understand you are looking specifically into are are we interacting with our product development teams correct? >> I mean more like I guess like you know Simmons has some products that you help uh offer um instant response when those products are hacked before these products are installed in hospital for instance. Do you have any point of interaction with the hospital security team? one one of the uh with the hospital security teams uh that's something that we could improve to be very honest internally we have we discuss uh attacks attack patterns controls uh are those efficient are those sufficients or not uh with hospitals we have exchange over the health health ISAC so there is enough

ISAC for the healthcare sector uh our company is frequently presenting there and bringing exchange especially vulnerability management is a very present topic if you um have more interest on this, take a look on the health is so it's a threat sharing group for the healthcare sector and you'll find not only us but some other some of the other vendors and and the key players from the ecosystem being coming there sharing insights and helping improve the situation as as a whole. >> Thanks. >> Thank you. Last question in the back. >> Thank you. Thank you very much for your presentation. Um, I would like to ask something related to what you just asked before. Um, how do

you monitor if something anomalous happens? >> Yeah. in your network at the hospital. Do you have some sensors that detect anomalies? >> Let me rephrase that question. Uh are we as a vendor of the medical device vendor monitoring devices on the field? No, we are not. That's that's when I mentioned this is a there is a there is a responsibility model on healthcare and that's these device they are owned and operated by the customers. Therefore, we support we provide guidance. there are even some uh uh initiatives towards that but that's something that's not done by the man by the manufacturer. >> Okay, thank you. >> Thank you. >> Thank you very much. Thank you Hall for

sharing some um interesting insights into the the specifics of the