Adventures in Edge Device Memory Forensics

Okay. So, hi, I'm Rich Tuffin. Um, I guess to start the theme, I got involved in tech programming my dad ZX Spectrum in Basic. Uh, I wrote my first uh hacking tool, password reinforcer on an ARM 2 that ran at 8 MHz and 2 megabytes of RAM. Um, and I've been working professionally in cyber security for about 20 years now, which is a bit scary. Um I've never actually presented in public so you know every day is a learning day. Um so I've been with Vexity for about two years. Uh and we specialize in uh cyber security products and services built around memory analysis and memory forensics. Um and I started getting involved in a lot of

cases involving edge devices. Um which is kind of handy because I used to be a vulnerability researcher working on edge devices. Um and I found there's not a lot of public information on how you apply memory analysis techniques to these kind of systems. So I thought well that's a good opportunity. I'll turn up at B size and present on how you can do that. So what am I going to do? Uh I'll start just by introducing why edge sec device security is a problem. Talk a bit about why memory analysis can help you with that. Uh we'll talk about how we collect memory on edge devices. >> So sorry I was what is an >> I'll explain that in a sec. Don't worry.

um how you can analyze memory from edge devices talk about a couple practical examples and then and I'm standing between you and lunch but there should be time for questions at the end. So what do I mean by an edge device? So some kind of self-contained virtual or physical device that performs a particular network function. So a really common example uh the box that your internet service provider sends you, you plug it in, it produces Wi-Fi. good example of an edge device sits between the internet and your network. Um, a lot of corporate versions of these things these days you can also get as virtual devices. They run as virtual machines either in the public cloud or on your

enterprise. Uh, but do very similar things. They filter traffic. They inspect traffic. They load balance across your website. Lots of different things. Um, they usually have some kind of custom web app or command line for management. And you buy these things as a black box. You don't go digging underneath. You don't really update them. You don't look at the operating system that's underneath. You just read the manual, configure it, work with it. Historically, when I started looking at these things, um, a lot of them ran proprietary real-time operating systems because they were operating on network packets. They needed to be really fast. Processors weren't that fast at the time. You needed every every millisecond you could get. Um, these days, most of

them run Linux. Not all but almost all of these kind of systems run Linux. So why do they run Linux? Um one of the main reasons is vendors want to run on offtheshelf physical and virtual hardware. So if you want a virtual device that runs in Amazon's cloud, Microsoft's cloud on premise um virtual machines, easiest thing to do is to run Linux because all that compatibility is built in. And a lot of these vendors don't build their own hardware. They buy offtheshelf standard server hardware systems. um put a snazzy front panel on the front, put whatever accelerator cards they want in the back um and sell it to you so they can just run on commodity

hardware. But if you do have some kind of proprietary feature, so you want to implement some snazzy VPN or you've got some AI accelerator card and a driver for it, the advantage of Linux is it's open. There's quite a pool of engineers who can um develop that for you uh or quite easy. And the final reason is it's it's low cost. Um, so the actual technology itself is obviously open source and free and you can buy or get open source toolkits that do a lot of the hard work of how do I package it up? How do I turn that into updates? How do I add other packages? All that kind of thing. It's kind of done for you. So it

reduces the cost to companies of um kind of putting this thing out to the market because they want to be a load balancer or a um snazzy AI data loss prevention tool. they don't really want to worry about all the underlying how do I get network packets out. So, what are the problems with edge devices? Um, well, they're often exposed to the internet by design. A VPN that doesn't let your employees connect from the internet is a fairly useless VPN. Um, the standard, oh well, I'll just not expose it doesn't really work for a lot of these devices. They're supposed to be accessible from everywhere. They're not always the most secure of devices. And one thing that I've often

talked about over the years, just because it's a security device, because it's a web application firewall, because it's trying to make some make your organization more secure, um does not necessarily mean that device itself is secure and well built, particularly the management web interfaces. So most of the vulnerabilities you see in these kind of things are in the management web interfaces that you use to manage and administer the box. Those were never intended to be put on the internet. Um, but people do all the time. They're really attractive environments for attackers. So, not only is it the initial point of entry, but it's often a place that lots of really interesting traffic goes. You can see all of the VPN

traffic. You can mess with that VPN traffic. Uh, you might see all of your customer website traffic. You can put interesting things in that um traffic. You might be scripping your HTTPS to your website. So, all of a sudden, you can see lots of credit card details. lovely places to be. And because underneath this is a Linux box, all my standard tools work. If I want a socket forwarder or I want some tool to uh capture data, I can just go grab an open source tool. I probably don't even need to compile it. It'll just work. Um with these older historical um proprietary artses, you have to do loads and loads of work to figure out how to run your

own code. Um really lowers the bar for entry. uh and they often lack the observability tools that we have on like standard servers and endpoints. So you don't have any antivirus, you don't have any NDR and you've got really long limited logging and inspection just what the vendor chooses for you. So what's the impact? Okay, so you've got potentially open door to your systems that's really difficult to monitor, that's really difficult to inspect and is a great place for attackers to cause trouble. Um, so if you follow the cyber security press, you've probably seen a whole whole load of examples about this. I'm deliberately not going to pick out any vendor in particular. This is an

industrywide problem. Um, pretty much all of the major vendors at some point in the last couple of years have had a fairly significant vulnerability, a fairly significant campaign of people actually exploiting these for real um, and using them to cause pain for for companies and organizations. Okay, so that's the problem. How can memory analysis help? So first off, let let me be clear kind of what I mean by memory analysis. Uh so I say capture the state of the system by making a copy of physical memory. Um my kids are four and seven and ask what do I do for a living? And I say I help people take crime scene photos of computers. I take a very

accurate, very sharp picture of the state of that computer at the time and then I help people dig through that um photo with a with a magnifying glass and work out what's gone on on that machine. Uh so yeah, use analysis tools to capture and understand what was happening, what processes are running, what network connections are open, what files are open, and also what's happened in the past. So what's in the memory of your machine now is not just what's happening now, but also vestigages of what's happened recently. So network collections that have just closed, files that you read that are still hanging around in the cache, uh command history, most shells will keep your history and

they'll keep that in memory. Um and logs, even if the logs haven't been pushed to disk, chances are they're hanging around in memory. So why do we want memory analysis on edge devices? Well, because they often have a very in very limited ability um to inspect them. It's whatever the vendor gives you. The vendor will probably give you a few few commands. They might give you some nice little pictures in the web interface that says, "Yeah, yeah, it's all fine. The CPU is fine. We're not using much memory. It's all great." But you can't dig right down to say, "Okay, what exactly are the Linux processes that are running on there and are they supposed to be

there?" Um vendors just don't want to give that to you. Um the other thing is they often run very largely in memory. So with whereas with a traditional Linux server you might be pushing quite a lot of stuff to disk. Uh things like swap files, things like hibernation files, things like um a lot of detailed logging. Very often with these edge devices, they boot up out of a compressed firmware image and then everything is memory. And if that system gets rebooted, it's gone. you've got whatever the vendor chooses to log, which is typically an awful lot. So, we really like the same kind of gold standard memory analysis we get on standard Linux, Windows, and Mac OS

systems. We want a really in-depth understanding of what the attack has done on this system. We want to be able to triage really quickly. So one of the nice things about memory analysis is you can very quickly include or exclude um a system and say yes I'm pretty confident in from what I can see nothing malicious has happened here let's move on. Um and it's also quite difficult for attackers to disable or evade. So one of the things you'll commonly see with edge device um compromises is the first thing they do is they turn off logging. they disable your ability to take whatever um the vendor's got for kind of providing information uh and then you're pretty

blind. Um being able to defeat a full memory capture and completely hide from that is pretty tricky. You usually leave some kind of trace that an analyst can go and follow and find. So I'm going to talk today about Volatility 3. Um so probably the leading open-source memory forensics framework. um written in Python, runs on the command line, uh plug-in based, uh fully open source, supported via a US uh foundation, the Volatility Foundation. Uh lots of powerful analysis plugins for MacOss, Windows, and Linux. Um there's lots of training, presentations, resources. I should say I mostly professionally focus on the kind of capture side of things. I use volatility 3 a bit to kind of prove that what I've

done is correct, but there are lots of resources out there. I've highlighted the foundation website and the foundation's GitHub as kind of good places to start if you want to know more about volatility 3. Okay, so how do I go about collecting memory on an edge device? What do you need? So you've basically got three options. The first one is you can get privilege if you can get privileged access to the device. So the ability to download your own binary, run it and thereby collect physical memory. Um, not all devices allow that. So, some of these edge devices will give you essentially a root prompt if you type the right commands in the right sequence

um and ask it nicely and say yes, you promise not to break anything. Um, but not all do. Some vendors lock that down and don't give access. The second option is that the vendor has privileged access. So, for some vendors, if you open a tech support case with them, they can remote into the box, they can run a privileged process, and they can capture memory for you. Um, not all vendors will do that. Um, which vendors will do it seems to depend a little bit on who you are when it is, what's going on. Um, and it does require giving the vendor access to your device, which depending on what the device is and where it's running,

you may or may not be comfortable with. And the final option, which actually I've been surprised how often this is useful, is to take a memory snapshot. So if you're running a virtual appliance rather than a physical appliance um you can ask the hypervisor so be it Proxmox VMware um Microsoft hyper whatever to take a snapshot of that machine for you that will include all physical memory and that's a really useful analysis technique. Um it obviously only works if it's running as a virtual machine. Um and unfortunately none of the public cloud platforms I'm aware of have this functionality. So this is private cloud only. Uh okay. So collection tools. Um so I should probably say like we as a company

produce a collection tool. Um this is besides it's not a sales pitch. There are lots of commercial open source tools available for memory collection. Um they all use one of two approaches. One is to load their own load this kernel module. So that requires that you have enough files about that specific system to be able to compile a kernel module. Um, and it requires that you've got enough privileges to be able to load that kernel module and that should then dump the memory for you. The second option is to use uh a Linux kernel feature called prok core. This is intended to be able to attach kernel debuggers. Most uh desktop and server Linux distributions

enable this and actually a surprising number of edge devices if you can get in have this enabled. So generally if you can get privileged by which I mean root really access at least one of those approaches will work. Um and if you kind of go through the set of tools that are available if you can get that privileged access uh usually you can collect memory. So that kind of gives us a grid. Um so depending on where is it running and what access have I got to the device generally I can collect memory. What happens if I end up in the two red boxes here? So I've got a device where it's physical and the vendor is not letting

me run my my collection tools or what I've done in the bottom right hand corner. So I can't run the collection tools and I'm in public cloud so I can't ask the hypervisor for a snapshot. Well, during the middle of an incident, you haven't really got an option. You have to move on. Um, but if it's your network, you can choose to change vendor. If it's your network, oh, jump too far. You can change the platform. So, actually, if you're thinking about this up front and thinking, how would I do forensics in an incident, you can help yourself in the architecture. So if you've got appliances you know are high risk, you can choose to move them and go actually

I improve my forensic posture if I use this different product that does this give me this capability or I run the same product but actually I'll migrate it into our on premises instances rather than running it in public cloud. So yeah, uh otherwise you just have to use the best information you've got to triage it and often end up replacing a box that you're not sure if it's compromised or not because there's nothing else to do. Okay, so we've got lucky. We've got a vendor who supports collecting memory. We've managed to collect it. We've got a massive great memory dump. What do I do now? Well, doing memory analysis requires two things. The first is it requires what I always

refer to as types. So this is what's the structure of kernel data. So a task strct for example on the next describes a process. So what's the process called? What files does it go and all of those things is all in a big cruct that you can go and inspect. You need to know what that looks like and you need to know what that looks like on that particular device. And the other things you need to know is symbols. Where is the data? So it's no use knowing what the data is without knowing where it is. So for example on Linux init task describes the very first process that has started um on a Linux system and

from there you can walk the list and the tree of all of the processes and find every process that's running. But if you don't know where that first process is then you can't start that process. You need both of these for useful analysis. And I'll show why that's a problem. So I want something simple. I want to know the name of my process. So if I have the right structure, I have all the right offsets, it's all in the right place, but the symbol's wrong, then I put that over my memory capture and it's in the wrong place and I think the CM the process is called some nonsense because I'm just reading the wrong bit of memory. Um, so that's no

use. Uh, okay. So I get the right symbol. So I'm in the right place. I found the right in it task, but I've got the type wrong. The structure is a different size. There are some extra fields. The fields are in a different order. Um, again yeah gobbledygook. So, I have to get both of those things right. So, just worth saying how symbols and types are handled in Volatility 3. So, it uses something called an intermediate symbol format. It's just a big JSON file. Um there's a utility called dwarf to JSON uh that generates those from dwarf debug data. I'll talk a bit more about that in a sec. Um you need an accurate ISF and volatility 3 matches up

images you've taken with the ISF you need via Linux banner which is a um piece of data compiled into every Linux kernel. It's available via prop version if you're running learner system and it's this big long string of gobbledybak um that if you work in memory analysis you get very familiar with very quickly. We'll talk about that a bit more in a sec. So where can I get symbols for edge devices? So really the best source of data for this is dwarf debug data. So when you compile a kernel by default as long as you've got the right options turned on you'll also generate lots and lots of debug information. So that's really detailed information you can give

to a debugger like GDB or LDB. So it knows what all the types are, where all those structures are, um what all the parameters functions are, loads and loads of stuff. Um for a typical kernel that's hundreds and hundreds of megabytes. So most vendors won't ship that with the actual kernel that you boot. They put it in a separate set of files. And for normal Linux distributions, um you can download that file separately and then you've got all that information. You can use dwarf to JSON. Use volatility 3. Life is good. Um, you very rarely get these for edge devices. Vendors are very very reluctant to give these to you because they perceive it is their proprietary

information. It gives you lots of understanding about how how their device works. Um, yeah, they don't like showing. So, the second source of symbols you can use is a text file called system.map. And if you've ever looked around in a Linux system, you might have seen this hanging around in slash. Um, and it's just a text file of addresses and symbols and it's used for various um, kind of translation things so that when you get an error, it tells you what function failed rather than some address. Um, it's surprising how often vendors ship this. There's no real use in them doing it. It's just that by default, if you use standard Linux utilities, you end up

with this file on the file system. Um, so you often find if you go onto a system, you can pull this off and that gets you a source of symbols. Um and then the final one is most modern Linux kernels actually have a copy of the simple table. So a list of all the functions and uh globals, their name and their address encoded inside the kernel itself. And that's used for lots of different things. Things like runtime instrumentation, things like producing nice error reports. Um there's a uh file in prop that will let you read them all back out again. Um, so in most cases, the symbols you want are usually embedded in the kernel that you're

actually running. Um, and there are open source tools available that will just read those out for you. Um, so in a lot of circumstances, yeah, you're good. Uh, one hiccup which we'll see in one of the examples I've got later, sometimes that can just be functions, i.e. the code and any globals are missing. Um, but there are ways around that. I'll talk to that talk about that later. Okay, so that's symbols. I can find out where things are. Types. How do I find what everything is? Well, if I've got dwarf debug data, I'm good. But we've said we don't often get that. That sucks. Um, if you have the kernels you need to compile kernel modules, then

that can be helpful. It contains a lot of the types you want and it was the default for the old Volatility 2 framework. Um, but quite a lot's missing and it's quite tricky to do. Um they're also not often available for um edge devices although you can actually sometime you can get them from the source code which I'll talk about a bit more later. Um I don't really like working with these. Um it's my kind of method of last resort. Uh and then another cool uh kind of technology which is a whole talk in its own right is uh uh extended virtue packet filter. So this is a really cool runtime um instrumentation framework, but for what

we're talking about today, it means that all of the kernel structures are embedded in a really compressed format in your bootable kernel image. So if this is turned on, you've got all the information you need in the kernel you're actually running. Uh sadly, either vendors are running kernels that are too old and don't have this feature or they deliberately turn it off because they um don't need it. So I've never seen an edge device that has useful BCF in manual on it. Okay. So but this is all open source. Can't we just build a new kernel with the debugger data in it? Right. GPL should have source code scope should be good. Um and modern compilers are

supposed to be deterministic and reproducible. So if I provide exactly the same input to a modern compiler, it should produce exactly the same output bite to bite. Should be good. But same input means precisely the same input. So I need exactly the same kernel config. I need exactly the same source. I need exactly the same compiler. And in a lot of circumstances, I need exactly the same system state. So the file system layout can make enough difference in a kernel compilation that a kernel that you compiled and the kernel I compiled won't match. Things will be at different places. So this is often feasible for types. The structures will be the same. I have very

rarely managed to do this for symbols because one bite off you cascade out and all of a sudden your init start of where all your processes come from is somewhere completely different. Uh so this sounds great but it's largely useless. Uh so this is a technique that I use instead. Uh dome kernels. Uh so most modern Linux kernels have enough symbol data to do useful analysis. As I mentioned earlier, um a kernel built from similar source and config should give you the same structures. Join the things two things together and you can produce an ISF is actually useful. Gives you usable analysis. So what do we need for input? We need a config. Um a lot of kernels embed that

in the actual kernel itself. Um you can often find them on the file system. I'm not sure why vendors do that, but they often do. Um, if you ask the vendor for their GPL source, they'll often include the config. Um, you can reverse the engineer the config out. I'll talk about more about that at the end if we've got time. You need the same source. So, if you're lucky, they're using some public source. So, something that the kernel developers have published on kernel.org or some other Linux distro. Um, it ought to be available via GP GPL compliance. I'll talk more about that in a sec. Um and you can sometimes work out what changes the vendors made via reverse

engineering. Uh the same tool chain usually made same m same GCC version is fine and likewise the build system if all you want is some matching types doesn't really matter. Yeah. So GPL compliance so Linux kernel and GPLV2 you must make sure that they too receive or can get the source code. Great problem solved. Um there are three typical means, typical ways that vendors comply with this. One is they publish it for everybody to see online. That's great. That's easy. Download it. Great. Um the other is they publish the source for customers to access. And that might well mean you're paying several thousand pounds for a support license for that particular um product before you can get

to that. That also gets tricky in situations uh like I encounter where I don't have a license to the product, my customer has a license to the product and that communication can get quite tricky. Um and the most annoying one that is entirely legal, they provide an email address or a postal address. Please send a check for $15 to this US address and at some point we will we will post you back a CD containing the source code you ask for. Um request of source can take a long time to process. I think my record is well over a year. Um, if you're working on an incident response where you're talking hours and days, that is essentially

useless. Um, the publicly available source doesn't always actually match what is what's running on the device. Companies aren't always brilliant about keeping that up to date. Um, and it might not contain everything you need. So, they don't necessarily include the kernel config. They don't necessarily check that it actually works and you can actually compile it. Um, yeah, it's annoying. What can you do? Um, if it's your device, you can ask the GPL source code well before you need it. Um, if you know what you're running, you can go and do that. Kick that process off before you're in the middle of an incident. Um, I tend to hoard source code. Uh, even if it's not exactly the right code, often

it comes in handy. Um, but yeah, if you're if you're if you know own your devices, uh, getting hold of source code is a really good idea. Um, you can often use the nearest available public source. Um a lot of vendors don't actually change their kernel. They don't actually care. Um it is going to fall over if the if the vendor has made some uh kind of proprietary change and you as I said you can reverse engineer it. It is time consuming. Okay. So that's kind of the theory of it. Let's put this into practice. Um I'm not using real examples here. A couple of reasons. One is client confidentiality. We work on breaches that often end up in the press, but we

don't necessarily talk about uh the role and that takes quite some time. Um so it's easy not to do that. And I wanted to focus on the techniques, not on can you do this on vendor X or vendor Y because that changes over time. It depends exactly which product you're talking about. Um and the final thing is although the kernel is GPL, usually all of the stuff that's running on top of it is proprietary and you click on Ulers when you download it to say yes, I'm not going to share this. Um this is Bides. Nothing interesting about learning. So everything here you can recreate using publicly available openly licensed uh content. Um but they're all based on

techniques that I've used for real. So hi Rich working on an urgent instant response come across a bid OS1 device that we think might be involved in initial compromise. Been able to collect memory. We need an ICF. Can you help? Looks like this is the banner. Let's break that down. So this is an x8664 kernel and it's running kernel version 6120. It's got a big line of gobbledygook in the middle which tells me that it's some Linux distribution and they're supporting this for a long period of time. So they've released a whole series of patches. Um one of the things that the kernel includes is what was the user and what was the host that it was built on. And

very often that's helpful. Rocky Linux are particularly helpful in the act they've put their actual domain of their actual build hosts in there. Uh so this is bogy Linux. That's cool. Uh and it was built using GCC 14.3 um with a set of patches that Red Hat maintain. If you've not come across Bobby Linux, it's a rebuild of Red Hat's operating system. So that kind of makes sense. Um and it has a time stamp in it. So it was likely built on Tuesday, November the 11th. Um that is almost always the case. Uh okay. So you figured that out. um apply years of experience or Google search engine of your choice. Um you can usually track

down what is this where does it come from. So in this case this is a public kernel and Rocky Linux make all of their debug symbols available publicly if you download and that's great. So I can match up um you'll see here that times don't quite match. So that says November 13th that says November 11th. That's fine. Uh what's happened is they've compiled the kernel, they've done some testing, and at some point in a couple of days later, they've put the package together and put it up on their website. Um yeah. Okay. So download the package, extract it, run the water JSON, analyze it, send it off. Thanks. We can't see any of this activity on that system. We'll keep

looking. And this is one of the cool things about memory analysis. If you've got all that really in-depth understanding of what's been running, what have we seen on it, you can very quickly go, nah, wrong place, let's move on. Um, that's really valuable. But things are never that easy and usually on a Friday night because a lot of people I work with in the US. We just got another bid capture from the customer. Looks like this is version two. Can't find a matching kernel for this one. Could you take a look? Here's the banner and the kernel. Okay. And I've highlighted a couple bits. Okay. So, this is 6.12.81-BOS. 81-B size OS. I'm guessing I'm not going to find this

on Google anywhere. Um, so, so what are we going to do about that? So, the version number says 61281 um with a custom version string. It was compiled by bides build at bides build.in internal. Um, so probably something to do with the conference. Uh, still compile rocky Linux. Um, and if we go and run, there's a script in the Linux kernel source called extract ik config. Uh, that gives me the kernel config. That tells me how they configured it. So, that was nice of somebody. So, one thing that kernel developers do is take a particular kernel release and support it for a couple of years. They're called long-term releases. And a lot of vendors will use these long-term

releases because it means that they don't have to keep changing things. The Linux kernel loves reorganizing things. vendors don't generally love uh having to keep up with that. So they'll typically pick a long-term support version and then use it for very long periods of time. Um so yeah uh hopefully that is somewhere close. So how do we build a donor kernel? Go grab a source, grab the kernel config, build the donor, and then we've got what we need for an ISF. Uh so we grab the symbols that we need. Um so I mentioned earlier VM Linux to ELF that comes with a tool called KIMS finder that will pull out all of those um symbol addresses. Uh and then I

usually just use a little Python script to turn that into the system. Map format that we saw earlier. Um I do actually have a patch for the tool to do it. I should probably upc um build the ISF and this is where we Frankenstein the two things together. So we take the types from our donor that we've just built and we use the symbols from the system map that we've just read out of the kernel. We update the banner to match. So I said volatility 3 uses the banner to match up the ISF and the actual kernel you're analyzing. So to make sure it's got the right one, you have to do a bit of JSON

munging using whatever tool you prefer for JSON munging um to make that match. Analyze. Cool. Okay. So I should probably show you some analysis. Uh so yeah, apparently they're very busy. So let's do some analysis. Uh so lots of people use uh volatility in lots of different ways. I like starting with PS tree. So this gives you a process tree. So starting with in it the sort of first parent process, what spawned what what's the parent child relationship all the way down. Uh so this is SHD calls a couple of SSHD sessions calls bash calls couple of pseudo calls bash again. That looks kind of interesting. Uh this is actually me logging into memory capture. I need root access to be

able to access uh prop kore and read memory. So what you're seeing here is me logging into the box via SSH um and uh yeah capturing the memory that lets me do this analysis. Okay, so that's not so interesting. So let's look further down the PS3. Okay, so why is there a Python 3 process and a bash process owned by init? So this is called demonization. So it means these um have been unlin from whatever they were started with and connected to in it. So they're never going to die. You wouldn't normally expect to see a scripting um program connected to in it. And you certainly wouldn't normally expect to see a bash process. Normally

it would be the parent of be the child of SSH as we saw or like something normal. Um, okay. So, let's switch to a different plugin. Let's call sock stat. So, that this lets me see active network connections. So, why is that bash process connecting it stat standard in and out to a network socket? That's a bit weird. Okay, so let's use the bash plugin. So, this lets me pull out the bash history out of a bash process even if it hasn't been written to disk anywhere. Um, so that's cool. Uh, okay. So at that time stamp there they ran ID and at that time stamp right there they ran echo. Hi there besides exit.

Okay. So kind of show what's behind the curtain. So my demo system has a web server that has a by design command injection. I send a um some data to a particular URL. It'll run whatever command I ask it to. Um not that different from quite a lot of edge device vulnerabilities sadly. Um, I used that to uh start a reverse shell that connected back to my system and then I typed a couple of commands ID and echo to check that was all working. Um, so that just lets you see it's a toy example but how you can very quickly go from the memory dump to what was going on in this system and kind of pull on

the threads to find out what was going. Uh, okay. Thanks. We managed to pivot from those IPs, but we can't figure this one out. and you have a quick look and as I mentioned earlier there's no data symbols. It's quite common for edge device symbols for edge device kernels to be compiled with no data symbols. This config ks all is not set. Um vendors think you don't need them. There's no particular use for them. They drop them. Um that's terrible for us in memory analysis because pretty much everything we want to know is a global in kernel somewhere. So init task where the process is that's a global it's a data symbol if it's missing we're in trouble.

Um so I promised in my abstract that we do some reverse engineering. So let's do some reverse engineering. Uh so who here has played the gra? Some of you. Cool. Uh open source reverse engineering tool developed by NSA um on GitHub. Lots of powerful stuff. We're just going to talk about version tracking. This is a feature that is intended to let you port analysis from one version to the next. So if I've done lots of analysis on version one, the company comes out with version two, um I can I don't lose all of that. I can copy it. We're going to use it in a slightly different way. There's lots of um information on the

training that's built into every DJ release if you're interested. Uh so version tracking and um has this feature called implied matches. It's very simple. If function one and function two both reference a global in the first version and function one and function two reference the same global in the second version. Well, obviously that's the case. In reality, it's more complicated because there's lots and lots of globals and lots and lots of functions and they can conflict and all these things. But at a toy level, that's what we need to do. So if I look at some uh decompilation. So on the left hand side here I've got my donor which has got the um symbols

that I want. On the right hand side is the one that I'm analyzing and it hasn't. It's got uh they're missing. So I can start up a version tracking session. I can match up the function names because I know those. Um and then down at the bottom you'll see there's this implied matches. And so you can see it's found uh those two data symbols we want and the source is filled in and the right hand side isn't filled in. If I press a couple of buttons, I can fill it in on the right hand side. And I do that about 60 times depending on how many um functions you want, how many plugins you want to run and you're there. So here's

the workflow. So we build donor as before. We use version tracking to recover the data symbols. Um that is automatable wire scripting. You can click through the guey if you want if you're learning but yeah you can script it all. Uh you generate system map from the recovered addresses. I just use a gedra script again to write out that really simple system map format. Um and then exactly as we've done before build a donor ISF analyze and we're good. Uh yeah, done our analysis, help save the world, go home. Uh yeah, I got a couple minutes. Um so there are a few more advanced problems. If you don't have a config, it is possible to invert it via reverse

engineering. So Linux kernel flags will turn particular functions off so they don't get included. So you can tell whether a particular flag is on or off by whether those functions exist. You can also start digging down into functions and go is it accessing this data structure or not? Okay, that must mean that that conflict flag is there or not. Um it takes a long time. Um vendor specific code modifications I me mentioned before. Uh they're quite common particularly for network appliance vendors. Um they're very time consuming. Um but it is possible to reverse engineer them. Uh and if you don't have any function symbols at all, I've never actually seen this in practice. It is theoretically possible.

I think you could use version tracking. I wouldn't want to do it and I certainly wouldn't want to do it under time pressure. Uh so there we go. Memory analysis are powerful requires quite a bit of additional effort compared to doing it on standard systems. Um it's often but not always possible to collection analyze. It's hugely useful if you can and it's much easier to prepare in advance. So find out what your vendor offers, get the kernel source, understand the collection options, and actually go through a test collection before you've got a live incident. Uh yeah, and then if you take nothing else away from this presentation, um reduce the attack surface of your edge

device. If it's management web interface is on the internet, take it off the internet. Um it's much better to avoid an incident than to do lots of funky memory analysis to work out what happened after the event. Uh yeah, and then my contact details are there. We're um lunch is next I think but happy to take questions. >> Do you have a vendor you like to work with and have you worked with any consumer devices? >> Uh so consumer devices, no because we tend to work with enterprise clients and they tend to be running larger devices. Um, I won't comment on the vendor stuff with the cameras rolling if that's all right. >> Okay. >> A little bit earlier you talked about

like an ideal system where you would like have access to all these things. I'm wondering would this ideal system be easier to exploit because attackers would have all these as well. >> So it is a balance. So like giving the access to allow you to take memory captures. Yeah. does require additional access to the system. If you're doing this really well, you design it so that that is a very limited set of access. Um so like if I were designing it, you'd make sure that only specific processes could access um the mechanisms you need. So there are tools built in I guess to let you do that. So you can keep it locked down while also giving the

visibility. But it is something that you think about. >> Um business question really. Do you provide consulting on the preparation side so that a company can be confident they can get memory >> all their devices possible >> as part of other services? Yes. probably worth logging out afterwards. But yeah, >> from the memory acquisition side of things, is is there like a public list available that tells you what kind of uh vendors support the acquisition of memory so that it also puts pressure on the vendors or to at least increase the level of transparency. So companies or enterprises that are thinking of purchasing these product know the limitations. Is there something I think I already owed it for documentation.

>> No, no, not really. Um, I didn't put it in here partly because it's a moving target. So, you know, people watching back on YouTube, it wouldn't be the same in three months, six months time. There's a lot of pressure from organizations like NCSC for vendors to do more in this space. Um, but no, there isn't at the moment a really good way of telling. Um yeah, >> it's a hot topic. I mean I think yesterday there was even uh something there to China Nexus on edge devices. So um I'm still very surprised that we are then from a defender side of point trying to catch up and we still even have the capabilities to catch up

because vendors are preventing us. I think it's the my experience is it's hard enough to get companies to put a security check into the purchasing process, let alone uh we want to be able to do some fancy debugging after we bought it. >> Yeah, sure. >> Yeah. Yeah. It's it's only one element of lots of choices you've got to make and you don't always get to pick. >> Uh yeah. So in in in the event that you have one of these devices where you can't like get room to like take a memory capture um I to the best of my knowledge most of these devices are built not dissimilarly to like a normal computer. There are like a lot of for like mostly

for shooting in video games like hardware memory type devices. Yeah. >> Could you potentially install one of those in your edge computer in order to like allow the you to take a memory capture even though you can't get in like on the operating system. >> Yeah. So in theory, yeah. So I didn't include it in this because very few companies are going to be comfortable with you installing a piece of hardware and certainly invalidates your warranty. uh but yes in principle like if you can use PCI luch or one of those similar technologies to read out of memory and you're prepared to take then then that is a that is a technique that that would

work. Um yeah it has obvious obvious challenges if you want to run supportable IT. I'm afraid we have run out of time, but please give it up for the speaker.