Hack the system: exploring libpam with uprobes and eBPF

all right thanks so uh I made a last minute decision to turn this from a 30 something slide talk and de about five slides and a live demo so there's every possibility this could go pretty bad um so please bear with me um but thanks for the intro uh yes I'm Peter uh I work for a company called up uh upwind security we are primarily a runtime security platform um that is also more recently started looking into shift left as well um I have worked in tech for 20 years my primary interest is uh is Linux I am an engineer and manager but don't hold that against me I very much still consider myself a software

engineer uh the goal of this talk is really simple um I think uh we're all nerds right I love computers you guys love computers we can be friends this is uh this is one of my favorite toys to play with ebpf is an extremely powerful piece of technology that can allow you to build really cool Monitor and security performance applications um with Linux so I'm really happy to hopefully share with you an example of how you can do that today and what I'm going to do is start off with a like a premise um and show you how to take what is quite a simple idea uh the hallway through into execution so we need a Target like a

story to chase if we're going to take an idea and go and build something we need uh some guard reels to to keep us focused um so I thought everybody likes to feel a little bit naughty right even the whitest of white hats amongst us um can get our kicks by being feeling a little like a black hat in AR game so I thought what is a really well-known piece of software that a lot of us use dayto day um that we can try and exploit extract some information from and open ass came to mind it's wonderful piece of technology what I'm showing you today is an a vulnerability um but hopefully the fact that it's

familiar will help resonate the part of ebpf uh I should also say that in my company we use ebpf a lot um we we use it so that we can help customers understand exactly what is running in very large clusters uh at run time we aggregate information by process Network information in a way that does not negatively impede uh performance um so it is close to my heart personally and professionally um so before we get started a quick gloss for your terms um so we're going to talk about probes today um probes allow for the insertion of code in applications user space or kernel space um and if that doesn't make too much sense right now I hopefully by the end

of this talk that'll be super clear we're also going to use ebpf um which really just allows us to attach programs uh to run a processes um I'd run time with I needing to restart them or modify kernel code code a quick side note just as the previous talk Shield rust a little I'm a a big Shield forward Docker um it is a fantastic debug environment um this setup and as part of this talk um we also have a an open source repository on my GitHub uh which I can show you here so if you do find that during this talk you're you maybe miss a step or two things are you can't really connect two

different uh points you kind just go and Gra this whole repository which includes the docker image um and hopefully start playing a bite with it locally PR is very welcome uh so uh on Docker really all we're doing is grabbing the ubben 2224 image uh we're installing open as server from the default op Source um all we're really doing configuration wise is making sure that the rout user has a password so that we can log into um the S server uh and we're also permitting root login you do not need root login to do what we're about do today it just save me about three lines on a doer file now we're going to go into the

exploration so we have a general gold in mind um which is we want to take open ass and try and pull some information out from it um specifically we're going to try and grab the username and the password as logins are performed to open s this is where I'm going to deviate on purpose from my slides I think this is much more cool if I just show you how to do all this stuff live so as previously mentioned we have a Docker file this is already running on my server you can see that uh we're running the sshd command and we exp expose 4 2222 and we forward into 22 this is just so that we can perform um

anass login like

so1 there we go so we we have a little environment setup that we can kind of start our journey and figure out well how do you pull this information out so right now we don't have a lot of information apart from uh s is running so what I'm going to do is start by for those unfamiliar with doer default uses overlay FS uh overl FS uh has a concept called merge directories this is really the the rot directory of your running container um so I can do a quick inspect and grab what is really the root of that container if I go back to Docker PS we can see that we're running the user Aspen sshd

command uh so I can check that out user hasin CHD and we can find the sshd binary that my Docker container is using so we know what file to kind of go and Chas after but we need more information about this file like what where are the possible places that we can attach ourselves to there's a common utility in Linux called ldd that helps you display the dynamically linked uh libraries of a given binary uh so we can run ldd against the SD binary and get a feel for some shared libraries that's it's using there's a lot of really interesting stuff here for the purposes of this talk this is the library that we're going to focus on lip Pam lip Pam in for

Linux is plug pluggable authentication modules given our goal that we want to get username and password out of an S session seems reasonable that uh an authentication module is probably a place that we should pay some interest to uh so in ldd output we also get the absolute path to this binary and instead of doing ldd this time uh what I want to do is say well what symbols are available in this binary the goal in my head is I want to be able to write a bit of code and basically attach it to a certain point in that program that hopefully handles usern names and passwords so instead of ldd we'll just do relf

ask get to dump the symbols so we got a whole bunch of stuff here um I'm a little too lazy to read so crisby GP and we can really quickly get a a fairly small subset of symbols that have the word offet or you can try password a lot of this as you're going through it for the first time is trial and error and Discovery and just trying to think of different ways that programmers might liable functions but uh spoiler alert I've done this before so it's definitely one of these uh so we can see if as one of the symbols here is called Pam get offto and for the purpose of this talk uh we're

going to focus on this symbol the get here we've made a whole bunch of assumptions right we have the high level goal and we're trying to Chas on a narrow scope of what we're looking at but we need some way to validate ourselves are we actually on the right path right before I go in reverse engineer or code Bas I want to make sure that my time's being invested in the right area so how can we do that well there's a tool called BPF trace and BPF traces allows you to uh attach really easily um ues uh to a given binary at a symbol and perform a simple tracing function um so if I take the previous command and

instead of readelf and I say that we want to run uh BPF we want to create a uo on this binary at this symbol we're going to execute a really simple I don't know a sites uh trus message and assuming I haven't done a typle thank God uh it will give us some feedback that it's attaching one probe um so this is still pretty useless right that we're attaching to a point in the program um but nothing's really happening with SSH session at the minute so what we want to do is perform the action that we're trying to trace and see if that Trace fires right that should be a reasonable signal that we're on the right path if it doesn't if we

perform a login and that trus doesn't fire we can exclude it from our search parameters and go look for a different symbol so what I'm going to do now is just log in again to uh my ass server and what I'm looking for whenever I log in is that little uh trius message should be printed to the iput which it isn't

classic okay so I'm guessing I've attached to the wrong place BPF

oh there we go okay so it did print okay I'm not entirely sure why that happened let me just try it

again cool okay so it is working um so this is thankfully enough validation that hey that that symbol is probably going to contain the information that we want or at least it's in the right flight path um so at this point we can we can decide to invest a little more time in understanding what Pam off talk is understand the structure uh of that function the parameters that it handles and as part of that try and identify where it could be handling the username and the password so prisbe open source uh you can go and clone uh lip uh I can print the repository here uh inside the LI pal repository we'll just do simple search for pal gut off

talk we got a small set of uh results to look through and it doesn't take too much effort to find it so we can see the pal get off Tock returns an integer and the first parameter it takes as a pointer to a pal handle T at this point I have no idea what a p handle T is right but we can go and try and find that in the same way so looking for a pom handle T I can see from the first result it's an Elias for a palom hle and thankfully in the same results I get the struct for palom hondle so here this is the structure that we're getting in the first

parameter of the symbol that the function call that we applied the TR on to from the results I can see that the first uh attribute is off talk seems like a reasonable um parameter that may map to a password we can also see user that one feels like a even safer bet that it's probably the username so at this point I feel like we've done enough investigation that we can go ahead and start investing some time in writing a little program uh that we will load onto this function call to extract uh the first parameter that is passed into uh that function symbol so you can go into do a quick sit rep my just to make sure we've got all our

ducks in order before we go invest some time and writing code so we started with a goal of let's go hack open we quickly find that Pam is a more interesting Target our goal is to pill out the username and password as people perform actions like passwd or or they ASN we know the symbol and we also know that in that symbol that function it is the first parameter uh that it handles which possibly contains the username and password again spoiler it is uh now the approach differ you can you can Implement uh a BPF program in a bunch of different ways for this demo we're going to focus on restricted C on goine and

we'll use BPF to go which is a little help utility from pelium um to compile our BPF B code but implementation details I will show you guys some code here honestly the code is probably the least important part of the talk and you have the GitHub repository Peter mcconnell/ Whispers if you want to just pull it and read through it yourself all right implementation time we have a game plan what are we going to do um so the first task is to build our BPF program and uh it is surprisingly simple so this uh this struct definition you eagle-eyed folks out there may notice is a literal copy and paste of the last piece of code that

we on the reason we need this is that we need whenever we load our program in we have to understand the structure of uh that parameter so that we can read properties um out from it uh the struct above is simply the event that we're going to store these username and passwords into uh for fun sake we'll also store the process ID and the command name just so that we can see if the username a password was linked to SD or pass WD or FTP or whatever it was uh the actual C code is pretty light right we're down to 54 lines here and this is all the C that's in the project um we have a what's known as a BPF map

which is data structure for BPF that allows you to pass data easily between uh system space and user space and we'll use this BPF map store the events into that have the the PID the com uh the username and the password we then have a BP BPF uh section defined which is our Trin function um I'm not going to go through this line by line um I'll just call it the highlights even though there's not too much um but really all we're doing here is saying get me the first parameter uh of the function call uh we get the username and password address those uh properties of that first parameter structure uh we then read the usum on password uh from

that address uh into a structure that we can then put into the vpf uh data structure which in Rus is a ring buffer a very uh popular data structure um for vpf programs um and that's basically it for the C uh all we're really wanting to do here is say Hey whenever I'm called whatever I'm attach to get me the first parameter try and read the uh username and password and store it into our custom event object and put it into the vpf program so we have a vpf program but what do we do with it how does that get attached to the running process uh that we want to Target um for the demo S I chose go you

can choose C C++ rust whatever you like um it doesn't matter terribly I'm not going to get into too many Holy Wars about that one um so I WR a really simple CLI the idea is I want to build a tool where anyone can just run it and pass in a custom symbol name or custom path to the binary um so this code is not terribly interesting I'm going to jump over to uh the listen

method which where most of the interesting stuff happens so this code is little longer we're at what 90 lines still not terribly much the reason why this doesn't take a lot of code is in this instance I'm using BPF to go um which handles compiling the C and the BPF by code but it also generates a bunch of helper functions in go that we take use of here so likes of load BPF objects this function here is not something that I Define myself we just got it for free as a result of BPF to go I don't want to talk too much about BPF to go but for the curious to see where it's being called

from uh this is the build file that we're using for the project and you can see that we R BPF to go for program we pass in uh the path to uh our BPF code and it generates the go uh so what we are doing here is saying open the executable the idea being that we have a path. li pam. .0 file uh so we want to pass that in as a parameter to our function and here we're just saying uh to open that executable so that we can start attaching our program to it once op we add Ur probe so Ur probe uh hooks into the response of user space uh function there's also U

probe which is the entry point and you also have K probes and K Pro you can see that the UR Pro is taken in symbol this is from a c org in our case this is Pam get off tuck um it also passes in the program so you'll note that at no point in the code above this did we generate this object trus Palm get off tuck um just to uh help map these things together this is also generated by uh BPF too and you'll see an annotation here that maps to trace Palam get offto and for the eagle ey out there there you may remember that this is the neem of our C function trius pal get off

talk so hopefully that dispels a little bit of the magic so we've uh we've opened the executable we've added the UR uh Ur group and now we just want to read from the ring buffer that we created NRC program um so a little further down here you'll see there's a go routine with the loop that's just constantly trying to grab as much data from uh the ring buffer is possible uh for every event that we read off that ring buffer we need to be able to parse it right we don't we don't yet understand the structure um so the way that we do that didn't like that c let try that again uh the way that we

do that is in through BPF uh to go um so we have let we open up believe it's in utils the simple parse event data that takes a slice of bytes um and all we're doing here is reading the bites uh bytes into a data structure uh that we can understand and go called uh event T if I open up our types you can see that event T is a very simple struct but the important thing to note here is the data structure of this struct is identical to the struct defined in uh ourc program because we need to be able to uh know the size of each uh attribute of that structure in order to read it so if you

get really weird jargon back in your BPF program it's likely that there's a mismatch between your go line or your users Bas structure and your uh cbas

structure so once we get the event back in a format that we can read we just print it uh to screen so to recap there I appreciate there was a lot of steps but we created a user Space Program um we use BPF to go to act as our BPF compiler it took in that urat probe C that we started with generated some BPF B code and it also generated a bunch of go helper functions that allows us to really quickly build these little applications in about 100 lines or so and we print the result to the screen now for the fun part Sienna and action so we can go back to the previous command uh which already has the path to

our uh lib pound that's running inside our container take this and let's validate that it is still running everything looks good and we can still log into it 1 through 7 01 if we specify the right Port cool so we have a functioning s server uh ready to go what we're going to do now is run our user space program and point it at the lib Palm that's currently being used by that SSD uh binary excuse me in uh in the container so we compile our program which is called Whispers just I'm a grief ball and figured that was a colum for it um and we will specify a binary path uh the we can also optionally specify a

symbol symbol has a default value of Pam get off tuck but we can see our user space program is not uh blocked it's waiting for information from that ring buffer if you remember back to the C uh we only publish uh events into that ring buffer whenever we receive them as you would expect so the next part of our test is to try and validate that uh we actually get the the data that we want so to do that we're going to just do another login and if everything's done right up to this point and I've done a login uh I should be able to see the username and the password the demog goods have been

kind so because we have uh written a program the targeted Le ponut uh open S H and we also get some extra things for free right this is a shared library that is used by many different processes um so even though we didn't go after password let me change this to bides the exact team users based program also captures username and password for other commands on the host because we're actually injected into uh the lip Pam process so that at a high level is essentially it um thankfully it went relatively smoothly um as mentioned before you can get all the source code for this project um it's my GitHub um I also have a website called Peter McConnell tocom the

most uh recent blog post uh covers a lot of the same material if there are any gaps that is it any questions

uh is it possible to extract MFA token too if MFA is set up on necessary I I'll give a really generic answer to this so the way that the probes work is uh you need to have privilege permission on the C system that the run and process is on so if you have any function in either user space or inernal space that handles the data that you want to extract then yes you can pull it out but so one of the questions that I predicted uh I would get here is like would this apply to SSH Keys which is roughly in the same kind of ballpark as what you're asking um so in that model we've hacked

the server here right that's the the target where lip Pam is running for private keys for example the private key never leaves the client you're not sending it up to the server um so as long as you're running on this privileged on the CM machine where the sensitive data is being read um you can see it if you're running on a system that never gets that file uh you're you're pretty safe worri

yes uh exactly so the the question if I heard it right was given you already given the privileged permissions that you need to load this program because I kind of invalidate the need so I need to be a little careful here what I didn't show you today is you can extend this idea with uh kres so kres is a very similar idea but you're hooking into kernel level functions so you can hook into the lik of exact V or fork and you can uh filter based on the command name for example so we could use kres for this exact same demo that where I passed in bin path and I invoked the uh The Whispers binary you

can replace the need to manually do that entirely by having one program that listens for kprs so sorry just your but I guess Oh you mean like if it's a different version of lip pump yes sorry gotcha yeah yeah so the um so you need to have so there's like version compatibility issues here right you if you change the structure of that struct palom handle for example you're no longer going to be able to read the data um so we hardcoded the definition for that struct another approach is that you can try and find the header files on the VM that you're running and use that which gives you better version compatibility um or you can also take

their approach it not using dynamically linked libraries stripping your symbols there's a lot of ways uho it uh H in the middle

so if it's running inside the VM yes yes so it should it should just work for free yeah oh to get information about the host no it would need to be running on the actual uh host system and not the

VM uh any other questions I have no idea how but I did it on time thank you very much sh [Music]