eBPF For Security Monitoring

time will start welcome everybody thank you for coming along to my talk this is e by packet filter EBP F for security monitoring as it could otherwise be called EBP eff all that your wildest dreams will come true terms and conditions apply and which sort of reflects the way that I feel about EBP F at the moment and but we'll get into some of the good stuff that we can deal with it in a second cool so a little bit about me first of all I am John McAfee wearing a week trying to escape on the boat somewhere and somebody had presides last night rather and charitably said since I've cut my hair a bit shorter my name is

James Ogden otherwise sometimes known as Linux Thor you find me here on job and on Twitter as well don't tend to be super super high volume on either platform and occasionally put things on there that I find maybe mildly interesting in my day job I'm a tech lead for security engineering sly betting and gaming so in security engineering we look after all the kind of security filling that protects the platform so things like idf's multi-factor authentication session recording a lot of different security tools and a big part of what we do is to feed data into our sim tool so our security operations guys can monitor that they can have a look at graphs and

alerts and so forth so I'm very interested in ways that we can either get more data and data that might not have been available before or ways that we can get the same data but much more easily and without having to jump through perhaps so many hoops so we'll move on to the the technology that we'll be talking about which is EBP F who is said of EBP F school and who is used he BPF before a couple of people who visit somebody else over there as well and who has heard of it maybe is interested or has never heard of it and this may be interested would that be sort of the rest of you is

there anybody who is not at all from a Linux background okay cool so hopefully it's going to going to address what most people might want to get out of the talk the people that have used it before I'm not sure that's going to be anything super new but I can kind of show you one or two things that I've been playing with with it so in terms of what I can hope for you to get out of the talk we're going to kind of cover what it is who might use it what are the limitations some of the limitations and then we'll have some demos at the end to actually show you it in use this isn't a

running order as such but hopefully by the end you might kind of get a feel for this and then decide if it's something that you want to take away and maybe have a bit more of a play with so I'll show you one or two ways that you can get started with it really quickly so ebps the next thing sorry I've got a bit of a smoker it's a salinity thing it's built into the Linux kernel you do need perhaps a bit of a newer Linux kernel you're probably going to get the best results from having the newest kernel that you can find I think is a fair way to put it so if you have something like

maybe a 4.15 and which I believe comes with with the bun to long zone support that would be great and they have packages in repositories for working with beef ef4 come on so that and the second if you have a slightly older kernel maybe back to maybe four three ashore so you probably more or less okay but you might find one or two things are not there yet so if you want to have a play with this you probably want to have the kind of newest kernel that you can get your hands on so a little bit about what it is it's a tracing framework essentially built into the kernel so you can write programs

which are compiled into a bytecode and then load it into a virtual machine and you can do I think of some interesting things with those so we have a quote here from Ingo Molnar who's one of the Linux kernel developers as you might know so when e BPF kind of first came around or EBP F in conjunction with K probes which we'll come onto as well this was a quote and I think this kind of sums up quite well what it is and what it does also somebody else gets to say this bit which I would be a little bit perhaps hesitant to say so we'll let somebody else say that part and then we

can just put it up there and it's not me that actually has to say can never crash never interfere with the kernel negatively but it's less likely to cause a problem that other things that we might have used in the past or similar jobs such as loading in custom kernel modules that might be hawking things in various ways and so it's much safer and we'll go over some of what I think use cases for EBP F so this is just my own opinion you might have different ideas for what you can do with it but I imagine that it could be useful for blue teamers so this is a term that's become a little bit and a little bit overloaded

perhaps a little bit meaningless with time but I'm talking about kind of the good guys the people are deploying tools in the network and the monitoring for threats maybe it's as part of an exercise of some sort a Red Team Blue team exercise or it's just your kind of general day-to-day they'd say role of blue team of that and then perhaps a kind of a subset of blue teamers or maybe a slightly different discipline our security operations folks as I mentioned before so these are the people that sit in maybe your security operations center sitting in front of all these interesting graphs and alerts and things like that that have been produced for them and then maybe they're

taking some action for you maybe they're performing some kind of blocking but as with many things in security they need data security as as a very wise person once said his name has escaped me now but security is kind of a data-driven problem so the more the more data you have perhaps the better your posture is if it's the right kind of data another use case that we can imagine for EBP eff is testers so anybody here is poking a system that performing some kind of test and they'd like to get a bit of extra information out of that about what's going on so EBP F can be really good in that case as well and slightly related to the

testers we have hackers and the old-school sense and the old hacker versus cracker thing don't worry few too young to know that or it's passed you by it was a holy war that was fought many years ago and wasn't very interesting to anyone okay so a little bit of history of BPF so we're talking about EBP eff and but first of all we have to kind of know what BP F is so BP f is Berkeley packet filter old old Berkeley packet filter from like 1992 or there abouts I won't ask for a show of hands of anybody who wasn't even born and I've got my suspicions about one or two people I can see in the room but it was

originally used for one or two different use cases so this has become known now with EB bf as C BPF or classic BPF in the past was always just Berkeley packet filter so what we have with classic BPF is a very simple virtual machine just has a couple of registers has a little bit of scratch space there and then we can load up some byte code to form different kinds of operations and that piece of documentation there on the right is from one of the Linux kernel documentation files and I put it up now because it's such a short thing but it kind of summarizes almost entirely what the architecture is of the classic BPF

virtual machine it's super simple in evolutionary terms you know what kind of we've got up we're staggering around but we haven't yet worked out loud to pick up a rock and mush days over the head so we we can still do some interesting things here so I'm guessing maybe most of you have used TCP dump in the past sing-sing notes in the room so that uses Burton packet filter BPF so you can pass the minus D flag on your TCP dump for commands and then it will print you out a dump of the BPF code that so fill it there for a couple of different tcpdump expressions so we have port 80 and port 81 so you can see there I think on line

7 and 1750 on one has become 51 on the other so you can can kind of get maybe a small feel for what's going on there so moving on to EBP F what is EBP f eb bf is exciting vp f it's not it's extended vpf and what we have here is another virtual machine but it's a little bit more advanced this time so we have ten registers we have a stack is a small stack 512 bytes and then we have this concept of maps and maps and something that we could load data into we can share that with user space but don't get too excited yet we're only a little bit further along in

the evolution we've picked up a rock we've sharpened it or bashing each other over the head so we're doing interesting things and but it's not yet super super advanced but we can do some very interesting things so EB PF is something that applies in a number of places the ones that I'm going to be talking about Cape ropes what Cape ropes are incapable by the way are older than eb PF you could use k probes in the past but you had to load a module and then you had to inject your k pro that way and what we have is the ability to be called whenever a function in the kernel is called so we attach a

k' probe to a particular kernel function that gets called were called first and then we can extract some information from that if that makes sense to everyone hopefully it does we also have you probes and new probes are like K probes that we can use these in user space so functions within user space applications we can attach our probe to and we can also do this with libraries as well so system-wide we can do something like Lipsy for example we can attach a probe to particular function in there and then whenever that gets cold you probe will will be called as well first and then as well as those two we have K rep Propst K

reprobate cape ropes but they get attached at the end of a function so just at the end will be called and then we'll have the opportunity to have a look and see what the return value is which is very useful and we have the equivalent and user space as well so we have quite a bit of power there in terms of where we want to attach our probe and as it said before this is all be BPF bytecode running in a Sun box or a BM so there's a little danger that we're going to cause too much harm you can crush it but it takes a bit of work that's right very hard so how to use it how to get

started with it you can potentially write your own EBP F by code in some way it'll be much much easier to use BC C which is the BPF compiler collection this makes it very easy to write EBP F code and C in Python I don't mean C or Python it's C and Python together so it's very often Python when a big lump of C in the middle of it yeah it gets kind of interesting so to look at a really simple example there what we've done here so we've got our kind of Python wrapper and then we have a little bit of C code in the middle so we're going to be doing a cape rope we're

going to be attaching it to sis look there so that gets add a function that's called in the kernel for making directories and then we can see C just there we have the PT Rex which is otherwise sometimes known as the context so whenever our probe fires first argument we'll be given is a structure describing what the registers are looking like currently which may be useful some things and then also crazy okay that's better so then we'll also have the kind of signature of the function there as well so mukhda takes path name and a mode argument and then with our BPF trace print k there we'll be able to say what is being told and then we can exit from

our probe we can return 0 there and then you can see underneath the result of running this so when we run this as Ruth you need to be roots will have discipline in order to use EBP F but when we run that whenever we make a directory system-wide our tape rope is called and then we can potentially even extract some information from there so we can take the argument there that is the path name and then we can perhaps print out so the thing that you can see there with the BPF trace print k is a kind of very quick and dirty way to get some output from your EBP F program this isn't really ideal this does allow you

to do some kind of quick prototyping and that kind of thing but whenever you're using BPF trace print k you're using it appears under the cysts under slash this as a file there and then that's shared so every every thing that's using this across the system is all trying to share the same file essentially it is so you might find this that if you have two of them running together and they're both using the BPM trace print K you might not get the output that you expect where you expect it you may also find the if things terminate without it having read everything from there then the next time you fire something up you'll get essentially some output

that's still way too from the last time which isn't great so the way that you're supposed to do it and it does add a little bit of complexity is to use so if you see there we have the structure that we find so what we're going to do this time is create one of these BPF per CPU arrays and then we can copy data into that and then we can share it with user space with the with the maps so has I did a little bit of complexity in the C code I've actually split it out into a separate file I'll I'll come on to the options for what to do that in a second

but I think it is a bit cleaner to do it that way it's just a matter of preference unless you're doing some kind of a demo or teaching exercise it can be useful to have everything all together but I like to separate them out if I can but then also if we look on the Python side there's a little bit of extra complexity there as well as we've had to introduce Python C types which is a little bit clunky to work with in my opinion obviously it's just it just a matter of opinion the thing is there like when we have to cast it's kind of a bit more clunky than we'd hope it to be

but then this allows us to run many of these programs without them ever interfering with each other so that's the reason for for a little bit of extra complexity there so as I mentioned you can load your C code from a separate file you can either do it in the top way there so we are loading that in and then we're getting it as a string we might want to do that because we may wish to rewrite it in some fashion we might want to insert some extra bits and pieces in there or if we're happy that our see code is all that it can be then we can just BPF with the source file there

and load it in it works like magic not quite it doesn't quite work like magic it works by essentially patching the function and then jumping in C okay so there's nothing to magic there that's all that might be a little bit of an illusion to sign of check time of useful abilities that perhaps you could relative to if you're not careful about what you do so the next thing to mention is the EBP eff verify so as i said your code is loaded into at the end in source unbox and then it is subject to inspection by liam verifier and it can feel a little bit frustrating sometimes when you're trying to get something to

work and some of the errors are not always easy to understand at first you can kind of get a feel for them over time there are strange things that can happen that I've seen as well where maybe the order of things can matter even if it looks like it shouldn't sometimes you can find that things aren't quite working the way that they should and by making what appear to be small fairly cosmetic changes often in the C code you can find that it's all of a sudden working which can be can be a little frustrating so if we if we have a look at this here so I've deliberately put some broken EVP eff code in here I

think what I was trying to do was to copy more bytes into a stack-based buffer than was available so I was trying to overflow the buffer and the verifier has said no the error is reasonably okay there it's saying invalid stack type giving us an offset and it's saying the exercise 400 I think that's trying to copy 400 bytes and so you see 4 bytes there so that's that's kind of fairly helpful it does take a little bit of thinking about it at first to work out what it's telling you but then you can get other ones like this the back edge from instruction in 33 to 23 so what I did here was deliberately introduce a loop from which

I wasn't it exiting so that's another thing that it will check for so it does have a jump forward and jump back so you can you know a simple for loop or something is something that you can't have but if you get into a loop that you are getting out of it will say no so the way the verifier works is to kind of do your modeling of the code and work out where where it's going and whether any of the any of the operations will violate violate its own inbuilt set of rules so as other things as well like accessing memory that doesn't belong to you as well as the size of the program

exceeds a certain number of instructions then also it will it will be a problem so back onto this thing of monitoring and monitoring very often is rightly or wrongly unfortunately a necessary evil that is driven from logs the logs are okay you can get some good stuff from logs but sometimes you end up having to read the data and chop it up and that kind of thing and of course logs will only have the data rate that the developer thought was useful and thought somebody might be interested in and another problem with logs is sometimes you have to turn on some sort of debug mode to get exactly the level of logging that you would want

even though maybe you don't want the whole firehose of everything to do with the applications internal state you maybe want something related to a library it's calling or similar information so it might be that the logs just they don't have the information for you and even if they do have the information you do have to do this process of taking some logs from somewhere I've deliberately chosen the patch I think of mine he's doing a talk right now truck 1 glad he likes to say that security isn't always the project zero and I think that's very true for some people is some browser exploitation and everything like that is wonderful but for a lot of people myself

included security is probably just taking some logs or events or data from somewhere and then putting it somewhere else where people can use it so in this case we have Apache which is writing its logs to disk and then we imagine that we we're perhaps tailing them and then we have this thing in the middle which is represented by the meat-grinder and that's all the nasty stuff that you have to do with it that's how the sausage is made when you have regular expressions in there you know you have all kinds of different things to transform munge the data maybe you maybe use Lunken you know you don't quite have deep enough pockets to send all the data

into it that you want so you have to start trying to filter stuff out that doesn't look interesting or put stuff in that that does appear to be interesting and eventually the aim is to get it to your security operations guys perhaps and so you can a security operations center and they may wish to look at graphs and that kind of thing or we may want to produce alerts from there few people and actually open these systems to take some action so wouldn't it be better if we could use the magic fist glove of EBP F to reach in there take what we want and give it directly or more or less directly via a little

detail to the security operations people or however it is that might be interested in this data and then also produce the alerts so we want to extract the good stuff and profit easily without too much fuss without too much messing around with regular expressions and open if they just get the good stuff out and make money or something cool so I'll move on to a few demos so you can see this for yourself and hopefully gonna cover cover a little bit of what my workflow is and in doing this kind of stuff I find what might be interesting oh this is going to be right can I look okay this is gonna be really fun

[Laughter]

all right I'm gonna blow this upside fun I'm not got quite the screen mirroring what I want so I'm gonna have to kind of lean around the corner to see what I'm doing so hopefully that's the worst thing that happens at the demo that we shall see cool so when we want to shape the screenshots before I was talking about using this with the master is it poison at least well there we go I was talking about having a look for McDow calls and so they the easiest way that we can get started with this so i've installed on here bcc and i also have the tools the examples that come with out so I have a very very helpful

tool installed here called trace or because it's called on here trace BP FCC yep so what we want to do first of all is just to a real simple cape rope and we're looking for our sis look that okay so that's running so by

so we can see that that's being called cool sold there again okay that's that's from where the oh and I was practicing but not to worry and you can see that it's been called anyway so we might be interested to have a look and see what the argument is that we are calling it with so we can do this something like it yep so now I'm calling this again but now with the cape rope we're extracting the argument there so we can see what directory we're trying to make which is potentially useful and we can see this system light so any process now which is making a directory we can see that the other thing that we can do as well is to

potentially stuck on and two of these together so we've done trace there so we can also do this that's just about that okay so what we've done there is to attach a kaypro to sis McDow so the thing within the kernel is responsible for them look they're Cisco and then also we've attached one to mcdr in Lipsy so any application dynamically linked which is calling mcdr there we'll pick that up so we you know we don't need to launch the programs in a special way running under soapy trace or something like that we just ask the kernel to tell us whenever anything is doing these operations and then our our code gets called there so

this is kind of the way that I normally start off prototyping this kind of stuff I have an idea that I might be interested to have a look at something so then I'll use trace to kind of attach to a few different points and see is it firing what I expected - am i seeing the right kind of information without kind of going and reading any source code or anything yet that can come a bit later on to work out perhaps exactly what the the functions arguments are that it's expecting but that's just a really simple way to kind of go is this interesting or is this not interesting is this worth pursuing a little bit

further so when I have something that maybe looks interesting like this yep so I can then progress to doing a bit of - no so that's the example that we saw before on the slide so that's a really simple version of it and then I've got also the example where with the with the string the argument from the path name and then if this is all kind of looking good and interesting like something that's worth pursuing a little bit further then I'll maybe woke up to something like the other example there so we've got our so that's our C code as we saw on the slide before just move that down a little but you will see that

and then it has its corresponding bit soft - as well so by now that's maybe looking more of a kind of a useful thing that could be used for found myself about say use cases but then I realized if they do use far too many times they're cool so is that kind of clear for everybody do you understand any questions do shut out if you have any questions as we go along I hope there'll be some time at the end but just give me a wave if you have something and we can come to you right away and deal with it while it's fresh cool so have I still got my thing in focus not necessarily but it makes a

good example just for the kind of the hello world example here and you yeah yeah yeah yeah I mean it's it's good to use something like mcdeere for an example because it won't be being cold all the time so sometimes people might use open or something like that but often a lot of stuff is is calling open so look there is maybe not the best real world type of example of what what a real attacker might do but it's a good maybe place to get started and have a look but the next the next example that we have here so what we're gonna through here just see if I got the right will note than that yep cool so we'll go back

to our example of Apache actually web server so I've put a little bit of digging in the in the pain in the bottom right hand corner there and I found something that I think looks interesting this ap send a response so I am interested in requests to Apache that have ended up in an error so they're they're the ones that haven't completed normally so rather than taking my tail of the logs and you know the whole kitchen sink in there I'm going to take just the ones that are interesting to me the ones that didn't complete successfully so we have a piece and our response there which is what Hutchy uses when it's sending an arrow response as

the name suggests so I'll see how we do at the time not see but cool so I can show you then what we have here with with a basic example it is unfortunately a little a little bigger than will fit on the screen at that size and what we've done here is we have a probe and it's a probe we're going to attach to a piece and error response so it's a you pro user space that's a little bit horrible that I've had to do the the ass-backwards thing is is a genuine member of the Apache HTTP request rec structure I didn't make that up but that is the thing that I found that so what I've done here so the

script requests rec is quite long it's quite a long structure with a number of different things in there and then pointers to other structures and so on and so forth I was just particularly interested in this case and pulling out the request so I just needed the top part there so I've kind of cheated a little bit rather than have to include a lot of extra include files and that kind of thing I've just Anna quick-and-dirty definition of the structure there but just the top part of it if that makes sense some of those pointers there are not boy pointers as such they were pointers to other different Apache structures but we don't worry too much

about them I'm just gonna take the request there so we go a little bit further down well so we are attaching a youth rope to Apache 2 and then it's attached to a piece and arrow response and the function that will be called is that one there I camera but I called it probe ap send a response so that's our bit of C code and then we open the perf buffer and then we are watching for data coming back from our probe so I'll show you what that looks like in a second

cool so he's listening right so I'm gonna run my little bit of EPF code there so that should have attached as you broke to that particular function in Apache so I'm going to do this curl that I know will generate an error and then just extracted from there what the request was at a high level I could potentially pull out more more information there so IP address and that kind of thing but just to keep it simple just both the he requests out that so if we do something else so as a file that we know doesn't exist that's can we do so if we so a request that completes successfully doesn't fire our probe so

we don't have to worry about them any requests which is ending in an error is looking much more interesting so we'll use our probes take a bit more information there I'm not saying that they should replace the locusts and that kind of thing they're useful too but it maybe take some of the demand off to get them there in and can it real time or real issue I'm we can maybe ship them ship them a little slower but then have this kind of information be be extracted quickly and used to drive things like dashboards and alerting in graphs for security operation in spokes and like anything so that so that is our our basic example I'm going to tempt fate

by trying to do something a little bit more which is probably a really horrible idea but we'll see what happens so we've got this here so I've imported a library therefore elastic search no I don't I don't particularly have strong leaning towards where you would want to put the data and there's no particular affinity between EBP F and elasticsearch it just happens to be somewhere at the moment that I like to put data and because they they have something called Cabana that I'll show you in a second that you can get some interesting visualizations out of but it it does go to show it's quite quick so we've got our more or less same program from before but now we are

taking the data I'm rather than just printing it we're putting it into a document and submitting it to elasticsearch and then this allows us to have visualizations dashboards that kind of thing also means that it's searchable and also we can we can do kind of time window based reporting there so we can say these are the top requests in the last 15 minutes and then maybe these these might look interesting so far rate of errors and suddenly shot up and there's a lot of these strange-looking requests then that could be something that we take action on so if you look there there isn't a lot to actually get that talking to a lasting search which

is another reason that I like to use it for this kind of thing it is very very easy just to drop a couple of lines of code in there and then we can get that submitting so let's see if this one works okay looks promising so I'm just going to fire up my web browser overlay look over my mouth coincidence

here we go so this could this could get a little bit interesting in terms of me being able to see this properly but I'm going to try and do my best for you cool so we have I know I'm not supposed to leave this spot just give me a second okay cool so that looks looks kind of about right so if we go back into a terminal so that's running now if I good couple of requests there can we see them

absolutely no idea on almost

okay I was going to show you the day check coming to there and it's a little bit awkward for me to do so so I'm just gonna skip over so hopefully something here I've been working cool so taking the data I'll just um refresh so that would date torence so I'm taking the data so there's a couple of lines of code I've sorry you want to say something yeah sorry I should be zooming in ready let's blow it up nice and big cool better it's better for me too thank you for that cool so what we've done here is to take the data that was being sent to elasticsearch and then we've used this thing called canvas

which is part of Cabana another thing from elastic so the data that we've submitted there we can use it to draw kinda graphs and charts and like anything these are these are very simple to set up I was going to show over to that but as for narrowing isn't working the way that we'd wanted to probably just leave it at that for now but I'm just gonna fire off another request into that and then we can see that that gets picked up down this

okay so if people have gone in there and then we can see that the dashboard is updated with with more requests we can see accounts of how many of them are unique requests that's not not the best formatting now that I've zoomed in I'm not by the way designer by any stretch of the imagination yeah I'm more from the old school where people wrote horrible Perl programs and drew nasty graphs which is what I've typed into here I think in my subconscious but hopefully that shows then that we can we can take this data which is from an e b PF program we can move that quickly back into python submit it off there to elasticsearch or

somewhere else and actually start to do things with it draw graphs that kind of thing the kind of thing that you could put in front of somebody and maybe generate alerts from if you wanted to so one more demo and we can hopefully have some questions something we'd still locate the time so move on to another one cool so so the idea here I don't know if anybody caught Tom nom noms workshop this morning you're in that one cool so he mentioned in there that whenever you load a script on Linux it's actually the kernel that is responsible for for taking a look at that and deciding how to load it so for this Kay probe example

what I've done is - I'll show you in a second is to attach a que probe to load script so that's the bit of magic in the kernel that kind of has a look at something decides whether it is a script or not and then goes ahead and sets up an environment and launches it so if we have a look at this one I didn't do there was a basic example for this but we're kind of going over the same ground again really so I've gone straight to the one that will submit this into elasticsearch I've got another one of those horrible looking dashboards for you in a second but there can be much nicer is the problem with me the problem

isn't the tool I've seen some lovely dashboards and visualizations yes it's just not the way that I think so what we're going to do this time so we have our clay probe on load script so we're going to be passed a struct Linux in one of them things which is a structure described in the program that we're trying to load essentially so we attach that what we need to do we need to do a little bit of checking so one of the interesting things about when the kernel is doing its checks you would imagine that it already knows that this is something sensible and looks a bit like a script for it's looking at it and but

very often you can find yourself past a regular elf file and all sorts so this is kind of the first place where it does the checking so I've married a check that is in that particular function which is just making sure that in fact in the in the proper check that's in the function it checks for for a hash gonna bang a shebang but I'm just checking for the hash because by that point somebody's trying to execute it it starts with a half probably interesting and so we'll take that and then we'll submit that over to elasticsearch so this isn't perhaps an absolutely realistic example but this is the kind of data that you might be interested in

in your environment so are there new scripts that are showing up so we could in one elasticsearch submit data from multiple hosts and then we have searchable list of all the different scripts that were being run throughout the environment we could choose to do some kind of eg application now if we wanted to but what we're taking isn't super heavyweight so we're taking the page the inode number because one thing is when you execute a script you can maybe give a full path or you can just put slash and run it and with an ad within a directory so you will see that when your probe is called but the inode is something that tells us that

this is definitely that copy of test felt SH or whatever it may be in some fashion there may be there may be better ways to do it but it gives us something that ties it back to and I note it isn't unfortunately super easy to go from and I know to what file it points up unless you have some kind of it a cache built up or maybe some sort of debug at first type of stuff but it is something there that we could get quite easily and quickly and then we can send that to send that to elasticsearch so the whole thing there so we're gonna pass in a host name because we're kind of

imagining that we might want to send this data from multiple hosts and then if some gnu/linux malware comes out linux malware I don't if anybody knows but a lot of it isn't super sophisticated kernel module type stuff or anything fancy like that a lot of it's in the real world nasty Perl scripts and things that get dropped and call other Perl scripts and download who knows what so that you can become part of a a botnet of some sort so yeah that's that's kind of what it looks like here again just go ahead on this one if I'm in the right place every script which is being executed every thing that starts with a shebang so you

know when you have like a bash script and it starts with the hash bang so underneath the hood the colonel will load that up so the structure that we get it has the first hundred and twenty eight bytes of the script if you're the few other bits and pieces they have a useful but yeah from that we can see the interpreter so we can see is it a perl script - script bash scripts or something altogether weird and wonderful yep

absolutely yeah yeah you could build up a whitelist of no scripts and then if something new shows up in the environment on any one of your hosts if you're feeding them all into a lipstick which is just what I like to use other people use different things then yeah you have then a search engine in for unknown scripts in your environment which could be an instable thing and we've got it at very little cost in terms of lines of code just feed it some data I think if we go into here there should be a script to run so so the other thing that I did there which you don't have to do is just just the

thing that I did was to base64 encode it and that can be a sort of a decent format for for working with data in so then we we just have a string in hand we can see then differences quite quite readily and so we can see perhaps the parts that are the same on the parts that are different if we're doing some kind of comparison so if we take another copy of that same script and call it first one dot SH and then we're going to run our message just so we have a little bit more more data when we put back into elastic so if we go into a I'm not making this up this is the longest trip to move my

mouse pointer across you see it that there we go there we go also will will fire up another visualization it is another horrible one that looks a bit like Gd graph or something out of the 90s but we can see here so I've got some other data here I've left this running in the background for a while in the past and I can see other scripts which are being invoked and as the gentleman said I could filter those out I could just say that they are known good and then I would have a dashboard or some alerts but only related to unknown scripts if we can so it's having a little bit of difficulty seeing this

before if we can get into here and just choose this networks so we can see the two there but if I if I expand time range let's check rotate the time I'm just gonna have a few minutes for a couple of minutes for questions at the end to say the last seven days so we have more data in there and if for example I was interested in pythons grapes driven your loan spell scripts there we go so there's a Perl script there so potentially we can have a look and see is this no good you know how to really have very categorize it so that brings us to the end of the demo I think

we were kind of getting up towards time we've got maybe three or four minutes for questions if anybody hasn't it

[Music] that's a very good question that isn't something that I've looked in too much the things that I've read suggest that it's extremely lightweight so when you have tape rope originally and I think on in some cases still it can be implemented with a breakpoints in single stepping but now as far as I understand it it uses a jumper kind of a trampoline into your code so it's it's super negligible in that sense whatever you build on top of it might have a cost though when it comes to taking the data out in user space you do need to keep up with it so it's kind of super high frequency you may get to a point where

the thing that's consuming it isn't catching up so you can ask to be have something called there so you can take action by default it will just print out that you've exceeded a rate limit and then discard those I guess maybe if you do something very very high volume then you probably do need to be concerned with performance a lot of the stuff I've been looking at is much lower volume so I'm trying to put the probe in it just the point where it's the data that I want I don't want to take all the data like we did with log lines and then throw some of it away I just want to go straight to the no but really I would

say not so if anybody didn't catch the question and we're talking about whether we need to kind of beef up the size of the instances if we're running in AWS or somewhere like that and we want to use EBP F I don't think so so in the past there was tres and that kind of thing and that would would be quite heavy weights but with this we are touching essentially so we're patching the function so that we get cold so if we put it in the right place so for example there with the with the arrow is the Apache areas I'm just knitting the ones that I'm interested in there with the arrows I don't have to do

anything with every other request and throw it away or anything it's just if you get it right quite surgical you know what I mean this does work with containers there are implications of this so in short this works anywhere that you have cap sysadmin you may have caps to salmon in your container this could maybe allow you to look at things that are outside of their might even allow you to anything really yam and you so a container in my mind rightly or wrongly is just a kind of a glorified Sun box open I'm going to upset anybody by saying your help but there's the shared kernel underneath so if we have caps this admin and we can

attach a que probe then we're looking at the you kernel across all that contains so yeah we can we can maybe see things that we might not want to see it's very powerful way of getting inside to really targeting something like the internal function things that's really fragile any minute legs yes what you know you can just put between releases and I'll change visual all see is it's like we are worried that change so I guess happy everything that a lot and that would be a good idea most of the on

quite possibly so the question if anybody didn't catch it were asking about these kind of api's in the kernel and the fact that things may change between different versions and so functions may take different arguments extra arguments that kind of thing one of the things that I've seen suggested in the past and it was somebody from Netflix he suggested it it wasn't Brendan Gregg it was one of his colleagues if you don't have Brendan Gregg he's got really really good stuff online about eb PF much better than my efforts so if you want to learn a bit more definitely have a look at his stuff but the suggestion was to use things like the allison and spaces so the linux

security module which is maybe a bit more stable you can pick things that maybe aren't going to change by your own kind of finger in the air does it look like something that's been that way for a while and it is potentially tricky you may need to rework things if something changes between versions but hopefully you can reason out somewhere to put the probe that is fairly static fairly unlikely to change yeah those things as well yeah so I would say be cautious that's definitely one of the terms and conditions I'd say it does apply to this you need to make sure they eat the probe in the right place I'd say also maybe don't try and put too much logic in the

probe maybe lift the day throughout you're interested in and work with it elsewhere I'm thinking that we're maybe a couple of minutes over on time and so I'll leave it at that but if people are going on to the after party afterwards so if you want to grab me outside you can have a bit more of a chat about it but thank you very much you