ELK: Not P.O.S.

so just to let you all know the most awesome hacker tool isn't strings it's tracer tee i love the fact that you scheduled like a blue team right after a red team because we all know red team is sexy and blue team is not all right so this is what he said the ubiquitous about me seven years of is investigations stint as a consultant i know a thing or two about logs because i worked in instant response for a long time and most of you in here is also token canadians but you know it's all good come on don't go too fast hi here we go how about those ssl bleeds really fine i make money by keeping my employers

happy um and the community because i'm a community development type person yes i work for elasticsearch so elasticsearch sponsored this place and they pay me to gumdu talks i'm not a salesperson so don't expect me to actually try and sell you anything because i don't like sales people the good thing about this is that this isn't a sales talk it's actually all of the stuff i'm going to talk about is free as in beer so you have no excuses not to go back to your organizations and install logging software that actually works of course i have to give you the shameless plug that you know if you buy some sort subscription you know you make

me eat which i like to eat

really logs they're just events events are just documents elasticsearch is stores documents we'll get into that it's all good the elk stack who has heard about the elk stack prior to today from the etsy guys sweet how many people use it fantastic working on it is okay who's who's in production a couple of people nice who's helping me you know who's helping me pay for my dinner yeah that's what i thought so three parts the elastic to the elk sacks log stash last search in cabana log stash is a way to take events because i'm hesitant to say logs because it's not just about logs it's about any kind of event you want do something with it and send it at the

other side elasticsearch is just a storage mechanism it's based on the apache blue scene we have we employ a numerous leucine core developers so we support not only that our stuff but we also support the whole lucian ecosystem kibana it's the front end to visualize stuff that's in elasticsearch not just logs but pretty much anything that's in there we have also of course some tools for success i'll let somebody else talk about that ubiquitous hey look a whole bunch of people use their stuff and logstash three stages input filters output also known as generate the events modify them ship them out what does that mean typically when you're doing the lodge dash you are going to have three parts

to it it can be in one device or you can break this out into multiple devices it's totally dependent on your setup one of the best things that drew me to the elasticsearch and logs elk kind of build when i was in your shoes as being the guy who had to deal with the incidents was that you could pretty much build it any which way you want if you want to have just something that listens to things and then pipes it into some sort of message queue you can do that if you want to run it as a single device because you're a small shop and you only really don't need a message queue you can do that too everything's

very very configurable very very open and of course it's all ruby so you can even extend it if you really want to we talk about inputs some really cool stuff twitter if you're worried about brand type stuff or you want to see if somebody's talking about your business or something like that you can monitor twitter you can monitor irc you can monitor snmp traps those are you know pretty easy something that how many people have uh message or uh packet flows in your network one one person really packet flows are a wealth of information logstash can take packet flows natively so you don't need any extra tools or anything like that you just point the

flows right at logstash and it just ends up in the rest of the system imap is also really cool so if you have uh if you want to if you have a mailing list or something like that you want to ingest all that information you can parse it you can add extra data you can add metadata to it so on and so forth we talk about codecs codex is just a way to manipulate data so for example let's say you have a tcp stream coming in and it's json based instead of just single lines you can take that json and just pipe it through the rest of the infrastructure you don't really need to do anything

special with it it just works like i said netflows message box cow fun you name it i was talking to some other people they're gonna start working on doing uh encryption so that you can encrypt on the far end have a codec for decrypt it and it stays encrypted through the whole process filters is where the power comes from when you have an event of some sort you usually have some sort of data that you want to extend some kind of metadata typical one being ip address what can you get with an ip address well you can do a reverse dns you can do a who is look up you can do guip you can do

what else you can look up reputationals sif data you name it you can add these metadata pieces of information as the event is going through how many people do incident response a couple of people how many people do instant response and realize that something happened three months ago how many people went ahead and tried to figure out where the ip address was resolving from three months ago wouldn't it be really cool to have that as metadata to the event as it goes through the pipeline that way you don't have to go back you can see it right there fantastic grokking we're going to get to anonymize is really cool if you're sharing that data with a third party

if you're going to be doing other kind of things where you're going to be displaying it to certain people that may not need to see certain functions outputs the comma one being elastic search it's not necessarily where everybody wants to send things you could do things like cvs fees you do page who knows what page of duty is one do you use pagerduty fantastic so pagerduty is a way to send alerts to yeah everyone's going home no jet i've been there there's nothing like waking up at three in the morning to back to back pages um and i carried a pager pagerduty lets you just you know get those escalations it's kind of magical it's kind of nice

um also things like hipchat or you know or irc or anything like that so if you have these kind of things and you have like a team irc or a hip chat kind of infrastructure where you want to be able to get your team acknowledging alerts right away just dump it straight into your team irc channel or you're turning the hip chat channel and away you go right of course you can email file it so forth windows is a special beast although this isn't technically part of the elk stack i'm actually really really big fan of the nx log function nxlog is an open source project with commercial ties so on and so forth the what's really nice about it is that

it will run as a very very thin layer on top of your on your windows hosts it uses pub sub to the window event log instead of having to sit here and query the logs every time so instead of you know 5 000 messages go through and it's gone well instead of having that it just sits there as pub sub and stays in memory the other nice thing is is that you can actually keep all the structured data how many people know that the latest microsoft windows versions actually all the event logs are structured data they're all stored in xml so instead of you know sending all this stuff with like into a syslog i use syslog because nobody

actually ever uses syslog rsc but wouldn't it be really cool instead of having this one line trying to guess what it was and just keep all of the metadata log all the things data all the things right grock what's the first one ip address yeah super easy what's the second one nope nope harder than that no it's kind of hard to figure out what it is isn't it closer no not windows it's actually common log it's apache common log that's a real pain to read isn't it could you imagine like changing your common log with one thing and then having to you know tweak that how many people do that with other systems yeah wouldn't it be awesome if you could like

use human readable things grock enter the garage grok is just regex but instead of having regex you're actually tanging it tagging it with human readable formats so ipr host and we're going to name it client ip now if you want to change something in your common log you make that change in your apache hosts make a new pattern and away you go and it's not hard to figure out right heck of a lot easier try doing all those regexes for every single device on your network versus doing brock because if it's an ip or host you don't have to redefine what that is every time define it once reuse it it's kind of beautiful isn't it

checkpoint gabana so i don't really go into elasticsearch because it's just really a storage mechanism and from a logging perspective or something like that it's not a whole huge deal although one thing i will say is that with elasticsearch a lot of databases so and so forth you have to extend it vertically so you have to add more ram you have to add more disk you have to add more cpus the nice thing about elasticsearch is that you don't have to only scale vertically you can scale horizontally previous gig i had from bare metal to joining the elastic search cluster in 15 minutes so you need more space 15 minutes slap a server in your way you go

this is and i'm not going to blame the monitor i'm going to blame all your eyes you all need glasses

this is cabana this is chicago's um chicago crimes for the past 14 years so what we did is we pulled all the chicago data sets and this is what cabana can show you this is cabana 3. so cabana 4 is in beta and i'll get to that in a second this is visualizing all the crimes in chicago for the past 14 years and you can't see anything on that wow and this is kibana 4. it's a lot more sexier i think personally yeah thanks all right it gets sexier i promise that i wanted to show this slide and you can't really see it but this here this is cool i love this this here everybody hates who hates pies shirts

nobody a couple people couple people hate pie sharks yeah i'm not surprised this is more than just a pie chart if you note if you if you look at it later there's two halves here that's male and female and what they did is that throughout the rest of the second ring it takes the next subset of the data set and adds that so you'll see that on both sides it's the same colors and then the third ring out it's now only two colors because that's the next subset so now what you're seeing is that when you get further down you're seeing the subset of a subset of a subset i personally i think that's pretty

meaningful when you start looking at things right

of course you can do the mapping if anybody's seen the old map we'll show you the old map because i've got a demo this is a lot better than the previous demo or the previous mapping but it gets even sexier how about heat maps that's pretty nice so there's still some things to do that i want to personally do and i'm looking forward to seeing it in projects either from the community or from the organization who knows of the sif framework intelligence framework so you get all this reputational data from lots of different places dump it into a database by the way sif version 2 is based off of elasticsearch so again this goes back to adding the

metadata so now you can go ahead and query all this beautiful reputational metadata right from all open source intelligence feeds uh ldap i'm pretty sure most of you probably run into windows world wouldn't it be nice to be able to get the ldap information about a user as the event goes through and reporting an event correlation those are kind of important for us as well let's do a quick demo time there's my mouse all right can we see that not really huh okay so last night before i went to bed wow that is terrible to look at all right so it's okay we'll we'll try and make do so last night i went out to one of my

servers on the internet and pulled the uh the secure log and the message log and wrote a really quick about a couple lines just to kind of give you an idea of how this can work i didn't want to spend a whole lot of time on it because i don't want to you know get too excited and go hey look at this look at this look at this so i really only kind of keyed in on ssh right you have an internet on the server you have a server on the internet you if you work from home you probably don't have a static ip address so you've probably got you know ssh open to the

world right so what i'm trying to do here is visualize over the last whatever the log was in this particular case close to about seven days worth of logs and then we're going to dive in to see what happened with the failed logins so this particular place where you can see as you can see as you would expect you can zoom in kind of get the idea of what it is and then down below if you could see it again i blame all your eyes um you've got particular details about the event that happened if we go ahead and kick up over here i've got the next in down there's all the failed logins and you can see how fast it was

that's seven days worth of data on my laptop and a virtual machine so it's designed to be fast so elasticsearch is a search engine if you're doing major search things with thousands and thousands of documents that's what's geared for therefore it's got to be fast so if you take that same extrapolate that into logs it becomes super fast for logs as well if we go in we can dive into this and we can see for example that i've gone ahead and parsed out the failed passwords for x again this is all using grok and that one

and you can kind of get a feel for what it is now so think about adding all the metadata to that particular event so you've got an ip address you're doing sif you're doing who is you're doing all this kind of stuff all in one single view that you can query at any given time then let's take out the ip address let's say we want to see everything for the ip addresses that some of you might have recognized this but there's and you can't well you can kind of see it just as you would expect i can talk louder than that is that better all right sorry so just by clicking on the field names on

the side it pulls out that drop-down of the last x number in this particular case 10 of that particular field and then categories it by the number of events for that particular thing just like another other kind of solutions that do this right and then of course i can go ahead and click on it and really dive into that to see what happened now this particular server it will block you after five attempts failed attempts so i'm ever really going to see a handful of attempts from a particular ip address and they'll have to you know move on we go ahead and of course add the guip functions to it so over here you can't see it but

here is a particular timestamp we have the user that was attempted to be logged in as we have the source ip address and then we can go ahead and add additional information such as the city name and country name and as you would expect there's a lot of abd a cute little map as you would hope to see so you can see what's going on and you can see that there's a couple of things over here in north and central america we've got some things in europe asia and then just a couple in the apac area you can't really see that color represents more we'll just leave it at that that way you can kind of guess

and of course if we go ahead and click in we can zoom into those particular things just as you would expect with a g with a any and i don't know if i have good internet to pull down the tile maps so that pretty much wraps up the demo um i'm going to say that we have a user conference coming up and because everybody else did it i had to add this slide and if you want to find me that's how to find me anything i can answer

and you'll notice i'm on time yes

so the question was do is there a release date for carbon four beta three um you're gonna hate me soon yeah it's i don't have an exact date but i know that they're pal they're putting the polish on it right now any other questions fantastic thanks for coming out