Realtime Cyber Alerting with StreamAlert

hello it works oh yes I'm here real-time cyberlearning this was a title I came up with before I'd really figure the rest out so sorry about that then it was in the schedule already so I didn't really I didn't really want to change a bit cyber living real time it wasn't with the little asterisks can you read that bit yeah maybe more like 15 minutes sometimes Who am I I'm Jeremy I'm a security operations engineer at vind and in the past I've dabbled with back ODEs broken barcodes capacitive touch opening doors from the wrong side or the right side unsuccessful bug bounty hunting recently I managed to make a talk out of it so how do you find

out you've been hacked right this is a blog post on the second generic blog I found quite cool maybe an engine maybe an employee notices something strange you know that's that's that's great they notify you maybe a third-party vendor reports a bug but embarrassing you know to have to answer to them ago oh yes thank you very much maybe a hacker submits a vulnerability and so it's a vulnerability that and then you go back and have a look and oh yes actually we were hacked and or maybe you don't find out and this is like an order of biscuits - worst case maybe I've sort of given rough times if maybe this is like a day you know some production code goes

out an employee notices something strange maybe a week a vendor notices a bug that could be longer - but sort of like maybe a heck of submits that could be like a month you know or like you never find out so what we want to be is gonna be in the top you know we want to be employees noticing something strange really fast so all everywhere I always see you know watch your logs what she logs what she loves watch your logs you know what she logs but the problem is with watching phones it's really boring you know they're not doing anything there's lots of them they're kind of brown looking and things you know so

really that's like no one ends up watching your locks that's quite say it but everybody watches a fire right let's go away yeah Wow just can't stop watching that fire this is the slow mo guys the YouTube channel so I had this brainwave idea what if my new startup log fire still working at Ben this is just on the side right log file IO you sent us your logs we set them on fire so that you will then look at them it uses some incredible technologies you know it's a disruptive advanced machine learning nearly tweak-d blockchain has got the blockchain in there multi-religious doors for logging so this is a method of procrastinating this talk you know and I actually went

ahead and implemented some of this so when you post your log to my API there's only one endpoint it just accepts logs it actually adds it to a blockchain mostly I did this because I found this naive chain implementation it's JavaScript as 200 lines seemed perfect and if you get on that end point it just gives you the whole blockchain so those are the only two operations so I had the blockchain Pat I was on my way to to instant success but I needed to add the fire right so maybe you post your log it's just 200 okay you know maybe you post another one that's 200 okay no everything is fine hang on 14 gone that just deleted the

whole block chain mostly because it's like an in-memory database and then you run out of memory eventually so I just randomly deletes the blockchain and then you just you're okay right hopefully the JavaScript garbage collector picks it up will see sometimes is a 4-5-1 and available for legal reasons and this is basically the gdp are kicking in so anytime there's a UK europe mentioned in your logline it just doesn't accept it as unavailable there's also a 2 or 3 which is my favorite and it also includes the a JSON error message LP LP 0 on fire this is the this is my favorite message to return I think it's 5% of the time you get this one printer on fire

actually is was in the I don't if it still is in the UNIX kernel as a message for around during a printing stall occasionally during normal operation the fusing oven would heat paper to combustion this message does not reliably indicate whether or not the printer is actually on fire so they decided to put this on fire message to motivate any system operator to immediately go check on the printer but it might be perfect so that's a terrible idea right because they're on fire what else could we do well there's actually some real things that other people were written that do alerting so there's the elastic have their product stream alert Argus a lest alert there's a couple

others we looked at we looked at a couple of but they're nice sort of looked more in-depth ad streamlet and the reasons we liked it was it was service scalable a like this so we were already using AWS for a lot of other things so it kind of that kind of made sense rules written in Python and I like Python that's the only other reason any reason really terraform for infrastructure deployment so we're also using terraform separately so that was great sort of alliance there to the input and output sources that they had ready to go they also kind of matched what we were interested in and it was open source so it looked really

cool and roughly this is their their Wow overview slide with icons from other companies you take logs in and they can either come in through Kinesis s3 or any snis topic they get processed through the weird streamlets squiggly and then they go out to you know phantom page of duty slack it's three maybe other lambda functions so you really put a whole lot of flexibility for example you might want to be logging quad trial so cloud travel and AWS logs your api calls things you might want to look at are like a delete VPC event maybe you stop logging the cloud trail maybe someone's using root credentials to sign into their account and maybe it isn't I am

policy change that you're particularly interested in or an insecure bucket goes out to the world so there's a couple ways you can get these to your stream molester either through cloud watch events or through the history bucket so that's the round like 10 minutes version tribe watch events is great like 10 seconds but there's some some problems with regions there so regions are a nightmare so we'll pop crowd crowd try that front we can either go to it Kinesis or s3 and kind of looks like this crowd trail in AWS you'd basically click a few buttons and you enable it or use tear form that does this for you but you do this first try and play around and in the cloud

watch part it cloud watch clicks the logs from cloud trail and then lets you ingest them either with the Kinesis or via SNS topics so they basically come out like this JSON message and there's this JSON messages something's happen today the obvious you might be interested in and you can search with this weird filter which is kind of JSON but not really there's a dollar and a dot I don't know why they did that they should just use JSON but maybe it's faster or something so yeah this is that this is like an example log and it's a root sign in which is bad so someone's use their root credentials so how do we how do we alert on this well happily the

Adel you serious foundation is benchmark this is a really great resource for getting started and they have a whole bunch of default so I think I guess foundational rules that you can use to scan your logs and they explain how to filter in cloud watch so it's really nice and they also explain a little bit about why the rules are useful but it's usually quite obvious as well so in that PDF there's like one of the things we look interested in is root access they say here's that filter pattern you might want to be using for root account access so if we apply that back to our JSON log that we captured so you can see that

this user identity is root there's a key missing for invoked by it's terrible the schema-less schemer it's very hard to work with sometimes and then it's making sure that this is a idea was not a NATO base service event so the other the second two are anyways can assume your root credentials on your behalf to do a few things which I'm not sure why you'd not want to alert on that but anyway so let's do this and string it kinda like the workflow that I've been doing for stream load is to find a sample event like we just did you can write an integration test and then with the expected outcome the test fails you

choose an output to alert on you describe your alert message and then you write that logic test passes and deploy happy happy days so luckily actually I didn't have to do any of that because streamlet have some default rules already and writ account usage is one of them so this is the entirety of a rule that you'd need to write and it's a Python function that ends up with a lambda so you've got a bit at the top that's kind of describing you know what log sources is coming from maybe what select channel or maybe what page of duty integration you want us to go to is a handy helper for required sub keys because of this tricky schema-less

schema and then the doc string is actually your description of the rule but that becomes the message that gets alerted then you implement that logic with Python and return true it's gonna alert false it's not gonna let so yeah ploy that it's a pretty pretty easy to get going because I did it at the morning tea break on my hotspot deployed to tear off just to make sure it's still work so yeah it's not too not too bad you can do it in about 30 minutes or so but you can yeah you basically check out the repo set up your config and then you can deploy it so just gonna show you that quickly running so we're it's three

here that root account usage let's just do something let's just create a bucket

so it's gonna create a bucket that's gonna create one of those vlogs because it's like an API call to a double used to create a bucket not everything's gonna create grab trail logs so if you're just browsing around it's not going to create any logs but say it's things that are gonna action things that you notice account so a in my slack we have the thumbs up emoji stream alert triggered real account usage and it gives you gives you the rule that triggered and then some and in the details so this is you get all that output there wasn't the rule but you have that description that came from the beer right so there you go came through

to my select channel that wasn't me that was all stream load guys really clear guys at Airbnb and it's yeah works really well now this was in case a demo didn't work so I said face but it did work so anyways and so other inputs that you can grab OS query and Google Apps admin one login duo there's a bunch of others as well you can add your own it's pretty easy this was a slide to explain the difficulties of cross regions I was gonna skip over that because it's too much and some outputs that you've got slack page duty and Fanta mystery buckets maybe another lambda function so you can really go crazy here and the

output to write your own output it's pretty simple it's not that much and I think the Select integrations all of I know 100 lines or so so it's it's pretty good so at the end what are we what kind of things are we loading on this is one of the rules I've written so it's assuming a role so we're using role based role based I am roles to assume assume roles that have permissions attached to roles so we might have a really really high score the administrative role and if someone assumes that they can still assume that they got the permission but we might just want to know and then we can perhaps message them and find out

what are they doing with the super awesome privileges that you only really need to access a customer data so we can just kind of follow up so yeah it's quite easy to write that I've kind of modified the description a bit to allow me to insert just path and formatting syntax so that we can get a little bit a little bit more info brighten their message make it like a really impactful statement so user Tom Jerry has assumed administrator from this IP address and in a bit of information and in a link maybe it's not the security nurse looking this maybe it's someone who's on call they might know a little bit of help on

what to do yeah so a question I got asked when I was running this through a colleague at work was like well how do how do you make sure that streamlets like who's monitoring extremely you know what happens if you delete something there and they've thought about that and they use a TOS its own sort of a loading think I'm cloud watch matrix so you can set up separate metrics to alert you on things like errors or processing areas in-stream alert or things like that that so maybe someones added a syntax error into your rule and it's just not running you might want a lid on that before you miss it a really important alert so

coming back to lp0 on fire and i was wondering how do i do this with lastrella right so they're still thinking this is chaos engineering from from like chaos monkey and things maybe there's a chaos security where there's a look going on you want to sometimes like you know in an important event just test test if everyone's still still on their toes maybe we could add a Twitter output you know open history bucket I just post to Twitter so that people really quickly go and fix that actually using chaos monkeys so you could maybe trigger some chaos monkeys chaos monkey and going like destroy ec2 instances and a whole bunch of other things as part of the chaos engineering

at Netflix so they basically trying to make your code more resilient but maybe you don't use Karis monkey and you just edit in here just start destroying things whenever something important's happen on your network like someone's got Rudy can access and suddenly other things are starting so one of the thing I was thinking of as balloons balloons reduced stress right it's an important important incident happens you need to maintain calm so I added added my own slack output my own output type called balloon let's see if I can switch to it so for example here critical critical API calls it's gonna go to our security alerts slack there's also going to go to balloon slack so

hopefully it's all works because it almost didn't so I should explain have I got a have we got a young on this night yeah cool so I bought an air compressor but I have a problem I think it's kind of so and I went to I bought a this is like a smart plug you take not really knowing what I could do with it at the time it's like looking at it it's really odd in the shop when you have to decide like can i hack this for what I needed to do like it's gonna come with some crappy app and it's gonna say I have some cloud-based thing and maybe it's like if there's still I don't wanna do

that like mmm how bad is this and that is that good right I managed to pick this one for $50 and luckily two guys had already reverse engineered the protocol and they did actually use XOR encryption sorry X or whatever XOR is right an XOR cipher to encrypt from your phone to the device on first setup which is absolutely terrible but it meant that there was a Python script away welcome to control this device I didn't even have to install the Android app it was just great so that's controls this there controls the air compressor and then we've got this balloon so is it gonna work let's see if we got back slides so let's do some

critical API call we could delete a trail so let's go jump over to cloud trail

I think actually it's going to trigger on the route account usage as well so we might see some some goodness heading yeah so let's grow rails let's create trail like even this creation of the trail is going to trigger it actually because it's a it's a I'm using my root credentials to do this I was doing this the last few minutes before the talk don't judge me why isn't letting me create a trail

ie credit so let's see if we can get an alert on that that's it [Applause] what is it doing I was always supposed to go once what happened are there's mini posts to inflate maybe they created multiple or maybe it's creating the bucket creating the cloud trail creating like the cloud watch event group yeah that's not good okay so this go into some more things so you can imagine you're working away and that thing like right that's create let's create a new bucket bucket insecure because you guessed it it's gonna be read only I mean read wide to everybody hey this is gonna be real read one from the world create bucket let's go back it's about

10 seconds to go through there

I thought the smiley face would like help calm things also it was like the only one at my Pattie store that look do any good it was like that in a pirate-ship it's hard to choose so here's all these extra lips that have come through so we had a cloud trial put bucket root account usage wall root count usage basically music root account so that it's just gonna go keep going off for entertainment sake what can we do that creates a lot of things maybe we can creep maybe we can create a bucket from the command line let's delete this delete this bucket a turrible is cute

you understand how amazing this is that this demo is working so I'm relying on a crappy IOT device that uses it so to do this from my laptop which is in u.s. East one and it's doing like a streamlet thing Wow we gotta go faster let's just go create more buckets

that's fine he always amazed it like you do then sometimes it's not unique mmm who are you more we need more I know we could do we could just up the time that it's on for as well you could appreciate this all this this is my terrible idea there's nothing you'll appreciate how amazing this encryption is yes let's just go for gold tweeny do you guys get nervous with balloons that keep inflating let's make sure I restarted our thing oh no no syntax where there was one one five ah there it is that's rid thank you okay let's go let's go let's go more what else can we do let's go and it's that slowly deflating as

well and this thing gets hot too so I thought if I put it what else have we got I could actually do some critical API calls but then it might stop the demo oh I know we could do we can delete that trail that we created that's why I created it because it'll delete a whole bunch of things as well trails bed trail oh well I haven't done it yet it's gonna go for 20 seconds what was it

I can't think with all this balloon going on oh wow this making your elf uncomfortable sorry okay now we can delete it turns out I chose that 20 seconds just great I only tasted this once because I had to hold it while I was testing it that was there was intense takes one minute with this crappy compressor yeah deleted it now should be 10 seconds oh yeah

okay okay this is from the tool shop come on [Applause] Wow I said thanks [Applause]