Distributed Blocking and Other Security Fun for Your Cloud Using Consul

hello

I just just before we get started can we please give it up for the guys who actually got the conference working I think it's amazing [Applause] so distributed blocking and other security fun this would be kind of a framework so I want to get your head booted in the morning and we'll see how that plays out so Who am I I'm this bald German I'm dirty deaf on Twitter and very much on Twitter very active on Twitter I don't know how it happened but it happened I'm an ops guy so this is actually my first security conference yeah well but this is a bit too formal what the hell this is b-side so yeah I'm also a biker I'm doing stuff

with the cloud and we're a startup from Germany and what we do is yeah well how did I get into security who can tell me in German who can tell me what this tells me what does this mean what could this mean no it actually means there are images being transmitted over non-security 9 the the funny thing is if I don't know if you can read it the article says that your stuff is being more secure within docker so I don't know how taco played out in the security community yet but we're constantly making fun of it but the guys they try their best just with tor yeah well so what in an obscure door on a security conference well just

like with everyone else it started with Twitter and this horribly explode exploded and other stuff happens on Twitter as well when some American guys asked me to provide them German locks just the stuff that commonly gets sold in Germany then this got a little bit out of hand and suddenly you realize your lock dealer it's tricky thing with Twitter so just before we get started it's I work for for a start-up so you've got to get that one a shout out in Stannah it's called what we do we are a p.m. so application performance monitoring don't know if you ever used one monitoring is kind of a shitty field and a big unsolved problem which we're trying to attack because

basically this is what you're being told should be monitoring when it's really not so we try to tackle this problem a lot of them are a lot of the existing ones are either open source and they're pretty much useless because the time is being put into the wrong places so what we do we provide you a 3d map with your machines these are little boxes on the hillside in these boxes there are little sub boxes with applications in them with stuff like my sequel or an engine X they get red when something's not right and these are the kind of informations that you need what what would you we do when you set up open source stuff

you define thresholds which doesn't help you at all that gives you stuff like informations like your CPU load is high and CPU load is high can be anything and it doesn't tell you that and that's why we're building knowledge into this thing so it can actually later tell you that you have either have a split brain or that one thing happened when one of the data centers of Amazon got burning in fire these are the kind of informations you need to have when you being woken up at 2:00 a.m. in the morning by page of duty your CPU load is high is it's not the thing you want to read when you just got woken up so we become totally

batteries included so you really don't have to do anything you just deploy the Java agent to all of your machines I said Java yeah the first person who can tell me why we need to deliver this thing in Java gets a beer why when you monitor software in Java why do we have to why do we have to pick Java just ok huh yeah when you want to monitor a Java Java process you need to have Java otherwise you can't get the metrics out of it so that's basically why you have to pick Java you throw Java agent on all of your machines and they show up then you can get all the information you need

on the machine on the software it is and all the metrics it has and we kind of we build a knowledge graph of this machines and how they interact with each other and as soon as you do this you can do amazing stuff like like well what the competition does is basically provide Gras but this is boring this it might indicate the narrow when something occurs what might have happened that other stuff is more fun like graphing all your all of your logical connections from from one point to another and you can do other funny stuff like show me traffic and when we have those logical connections between applications in your cluster you can do funny stuff like we

also give you traces another problem that hasn't been solved properly yet by our competition when something occurs like an error where one of your machines gets gets unavailable which happens all the time in the cloud and especially in u.s. East one when you want to freak out an ops guy just say you as East one it will probably show an error here and you can see what happens and this will be the information that you need when you're being woken up at 2:00 a.m. in the morning and yeah this pretty much says all about docker yeah it's our product and our incidents are as we build this knowledge graph of your applications we can not only figure out

what might be what might be the indicator but also what might be in where the problem where else the problem can occur so we have a CP oh wait that's high on one machine and we see it's connected a lot from a logical perspective to MongoDB and it says that this MongoDB ax is going to maybe run out of memory in two minutes just because it's being fired on with with requests it cannot process but back to the presentation and yeah we basically support almost anything case you wonder so what do I do all day actually I'm a classical ops guy in the cloud world so what do I do all day the one thing comes with another I

can someone do you tell me which output this is of which program this rainbow text its kitchen CI that is a is a ruby program that you can use to fire up cloud machines and test your stuff on and when you're testing cluster stuff you have to get a lot of machines up and yeah this is the reason why I drink basically up there which program is this you should note that one I just want to get your head started in the day it's Jenkins it's a Jenkins pipeline so conlou why Khan Zulu what's consul it's a service discovery tool it's written in go and it's you can throw it on your machines and your machines then later have the

ability to say which IP does the clock cluster software I need to access right now run on this is one of the one of the use cases but it comes with all funny also all sorts of funny things and it's a distributed service discovery tool so it's also a key value store you can throw informations in it and access it via Jason speaks HTTP it's also coming with service health checking and it's modern so it's as I said written in gold just row a binary and all of the machines and it's comfortable and it's actually working it's when you're a startup and you have to shift between different technologies all day to see what works

you really can rely on a software that doesn't break or breaks for obvious reasons and when these reasons get more and more obvious as software gets usable for a startup it works across availability zones this is not Amazon speak not cloud speak but can be in the traditional world and it's being used by very large companies like digitalocean run their service discovery with console they I think they made a kernel patch to raise the arm up cash in the Linux kernel just for that as I said you write a bit of Jason and I'm apologizing for the bear readability I didn't know how the light was set up will be you just throw a random throw some Jason in it

and use the binary give it to Jason and pay a BAM you also get a this nice-looking web interface which you can use but don't have to basically it's just HTTP so the cluster speak with each other and your machines can access the information locally so it's written by Hoshi Corp and it's a startup that writes nice software maybe one of you recognizes one of these logos anyone maybe you know vagrant ok yeah these are all tools that help ops not having so much despair and they're all they're all pretty good actually so first of all you install a server but you don't install one server because it's distributed software one server might fail and then

your information get might get lost so you don't install one server you install three servers at least well this this distributed software fund where one has actually needs at least five minutes to explain why installing an even number of servers is a bad idea but it's basically just to have it reliable you should install three or five service in multiple availability zones and how would you do that you go to the to the console dot IO page and download the pre-built go binary yeah it's basically called bash but if you want to talk about if you want to talk to me about the security of package managers I think cult bashes may be one our best options because have you ever

tried to sign our RPM file it's well sad you download the thing you write some Jason it basically just said where it can store data how much service you expect to bootstrap when the bootstrapping happens between the servers and then you write config for your clients there's all sorts of informations but this is just trivial stuff and throw in boxes with agents your cluster setup you could to go and what can you do now you can ask your asked by a local HTTP where is my Kafka or where's my elasticsearch where is my Application Firewall you can check if via local HTTP is my stuff healthy this may be my webserver not available on port 80 you have all sorts of options

there you can even throw in a bash script that gets fired every define a time interval with vault another of there another one of their products is a as a security store it also got audited and got pretty good ratings that can also generate open SSL certificates on the fly it's amazing tool and it also comes with a command pipeline that means you can define watches that if it's somewhere in my keys keys value store with this prefix gets changed then do that and this is being built into very much products almost any of the Cape's vendors have one like ansible tower you can throw money at chef you can throw money at puppet you can throw my

at saltstack if someone would actually use it and theirs off this problem they give you a command pipeline here you get it for free but yeah this is beside so DevOps what does DevOps mean DevOps means also that we might just as well for our purpose which in this case is security use the tools that Ops guys use anyways maybe for a different purpose maybe you need to up the cluster but it's doable and you have the knowledge in-house which can be which can be quite a benefit and basically all we want to do is this so yeah deficit hawks what do we do we do this we use this tool for sex stuff so the

first thing I came up with was just this trivial thing that someone actually recognized this logo no fail to ban rings a bell yeah does iptables ring a bell this awful API that no one can come up with an instant answer of how can I block this IP because you cannot memorize all the parameters and the order of the parameters fail to ban basically just watches your locks and if something weird occurs that shouldn't occur then it goes there and blocks this IP where this originates from of course there's lots of different lock layouts that you have to specify where is the IP and where is the indicator and stuff but it basically just does that and it does it

pretty good it's doing a good job so we using these console watches which I talked about you can set in console you can set these watches on the key value store you can set it on on changing of service health you can set it on

entering of new machines in the cluster entering of new servers in the class almost anything so these watches are integratable into the adjacent config as you set up the cluster there integratable into console via the CLI on-demand like you can when you use Jenkins or something similar you can build a job around this and let's the sec guys enter rules in there and then it just bootstraps it into the cluster its trigger above are all across the cluster and this is what we're using for this particular use case it can it's basically you can fire off just the script and this is super super useful because this creep can do anything you want it can even register some new users in

your website it's anything and therefore it's as powerful as you are all your options that you're doing manually on the CLI you can just throw it in there and it's reliable means that the the trigger will the watch will trigger and if it triggers once it triggers reliably on all of the machines and if they're not available they're the it triggers afterwards and it's exchangeable so you can flip the scripts and give them different names and add new ones the old ones and it's cheap means it's it's not a lot of effort that has to be put in for adding a watch fail to man I can come up with an operating system that doesn't come with it come bundled with

it and I'm again to user so yeah it utilizes IP tables yeah this is a trigger for some people really ops guys get disgusted when you say IP tables there are a lot of existing plugins that tell failed to ban how the log files are being structured so it can pull the information from elasticsearch it can pull the information in push the information into elasticsearch it it really comes batteries included and you probably already use it though you don't know you use it but that's another DevOps thing and it's no biggie to integrate it's really just a package to install and when I'm talking at a security event I'm talking to people probably using huge and probably pretty

expensive firewalls stuff and this is also just because it's just a script trigger able from the outside so what do we do here I have this in sadhana installation that says I'm not monitoring yet but that's not important important is that before that I have a login field and in this login field I enter my credentials and I entered them wrong and this of course triggers a log entry and in this log entry I can see that there was a that was a that someone entered wrong credentials and this cost this Lowcountry and this comes from that IP and caused this GDP status so basic stuff we all seen that so the next thing is we're writing

a failed to ban entry that just tells failed to ban how these log entries look like where the important information and then fail to ban can go on and do it stuff block the IP when it entered the wrong credentials for three times and this IP can either it's your choice not not access the port of the website again you can manage to get them have a login lock or you can just bend the IP from the machine in general so it cannot access it anymore anyways but that's basically just simple stuff you this fail to ban has this CLI that you can use while you're developing these regular expressions because regular expressions you'll make errors you won't see them so

this tool is pretty handy you give it a lock file and you give it a reg X and you say look for this and then you say and then you see it didn't found anything and then you see that you made a error in writing the reg X it's called failure to ban reg X and yeah it's just well the thing is I want to it's still in the morning hour still booting so I want to know from you having the information that watches do one you can use watches to trigger one thing all across the cluster and having this information that we using fail to ban to block an IP what is obviously the thing

that I want to know or what is obviously the next screenshot you would see

no you use fail tube and been an IP and use console to watch stuff all across the cluster so what is obviously the thing that comes now no you get this IP block from one IP and all machines block it instantly you also have this key store so you can either use the use the console watch in the key store enter a new IP into the key store and all the machines read it from that or you use the you use the watch on the on the trigger of the failed man and all I piece block it instantly so that was basically the thing I was trying to say now so that was basically it

so the thing is why I actually gave this talk is to make people aware of this program because it has all the abilities you can come up with you can do this exact the same thing with your own security appliances and yeah it's really no biggie to set up it bringing up a new cluster really it just takes ten minutes reading the API it takes maybe 30 minutes and they are all set this reg X can though if you're using fail to ban cause pretty much a big headache because reg X but it's basically I just want to make you a show aware of the program so are there were any questions I wanted to

leave some time for that and especially since these this is not an ops conference did I make myself clear are there any questions yeah

are there other security yeah well you can distribute what do you call it threat threat Intel you can distribute that eyepiece you can you can well this is basically the thing you should have come up with being obscure being SEC persons because I want to know from you what the use cases are I just gave you the basic features and the thing I wanted to read is the word that it's really the pipeline is powerful but it's as powerful as you are and if you have those things like a mail cluster like like security appliance they're all just being trigger abou either via CLI or API and it's just just a script so just this

script is really the most powerful thing here and having those information informations being spread distributed and reliably is a very powerful feature so thing maybe well this is the first thing I came up with give me two beers and I give you probably 30 more of but it's really you guys that can probably come up with a good use case but I just wanted to spread the word about this thing so questions anything yeah

i we use console and I'm working for a start-up so when you work for a start-up you show what it does so everyone knows that it is this anything yeah no it it uses that the trigger is actually lock files yeah that's that's one way or you can use any other appliance basically you can tell it what to do in rules files written in written in their own weird seal I syntax and you can show it's also just a script so we can do almost anything you can send your your security appliance just an IP and via HTTP perhaps and this gets blocked and the thing you're using console for is then to block it instantly all across

the cluster yeah failed ban is one of these one of those things that a lot of people know when they touch Linux boxes and maybe operate their own website and it's it's a lot of fun I really can recommend setting up a wordpress site on some shitty VPS server and enabling failure to ban and then see what happens it's magic in the cloud it's really it's fun I think

thanks guys you [Applause]