Managing Security with the OWASP Assimilation Project

all right can everybody hear me okay welcome to breaking ground I would like to Welcome to our first talk of the day in here big uh thank you to our sponsors for making this all possible otherwise I'd be at home wishing I had money to go to black hat um this is Alan Robertson he'll be our first Speaker talking about automating security with the oos assimilation project and with that I'll hand it over to him and thanks as he said I'm Alan Robertson and this is about uh an open source project which is recently Affiliated itself with OAS but uh it has security aspects and non-security aspects as well so you might hear of it as the OAS assimilation

project or you might hear it as just the assimilation project um and a little about me I spent a long time in it uh coming up on 40 years actually and I founded the Linux ha project or pacemaker people here heard of pacemaker for failover or Linux ha I founded that project ran it for about 10 years and I started this project in 2010 inspired by some work I was doing uh for the government on a uh uh a 2.2 million core supercomputer and it was not a normal cluster it had one clock for everything it wasn't like a bunch of machines operating separately and started making me think about scale in a way I'd never

thought about it before and that's what prompted me to start this project and I I started a company around this in 2013 after IBM showed me and 8,000 of my closest friends the door all at once and um I'm worked at Bell labs for 21 years worked for Souza for a year and IBM for 13 years or something like that so uh let's go on That's So as you think about security uh how many people think that it's easy to get good security staff okay do you think security is going to get better anytime soon do you think you have enough staff if people would to switch to devops or agile things you think that that's going

to make your problems easier and you're gonna be able to keep up with that what's what's up with you guys what's up with you guys right experience oh experience there we go that's it now something else that's true about me is sometimes I make claims which seem a little outrageous sometimes people and I'm told they didn't let you bring in they don't let you bring in Rotten Tomatoes and eggs so I brought some for you and so you folks can uh oops that's the pro I shouldn't do overhand throws I hit people when I do that so yeah yeah there we go so if the if the time comes when you say you know I know why

his eyes are brown it's full up to here with you know that brown then then you'll know what to do and I've uh I've never thought about it before but I guess what I did is armed the audience maybe that's not such a good idea a security conference so let's think about things that happen here that people don't do really well 30% of all break-ins come through systems that are not in the inventory oh we had that system well I didn't know were we patching it no no no Maybe not maybe not didn't know it was there 90% of everybody has had failures of services that they're not monitoring and this is a by the way this statistic

from James Turnbull comes from people who are running chef and puppet which are bleeding edge compared to the industry as a whole right um that's as good as it gets and of everybody isn't monitoring all the servic they know they have 71% of people once they get in compliance with something like a PCI oh a year later we're not in compliance oh what happened I don't know um 30% of people well we really only start monitoring things once they break um and these are of the best people in the industry right so how many people think the real world is probably worse than this yeah yeah exactly 30% of systems aren't doing anything useful at

all but they are making a nice contribution to global warming they are space heaters right they space heaters they're doing nothing these are all other people's statistics which I have every reason to believe are reasonable and and I think you guys have seen the same stuff yourself when I talk to people who work in security they say yeah yeah but you should have met my customers you wouldn't believe what they did they go on to tell me something that well yeah I kind of do believe it unfortunately um so a little about what the this project is for and I apologize for the size of the uh oh that's not interesting that's it's not U there's

some things not oh that's interesting the automat yeah so anyway sometimes your animation doesn't work right you know so we do things like track IP and Mac addresses uh validate continuously validate configuration against compliance rules uh we track software versions uh we track check sums of network facing processes uh monitor systems monitor status of servers and uh systems servers and services uh score we we produce scores for your machines basically risk scores you know it's like golf except you can actually get zero in theory a low score is better than a high score so you can think of it as correlated to risk sort of correlated to risk you know anybody's mechanical scoring is never who can

agree on what the risk is anyway but the point is that the low numbers are better than better than high numbers right uh we also uh uh we also uh discover the same kind of things for Docker containers as well as VMS and and uh VMS and bare metal and clouds um we alert on CH we can help you alert on changes in configuration or status so that you can actually be aware of when things happen and we do this in approximately sort of real time like uh that is to say within minutes to hours of when the changes are made that that doesn't mean daily annually or quarterly that means when they change and we do it

with basically nearest zero configuration and we do it without any pings or Port scans so you know it's always bad when you have a security tool it sets off your other security tools that's a bad thing you know uh so you have to turn off your your your your uh other security tool in order to security tool a to run to be able to run security tool B uh so this is what we do and we do all these things together in a way that scales really well by scaling really well I mean is like on the order of 100,000 systems um so this is when I make this kind of claims this is where

people start thinking that maybe those eggs and tomatoes are good ideas uh we do our continuous Discovery we do this continuous discovery which drives everything and we and we create from that a configuration management database a similar logically similar to the to the graphs that are passing around through the audience now um we have the discovery is done in a way that we don't actually do any network footprint to discover anything now to communicate the discovery yeah we do that we do talk on the network to do that you got to get the data there somehow uh the discovery though interestingly enough eliminates most configuration it is not quite configuration free but it's pretty close

to configuration free so the thing about configuration how many people think when you configure stuff it stays configured correctly forever no right right oh it's that experience thing you talked about that's it yeah the so if it's configured automatically it's more likely to be figured figure correctly um we also Drive best practice analysis which you could also say is compliance analysis because what you're fundamentally whether you call it compliance or whether you call it best practices the idea is you want to be following some rules that you think are good ideas and when you don't you want to know whether that's an official audit process or whether it's just good hygiene on your computers I don't care

it comes to the same thing and people say well that's just the basics okay most people don't do the basics very well so let's start by doing the basics well which is a lot of what we do here and as I said we we put it in a graph database which basically means a graph database is basically like what you draw on the board you have circles and arrows you know relational databases have tables rows and columns in a in a in a graph database you have nodes that have attributes and relationships which are the arrows that have attributes so it's circles and arrows in a database instead of tables uh the nice thing about there's a

number of nice things about it but I I won't go into that a lot more but it's in a graph database instead of a relational database so scalability without a lot of extra complexity you know this is just out of the box Discovery without pig is report scans how many people think that sounds reasonable able now I okay guys I guess I should have said how many people think that doesn't sound reasonable yeah yeah sounds like sounds like that maybe this is a good opportunity for those tomatoes and eggs but hang on a second here let's look at the scalability aspect because this is my favorite thing to talk about so I can explain how we scale so that

your grandmother would understand uh even if your grandmother isn't Grace Hopper um so at my church on Wednesday night we get together for a meal we stand around hold hands like this you you know and we kind of a circle with our eyes closed and while we're there with our eyes closed and the pastor's praying over our food if Aunt Sally passes out who notices first in this Arrangement the two people holding your hands now for you to participate in this human monitoring Arrangement how many hands do you have to have how many there we go two you have to have two hands because everybody doesn't have pretty much most people have two hands and and so if I add a

thousand people to that Arrangement how many how many hands do you have to have do you you have 2,000 hands oh my goodness do I said how many hands do you have to have you have to have two hands right did your work did anybody's work go up because I added two a thousand people to the arrangement nobody's work went up so that's how we scale we have our computers hold hands well okay they're actually heartbeats but if you wanted to get like really literal you could put your fingers on their on on you know feel their heartbeats through their hands and that's literally what we're doing here we're exchanging heartbeats with two neighbors and now the interesting thing

is now if you have this kind of arrangement everyone is being monitored by some by two other people so you can reliably know if a machine goes away and you can just sit back and wait for the phone to ring wait for someone to call 911 and you don't have to do anything at all because the monitoring is all fully distributed among the systems that are being involved so um that's how it works that's the fundamentals of how this scales the way it does and once you have the scalability then with agents everything else falls out because most of the I mean you can have a system how do people think people like to change systems or

is it more that people when you screw with them you mess them up right people like to leave them alone so once you've told us your configuration we only get notified about changes which means that we might have a machine we haven't heard from at all for a year but we know exactly what its configuration is and we know it hasn't changed because we have a reliable way of knowing if that changes through our agents and if our agent goes away then the then the two uh neighbor agents complain that it's dead so we'll know it's a nice reliable way of doing it's no news is good news but it's reliable at the same time which is sort

of contradictory but it's actually works that way so a little about the architecture here and now we're going to do a demo here in a minute let's see how far I am into this I'm about 12 minutes in um so there's a central system called the collective Management Authority this is the assimilation project and yes we we have Nano probes we inject your machines with so that they can be assimilated and join the collective those of you who don't know Star Trek may not appreciate the joke and the rest of you may be groaning that's okay I'm I'm I'm okay with bad jokes um they're here somewhere now bad bad bad jokes don't count just if I'm full of brown stuff oh

they're over here so I did actually collect an egg once U but I deserved it I sort of Drew it you know like you can draw a foul I drew an egg um so we have Nano probes which are the agents on all the machines which then do the heartbeats and and then do the discovery and do the monitoring and send everything to the central server but only when it changes only when it changes and the the Nano probes are written in C the central server is written in Python and we have this graph database that we use called um um neo4j which happens to be written in Java I don't particularly care I talk

to it over rest um here are the some kind of analysis of reports we provide comparison against best practice hardening rules uh they default from a project called the it best practices project which is an open- source project whose purpose is to collect uh particularly hardening type rules things you can observe mechanically which people think are a good idea to to be in compliance with uh they current most most of the rules there were taken from the dis stigs uh which are mostly good rules uh some of the implementations of them sometimes people have are bad but they're mostly pretty good rules might be a little excessive but you know there's nothing really wrong with them

either uh unknown IP addresses no it's what IP addresses are out there that we don't know what they are uh we monitor Services as well as I think I mentioned this and we can tell you what services are unmonitored and we do a triage I let you so imagine that you have a hundred let's say for for example for the dist digs there's about 250 rules for a Linux box and the average machine's going to fail about a hundred of those out of the out of out of the box so if you have a thousand machines and each one has 100 failures you have 100,000 failures so now you need way of approaching this to

let you eat this elephant in a rational reasonable way so we provide a a scoring system in a triage sword order ficially says here's the machines with the most problems or if you prefer here are the areas and rules that cause the most problem across all the machines and you can do it either way and of course we provide you the information you don't like the way we sort it sort it your own way um but that's the idea of it if we want to help you manage that process and then you can graph the total score score and see as how how it's going down over time because if you don't intend on fixing this you actually don't want to

know what what ways you're not compliant you only want to know if you plan on fixing it because otherwise it looks like uh you know when lawsuits come and there becomes disclosure and things like that you want to show that it's going down not that it's going up or staying flat you want to show that you're doing your job but the good thing is now you can how many people think they have managers that like graphs yeah so we could do a graph that let them show how you're actually getting your job done which I think that's a good thing right by by by by you can eliminate rules yeah that that sometimes that's the right answer but you can also do it

by actually fixing the problems I mean I know that's a radical idea but you could actually fix the problems but uh and uh so anyway a little more in the best practice analysis they're triggered by Discovery updates and we don't EV valate them daily weekly quarterly or annually we evaluate them when something changes gosh wouldn't that isn't that interesting in other words you know almost right away that something has changed and then you know that you either got better or you got worse when that happened right uh you can discover more or less anything you want writing a discovery agent basically means whatever domain you want to discover for example maybe Pam rules maybe proxis uh maybe login uh SE login depths

whatever you write a script that spits that transforms that into Json which is not rocket science and you then you teach us where it is and we run it for you and we keep it up to date in the database not hard then if you want to write rules based on that you can or if you just want to have it in the database so you can query it there you go it's there and as I said there are alerts and various kinds of reports available and and here's some of the stuff we want to do in the future because once I get into the demo I get carried away so I want to tell you this because I don't want to

forget this we want to coordinate with security vendor updates so for example when red hat comes out with a new patch we want to then ding the security scores of all the machines that have that vulnerability and not unpatched vulnerability and of course when because of the way Discovery Works once you apply the patch uh then then that score would go back up uh because that's integrated into the process as well what we're not what we know what packages you have in the machines we just don't currently update the security score based on that we ought to uh so we want to do interesting things with check sums we currently collect the check sums on

everything that's Network facing we you know of course then going to your vendors and getting white lists and bringing them in that's a good thing black list for some things that's a good thing and one of the other things we can do because we have it all in one place we can do what I call minority reports which is to say did you know you have 99 copies of this library that are the same and one that's different oh maybe that's not a good thing right right or maybe it is a good thing maybe it's the only one that's not hacked but it's something worth looking into in any case and and we want to

integrate with Sims and we'd like to do add some role-based access controls on on the uh on some of the queries so that you can do this more in an environment where you don't have to be a security person to be allowed to get out this data so that people can see the data that's appropriate to to them according to their role and the question I want to ask you and you can think about this and I want to invite your in uh input is what else should we be doing right imagine you could know anything you wanted to know about your systems and then then what would you want to know and what would you do with it what would

you do about it right and so that's kind of our that's the space we play in here the question is what's interesting to you because it doesn't matter what I think's interesting nobody cares what I think uh what what matters is what you think what matters is what you think so a little about the demo uh that in this demo everything is discovered nothing at all is configured manually I don't even tell it where to find the central server it happens to be on the same machine because that makes the demo easier but it doesn't have to be it just it discovers that as well we we'll we'll show you things that need hardening show

you I'll talk show you walk you through the process of what this sword order looks like for the triaging of of of problems uh how to how to um uh um how to visual I'll do a visualization of an attack surface because I know all the the I know all the services you're offering I know what ports are on uh and and it's in a graph database so it's natural to be able to visualize some of these things and I'll show you a little demo of what packages and versions are on machines and and it'll include a little Docker Discovery Well I may not be running a Docker instance right now so if if I'm

not it won't Discover it uh because I don't think I set that up this morning so but it would if I had it running so uh so the demo now do this and go over here okay so I have a script cleverly named script called demo and it'll Wipe Out the database oh yeah right sud do

so it's actually going through here this thing is back to not working very well uh but anyway doesn't matter up here at the top it says erase DB run into foreground so we're starting it up it's going in foreground we've uh now it's it says in the resetting connection that means that it's now gotten a connection from one of its agents right uh and now it's starting to do Discovery and it it's showing you what rules it's failing here this particular first Discovery is off of etsy audit.com and it's saying that we passed some rules and failed some rules and the security ID is it says n those are actually disa uh Stig identifiers um and we'll show you that

show you something else that goes with that in just a minute uh and now we're EV evaluating some best practice rules on shd shd configuration um and we passed some we fail some and this is all happening as the discovery comes in uh actually I deliberately kind of slow the discovery coming in so that if you start up a bunch of machines at once we don't get flooded so much it you know spreads it out over a couple minutes um and now we're going on to some things uh where are we here uh file and directory attributes and we're passing some and failing some and um so in this in this case by the way

this failure here was on uh is says that one of the Shadow files was readable by group readable by anyway it doesn't like something about that I don't remember exactly what it is um and we're going on here and it's disc as it's doing discovery on this set of attributes that set of attributes this set of files and so on and it continues on through here now and it also says here about some services are now operational so once it's discovered what services the machine is offering it's actually going out and monitoring those Services as well it says well you know I see you running running uh neo4j why did I just monitor it for you I see you're running

sshd why did I just monitor it for you so it's doing this kind of stuff here it's coming down through here and the last thing it's come up with here is now I did a tail on yeah a tail on uh I think this is I have to go over here yeah so I did a see this is in the this is the tale of syis log so if you look in there it's got a it's got a URL into the it best practices. info project I people in the back can kind of see that I hope a little bit but if you open that up open link um I'm not sure quite where my

browser is it's in the wrong window let's put it over here no I don't want you to restore it so people in the back can kind of see that too so it tells you that this is a medium severity problem and you're not supposed to set icmpv4 redirects by default and it goes on to explain it in a little more detail and how to check if it's correct and if you continue on a little further it then tells you how to fix it because actually checking this this value is like is value equal to zero right that's a very simple piece of code but the explanation for it is a lot harder it's actually because if you you

don't know what these rules mean or what their significance is you're probably not going to fix them right and so the the tying of these two together I think makes it much much more useful and this basically is the fundamental of what the it best practices project is is this kind of information right and it there's a website for it and there's there's a um Source control under GitHub and it's an open source project right um and there are a few rules there that don't come from dis digs that I've written and actually it doesn't have to be all security rules there's actually one networking rule too but this is the idea of what this does now let's go on and

see what some of the other stuff I thought about doing in the demo here is oh yeah this this is cool

um so let's make this bigger oh yeah it did make it bigger oh no went to something else that's not the right file what did I do I think I went to the next file I'm not sure exactly how I did that so let's get rid of this program and try again view zoom in okay so this oh it's on my screen but not on yours oh let's put it on your screen okay so this is a visualization of the attach surface for this machine right now and I'll see if I can explain this it would be a lot better if this thing were working other sort of working no it's not working again oh this is the

one that worked better that's why it's up here yeah okay so this is the this is the system this is a representation of the system itself you see it's in bold red and it says we have a risk score of 57 the way the risk score is calculated is if you have a low priority violation it's worth one point medium priority is worth two points and anybody gonna guess what high priority is three so uh we have a total and that's why it's red is because it has it you know basically illustrates it Bolder and Bolder up to a certain point and and of course if you don't have any it would be black and thin right rather than bold

and red so now it's decided to stop working again I need to replace the batteries before I got here I guess but um so this is so what we have here is a bunch of different Services here is a Java program uh Dropbox now it's unfortunately it's scrolled off the screen here let me see if I can scroll this I'm not sure if I can or not yeah okay um so sshd is running here as root now this looks like either I had a bug in oh this is an old version of this file that's the problem um this should be read uh and in fact in the current version of the software it is I just I cach this file

because it's takes a little while to run so this is SSD which is listening on the these IP port combinations this is Dropbox which is both talking out an outbound connection to Dropbox here and and an inbound uh connection to a lot talk to other people on Dropbox uh RPC bind uh this is neo4j and I was apparently running Veno server at this time as well like I said this is I'm not sure what happened to the version I had here that was oh right and this is a different kind of visualization which I can't show you because it's too um it's too it's not bold enough uh so anyway the idea is that these are the

various these are the IP ports that people can get in that that you're offering services on up here are the IP ports for services we're requesting outside thanks um so anyway this this is an idea show you an idea of what the visualization looks like um let's let's go on to something else um sure this is a good one um so let's look at all the packages on the machine and if I can go back over here get rid of this okay maybe I should sit down apparently I'm not very good at mousing there we go so let's get rid of this now okay so let's go um go here control shift V so this is a query on all the packages

on all the machines so that's um so what we have here are this machine system 76 has a package called Ubuntu one client data I mean this is just some of the examples of things on here that's all the packages and all the you see the version numbers out there too and what kind of pack these happen to be pip packages it finds pip packages uh RPM packages dab packages and so on right all the different kinds of packaging um node.js packages and so on and and you can also query for specific packages um let me go back here and find this again um so let's look at the version of the database I'm using here right

um and now it just came back with one answer I don't if people in the back can see that but it basically says I'm running version 3.0.4 of neo4j so we have this stuff in the database whenever it's updated the database is updated on this kind of stuff and lots of other kinds of things as well these are just simple examples of the kind of stuff uh the the kind of things that uh the kind of things that uh the kind of things that that that you can have in the D in the database here um as I said it concludes Deb packages RPM packages python pip packages ruby gems PHP packages and node.js npm

packages um you can do others as well it's just I know how to do those and this is as I said before an open source project and let's see what I can have time to do maybe one more um oh yeah this let's do one of these so let's look at the most uh critical uh problems that we have from a overall perspective and I'll explain this I talked to you about triaging before so what we have here at the bottom this is um this is not the I guess I should have just gotten fresh batteries but if you look here I'm going to have to come stand next to it sorry if you look this is the head at

the bottom so it says here that process accounts to 16 points on this machine and here are the rules by order of how much the the individual rule costs you and then the next area that cost the most is Pam rules and and and so on and so on the idea is if if you wanted to know what to attack you'd want to attack the things where you have the biggest bang for your buck and that means let's go learn how process rules work and how we set them up so we go investigate that look at the process rules look at these particular rules starting from here come to understand them fix them and then

distribute them out and when you do you'll get your biggest bang for your buck then you can attack the the uh the Pam rules and so on Down the Line it gives you a way of attacking it to help you know what to do in order to fix all these problems so this is the basic idea of of of what we do here and is this making sense to people and and and as I said we do the rules we we we do all these different kind of things it's really it's a it's a very broad project perhaps overly ambitious could be um not an not not an Fair comment um let me's see where is my where's my

mouse control shift five no function shift five F5

okay oh I'm going the wrong way oops okay come on I don't need that now I went too far so a little to summarize what we're talking about here we know more about your systems than you do anybody here think that security is like Warfare yeah sunzo says if you know yourself and you know your enemy you need not fear the result of 100 Battles we help you know yourself like nobody else we know more about your systems than you do your admins do and every anyone knows we can discover anything you'd like to know Discovery keeps everything up to date more or less quasi real time best practice auditing of course is is is this best practice

auditing is continuous it's not quarterly annually or whatever our scoring system as I mentioned helps you figure out how to Res how to attack your problems uh Discovery includes it has near zero configuration this is just summary of what I think is cool about it uh we include Docker containers Discovery system monitors servers and services for good measure so in terms of your assist admin staff getting value out of it this helps you give more value to encourage them to you know to be able to use it and everybody to work from the same Playbook of what's on these systems for Real uh system includes um we have an event API that says it'll let you know

when things happen so you can you know trigger whatever events you want hook it into your sim uh into custom scripts whatever and it scales and unusually well so this is this is uh a little more about a little more about this if you go to uh category getting started it'll show you walk through four different places you can uh that you can go to find out how to get started with this and to see some similar demos there are some demos online as well these are in the slides slide deck and of course the idea is get assimilated try this out give it a shot and contribute to it and of course resistance is futile question

yes over on your agent so the agent the a the question is what's the overhead on the agent and the answer is it is written in C and and uh U its footprint is measured in megabytes not gigabytes and being so so that depends on how often you want to check these things uh it's still in measured in the you know a handful of percentages and the worst case uh you know like 1 2% something like that it can be less than that if you want it's a question of if it's a problem make it check less often you know right now we're currently having it check in in the on intervals of seconds but you can have it check on intervals

of minutes how often do you need to know these things what's the Frequency that matters to you then you set it accordingly does that make sense it's not a very good answer but it's the best I can give you

this is this is this is as small and lightweight an agent as you're likely to run across I've been perfectly understand that give it a shot and tell me what your feedback is more questions uh blue in the

back right so one of the things we if if if we're running on on real or virtual machines but not clouds we can we also listen for art packets so we know all the IP addresses that Mac addresses of everything on your subnets and that goes in the database that's a query that I didn't have a chance to show you because we don't have enough time yes question over here two questions uh there's a Windows port in progress but it's I don't know what when it's going to get done uh the answer so the question is is there a Windows client the answer is it was the architected to have a Windows implementation from the beginning and I

have somebody working on a piece of it I don't know how fast it's going to get done it's open source come help me and also my stuff oh absolutely so let me make something absolutely clear what this database it is the buried treasure map for your environment right totally understand that and but I've been aware of that from the beginning and that's all you can really do the point is if you put make valuable information the problem is it's valuable right and you need to protect it I mean the so so every the the all the communication is encrypted with public key encryption uh you you we can talk about that that's obviously a complicated question and it's a good one

to discuss here I've been given a lot of thought and I've done what I think I know to do and I'm certainly happy to hear more input let's go on though if that's all right for right now and I would like to talk about that some more yes you how does so there are there are heartbeats that UDP heartbeats and the communication with the central server is also UDP but the stuff with the ud Central server is authenticated encrypted compressed blah blah blah blah and and um with public key encryption but more in terms of how do they discover each other they don't discover each other the C they wake up and say hello here I am

the central server says okay here's your here's your neighbors because we need to know about them so while we're talking to them we tell them where to where to splice into the ring uh uh in the back in the very back yeah this guy the guy yeah with a hat and the Little Beard sorry it's like how we gonna describe

it is I didn't hear the question I heard about common checks does it use um raw scap content or did you pull out like a small subset of common checks between the different Stakes well first of all we have the checks we have which is not all we want to have the checks we're currently doing are mostly based off the N disa stigs um the thing is it has to be mechanically observable I mean I can't observe whether somebody signed a paper log coming into the data center um all right one more question one more question in the beard

there that's just heartbeats with there's no content to it other than I'm alive okay and because the key the key distribution problem of that is just hor effic so we don't do it it's just we trust at that level we provide that level of trust for free and yeah you could spof it um the problem with when you screw with a monitoring system you're likely to get observed it's not the it's not the thing you usually want to screw with if you're an attacker because if you do it and you don't do it right you're going to get observed because it's the point of this system to observe you all right thank you talk let's give

a big round of applause to [Applause] Alan and feel free to stay in here for our next talk or go check out some of the sponsors out in the chill out room wander around