Why it's all snake oil – and that may be ok

que habrá all right everyone good afternoon hey Pablo Brewer if you're on twitter i go by angry Hobbit talk a little bit about why it's all snake oil and that that may be okay so any any vendor sales people in the crowd good because you're not going to like me very much so mandatory disclaimer user my own they don't represent the views of my employer as a matter of fact my employer doesn't even know I'm here they'll get to find out when the YouTube video gets posted by a green that'll be interesting so why are we here we're here because frankly most info SEC professionals are not computer scientists they just kind of fell into it or they were enthusiasts

some have some formal education some not but computer scientists get a lot of computational theory and computational theory allows us to text snake oils so let me know if you've ever been part or heard this discussion boss comes andy says breaches are just costing us too much we can't afford it so slick Willie the contractor comes bak died and he says hey come by our next gen box full of zero day detection right we start to get a little bit hopeful right really you can detect nowadays totally do it boss totally detecto days really you're going to keep me safe and then they'd launch into this field with bunch of marketing speak with a bunch of pew pew

technology that you don't understand it doesn't really explain what the product does regardless of that we give them our money and then a couple of months later an apt comes along and that's what our network looks like so i went to a bunch of vendor websites in preparation for this talk and i found all of this great language can somebody tell me what any of this crap means because none of it tells me what the product does or how the product works so this good-looking guy right here is a guy named Alan Turing some of you may have seen a really bad movie with him recently Alan Turing basically came up with the math that is the basis for all

the computers that you run all of them and a Turing machine is just an abstract machine used in mathematics to prove fundamental limitations on mechanical computing this is how I prove that your problem can or cannot be solved by computer and something is said to be turing-complete if it's theoretically capable of expressing all the tasks accomplished by a computer so if your computer scientists you get to spend a wonderful semester learning something called utama da it is really painful math notation but what it really is is a field of scream at that studies computers in the problems that can be solved by them and more importantly the problems that cannot be solved by them ever some very generic charge of Tom two

splits problems into three categories there either solvable easy intractable meaning that they're hard and we have to make a lot of assumptions or they're undecidable and therefore impossible you will never ever ever be able to solve these problems of the computer they're just not capable so let's talk about an intractable problem the first intractable problem is computer scientists get is something called the Traveling Salesman problem fairly straightforward given a list of cities and distances what is the shortest possible route that visits each city exactly once and then returns to the originating city seems fairly straightforward but it turns out that her computers is an intractable problem and you're running time is a factorial of the number of cities that you've got

so imagine that you're a Salesman and you've got to go from San Francisco to New York that is a ton of cities the Sun will burn out before the computer finds you the optimal solution now somebody out there is getting ready to throw the BS flag and they're going to tell me that hey that sounds a lot like my GPS works and you're right it is how your GPS works but your GPS works because it makes a bunch of assumptions it assumes things like interstates are faster than rural routes rural routes are faster than farm roads and then you may tell it some things like hate don't show me any toll roads sometimes these assumptions are

good and if the assumptions are good you have a workable but non optimal solution what is not optimal mean well in the security world non-optimal means you're going to have type 1 and type 2 errors you're either going to have false positives or worse yet false negatives so let's move on to an intractable problems talk about produced by contradiction so certain problems are mathematically provable to be undecidable and cannot be solved by computers again ever so often times vendor will come to us and they will make assumptions or make claims that they have solved a bit difficult problems here is how you handle that let's assume that they have let's take them at their word and imagine that we

can take their solution that they're trying to save you sell you and package it into a chip if we can then use that chip to solve a provably undecidable problem we've got a contradiction and if I can't solve that undecidable problem then they haven't solved the problem that they've suggested that they can solve so there's the halting problem fairly straightforward given our chart machine m and language w/o my punctures missing with a very confused baby alright let's try the let's try the English edition of a halting problem here's what it says very simply given an arbitrary computer program and input you will never be able to determine with a computer if it will finish running and

if you think about it logically this is very straightforward it's either going to finish yay or it doesn't and the problem comes in when it doesn't finish if it doesn't finish maybe it's still running or maybe it's stuck in an infinite loop and you will never know at some point you may decide to break the program after say 30 days but what if it needed 30 day in one minute to finish processing you're never going to know and that's the problem so let's get back to our vendors vendors going to come to us and your say we detect all zero days and what we hear is what we hear is we detect all zero days

all right let's assume that's true for a minute let's take that zero day detector let's put it in a a chip I let's put in our halting problem and then let's redefine the halting problem this way if it's a zero-day will halt and if it's not a zero day we won't halt so we've solved the halting problem but we can't solve a halting problem at all ever mathematically provable one hundred percent so if you can't solve the halting problem and I can use your chip to solve my halting problem you haven't solved your problem either so there is no zero day detection and there won't be ever all right so no odea detection all right so how do vendors claim this well

they claim that because they can detect some zero days and some of them can they've got some really smart people that work at these vendors and they do it using signatures and heuristics and statistical models and yeah you can detect some things like Rob chains on occasion but you're not going to detect all of them they make assumptions to reduce the problem space so that they can solve part of the problem so the trick as a customer is to ask them what assumptions they made and then validate those assumptions how are you doing what you are claiming that you're doing what assumptions did you make and will make sure that they make sense in my real

world so at about tcp/ip you're going to get some firewall vendor it's going to show up and talk about their next-generation firewall and they can detect all malicious traffic tcp/ip turns out tcp/ip is also turn complete since its turn complete the halting problem applies since a halting problem applies they can't detect all malicious traffic in tcp/ip so that next-gen product it may make things better but it's not a panacea and it's just going to have different kinds of signatures and heuristics so for the vendors first be honest in your claims your customers aren't stupid we're going to challenge you second of all when we ask you about your something assumptions and methodologies they're not secret sauce

you need to be able to explain it and if you're uncomfortable with that then have a an NDA most of us are going to be fine with that and for goodness sakes quit selling us marketing speak that next-gen cloud-enabled software-defined heuristic based virtualized threat centric multi-layer end and analytics adaptive side of the fence pew pew isn't worth the shiny paper you printed it on so for the customers the vendors do have some smart solutions and that's fine but they also need to make a living if the claim seems too good to be true it probably is so ask them how they're solving these impossible problems ask them how they're reducing the problem space ask them what

assumptions they made and then make sure that those things make sense in your operational environment most importantly arm yourselves with little knowledge understand what problems are not solvable by computers in which problems are solvable that are hard so the next time a vendor comes to you with rondo the Otay mutilator ask them how they solve the halting problem for those of you attending our sack there's your our sack bingo cards with your sales terms I am NOT responsible if you die of alcohol poisoning while playing are there any questions please but i do not have pony some of my graphics didn't make it any other questions all right ladies gentlemen thank you very much for your

time you