BSidesTO2017 Chuck McAuley

hey guys how's it going today I'm going to talk to you about being lied to I'm sure most of you guys have had to buy a firewall at some point and I'm gonna tell you all the dirty secrets about how the data sheets are made and how you can make better decisions about buying better ones because that's all I've done for my entire career so speaking of which my name is Chuck I work in a company called exceed communications I work at home I'm very lucky to have the privilege of doing that I live up in the Boston area so you know don't give me grief about the black and gold's guys I've been paying

attention yet so and I've been testing network security in line network security devices for 15 years basically my entire adult life since I got out of college and today I'm gonna talk to you about some of the findings and some of the stuff that I just sort of know about how that industry works and hopefully help you guys figure out better ways of making better informed decisions so firewall data sheets have some level of truth to them the question becomes what is that truth and is that actually of actionable value to me is it something that I can actually use take lay them out in front of each other and compare them against each other and

figure out whether or not they actually are like for like right I'm also going to talk to you today about how do these sheets are made themselves so we're gonna look at the testing methodologies that are put in place for this we're gonna look at a publicly available information that that you can go and look at yourself to see how these methodologies are actually used I'm going to give you the crib sheet on the problems with them gonna talk a little bit about false and incorrect assumptions about you know how everything must be the same because it looks like it's the same just because something looks like it's the same doesn't mean that it is and I'm gonna

suggest some improvement in terms of like what what could be done to make life better for everyone in here so we all know this lovely quote there are three kinds of lies lies damned lies and statistics this is commonly attributed to Mark Twain but I went and found out that he didn't say that so I don't actually know who said it it's a really really old quote which is kind of fun to dive down but in reality what we're really dealing with here is the fact that you've got these different statistics on different data sheets and we'll just load up the first one to look at it this is Cisco's a sa matrix for making a decision about which firewall

you want this is a little bit old but I went online last night and they have the exact same footnotes and everything everywhere and you'll notice that just about every single statistic that you're looking at has a little note next to it and then it looks like it's backed up with something that sounds very technical and serious about how this is done so it's been you know vetted and and and the reality is is that this is how its tested right so maximum bandwidth and throughput with UDP measured under ideal test conditions so they give it they told you that they've given you the best possible number that they can get out of this right number

two they say it's multi-protocol traffic with primarily TCP based applications like HTTP SMTP FTP blah blah blah blah right so you have a certain level of assurity here and if you're sort of just sort of wrapping your head around this a little bit it looks like this is this is this is this is the works been done and the work has been done but you don't know enough so then you go on the check point website and you look at their stuff and you can see that you've now got RFC's mentioned here so this must be this must be technically accurate if it's an RFC it's been validated and it's good right you have recommended iMacs

traffic profiles acronyms are always good acronyms make you feel like you're dealing with other technical people and they're right and then we have coordinates and we got even more footnotes all over the but I really like this no all performance values are up to and may vary depending on system configuration great so what is it what what can I expect out of this thing you also have other things that can mislead you or make you confused so for instance they have a firewall throughput which is actually this is the technical definition of it it's measured in packets per second you can push across a network interconnected device without losing any packets as per RFC 25:44 I

told you I've been doing this for 15 years it's in my head and then we have Palo Alto who just basically sort of seems to wing it and just say all performance capacities are measured under testing conditions that's the only footnote it's there's nothing there so okay I mean I mean all these devices are great to keep you secure people rave about and I've got their favorites I I've got friends who work at all these places but at the end of the day once you go to the marketing machine and you go through the sales targets and the fact that you know you gotta keep up with the Joneses this is what you end up

with okay so this is really what you end up with if we sort of go back and we look at that we've got we've got concurrent connections we've got throughput and we've got connections per second those every single firewood manufacturer will put that out there and they'll say they'll get the most of those but the trouble is is that even the agreements over what those are and how they are defined even if they are technically defined are still technically defined loosely for instance throughput voronet actually put the actual technical definition of throughput up there but everyone here thinks of in terms of gigabits throughput megabits whatever right so let's talk about let's talk about throughput first

this is uncle Massachusetts as you know and this is not my senator this is someone else's senator his name is Ted Stevens and about five years ago he stated that the internet is not like a truck that you dump something on it's more like an interconnected series of tubes which I don't know what that means whatsoever but basically what you're dealing with is a throughput as a definition is the number of packets you can forward across the network with a packet loss but the reality is is if we think about just as throughput as in bandwidth right which is how much data can I push across this network if I look at an intrusion prevention device and it's got

signatures enabled it's got an app ID enabled it's got NAT turned on or off like all these things impact the performance of the firewall in different weird ways but even more fundamentally the packet size that you push across that firewall severely impacts performance but how is it measured when you're looking at it from a perspective of these data sheets it's always maximum MTU UDP packets that don't hit any regular expressions or anything else I just hit the session tracking state table as easily as possible so you end up with like a firewall that claims it can do 100 gigabits per second whereas when you deploy it you're going to get 10 gigabits out of it and that

that you know normally drives a question how often do you see a 1500 byte UDP frame across your network unless you're running IPTV for Bell Canada you don't never like a DNS packets was 70 bytes normally 80 bytes so this is not a good way to know what you're gonna get out of this thing so they came up with a better way I call it multi-protocol there's a problem with this though there's no RFC to define how to do multipolar protocol throughput testing so if we look back at Cisco's definition again they get a whole bunch of protocols we don't use anymore I mean they put that up there and then if we go and look at checkpoint they just simply

say that it's either this thing called a mix or a real-world traffic blend with no definition of it whatsoever I mix is a grouping of different sized packets that are typically seen across the network it was developed by service providers it's stateless and it's designed to test routers and switches switches which don't care about state what does a firewall care about state it's the number one thing it does right well it's like so you can't figure out what you've got going on there they don't say what percentage of which traffic is running how long the session runs for what it's doing how big the emails are getting delivered over IMAP that's just not in the documentation it's not in the data

sheet then we have connections per second does someone want to without reading my slide first yes tell me what you would define as a TCP connection syn syn ack ack that's it what about data well I got send data right and then I gotta get X for the data how do I close my connection are ESPYs ban if you're making a firewall data sheet use ours T's it's one pack instead of three or four come on remember packets per second reduce performance so what you see four connections per second testing is all kinds of janky stuff I've seen since in ack-ack and that's it that's the connection and that's they just drive that up and they fill up the state and

that's how fast it goes I've seen syn ack ack reset people try to keep it a little bit more real will request a one byte webpage yep because we see lots of 1 byte webpages all over the world right other performance factors that are consideration that never really rarely come up at all you can end up with IP address hashing algorithms so different to nobles turned on the firewall to make it more optimal for a certain IP pool many of these data sheets will just use say 500 clients and 10 servers because when you're you buying a hundred gigabit firewall you are only going to have 500 clients and 10 servers for your entire network all the time right

if you blow that up if you put like a simulated gateway router and have the internet like millions of IPs as your servers you I've seen crushing performance like brings it down five to ten percent of what's put on the data sheet there also just fundamentally what is the benefit of this metric if it's not testing anything that you're gonna see in the real world in unemployment right this this is a repeating factor here but if you're not aware of what you're doing you got problems and then we have concurrent connections long story short they like to do one byte web pages so send a get request to inter bytes sent a one byte web page back and then they sit

on that socket connection for I don't know ten fifteen maybe two minutes and then they send another request so you've got this very low throughput trickle designed to layer up the entire session table but more fundamentally I like to say stupid math is stupid if you take the maximum advertised throughput and divide it by the maximum number of concurrent connections you end up with like half of kilobit in some instances in terms of what the firewall can do so you really are they're over killing it in terms of trying to brag over each other for this and then we have measuring protection guess what's not on the data sheet how well the thing blocks attacks seriously go back and look there

is no security effectiveness on the device it's all about how fast it is and cool but the way that this is tested is basically these steps you start with my sales engineer ran a pcap and showed it worked and then level two is let's run Metasploit layer 3 is pcap replay everything that i have here yes and then level 4 is of course plugin a live network will prove it works and once it's there then you have to buy it because it's now part of your network right because everyone just puts new stuff in their network every day nope all right blame the blame is RFC 35 11:35 11 was written in 2001 back when calculator

watches were cool and it would it said to test with HTTP only it said to use giant UDP frames for throughput and then for connections per second concurrent connections use HTTP it made no specification as to the object size object size means size of web page so that's why you get all of this all over the place stuff like my connections per second with one bite Paige with the one kilobyte webpage with a no webpage it you have no idea I don't hate RFC 35 11 it's sort of like hating the calculator watch the calculator watch is really cool if it's 1999 again right the problem is that these devices have matured and changed there are no longer just session

tracking machines they do data loss prevention intrusion prevention they they make VPNs and cook your mom dinner when you're not going to get home on time right so you can't get upset at a testing methodology that's just not moved on with time other things to take into the note like I said large packets are easy it's like Big Papi visiting Toronto you you can do a lot less frames per second if you do large frame sizes so ask to see performance with the smallest stream size that's a nice little trick to get know how badly this thing will suffer right if they're going to show you how good it is as to see how bad it will be

another thing of note is an obsession with average packet size I see this over and over again people will see metrics from their net flow data or other things say my average packet size is 7 or bytes so I need to have all my packets look like they're 700 bytes that's the wrong way around that's bad thinking because when you look at your TCP traffic you have all your sins your acts your resets all those things sit over here and unless you've tuned your stack you get an act for every data packet and then you have all your data packets which fill up the MTU as much as possible right the stuff in between is negligible

in terms of seeing it so it's basically this is this is very important to keep in mind when you're thinking about what your traffic looks like when you want to know what the performance looks like the other thing on these data sheets is that they're all red line numbers so if you look at the maximum concurrent connections and you look at the throughput it can be logical to draw an assumption that you can do both at the same time this is wrong maximum throughput is tested differently than maximum connections per second total number VPN tunnels brought up is different than throughput all of these things happen in isolation with a one configuration option enabled and everything else

disabled when you start turning them all on and off and fine tuning turning on signatures on and off it all goes out the window I mean you can't get really upset at them for that because they can't make every single possible configuration on these things but you need to you need to know that that's how that's done and then if you look at just TCP stack parameterization on its own like I said number of IP addresses matters but act prioritization matters if you use delayed acts if you're using Windows 10 stacks versus OS 10 versus Linux from 15 years ago the number of package generated sizes and delays used are all going to be different

same with icw same with retransmission timeout values and packet fragmentation all these things you need to know what you've got going on there so solutions first and foremost challenge your vendor your vendor does good work I visited every single one of them I've seen their tests LEDs they know what they can actually do with these things but they you know also are under the gun to provide the right sort of metrics and values to compete against their own competitors right it's keeping up with the Joneses look at independent test lab methodologies NSS Labs is a good one they actually do testing I mean all these sort of Gartner like groups are have some problems but they're better

than nothing right better yet I'd like to see a new RFC the benchmarking working group at the IETF which is probably the most boring working group at the IETF they're cool people I watch what they do but they just reissued their Charter and restated that they are only interested in achieving lab performance results they don't want to get in the messy weeds of realistic performance results so there needs to be something that at least is a methodology of coming up with a way of testing that everyone can agree on I like chaos chaos in the network makes everyone feel safer right there are better test tools available there are ways to do it but basically by and large

what I what I witness in the field is inertia it's my occupant Optive guy told me I should buy one of these two devices so I'm gonna buy one of these two two devices right you already sort of getting narrowed down in the field or having suggestions made to you and and you know they got good tribal knowledge they know what this stuff can do but you need to you need to challenge those assertions if you're right the first time it reduces headaches all the rest of the time money I think and and that sort of leaves me with the last bullet No I hope you guys I know I'm burning through this real fast but I hope you

guys have gained a little bit of a deeper understanding how these things work and know the right questions to ask now like you said everyone spends a lot of money on this stuff it's a multi-billion dollar industry if you ask the right questions you can really sort of needle your sales guy which will give you a little self satisfaction but it will also challenge everyone to do gain better confidence in terms of what they're doing aside from that I'm the noble trout you can't see it because it's all just that slight and I'm done if you got any questions [Applause] there you go there's my twitter handle yes testing on DDoS well if they are

good at preventing DDoS um they normally are tested for DDoS and they can hold up the DDoS what's more interesting is think about this way if you meant to add a about another slide in here that sort of showed the multi architectural design of a firewall you sort of got two kinds you got software based like likes a check point McAfee and then you've got hardware based like Porter net kind of hollow Juniper Cisco right those hardware based ones if you can hit an IPS rule that gets popped up to the general purpose CPU and you just throw that over and over and over again the firewall you will crush its performance capability of handling traffic overall

which is kind of a neat trick

Thanksgiving rule yes

a crowdsource method of testing I the closest that I've seen to that is like independent testing labs I haven't seen any or sometimes sometimes the service providers get really really upset and then demand a standard gets ratified but I haven't seen anything specific to firewalls and and traffic mixes a lot of times like some of these vendors will actually open source their own tools and then try to use it as a means of like storing chaos in the mix so that you test with my tool on the other fire one obviously performs poorly but it's about the best yeah yes right all right he's asking the question how do I know what I need that's what my company does we make

big giant devices that actually can do this sort of testing and they're very cool and they're all the people I mentioned today or my customers which is why I'm kind of not being such a jerk towards them well not my customers environment developer but yeah that that's how you know you've got a you you can do your own testing or some of the bigger players have proven concept labs with test equipment that they can use to demonstrate to you how it will work in your environment which is a really good way to gain confidence yes

okay the question is the amount of time to spend testing a firewall to get performance out of it the size doesn't matter of course there's a dollar cost associated with that but normally you can get pretty confident results within three days with someone who knows what to do it alright thanks guys [Applause]