Firewalls – Threat or Menace

comic book from the 19th from the 1960s is this working yeah okay and it was something that the Daily Newspaper said about Spider-Man so the question was is Spider-Man a threat or is he a menace which was meant to be a rhetorical question firewalls are terrible and I'd rather not have them the only thing worse is not having them but before I get to that I want to rip something else from the headlines more recently um you may know that I wrote a version of the crown program some decades ago and it was originally written for the BSD team because they were trying to excise all of the a t source code so that they could release a completely

unencumbered version of the SDS this would have been uh late 80s early 90s and I thought this would be fun and it was and I learned a lot and it was good but eventually Linux came along and became far more prevalent than BSD and I understand Linux now has something called systemd which incorporates the functions of cron and so chronus starting its long delayed death which is probably good um but anyway I still get bugger forts I got a bug report recently from somebody you know very polite they don't know you know how old I am or you know how to address me exactly so you know Mr vixi I'm afraid I may have found a bug uh and

basically what they did is they use the cront tab command which is used to either list or replace or edit your your per user Crown table but they put a space they wanted to say cront tab Dash L to list their con Tab and they put a space they said cront tab Dash space l and they they were just mystified because this caused their cront tab to be deleted and they wanted to know was this something that I had intended you know and it doesn't see it in the man page um so I was thinking about this um and I want to show you the answer I did not give him

so I want to say that I am privileged to be the CEO of my startup which means that my boss is a board of directors and their boss collectively is the shareholders um but I also feel quite beholden to my customers and to my employees and their families because without all three of those sets of people shareholders customers employees I got nothing and so um I fully understand why Intel would answer this question in this way and it's because they're publicly traded companies and they work with lawyers and the lawyers tell them what they must do or what they must not do and I am tiny and I also work with lawyers and I tell

them what they must do and what they must not do so I don't necessarily blame Intel for giving a lawyer friendly answer but I do want to say that what I told this guy about KRON is you know what you actually said was Grand tab Dash and it interprets that as wanting to copy your new cron table from standard input you didn't give it any standard input so it copied nothing into your system cront Tab and that's that's what happened here and then I realized that I was in exactly the position Intel is in with this heart or the the Specter meltdown thing which is the program is doing everything that it's documented to do

but it's also doing something deleterious that it is not documented to do which is to say that that L that comes after the space is extraneous input the syntax of the command line does not actually document any options there so I could have caught this for the guy I mean if he typed crowntab Dash and his return he's on his own but if he types cront tab Dash space and then something else I should be able to say you gave me something I wasn't expecting I'm going to choose to do nothing because I don't I'm not sure that you know what you just asked me to do and that's what Intel should that's what their chip should do

this whole side Channel thing yeah there's they're talking about there's no bug in the macro architecture the instructions we told you would take certain would have certain effects have those effects but the micro architecture has some side effects that we didn't document but everybody has them so it's not our fault um I hate this kind of stuff and um I'm gonna I will publish a patch to KRON to fix this but of course most copies of cron are inside Linux things that have seriously diverged from the main line so watch yourselves using cron is the moral of that story but also watch yourself when you buy chips a lot of us did not know that

there was a copy of Minix inside of every Pentium CPU to run that little system administration engine that we talked to for updating bioses and whatnot and that thing hasn't been patched since God knows when and the Southbridge chip is able to lurk on the port path to your ethernet controller and use it without your permission and see what you're doing without you knowing it so I didn't ask for any of that and I am going to be checking much more carefully about these complex designs that we use in our work because I don't think these people know what they're doing

yes

you know you ask a good question um I have learned everything I've ever learned the hardest way it could be learned so that I would act and I learned it over and over again until I learned it often enough and hard enough that it's stuck so chances are I could write a c program now that would not have an off by one error in it but if you're looking at my code from the 1980s that's something you should really worry about so yes I I last gave this talk in Estonia in May of last year so um this this is kind of a repeat showing but uh I offered your program committee a selection and they

picked this um and it's still quite topical so let's talk about firewalls um the the internet as well as any other reasonable networking standard lets you exchange a whole lot of arbitrary data that is not necessarily uh known by the designers of the internet they didn't consider voice over IP they just said we'll give you General enough functionality that you could build stuff and then eventually somebody said hey telephony um but not all the traffic that it is capable of carrying is in our best interests and I'm referring both to outbound and inbound traffic right there are I remember when firewalls first came to the fore I worked at digital Equipment Corporation which was the

second largest computer company in the world at that time the first was IBM and so a lot of times we would compare ourselves to them you know how what are the trend lines what are the approaches uh we had a firewall that um pretty well kept information from getting out that's right so we we had a lot of protection against our intellectual property leaving the company and IBM was the other way around they had a lot of protections against intellectual property entering the company and that simply was because we were concerned that people would copy our work and that we might not be able to compete if everybody knew everything we had IBM had as many lawyers as

Engineers even then they knew they could protect their work if it was copied inappropriately what they didn't want to do is have to defend against an attack where somebody else's intellectual property had come inside the company so it's totally different approach but it all refers to the fact that for some value of local policy not all traffic is good for you the network operator or the network user and so that's where firewalls came from which has blocked the part that that we can't that we're not sure about or that we are sure is bad Forum since really Cheswick and bellaven wrote the firewalls book in the late 80s there were a whole bunch of people doing

packet filtering is what we called it but it really got traction and got some names and you know became a market Trend with the first publication the first edition of that firewalls book um and what I want to say about these packets is that we hardly ever have a rule that is designed to allow or disallow a single packet because a single packet almost never conveys useful information or does anything either harmful or or harmless or useful you need a flow you need a whole bunch of packets that can be grouped together and you would look at their five Tuple which is to say the IP address of the sender of the receiver the port number

of the sender and of the receiver and you would say that all the packets that are going between those two endpoints were part of a flow and you would describe in your firewall rule a thing that said flows that look like this are allowed and that's good in a sense except that it turns out you can tear flows apart in interesting ways again probably side Channel attacks in a way if you ask the IP Architects about this um and of course the sad thing about working with flows is that if you wanted to create another protocol that was not TCP and was not UDP you know and there are some examples of other protocols that have been developed that are far

superior to TCP and UDP for certain use cases um you can't do it because everybody everywhere has got a firewall that allows things that understands like TCP and UDP and thus anything you tried to enter the market with today that isn't one of those things you won't find customers or you'll find a few but they won't be able to talk to anybody and you will fail in the market so this is a clamp not just a glass ceiling but a uh a blood pressure clamp on Innovation itself we are stuck with what we have and whatever was on the test Network at some Network Lab at some Gateway producer 20 years ago is the only thing that your home DSL is going

to allow in or out of your house so I kind of hate that but I don't know what we could have done to prevent it anyway um there I mentioned that flows are not as cleanly uh delimited delineated as we'd like them to be and I'll give you an example um if you have an IP datagram which could contain either a UDP datagram or a TCP segment and if that IP datagram is too big for some link if it has a maximum transmission unit size that is smaller than the size of the datagram you're sending it will split your datagram into pieces and send those pieces and this happens in the middle of the network which a lot of us thought

was kind of crazy and IP version 6 doesn't do it which creates other problems I'm not going to talk about that I just want to say that your original datagram becomes several datagrams and then there are various little header fields that say which part of the original datagram this one is and so we distinguish between the non-initial fragment in other words everything after the first one versus the initial fragment so if you have something that is flow based firewall that is flow-based in terms of what it's going to allow like a web is okay the web port numbers are okay or DNS or something like that it is only going to see the port number it

needs in the initial fragment because the UDP header or the TCP header is there it doesn't even have to all be in the initial fragment you can have a fragment that starts in the middle of the the protocol header although that will generally not work anyway so what this means is a lot of us have firewalls that are set up to say all right UDP is okay DNS Port 53 is okay and any fragment is okay in other words I don't want to do the work of trying to keep track of which initial fragments I have let through so that I can only allow the non-initial fragments that went with the original datagram that's just felt like

Madness now everybody does it but originally it was seen as a bridge too far um and this goes on right you if you have a rule that says uh that a sin packet in other words the first packet of a TCP session the thing that tries to get the you get the session open is controlled but all non-sin packets are just to be allowed through and again that's to keep the firewall from trying to remember which TCP flows it's letting through and just say I'm going to control the first one and if the first one doesn't get through then the non-first ones won't be useful and so life is good the trouble is if you wanted to DDOS

somebody you could send an awful lot of non-initial fragments or non-sin TCP and you would be able to hit inside their so-called hardened perimeter simply by sending packets that were outside of flows but that they weren't doing the work to note weren't part of flows so this is the kind of weakness that you get that proves to you that none of this was well considered in the days in the let's say the 1970s when this was being designed trying to it's a miracle that works at all and then it becomes an even more astounding Miracle as you start to study the ways it's been abused finally you have this idea of UDP state where for example

you might want to say I should be able to make DNS queries of servers outside my network and they should the answer should be allowed through and so the simple way to do that is to say if the destination Port is 53 or the source Port is 53 then allow it through and that's what a lot of us do which is why a lot of us are subject to ddoses inside of our firewall because somebody just pretends to send us a lot of responses to questions we never asked on our firewall isn't keeping track of which ones we asked and so it all just gets through so I hate this kind of stuff although I recognize

that it keeps me in business and I'd be driving a tow truck if this problem didn't exist so um because of all of that state firewalls have become stateful in other words they keep track of which flows are in process so you can actually have a rule in most modern firewalls that says let a DNS query go out and let any let the response to that query come back and it really is just that one response once the response comes through the state is collapsed and it's it's you're back to to the ground state without that rule so these are temporary little rules that go in but state has mass these are not free just because they're

made of ones and zeros and we don't normally think of ourselves as paying for ones at zeros doesn't mean we have the ability to have an infinite number of them ever no matter how many terabytes you have there are still terabytes you don't have so if you fill up the state table with junk then the firewall will slow to a crawl either because it doesn't have room for all the state and it has to discard State and then go back to some secondary mode where it's letting things through it shouldn't or it is not good at having a million flows it was only tested with a thousand and with a million of them it's doing some linked list search that's

quadratic in terrible ways one way or another too much state is bad for you now there's just no way to avoid that that's in the physics um so if you think everything I just said and applied specifically to the DNS protocol and think about truncation then what you need to need to know about DNS is that UDP is the primary transport you start any given transaction on UDP and if it fails in a certain way then you would try again using TCP and the reason for this is TCP takes a lot longer to set up and tear down the connection what you want to do is get your question out and get your answer back as quick as you can so it's done

with a single exchange of packets but the answerer is able to say that the answer I want to send you is bigger than will fit in UDP and therefore I'm giving you an empty answer and I'm setting the truncation bit I'm telling you that you're going to have to try again with TCP which allows for me to send you a longer answer that will be long enough to send the one that you need so this gives the attacker or really the initiator anybody the people outside your network outside your firewall the ability to control your behavior and anytime you see that you probably want to treat it as a red flag and really study it carefully because what's

happening is I am able to tell somebody that they should hold more state and if I tell them that often enough they will not have enough state to do the other things they also want to do so it is a denial of service attack but it's not done with you know tons and tons of packets it looks very small on the wire but it still creates a denial of service condition you really should not have a protocol that allows other people to put you in that condition but we have them we have not just this one but a lot of them but this TCP retry after truncation is a big deal for DNS and this is what it looks like

question comes in so QR equals zero means it's a question QR equal 1 means it's a response so a question came in the answer was too big so we sent you back something that said truncated and then you come back and say okay here I am on TCP here's my question and there goes the response now that doesn't look so bad but it's also not true because what I'm telling you here is DNS messages these are not packets on The Wire these are the packets on The Wire which is those top two are the ones you saw from the previous one and then the next eight are TCP setting up and tearing down connection state

um so you're not only causing other people to hold State uh you're causing them to send and receive more traffic and if you wanted this to turn from a non-congestion non-volumetric attack into a volumetric attack all you have to do is turn the knob and do more of it and uh that's a little bit scary now there's something you can do a lot of attackers do do this it's a slow lorus attack where after you've got the other guy holding some State you simply deny him the response he's waiting for and let him eventually time it out and he'll have he'll reach an air condition close the connection without permission and that'll get CIS logged on both sides

which becomes another element of the attack so this is like a kick me sign this is a protocol that has put a kidney sign on on your back but in fact this is still not what's happening because there's also going to be a web server inside that firewall and um so if you didn't turn this into a volumetric volumetric attack but you did cause that stateful firewall to remember all of the open TCP uh flows that you were causing people to to do by how you answered their DNS questions you can fill up the firewall State table so that it has no room to store the state about web now there's an argument to be had about

where that web server belongs in the topology but pretty much every time this has been reported anywhere a flurry of responses comes back that says you don't want to do that you want to do this which is to put the DNS server outside the stateful firewall because the DNS server has a slightly better chance of being able to manage high levels of state by discarding it opportunistically then that firewall has the firewall wants to do the best job it can for you and it does not know what is going on inside the DNS protocol the way that the DNS server does so you can argue about whether the web server ought to be back there behind the

firewall but you can't argue that having a DNS server behind that firewall is not just a kick me sign but actually walking up to strangers and asking for trouble I originally put this presentation together because a large National Bank in one of the top 10 economies in the world got taken off net to the point where they were not able to answer user Commerce web traffic and this was what this this slide presentation was made for them which is Doctor Doctor it hurts when I do this well I'm going to stop doing that um so anyway once you get your DNS server out as I mentioned you have the opportunity to put some protocol specific logic into how you're going to

manage your state and when I say manage I mean discard the stuff that is least useful or least likely to cause harmful side effects and so we had to build this and I'll I'll show you in a moment the impact of having built it which will tell you why we had to build it but first let me give you the theoretical background here so if you're running one of these name servers and again in this case it's a Content server it is answering questions for example.com or wherever it is that you work that server has to be extremely well provisioned over provisioned I might say you might want that to be able to ask and answer a thousand times as many

questions as it will ever do in any healthy scenario because when it's your turn and you know the roulette wheel is always spinning so when it is your turn to be on the receiving end of a DDOS attack you want to be able to answer legitimate traffic at the same time that you're answering all of the non-legitimate traffic and so it is crazy to build a name server that can only accept as many packets as our normal and healthy you have to over build these it's uh that's one of the reasons that so many people Outsource their DNS is they can't afford to over provision this their whole business might only need 50 to 100 megabits of connectivity but

their DNS server needs 5 to 10gb gigabits of connectivity just because it will get ddosed okay so once you've solved the problem by over provisioning the server then you've created a different problem which again indicates that the bad guys have the option of forcing you through a flowchart um do you want to die here or do you want to go over there well let's go over there oh do you want to die here do you want to go further so um because of packet Source spoofing which I shall explain in a moment um I can send a packet to all of you claiming to be from that guy and you will answer him and he will be suddenly in the position

of having heard a lot of answers to question he didn't ask if I do this a thousand times a second and there are millions of you then he's probably off the network and he could call you let's say one of you is running an important server like the top level domain server for.com and this is participating in this reflected spoofed attack and he calls you and says would you please stop ddosing me all you can really ask him is say well all I'm doing is answering the questions that appear to come from you would you be would you rather I don't answer the questions that appear to come from you and then he's got to answer the question

do I want to be unable to reach any.com name how do I feel about that um so you did this by over provisioning your name server enough that you would be able to withstand a DDOS you made yourself into a perfect DDOS amplifier reflecting DDOS amplifier congratulations welcome to the next level so we built some technology called response rate limiting we means I was back at my non-profit before I came to farsight so this was 2011. that simply kept track of how many repeated questions it had gotten where the definition of repeated is kind of fuzzy because it has to be a nearby question and it has to come from a nearby location but the idea is to be

able to excuse me preferentially answer the queries that don't look like they're part of an attack and we were able to do this because we have a fairly deep knowledge of the DNS protocol we know what state everybody's got to be holding and we know in many cases we can see that a a non-attack person would not have repeated this query we have answered that question for him two two recently and too often so we had obviously some State Management problems you do not want to uh change from having unmanageable State over here that makes you offline to having it over there instead which what we did is we said what is the worst case how much

State could we be made to hold and how can we design around that assumption and really uh probably when BSD Unix was first being built having two megabytes on a machine was a million dollar problem and so the idea that we would burn 10 megabytes of hash table just for this problem strikes me as a little nuts on the other hand 10 megabytes costs a nickel so yeah we have a fairly large State table but it's not infinitely growing so we make sure that we will not become a source of a new state Mass Attack and all of your Authority servers needs this need this and if you're running bind if you're running NSD from nlnet

Labs if you're running cannot from cznic labs in Prague then this is part of your server and powerdns has this now it's in their DNS dist front-end utility it works a little different it's not the way we documented it but it's better than nothing every server needs this and we have documented it we have made sure it's unencumbered there's no patent nobody owes any royalties when they when they deploy it um and uh it ought to be the default and now that it's been out for five years I'm now making the rounds trying to get all of these name server implementers to turn it on by default so that even if the config file fails to mention it this

feature will be in place so some background OPN is other people's networks these are the ones that cause most of your problems and they are in fact the least safe elements of your lived experience on the internet because you cannot pay money to get them upgraded or get them get their code updated you can't expand their capacity you can't re-educate their administrators you are stuck with other people's networks as just part of the equation when we talk talk about costs and benefits other people's networks are far more cost than they are benefit I mean yes we're I'm glad other people have networks or I couldn't talk to their customers but that's about the only reason I'm glad that they that they

do it Source address validation Sav is the thing nobody does which is the real fundamental enabler of that spoof Source attack right because I mentioned if I send a packet to all of you claiming to be from him you're going to answer him but my ISP ought to know that I'm not him so when my package is trying to leave my isp's network they ought to say that is not the ad The Source address we gave you why are you trying to send a packet that did not come from you and it should just drop that and that's what you're seeing here um the attacker on the top is using the Target's address as its source address

on packets that go through that internet Cloud that's on most slides now and it hits the reflector and the reflector has no reason to disbelieve the source address and it sends that the red dashed line out to the Target and if this was only one packet or only one reflector or only one attacker then we would not be talking about this but it's always a lot of packets and as I speak to you there are at least a thousand of these going on on the internet I see everything now I have a Global Network of DNS passive DNS sensors and so I can tell you this is done all the time now the reason they

do it is because that Target tends to be an online gambling casino know and that casino is able to measure in the tens of thousands of dollars per minute of profit they don't make while they are offline and so if you can send them a big DDOS for say 20 minutes and then send them some email with your Bitcoin address and say I need this many Bitcoins or I'm going to turn that on at the top of the top of the hour and leave it on all day pretty much you can count on those Bitcoins flowing um and this is something that I can't seem to stop the reason I can't is again this attacker should have an ISP who

stops them from using an IP address as for their source that was not allocated to them they don't and it's not that they can't all modern routing equipment has this ability they don't want to turn it on because there will ultimately be some multi-homed customer who will complain and they don't want to do any work to find out who that customer is um they also have the problem of the brake fix flowchart right if somebody calls in and says hey my internet's not working you have this list of places you have to check to see what it is that could be causing this problem they are your customer you got to get them back up if

you turn on this feature that is one more thing that could be the cause of that problem and you don't want to answer the phone anyway but when you answer it you sure don't want to have a long list of things to look at so it's simple economics that keeps the source isps from turning on the feature that would save the rest of us now I do want to recount kind of a fictionalized account of a real conversation that I had which is um I want to turn this on and that means we have to delay something some the the deadline for something else the Milestone is going to move out a couple of weeks and we need to invest in some

training documentation okay so how does this make us money you want to make an investment I'm your CFO explain it to me now if I'm the CEO I can tell them to shut up and go away but I was not always the CEO so I can remember having a conversation where I said well actually it doesn't make us any money uh oh okay well then how does it save us money right that's the other reason you might make an investment well actually it won't save us any money well then why do you want to do this who will it help oh it'll help our competitors oh Mr vixi get the hell out of my office

it's more or less what I remember hearing so the internet is not good at aligning the interests of uh third parties in fact it is very good at misaligning those interests and having the The Practice still be practical so we put out a patch to implement this response rate limiting uh and that was itself a bit of a political headache because I hadn't really checked very closely with my colleagues about it but again I was the CEO I was getting away with stuff like that um and um immediately my friends over at ophelius who run the dot info top level domain put it in production and that conversation was interesting they they said uh Paul I've heard there's a patch

well yes there is a patch well we need the patch well no you don't need the patch because you know I was selling them buying support service at the time and I reminded them whenever we send you new code for bind you put it into a test lab and you beat on it for six months and then if it survives then you put it in production so yes we have a patch it'll be out in a few weeks and you can start your process they said no Paul you don't understand I need the patch I need the patch now and you know somebody is in the habit of writing your checks and they make you

know demands then you you might have to just say well here's the patch but good luck with it so the the the patch was running in production within 15 minutes of that phone call and this is the result so here's how you read this chart the the why negative is uh request volume and the Y positive is response volume and the x is of course Time marching forward to the right and what you can see is that their request volume was between 103 maybe 300 megabits a second and their response volume was anywhere from one to two gigabits per second and that's simply because responses are bigger than requests um Thursday afternoon they put our patch

in and as you can see the request volume continued but the response volume went down to something very very small and um what I later got the back story over beer by the way I've not been able to pay for my own beer whenever affilias is in the room ever since so I guess there are some beneficial side effects open source um and they told me that they were getting a lot of calls from people saying exactly what I told you earlier please stop ddosing us and they kept answering do you want us to just Blacklist you so that we no longer answer any question from you about dot info now some people think dot in info is crap and said yes

but most people were afraid to do that and um I guess the ones who said yes probably were worrying worrying for a different set of reasons anyway the big concern that they should have had and they would have had is not for the fact that their house was on fire is what if we put in this patch and we stop answering legitimate traffic in other words what if we increase the latency increase the number of retries you know what's there are some bad things that could happen to this thing that we're the sort of the public Steward of this public resource of dot info um what if there's a false positive what if the code has a bug they didn't have

time to really investigate that question but what I will say is that the grand total of complaints from people after Thursday afternoon who said hey you're not as reliable I can't you know I run the Dig command and you don't answer or whatever I'm seeing retries the grand total of complaints that they've received zero so we got it right the first time and that's why you need to be running this in all of your servers however that's not the end of the story how much time have I got oh good all right um because this is just DNS and the reason we were able to write code that was able to be used in production without much testing and

didn't cause any problems is not because we are especially great coders like take a look at KRON from the 1980s um it's because we knew the protocol really really well and we knew exactly what a good query had to look like and what a bad one probably looked like and we did our own testing did a bunch of uh sort of self-red teaming to make sure that we weren't creating something that was trivial to bypass but um not everybody knows the DNS protocol well enough to think that through and their DNS is not by a long shot the only protocol that needs this attention every single protocol needs every single speaker of that protocol to be

considered in this type of light and then we need to add all of this new state and new complexity to every endpoint because opns don't have Sav and aren't ever going to have SUV so um give you a couple of examples of what I mean by protocol aware this is something a lot of people think they can do rate limiting in their firewalls in particular eye with my IPFW firewall I've got various kinds of rate limiting but it's very coarse grained what you can do is essentially you're simulating a T1 at a at whatever speed you want it to to run at with a certain queue size and uh anything beyond that gets random early dropped to just normal congestion

control as if it was a router but it's actually in a socket and that's not good enough for this because there will be too many packets that will match that that will be both good and bad so you have to look deeper into it than just what are the the five tuples and how can I maintain a quota on each flow also if you did what I just said and kept one for each flow then you would be subject to a state Mass Attack which would just move the problem from the first one you didn't know how to solve to be some other problem you don't know how to solve so you can't do it in your router

and there are some companies in the DNS space who sell DNS appliances which are really Linux to you boxes with special plastic on the front of them but um there's a couple of them now that have got an interface card that does the rate limiting for them so that this does not take Precious CPU Cycles from your main processor and I quietly went into a customer of one of those smart firewalls with or smart DNS appliances that had this wonderful Hardware stuff and I quietly wrote a program that would send the packets that would bypass that thing because it wasn't looking deep enough into the packet in particular if you see a negative answer you need to know why you get why

that negative answer was generated and that's something you won't be able to detect on The Wire this is not a bump in the wire answer there is no bump in the wire answer for this or we'd do it in our firewalls this is something every protocol speaker has to do by uniquely being able to determine which things should not have been repeated now you can game anything including us I know how to bypass our our stuff and a couple of other people have figured it out also it's not that tough and so what you're really doing here is not making yourself perfectly safe but rather making yourself less attractive um there's an old joke about two hackers

in the woods who encounter a bear and one of them reaches down to lace up his shoes and the other guy says there's no way we can outrun this bear and he said I'm just gonna outrun you and uh so it's kind of a horrible joke except I have friends like that and um and yet it's also the case that if we Harden the heck out of every name server if we make it the default if everybody within the sound of my voice and all of your friends as well turns this on then there will be fewer name servers available to bounce this type of attack off of and that means that the ones who

are left will get used more often hopefully they will get used often enough that it will start hurting not just their victims but themselves and they will like I I can tell you that philius when they were sending two gigabits of response traffic they had to pay for that right that changes how much IP connectivity you have to buy so if we can get a smaller number of people to amplify more attacks then we can eventually make it their problem to figure out why their internet connection costs too much and then maybe they will also turn this on I wish I knew a better answer this sounds quite darwinian to me but it's what we do

because it's the thing that everything else we tried didn't work so ultimately it's a token credit scheme which is as old as the hills when it comes to quota management and you just have to be able to remember which start of authority record they were in at the time the response was generated uh so that's basically which zone They are in and you have to know which client Network it is but we don't know the size of the remote Network so we're making gross generalizations and assuming 24s for ipv version 4 and slash 48s for IPv6 and we allow you to override that in the community config file but so far nobody has yet overridden it and said hey this

works better generally the defaults we chose are the ones you should choose so just mention this briefly in your config file and the rest of the logic will just kick in and dare I mentioned it won't cost you any CPU time it'll save you network bandwidth and it'll save you the psychic pain of having caused other people to receive traffic from you um right so last but not least I want to say that there is a protocol that we think of as not being susceptible to this that is and that's TCP so the reason we don't think of TCP as being susceptible to a spoofed source DDOS attack is because of all that setup and tear down traffic right somebody

sends a sin you send a sin act they send the act and by that time you're fairly sure that you each know the other person's random initial sequence number and that they really have to there has to be somebody at that address who is uh who knows they're talking to you in other words not spoofed and it is absolutely the case that at an application Level after the TCP session is set up that that is true however that initial sin has a problem there's a bug in the TCP specification because the a sin the syn uh has a sequence number it's uh just like a one character of data would have a sequence number so it is that the sin has a

sequence number and the specification for TCP says that if you transmit something that has a sequence number like the synac in this case and you don't get an acknowledgment then you have to retransmit it now this is the one part of the TCP State machine that didn't need this because frankly if you only sent one synac in response to a single sin right you didn't have a retry timer on that synac it's fine because no state will exist for anybody and the the original initiator if they're a real person will eventually send another sin to which you can send another sine Act but that's not what the spec says so what I've drawn here is the BSD case

which is where I can send one sin packet to each of you using his Source address and you will each send him if your BSD systems five synacs and that's a pretty decent amplification Factor people don't think of it as an amplification problem because your pack your the the synax you send him are not going to be back to back we normally think of DDOS as when your link is full and yet if there was a million of you instead of just this classroom you would each be sending enough packets with your 30 second retries or five second or one second whatever your retry was you would be sending enough traffic to congest that guy's link

that means that ever to fix this every TCP responder in other words every web server in every Home Appliance or iot device everything on the internet that is able to receive not just TCP 80 but TCP on any port number has to have rate limiting they have to remember what's been done recently or they have to violate the spec I've been trying to get the ietf to change the spec but the TCP specification is carved in Granite and they do not want to admit that this was a bug in it and so I don't think we're going to see a spec update I did not discover this I've given the authors of a woot paper that was in 2014

I kind of knew generally that this was a problem but they went and experimentally proved it and also found about eight other variations that show that the problem is even worse than I am describing now I do want to say that Linux behaves differently instead of five packets it will be 50. so I think sometimes about the the long tail problem that we have where you know I put maybe some bug or didn't understand what I was doing in get host by name in the original C library for DNS lookups and then someday we fixed it but there was so much of it that had been written into ROMs for of small cheap devices that they weren't going to

get patched and we see that more and more and we're going to see more of that on the iot front right these iot devices cost an awful lot less than the ones that they don't have ROMs there the whole thing the whole iot controller in the bottom of that light bulb sold for a nickel so they don't really have time in their schedule or money in their budget for red teaming it to see if it has a problem and you know their resources are are so constrained because they have so many competitors that really uh if you game this out you'll decide well probably it can't be patched and that's a problem or it might be patchable and that's a

problem because you never know who's going to be sending those patches and so there's not a real good outcome there on the other hand we have determined that UV radiation is the great killer of electronics and the the plastic cases that they are in and at least with refrigerators and toasters and other Home Appliances they tend to get replaced every couple of decades so there is a chance we're adding 6 million iot devices to the internet every day and there's a chance that within 20 years or so the ones we're adding now that have this bug will start to die and that sometime between now and then we will stop adding new things that have

these bugs so I think we'll probably all be fine

so I am I'm often criticized for coming into a room like this one and telling a story like that one and then not having some call to action you know okay Paul you've described the problem what should we do well change Human Nature um I I don't really know what to do I mean we did we fixed this for DNS because I know how to do that um I don't know how to get TCP fixed and I don't know how to get ntp fixed or NFS or SMB or any of the other things that use UDP I definitely know that people are not going to turn on Source address validation so brainstorm maybe you've got an idea I

know I'm I'm fresh out

yes so the idea of using liability law as a way to get people to clean up their acts um is probably well it's better than nothing but it's not as simple as that because these online casinos are often in offshore Havens Haven countries so that they won't be subject to National Law they may not have a national police force or even National courts and if they did then they are subject to various kinds of treaties to try and figure out how you as a California ISP or whatever would end up owing them some settlement after some lawsuit it's a very complicated Arrangement it's also very difficult to find out which ISP that was right so again we use the case

where I send you guys a packet and I claim to be him and you all answer him and he's wondering who are your isps well he won't be able to find that out very easily and certainly if there's a million of you there's probably a thousand of those isps so by making sure that this is death by a thousand stings we have made it almost impossible to for any success that you have in getting recourse to have any impact on their resources going forward I'll come back to you in a moment over here

yes there is something called sin cookies that Dan Bernstein successfully got into the TCP State machine about 20 years ago but right now the logic that uses them is congestion based in other words it goes into sin cookie mode when it has seen too many per unit time this isn't going to see very many per unit time so what I'm happy to say is that there is now a a kernel option that you can use the sys CTL command on on FreeBSD at least to say please violate the specification in this way send only one retry or you know send only one one response eventually I'm hoping to get those people to make that the default but

FreeBSD is a very small part of the University at the moment

sctp is one of the protocols I was thinking of although I can never remember where all the t's and P's go but sctp is like TCP but much better TCP only has a 32-bit CRC which means that the probability of receiving data different than what was transmitted approaches Unity at four gigabytes so if you're transferring large files and having problems it's because of cosmic rays sctp uses 48-bit crcs and is not vulnerable to any attack of nature or man and it does not have this problem because it has cookies all the time I and I would be we would so quickly have used sctp for DNS we would have moved off of this UDP horrible thing in

the TCP and the fallback and that we would use sctp but since no middle box allows packets that it did not understand when it was made 25 years ago when sctp wasn't here then sctp is hitting a black hole it is really only used for voice over IP which is that was how it got funded and all of the people who are doing voice over IP with sctp are ready to fall back to UDP when the other end can't hear them um not long ago the industry were talking about um an increased volume of hack on DMS poisoning and all that and then of some regulator Authority researchers came up and say Let's uh let's all put DNS set

on and then later researchers came out and say oh wait a minute when we put DNS Tech on we're going to have massive flood of uh DDOS due to similar patterns that you're talking about I thought could you please uh share your thoughts on this do we want dnsa or do we want something else um so dnsc is a bitter pill it creates almost as many problems as it solves but it isn't that good no matter how much it hurts and so I have deployed it I recommend that you do the same um DNA SEC is a story as old as mankind it took 16 years uh we restarted four times from scratch because people kept coming in late in

the game and saying wait but it doesn't also do this other thing that I also need and we've come up with something that doesn't have a backward compatibility mode there is a uh a new record type in the system called DS for delegated signer which exists in the parent in the same node as the delegating NS records and that means you have to have an exception if you're a validating resolver you have to know that if the thing you need is a DS record you don't send it to the closest in closing name servers you send it one hop up from them because that's where the DS record is and of course a lot of

middle box DNS forwarders don't know that rule so you can't deploy end-to-end DNS sect behind any middle box that was made more than say three years ago so it's I can't say it's an unmitigated disaster because it is slightly better than it is worse but ultimately this exposes the flaw in human engineering people solve the problems they have they don't think about the problems other people are then going to have that's the when when you ask them about that um they will generally imitate a tea party member and say oh let's just let the market sort that out and well so the market is ddosing that guy because I'm sending you packets pretending to be

from him and the market thinks that's just fine because for each one of you no Pebble feels itself to be part of an avalanche

we're out of time thank you all for coming [Applause] thank you great job sir thank you it was fun yeah they don't want to let you go