Network Hygiene and DDOS Prevention

John Brown is a sec senior security evangelist and member of the Cy Outreach team prior to joining Cy he was a CTO of a regional ISP that provided internet and voice Services via fiber optic and microwave technology please welcome John let's see if uh I can make technology work my passw Mo by the way

all right are we actually fil oh okay I'll try not to make your your life too hard I I wander a little bit so you're good all right cool good afternoon folks my name is John Brown I'm a senior security evangelist for team Cy uh we're based in Florida we're about 20-year-old company we do uh very interesting thread intelligence things um today I'm going to chat a bit about Network hygiene dos mitigation and prevention methods this will be a fairly quick fast talk um if you have questions about things feel free to ask them in the middle of the presentation when they're relevant I don't mind that if we need to get into a further discussion we'll find me in the

hallway and I'll be happy to sit down with p and paper and we can geek out as long as we need to geek out trademark disclaimer I might have some trademarks in here those trademarks belong to the people who own them not me not teamy we're just using them uh quick uh view on our agenda uh who is Team cing what's Network hygiene why do we care how are networks abused what things can we do about that um some examples and then some closing comments questions and answers so uh uh as I said my name is John Brown I'm a cissp I'm also formerly an ISC squar cisp instructor so I certified by ISC Square to teach those classes as well I

also used to teach microti training um 35 years basically is a networking internet geek was ran Ian's L rout DNS server which is one of the 133 critical DNS servers ran out for about three and a half years for Ian how many of you know what peing dd is okay I got one hand two hands that means the rest of you should homework assignment tonight is go look at what puring DB is uh I'm an to admin on peing DB it is the database of peing and interconnect for telecoms across the world and internet exchanges very useful information if you are doing uh any kind of cyber investigation you're trying to figure out well who is this as where

they connected what's going on what are their net blocks along with some of the stuff like at bgp do uh a.net um who knows what nanog is anybody okay I'm on the program committee for nanog if you got a great talk we'd love to hear about it and get you in and how you present it n so there's my my picture for N I am a recovering past owner of a regionalized p business R for about 17 years some of you here may know me from that world I have been there for almost three years now y oh one other thing I do real quick I teach people how to fly airplanes so uh I'm also a flight instructor who team Cy

we are about 15 20 year old company this is the marketing slide that's how I get paid to be able to go I got to have the marketing slite up there um we are a 15 plus year old company we do thread intelligence and we have lots of reputational things we have a commercial side and we have an Outreach side the Outreach side is stuff that we do for free and we'll talk a little bit about some of those but this is not a sales pitch you want to know more about tcy there's some stuff there go to our website one thing I will just sort of pimp for a second we have an email mailing list uh

dragon news bites it's a really great list it's free to sign up you have to be vetted to be on it uh but it's a great list of relevant cyber security issues that go through our S2 team Etc and then we publish that out so if you're interested go to the try website sign up for the dragon news bites uh it's free comes out daily weekly uh and will be happy to get you set up there so into our talk what is hygiene hygiene you know it's a series of practices it's about washing your hands and keeping yourself clean and uh helping to promote a healthy environment for not only for yourself but for your

neighbor right um and we all sort of want to have a healthy environment we all want to be able to have a hygiene we don't like having stinky neighbors as it were uh and so we do some basic things we brush our teeth we wash our hands we take showers etc etc and all that helps promote a much healthier environment for us humans to live him uh but what about Network hygiene well just as it's important for us to have good healthy habits for how we as humans interact and we we need to have good healthy habits about how our networks connect and interconnect so generally Network hygiene is going to be about keeping your Network clean so quick question how

many folks in here run a ISP or service provider type Network email hosting web hosting internet service provider uh Etc raising hand anybody okay couple how many of you run or a part of an Enterprise Network it's a un campus CNM um City of Albuquerque state of New Mexico Etc right okay so both service providers and Enterprises we need to have have the ability to have good Network hygiene and be able to have clean networks by doing this when a problem pops up it'll be easier to spot and resolve back in 2006 my friend Darren kenberg from Ripe MCC ripe is the regional internet Authority for Europe similar to what Aaron is American register of Internet numbers is here in

the US Network hygiene pays off there a link to that at the end of my slides um that talks about how Network hygiene can actually pay off in your operational cost and how your network runs why do we care each of us Network operators should care about Network hygiene because quite honestly if I'm running a bad Network Kristen probably isn't going to be as trusting or as thrilled about that because I am potentially a mitter of problems I'm potentially a source of problems for her network right so there's a trust relationship if I'm running a good clean Network then she's got oh I know I can call that guy John Brown up and he'll probably be able to

help me solve a problem in his Network because generally he has the mindset to run a clean Network and not create problems right it helps reduce the problem resolution also like I said it's about trust it's about your internal and external customers external customers if you're a service provider organization so uh even if you're a UNM or CNM you're doing distance education telea Health tele medicine those things like that you have external customers knowing that you are having a clean network will help build trust and help build brand with those customers out there we each have a duty at the end of the day to run a clean network not only for our ourselves and our own customers

but also for the person sitting to the right and sitting to the left of us well how do the malicious of act actors abuse networks IP spoofing is my big wonderful thing I love to talk about how many people here do not know what IP sping is feel free to raise your hand because I'll I'll dive into it for about a minute we'll talk about it okay real quick it's where I Forge The Source IP address in the IP packet right Dell is sending out a packet but he's not putting his address as the source he's putting K Christ in yes I do use people in the audience as examples I owe both of them beers now I think so D is going

to send out a packet and he's going to put Kristen's IP address in it well let's pretend that Kristen is still back over at Water Authority right so he's going to put water authorities IP address in that Source pack address and you're going to send that out on the internet source f well there's more than just one victim right there's the victim of Water Authority because their IPs being used probably because they want to turn around and use that to send packets over to Lea and deos her making it look like it's coming from Water Authority right that's the price of helping new right so there's all kinds of victims in the middle it's not just the Dos attack

victim it's the transport and peering providers in the middle it's the folks with the open recursive name servers in the middle of the network or out of the edge that are being leveraged in the internet engineering task force world we have what's known as best current practice 38 it's been around for 24 plus years it talks about how to prevent Source food package from leaving your network I'll leave you a little tidbit real quick I did a study about 6 months ago of New Mexico networks yeah we're not doing real good about Source moving folks I'm not going to put a name in shame up but yeah most governmental organizations are allowing Source Bo packets most of our isps are allowing

Source grp packets to leave their networks not a good thing there are simple ways to prevent Source proof packets from leaving your network unicast path forwarding we'll talk a little bit more about here in a minute is a great way to do it your Cisco your Juniper probably your found probably your P probably any other routing device out there knows how to do your RPF today so Source poofa packets allow us to leverage open recursive name servers and allows us to send that right so let's call an amplification of attack so how does that work very quickly so I'm going to use my three uh victims here in the in the room again right but I'm

going to mix it up a little bit we're going to Dos Dell pretending he still works at the city of Al which I know he doesn't we're going to Dos Dell back here okay Lea over there is an open recursive main server and Kristen just because she has a ton of security uh credentials we're going to use her as our malicious actor right we know she's got her white hat uh pen testing and all those other white hat Pion we're going to make her a black hat today so she's going to launch a packet what she's going to do is she is going to put in that packet a source address of Dell and she's going to send that packet

to the open recursive name Ser over at Lea Lea is going to receive this tiny little packet of a DNS query and she's going to answer that query and she's going to send it where audience participation where is she going to send she going to send it to Dell right she going to send a big packet to Dell a big answer right we do a DNS query for the resource record type any which a lot of folks are now blocking thank goodness but we're going to do it any and then all of the NS record and um DNS sex stuff all that junk is going to go flying towards Dell and we're going to end up crashing his server we're going

to end up flooding his Transit repairing links over there well that is happening that amplification is happening because the network that Kristen is connected to is enabling or allowing Source boo packets to leave if christen's Transit provider did not allow Source boo packets to leave their Network this would be a nonissue DNS is not the only one ntp can also be used as a reflection how many of you have ever heard of a very large hosting company called ofh okay they had a big dos a couple four weeks ago I think months or so ago and then their report about that they came out and said that the purpose part of the problem was open ntp and open DNS

and Source move package so let's look in New Mexico how many open recursive DNS servers we have that can be used and have been many of them have been used for DNS amplification attacks throughout the state I saw this morning 950 servers in the City of Albuquerque being the largest of that there are 472 if you'd like to take a look go to showen I have the happy little showen query that I use right here all of these servers are able to be used to attack others on the internet I do not see a reason why fill in the blank large organization in Albuquerque needs 26 open recursive Main Service reachable on the global internet especially when you look at the

DNS for their domains they're using two of them and they're in the same exact IP space all right moving on let's talk about what bons are because bons is another hygiene issue bons are IP addresses that should not be out on the Internet rfc1918 is one of those typical blocks you know the 10x the 192.168 the 7212 those networks shouldn't be there but 192020 I think if I get that right shouldn't be there either there's a whole list of special use addresses that are available through an assigned by the internet assigned numbers Authority right but there's also other bones there are routes that are reclaimed by the regional Registries all right so we have five Regional

Registries in the world we have Aaron in North America we have Wright in Europe we have AP Nick in Asia we have Latin Nick in Latin America and we have afron Nick in Africa those organizations will reclaim IPS an ISP goes out of business the company turns in their addresses some other Reclamation action happens when those addresses come back they are not out on the public internet they shouldn't be used on the public internet you shouldn't see them in your routing table and so whoops so these are reclaimed and there we maintain at Team cry a list of bogons and you can get that fee it's free but here's a list of martians these are IP

addresses that should never be on the internet how many of you are filtering these so they don't leave your network you should be you should not allow packets to leave your network going to these IPS because these IPS should never be out on the internet to your translit appearing providers right and you should be receiving packets from the internet with these Source IPS because they shouldn't be there in the first place so you should make sure that you are blocking and dropping that traffic whether it's ip4 or ip6 we also have bogon ASN autonomous system numbers so here's a list of reserved as's it constantly changes so you want to keep up to date with it but there are

folks out announcing prefixes on the global internet using unallocated or reserved as mbers malicious actors use these things because they know attribution is not something that can easily be done right because there's no who is looked up for this you're not going to go to Hurricane Electric's uh lookup tools and see oh that's Billy Bob's company right because they're not allocated so from an attribution perspective it makes it harder for an investigator to figure out who and what and where mostly we have misconfigured that happens in the BG routing but both guns can also be used for doos and they can also be used to subvert traffic even if it ends up being a regional

subversion bcus as is can be used for Route hacks route hijacks route leaking ET Etc looking at into the global bgp table I typically see hundreds to thousands of Bon IPS in the routing table that shouldn't be there somebody somewhere is announcing them which means somebody somewhere is allowing them into the routing table which then propagates across the rest of the world criminals love this because it's harder to catch them when these routes are this shortlived so how do we reduce it well you can R do is by filtering your inbound bgp routes not to accept that if you're running bgp also make sure your outbounds aren't sending out anything we're not leaking our private IP space

via bgp to our Transit Andor peering providers same thing as Internet exchanges or you can also make use of Team C bgp feed and that feed will allow you to bring it in through pgp and be able to drop traffic to or from those prefixes all right so the next thing we can deal with I'm trying to help mitigate and keep the network clean and good Network hygiene is rpki how many folks here know what rpki is okay rpki is a public key infrastructure that is used to sign your routing announcements it's used to say this prefix 1981 125.5 sl24 belongs to me and it is cryptographically signed and maintained so that we know it is

actually belongs to me doesn't belong to somebody else and if somebody else is announcing it they're not correct rpki is fast and furiously coming folks to a network Transit provider near you in other words the big isps now are no longer accepting routes from customers that do not have rpki signed routs you can go to Aaron American registry of Internet numbers it takes less than 30 minutes if you own your own IV block it take less than 30 minutes to use their posted solution to sign your prefixes your route the benefit is is that when Dell signs his Rous at the City of Albuquerque again we're just pretending we went back in time we know he's not

there today we go back in time D signs those routes to the City of Albuquerque that is an attestation that says these prefixes Belong To The City of Albuquerque if somebody in Pakistan decides to snag those IPS and announce them they're not going to get very far because that attestation is false they can't put it out there they can't originate it with the right origin as and other data they can't pass that certification that is done in the routers in real time all right so all of the regional Registries right eron IP Nick Etc have an easy process to do this right soon your routes will not be accepted and soon is probably 2025 route rpki they said also helps keep

route hacking from happening if we look into New Mexico I can tell you that there are multiple cities within the city state of New Mexico whose routes are not signed by rpki at all I can also tell you that there are very large governmental state agencies here that have their routes 169 something or whatever it might be um um you have no rpki entry and no internet route registry entries either those networks are at a higher risk of hijacking than those folks that have signed prefixes signing rpki I saido is a 30 minute process and uh or less and it's very easy very simple to do doesn't cost any money cuz you're already an errand

customer and poof now your vs are protected if you need help with it Aaron is perfectly happy to help you you can send us an email at team cing I'm happy to help you there's plenty of folks that are willing to help you with it and a lot of us will help you with it for free or a good breakfast Breo so dos Network hygiene we talked already a little bit about it but sometimes we want to make sure we hit it number one biggest thing is stop preventing Source stop allowing Source proof packets to leave your network your neighbor to the left of you your neighbor to your right of you will be very happy about

that if we did not have Source Bove packets capability out there today I bet you a huge amount of our dos uh problems would disappear we wouldn't have those OS issues because the ability to launch an amplification Tack and leverage a open name server or an open ntp server would be removed charge in is another of use so become an internet hero stop Source boof package from leaving your network so how many folks here run a network that uses bgp a few of us okay cool so you can use bgp for security a lot of us will be jump up and down and say well wait a minute bgp is not very secure John it's got all kinds

of problems look at all the security issues and you're going to use it for security yeah because it's a great signaling protocol to quickly announce IP prefixes across your network or between peers that are bad IPS right so you can get IP address feeds that have reputational data from folks like spam house teamc and others ingest that into your bgp set a community string on it set a route policy up that allows you to drop traffic to or from those IP problem goes away when I ran my ISD there were times where I had over 30,000 32s in my routing table that were malicious IDs that none of my customers could ever get to or receive from

because we would drop it at our Edge to the outside world and what that did is is that cut off a huge amount of problems that folks could have had the moment that fishing attack clicked on it and in the malware tried to go and find it C2 oh I'm sorry the C2 is dead because I'm dropping it up at the transit side so now we can't phone home and we can't continue the malicious threat because I'm dropping it in 15 years of doing that I had zero false positives it works you C ver path forwarding is your friend and since I'm get I'm between you and lunch and the kind lady here just told after 5 minutes we'll keep going

you want to learn more about urpf find me in the hallway remote trigger black hole is another tool basically you're running bgp so let's say Kristen is my Transit provider I set up a deal with her where I announce a prefix with a 6555 col 666 uh Community string on it that tells her to set a policy in her router automatically no human intervention required to drop traffic to that I just sent her so now she can drop all that traffic before it comes to me so if I'm getting dos I have my one little 1 GB connection and there's 100 gigs coming at me I can have her drop it at her end where hopefully she has more capacity to

be able to uh absorb that traffic very quick easy thing to do um yeah we have a free service for that too what can you do if you're an ISP validate your external phing route filter make use of rpki please filter based on rpki make sure your Internet route registry data sets are up to date most of us in the state of New Mexico do not have internet route registry data for IP prefixes that is up to date or valid we also don't have who is Data there there's a very large governmental entity that still has its IP addresses under the name of somebody who hasn't worked there for at least 15 years and they know that it needs to be

done and it just hasn't never been cleaned up so if there's a contact issue who do they go they send an email to the person that doesn't exist that doesn't help in in in fixing problems make sure you're only anouncing your prefixes and that they're valid if you're not sure what's happening with your bgp world set up a test router a little $60 mic check would be great for that set up remote trigger uh with your peers set that in advance know how it works have an sop document created so that you have the ability to understand that policy filter prefixes that are on B boms if you're an Enterprise please please please make sure you don't have Source boo packets

leaving your network if you're running bgp do many of the things I just mentioned right make sure your route registry data is up to dat additional things go read BCP 38 best in current practice number 38 like I said earlier it's been around for about 24 years look at using Community Services after to help drop traffic and improve your network we all in a community need to work together to improve our Network hygiene if someone sees something see it say it talk about it help your neighbor improve their hygiene regardless of what organization Consultants that are in here msps that are in here if you see an issue with one of your clients help educate them on how to go about doing

things to prevent Source mve package and other malicious activity open recursive name servers stuff like that from being on the network uh real quick M take example I won't bore you with it but it's really short and simple to set up bogon filtering closing comments we're all responsible for Network piping that's probably the biggest thing netf flow use netf flow to see visibility in your network take time time to validate your routes take time to validate what you're sending out take time to make sure you're not running services that can be exploited this is an ongoing practice make sure you're not doing Source food I think I talked about Source food package a couple times uh at

the end of the day please do not want smelling networks if you have questions if you have comments feel free to send an email to outreach at com.com or find me in the hall uh many of you probably know where to find me otherwise and I'm sure we're out of time but if there are questions I'll be happy to take them if we're allowed to get questions and there's some documents up there you want to screenshot that of stuff if you want a copy of my slide deck find me I'll be happy to swap business cards with you and send you a copy of my slide so I'll stand for questions or comments or throwing

Tomatoes think the gentleman over here has a question

by the way I she walking over these Blinky LED things that there were in the sodere lab that is such the dope that was cool Whoever thought that up yeah Bravo yes sir I'm wondering if there's um an equivalent of a milici IP list uh some sort of reputation service for speaky networks is that feasible or valuable um you heard such there there isn't directly but if you go to maners ms.org and start looking at how the mutually insured routing World wants to talk about how routers folks with routers should play with each other you'll get a list of people who are not stinky um one other thing I'm going to add real quick with regards to Source Bo

packets and why it hasn't been done and why it needs to be done for a lot of reasons Vos was a money maker for large isps they charged you B B on usage and and consumption so they were not fiscally motivated to really help solve the problem because they're making money off of that overage that you are experiencing right now right the United Kingdom the UK a few years ago came out with a rule that said that you could not sell internet access to the government of the UK his Majesty's government unless you could validate on an ongoing basis that you did not allow Source boo packets to come I wonder whether or not our procurement folks here in the state

need to start thinking about putting that into rfps so you're buying internet access from those that are definitely not going to run a stinky Network any other questions I hope this was uh valuable I hope you guys learn something if you got questions see me in the hallway thank you very much for the opportunity and uh enjoy your lunch thank you very much