Digging and delving in DNS

all right good afternoon everyone um first of all it's lovely seeing people in person again it's been four years for me since having a a live audience at a conference so having my had my thunder stolen it is always dns's fault um i've been a long proponent the dns is the most unloved protocol on the internet everyone just assumes it works but when it breaks everything breaks yet nobody really wants to care too much about dns so a little bit about me i'm currently preserving in norway as professor of cyber security at norov for over 20 years i've been working in networking cyber security i've worked in higher education defense telecoms finance etc but all of this has generally been fairly low level i am unashamed lover of packets if you want to see some of the work i've done the background here comes from a tool that we've i've developed with students over for 10 15 years ago for visualizing packets and it's still one of the things that i really enjoy working with so i love packets i love the command line all the analysis done for this was done with bash said grep awk and a few other things but no no gui was used no pixels were sort of assaulted in it you can find me on on linkedin and twitter the some i've already had two people ask me today what is rykenology um as in the bio it's the study and collection of old woodworking planes it's something i'd like to get back into i have yet to find old stuff here in norway that seems to be a challenge all right so on to what we're actually talking about um why is it worth looking at dns and i'm sure many of you have seen this meme um over the years anyone who's worked in networks and everyone says of course it's not dns there's absolutely no way it's dns of course it was dna so this has been attributed to to s what's it says brian brenski um but it really is true and it's funny because it is true the dns is so unloved and it's largely because people don't understand why you need to have dns dns for most people is some random setting you get on a dhcp or it's something that your isp gives you you've got these random eight numbers that you have to put in and if you don't have them nothing works but beyond that there's actually a depth a lack of depth of understanding as to how dns actually works and every single click on every single web browser and every single mobile application pretty much relies on dns so my thought was here i've been playing with dns for years and i was i was largely inspired by two final year students whose projects i worked on with them last year looking specifically at the current state of affairs of dns in norway one of the big challenges though was finding data and i'll get to that but the question i had was how do we start quantifying what resiliency is in terms of national infrastructure with all the brewing things that were happening in sort of january and february this year there was a lot of discussion on some of the sort of the network operators mailing list about what is the viability what is the risk to dns should we have global cyber warfare breaking out and it kept coming back to the idea of one needs to have resiliency and that means redundancy and high availability resiliency means when things go wrong you can recover but ideally no one should notice things that go wrong and start so from this idea go and start looking at what makes our best practice diversification why should you not have your two dns servers for your organization as dot 46 and dot 47 in the net block you've got there's nothing saying that won't work in fact it works wonderfully until you lose routing until you you lose connectivity and so we start looking at that kind of diversification what is the logical distance between your dns servers what is the geographical distance between your dns servers and logical is we generally start measuring that as to how diverse are the autonomous systems the asn's that these systems are actually working on none of this is news this has been around since the early 90s anyone who's worked with dns will have come across the infamous bog or the bind operator's guide all this good advice is there as well in the as in a whole number of rfcs so coming back to where we were in sort of january february this year and there's this discussion about what happens is dns really critical infrastructure um yes it is but nobody you look at sort of the government definitions and sort of cyber warfare definitions around the world when people talk critical infrastructure it's power plots sewage telecommunications and yes dns is part of telecommunications but i have yet to see a document that specifically calls out dns as being critical infrastructure this further spurred me on by the fact that when sort of in the as sort of the late february russian tanks rolled over the border into ukraine and sanctions started being tightened and a mad rush by russian organizations largely instructed by their government to pull dns that they had hosted in france and germany and sort of west west and europe the us pull it away and try and re-host it back in the country in the event that a the the country had total internet isolation and that really to me reinforced the the question of saying well who actually owns your dns so we've come up with the experiment what are the questions we're trying to answer so we can answer it on a global scale and it's a project i've got running now that i've actually got code running fast but norway is a relatively small country in dns terms it's minute so let's let's try and sort of do some exploration what does norway actually have what's within the no domain and start asking some of the high level questions as to how much of norwegian dns is actually hosted in norway versus friendly countries versus possibly unfriendly countries and on based on this and a number of other diversification factors say what is the resiliency for the for the country's dns um and then since we spending all this time asking questions of dns servers all over the internet we may as well grab a whole bunch of other information while we're at it and i've looked at at the adoption rate of various extensions and features within dns um there's a whole lot i've looked at in terms of the time we've got today i've just focused chosen to focus on two of them one which is spf many of you should be familiar with that it's effectively it's not to stop you getting sunburned but it's to stop the spam coming in and then a record that many people haven't actually heard of the caa record unsurprisingly caa very very low usage that's your spoiler for a little bit later so norwegian dns run by norad relatively small and global terms the latest stats i can get from their web page about 830 000 domains and there's been a fairly rapid growth over the last couple of years and it seems to be slightly tailing off over 21 and sort of early 22 so far one of the really interesting things that's held up is the norwegian sort of country code the um domain is held up as a case study of good governance within the structures the fact it's really actually difficult to get a no domain not that it's it's difficult in terms of technology side but you've got to go through the process of you pretty much stuffed you can't register a norwegian domain without a person number or business number um net result norwegian domain space is actually fairly clean in terms of it's not populated by phishing and scammers and all the all the sort of mess that you find in the likes of dot biz and sort of xyz and all the other um really cheap and easy domains so with all of this one of the big challenges we need something to start with now i didn't try again for this experiment because i tried last year with my students and we just got no response back so if anyone knows someone in norit that would like to extend this to a more complete result please let me know put me in touch with them um so because we'd asked for another idea and said can we get a list and we never got a response back which is not surprising because most country registrars they don't want to distribute lists of domains because they are prime source for people running scanning and exploitation and domain and domain squatting and all the rest of it so i had to put a list together this was somewhat harder than i initially thought but with a little bit of digging around and playing and to rattling around the dark parts of the internet i came up with a list of around 570 000 norwegian domains um now 570 000 is obviously not everything the new domains are being registered every day domains are expiring but it's a good enough samples 570 000 out of 840 000. statistically it's a good enough sample to be regarded as representative so it's about what 68 percent of the what what norad says are the number of registered domains currently in place so a number of caveats that sort of apply to this and everything else as we go through it is data's volatile domain's expired new ones are registered however imperfect data is better than no data at all again everything you can run the exactly the same tests i've run if you run them today you're going to end up with different numbers that's a guarantee just because we're working on on sort of the a snapshot of data being a great lover of the command line i thought ah this can't be too bad and i whipped up some bash and a bit of orc and some other shell scripts magic to make it run and it took three days to run through the list that was actually not too bad because i'd forgotten it was running but i'd asked the wrong questions i actually got very little useful information back other than it took three days fast forward a number of other iterations tossing out what i was using and i delved into that horrible place called golang came across zdns it's written by the same guys that develop zed map and stuff really a team of guys really good in terms of high performance network scanning highly highly recommend if you're doing anything around dns in terms of hunting around the dns is amazing just cutting to zdns and a little bit of other cleanup went from three days to 38 minutes to scan the whole of norwegian dns and i reckon with a little bit more tweaking i can probably get that under 15 minutes while still being a good network citizen in terms of not overloading web servers or name servers etc because that's something one needs to be aware of when you're doing this kind of open data collection is you need to do it in a way that is ethical and is not going to cause harm as you go along so it's not about running queries as fast as you can so in summary what we found from those 570 000 domains we distilled it down there's about 16 000 name servers so 16 000 name servers the machines that actually host the dns zones and will reply to queries at a minimum a domain needs to deal with servers some went to three there were a couple that went to four or five but generally two is is is the norm we take the sixteen thousand name servers they collapse down to effectively five thousand six hundred second level domains um so that's and it is inaccurate and the whole of the uk just got lumped under code at uk um in the analysis um but it's a good enough metric given that a lot of these dna servers actually are sitting on your traditional tlds of dot common.net what was really surprising when running this is there was probably less than one percent of the data that i had was bad in terms of that there was no such domain in existence or the the server just refused to respond which in my mind points to a pretty well run and functional dns infrastructure for the country also it was really interesting to see a fairly significant number and i completely forget what it is but we're actually making use of the internet internationalization extensions to dns the so-called puny coding encoding allowing for the the three additional um norwegian language characters which is really really nice to actually see what also came out is there are a couple of giants and and one can only call them giants or leviathans within the dna space um within nor and we'll look at that in the next slide very very dominant um but all of this there was a whole lot of data to to work with and collecting the data was just the first point then it went about filtering and here again i came back to my good old friends grep said orc cut all done at the at the command line so coming out of the the findings we had um yeah top domain hosts the top 10 name servers um in the list served if you if you add up all the domains that's 72 nearly 73 of norwegian domains are served by 10 machines top 20 systems it's nearly just under 80 this is incredibly heavy density and while this may be great from a performance and all the rest of it it starts raising the question of what happens if if we can take 20 machines out on the internet be that via physic sort of a kinetic strike be advised sabotage null routing denial of service hunting that's potentially 80 percent of the norwegian dns space that fails to work what those addresses are i will leave to you as a as an exercise to yourself but what i can show you is sort of collapsing down the just looking at the the top level domains that host these dns servers hype.net 44 of domains have dns servers sitting in hype.net i don't know if anyone from hype.net here but you are by far the leader by an order of magnitude over any other competitors one.com pro isp it goes on cloudflare is no real surprise etc and the list goes on unsurprisingly it matches the data that nor id publishes the menu shop is by far more than anybody else and this is this was taken off their nor id info page today so having a look at all of this and one could spend an entire day just sort of pulling out some of the nitty gritties um but to try and summarize what we've got in the time there is relatively good dispersion the the big players in in dns hosting or big isps that are providing dns service to their clients have understood the basic rules they are geographically separated they are different net blocks different as membership if we look at just the top three name servers in s123hype.net that you can see they're on three different as numbers so it means three different separate routing paths around the internet good for resiliency they're on three different net blocks we haven't got any of them sitting as dot 47 and dot 48 two machines sitting next to each other um there are places where you still see that big companies around the world that have dot 46.47 as their name servers and they run them on premise what happens when a line goes down and we can see that the different organizations run that so i've used this as one example but if one goes through sort of the top 70 odd percent one actually sees there is relatively good adherence to to guidance coming back to the question of who owns your dns actually taking all of these dna servers and doing some geolocation on them and i've got a big star there because yes geolocation has its problems i tried to to minimize this by using both the max mind sort of repositories as well as the live lookup service that team sumaru offers through their whois operations and there was very little discrepancy so i'm comfortable enough that in broad strokes these numbers are right 37 of name servers are hosted in the us only 16 of name servers of the total and again it's not weighted by the number of domains they serve this is so this is lies damn lies and statistics we'll see a little bit later are hosted in norway then we've got germany sweden france and then rapidly following off that is finland etc the interesting ones i found is we have 51 domains whose name servers are located in russia 16 in china and 5 with name servers in iran why did i choose those because they're interesting countries when one starts looking at some of the cyber warfare doctrine in the world so what is the impact on this 20 ip addresses potentially we could paralyze norway's infrastructure it would be a fairly great achievement for someone to take those 20 out but it is a possibility but they would have to be taken out simultaneously if you fail to take them out simultaneously the attack won't work because that is what dns is designed to do and to remind you that the original design behind the internet was when the nukes dropped the metric the u.s department of defense through darpa wanted a network that would self-heal and the internet that has evolved from that has done a really really good um show of that today so quickly to finish off some of the other things i looked at spf records generally used for providing authenticity and anti-spam stopping people spoofing your domains very useful for stopping sort of guided phishing attacks against users etc of the 570 243 or about 42 percent of the domains surveyed have some kind of spf record that looks good until we look at some of the other data spf has a whole bunch of other functions but i've looked at some of the the key things the minus all means effectively unless i'm explicitly allowing it it's not allowed everything is denied other than what i that's the the sort of the optimum setting so about 42 percent have it right 87 domains have plus all which means doesn't matter who it is they can send i authorize the entire internet to send email on my behalf i thought initially this might just be one provider that had set things up incorrectly but it spread across a number of domains on different providers which was was an interesting observation the remainder comes in um with the sort of the tilda which means i don't explicitly disallow it but if any sort of male processing software should flag it as it wasn't sent from an explicitly allowed source um looked at some of the include stuff but 149 000 of them include records they don't explicitly state they include from somewhere else chief among these is they include from the the settings from sps protection that immediately tells me that number of domains are using outsourced hosting on microsoft o365 infrastructure because that's the only reason that you would go about including that certificate authority the caa the extension it's been around since 2010 but very few people know about it it basically says who can actually go about issuing certificates and the results i've got here is very very few people less than half a percent actually had this there were some really weird things in place null records but the most important and the most interesting to me was the significant number of of domains that made use of let's encrypt in terms of the free encryption um some domains had as many as nine um ca providers listed um and and that was interesting so these are the top cas let's encrypt 88 of the domains that had this we're making use of let's encrypt so it's a bit of a question of is this the chicken or the egg are they using it because they're using providers that require it or are they using it because it's good practice it's i'm now currently running exactly the same experiment looking for some busy running through the majestics million top what what are the top million domains and on the inter seeing what those results come out so the last sort of 10 15 seconds is so what um what's what's the takeaway from this well dns is important there is no hard and fast answer other than adhere to standards and be aware think about with you as an organization where do you host your dns service or is it something that you do you run or do you blindly trust that your upstream isp takes care of it are they actually taking care of it it works until it doesn't work ask these kind of questions around resiliency and that brings us to the end we've dug a quick tunnel through dns and we've come to the end with probably more questions than actual answers thank you very much thank you barry