BGP Hijack — Who's Stealing Your Prefixes?

and that's going to show me a sequence of autonomous system numbers and tell me how how my router thinks to get to that prefix which organizations am I traversing to get there so the first one is the shortest buff because the there's only one as in the as buff 32 934 which happens to be Facebook so that's the ASN of Facebook and what's what this tells me is that open ens or at least this router is directly connected to Facebook over the Internet exchange so obviously this router is going to prefer that path in in a normal circumstances then the alternative so let's say that that link goes down the alternative is the green one and you can

see there are three asns in that path the one on the right hand side again is the same as the one above so it starts with the Facebook as so that again is 30 to 934 um then it shows 12 21 and 4637 so this tells me that my router is connected to 4637 which is uh in this case Telstra International which is connected to 1221 which is Telstra domestic which is a different organization which is then connected to Facebook and then the last one starts with 2914 that's NTT entity sends it to entity Australia which sends it to Telstra which didn't sends it to Facebook um anyway so this shows you that the router has multiple options and it

determines its path um based on the as buff length by default so there's all kinds of interesting things that you can infer from this buy Nas by just looking at an as buff you can see who originated the prefix by looking at the rightmost ASN you can see how we learned it by looking at the leftmost one and then you can learn something by the Relationships by looking in this path you can see that and in this case Facebook is connected directly to Telstra probably paste Telstra for Transit connectivity and the same with NTT there's all kinds of interesting things you can do okay so I think that's that's sort of the introduction into you know routing

on the internet how that works and and what that looks like so now let's take a look at some examples where you know it didn't quite work as expected at least not by most of us so let's talk Bitcoin I'm sure we already talked quite a bit about Bitcoin uh bitcoin's popular nowadays you know everybody wants to be in Bitcoin there's lots of money to be made and lost depending on when you got in um but there's also lots of uh Bitcoin mining farms and and this is an example of a company called megabit um megabit power in Washington state um they literally have thousands and thousands of boards working on mining bitcoins and all together they're trying

to solve this complex mathematicals puzzle and you know every now and then they find the answer and they get some reward for it lots of people do this this is an example of a five megawatt form in in Thailand um didn't work out so well for them um caught on fire millions of dollars in Hardware went up in in flames and apparently at no insurance um but either way I think what I'm trying to say is a lot of people are trying to get into this uh but not all of us have the money or resources to to actually start a Bitcoin farm so what you do when you don't have your own data center your

own Hardware well you know you can do the same as as we all do in our Enterprises or in home you go to Amazon or you go to digitalocean and you use their CPUs to to do the calculations and as long as you know the rental price is lower than the money you can make with it it's all works out and this is sort of where this story starts um a little bit about Bitcoin I am by far not a Bitcoin expert but um I think in the simplest form what happens is when uh when a Bitcoin miner comes up it basically joins a Bitcoin pool Bitcoin pool has a pool server and basically gives out assignments and gives you the

credits for it so this attack was officially detected or initially detected by Dell secureworks and it's something that happened between 2013 and October 2013 May 2014. and what happened here was um when miners would come up and register with the pool server um the pool server would send a redirect to another server as part and this is part of the protocol where you can base it's part of a load balancing mechanism you can say well you shouldn't be talking to me you should go and talk to this guy because I'm too busy or whatever um now the question is so that other server was controlled by the attacker so now the question is how was he able to

initiate that redirect well what he did he was basically hijacking the IP addresses of the pool servers so these pool servers in question were running in Amazon digitalocean ovh all your standard hosting providers so let's say that Amazon typically announces their prefixes in 16s he would find out a pool server isolated which was a single IP address isolated to a slash 24 which is a more specific block has about 256 addresses in it it would then announce that particular range of IP addresses and saying if you want to go here you need to send it to me and because there's no you know it's it's all based on trust everybody on the internet who saw that bgp update started

believing that and as a result all traffic for that slash 24 was now being redirected to you know this other server or at least the network he controlled at that point he initiated his redirect and the minus would go to a yet another server um which is where sort of the interesting thing came in um the redirect or the bgp hijack was only visible typically for a few minutes two three four minutes or so um fairly short but it was enough to you know to find a fair amount of miners initiate the redirect the minus would still be sticky to that other server and mine for days or weeks and basically mine but not got a reward for it so if

you look on the Forum several people after a few hours or days find out what happened um so this was a really interesting case because it was sophisticated on multiple levels um and Not only was it you know it was combining the Bitcoin stuff with bgp stuff but he was also trying to hide what he was doing using a technique called as buff spoofing so remember the path the the range of asns it was actually you know trying to change that and and lie about it so interject other ASN so it was very hard to determine where the actual attack came from so this went on for a few months affected several Cloud providers again digital ocean Amazon all

the ones that you know we all use for for different purposes um actually was a Canadian thing as well so um we've determined that the the networking question who was announcing these things because the thing is we can't go home tonight and try this like we need to have access to a router which is bgp and at that point you need to be connected to another provider or even better to an internet exchange so an internet exchange has a new term is is basically a large switch with lots of Internet providers connect to and then you have sort of this direct connection with all of them and then you can set up this bgp session with them

Transit providers are providers that are very large providers like level 3 or NTT or more Regional here let's think about Talus and bell and Shaw and you pay them to provide Transit they typically are pretty strict in what's called bgp filtering is in they have a list on their router and say okay I know you are open DNS so Google you're only allowed to announce these prefixes because I know those are your prefixes I'm not going to allow you to announce anyone else prefixes but that's a lot of administrative work so if you are an internet exchange where you typically peer with hundreds of other networks you know and those IP addresses change all the time people have just basically

given up on that it's like I'm not going to filter that that's way too much work I'm just going to trust that if you're announcing 20 prefixes today I'm gonna allow you to do 30 and they just have sort of a hard cap that's why this attack really works well on for example the Toronto internet exchange where or any internet internet exchange really but this happened on the Toronto internet exchange um so this was primarily affecting peers that were connected to the Toronto internet exchange who either appearing with this part with this organization or with uh what's called a route server which is really a proxy for bgp announcements um in summary Dell secure Works guesses

that this or estimates that they made about eighty three thousand dollars with this um we verified that there were at least 51 prefixes hijacked uh over the course of a few months which is a fact that 19 unique autonomous systems and you can see a few of them listed here but I mean none of that was really targeted it just happens to be where all these guys are right so this is a really interesting case where um you know people were using bgp hijacking to try and make a good buck so let's take a look at another example censorship so the internet is no longer truly open we all like to think so but every now and then there's quite a bit

of censorship going on and you know certain parts of the world this happens more than others turkey is this particular example but we all know about China um and several other countries so this is um this is actually a really interesting case because uh it affected our Network directly Open DNS but also uh Google and level 3 who also run large open DNS resolvers um so what happened here it was election time in Turkey um so March 28th till March 30 which is about exactly a year ago now and there was some stuff on YouTube and Twitter that the Turkish president didn't like and basically ordered his government to block both YouTube and Twitter for the

whole country um so then you as a techie are ordered to do this and you try to figure out how do I do this for a whole country right so there's different ways of doing it um but in in some cases especially um cases where a certain country only has one or two isps and they happen to be controlled by the government definitely makes it easier and it happens to be the case in turkey turkey has Turk Telecom which is the largest provider there and obviously most people there use the Turk Telecom DNS Services right um so they said that we can do that so what they did was they basically ended their DNS service at a static

entry so if you want to go to YouTube or Twitter this is the IP address which obviously wasn't really the IP address of YouTube and Twitter and as a result your computer was trying to connect to something that you know either didn't exist or was something that was controlled by the third Turkish government and says hey you're not allowed to go here or something like that um so you know succeeded Engineers happy or not at least they did what they had to do um and um so that was good for them um but pretty quickly uh these images started appearing on uh in mainstream press so as a DNS geek and most of you maybe like you see IP addresses spray

painted on bus stops and walls that's pretty cool right we don't really see that every day so we're like oh that's cool um so um what happened was people figured well if we use something like Open DNS or Google or some of the other you know free open resolvers we are able to bypass the the censorship so people started using this like crazy and at opening industry so it's a huge spike in in an uptake from users from turkey and I'm sure Google saw the same and some of the others as well so citizens of turkey happy we are now able to bypass the government censorship We Rule awesome right um Turkish government not so happy

um so uh within a few hours we noticed all traffic from Turkey dropped off it was basically zero traffic left um what happened was that now in um null routed all traffic to some of these IP addresses you know a popular open resolvers like Google and open DNS um well that sucked and um you know but they pretty quickly on realized that you know the collateral damage is pretty significant because not only did they now know route you know traffic to our name servers um they now also broke the internet of I don't know a very significant portion of the country because everybody had changed their DNS servers to us and I was like well I don't know I type in

Facebook and it doesn't work or I type in CNN or whatever they're visiting didn't work and you know we all know this like DNS doesn't work or I don't care internet's broken right um so they had to sort of undo this because people couldn't work anymore stuff was broken uh so pretty quickly on they undid this they realized that that hammer is a little bit too big um so you know we're happy again we see traffic all awesome um next day though they did revise their setup and so that's sort of the final setup where they came up with they had now configured a server or set of servers with the same IP addresses as

OpenDNS and Google's 888 and so on and so forth a few others and they started inserting new bgp announcements and this is an example of of that announcement so they started to announce for example 8.8.8.8 32. so this is a screenshot from when this happened I took it it is a normally if you want to if you want to go to 8.8.8 it's a slash 24 route that's sort of the accepted minimum route you can announce so that's the normal healthy route that we would be looking for that I know Google really announced but now we're seeing a different route and in fact the as buff is empty which means that it's locally originated so

this is the proof that they inserted a new bgp announcement a new route and he set the next hope to reserve that they controlled and they were in fact answering DNS queries it was actually a DNS server but now again they had applied that same mechanism as they did initially if you type in YouTube or Twitter they would give them back an IP address that obviously wasn't YouTube or Twitter and you can see an example here this is uh from a tool called ripe Atlas it's basically a very large measurement project so we did a DNS lookup from within turkey which is the one in Orange using 888 for example you can see the IP

address that was returned to 195 175 blah blah blah which is a an IP address controlled and owned by Turk Telecom so there's a clear proof that this is not a real YouTube server you were being redirected to this IP address if you were in Turkey but it wasn't actually YouTube and then in the green you can see an example of what it would actually supposed to look like where you can where it returns a set of of YouTube IP addresses so in summary the Turkish government has now achieved censorship using DNS and they figured out a way to also censor DNS service that they don't actually control Google open DNS so this is all

really really scary stuff fortunately for us um about two days later this was done with a lot of pressure on the background but eventually it got solved but it does show you know how vulnerable the system really is because there was very little we or even Google could do I mean just insert that announcement there and it's there it's just you can't just delete it so this is an example for um bgp hijacking for censorship and setting up a man in the middle again another example spamming so what's interesting about spamming I mean spam has been around forever we've talked about a lot um what can we say about that Well turns out there was actually a significant

operation going on last year where spammers were using bgp hijacking or bgp as a technique to solve their biggest pain the biggest pain for spammers as we all know is finding clean address space right so someone starts spamming you know within a few minutes or hours it ends up on The Blacklist and you can't really use it anymore it's also really easy to track you know who was doing the spamming so if you know maybe you don't care about that but sometimes you know that server maybe it was owned or whatever and takes taken offline so on and so forth so the biggest problem for spammers is finding unused address space or clean address space

as we all know we're running off IP we're running out of IP address space right ipv4 is it's almost gone we need to go V6 et cetera Etc but there's actually large large squats of ipv4 address space still available it's been allocated to organizations but many organizations are not actually using it so if you look in the bgp table you'll see oh it's allocated to the usdod or even the Canadian Forces or I don't know GE or whoever like some of these really large organizations have large spans of address space that they're not using at least not publicly maybe internally but it's not visible in the global bgp tables so what did these guys do is basically

start searching for that just Loop through it you know 1.1.1 is available 1.2 to ends Etc and so at some point they found literally thousands of networks that were not currently being announced and they said okay I'll just claim ownership of that and I'll start announcing it um configure it on my server and now I have a new IP address I got a whole slash 24 a whole slash 16 or whatever I looped through it starts sending spam and once I'm done I'll just use the next one I literally have thousands of networks to use for this very scary stuff um so that's what they would do so let's take a look at what that looks like if

you visualize it we spend a lot of time on sort of fingerprinting this and figuring out what exactly what happened so this is an example for one ASN um but we found different other asses that all had a very similar fingerprint initially we assumed they were all different operations sort of using the same technology later on we found out there was actually all well actually two organizations uh behind it but what you're seeing in this graph or in this visualization is basically on the bottom line you see that the timeline on the vertical line You'll see the basically list of prefixes and the green line is the lifetime of a simple of a the announcement of a prefix

so what you can see here is a very clear fingerprint where spammers would start to announce the address space eight prefixes at a time for a few hours stop it and immediately start a new batch of eight prefixes and sometimes that would be a few hours sometimes it would be two days depending I I don't really know what's going on but if you visualize it it's very very clear that you know a new cycle of announcements would start um so we contacted well not the asns involved not the origin asense but sort of their Upstream providers and say hey do you know what's going on here like it's very uncommon for a network to constantly start announcing new prefixes

typically you get an announcement you get a prefix allocated you announce it and you use it for years and then maybe you bring another one in but definitely not for hours at a time like nobody really owns that many prefixes that you can do this on uh basically for weeks or months so you start talking to them you said hey why are you announcing why are you accepting these prefixes remember we talked about filters um well they said we have an automatic filtering system in place um which is called um sort of a who is system for uh for routing health and they were using redb which is one of the more more popular systems

um and so we're just automatically generating these filters so as long as the customer the spammer in this case would add a who is entry for this it would just work and then we started a little bit harder so well why they are announcing all these prefixes all the time and they would say well if I look on my file and I talk to the customer they're what's called a DDOS mitigation company so they're mitigating attacks for different customers which is really the only reason why something like this could happen but then we would go back and say well who are they mitigating a tax for all these prefixes haven't been seen in years are they

mitigating attacks for ghosts or whatever right so that that basically that really got their attention um and they helped us a lot with looking at netflow data pcap data and indeed showed that you know 99 of the traffic was basically SMTP traffic in fact it was almost 100 percent which is actually interesting um because where did all this control traffic go wherever the DNS lookups where is the SSH traffic or whatever it was basically missing it sort of leads into a different uh conversation but it must have been definitely sort of another back-end system that was doing all that kind of stuff this is an example of the who is entries that they that they were adding

um so you can see the prefix there's a description and then there's the important part there's the origin as so they basically said I am hereby authorizing myself to announce this prefix from this origin ASN and then there's the maintainer object which is basically sort of the owner or the uh the guy who added this the uh so you know this is good and bad because now we knew oh crap they actually figured out how all this works they're smart guys but at the same time we can now look at you know what other routing opiates have they added so for this particular as or even by this maintainer and indeed we saw literally thousands of Route objects

um typically for prefixes that were announced in the past by the same ASN and we used to send spam we could validate all the items it's like okay now we validated it the cool part was there were also route objects in there who has who has entries for prefixes weren't announced yet so it also provided us with a Looking Glass into the future and we could start predicting what they were going to use next so really cool stuff again we work with the providers and got a lot of this stuff shut down um this is an interesting graph from another provider this is uh I believe a Russian one um the Russian ones were really

interesting they would start strictly at 9am Moscow time always that was really cool and they also take vacation so there's two distinct gaps in this graph I'm not sure if you can see it but it's basically over Christmas and New Year's that was pretty cool so you can definitely see um that there's humans behind this stopping and starting these operations so while we're talking about spamming um we've all heard about spam house and how they try to you know do something with the spam problem their reputation among some networks is somewhat dubious not everybody likes them they do good work but sometimes they're you know their Hammer is pretty big so they tend to make a lot of enemies especially sort

of in the world of spammers and the more gray areas of the internet so this is about I think in 2013 spam house got into the news because there was the biggest deed also on the internet ever and it was significant it was 300 gigabits of traffic this is remember two years ago so that's a fairly significant uh DDOS supposedly the biggest up to that point and that's not something you want to deal with Spam house at that point moved to cloudflare to help them with that and you know lots of isps were scrambling to uh to make this all work um and and sure there were a lot of bits and but it was really a blunt Hammer a

big cannon to try to take spam house offline but didn't get as much attention was at the same time the guy who was doing or supposedly behind this attack or the the DDOS also started to announce a more specific bgp announcement for one of the spam house rbl servers and you can see that in this uh this screenshot shot where they were announcing 204 16 254.14 which at that point was an rbl server it was running a DNS thing and basically you feed it an IP address and say is this a spammer or not or can I trust this and it basically says false or negative false a positive result so what happened was this was announced over the

Amsterdam internet exchange people connected to the Amsterdam internet exchange which is the largest internet Exchange in the world so there's hundreds of organizations there we're now routing to this new bgp route so it wouldn't end up on a server controlled by spam Mouse it would end up on a server controlled by this guy he had actually set up a DNS server and was basically giving back positive replies for everything so rendering the spam House service on reachable because some of these guys that noticed this said well my spam level went up like crazy and some of these are obviously not spam so as a result they had to shut down or deconfigure the spam house configuration

their mail servers because it was unusable so that's sort of another um you know dos service really and it was another Factor as part of the the larger DDOS and again a really interesting case how you can basically render something unusable or just another man in the middle really so let's talk a little bit about some of the I guess larger scale hijacks so what we've looked at right now they're typically pretty targeted trying to figure or a specific purpose some of these things are targeted you know if you look at the Bitcoin one we've looked at the third Telecom one there's really you know certain things that they're trying to achieve there unfortunately now Bishop even been

around the well and there's been more than one of these accidents sometimes they're just plain accidents like you know someone mistyping something on a router and you're you know forgetting a digit somewhere and all of a sudden you're announcing someone else's prefix um at bgpmon we've blocked quite a bit about Syria uh they've been in the news from an internet perspective uh because they basically disappeared for hours and days at a time the whole country would disappear obviously their infrastructure isn't as well established this year there's a lot of stuff going on power outages you know lots of stuff so we'd blog about you know Sirius offline they've been offline for 48 hours now this is this is awful

um but in December and again in January we blogged about them for a different reason because basically they now started to originate claim ownership of about 1500 prefixes that didn't belong to them they basically said oh you know I have this route to YouTube and to the usdod and to Reuters here in Canada so if you want to send traffic to them you can send it to me and I have a really awesome path to it so she really used me so this affected about 1500 prefixes from roughly 300 different organizations worldwide um the lifetime of this event was typically a few minutes but some of them tended to last for a little bit longer

and now the whole discussion sort of started so was this intentional was the Syrian electronic Army behind this or you know lots of conspiracy theories or is it just an engineer that mistyped something which is just as likely we've seen many of those cases but obviously when you mentioned Syria or China or Iran or Ukraine um you know it's it's it's definitely gets people's attention um so let's take a look at an example um so in this this is a visualization of what the internet looked like for one particular prefix at that point or at least a snapshot um so we're looking at the uh YouTube prefix 208 117 232-24 normally announced by Google YouTube now it was announced by uh by

this ASN 29386 which basically is the Syrian National Telecom establishment the main ISP the only ISP there um so that was he originated that it was connected to the next one 67 62 which is uh Telecom Italia which is one of the largest isps in the world so if you can make them believe that this route is the best buff and really exist you can be assured that significant portions of the internet will redirect traffic towards Syria so other prefixes that were affected DOD level 3 YouTube Telstra Rogers Time Warner Cable Akamai telephonica way way more than this but this is a few examples um and you know again whether this was intentional or not you know it's an

interesting discussion to have but from a purely infrastructure or operational perspective it doesn't really matter at this point traffic was being redirected to Syria and I know that's where it got dropped because there's no way that they have the capacity to deal with that much traffic from the YouTubes and the AKA mice and all that kind of stuff right so although it's an interesting discussion to have as an engineer I don't really care it's my man I'm down and that's how easy it is unfortunately in some cases uh to bring stuff down we've seen other cases uh indo-sat is notorious they're a large provider out of Indonesia we've seen it several happened several times there as well so it's really scary stuff

I'm not sure if it was intentional or not um what I will say though is it's very easy to hide stuff here as well similarly with a DDOS where people get DDOS and actually some other malicious activities going on it's just a distraction same could happen here right I could hijack 8 000 prefixes but I'm only really interested in one you know at that point it's hard to figure out which one I am really targeting and which ones are just noise but again we we can just look at the data there's can't really see what the intent was here um so one other example uh this is a little bit more advanced um it's actually almost the same but in

this case you're not really re-originating the prefix you're not claiming ownership where you're saying okay so this is an example from cloudflare cloud fire went down uh had a significant outage in October 2014. I'm not sure if everybody knows that cloudflare is it's a very large sort of content delivery Network some really really large networks run on it um so if they go down significant portion of the popular websites go down they had an outage affecting many many of their users in in October turned out that a provider in Colombia um or I think it was Colombia basically said okay I have this route to cloudflare I've learned that through my provider and now I'm going to

re-announce it to my other provider basically saying oh okay I learned this path and I'm going to share it with the world and so the rest of the world can send all this traffic to me and you can trust me I will send it back that's perfectly fine but it has to come through me um no typically these things are in accidents they're just a leak a filter that disappears somewhere or a filter that isn't configured correctly but again the same incident also affected AWS so if you start attracting traffic for Amazon web services and cloudflare you can be sure that you receive dozens of gigabits if not hundreds of gigabits and typically you know a regional

provider anywhere in Colombia in Canada that is not something you can handle and as a result traffic conditions the service for cloudflare went down again no bad intent probably still an outage so you know as an operator you're like I don't really care what the intent was here I'm down and we need to fix this all right so quick summary we looked at a few of the accidents or incidents that were definitely suspicious we looked at the Bitcoin one the turkey one uh spamming spam house you know all those things I think there's clear indication to at least mark them as suspicious um we also know that some of these things are just fat fingers in fact the

majority of them are probably fat fingered and then you can argue about if that matters or not but still it will cause outages so some of the thing typical things when we talk about fat fingering is configuration mistakes when people necessarily for the network Engineers out there when you start redistributing bgp into ospf and then out into bgp again that's a very traditional way to sort of originate the whole internet then there's these things called bgp traffic optimizers that will just make up random New Paths and say you know you should really use this buff and then they link it out to the rest of the world people start using it this is technique called as buff pre-pending

where you can make your path longer you say okay I wanted to make twice as long or I want to prepend my own I asked two or three times people don't always understand how this works and they said Okay I want to pre-pend it twice but what you're supposed to do is you have to manually say basically add a string to it and you say okay this is my ASN and I'm going to just copy my ASN a few times you can't depends a little bit on the operating system that you're using on your router but a lot of people say I want to pre-pan two or three times so what it actually shows is the origin as

is to or the origin as is three so that looks as a high check isn't really a hijack it's just a fat finger um but you know there's a lot of these examples uh we tend to post them on our blog how do you deal with this so there's some work going on in the ITF today uh around um called rpki which is really a sort of out-of-band mechanism sort of a who is you can think of it uh but with some crypto involved that you can cryptographically verify that this prefix really belongs to this guy and that's something we can't do today but there's still a lot of work to be done it's probably years out for it

before people will use it or are willing to use it so right now unfortunately the real answer is you know you have to watch for these kinds of things there's monitoring Services out there commercial free you can roll your own but that's really the only thing you can do today because you have to know that these things happen I've had users tell me that you know we just stopped seeing traffic coming in so I would reach out to them and say hey did you know this happened to your network someone rerouted your traffic to Latin America and it would be very skeptical goal and they said well who are you like I would just you know as a courtesy reach out

um and then they would say they would say no I said you must have seen an outage based on all this data you know you must have seen it out it's like no no no okay so I would explain them and after 10 minutes they say okay you know I might as well be truthful you know we didn't see indeed see an outage um we looked at all of our firewall logs vpns IDs blah blah blah and nothing nothing showed up and that was the case because the traffic just wasn't there the traffic was being sent somewhere else so it was a really interesting example of where some of these you know very Advanced security teams just you

know didn't click in that traffic could have just rerouted somewhere else um so it's definitely something to keep an eye on um because you know you can't log what you don't see um anyway so that's sort of a summary of what's going on in the world of routing um some cool cool examples and um yeah let me know if you have any questions feedback thank you foreign