Adventures & Findings in ISP Hacking

Afternoon. We have Ian who'll talk about adventures and findings in ISP hacking. Uh as a reminder, uh please put all your questions in Slato. Uh besides SF.org/q&A, there will not be a in-person Q&A. So if you have questions, uh that's the best place to address them. With that, thank you Ian. Thank you everyone. [Applause] So, as I was just mentioned, I'm Ian. I go by LANRA in social media. I blog about things like this and other main things at land.com. Um, I'm an offensive security engineer on a red team. I run the besides SF network. It's like the Wi-Fi some of you may have been using. I run my own hobby internet service provider for fun. Um, and I run a

historical dance database at dance. Coffee. This talk is about none of these things. I'll be covering some security findings and methods used on two different Bay Area ISPs that I looked into over the past few years. And while not a traditional supply chain security shock, I will be talking about how some really basic vulnerabilities can be hidden within an ISP's network and how they can impact you um as a customer or a business that might be downstream of them. So without further ado, the first ISP um pigsplain. This is a local ISP whose name I've altered for the purposes of this talk. That'll become apparent later. So it all started when the internet out. Nothing motivates me more than something

getting between me and a functional internet connection, especially when stuck at home. And this was amid like COVID pandemic. So with no internet access, I end up checking my firewall rules and noticed a lot of inbound traffic on the WAN port from private IP space. This is RC1 1918 space. It's like know what you just see in your home network behind NAT 192 168 172 16 like 10 networks. Uh you shouldn't be seeing this on over the internet is not routable. Um I was curious why was my home's router receiving these broadcast traffic. Someone was broadcasting was not from this internal pspace especially since my ISP did not use carry grad. So I added an interface or an IP

address on the interface on my WAN on the W port of my router in these subnets so I can start actually interacting with this these subnets send my own packets and see see what's there and like do a network scan and I end up finding all the or a lot of the ISP's internal switches and routers this turned out to be their control plane there were no VLANs so for those who may not know what is a control plane this is the default network within the networks infrastructure it's contains most of their management interfaces it's used for things like SN tenant P IBGP um stuff like this. It should be very privileged, no user data. It's usually

used to like admin or configure the network and ideally should be logically separated from other networks including user user networks. So this is kind of an outline of how a like a typical ISP might um set their network with a control plane and VLANs between various various customers. You know, you have a different VLAN per customer, so they can't talk to each other. um in the I the ISP modem like the your your home modem would be the edge of their network. The ISP can't get into anything beyond that and you have no access to anything beyond that into the ISP's network. Um the control plane network allows for the ISPs engineers to access the management interfaces of all

their routers, switches etc. And again like they can reach into like your your home modem but nothing beyond that because normally the ISP owns that modem. uh nobody else should have access to the control plane network especially the customers and depending on how it's configured routers on the network um on different sub networks between them can selectively choose a route between them and chooses pass traffic through but that's all like software find and firewall rules so this was this network as one giant flat network the DHB server would is only issue leases for the WAN subnet the internet so like you plug in your home router you get an IP address you can route and it just works um but it's

still all flat like this is a in my opinion a very bad network design. Um and lots of modern network gear is very talkative. It would send lots of broadcast packets um along the control plane and trying to like auto discover other switches, other routers. Um and this is also polluting the customer's network like if you're on a a Wisp or a limited band connection. The all these packets are consuming up your bandwidth that you're paying for. Um so you're getting slower internet. So this is what I did like we mentioned earlier. I just added an IP address on my router here like stand up by the top right one here um on the on

that theoretical interface and I can start inter um interacting interacting with that traffic by adding this IP address you use tools like end mapap and see what's where so this is why finding you know found lots of internal services SNMP NTP SA server tnet web servers for all kinds of internal devices um and all accessible through the control plane um I also found you can use the default gateway on the control plane and just get internet that way then you're like you're basically opting into carry net but it's not via the normal path for the internet. So like you kind of get anonymous internet access because you don't you're not using your WAN IP using

the ISP's WAN IP address to get internet access. Um we start integrating a lot of the devices found you can see via the HTP responses or by the banners and tnet or SSH what types of devices they are what types of routers or switches what model numbers. Um some of these devices have default credentials some of them worked some devices had no authentication at all some had authentication but those weren't as interesting. Um, and of course you do SNMP scans will kind of tell you what interfaces on what switches are plugged into what other interfaces and where how the bandwidth flows at the network. So you get a really good uh like a virtual layout of like what where the network is

or logical layout. Additionally um one particular switch allowed guest access read only access to the whole switch. Not that bad but one interesting thing is the guest access has the ability to dump the config. Then the config contained the admin password and plain text. So that was exciting. Um, I also found I started googling for some of the uh MAC addresses that some switches found and I found form posts of employees of this ISP posting their entire switch configs on various forms on there asking for help to various things. So, um I I have two screenshots of here. I added these redactions. Um they're not redacted at all online. These configs contain their passwords um

of admin users um IP addresses even like their own corporate office wireless credentials are in here. Um all kinds of other stuff and the firewall rules. Uh I didn't test any of this works. I just saw them. I don't know how old it is. So at this point I have a really good idea of how like the network is logically laid out. But let's talk about like a physical layout. This is an ISP that spans the whole Bay Area. Let's try to figure out where are these switches physically located, which buildings are talked to which other buildings. Um so like many home ISPs they have their own name server and you can use reverse DNS

which is a DNS PTR record to resolve an IP address domain name to query it. So just by quering their name server for the IP address on the control plane of these devices, I'll get back their internal host name and they like the format you know location type of device dot you know their domain name. So I can start doing this for every device I found and it would tell me like you know oh they put like the building address it's located in in their vers. So I can start physically mapping them out which which routers talk to which other routers and which ones are linked out. Um it gave a really good physical layout

the entire network map. I work for another ISP not kind of my hobby one. So you're really good for competitive intelligence. Um, this is one of the theory I had. I didn't actually test this, but this ISP gives you a modem like you'd expect. And the authentication for the modem into the network is done via the modem's MAC address. The modem's MAC address is stored on SPI flash chip point but to by the little arrow right there that you can just dump and rewrite. And I found my MAC address. Um, I guess it's in that slide right there. Um, it's at Office location 24. You can change that bas to any other MAC address and their network

will think that you're that you are the other device. So, say like you get get on the onto their network, do an ARP scan, get the addresses of all the other customers, clone any of those, you now get free internet that way. Um, of course, you still need the the hardware, but you can buy that on eBay. It's not that hard. So, for the disclosure of this, um, like I said, this is during the pandemic. This is information is a few years old already, but I usually find these like early August, I started discovering a lot of these. I emailed their support with some of the high level findings trying to disclose this. Um, I don't hear back from them. I sent

some follow-up emails over the next few months. I don't hear back from them or they said they're going for network seam. Um, I've tried to meet one of the lead engineers at a local barrier meetup. I told them in person and to I'm kind of paraphrasing him like as long as the internet works, they don't really care. Um, it all started when there it didn't work, but like that that issue was resolved. So, I don't know. Do it that way you want. Here's the second ISP. Um, Sonic. They're in the Bay Area and I highly recommend them. Um, I've since moved and I used use them. Sonic is great. So, Sonic use, if you're unlucky enough to

have Sonic fiber, they give you an it's an optical user optical network terminal. Converts fiber to Ethernet. This is what the gigabit fiber modem looks like for Sonic. Um, it uses the GPON standard. The particular one that they use or that I was I I had was a 80ran 411. Um, does gigabit GPON. It also does VOIPE and has a MIP CPU. And you can see there's the physical characteristics of the device. You know, there's a Ethernet port, a phone port, power port, fiber port, and you can see the inside. A big heat sink, and there's one little header on thing out the side of it. That's actually just for a UPS. There's nothing interesting on there. So, first thing I

want to do to learn more about this device is to find try to find a UART serial console. The device allows you get logs and possibly more and to interact with the device at a very low level. To do that, three pins are required. You need ground, transmit, and receive. You can identify ground pretty easily with a multimeter by finding like some shielding somewhere. the negative end of the power and look for zero ohms from any of the debug pins. Um, transmit can be a little more difficult, but it's usually identified by fast fluctuations if the devices transmit anything between zero and 3.3 volts. Um, ideally though, you want to attach a log logic analyzer

to a lot of these pins and look for async serial output and decode that nasty and look for something like this. So, this is a screenshot of like the sal program which I use um just like what the UR would look like output for the hello if that was to be reprinted. Um, you can use the timing differences between the peaks to figure out the baud rate, too. And but there's usually like two or three common ones. You try those and move on if it you don't get anything. And the RX pin won't output anything, but it most likely be like literally the pin next to TX almost always. And it's important to remember uh to connect the TX to RX RXTX for

whatever UR adapter you're using. If you connect TXT TX RX, nothing's going to work. So in this case, UR was identified on a hidden form pin connector. you may have not noticed in the prior slide. Um, you can see the pin out here that you'll figure out, you know, TX, RX, 3, 3.3 volts and ground. Um, and amusingly, this actually can be accessed externally. You don't have to take the device apart by pushing like the little wires through the air vent holes and you connect right to it. Um, but now we can read the boot logs and we can learn that the device uses the Uboot boot loader, which is really common on embedded electronics, especially ones that run

Linux. And this device does run Linux. Um and after we see we see all the boot logs and boots up and when it's done booting we are presented with a login prompt and can't go much further than that. So the next step is let's get past this login prompt. So the ideal way to do this is to desolder the nan flash storage and dump it with an external NAND uper. Uh there's a little risk though to damaging the device when desoldering and resoldering. And at the time I didn't feel like taking that risk. Um so instead I decided the very slow and hacky way. Um, so I used u art to enter the Uboot menu and the Uboot

bootloader has a few commands that are of interest here. So we have uh one command that lets you take all of the nan flash. I think this one was 1280 megabytes and just read it all into memory. You give it you give it a memory address for flash to read from and a memory and a number of bytes to read and remember memory to write write to and just it it just does it. It's very very low level. So I did that. Then there's another command to print memory print the contents of memory as hex over serial. So I just issue that command um and then it prints hex and this is a screenshot of my laptop uh receiving

that dump um both in code and hex and then the ask representation over about a 115k ba serial um you'll slowly read out all the content of flash as hex this is incredibly slow um once you get this file uh you can use programs like you know screen with the -l to log the output of the serial outcome serial output gp set and xxt to reconstruct it to like an actual like binary with the flash dumper would actually given you um this particular doing this on 120 megabytes doing it this way took about 15 and a half hours to actually dump. So a normal dumper would take maybe a minute or two. Um so left running

overnight. Um there's also a tool called BCM CF dump that is meant for the same model CPU that's in this device that will do this. It automates the process. It's not any faster. It still takes 15 half hours, but it just means you don't have to use like bunch of shell scripts to reconstruct the binary. It does it for you, but it's still incredibly slow. But now we have a flash dump. We can try to see if we can see if we can find some um find anything there. So the file system has a bunch of partitions and you see here this is the top screenshot is from the boot logs to see what the

various partitions were and we have the offsets into them. There's no partition table like you'd have on your normal computer like GBT or MBR that does this. We have to you know just know the actual offsets in the file system and we have those now. Um the most interesting one the root file system is J FFSS2 and we can pull that out using the DV command you know just set the start offset and end offset or number bytes in the in these dumps and create a new file and then extract you know the the JFFS2 file system and extract your local directory in your system to look through and see what's in there. Of course the first thing you

want to look at is Etsy password see what see what users exist. Found four users admin support user and nobody. Um the interesting things here are these users or this is step using just MD5 for the hashing of the password which is not that great. Um also know it looked Etsy group. Um all these members even nobody is a member of the root group G0. So they're effectively all root. Um and there's like on the slides there like two links like online MD5 hashtracking services. Um there two of these users um their passwords are found like instantly. The M5s are known. The users the user user password was user. The support users password was support.

I am a bit embarrassed that I didn't guess those. Um yeah, next. My gosh, I should try a little harder on that. Um the admin and Nobody users weren't weren't found that way. Um and also just interesting note here, why does nobody have a password? Uh the nobody user should not have a password, should not be allowed to log in. Why was it shell bsh? Also, there's a comment there saying it's for FTP. That kind of makes sense, but then it still shouldn't have been SH as a shell. So, the your FTP user can log in as, you know, get a shell in theory. But let's move on a little bit. So, also in the boot logs was a strange

message here about CIS RQ. Um, I had never heard of this before. Um, stands for system RQ. Um, I don't know if anyone else ever heard of it, but there's this button on your keyboard you may have once seen that literally says the same thing. Um, so it's called the magic ciscue key. Uh, it's a key combination understood by the Linux kernel which allows the user to perform various low-level commands regardless of system state. So you can do things if the if the kernel freezes or current locks up, you can do this magic key combination with that key and basically start debugging your kernel. It's meant for kernel development. U, I'm really surprised that they actually made a

dedicated keyboard key for this. Um, you won't see in your laptop, but it's on like full fulls size keyboards. Um, so how does this work? Uh basically you type uh you know alt sq key and then another key and it does an action based off what that key is. This is a screenshot Wikipedia some there's lots of commands and actions you can have it do but one of the interesting ones you can send a sig kill to the P1 which is a init system um and that will almost always just drop you to a root shell and this device was that enabled so I can just boot it via the serial console um send this key combination and

it just gives you a root shell and that works. Uh but no I was I was accessing this over the serial UART which doesn't have like an actual keyboard. It just sends like UR commands and I was using the screen terminal emulator. So for that it's a little different because there is no the CRQ will get interpreted by your computer not by the computer that you want to pass the over. So you have to use um on the slides is control A control B and then the command key. You don't actually use Q key here. Um that's was way off the first way. Um fact I got the root shell. Uh I haven't actually even needed the passwords yet.

But also in the boot logs u the device we have its IP address. It does boot up and show a it has a tonal local IP address for you to access locally if you need to debug it. Um it happen 1681.1 you can do MAP scan of that and you see it's running a TNET server and HP server. So now if you set like you know the computer you plug directly into it it's IP address like say 1921681.2 you can talk directly to it interact with these services. So I start taking a look at those. The talent interface drops you to the exact same login prompt as UART. Although now we have the passwords here.

So you can log in. Um we get it's a very limited custom CLI. It's not a normal normal shell. There's no shell access but it has a few access to a few normal shell commands like ping. As a matter of network diagnostics um can can we escape this limited shell? Of course the answer is yes. So see this auto plays. Yes. Um, so the ping command allows for a command injection via semicolon. Watch this go in. So like I'm logging in as user user. See all the commands that are available here. So you know I'm attempting can I do sh can I do bash? Nothing nothing's working here. We have access to ping. And this looks just like the normal ping

output that you'd have at your system. It's so I know it's not like interpreting ping. It's actually just calling actual ping binary. You can ping yourself.

can run bash um have myself again but then it drops you do a bash shell and so what's really interesting here though is like this is surprising it actually works with a full anchor interface so you can run interactive tools like vi top um I didn't expect that to work no tab completion unfortunately that did not work but there's empty password this is my favorite part here when you exit it tells you to have a nice day. I love the fact the engineers uh put that in there but didn't fix the command injection. All right, so moving on from that, the web interface uh is also accessible. We can login in I guess the same credentials we have. We just have

the user and support users. Um but it's very limited again what we can do. It's like read only access. Can't see many interesting details here. But when you go like the password change website, it mentions like you know there's a lot more features available as the admin user. Um but that's you know we don't have that password yet so we can't can't can't see what's there but I did find some some of the web interfaces even for the guest user have some very interesting feature on it such as you can enable uh mirror ports which could enable like a local LAN tap again not not remotely but locally it's probably for engineing debugging but one of the

most interesting is you can do a pack capture and this does sh TCP dump so the network you know the network engineer can do a pack capture of you or you know assuming they're trying to debug a session or fix something um which is like actually very privileged for a non-admin user Um and much like the the tnet like interface uh there is also a ping utility. Um it does not support command injection via semicolon but it supports it a different way. Um you if you open it basically in two different browser tabs at the same time and you can send a command and say that by default it just says command equals ping. You can change

it whatever you want. In this case I just did cat proc CPU info and another command tab. You can refresh the output of that. And here's the output of the CPU info page. It's going to architecture so you can see what's what what you're running on. um does require two different requests but you know it's still command execution and remember the web server is running all users but including the web server is running as root so this is root or root command injection here um there's also um now we have the file system dump I can look through like where the uh where the um web directory is being served from and there are a

bunch of pages that aren't linked to these hidden web pages and one of the very interesting ones is dump sis info html which creates like a debug dump of the entire system the again the non-admin user the user user can't access this. We can create the system information dump. Let's go do that. Click the dump. Picks a minute. Open up the file. It's a very large XML file. Um, and it dumps all the system passwords in plain text. Um, we now have the admin passwords. Um, yeah. Um, see anything here? Yeah, there's also dumped another user that wasn't password. Um, it looks like manufacturer user whose password was manufactured password. It also contains other things like, you

know, more all the SIP server configuration and the SIP password too. So, we want to access their control plane. We can have run our own sub service if we want to um if we don't want to use their own T. So speaking of the control plane now I'm actually logging in as root or the actual users on the shell. We can see all the network interfaces on this device. Um see here the main interface the main bridge interface BR0 is the customerf facing internet connection. This has the 12161 interface on it. We can also see an IP address on a VLAN 702 on the GPON fiber interface and the 10.81289 81 1289 interface. This is a huge subnet over

8,000 IP addresses. This is their control plane. We're now on the control plane, similar to what we had in the very first ISP, but no hacking required this time or sorry, some hacking required this time. Um, I was able to confirm with some other friends of mine who had the same ISP that we could ping different devices through this control plane um, behind the which also means that these modems are routing between the customer's network and the control plane um, for certain types of packets. So you can ping parts in the control plane but not talked to that much on all network services like the TN net web server. Um we're listing all interfaces. So those on the control

plane and the users interface which is what I was asking from which is probably intentional. So you know like say the network administrators need administrators need to debug something they can access it via the control plane can access the same web interface or tnet CLI that I was accessing remotely. So this got me thinking does this mean that you know one compromised such as mine uh could on the control plane get code execution on another customers by accessing its debug interfaces over the control plane. Um if so they could literally have a man in the middle like you know someone else on their on their network and lots of great network debug utilities are pre-installed in here like

TCP dump for example. Um so I did end up disclosing this to Sonic because this was you know as a customer I wanted this not to be done to me and I wanted to be fixed. Um they are very receptive and they actually gave me permission for this test. Um that was what I was really after because I don't want to actually do actual crimes. Uh so they gave a test in lab environment to test this. They were also really eager to see if this would work. Fortunately this did not work. Uh their system was configured in a way to block tot communications basically at their router level. So even though it does route uh you can talk to

only infrastructure on the control plane that isn't other customers which is great. But like I mentioned earlier, you can still talk to other things on the control plane like their gateway DNS server and uh the VoIP servers and maybe more. I didn't explore that too much. So as I mentioned for like disclosure, you know, this was a much quicker disclosure timeline at least for disclosing them to acknowledge it and work on a fix. Uh first I want to say that none of these findings were Sonic's fault. Um I initially they're all in the like bugs in the firmware of the hardware device manufactured by Adran who I initially tried to disclose to twice. um they are not very receptive

because I am not a customer of them. I have no relationship with them. So I disclosed to Sonic who then pushed a train to actually release a fix and they did uh they initially pushed the fix just to me. I tested it. I they the fix basically just cut off the TNET and web server interfaces and turned off the UR interface um on the device. I don't know if these interfaces are still enabled on the control plane. I guess they are but I don't know that. But that basically cuts off all user access to that without finding another vulnerability and effectively mitigating all these vulnerabilities which is great. So this is a security success story. Um there were five CVs issued for

these findings um for the for the ANT. Um one other comment I just want to th out there. Um when I was looking through the firmware on this there was a lot of references to other hardware that makes um other types of devices. So I wouldn't be surprised if these same vulnerabilities might be in their other products. I'd hope that you know when they receive these this report both from myself and Sonic that they would look through their other products and make sure that these issues are also solved. But I don't have any other hardware. I have no idea. So some of the like the takeways and impact from this is you know all the

web vulnerabilities I've shown you all here are very like basic stuff you know like simple command injection or really really simple guessable passwords. Um it was very waring to find these but these are like really really low level stuff and this stuff's still out there today affecting lots of people and it's going to affect like your network your business network if this is the type of stuff you're relying on. Um my I'm not trying to say you know everything is bad. I'm trying to say is like these still look at embedded stuff um these rules like knowing how to look at these embedded systems can in combination with these basic vulnerabilities can be very rewarding. You can find a lot of

vulnerabilities and hopefully get them fixed. That's hopey a good success story and uh that's my talk. Uh a version of this will be published at my blog lard.com in a few days hopefully. That's all I think there's questions. You have any questions?