Vigilante: Bringing a nail bat to a gun fight

hello oh can you guys hear me okay or should I okay cool hi yes so my name is also Chuck I have a regular name too and I'm here to give a talk that looks really really aggressive and mean I'm a big fan of like nail bats and zombies and things like that so that's actually the Twitter background picture of my Twitter thing so it was easy to grab and just use for my slide a little bit about me the intelligence is way too high on this special chart I'd probably put myself at a three but my Luck's definitely a 10 I've given this is my fifth talk at b-sides Connecticut which is nuts for perspective there's only been six of

them and my talks have gotten more and more increasingly weird and less related to what I do for work so I think the first time I came here I gave a talk and it was in a polo shirt and it looked nice and it was really cool it's recorded somewhere you can go watch it and then I gave a talk the next time was on hacking mobile networks the epc core basically how to take down the stuff that routes all your cell phone cellular data which was kind of fun and neat but also work-related ish and then the next year I gave an impromptu off-the-record I'm not going to talk about a talk I'm not telling you

I just let's just say if you do something stupid they they like it and then the the last year I gave a two-fer talk because I didn't have enough material for one talk it was about hacking back at Monet our miners drive-by exploits and threat intelligence and then I talked about how I broke into my Audi which is kind of neat so I'm sort of all over the place and now this is like continuing the slide down this year I'm just in a t-shirt and sneakers that have spray foam on them from repair in my house and yeah so let's get weird so I'm not sure if any of you guys who has successfully deployed used tested or

tried out TLS 1.3 or okay like let's do the old fashioned show of hands who knows what TLS 1.3 it's okay good and only like one person in the back over there put a hand up and someone over there it's been a total disaster in terms of rollout IETF RFC all this sort of stuff it was fascinating to sort of watch as someone who's involved in network test and measurement that this thing could be such a fiasco and what was going on there you would repeatedly see it sort of rolled out in the browser's and then quickly taken back or cloud fly return no no no no no no it's bad and you'd sort of start scratching your head and

sort of looking at well why is this happening what's fun is is if you ever are interested you can subscribe to any IETF exchange and instantly you're now a member you're part of the advisory stuff right that's how open they are about this stuff you can sort of sit on the sidelines and watch watch the arguments and bent or go back and forth on their mailing lists and it can be quite entertaining the reality is is that every time they put it out the way the way that Google likes to test things is because their customers are free they just update their browser and they sort of do a a bee analysis they call it

right like I'll turn it on and see how many failures I get and then I'll turn it back off again so then it was sort of finally announced in August last year that it was rolled out but as you saw from the show of hands it still hasn't rolled out why is it not rolled out it's because like I said it breaks everything and what was really interesting was why it broke everything the people pushing this standard wanted to stop people looking at the network traffic going through the network TLS is meant to keep you safe right some of us might call it SSL if you're really old but basically it it's designed to stop you from looking

on the inside so if you go banking or something else all that data is encrypted does a public/private key encryption pair exchanger then share some sort of megalithic awesome key and now we have ephemeral ciphers which make it even tougher to break in so but what's the problem is is if you're a blue team or if you're a network defense person you have policy after enforce you have to stop people going to you porn all day you have to stop people going there known criminal websites or make sure that they're not downloading malware over TLS almost all malware is over some kind of SSL let's encrypt makes it nice and cheap for them to do it right so if

you go look at a blue coat fortigate squid proxies whatever they all have an ability two men in the middle that that network connection and then your IT staff will ship a certificate that says that's okay and your browser doesn't throw up an alert right this is sort of how the submit the work and in America we understand our computers don't belong to us they belong to our company and everyone's okay with that and they then just buy another computer for porn at home there's there's a lot of good reasons for doing this right like you do want to limit access to stuff you want to make sure people are safe it's a headache wouldn't people download

malware and also it's a handy thing to use if you happen to you know own the route signing certificate for calm and have a device that can intercept SSL to to spy on people so the trouble was was that these guys it's somehow magically appeared out of all this design and thought into this network protocol stack that they never even imagined that these devices existed out there or did this and it's mind-blowing but if you read these blogs and stuff they're like we found out about something we're calling middle boxes if you're me you call them a firewall or you know something like that but it's it's it's amazing how disconnected you can be as a developer a huge company

like Google and not understand that this is also happening in the same world right but it really looks like that's sort of what happened so the the security vendors from what I could tell weren't brought into this discussion until way later the guys kept trying to get increasingly clever but the the weirds group they did was at the top of the protocol header it says what the version is and they just put 1.3 assuming that that would just work and all the decryption devices out there sort of assumed that 1.3 would work kind of like 1.2 and it doesn't and so that sort of broke the whole thing and so if you start looking at it from up from my

even bigger point of view these mailboxes don't just do TLS decryption they intercept your IMAP they intercept your pop3 your SMTP communications they have really neat sort of proxy mode slow mode whatever and they if they'll even hold back a data exchange as you're downloading an email for the last file send it a cloud hash it if it's bad it drops that last packet and hopefully your browser doesn't stitch that piece of malware together because there's no bytes padding on the end doesn't do anything the other stuff that middle middle boxes firewalls do is NAT they perform fix up some protocols so if you've ever had a headache where FTP isn't working or voice over IP isn't

working you haven't turned your fix-ups on they rewrite the protocol headers for that squid caches your data so it inserts headers it looks at stuff and that block spam right so then for me and my point of view the question that I got provoked with was can I figure this out can I can I figure out what's between me and somewhere else for a box that's trying its best to be as transparent and go along get along as possible this isn't so much identify as oh it's a firewall but I want to know what specific device that is and how many there are between me and the rest of the world do I have the right to know that

there's a follow-on question and I think the answer to that is yes and more importantly can I do it can I figure out what these things are and how they look so here's a chart for you but basically this is an FTP transaction across a firewall it says FortiGate and the pcap file out there so you can tell that that's a peek now FortiGate device and if you look on the left you'll notice that's a little shorter than the one on the right so now it becomes an eye-watering like visual diff of trying to figure out which command it is that was snuck in here that's not there yell it out if you see it otherwise I'll just

say it right now mm-hmm sighs yeah it's size on the right size is an esoteric weird FTP command that you can issue before you do a retrieve for a file and it'll tell you how big that file is now if you're doing malware interception why is this important why do you want to know how big that file is gonna be because you'll have a separate socket it's just a binary blob of data that comes across at you right for that FTP transaction it has no context to it so you need you need the context of how big it is if you're gonna do malware analysis on it because what it will do is it will snarf up all of the packets

pass them to the customer so the customer sees their download going just fine and then I'll leave those trailing bytes and it will just hash that whole file the last somewhere in the cloud is this malware with md5 there's a reason to use md5 or sha-1 sha-256 or something and then the cloud will come back and say yes this is malware you know terminate that connection then so we have our first test I can do an FTP transaction and if I see on the server side that size request and I didn't send it on the other side I now know I've got a 40 gate between me and somewhere else right here's an easier one so I can't

remember which one this was but what's interesting is is there's no effort to mask any of the HEV headers or anything this little ie friendly error message is built right into the thing to try to navigate around an ie bug for like ie6 where it basically just says if you get a 404 and that's too small it just displays the Internet Explorer it's a bad webpage rather than the page and on the right I was sending a car so I'm testing the AV filtering mechanism here and this is drifting slowly downwards there we go okay so this is easy to tell that the content I requested is not the content that I asked for but that's not

the question I was asking the question was asking is who's actually doing the blocking and what it is so you can see different headers are inserted as long as well as Jesus looks like some other fun things right but what's also interesting is if you enable proxy mode on one of these firewalls but you turn off all detection you turn off all the engines it still flips one little bit changes HEV 101.1 so now I even know if you're trying to sneak something in there and not do anything at all I can still see you which is kind of neat too and then this is what I did last night I just googled how to set up a transparent

squid proxy and you can see squid is super noisy puts a lot of stuff in the requests some of my stuff is also looking at a also looked up a lot of the TLS stuff but that'll make you go crazy trying to see those binary blobs so the question becomes who watches the Watchmen this is actually from a terrible Latin Romance poem but it's been a SERP I Alan Moore and I went and tried to figure out what's the Latin for watching the Watchmen and I got vigilante so I thought that sounds really cool and I'll use that for a name I wrote it in Python because I don't know how to write Python so I figured

it'd be a good opportunity to learn how to write Python it recognizes middleboxes based on the traffic characteristics so you can think of it it's almost sort of like

a procedural programmatic tool kind of like an IPS uses regular expressions to find malicious stuff that's on the network this is trying to find what is watching me right using a similar sort of framework or idea it's currently capable of detecting FortiGate in two different modes if you've ever deployed the FortiGate you know those two different modes make it up or like I like operate like a totally different firewall it also detects Palo Alto because our IT department uses Palo Alto so that's easy to test and as of last night I added a little bit detection in there too it currently can it has test cases for HTTP content filtering AV filtering proxy filter proxy detection TLS and FTP

and yeah there you go content filtering AV filtering and all that so now I'm going to try to do a demo my laptop fan is going nuts because it's a little hot in here or maybe just here so we'll see how well this works out so if I go here I need to establish an SSH tunnel to a Jeff piezo server and this is basically port forwarding three different ports that I'm listening on so that my software just runs locally so now if I run you can see I've even been working on it now oh you guys can't see that oh my god what just yell that out right away guys don't let people dangle out

there when they're presenting it feels terrible alright so this is guy go fast but I added color tech support so it looks cool and amazing and if you look here you can see that it can detect am I getting inspected by a Palo Alto or am i get inspected by squid or whatever I'll scroll back up here so if you look here I'm testing the presidents of proxies it didn't detect any proxies now doing clear-text heg inspection so here's a neat trick because dns is also a dumpster fire when these things are doing a content inspection and trying to stop you going to malicious websites most the time they don't see the DNS traffic your dns

servers over here firewalls here users here better than up here right so instead of using DNS queries to to block stuff they look at the HTTP header they look at the certificate name yes and I filled the whatever like those those sort of Network artifacts instead right so all you have to do is pick a server on the Internet and then issue a get request in the host header that says playboy.com and it will nuke it so that's like the easiest way to test one of these things so what I did was I generated a list of places that I figured you might want to stop people go into playboy.com google comm example CNN BoardGameGeek

they blocked my they blocked my board games at work this is number one reason why I set up a tour proxy oh I mean I just want to find out when Magic the Gathering is draft night is at 6:45 or 6:30 to take my little one - and they block it Oh price-price Clearing House they can block that screw those guys and then it's kind of neat runs a shell script that generates new TLS certificates every time you run it these are fake TLS certificates but they put in the right certificate name sni headers and stuff um and it puts those in there I wrote basically like a fakie a fake TLS handshake II thing that

pretends to do 1.2 which was lots of fun and a huge headache if you've ever done a SN dot one parsing no and then at the bottom here I've got an FTP filter the FTP filter test and the the only one I've got a test for that is the FortiGate right now and then this one doesn't have the sexy syntax highlighting but you can see that the AV filtering was pop down there right so that's on a pan and you can figure that out but then it gets cooler but wait there's more so this is the like Linux box I set up squid with am I still connect to the Machine I am good so hopefully you guys

can see that it's a little smaller I know but this is basically a Linux box it's got a squid server set up and it's forcing all the traffic through that squid and then it's going to go through my VPN at work so now I can see if there's two devices layered on top of each other and who's doing what just kind of neat right um so if you look here at the top it figured out that when I say your browsing is being inspected by squid transparent proxy mostly used by Airlines and cheating it off and then further down here it says that your playboy.com is blocked by the pan so you can actually sort of see who's blocking

me where which i think is really neat so that's the demo part I I think it's kind of cool I don't know and so now we have the necessary concluding slides and things so let me try to explain how it works I won't lie I made this about 20 minutes ago because I realized that there's a lot of Voodoo going on in there you might not recognize but basically you're here you are over here with your computer and what I do is I set up an SSH turn out to an AWS instance the world's tiniest computer I can find and I say do remote port forwarding from there for 80 88 443 and 2121 you can recognize those always like

alternate ports I don't have to do user privileges or anything and then I set up IP tables to forward 84 43 and 21 down to there and then I run test cases one at a time I do comparisons of what I sent what I received on both sides so my test cases the the function absorbs it says here's the original sent here's the original received on the server and here's the original sent by server here's the original received by client it's working so far and I'll just keep adding in new fix ups for it as I find problems I guess the code is growth so I'm not sharing it I really want to test the

blue coat TLS device did manage to find one for a hundred bucks on eBay but I haven't found the time to set it up and probably there's no license on it so boo SNAT detection if you look if you saw it's like I call these test cases layer seven but also one of a source net detection using scape II or some other packet crafting tools you can see the same sort of little network artifacts and identify if you're using pfSense or FortiGate or pan for doing your net they all rotate source IPS differently source ports and things instead of using SSH for port forwarding which is really cool I should probably learn our WebSockets work to make it a little easier to set

up and run and then I don't have to rely on SSH working not being inspected which leads needle last one I want to know if SSH is being intercepted as well because how many times do you just type yes when you're logging into your server and it says new key you know like okay okay I get work done I guess it's right who knows um so why did I make this this is the the obligatory pontification slide I had an itch to scratch I guess I was like I have to deal with this in my work somewhat I have to make these application protocols work for testing across firewalls so I'm already learning how these things all work can I apply

that in some other way also I was reminded of a college roommate who took a map of the Lakes region of northern Minnesota I don't know why he had that on his wall I went to school in Worcester and he scrawled on it giant Sharpie markers the map is not the terrain probably in a fit of madness it was college using imagination so the the point is is like even on my way down here today from New Hampshire knowing what you're going and knowing where you are doesn't tell you what's in front of you right it's not telling you how you're getting there it doesn't tell you where the toll booths are the police are

hiding there's a pothole or you know think break down some of this stuff you can use Waze now right they sort of enhance and they're giving you terrain data now right but it's it's really that it's if I'm driving I can see what's in front of me as I'm going to my destination on the internet you know where you're going but you don't know how you're getting there it's a giant black box of I don't know I figure people have a right to sort of figure that out like what's going on between me and the rest of the world and how could it impact me and I'd like to know if they're doing that so I've got 7 minutes

left if you got questions but basically you can follow me on twitter at noble Trout I'm part of like three random people in Massachusetts that are trying to start a ski slash security convention thing up in the White Mountains this winter it's going to be a total disaster and I'm gonna love every minute of it so if you're interested follow the ski con and e Twitter handle and yeah look forward to think some of you guys there I'm gonna measure there's gonna be a lot of beer and worn out people so it'll be good that's about it thank you [Applause] yeah yes I did I did and I didn't put it in here but yes I did

um so actually I can show you that if you want to see it let me see here I happen to have my private github repository right here here we go so it's it's a bunch of binary blob comparisons but basically if we look down here there you go so this is sort of what you're dealing with basically right what's kind of neat let me see if I can find it here there we go so the pan takes the random number random thing in the TLS handshake and it just decrements the first byte wave 1 so my test string is Fifi Kathy Vika Fifi and becomes befefe Kathy Vika VP so that you know this is the sort of stuff that

I'm trying to do to make it hard I set up because there's a lot of back scatter all my almost all my requests I try to smuggle UID in there and I pattern match on that to know that I'm talking to me otherwise it can get confusing