Mallet — an Intercepting Proxy for Arbitrary Protocols

okay so your last couple of minutes maybe all right so what I'm going to be showing you is a tool that I've been working on for the last year or so that is designed to assist people who want to understand how a client is talking to other systems on the network so it's a demonstration of mallet which is ultimately actually a framework for creating your own proxy's hands up how many of you have used burp or zap or Achilles anybody going back years and years are nice thank you and web scarab one or two all right okay bigotry Who am I slide I work for a company called sense post I'm a researcher I've been

with Saints pose for eight years since Bose was founded in 2001 so we've been around for a while I guess everybody around here probably knows who we are yeah started in rule of Spade room in 2001 and the 14th of February trust hackers not to have anything to do on Valentine's Day okay and if you want to get hold of me that's where you can get old me okay so why do we care about flows across a network the the essential systems that are interesting communicate and those communications happen across a network of some description so a lot of that communication has been done using HTTP and HTTPS because it's as we like to call it the universal firewall

traversal protocol but there are a lot of other protocols as well that that you may encounter so I think what's what's been happening in the past is that when people have run into applications that are using non HTTP or HTTP protocols they run Wireshark and the go me okay maybe it's SSL it's it's fine you know I'm gonna call it good I don't really know what's going on here because I don't have tools to actually do anything with this data so the one thing about Wireshark is that you can see what's going across the network but you can't tamper with it and this is why things like the intercepting proxies have been so powerful for reviewing web

applications but in many cases if you can just get into the stream of the communications between the clients and the server you can tamper with it in many in many cases in very similar ways as you would with an HTTP or a web application okay and the other thing I just wanted to mention there is that in many cases when you start looking at non HTTP applications it very often takes you right back to the 90s where we saw things like shopping item prices in hidden form fields in shopping carts and that sort of thing that you could just tamper with set the item to be one cent and check out because you know nobody's

actually looked at these protocols in any detail simply because tools haven't really been available to do that okay so I just want to point out some of the important differences between some of the things that make HTTP a more easy protocol to deal with and then contrast that to some of the the complications that may come up in in other protocols so on the one hand HTTP is a nice orderly request response based protocol you send a request you get a response and you're done it's stateless which means that you can establish a TCP connection send the request get your response disconnect and open a new connection and send the next request so you don't

actually maintain any significant State on a particular TCP connection other than if you're doing into your lane but that's not really important because you can just disconnect and reconnect and that's that's real that can be ignored for the purposes of an intercepting proxy contrast that to something a little bit less orderly for example OpenSSH who talks first when you establish connection to an SSH server does anybody have an idea does the client talk first or does the server talk first do you get a banner from the server or do you send your request in fact they both talk simultaneously the client doesn't wait for a banner it starts talking immediately that it connects and the server sends its banner

and waits for the client to actually send its own piece of information so it's a simultaneous connection or simultaneous talking kind of protocol just to illustrate that other protocols can be a lot more messy than our nice orderly HTTP that we're all familiar with other things that we need to worry about are things like timeouts how many of you have intercepted a request in burp you know shut down your laptop gone home for the weekend come back on Monday morning opened up your blood your laptop gone uh-oh here's this request that I had intercepted send it on or forward the request and you get a response no issue so burp abstracts things like timeouts and disconnections

from the network in a very convenient way other protocols may not be that forgiving one thing in particular that you have to worry about is timeouts so if you send a request to an HTTP server in fact and you start typing really slowly or you just open a TCP connection after a few seconds that connections going to disconnect again and that's because the server has been footed with some protection against an attack called slowloris which is a way of denying service to an application you open up lots of TCP connections and eventually you can exhaust all the resources of their of the HTTP server so if you don't send data it will disconnect but you don't see any of this

in burp because you track their connection and you can hold that as long as you like and then send it on and it's just transparent and this once again is because HTTP is nice and orderly and you can disconnect and the requests are stateless the timeouts happens on the server end but burp hasn't actually made a connection to the server and told you forward that request and get the response back contrast that to a stateless protocol where you have to maintain your connection to the servers continuously and if you don't respond fast enough the server's going to disconnect you so these things can be a lot more difficult to deal with and particularly in malat i have no sort of

magic bullet to address timeouts unless you can find things like a sort of a keepalive message that you can send but one thing that mailer does offer you is the ability to script the changes that you would like to do so that it's not a manual process ok so really just to show that there is a lot of complexity in dealing with protocols other than HTTP have any anyone here had to deal with a protocol work with an application that was using a non HTTP protocol can you give me an idea what the protocol was AMQP Queuing protocol ok you're trying to review the security of it so just see how it worked ok all right

sure so I think for a lot of people they the solution to dealing with custom protocols is to write your own testbed you'll you know hack it up in whatever language you're most comfortable with it becomes a one soft solution suited for that particular protocol and probably that like one particular environment and it doesn't become a more general solution to the problem come on forgot to turn off my screen saver okay so a lot of solutions of it I've seen have involved people using so cat you script up something with so cat running as a server and you pipe that through a bit of you know said or or or core whatever you're going to use to make

your manipulations and then back into so get to to make its ongoing connection and again these are very point solutions that are mostly uncomfortable to use and so what I was trying to do with mallet was to provide the infrastructure that is required for all intercepting proxies and allow you to focus purely on the the protocol itself so there is prior work I wrote OS proxy this is obviously a problem that I've been thinking about for a long time I was proxy I started in 2007 but gave up on that fairly quickly when I realized that I was building a programming framework which doesn't make it easy for a tester to actually just get on with dealing with the protocol

itself Martin Hollander wrote etiquette proxy based on Ross proxy didn't get very much traction again unfortunately then there's Mallory Mallory is probably the most famous of the intercepting proxies it was released in 2010 and I think it's last update was in 2011 from that I'm assuming that it wasn't particularly popular it didn't get very much take-up and I think one of the reasons for that is the complexity how difficult it actually is to get started with it so Mallory is shipped as a virtual machine which means that in order to get traffic into Mallory you've got to deal with iptables rules and routing configurations and you know it's just a it's a lot of overhead to

figure out whether you're actually going to be able to use a tool or not I think a lot of people go yeah I'm not even going to look at that now and there's a bunch of others all of these ones with the slashes in our github repos so probably the most famous of the remaining ones that I haven't talked about yet is beta cap so I think better cap is really good at very low level protocol so there are poisoning and those sorts of things but in terms of intercepting arbitrary protocols the only support that beta cap actually has is for HTTP and HTTPS so we kind of right back where we started we don't

actually have a solution for non HTTP protocols so to address the issues that I saw I wanted to make Mallett as easy as possible to get started with in contrast to the the confusion with routines and routing tables and IP tables and so on and then the other thing that I wanted to do was avoid reinventing the wheel and doing it badly and I think that's another reason why tools like Malory didn't get any traction is because they had very limited protocol support so that if you wanted to support something that wasn't on the list of things that they already provided to you you had a fair amount of work to do and it was using a sort of

custom framework that they had developed so as I mentioned one of the problems that I ran into when I was writing OS proxy is that I was building a programming framework that would have been very special purpose you know the only people who would have known how to use it would have been you know I would have had to support every single person who comes in trying to write their own protocol decoder and then I ran into Nettie and I was inspired because the way that it works lines up very nicely with the way I wanted to be able to to make this available to users so Nettie for those of you who are not familiar with it is a

Java network application programming framework designed for buzzword compliant a rapid development of maintainable high performance protocol servers and clients so they do have HTTP but they've got a vast wealth of other protocols that they also support and more importantly they have a large community of developers who are creating their own protocol implementations and would be in a position to assist anybody who wants to know how to create their own protocol decoders and encoders using the native framework and just for for illustration the lead developer of natty works for Apple my understanding is that they ship the natives used to ship in order of four petabytes of data a day using the native framework so it's very

high performance and they've thought about its architecture in a lot of detail and that actually makes it really really nice for my purposes so this just illustrates a sort of brief overview of the native architecture at the bottom layer they've got their byte buffer which is their abstraction of a byte array an extensible event model which we're going to be using to manage the connections and the messages going backwards and forwards across the connections and then a bunch of different protocols that they've got implemented okay so a core abstraction for the native framework is their channel pipeline and the channel pipeline is basically a holder for a number of channel handlers and the channel handlers basically implement

your protocols so you'll have data coming in from the network that's a socket read level it hits the first inbound handler who might process that data in some way or update two states or something along those lines forward information on to the next Handler and so on and so forth it may also choose to write data out to the networkers in response so for example this might be an SSL Handler so it receives encrypted data and does that sort of handshaking with the SSL server and then it forwards clear text data on up the channel and eventually it's going to get to the top of the channel generally that would be an application that is implementing your business logic

and that's going to then respond back down the channel layering on the different protocol modifications as it goes so how do you turn this into a proxy very simple for our business logic all we do is we implement a relay so it receives bytes coming in or messages coming out the top of the pipeline and copies them into the next outgoing pipeline and any messages coming up in the other direction get copied across very simple but very effective and of course our relay then has the opportunity to provide information about those messages and about the the data going through to a user interface that that we can then interact with those those messages so I've mentioned one of

the problems with most of the the intercepting proxies being that you have to deal with IP tables and routing tables and so on I wanted to avoid that using you know it's something that's a lot easier and more familiar for people to get started with and that led me to the socks proxy Sox protocol so the Sox protocol is a fairly well supported way of establishing TCP connections through a Bastion host so a an upstream gateway but of course it doesn't have to be very far upstream it can also be running on our local machine and again giving us that ability to intercept the data and and interact with it and so the simplest or

the sort of this is the an example of the standard pipeline that Mallett will start up with so at the top we have a listener and in this case it's a Java in IO server Soccer Channel and nature supports a bunch of different kinds of transports for example UDP serial connections SCTP and so on once the data comes in it will be passed over to a Sox initializer and that's really just a sort of a wrapper for a few different protocol handlers that negotiates the Sox protocol with the caller and really what that's doing is its allowing that caller to say I would like to connect to this IP address on this port and from

there we can then pass that on through the pipeline and make that outbound connection to that to that destination and then once that's done the sox initializer essentially just gets out of the way the intercept handler is a like it sounds it's a way of intercepting the messages that are going through the pipeline and it presents those in the user interface the relay just does the copying across from one pipeline to the to another and you can see the different colors here the blue color indicates the server side of the connection and the red color indicates the the client side of the connection that's the outbound or outgoing direction so you may wonder why they are to intercept handlers the

reason is sometimes for debugging purposes it's handy to see that the messages that you're writing or that you're receiving are actually being written out to the server and that the messages that the server is sending are actually being written back out to the client so it's handy to see both sides of the connections so in a way it gives you the sort of debugging view of it which is quite useful so this is the sort of simplified version of a native channel handler showing really the methods that you may want to implement in your own protocol handlers or just in a sort of scripting fashion if you were looking to do that so channel active you might override if you

wanted to set up some initial action on a on your protocol handler being added to the pipeline channel Reed gets invoked whenever a message comes in from the network and you've got the message object that you can interact with channel right similarly you is called when a handler is writing a message back down that pipeline and again you've got that message object that you can interact with and then user event triggered can sometimes be useful for putting logging messages just into that connection data or just letting you know what's happened so here is a very simple Handler or scripting handler well it's a it's a handler it can be implemented as a script in this case it is because it's

got the sort of return new channel duplex handler at the top which is just sort of scripting mechanism but what it's doing is it's looking for inbound messages checking to see if it's a text WebSocket frame extracting the text of that turning it into an uppercase version and putting a new text WebSocket a frame with the uppercase text back into the pipeline so not particularly useful but it demonstrates that in a few lines of code you can make fairly your advanced changes to two messages that are passing through the pipeline and you can discriminate based on the type of message that it is however it's been so I think that's enough talking what I'd like to do is show a demonstration

and I didn't bring any children with me so I can't sacrifice them and fly CFA wouldn't let me bring any chickens on the wing okay let's see

that is a very strange thing that VMware does it doesn't lock your screen if your masters inside your VMware very odd okay

if I can set this up as a single monitor because otherwise I'm going to be breaking my neck to see what's going on

aha

okay so mallet is packaged as a maven project so if you clone the repo simple enough just to run our maven oops maven clean package and it'll build it relatively quickly after downloading a few files you can then just run it as a jar and it will show you something that looks like this so in the big white area you've got your standard pipeline which is that Sox initializer and it will just forward by its across and you can start seeing what their traffic looks like and this is very analogous to looking at it with Wireshark you can see the bytes going across but in this case now you can actually start tampering with them

as well should you choose to and as I mentioned in order to get traffic to go through mallet you would need to sock safai it so Sox is the Sox implementation is relatively simple to do a lot of programs already have built-in support for using a socks proxy but if they don't you can also use a wrapper program like Sox afire proxy cap T Sox there are a bunch of apps on Windows I think sorry I mentioned proxy cap is also available on Windows that you can use to specify you know which particular executable you want to intercept so that's a nice way of getting Windows traffic into mallets as well even if it doesn't have native Sox

support on the left hand side in the green tabs you've got a bunch of sort of predefined protocol handlers that are fairly common in the basic section you've got sort of mallet specific things such as listeners the intercept handlers and the relays and they're your sort of Sox proxies and such like Sox handlers on the protocols tab you've got some sort of fairly simple things like SSL servers and SSL clients HTTP servers HTTP clients string decoders JSON decoders and so forth and these are just a variety of different protocols that I've run into and needed to support directly so I've added a shortcut to them but it's simple enough to add your own protocol support using the default

handler if you just drag that let's make a little bit bigger if you drag a handler into the user interface all it's really containing is the class name of the handler so you can change that to a different class and you can specify any parameters that are required for that particular instance constructor parameters for that for that class name okay so what I'm going to demonstrate is a simple client-server application that is packaged with mallet just for experimentation purposes so it's a JSON based protocol that receives your first name so name and date of birth sends it off to a server and gets a response back with your age so if you go to the

connections tab and select the first connection it collects a list of all the events that have passed through including things like the connection establishment and so on but the important ones that you're going to be interested in are probably things like user event triggered which is this this is reported by the sock server so trying to connect to this particular address and then things like the channel read and write events so here we have a channel read you probably can't see it too well but there is an open curly brace and if you go down this is the channel right on the client side again showing that open curly brace going art and then incoming

a line of text birth year 1972 art going the same etcetera etcetera so this is the list of individual sort of chunks of message that we received on the network now looking at it it looks again like this is a JSON message but it's not particularly easy to work with because it's fragmented in a number of different pieces so before we can really start working with us what we really want to try and do is defragment this message and an easy way to do that given that this is a JSON object is to match opening and closing braces so we can just count up when we get an opening brace and count down really when we get

a closing brace and when we get back down to zero we can pass on that completed message so let's create a graph that does that or let's open a graph that does that and these are all in the examples so if you want to play around with this in your own time the client and the server is in the mail it's JA very simple to play with

all right so let's go back to the graph editor and see what's changed and really all I've done is I've added this JSON object decoder now this isn't a piece of code that I wrote myself this is native native functionality I didn't have to think about this I just went and looked for natty JSON protocol support this as I'll show you is an natty handler that is existing so I just dropped it onto the the line in between the Sox initializer and my intercept Handler let's go back to the client and send another message so we have a second connection but instead of having a bunch of different messages read there are a bunch of read events for each of those

things but at the end of it we have a single aggregated message showing our entire JSON object so it's important to be able to to aggregate messages into a single logical thing because you don't really want to be working with just an open brace okay so this is all very well we've now managed to figure out where our message starts and ends but you don't really want to be fiddling with bytes when you're dealing with a text-based protocol like JSON so let's go back to our graph editor and open our next example

all right so what we've done here is added a two new channel handlers a string decoder and a string encoder and what these are going to do is they're going to take bytes in and they're going to emit a string object a Java string object and in the outbound direction it's going to take a string coming in and or a string going archerd I say and emit a sequence of bytes and we've parameterize this by telling it to treat them as utf-8 strings because once again this is an 80 handler there's not my own code this is built in native functionality that supports a variety of different character sets all right once more so now we have a new connection but

we can see that rather than a sequence of bytes we've now translated it into something that's a bit more useable so we've now got a text a string object that we can edit and while we could have edited it as the bytes the the bite the bite viewer does allow you to modify those bytes as well if you intercept the messages as I said it's not that friendly so at this point I'm going to demonstrate that you can now intercept these messages and and more and modify them alright so instead of an immediate response we now see that we have a few events which have been caught and held I'm gonna send the initial connect on

and that starts doing its protocol handling and here is a message which has been sent by the client fubar bears and I can now send that on and I'm going to turn off interceptor because it does generate lots of different messages but you can see that the message that was sent out was bears rather than foo and if I go to my server instance it received a lost name of bears instead of bar okay sorry they whoever's parked on the sports field that you need to move your car sorry real quick lots of people are parked there it seems ok alright so the the next step using a string is all very well but maybe you would actually rather

pause this into something that is a bit more useful to work with something like a structured JSON object so in this case I've created a script handler which is going to make use of the jackson library jackson java json api and basically what I'm doing is I'm making use of I'm extending existing native functionality and implementing an encode and decode method my decode method is going to take in a byte buff and emit JSON node objects and yep that's just a couple of lines to do that and then similarly in the end code I'm going to take in adjacent node and emit it as bytes again

sending the message to the server and now you can see that instead of a string we've now got a structured object and this introduces another feature of mallet which is the ability to edit arbitrary Java objects using reflection so a reflection just allows you to interrogate the properties of an object and to view its fields and to drill down into those so you don't have to create a special editor for every kind of data type although it can make your life a little bit easier in some cases ok so that's not intercepted so I can tamper with it but let's run that again

so for example if I change that to 1962 I can send that on and it thinks I'm not 56 years old okay so this is now showing that you can interact with all sorts of different kinds of objects mallet doesn't actually care what type of objects you passing across so long as it knows in some way or you tell it how to convert that object back to a stream of bytes and really this is what understanding a protocol comes down to you understand how to make sense of a sequence of bytes turn it into something usable and then turn that back into a sequence of bytes in some way or other I added them it's been since I've had my

twins okay so the one thing I like to point out now though is that there's this checksum invalid message that that's coming up when I temper with a message and if you look at the message itself there is the checksum field and [Applause] after doing a bit of reverse engineering it looks like this is an md5 sum of the first name surname and yeah so that's protocol understanding 101 eventually okay so in order to allow us to make modifications and send them on to the server without the server realizing that this message is now inconsistent we're going to have to make sure to update this checksum field and it may be that you're making some changes to a

different type of object that doesn't have a checksum but it has other fields that refer to the thing that you've changed and they need to remain consistent in some way so let's go back to our last graph

and I've added another script handler on the outbound pipeline that will recalculate the checksum for me so instead of me having to go to my shell and pass these things through md5sum I have a script which will do this for me automatically on every single message so on a write event if it is a JSON object it will get the name surname and here calculate the md5 sum of that and then it will update the node to put that md5 sum in the right place so one last time I need to intercept this

and it returns an answer but more importantly the server doesn't care but can't tell that I've been tampering with it okay so I hope you can see the progression from starting to understand what the protocol looks like what the data that you're seeing goes through just by its underline then we start saying okay well it's JSON so we aggregate that into into a JSON object make it something a bit more useful to deal with and now starting to automate some of the changes that we might want to make to those objects and and how relatively easy this is to do okay I've got a few other demonstrations that I'd like to show you to illustrate that

Mallett can actually do some fairly complex things so this is illustrating two sides of of Mallett so on the left hand side there is an HTTP pipeline and on the right hand side there is a UDP pipeline what's going on just to work sort of down the the graph we get our Sox connection coming in it looks to see whether it is HTTP whether it is SSL or not if it is SSL it adds an SSL server handler Mallett will automatically generate certificates for you signed by a CA much like Burke does and then negotiate that connection for you any bytes that come through it'll make a another check to see if it looks like an

HTTP request so if the first few bites look like a gate head post request and we'll go down the HTTP branch or else it will take the non HTTP branch if it is an HTTP request and you can see the same sorts of things happening down this HTTP branch for the non the sole option as well so for this HTTP we'll use an HTTP server codec so it reads in a get request for example and knows how to encode an HTTP 200 response back into bytes because Mallett because Nettie is designed for high performance it actually deals in HTTP requests as chunks so you've got the request header and then chunks of message content we

don't really want to deal with chunks of content so we have an aggregator that just pulls those content chunks back into a single message further down we have a special handler that I wrote to handle the upgrade process from an HTTP request from an HTTP protocol to the WebSockets protocol so WebSockets is a protocol that runs on top of HTTP but really in fact what's happening is that the endpoints tell HTTP to get out of the way and convert that connection just to a pure TCP connection without the overhead of the HTTP messages and so that's what that WebSocket upgrade Handler is doing it's listening for the successful upgrade message and then removing those HTTP handlers and then I

have a script handler which will make WebSocket frames uppercase so just to do a couple of quick demonstrations because I know I'm running out of time WS cat is a WebSockets client so I'm running that network disconnected

disconnected all right there we need to akin problems so right what I would demonstrate there is really that if you send that WebSocket message it gets uppercase and the response comes back in uppercase there we go

normally if you run it without going through the proxy you just get back exactly what you sent the other one I wanted to demonstrate was a UDP pipeline so in this case down on the right hand side we have a UDP listener that is establishing a DNS pipeline so it's processing DNS data gram DNS query decoders bytes get decoded come out as a DNS message intercepted and then there's a script handler that's really just going to replace any queries for google comm with a query for sense post comm now for dns it's not quite so easy Sox does support UDP but it's not quite as simple as just proxy changing it so you can see I did a big redirecting it

through port 105 3 which is where my UDP listener is for google.com but I got back an answer of saints post calm in order to do so I had to write a little bit of a handler that said if I see this message change the change the text from google.com to sense most calm but I also had to modify the response to make sure that it matched up because otherwise the dig client would say hey I'm getting a response for Saints Post while I asked for Google this is an invalid answer so mallet doesn't stop you from having to understand the protocol details what it does stop you from having to do is reimplemented a sand a lot of the the

decoders and encoders that convert messages to bites and bites to messages think I'm out of time any questions one one question so an encryption protocol would be a way of starting to encrypt this starting to prevent this much like we see SSL is intercepted well using a burp for HTTP if I control the client I can tell it to accept my certificate authority so that sort of thing doesn't necessarily solve the problem how you would prevent this is using robust check checks and validations on your on your server to make sure that your client is actually sending you correct data and the point of Mallett is to allow you to make sure that those things are

happening