Attacking Apache Reverse Proxy

what I do on a day-to-day basis is create vulnerability signatures to detect uh vulnerabilities on operating systems third party applications and stuff like that uh a few months back I was uh working on a uh interesting Apache reverse proxy vulnerability and um uh I was trying to create a signature for that particular issue and what happened was that once I created the signature um I I was just curious to look at the patch to see what was happening how they patch the issue and while reviewing the patch I figured out a way to bypass that patch and ended up discovering a variant of the vulnerability so uh in this talk I'm going to walk you through the entire

process of um how I um wrote the signature discovered the um looked at the patch and then uh discovered the new variant of the vulnerability so let's get started uh what should you expect from the talk we are going to be doing four things today first uh we're going to take a look at an older uh Apache reverse proxy issue then we are going to be looking at the patch to see how that issue was patched three we are going to create a proof of concept on the fly to see how uh to uh learn how to create new exploits and then finally uh once you've created the exploit we're going to try and uh

exploit the issue okay so uh since we are on a mission of uh finding a vulnerability in apache's Reverse proxy configuration first let's understand what proxies and reverse proxies are I'm sure all of us have used uh proxies to access the internet uh what we basically do is configure the proxy through our browser so uh what is happening is all your HTTP requests are uh passing through the proxy and um how the entire thing works is that a client makes a request request the proxy intercepts that request the proxy would forward the request to the server it would get a response back from the server and give it back to the client so a regular proxy is acting on

behalf of the client a reverse proxy on the other hand it acts on behalf of the server the basic steps are still the same you make a request uh the proxy would get the request from the client it would pass it on to the server and then what whatever response it gets back from the server it passes it back to the client but um what we're trying to get here is that a proxy is basically acting on behalf of the client while a reverse proxy is acting on behalf of the server so in case of a reverse proxy the client has absolutely no idea that the pages that are being rendered are from a reverse proxy to the client it's U just

like any other web server rendering the uh cont content back so let's see uh how this works in a typical uh scenario so here we have a company a actually just a second let me get the quicker

around sorry about that okay so here we have uh a company a it has a public uh IP address and uh a public domain at www.ws.com uh it has a registered DNS entry so whenever someone is trying to access this particular URL uh they're trying to access it from the internet so it's publicly accessible from the internet

now I think this is not working all right uh so we already uh we also have a couple of application servers owned by company a both of these application servers uh are behind the firewall and within the DMC so um the first application server has a internal uh web address at internal one. po.com and the other one has a web address called internal 2. po.com now both of these application servers have uh private IP addresses and the DNS entries are not registered so someone trying to access internal 1. quality.com and internal 2. qu. from the Internet is going to get a no such host error because these application servers are only accessible from within the

company Network they're not accessible from the internet now uh let's say the company decides that they want to enable web access to internal 1. wal.com and and internal 2. wal.com but enabling web access to these servers directly from the Internet is a bit dicey so what the company decides is that it should be integrated with the main web server address but a reverse proxy scenario would come into picture so basically now anyone from the Internet is able to access internal 1. equus.com and internal 2.com now how does that happen um they set up a reverse proxy and how internally the reverse proxy would work is that anytime a request is made toes.com onepath one the reverse proxy rules

would kick in and it would be redirected to internal one. qu. cpath one so what is happening here is that all requests to app one are being routed to internal 1. cal.com and then all requests to app 2/ paath 2 are being routed to internal 2. calls.com so this is how a typical reverse proxy scenario would look like in any uh company's U um environment so uh let's see how reverse proxying can be set up with Apache now um Apache is the most popular web server out there uh and uh it's quite easy to set up reverse tring with Apache uh there are a couple of things that can be done there is a module called modore

proxy which you can either uh build when Apache is configured or if that doesn't happen you could enable your Apache configuration file which is http.com for Apache 2.on and enable the module at run time I won't go into specifics of how exactly that could be done but it's a very simple process you just need to enable uh mod proxy when your uh Apache is being configured once Apache has been configured for reverse proxying there can be two things which are possible there are directives which you can configure um the first directive is called uh rewrite Rule and the next directive is called proxy pass match the reason we are looking at both of these directives is that when we look at the

vulnerability uh it exploits the way these directives work and the way these directives are written and the rules are written based of these directives so let's see what these directives do uh the first directive is called rewrite Rule and uh these are both configured in your Apache configuration file um the first thing that is there is the directive name the second thing that is there here is the regular expression that is to be matched so if you remember in the first slide we saw that any request from app one would be mapped to internal 1. qu. so here is the regular expression that you specify that whenever requests are coming to a particular web server how you want them

to be uh rerouted to your internal proxy so you have the regular expression and then finally you have the rewrite Rule now you might wonder what the P flag here is for this particular directive that is just a way of configuring reverse proxying with Apache so it's just a syntax thing uh that is there the other directive that is there is called proxy pass match uh it has the same syntax you have the regular expression and then you have the rule uh to where the redirection should happen so uh let's see how this works uh in regards to uh the first uh figure that we saw so I have the rule here um the the syntax is right

here uh now what would happen is that whenever you have a request to qu. sla1 coming in uh it's going to try and match the regular expression so um Here app one is going to match this particular regular expression we have a very generic regular expression here uh so I'm just using a simple dot star to see what's happening so whenever this request comes in uh app one will be matched with this particular regular expression and then what happens next is that it goes through the reverse proxy and then whatever is matched here gets extracted in the reverse proxy rule so if you notice you have a dollar one here uh the way regular expression matching

and extraction would work is that anything within the first parentheses would be extracted as dollar one uh most of the companies have a lot complex regular Expressions so there would be multiple levels of parentheses so if you have some multiple parentheses the first one the first variable to be extracted would be dollar one the second one to be extracted would be dollar two and so and so forth now what happens here is that we already know that app one was matched here so app one gets assigned to the variable dollar one and your final request is internal 1. qualis /a one so this is how a rewrite rule would work same thing happens with the proy

pass Mash rule uh is the same thing uh here what is happening is that we have a slightly more complex uh uh URL uh f/b bar. JF again matches with uh this particular regular expression which is everything uhar followed bych and then uh ending there so this matches here so this particular thing is going to be assigned to dollar one and our final uh variable a final reorder is going to be internal one. pool.com fbar doj because that is what dollar one is assigned to all right so uh these are all the basics that we needed to uh understand uh what will be needed what a reverse proxy is how a reverse proxy can be

configured with Apache and what are the different directives now uh let's start with the first thing that I spoke about uh during the start of the talk we are going to look at an older reverse proxy issue and um try to see what what can be done if you were to write a check to detect that particular vulnerability and uh what would happen next so uh this is the older reverse proxy issue that we're going to be looking at uh it's it has a CV uh 2011 336 8 and if you uh read through the CV what it says that the mod proxy module in aache HTTP server so I mentioned that to configure reverse

proxying for Apache you need a module called modore proxy uh it says and then it lists the version of Apache that are affected and it mentions that if it does not properly interact with the use of reite rule and the proxy pass match pattern then there is a possibility to allow remote attackers to send requests to the internet servers via a malformed URI all right so what we know uh so far is that if you have incorrectly configured rewrite rules or proxy pass match directives then there is a possibility to remotely access your internal uh systems and internal uh URL addresses and it also mentioned something about a malformed URI uh containing an atate character so uh

we'll see uh how that works now I have a couple of samples here I have a correct Rule and I have an incorrect Rule now uh I mentioned earlier that U you have a couple of things you have the proxy pass match Rule and you have the rewrite Rule now both these rules are not required at the same time in your Apache configuration file you just need one rule so the vulnerable is in both the rules and it works the same way but I'm just going to be looking at the rewrite rule but it's exactly same the proxy pass match would uh work the same way so uh could you guys tell me what is I have a correct

rule here and an incorrect rule uh can you tell me the difference between the two rules yeah please the slash is missing so it'sing right to the end of the U or the URL that's in there exactly so the only difference between the correct and the incorrect rule is that in the incorrect rule once the uh the URL ends there is no forward slash here right now this is a very small difference it could be just because of an administrator mistake or the rules that we're looking here are quite simple because it's it's for an example purpose but in a real life scenario uh your actual regular expressions are very complex you might have a lot of

application servers like something rerouting to Wordpress something rerouting to some other application so this expression here is complex and then the redirect URL is also quite long so there is a high possibility that someone might just miss this particular forward slash all right now we know that uh what the difference between a correct and an incorrect rule is now the vulnerability states that if your rule is incorrectly configured and attacker can access systems on the uh internal network uh let's see how um now I have uh I configured a vulnerable Target and on that vulnerable Target I had the Apache configuration file to have the incorrect rule all right now uh this cve already had uh

proof of Concepts available uh publicly so uh what I did was just for the sake of creating the signature creating the check to see how uh how to write a signature I was just sending those already available proof of Concepts to a vulnerable Target so um let's see what happens um what I'm doing here is I'm connecting via netcat to this uh vulnerable aparta reverse proxy configuration and I'm sending it a request uh get atate Local Host now um I'm trying to see how vulnerable server responses uh the response back from the vulnerable server is that it comes back with an HTTP error code of 200 okay and then you see that the entire content of the

website that was running at Local Host is being returned so what happened here is that you send the request get at theate Local Host now this particular rewrite rule because uh this server is set up as an Apache reverse proxy um it's going to try and match the URI to our regular expression so this particular URI at theate Local Host would match this regular expression and then as we spoke about it's going to be assigned to dollar one right so your final request is internal one.all us.com at the rate Local Host now can you guys tell me uh why the web server returned 200 okay and why it gave you the content of the website running at

Local Host yeah sign part of the exactly so the at theate sign here was appended here now since we had the forward slash missing our request was internal 1. qualis docomo host so in the context of the URL when you're seeing that what happens is that you view it as user at theate Local Host so because of that crafted at theate uh sign Local Host became the main domain and whatever was preceding Local Host became the username of that Local Host right so that's why it returned a 200 okay Response Code uh there was another uh the next thing that uh uh the next proof of concept that I tried was get at theate QQQ dod. qu. now this particular U web

address is any internal uh website address on your internal Network so this is something that you cannot access from the internet it is only uh visible on your internal networks but when you route the same thing through your Apache reverse proxy the results are that you would get a 200 okay response code and it would come back with the uh entire page of the of whatever web server address you requested all right okay so um now we still have the proof of concept uh we know that how a vulnerable Target with a incorrect rule response back let's see what happens if we correct the rule back and the target is still vulnerable uh we're sending the same

request to our vulnerable host and when you send the request get at theate Local Host it comes back with a 404 not found Response Code and the same thing happens with whenever you request an internal web page uh now the request is still the same but you don't get the actual page you don't get through uh the content of Local Host nor do you get the content of qq. dev. pais.com and the reason is uh because the rule is correctly configured so consider the same thing at theate Local Host gets extracted and it gets assigned value here so your final request is internal one.all us.com localhost so obviously the server is not going to be able to find that

particular um host and it's going to give a 404 not found Response Code so we see here that the difference is very small just because the the rule was incorrectly configured someone was able to get through the internal um system all right um now let's see what happens if you have a patch Target and uh the rule is still incorrect so uh the CV that we looked at um Apache released a patch for that and uh I set up a patch Target and I'm sending uh the request to the patch Target with an incorrect rule so uh the this is our third scenario um what happens is that when you send the request um the server responds with a

400 bad request error code now earlier it was responding with a 200 uh okay Response Code right so um once the patch was applied Apache was able to um figure out that uh the request was crafted and malformed so it gave a 400 bad request error code all right so what do we know so far far we looked at three different scenarios uh for the existing reverse proxy issue we tried the proof of concept on a vulnerable Target with an incorrect Rule and we know that it came back with a 200 okay Response Code we tried it on a uh vulnerable Target with a correct Rule and we know that it came back with a 404 not found and then

finally we tried it on a nonvulnerable I mean a patch Target and it came back with a 400 bad request error code so when I was doing uh my research this was good enough for me to create a check because I knew that how a vulnerable and a patch Target would respond back and then uh I could write a signature based on that so um that was great but then what next uh writing the signature to check the issue uh was the easy part and it did not take a lot of time but because I had spent so much much time uh investigating the issue and then creating the targets and figuring out uh what would happen if uh things change I

thought that I should pursue this further and do something else with it so I thought that uh for someone to check this particular issue they would have to launch a scan and then uh figure out how about if I could just write an open- Source tool that administrators could run against your Apache conf ation file and then it would tell them that your re your rewrite Rule and your proxy pass match rules are not correctly configured so I thought that okay uh um that's a noble thought maybe uh I could just make the tool open source and then people could just um see if or whether or not their rules are correctly or incorrectly configured so I started working on that

I spent a couple of days um trying to get the best Apache configuration file readers that were out there and then then uh when I started to write the regular Expressions I figured that real life regular expressions are going to be so complex um that there might not be a shortshot way of completely getting uh the parcel rules correct so um those were my uh limitations and I thought that okay this is not a very good option maybe I should uh look into something else so um a lot of researchers and hackers they actually look at the patch to see uh if there are ways to bypass it or how it was patched to see if there's

anything else that could be done with it so um this idea uh was attractive because when I was creating a patch Target uh I realized that Apache only had like 10 lines of code and um for me it's it's always uh I was sort of curious that okay within 10 lines of something it was patched and it was correctly uh uh configured so I thought okay let me just look at the patch to see uh what is happening so uh that's what I did so for the reverse proxy issue that we looked at uh this is the patch that Apache initially released and um when I configured the patch Target I just had to modify uh this particular file uh

called protocol. C and then uh recompile and uh reconfigure Apache so if you look at the patch here it's just 10 to 12 lines of code and it's just one uh if uh loop that is there and then it's returning and a bad request error code so if you look at the patch here what it's doing is that it is checking for uh some method underscore number variable and if it is not set to mcore connect and if it is not a pared URI and the URI does not begin with a forward slash and it is not a star or not a null then it would give a bad request code and return back so um the first thing that came to

my mind is that okay uh this seems correct because what they're trying to do is that they're trying to see if your crafted URI is beginning with a forward slash or not and if not it was just bailing out so I thought that okay it feels as if it's doing the correct thing but to confirm my understanding I just started reading the comments that preceded the actual uh patch so if you look at the comments it's it mentions the same thing some mention of an RFC and then it says that if uh if the request is not using connect the part s does not have a scheme and it does not begin with a forward slash and it is not

star then fail and give a 400 Response Code now this is the same thing that we thought about when we looked at the patch right uh so you feel that it's doing the right thing uh let's analyze the patch and see uh before we do that uh can any of you uh tell me if there's anything striking in the patch that you feel that you could look at it and think that you could bypass

it I'm sorry this one so both those things are true that's not

okay so that could be a possibility uh I did not look at that uh anything

else the you are

right uh that that could also be a possibility but I I still did not so the if the if Loop here as four different things and um when I uh gave this talk at one other conference actually two of two other conferences the first answer that I always got was that if the method is not mcore connect and I started thinking that everyone looks at that the first thing and how did I skip that uh but I did not look at the M connect nor did I look at uh the URI star or the U the null character I looked at something else uh let's see uh what I looked at so uh we know that it was a minimal code

change but the Striking point was that uh the comments mentioned that if the P URI did not have a scheme um so I thought that okay let let me show you what that is so here it says that if the P URI does not have a scheme and then the same thing is mentioned here so when I was reading through the comments I thought that if somehow I could create a URI which had a scheme then I would bypass the entire if Loop because this particular condition would not be satisfied so uh I think the same thing would have worked if if it was like if you could have done something something with the uh star or with the

null character or with the mcore connect method but um I thought that if somehow there was a way to construct a URL which had a scheme then we would literally bypass a patch and then you could still uh modify your proof of Concepts and uh the server would be vulnerable to the same um reverse proxy issue so all right so how to do that uh Google to the rescue uh now we know that we need to create a URL with a scheme now uh I had a basic idea but to understand that what exactly a URL with a scheme was uh we I I I made a quick search on Google and here's what uh it came up

with um this is a URL which has a scheme um the way it works is that the first parameter is the scheme name which is usually HTTP or FTP or whatever uh protocol that you're using the next part is your uh user information so you have the username and followed by a colon and you have the password that is followed by an atate sign and then you have the host name so it's uh ks.com again it's followed by a colon you have a port and then finally you have the PA and your query for the purposes of our uh talk we're going to ignore everything from here uh the main focus is going to be the user

information the host name and the port okay now that we know that this is a URL with a scheme we somehow need to create a crafted request which has some sort of a scheme so that it would bypass a patch but what would be our next step um we also need to figure out that how the translation is working like if I were to create my own um proof of concept or own crafted request I need to see how Apache is doing all the translations and uh what is happening so uh this my step two was to uh review all the Apache reverse proxy logs and to uh study the uh rewrite translation now to set up logging on

Apache was quite simple um I just had to do uh two lines of code change in the apage configuration file so um to set up rewriting all you need to do is you need to have a directive called rewrite log and then just assign it a rewrite log uh a file name and then you need to set rewrite log level to three once you do that you restart Apache you can monitor all your requests and how the reverse proxy translation is working so um uh let's see uh we're now trying to create a new proof of concept uh but before we create our own proof of concept let's just see um what happens if we use the existing proof of Concepts

that we already had um after turning on logging so I just have the rewrite rule here um and I also have the URL with a scheme and here's the first request that we looked at which is get at theate Local Host uh let's see what happens when we try and uh look at the lck file so uh it first tries to initiate the rewrite engine and then this is our requested urri right so it tries to apply the regular expression to the URI so it mentions that it's trying to apply the pattern dostar to our URI at theate Local Host the next thing that that happens is that it it will try to create the uh the

Rewritten rule so our rule will be hdp sl10 4.2.1 158 and then at the rate Local Host so we know this from our examples earlier uh but the thing to get here from the log file is that uh every time you try and send in a request just five steps happen and five lines of logs are written it tries to apply a pattern to the URI and then it comes up with a final uh request all right so um that is that let's um now look at a slightly different example um I have the same rewrite rule here but the redirect rule has been modified if you notice I added a port here after uh the U URL or the IP

address okay now uh what is uh most common HTT TP Port after port 80 someone might say 443 what else 8080 right so here we're doing a very interesting thing our rewrite rule has the rule 10.4.2 158 followed by Port 80 and in our request we're trying to request the page 80 now let's see what happens this is going to be our URI which is obviously going to match our regular expression dostar and then it's going to be assigned to dollar one and the final request that you have is 10.4.2 158 colon 8080 right so we managed to get access to the page running on port 8080 uh while we did not have any access

so from uh an outside Network you were able to create a crafted request and you got access to a page running on a different port and you did not have access to that earlier okay uh now let's try combining the first and the second request and uh see what is happening so mind you we are still at all the uh the previously available proof of Concepts that we already had we're just trying to see how uh rewrite rewriting is working uh what happens here here is that uh we are back to our original example the port has now been removed um when we try to request colon at internal server. qu. comom and followed by the

port 8080 it's going to match the URI and the URI is going to be assigned to dollar one and your final request is going to be 10. 4.2.1 158 colon atate internal server. follow.com now can you guys tell me what happens here when you have a particular username followed by colon at the internal server. fis.com in terms of the URI that is here password exactly it's a blank password so uh what happened here is that 10.4.2 158 which was our username um sorry which was our uh UR L is now being treated as the username and the colon and Then followed by a blank password so because of this at the rate it changed the meaning of the entire

thing and uh you can treat this as a username and then specify whatever internal uh or web address that uh you want to specify okay now from step one we know what a URL with a scheme is from step two we know how Apache is uh rewriting rules and um how it's translating um for me step one combining step one and step two uh is was equivalent to creating a new working exploit so uh let's see how we can do that okay uh so I have the URL with a scheme here and then now um we have a patch Target and we're trying to create new proof of Concepts new exploit strings um to see that if we can uh

bypass a patch so um I sent out around 20 to 25 different sample requests but I'm just listing a few here and if you looked at all the previous three examples we saw that from the logs there were five lines of code and uh five different patterns being matched up so here what is happening is that I'm sending the request get get at the rate Local Host followed by two colons and then 8080 I'm sent here the next request I have two at the rates so what my basic purpose was trying to leverage the URL uh uh dmit which were colon and at theate because I I knew that everything following the first uh colon was the

password and everything following the second colon was a port and then there was an adorate delimiter so I tried creating request which had multiple uh delimiters like multiple aerates and multiple colons and then U logging what the uis were being extracted now there's a pattern here in terms of the URI being extracted uh can you spot the pattern from the request and the

URI I'm sorry exactly so everything following the first occurrence of the colon is being treated as the URI right so here everything following the first colon is colon 8080 here everything following the first colon so I noticed that having two at the rates did not help a lot but having two colons did because as soon as you had two colons to the request URI everything that followed the first occurrence of the colon would be treated as part of the URI so you could leverage this particular part to either use it as a password or as a port so uh all right uh we're inching closer towards the PC here we have uh the exact two U

columns from our previous slide I just have the translated request here and we already have two proof of Concepts right here based of the the the premises that everything following the first colon was the URI Can you spot the proof of

Concepts okay so this is the very first one our request of aate Local Host colon 8080 resulted in the URI of http 10.4.2 do15 58 with 880 now remember from the outside world all you're doing is you're trying to access at the rate Local Host 8080 but because it went through the entire reverse proxy translation what happened was that the translated request let you access port 8080 on this particular page right and the next proof of concept is right here wherein you're trying to add a random string follow by a colon at internal 1. pais.com so your final request is 10.4.2 158 at internal one. po.com so what happened is that this particular thing was converted to a username and

then you were able to access internal one.all us.com right so we just created two proof of Concepts that worked on patch targets and uh now it's time to exploit it so um all I'm doing here is that I'm sending uh my first proof of concept get atate Local Host colon 8080 and uh seeing what is happening so if you uh look at the response here it comes back with a 200 okay Response Code and you see whatever uh internal page is uh there on port 8080 so basically by just understanding what a URL with the scheme is and then trying to leverage that and creating crafted requests you're able to uh create new proof of

Concepts uh that worked on a patch Target for my next request I'm sending the request uh uh a random string followed by a colon at qqqqqq do.com and the result is the same it comes back with a 200 okay Response Code and you're able to access uh the internal system okay so um after I discovered the PC uh we do have a responsible closure policy at quis so we reported the issue to Apache and Apache was very quick in responding back they assigned uh at a cve and uh then there was quite a bit of back and forth uh uh between uh the guy I was interacting with and uh they mentioned that they wanted to make the issue public because

the initial part of the code they were not sure of how they wanted to uh patch it so it was made public and a couple C of months later Apache patched it with Apache 2.222 let's see how it was patched because I was curious to know that so the first part of the code uh that uh was there it was yanked out uh the F the file that we looked at in protocol. C because that was trying to make uh changes to a a common file and what they ended up doing was they ended up patching the proxy doc module and the rewrite module and they're doing the same thing they're trying to um see if the URI begins with a

forward slash or not and if not they just uh bail out by giving a bad request error code now uh if you think from a broader perspective um finding the U vulnerability was not very uh not very difficult uh we just did a couple of things the only tools that we used were netcat and some basic Googling skills right so uh things like these happen like patches come out and sometimes uh there are loopholes in the patches that if you just look at the patch there is a way to just bypass it and uh go past it um I also have a tool uh which is available to see if you have a reverse proxy configuration and if you want to

check if your rules are correctly or incorrectly configured uh it's a standalone uh pole script uh you could download it uh from uh this particular URL and all you need to do is provide it with your uh vulnerable Apache uh um IP and then if you have or if you know of internal uh web server addresses I also have an option you could try uh providing that and it might come back with a 200 okay Response Code if the rules are not correctly configured and here are a few references uh I used uh to put up this uh presentation and finally if you guys have questions questions right thank you very [Applause] much