Log4Shell If You Don't Know Any Java

hopefully hi one i i know we're the last session before lunch we won't run too late don't worry so let's see um who remembers look for shell hopefully everybody um and it was a bit like the thing where in december mid-december look for shell hit and we were all kind of ready i think for christmas more or less and then somebody decided to join the party and it changed how things were working out a bit um i normally run the poll like what logger are you using if you're a java audience but i'll skip that for today i still have another poll if you go to slido.com and then sierra which is my twitter handle and everything or you just scan the qr code you can fill out the poll i'm just curious to ask anonymously who was affected by law for shell because nobody wants to raise their hand and say that publicly that we were affected so i'll let you fill that out anonymously also there if you have any questions at any point in time you can just type them in and i'll come back to those later so if you don't want to shout we can also do that you can just raise your hand and ask in person of course but if you type in any questions i will come back to those and we'll see how that's going so let's see um so i see that we have a 50 50 tie um and well i guess it keeps switching cool so i i see that we had a whatever share of distribution we had um i'll i'll put that up one more time so if you haven't joined yet you can still fill it out we can take a look at it at the end like who was affected and who wasn't affected um but let's um jump right in and see what's going on so when i was asking at a recent event like who is using log4j2 that was the one that was affected um for the java audience like that was the most commonly used logger um there are some alternatives some still use println and others are what what is logging um and in that case actually what is logging was a good thing because you didn't have that security issue then but overall probably not what you want so i have broken this down into like four parts what even is log for shell how can you exploit it how can it affect your products and how can you protect against that and well let's dive right in why am i talking about that i work for elastic the company behind elasticsearch we have used log4j2 and have because of that dealt with log for shell a lot um so we have that all of in our product so i spent a large amount of my december on lock for shell issues so what even is lock for shell um so those who have kids who play minecraft they might have come up with that question and you can then give them a very long explanation of why there was a lock for j update and what it means and we can take the entire day or night um to explain like where this is coming from from strings in c um down to jar files but we'll do the the compressed version of that i think um so what what happened was there there was this update coming out i think it was thursday night our time when somebody said like oh there's this security thing in lock4j and everybody was like oh well interesting i'll i'll look at it i don't know tomorrow or after christmas or whatever and then on friday pretty much the internet was on fire and everybody like yeah we need to fix that so what happened um or before let's looking at what happened like taking a step back what is even enabling this security issue so at blackhead back in 2016 um somebody said like there's this interesting concept in java called jndi the java naming and directory interface or sometimes the java people also call it gindy which basically allows you in your application to look up some other values from some source and that source could be ldap rmi or corva most of the attack vectors are now doing ldap and it's basically a way to dynamically look up some value from a remote source um to fetch that which back in 2016 people are already like oh this looks like an interesting concept to exploit and then it was very quiet until somebody rediscovered that in log for shell and basically what you could do is you would give log4j the logger a string like this which basically says use the gindy plugin and use this ldap address to fetch whatever code you find here and then run it in the context of the application and that's really what all was to it and it was a or is a straight 10 out of 10 rc um so that's pretty much how that worked um to take a look at how this works and i hope this is large enough to see but how loggers in in java generally work like you have data coming in and you have that logger that should write out the data and then there is a layout that defines like how how should the log be defined like this is the time stamp and this is the log level and this is the message and whatever so that's the layout and then you have the so-called appender that writes it out to a destination that would normally be a file and in the case of log4j it has a lot of so-called features and one of the features is that it has lookup plugins so you could use um that dollar curly brace syntax and then you have a plugin name and a value for that and what it could do is like you could say java colon version to actually look up the current java version which might be helpful to say like oh this is a java version on which this is crashing or the java version we are running on um and that's potentially a good thing um what then people figured out what with jindy which is another plugin you can then load this remote code and that's really all that happened so it's a combination of a remote code execution that you load this remote code and that you also then just execute that so it's not just loading a string which might have been fine but it's really loading that remote code and then executing it so what what happens is an attacker gives you a bad string and redirects you to their ldap server under their control gives you whatever that a thing is doing and you execute that and that's really the attack vector so somebody makes your application log that bad string it will fetch the code execute it in the context of your application and that's all there is to it you can also phrase it a bit more positively because it's not just arbitrary code execution but it's a surprise extension api which it really is to some degree but well it's not what you wanted and then there is the big discussion is this like more of a remote code execution or more of a serialization issue um because to make this a bit more complex depending on your java version and there are a lot of java versions out there um loading that remote code has in more recent versions been changed to false so it wouldn't just load remote code and execute it but it would just fetch it and then not execute it but there are still other ways around that or you could still use that to trigger local code that already exists so a new java version is not protecting you completely against it though it is closing some of the attack vectors in that so it's in part that remote loading that is a problem the other part is like do you even execute that remotely loaded code so that is kind of like the thing that is coming at play here um and then obviously how to get rid of that was you upgrade um log4j and log4j still supports jdk 7 which is very old by now or modern versions and so you need to update to different versions of log4j to get to the latest version which looked all good and people even adopted their everybody knows little bobby tables somebody adopted that to call it little bobby jindy so this is the the remote code execution that you could get with that and as we know from little bobby tables you should sanitize your inputs and that still applies for this gender vulnerability the other thing that we learned personally was that most security scanners for your applications are pretty bad who's using a security scanner for their application very few okay so the problem with security scanners for us mostly are that most of them just look for the version of a library and then they say like you're vulnerable you or you're not vulnerable but lock for shell is not that simple so what log for gel does is you have some options for example you had could say format message no lookups to true which basically says like don't load remote code because most loggers don't really need that and you could set that but most security scanners don't know that or don't take that flag into account also because of the jdk version it might not be exploitable the same way on all versions and security scanners also don't find that also the actual feature to to load the remote code in the log4j library is there is a java class called jindi lookup if you remove that from the jar you would basically remove the the attack vector or the door to it and no security scanner finds that and we have done that but then you have a constant fight against false positives funds from security scanners so we have been answering hundreds if not thousands of emails about automated scanners that were detecting the wrong things which caused us a fair amount of pain over the time so we had the first lock for shell vulnerability but why not a second one because you can have always not just second breakfast but second block for shell and what happened was that the first patch that they brought out was actually incomplete at first they only thought that was a limited denial of service attack but it turned out it was limited remote code execution again because what you had to do is you had to change the attack vector from this to this because the patch they implemented at first was looking at like is it only doing something local then it's fine but it turned out the way that this is using code or whatever is that it will ignore that and if it says sees attacker.com afterwards it will still load the code from attacker.com so this was a not so complete patch and the thing that makes it a limited remote code execution is that in that pattern how to output the logs um you needed to have something called uh or you needed to use this context um with whatever name you give it um in your log output and then if you ever log something into this distinctive variable um only then could you trigger it so it needed a non-default output pattern for your logs but you could still load remote code through that not as bad as the first issue but still um not pretty to get to that um and again you could upgrade or again this one could also be solved by removing that jindy lookup class um which we have done and for us it's kind of like it's a stupid hack but it's also kind of like effective or you could say it's honest work by removing that because it really closed that attack vector because very few people actually need that remote lookup of code to execute something it's just that i don't want to be too harsh and say like it's an unnecessary feature but it's something that almost nobody in the java ecosystem ever uses and it still opens this entire class of attack vectors so you can do a stupid hack to actually kind of like get around that and obviously there was another lock for shell security issue because this one kept giving and more people started looking at log4j and all the features it had and then they found another one again you needed a non-default pattern this was only a denial of service attack the reason was they had recursive lookups in the patterns so you could just kill your java application by having a weird pattern that did recursive lookups here the solution was this feature they just removed because again nobody uses recursive lookups or almost nobody uses recursive lookups in their java application it's just a full class of attack vectors that nobody really needs or uses um and it wasn't the last one there was one more so there was one more security issue here um this is a remote code execution but only if you control the login pattern because you basically needed to have a specific configuration to log to jdbc like a jdbc is like the database interface in java so you would need to control the pattern of how to log to a database and only could you then exploit that so that the cvs score is a bit lower on that one but they brought out another patch for that from that one there was a big discussion then how bad of an attack is that yes it's kind of a remote code execution but you also need to be able to actually provide that custom pattern to exploit it which then led to this discussion um if the attacker can already change your configuration files on the server is it really still a security issue or not um and then some people are insisting well it's still a remote code execution um even though you need write access on the server and so to actually exploit it you would probably have just to wait until you find somebody who can edit on your servers um so that one i think is more like a not such a big deal because it couldn't be really exploited in practice that easily okay the final point here is how many features do your loggers need um and that is even in the java ecosystem um which tends to throw all the features on things a bit of a debate because they're the two main loggers that are used are log4j and logback and logback doesn't have a lot of these features and to make things more interesting that the two groups of people who do these loggers don't get along with each other and they constantly fight on twitter and say the log back people will always say you shouldn't have all these features and the log4j people are like well we have all these features and we're the better logger so this is kind of like leading to these discussions and if you should really have them i'm not sure but for attackers it's a great field um because it has so many features you could potentially exploit um and by the way the the people who found the initial security issue they created that initial logo and then every time they found a new security issue they made the logo worse and this is the final result i mean if we find another one it would get even worse so this is the the final look for shell logo by now and that's where we ended up any questions for that part so far did anybody type any questions in here no any other questions otherwise i'll move on so how can you exploit it um there is plenty of code on on github to actually see that so here i'm taking this one somebody wrote a vulnerable application just to show you how easy it is or how little you need in terms of exploitation what you need here so this is using spring boot which is the most widely used framework in the java ecosystem spring boot itself by default uses lock back and not lock for j so it's not vulnerable but if you they have the option to switch out the locker so if you have done that then you can actually exploit it so for the purpose of this application um we have added that and all you need in your application is um we're having this class and on the under the the root url basically what you need to do is you need to log something that is vulnerable and here i'm just taking the the header if you provide the header x api version we're logging that and that would be the necessary vector to actually get that bad string into your application and then load remote code so any application that takes any user input and doesn't sanitize it and locks it anywhere is potentially vulnerable and attacker will just need to find some header or some other place where they can inject their strings somewhere that the application will pick it up so that's all that is needed with a vulnerable log4j version um and then to exploit that that the repository has a has a an ldap server that you can run so from there you could fetch the payload and then you would just need to run against your vulnerability application the the curl request where you say here we have we provide that header with x api version and then from the the gingy server up there we just load the code and that then it will be executed and that would then write something to the local file system to show that it has been exploited so that's really all there is you need to run your gender server you find need to find the vulnerable application provide the string that is somewhere being locked and then it will fetch your payload and you're done so that's that part um something that led to some discussions initially was like why was there no virus or should there be a virus or worm around that and there wasn't in part because every application is different and every application is locking some different things so there is not not one general attack vector but it is somewhat application specific what you're logging that's why it's kind of hard to exploit that automatically what you could see in the days after the exploits for this came out that people were throwing um their payload strings into all kinds of headers in your applications and you would see a firework of logs if you were logging that where people were trying to exploit anything they could think of like any field you could enter on the website any header they could think of people would um try to throw in their dollar curly brace strings to somehow attack your application um but because it is application specific it was not really easy to to write a worm or automate that any questions on the attack cool then let's continue um how can it affect products um this is a bit where we at elastic come in because elasticsearch um has been using a vulnerable log4j version basically forever um so 5.0 came out like five and a half years ago or so so for the last five and a half years if you have used elasticsearch or any of our products basically um we had a vulnerable log4j version in that but the vulnerability is actually not that simple to exploit and i have made some tables to actually show show you so what you need is you need to figure out what is the elastic search version that you're using and i'm this is just a slide for for seven um then based on that we have a specific log4j version shipped with that so for anything older it was a vulnerable version then it depends like which jdk you're using which attack vectors are open um so what we have done initially the the very old versions on an old jdk could leak some information but you couldn't do a remote code execution because of the java security manager and i'll show you the java security manager in a moment with newer jdks you couldn't actually exploit it and then we removed the gindy lookup class and said that don't look up any remote values in any case so this closed it this version was then still flagged by all the the security scanners um so we had to update log4j pretty quickly after that to get around those security scanners we are still removing the the gindy lookup class because nobody needs that feature it is still just a precaution of what we are doing here why is this an information leak so what you could do is you could do dns lookups from your application and through the subdomain for example you could exfiltrate some secret information from the server but you couldn't execute code and i'll show you in a moment why i've done the same table for six point x where it was very similar where it also was not a remote code execution but only on old jdks and old versions and information leak and then on five point x which is not supported anymore luckily um there it was a full remote code execution because we didn't have strict enough java security manager rules in place so there you could have both the remote code execution and information leak issue and on more recent versions you could set that format message no lookups to true and which basically doesn't do remote lookups on older versions the lock4j version is so old that it doesn't know that flag anymore there you would really need to go into the the jar file and remove that that class file from the the system um and if you're on an even older version which is definitely not recommended but then log4j was so