Down The Rabbit Hole Of A 20-Year Long Vulnerability by Olgierd Pieczul

okay uh welcome everybody for uh this talk thanks for coming uh so we'll be talking about 20-year long vulnerability uh my name is olgaert I work for security architecture in Oracle I help our teams to write secure code and I will be talking about code security so the flow of this talk is is quite unusual we'll do a lot of zooming in we'll zoom in quite hard into this problem and then we zoom out to understand what we can learn from that so at points it will be a bit hard but then at the end it will become very obvious I hope so we start with zooming into web applications then we look in Apache

struts which is our subject here and how it handles parameters and we'll discuss some vulnerabilities and then when we zoom out we look at some technical challenges the slippery slope um how people deal with abstractions and what we can learn from vulnerability reports and their quality in some places uh I will lie just a bit uh I will lie a bit uh so if you're like an expert on Java you will find that there's some poetic license in some of the examples there is a contained space and those things are complex so this wouldn't Hinder it's it's factually correct but uh there are some simplifications so don't don't tell me that oh this is different I know

um okay so web applications um from the very top web applications just work by requests and responses you as a browser send a request application sends a response if we zoom in um there is actual protocol there HTTP which deals with which supports that and with HTTP you have certain formats and conventions and and protocol how it's being used in particular for example parameters are sent in this encoded fashion and the responses are sent with the protocol and typical responses HTML so very early when we started building web applications we've been dealing with all of this by hand so like if you remember CGI bin and like if you were writing web applications in parallel I half of your

application was like dealing with this encoding and breaking things and decoding them and then your code would be half of your logic and half of like outputting HTML to the standard output and that's how it was done at the very beginning but luckily over time we developed Frameworks we wanted to shift the balance from handling all of these requests and response and focus more on application logic we wanted to write actual applications not deal with all this legwork of handling requests and responses and that's that's where we are now we don't deal with that we use those Frameworks and the framework I will be discussing is Apache struts stress was groundbreaking um in a way that it really helped

building applications um in a in a modular fashion using high-level language in Java in reusable Manner and at that time it was I think like 75 percent of Fortune 500 companies used Apache struts so it was very very popular it still is um so stress uses something like model view controller Paradigm uh in in in a nutshell oversimplifying a bit it separates the logic into controller which is struts that handles all the legwork request response isn't parsing and what you do as application developer you write actions which are the model part or your business logic lives and view which is the presentation the HTML part the JSP whatever so we can separate that you can have

different team doing the the business logic coding and different team doing layout uh you don't mix one with another so example of an application is all of this I will be talking about like a simple function that returns an order right so if you have like an online store you want to get an order by ID and that part of logic of this application will be covered here so implementing something like this in struts as you can see there are two pieces there is this action and there is a view action does very simple things it implements three methods one is to set the order ID this is how the action will get to know what order it is asked about

there's execute where the actual logic happens this is where it takes the order from database for example and return success which tells struts okay this worked I can render The View and it provides a getter for order which allows later part of the application logic to retrieve it so if you try to look what happens actually when stress works with an application so browser sends a request this is handled by struts my code doesn't run yet then struts goes and instantiates my object it calls set order ID on that so my object receives that I can remember it for later then it calls execute then my logic runs then it knows that it wants to render

the view to enters the view and in this view there are some special struts tags so the HTML part can refer to the the value returned by action so this is very nice because I can as you can see the left hand side part has not it doesn't really do anything related to web it just it's a pure logic it sets gets it doesn't deal with any parameters it doesn't deal with any presentation you can go and swap those views you want to return Json fine your logic stays the same you can just replace those views um yeah HTML goes back so now we're getting to the actual core of the problem what is this magic how this

happens that when request has order order ID struts decides to call my action set order ID I can name things however I want and those things will happen and this magic that struts uses it's called ognl ognl is a library that allows you to I can Traverse and address Java objects and hierarchy of java objects as strings so you have all this things if you can say like Foo will will call get Foo on an object if you have for equals bar it will set full to bar and you can change things you can access list it's pretty powerful you can do a lot of things with that so you can have simple strings on one side and have like

Java operations on the other side so this is what struts is using for that piece of logic so your parameter gets interpreted as ognl ognl parses that and executes the method for the uh pretty nice because I don't I don't have to go and configure everything I just have this magically working with my application unfortunately this logic was responsible for this many vulnerabilities I I'm pretty uh unhappy that there is not one more because it would look nicer so those are all the vulnerabilities related to OG and L processing by strats it's not all the stress vulnerabilities and they were what we have like we have many years of those vulnerabilities and some previous problems were not reported as cve so

they not have records I will cover them so what I did I spent weeks going through them going through every code change after code change every discussion on mailing list or injira um looking at every bug report and how it was reported and I was trying to understand how this happened how that one piece of code could have resulted in so many uh problems for so long so we will zoom in even further to those vulnerabilities and see what we can what we can learn uh by the way those vulnerabilities were pretty like I will not show all of them but this is what they were about this is the description of all the vulnerabilities as you can

see many of them when we were remote code execution vulnerabilities or vulnerabies that allow you to um modify internal state of an application and compromise it this were pretty severe vulnerabilities okay so let's talk about ognl so um first thing ognl does more than struts uses it for so with all gnl you can do things in the middle if you see you can do you can write things such as uh execute Java and or general will do this for you so if you put this whole thing into a parameter it will go and run your code inside that application luckily uh very early strat's team has disabled that functionality of ognl so they just disabled method execution altogether

okay so that worked fine another problem was that um it has something called context variables so you have variables in struts sorry in ognl they are denoted with hash and coincidentally struts also uses those this stack of of variables to store information about the state of application such a session and user information an internal state so you can just use that convention in ognl and like for example set user to admin in your session just by using ognl and the fact that struts uses that ognl so when these when struts saw that they implemented a fix and that fix was oh hash character is a problem we will have a filter that says if your

name has a hash so like in Java this means if it contains if if the name contains hash then reject the parameter but then it was discovered that there is more you can do and then it was discovered that you can use Unicode version of hash so they kept adding those things um they decided maybe that is not good so we they put a regular expression instead and that works for a while then it just was discovered again that you can manipulate that so that filter wasn't perfect and you can do actually two things first you can bypass that by putting this as a sub-expression in stress sorry in ognl so that's the the bracket but also

the logic that controls that method execution is disabled so you cannot write in your custom code was also controlled by custom variable so first you bypass that you enable method execution again and you're back to business you can write your you can like execute custom code on an application from the browser so they expanded this regular expression to remove that but there was more so there was uh you could run static methods so Dynamic methods were disabled static methods um things that like are always existing in the system they don't refer to any specific objects were available so they had to disable that you could run Constructors which was pretty interesting because some Constructors um

have side effects so for example if you implode if you run a file writer Constructor it creates a file so that was available and you can put odnl inside or DNL and that was also available and you could bypass that filter this way so they were they were refining and refining and refining that expression uh to this point but there was another problem so before I go there uh kind of thing to have in mind for the beginning like there was another problem that people use request parameters for other things than like talking to application in particular there was like a popular framework called Dojo at the time and Dojo was like using parameters to just denote that the page has a

different URL to bypass cash but those those things didn't have a corresponding methods and they were causing errors uh so there was a request put for strats to disable certain patterns of parameters so they are not used this is not for Securities but just so the errors don't show up right because many pages if they use particular JavaScript framework will have those parameters added but they have no use in the application they are just internally there so what they added was this again not a security mechanism it's just the convenience that certain parameters were just excluded and not considered in that processing just just park that for a moment okay so another thing in Java is that

it uses inheritance inheritance is that you implement an object you can Implement an object on top of that object well underneath that object maybe better and that object inherits that behavior so you can have parent object it has get full method and you implement get bar sorry child object which has guitar but the child objects also has get food it's like a an extension of the other object and that's like in object oriented programming that's that's the one of the main things right people do this all the time but also strats has some of those parent objects that you can use uh to tell strats that your action is interested to receive certain things right so for example if

your action is about to see session as you can see this is just a plain Java object uh you implement that interface and create set session in your class and then struts knows that it has to pass you a session so your class action implements session aware and it has this set session method added now this caused the problem because now you have a conflicting feature one is you want to see session but the other one is you added a public method that ognl can interfere with this is just because your class added that it didn't have it it just has it now um so that was a problem and when they decided to fix it but this is not this

is not specifically about OG and L this is just it's a method that is in the class but like we don't want that one so this is when they decided to reuse that feature they had for just like convenience now as a security mechanism and they added all of the things that their um interfaces that were provided with strats were implementing so they promoted this non-security feature to a security control uh to add that okay so that worked for a while but there was another problem in Java every class inherits from object called object from class called objects like every this is like a root object of everything and that class has certain methods uh like

gets class to string hash code if you know Java you're familiar with that so these are like things that every class has implicitly so the problem is that if you have get class which is publicly accessible yet class has class loader underneath itself and then you can you're just limited but by your creativity how you can exploit that so the point is that through these public Getters of uh through public interfaces of every object you can go and manipulate things like state of jvm so they were very creative attacks based on that where people were like stringing multiple requests one after another to modify the state of jvm to run custom code so what's just did

okay so they added that there was a couple of iterations of that regular expression because it wasn't really always working well so as you can see this expression got a bit complex but they again added that to the list and they were modifying that so there were new problems found this set of filters was was not good people were fighting ways around them uh then like they compressed it expanded it um so this is what's I think at the end is in strats today um I don't know what is doing that much like I could I could spend some time and understand but this is this is this is complex this is not it doesn't look good

so are we done with with ognl well almost so at the same time when all of those was happening there was another logic instructions which was cookie Interceptor which was doing exactly the same thing just for cookies rather than for parameters and only very late it was discovered oh actually it has all the same problems so this was added there so many of those CVS refer to cookies as well uh yeah so uh hooky Interceptor also has its own pattern and that was added and solved some of the problems okay so so this dragged on for like 20 years um what went what went wrong and what we can learn that's the that's where we

will be zooming out a bit so first of all um on the very like very obvious but very superficial as well like don't do regular expressions for security this just doesn't work they are hard and they are um difficult to manage in fact some of the vulnerabilities were just based on the fact that while making some fix developers broke those regular Expressions so they disabled previously filtered things so they messed them up themselves by accident here is an example when they just removed something by accident and they had to put it back thank you if we go further so this is uh pretty interesting so this is all the changes to all of those filters across all of

those years and when it's bold is denotes that like what was the actual change so this is very interesting to look at that from from that perspective because it's very easy to pick on developers and say ah why you did that and if you look at the final set of regular Expressions yeah this is pretty mad but it never it didn't happen this way right so at the beginning uh what happened was that there was pretty simple problem and pretty simple solution and then when the new problem appeared the natural thing was to extend existing solution and the more that existing solution was expanded this became like default mechanism for doing that so when the developer

um is told about the vulnerability that loads of websites and companies are angry about and they have to produce a fix as quickly as possible they naturally would would gravitate to just extending the solution they already had even though it creates this very complex logic that is not very good um there were hints where things could have been changed so for example moving from just list of conditions to regular expression or refactoring regular Expressions that happened a couple of times um uh but uh but yeah there was never a good time because the the right incentive to do something about it is when you're making change but this is also when you made to make as quick

change as possible so perhaps choosing Simple Solution at the very beginning is the real danger because it puts you on the slippery slope the right time to redesign it is probably earlier because the longer you go the more you have like people depending on it and so forth there's there's more to that because this changes were split between config users can overshadow them and it was it was there was much more complication in that um but yeah making early decisions to fix problem in a particular way will will haunt you uh later on so if we zoom out a bit more the other problem that that you can observe is that what stress did was they

used existing Convention of like public method public uh so so access modifiers for Fields you can have methods that are public private protected these are not security things they know they sound security but they are not they are just about um you know defining interfaces and how things get exposed across between objects what strats did was they mixed that convention that is used for many other purposes and people do use public methods because they have to because they want to have their classes extendable but they linked it with you can set parameters if a method is public and they created this this new thing that any public getter Etc is accessible by HTTP request this

is unexpected this is implicit this is this magic that this service that this framework provides but it's very um very unexpected and many of the problems result from that what they should have done and what other Frameworks did like spring is that make it making it explicit so don't hijack the concept of public matters and use that for for a decision what spring does it's a similar logic here uh it actually has ways to tag things so you say oh I wanted this part this to be a parameter and I can even name it differently it's not implicit it's explicit so everything else stays invisible to the world unless you explicitly say I want this thing to be

to be visible uh funny fact in one of the discussions very early strats were considering that so developers of stats were like you know what we should have this annotations those things with that here here called annotations we should have these annotations to solve that problem that never happened this was 2007 so this was very early this was this opportunity to fix this problem uh once and for all but unfortunately it didn't happen um so if we zoom out even further probably the core problem and the things that we should avoid in writing software is when we have a need and we look for a solution for that need uh we need to find one that has all the

things that we need but what they have chosen is something that does that but also much more so using two powerful tool is in this case dangerous they didn't need to have the full expressiveness and full capability of ognl but they use it because it was close it was from the same umbrella the same ecosystem and it was convenient for them to use because it had all the things they needed now if you discover that oh I have a tool that does more but now you you have all this behavior that you don't want inside your your code so what do you do well they try to contain it so trying to take something that is much bigger than

you want and trying to contain it for some sort of firewall doesn't really work in software so this is false dichotomy this is false equivalence those things are not the same the expectations is that they will be the same but you don't build smaller things by making taking bigger things and just slapping protections around this that's that's the main problem of of this design um so another thing that we could see is that um well so so in software we um always build larger things from smaller things nobody writes software from scratch we use components abstractions um and it's great because it reduces the complexity that's the idea you have to reduce the complexity you delegate some

tasks to some other abstraction uh so this is our kind of stress application so it has your app some the app can use some Library you have your struts it has all of those components you have ognl it does things and you have Java underneath all of that interacting with everything you have web server Internet it's a very simplified model but it already looks complex um the problem is that this wasn't a clean obstruction it was like everything was interacting with everything else in unpredictable ways rather than reducing complexities was actually increasing complexity so when we in hindsight look at that problem it's like oh setting a particular parameter through struts to ognl allows you to take

advantage of class loader that is on every object like this is like this goes all the way through all of those layers and it it's not a clean abstraction because of that chosen um way of like running code but trying to contain it another problem is like from a point of view of developer who sits in particular place it doesn't look this way we see it this way because we analyze this all but what developer sees is something like this right they they still see those abstractions they don't see all the details of all these complex interactions because they want to see that they know a bit about struts but maybe things get blurry at the edges they know that there

is some expression parsing in node gnl because that's what it does and they knew very early that there is Method execution that they disabled and access modifiers in Java because that's what they used but that's it and only later when the reports of vulnerabilities were showing up they like gained understanding of those things that were invisible but again never get probably like full understanding because okay this is this is very simplified model they just see bits and pieces of that complexity and trying to slap protections around them um so a lesson from here is like when you're building abstractions using abstractions do it cleanly not in a way that everything interacts with everything because that

actually doesn't reduce complexity it increases it um so the final pieces on bugs bug reports this was also quite interesting to see um so bug reports may be misleading um so the the cycle would be somebody reports a vulnerability and they explain what this is about then it's consumed by the team who who writes its own advisory and there is a cve record that has that so this is interesting example with the remember that you can create a file by running a public Constructor in the report from from from the researcher it was explained as arbitrary file override right sure that's a thing that can happen but this is not the problem it's like you can run public

Constructors so you can do pretty much anything that that is available as a public Constructor including writing files but this this bias in in the description propagated into into this other public information so when you go and look at what is this particular vulnerability we'll see oh there is a file override and maybe you can see well it doesn't really matter to me because my application Riot runs in a container with read-only file system I don't have to fix it but no this vulnerable is not about that it was just an example that was chosen in that report um so in [Music] it's it's important to know like when you're making life or death decision

about like a particular vulnerability don't assume that everything that is written there is actually correct probably is most of the time but there were examples when it's not another kind of interesting thing here is that this thing about like Unicode replacement and setting context variables that was reported multiple times it was the same vulnerability with the same impact that could be exploited in the same way they were just like slightly different attack vectors because there was like ways of bypassing that filter but this was always reported kind of slightly differently as there was more understanding and the score the Vol the CVSs score of those vulnerabilities was different it was all the way from 5 to

10 9.8 and this was kind of interesting um because this was really mostly how things were reported that impacted how they were rated not how the actual vulnerability worked so only after the final well deferred about that vulnerability was reported it had in the title or in the primary description it allows you to execute arbitrary commands and it bumped the score so if you're looking at what was the middle one about well it was about the same thing and it should have the higher score but it didn't because it wasn't understood this way um and like when that understanding was was available even though it was there maybe it was on the second one wasn't on

the first one uh like nobody went and like bumped up the score of historic vulnerability because um I don't know why uh but uh what we can learn from that is again those vulnerability records will work well in aggregation but they are probably not very good tool to look at specifically when you're looking at specific problems um so final thing more human nature thing when um when I was looking at the discussions of the development team and their approach to things I noticed that they had completely wrong assumptions about what people understand about stress they had this under this assumption that people really well understand the security model they have chosen even though it was a bit weird

um they were dismissing multiple vulnerabilities like the one that you can access inherited methods for years until the get class showed up it's like oh we didn't think about it either but there was lots of wrong assumptions that that were completely incorrect they were very close and to the problem and they thought people knew all of those things and like surely people should be doing this should be doing that that's not how all the other people wrote their applications so again don't assume that others understand your security model especially if it's weird so to close some highlights uh struts uh picking on Strat is easy right especially in the hindsight but that wasn't the point of this talk uh strats

was groundbreaking framework that was that helps people write good applications but they were also first one of the first trying to do those kind of things and trying to build those abstractions and there were mistakes that we can learn from um don't do magic if you're writing your application and you want and and it seems like oh this looks like like magic that's probably bad because people don't interact with magic very well so think about clean abstractions and don't assume that people will understand all the intrinsic nature of your of your model um powerful tools don't use two powerful tools for a simple task there's no way to contain them and especially not with string matching

design change uh the right time to change your design is now there will be never better time to do it uh I know it's I know it's hard to do but if you're thinking about well compatibility issues disruption this will only get worse uh bug reports they can be misleading um maybe inaccurate they work well in aggregation or just to point you to existence of the problem but don't take them at face value to like decide whether something is worth fixing or not um if you want more about that there is there's a paper I co-authored that discusses that in a bit more detail and that's it so thank you for your for your attention and I will have some time

to take some questions

yeah so the question was did it had to happen so so they could learn so yeah I don't I was thinking about that so yeah so nobody is doing things like this I think this was my opinion was that at that time uh people were really trying to model everything through object oriented programming people are trying to contain all sorts of problems into relationship between objects and inheritance and aggregation and all of that and Strat was also trying to do that was like oh we have this problem of objects and stuff let's use methods and Getters and Setters and Java beans uh this is not how we do things now that was the core problem it was like trying

to hijack existing thing for and and like shovel it to fit the problem but it didn't fit and it resulted in this this complexity and these problems

came out um

[Music] about all the buildings yeah so the question was about relationship with struts one and struts two struts one wasn't using uh all of this magic but there were certain things there were I think like I never used rats one I did use threats too but um I think eventually there were vulnerabilities to struts one and there was no patches because nobody was fixing that so that that's the danger I think there's another question yeah another way to characterize this year

it was so so principal yeah principle of History so this is probably what I try to contain in this picture with like this is what can happen and this is what you need uh so um we do that like this is common very common in software where where there's more expressive things like when people use HTML for formatting oh I need to allow users to put some formatting so they will use HTML packs and people put scripts right that's it's kind of similar thing so it's not uh trying to take something very expressive uh for a very simple task and open it up and then trying to patch it around um any other questions so for example if you inherit the Legacy

application built on top of trust that's what you have to deal with how would you secure it would you would you put something in front of it yeah Define what inputs you want some kind of web application file or anything else what could you do maybe not to eliminate the risk but reduce yeah with uh yeah so like the question was if you already have to deal with strats what you do about it well with pain uh you you do all of this uh so we cannot solve the fundamental problem you you have to rely on defense in depth that's probably the best and ask to migrate um

yeah yeah it was it was very popular it was it was extremely popular and uh so you've heard about Equifax Hack That was Apache strat's vulnerability um yeah it was there was no there was no good fix uh there was like they eventually like there's not that many new vulnerabilities in strats so this kind of works uh but yeah it dragged them for for long and and it's very hard to migrate off a framework especially if it's so like informs the design of your application okay I think we are at the time so thanks thanks again for for all your questions and for real attention uh thank you