Navigating The Open-Source Security Fog: A Case Study Of Messy Vulnerability Data

thank you hello everyone thank you very much for coming to uh to my talk um so yeah my talk will be a short one um but we have a little bit of a bigger slot here so the benefit of that will be I will talk a bit less quickly than I than I normally do um but my short talk for you today uh will be on navigating the open source uh security fog so uh the weather outside is uh pretty typical today um and and in here things are also going to get a bit hazy figuratively at least hopefully not literally um as we dive into a case study of murky vulnerability data so quick about me I very short and

sweet uh about me um I'm a vulnerability analyst with the synopsis cyber security Research Center Park based here in Belfast um where we specialize in reporting and Research into vulnerabilities in open source software and this is where I started my career uh five years ago so five years experience a fair bit less than some of the excellent speakers we've had here today uh but did hit one nice Wii Milestone uh this year which was discovering and Publishing uh my first uh cve uh Records uh at the start of the Year U which I don't know I guess that sort of puts me in an a sort of the small baby all small wind baby phase my career so um and one

of those cdes is going to be the basis uh for this talk um so I'm going to run through uh one of those cves called remote code execution in open tsdb via the Legacy cre AP we'll get to what all that means um shortly um that's a link to um The Advisory that we published on our synopsis blog for this issue um and the talk will be split in broadly two parts uh uh the first half we'll look at the vulnerability itself s um there's nothing particularly groundbreaking by the exploitation is vulnerability but it is it's a nice s example of High um uh seemingly innocent looking web interface this component has can lead to nasty

things and then in the second half hopefully um the part that will be a bit more interesting um and uh we'll have sort of useful takeaways from it um we'll actually look at the history behind this vunerability um and this fundability did have a pretty messy history uh behind the disclosure of it so hopefully there'll be a few things that all of us from users to developers and researchers of Open Source open source software um can learn from it before I go into any of the details of vulnerability um I'm actually going to start with the single biggest lesson I can teach first which is everything I'm about to say is much much easier said than done so I'm going to be

talking about this vulnerability and sort of the mistakes the oversights miscommunications um that were made with the handling of vulnerability um that will absolutely not be to cast judgment on anyone that was involved with the handling of this fundability uh because these things are difficult security is very very difficult and it's really only with the benefit of hindsight that um I'm coming in and time that I'm coming here and say saying hey here's some pitfalls uh that we can be aware of when handling open source security so what is open open tsdv uh somewhat predictably it's an open source tsdv solution um so it's a source code available component um and TSB uh means time series database uh so basically

data that you're collecting continuously over time and probably which you'll have very large a minds of so think things like processor loads stock prices whether data things like that um and the meno really can explain better um than I can what exactly open tsdb does um in their own words um it's a management application that lets you store index and serve metrics at a large scale um and make this data easily accessible and graph and the graph part is going to be the key um to the vulnerability I find being able to exploit it so that's open tsdb um the vulnerabilities in the Legacy query API so what is that um this is an endpoint

called Q endpoint and one of the endpoints that this component has and basically what it allows for is for user to ask open TSB um I want data for um a particular metric I'm tracking um with various uh query filters that you can apply to it um and the components will return graph plot data for that query um and the plot part is done by a library called uh G new plot um so how does the data get to this Library um well basically the functioning this end point is or this query API is um the user say is saying here's the my DB query here's the data I want to get for the particular metric I'm tracking my

database definitely not malicious at all um and eventually this data is going to be passed a GNU plot um which uh uh creates the plot that then plot of the data is returned back to the user and so how exactly does gnu plot get that data well in between these two and what open TSB does it says okay I'll validate that data and then I'll pass it on to gnu plot and the exact way it passes on that data um is it inserts the query parameters that the user provided directly into a shell script um so the example sort of endpoint query I've got up there um with those um query parameters in yellow um when that's sent

open the SB it'll create something something like this um a shell script um with those parameters after valida directly um inserted in um and then that'll be run that'll be run with those gnu plot commands and the plot will be created so I'm sure it'll be obvious to to many of you that the red flag here here is you know in the middle bit that it's very very important that this validation works correctly because if it doesn't then you've got untrusted input going into commands that directly executed by the system um and unfortunately uh validation for this endpoint um wasn't fullprof so a carefully crafted payload can bypass it break out of those prod commands in the

shell script uh and basically run any system command you want uh within the Privileges of the running open tsdb service so I'll show what that actually looks like the exploitation of it um in video form this is the most delicate part of the presentation because I've discovered that this video can crash power point if I'm not careful so um that's just High serious this vulnerability is that a video of it can PowerPoint Panic um so if we're very very careful here I can play this um and this is the front end open tsdb um so um you can insert you know your query parameters you know what what you're looking for for your metrics and you get

plotted graph data um and that's that's running through one API through the front end so the particular API we're interested in we can run queries on that directly um the parameter I'm going to exploit in this particular um video is the smooth SM parameter um that malicious parameter that's not actually the explo code um obviously it'd be nice if it was um but the specific details of how that works we'll keep under wraps because that wasn't um publicly dis disclosed as part of as part of our disclosure um but yes when you run this command normally you will get a Json response saying yep that data was plotted all good um but when you run it with the

malicious malicious playload I have which is able to sort of break the control foow of that shell script um and then a carefully craft C carefully formatted system command will let you run any system command you want so you know that is really it's really quite serious if you have access you know to this open TSB endpoint and you you're allowed to supply uh queries to it you know there's lots of really bad things you could do you can acil trate sensitive data from the server that you're targeting um you can set up a reverse shell so that you can have an ongoing connection to the system or perhaps most seriously at all when I run

it um you could run the system's calculator app um as every good uh command execution POC should show off so we can't can't be having can't be having Rogue calculations going on on your server very serious so is that where this story ends um it is actually that's the end um but it's not the start it's not the start of the story um this vulnerability had a very long history very unique history of disclosure um and this is where this is where the fog is going to roll in and the the very high resolution fog um couldn't afford couldn't afford a real life smoke machine sorry but um but yeah so what we're going to do is we're going

to try and navigate backwards in time through the history of vulnerability to illustrate What specifically happened with the evolution of this issue and sort of the pitfalls that everyone involved sort of tripped over um so this is going to get confusing um don't worry if it confuses the uh the bsid out view um in fact accept and embrace the confusion because that's kind of the point of of why we'll go through this uh and through that there should be some interesting things to

learn so I'll start off with um the most recent bit what I disclosed um and the first admission to make is uh so I presented that vulnerability as you know something that was brand new discovered in the vacuum it wasn't uh what I actually find was a bypass to an an earlier fixed intended uh intended to address the issue um what was that earlier fix that was able to be by bypassed well essentially I won't go too much into detail on this but essentially some regular expression validation was added to these query parameters which was intended to lock down what input you could Supply to as query parameters so intended to be you can only um Supply those options nothing

else no bad characters no bad commands or anything um but when we in when we inspected this fix commit and see what exactly was done we realized that whilst this is what that was meant to say um what the the way the r Jacks was defined specifically what it actually meant was uh you could stick in one of those four options in front of your malicious code and it'll pass validation and so unfortunately it it didn't work um because it just hadn't been quite the Rex hadn't been quite um defined correctly so that was able to be bypassed so obviously um because you know I discovered an incomplete fix to this issue there must have been a previous disclosure of this

specific vulnerability um prior to mine of course there was so let's try and go find it let's go uh wandering through the fog again backwards in time let's see um what we can find and what we do find is back in November 2020 uh was was uh a previous report um which led to that fix which then I identified as it could be bypassed um but this report also was for an incomplete fix um and which was very much and I won't bog D too much with the details of of all these various disclosures and fixes but basically this this uh this disclosure was much like mine um they find a way um a different way they were able to bypass

the parameter validation um and so it required and further fixing which unfortunately didn't quite didn't quite work so of course just like my report that also implies that there was a disclosure prior to this so let's look through the fog some more and what do we turn up well we've got a report here from uh June 2018 um and what did this one say Well it said that they finded and exploited this vulnerability this particular researcher um but at the time they said we think this is fixed we've confirmed that this is fixed um and that makes sense right at the time they believed this was fixed and it was only couple of years hereat in 2020 another

researcher said oh no it actually could be bypassed needs a better fix applied um so that all that all makes sense um but hang on what's going on now with more fog are we going back further in time well we are the fog clears again and we've landed in June 2017 now um and wait a minute it's a report saying that it wasn't fixed um it was known that this issue wasn't fixed it was still exploitable um a year before someone wrongly confirmed uh that it was that's not good obviously uh imagine you know if you involved in research is Vil or handling his vulnerability in some way um and you'd only seen that 2018 report

you would have got a mislead picture um at the time of sort of the state of this vulnerability you thinking it's fix fixed whilst everybody else malicious actors are like nope we can still exploit um I like this um yeah and I I also like this particular is report as well because just in the middle of it these are all public GitHub issues by the way just in the middle of it someone just comments ARG why is security so hard sad face which I'm sure every single one of us can sympathize with um so maybe we should spend a bit time to reflect oh no wait never mind there's more fog we've got to go backwards for

further there's still more to this issue um what are we going to find now goodness knows we what is it oh right March 2017 an earlier disclosure in the same year um that led to that fix that was known bad um but later misreported as good in 2018 before confirmed in 2020 before then being byp all that stuff um so that's clear now right that that that should be sort of the end of no no no we're still not done with this uh the data here is still so foggy um what we get next well we're in 2016 now um and even earlier report um and there is something particularly interesting about this disclosure of this of this

particular issue which was this report said hey you know command execution we can do this in this in this in this query API um but in the middle in the middle of the report they kind of quite confidently stated that the reason this vulnerability exists is when you're making a a malicious request to this API and you request the data back as PNG so you know just an image of the plot data um that it's calculating um if you cause an error then it's in the all the error handling that's where the command injection happens and that's where the where the bad stuff is happening um and unfortunately the maintainer at the time really just took this at face value uh

uh and so the fix they applied was Okie do we'll just chop off that send as PNG command that was part of all that uh job done but unfortunately that was you know as we've seen with the later disclosures that wasn't the root cause of the issue um you could and the indeed the exploit I was running was just sending the request but returning it as Json completely bypassing any you know PNG hits so so that wasn't related to but unfortunately that was sort of Taken and ruled with and so that meant it it wasn't fixed at the time is that the end surely this must be end nope nope it isn't um but it nearly

is I promise it nearly is now you know what could possibly be left well thankfully the the the earliest report that we were able to discover for this issue um was back in April 2016 so really quite close to that may report by another researcher who basically they basically discovered it independent of each other um and it does stop there so this is the full timeline of one vulnerability it went through a series of disclosures and a series of fixes and you know I did say that you know all of this so very very confusing and maybe being a bit flippant like maybe when you go forward for it it's actually quite simple um yeah it really is simple isn't

it you know two report in 2016 you know which which independent of each other which likeed the issue one of them reporting you know confirming the wrong root cause which led to a bad fix which then a year later uh was reported uh but that that led to another bad fix so later later on in that year someone said nope it's still not fixed but it kind of fell by the wayside so then in 2018 someone said oh actually I think it is fixed even though it isn't um and then two years later someone else said no it isn't fixed you got to fix it uh and then 3 years later we came along um and

said nope still not fixed could you please fix it um we disclosed that um and and they tried to fix it and we and we validated it and we think it's fixed um and then we publish it for clear as mud right yep clear as de so yeah that's that's everything I've just went through on one slide um that is pretty much exactly how my brain felt when researching this vulnerability probably how some of your brains feel now too um there is a point to all that um what I really have to stress is you know I've kind of raced through that hisory you know a few minutes you know saying oh so confusing all the f um but when I was

first researching this vulnerability it took far far longer uh to actually work all that out took a lot more time than just sort of racing through the fog in a few minutes so imagine if you were someone that was tasked with working like what happened with a single vulnerability whether you know you were involved with you know trying to address the issue whether you use open tstv you know in in in your software and you're like we need to address this you know you're going to have a bad time figuring this s and this is just one single vulnerability so imagine if you have loads of vulnerabilities in your open source software that you use um you know

this is this is you don't want to have to be sort of treching through all this all this fog and all this sort of very messy data so kind of just to summarize what exactly what all that sort of borderline conspiracy the timeline was um we had seven separate disclosures for one vulnerability so throughout all of that I was talking about just the single ility command injection in that EMP um and particularly should not particular that six of those disclosures were sort of fully fully public get up issues so they were just posted up anyone could see them you know even before even before the maintainer had you know seen it and realized the issue and tried to

fix it that was all sort of fully publicly available at the time um through those seven disclosures four patches were attempted three of them unfortunately uh didn't work hopefully the last one um is all good now free cbes were requested along that timeline and so that's another important note not every disclosure of the issue actually got cve disclosed for so some of some of them had cve records other disclosures just didn't they were just you would only know about them if you'd seen the get up issues and all of this took place um over seven years which is certainly a long time for you know a fairly High severity vulnerability to sort of be out

there um in the public domain so what meaning we take from all that well what's good about all this confusion is in one single example it kind of nicely encap encapsulate it's a lot of the things um that the work that we do with synopsis that we end up coming across and researching and Reporting vulnerabilities sort of the thing all the things that are hard um about software security research and management um we don't usually find them all in one single vulnerability but in this one we did so it ends up being quite a nice example of of the things that are difficult and the things that could go wrong things like verifying vulnerability is is fully fixed um

obviously with some of those disclosures um we believe you know that that likely it wasn't done or it wasn't fully done in some cases with with how these issues were disclosed a clear communication between researchers and maintainers um as I said you know a lot of these were just public get up issues um and you know not all the information lined up uh not a lot of it was clear um and perhaps one thing I should emphasize as well um on this point is um something that I think is very easy to forget is that you know if you're a security researcher and you're disclosing something someone something to a maintainer of a software component um it's likely that you you

probably know more than they do about sort of the the nature of of sort of the vulnerability um and security side of it flip side is they probably know a lot more um about the actual working and operation of the component so there's like there's a bit of a mismatch and sort of the the knowledge the two sides share um and that I think sometimes that can be easily lost and can easily lead to sort of miscommunications which means issues don't get um resolved properly cross referencing all this data obviously is very difficult and you know the most obvious example of that was you know the reporter who confirmed an issue was fixed when a year prior someone said

no it isn't um obviously if they'd seen that they probably wouldn't have said no it's fine get the latest version It's Grant all fixed um and identifying vulnerability scope and rate causes as one of the earliest reports sort of honing in on we think this is a right cause maintainer went along with that unfortunately that didn't that didn't uh address the issue um and another thing you know which is it's not unique to open source but maybe is exacerbated by open source you know available time resources to address issues so obviously you know seven disclosures over seven years large time gaps you know between these and this is just the reality sometimes open source software the um the sort of

popularity and the usage of Open Source open source software isn't necessarily related to the amount of available time resources that people can be can put into maintaining it and then if there's security issues making sure they're addressed in a timely fashion so they don't always line up um so sometimes you know a lot of time can pass before these things are addressed just sitting in the public domain for any bad actor to to realize that this can be done um and then from a user side as well it shows very well that you know when these things happen you know when the things that are hard about software security research and management when you know these aren't

handled as well as they could be it it shows that you know implicitly trusting any individual piece of data you know a b of vulnerability that's being reported um or even cve data you know you have to be careful um you know as I said with the CV not all this stuff got cve so if you were only looking at cve data you wouldn't have a complete comp picture of what happened with respondibility um so it's hard to implicitly trust you know any one piece of data you have to be careful about that and so what can actually help what are the actions that uh users developers um and researchers can can take to help

clear this fog um well ultimately what causes these situations to be so messy is that you know the data doesn't line up could be hard to De decipher um so what can on the developer and sort of researcher side of things what can they do to help make this data bit better um something that we think is particularly important in vulnerability reporting is that both parties have clear vulnerability disclosure policies that's maybe intuitive to most of you maybe not you'll maybe you know you'll probably know about sort of you know vulnerable disclosure policies that software vendors and software maintainers would have you maybe wouldn't have thought necessarily so much about researchers having vulnerability disclosure policies

they're communicating as well um but we think um that that's quite important and what I specifically mean by you know establishing on both sides clear policy IES is things like establishing the roots for um private and public Communications what you're going to what you intend to communicate privately with with a with a maintainer and what you intend to be disclosed publicly um at the end of the day um expectations and what information will be communicated so what details analy analysis uh exploit proof of Concepts um that researchers will intend to pass on um and what both sides intend to keep private or make public um research is pledging to verify uh any proposed fixes that are

made um and setting dates for public disclosure so the purpose of that being um you're saying that you set a point um where you believe it's best that users are aware a security issue exists and can act accordly before Bad actors find out an issue exists and operate it on operate on it for themselves so even if something is not necessarily been fixed in a timely fashion you get it out there into the eye of users so there able to um act accordingly so that's on that's on the uh the uh maintainer and researcher side of thing on the user side of things um we think it's important to be aware uh of the ways that this vulnerability data

can be misaligned um which you've seen plenty of the reasons uh through this vulnerability um to sort of summarize the key ones um it's important to be aware that one vulnerability um can be disclosed multiple times um so if you have you know vulnerability that's um not fixed properly several times or even just disclosures that that misalign and that can lead to multiple disclosures um and not all security issues run through the CV process this is something we see not necessarily commonly but more than you would expect to see uh we believe there's kind of there's a bit of a sort of um gray area where security researchers and maintainers don't necessarily know whose responsibility it is when a

vulnerability has been discovered who it is who actually goes to to get a CV uh cve assigned and and published out there um in the public domain um and when there's sort of a gray area it can then mean no one does it and so things don't necessarily uh get um cves um and whilst that last point there things to be aware of is not necessarily um an action item as such I think the point really is to say uh if you're in a situation where there's something so not necessarily you know software security but you're in a situation where there's something something or some data that you can't implicitly TR um then let that help you sort of adopt

a more defensive proster and so whatever processes you have you know to manage security risks in the software you use whether it's CV data other vulnerability data dependency tracking what whatever you you you you have in your environments um if there's anything you think you do or don't do because you're assuming well I get that security data from somewhere I'm observing the security data and the data looks good it's probably fine um then being more defensive in reaction to the fact that there's a chance that theat is maybe not so good it's it's not so fine um then that can only be a good thing and that's me and so yeah thank you very much for

listening I'm happy to take um any questions plenty of time for [Music] questions

interested personals Enterprises [Music] not

research immediately after yeah absolutely yeah so the question is asking yeah so there larger Enterprise organizations you know you know the open source side of it how it relates to them you know our understanding you know open source software its usage is so prolific um it's not just you know randomers using open source software you know you know all organizations of All Sorts sitzes use open source software and it is yeah it is unfortunately reality um I think of you know a relevant XKCD comic uh for everything we've seen one of them in in one of the earlier presentations and it's that one of like all these sort of um sort of like bricks the Jenga

blocks and there's like a tiny you know which is your software stack and then there's a tiny little block which is some random component maintained by one guy in Arkansas or something um and you know if that falls apart then the whole thing the whole thing um Falls dying um so it it it it's important it's important for everyone basically um that you know um that this needs to be tracked um and and but unfortunately sort of you know so many packages are used and obviously the larger the organized you gets the more packages are used um um if if the data like this is per then you know it's just it's it's not you know no Enterprise organization

is going to be like no problem all our size severity vulnerabilities will just uh manually go and look through all this data like I have and just double check what was good what was bad that's not sustainable so you know that you know we we do need to make sure that you know uh this data is clearer and sort more lined up um so that um so that you know users developers can have more Trust yep

you I don't have any specific figures for that now um I can just say anecdotally that it it does happen it happens more than you think it would um um so yeah um so I don't have a specific figure but there there there are several occasions where that happens and it's not necessarily a bad thing when usually what most people do if you have vulnerability it's disclosed try to be fixed people think it's fixed and then later on someone discover the isn't and usually that gets a second cve so you essentially have two cve records which are tied together um and as long as everyone knows that they're tied together then you know the picture is

clear um you know and it's just it's not that much different to if it's a brand new vulnerability is to disclose right unfortunately there's cases where that doesn't line up to I don't have I don't have a specific figure but um um a worrying [Music] number [Music] particular absolutely yeah there's enough there's enough there's enough unclear data to go around so yeah I'm not going yeah I won't go as far to say again you know and it's it's it's it's came up several times if anyone attended my colleague R talk on mobile wout security he made sort of the same point that you know like this isn't to say like oh these developers what are they

doing you know so hor like this is difficult this is all difficult and you know and like I'm inviting myself you know I'm saying you know we discovered this issue that's hopefully the last bit in the timeline um so definitely no one else is going to come along and find actually that fixed as bad as what you know new serle I you know um so so it's all very difficult but yes um you know there's a lot of unclear data out there that yeah you can

inspect yep the [Music] back

[Music]

no it's a good question the question is yeah when you have these sort of you know multiple cve so the one issue um and tying them together whose responsibility for that is and I think the answer to that is basically same one for the signing the CVS in the first place um it's not always very clear um you know you know anyone this is the thing anyone is able to request CV um and we see it often like sometimes people who aren't involved in any way with a disclosure of vulnerability come along a l Point say oh that should get CBE and they go assign it um so so it it can be anybody's responsibility but

unfortunately when it becomes anybody's responsibility then you know if no one actually steps forward because they think someone else is going to do it then then it doesn't happen in terms of how can be related together you know at the cve level like when cve records get published by miter and then analyzed by MVD and National vulnerability dat this um there's no special way in that they're related it would just be it would be incumbent and whoever was disclosing the cve to say this is related to this prior issue due to an incomplete fix and there'll be plenty of CV records where you'll see this exists due to an incomplete fix of earlier cvu and so it's incumbent on whoever

discloses cve yeah unfortunately that's it's not always very clear um who should do that that um which is why if but if you have you know researchers and vendors have clear disclosure policies and as long as someone so if you're someone who's like a discovered on thebody want to contact the vendor make them aware of it if you have clear policies then as long as someone establishes I'm um I'll be the one to do cve then yet that you know then I can be incumbent on them to sort of do do the research and establish is this related to priv disclosures make sure that de is

clear that's us thanks again thank you very much