Incident Response in Cases of Supply Chain Incidents

of supply chain incidents and I'm not sure if you're here or not but if you have a flipper zero and trying to connect to iOS please don't okay thank you very much and let's Lucas it's yours thank you thank you hello everybody so as said I'm Lucas I work as um incident manager at Cloud flare and I'm here to talk a little bit about um what changes in incident response when we're dealing with um supply chain incidents um so the idea today is that we we talk a little bit about yeah what is supply chain Security in general um we're going to go into some real word examples that we have seen um and then yeah what where the

challenges what did you did we learn uh with those uh real incidents and yeah how can we make things better so what are the main risks or the main issues with Supply chains so I I classified the supply chains in three pillars the software as a service the commercial software and open source software a lot of the risks are similar but there are specific the details that we need to be uh careful when when dealing with with any of those um different types of Supply chains so for uh software as a service um maybe the main thing is I'm I'm using someone else's software running their infrastructure so what if they get compromised what if they are

attacked that that's a big risk that I need to take into consideration and looking from the incident response side how do I respond if this kind of uh risk is is going to happen eventually so uh going further than that is what if not only they are attacked but they are they can also be used as a proxy to attack my own infrastructure my own users um indirectly so when I'm using software as a service there is usually some kind of trust in between the infrastructures that that is needed and that can be abused by attackers and uh the the trust used to attack my own infrastructure so um also because the software is running on

someone else's environment we usually don't have false visibility so for from the instant response perspective if I'm running my own software if I have my own servers my own cloud I have in general full access to all logs I can generate the logs I can look at everything I can go deep into anything I want if that's a software as a service I don't have this access I only have whatever logs my provider is going to give me and that may be enough that may not be enough so this is uh something we need to to uh pay attention to uh and also the communication uh I I have a dependency on the provider so I need communication

with the provider I need them to tell me what's what's going on in their environment um when we go to commercial software that I'm running myself then um again I have trouble what if the developer or the company is is compromised so I guess solar winds is probably the biggest example of that uh someone if I am a solar wind user I I bought the software from them I'm using it but but they got compromised and that eventually became a problem that I need to respond so I that may become an incident in my environment and I I'm I'm the one that I need to respond to that um source code leaks may lead to uh

leaking vulnerabilities zero days and all that kind of stuff um so that that could be another problem bad updates so not only the updates that they're going to make the software fail but also if their infrastructure is up uh is compromised the updates that I'm downloading may be compromised as well so I need to be careful with what I what I'm I'm downloading and there is a really strong level of trust that I need to have in the provider when I'm downloading updates because yeah uh I'm downloading software that I'm going to run on my infrastructure if if someone is able to compromise that that they're going to compromise my my infrastructure again communication so if something

happens to their their environment if something happens to their source code I need to know I need to be able to to prepare and sometimes respond uh and also zero days it's always a problem for for instant response so things that come out of nowhere we don't know about and we start having to respond to them for open source similar but always a a bit different so they can be attacked um I I can't have bad updates someone may compromise there get Hub account and start putting m in there um new version um yeah communication is a big issue with open source a lot of times especially small projects that are a small group or one person is there

so something happens that person doesn't have the time they don't see it so they they are not able to tell me what's going on um fake packages is kind of pretty common today that uh people either generate a fake package in something that existed or a name that's similar to something that exists and then the package manager uh confusion and then it's downloaded and yeah we're going to have to respond to this incident um and again zero days may happen in any kind of uh of software so how exactly does it affect um incident response all those things so first one is logs and visibility so if I'm kind of Outsourcing either a portion of of infrastructure someone's going to

run the software for me or I'm getting the software from someone I need to make sure I get proper logs I need to have visibility into what's going on that's the only way so logs are the Big Tool of an incident responder if we don't have logs there's nothing we can do and so we need to make sure we have them um we also need to have good logs they need to be complete so this is more of an issue in uh software as a service environments that sometimes the provider does not give us full visibility into the logs we have partial things uh we don't usually have logs in the of the infrastructure so we only have at the software layer so

all all those things may affect the way we're going to be uh responding the thing that's common to every catalog I need to understand them so your incident response team needs to be able to go to the log and really understand what's going on understand what those events are and what do they mean in real life so uh a lot of logs are a bit cryptic they they have kind of keywords that are not extremely clear uh they can be short and so all this means that uh in terms of preparation for an instant response team I need to be able to understand what is in the logs when I start looking for something in the logs

I need to understand and know how it looks like otherwise I'm never going to find it uh and costs that's mostly in the the software as a service side because providers usually charge more if you want more logs but also if you are running on your own infrastructure you need to store those logs so that there is a cost that uh associated in in that um time to respond so any provider will need to or should tell us when something wrong is going on so be it a SAS provider it's even more important but uh commercial softare uh I guess yeah again solar winds is the big example people need to know what's going on people need

to uh be able to prepare to uh to roll back apply patches or take things out of the internet so all all this is is crucial so uh uh a timely response will will need a timely communication so how long they it takes to fix the the things and um yeah all all this is is part of what I need to be able to respond um again communication so not only uh do things quickly but they also communicate clearly tell me what what's going on uh at some point that I'm trusting a company I want them to tell me if they hide stuff it's going to be worse it's going to be the worst scenario that I

can uh and the worst scenario we face in incident response is when the provider hides things because then we're we're fighting not only the the attackers we're also fighting the lack of information because the company is not providing it to us so um in collaboration with the the companies and open source projects so that we can get fixes out so we we understand how to fix so that that uh they help us in the investigation they help us find what's wrong they help us find what's going on and and then we eventually uh can quickly fix it so given that um I want to talk here about a couple of uh examples that we had uh in real life about in incidents

that that we saw and we had to respond incidents related to those different kinds of uh Supply chains uh dependencies that we have from from a vendor so um the first one from uh 2022 is an OCTA compromise so it's not the one from this year it's the one from last year the one from this year didn't have time to uh include in The Proposal nor uh in in the presentation itself so uh yeah last year we got this so this is what um for me it was the middle of the night so it was waking up in the middle of the night and getting this information for the my colleagues in the US I think it was the end of the

afternoon be kind of dinner time some something like that but basically uh in January 2022 OCTA was compromised and they basically didn't say anything until the group that compromises compromised them which is called lapsus started uh publishing screenshots of octa's internal tools saying that yeah here's the proof that uh we we managed to compromise them uh from my perspective the big issue was if you look at one of the screenshots you see Cloud Flare's logo in there which means yeah well someone that took the screenshot was looking into my configuration inside of my software as a service provider and on top of that uh it's reducted somewhere in the the bottom here it's not really

uh legible but there was a um the email from one of our users and obviously it was a valid email for a valid user that was uh doing there so that's what the user we came to call user zero that's where all our investigation started so basically it started with us getting a big surprise there was no communication whatsoever and eventually someone publishes these screenshots and says hey you have been compromised and for us it was a sign that we had to uh quickly check what was going on so yeah as I said they claimed that they had compromised OCTA um so without any communication from OCTA yet we we started getting through the logs so we

have logs from the the uh software as a service console we download logs we store them in our own systems and so on so we started going through the logs um but first we found out that the logs didn't seem to be complete so there there were a few events that we could not find in the logs that should be there uh and yeah in conversations with the provider we were able to find out that uh what we could see in the console was not exactly everything so we were uh the the provider did a filter on the logs that were given to customers that was the the explanation we got So eventually we managed to get the extra information see

everything that uh all the events that we expected to see there um but there were also a lot lot of event names that were not clear we didn't clearly understand that so during the response and the house is on fire then we need to stop go to the documentation go to the support and start asking okay what's this I see this keyword here and I don't understand what this is so that's another thing that happened in the middle of of this whole uh exercise um so yeah OCTA is an ID provider provides credentials stores our passwords and yeah uh multifactor authentication so yeah what you do when you have a problem with that you rotate credentials you uh

clear the sessions make sure that everybody needs to input a new password and obviously we started with the the users that were more directly affected with from that uh incident um from all the analysis of the logs and we eventually we got the help from the the provider there was no event showing that the attacker had done anything to that uh configuration that uh they they were able to see and and the user in question was also not affected by uh by this attack so at the end it was good that we were not uh indirectly attacked but it was very close so yeah and maybe another day I can tell you the story about the

this year's OCTA compromise and this one yeah we were affected it was an indirect attack that that directly affected us um but yeah maybe a story for another day um moving on uh that's an interesting uh bug and this one is on on the pillar of the open source things so there is a very strange construct for IP I pv6 addresses is that you can map an ipv4 address inside an IPv6 one so that's RFC compliant and it exists and yeah I also didn't know about that when we started investigating this thing so basically this is the format you do you basically put a bunch of zeros a bunch of ones then you just put the btes from

the ipv4 and it yeah it's there it's uh the the way to map it's supposed to be a valid IP uh IPv6 address and it should work work so fine so what did the this researcher do so this was something that came to us through uh bug bouny so the researcher did a demo and put this so they they uh basically created this DNS record uh that's an IPv6 ands record uh the the 4 a and it's basically mapping the loop back back address so loop back address for if anyone doesn't know that's an address that uh allows me to send packets to myself basically so it's a uh a way for for us um any machine to

talk back to uh to itself so in ipv4 that's the uh 127.0.0.1 it's pretty well known but if you map it into an IPv6 this is what you get so we have a lot of systems that are written in in go goong uh and one of the the portions of goong is the net library and that Library does all kind of uh network uh stuff and one of the things it does is it reads DNS records and it um uh gives us the results of the DNS records that that it's reading so our software was basically calling that Library that's an open source thing and asking for what is the IP of exploit. example.com and the structure that was

being returned is this that is on the bottom of the the slide here but that's basically something saying oh this is an IPv6 but the string that we see is absolutely not an IPv6 this is an ipv4 so with this the attacker was able to use another problem that we had in the software because we use um block lists to prevent uh those kind of ips to be used especially loop back IPS to be used trying to connect back to ourselves so uh we don't allow any um external code to connect back to this the our own uh the same server so that's that's uh the the idea of the block list but when we

started searching for that so um because the structure was saying this is an IPv6 we would look for that 127.0.0.1 in the list of IPv6 addresses and it wasn't there because it's not an IPv6 address that's not a Lo back address for IPv6 so it was never found and the attacker then chaining those two things was able to make connections or start connections to to Ser services running in the same server so that service running on on the server had no authentication was not checking anything then they would have access to whatever they wanted which is a reality for a lot of services that uh when when they are bound to the loop back usually a lot of

times they don't have any authentication they just say okay it's myself that's good let's just go for it so yeah that's another mistake so uh what happened is well the the guy was able to connect and he was able to demonstrate that he was doing uh connections and he could iterate over the ports and and try and eventually he started using uh internal IPS like the the ipv4 10 Network that's usually used used for internal only Communications and trying to establish connections to other servers so yeah um it was pretty bad so how do we address something like that well it's an open source Library we can go into the library so we debugged it

and we found the thing we found the the open source code we look at the open source code and we find out yes that's how it's working so in this case it was considered critical enough that we didn't wait for any patches or anything from the open source project we started work doing our own workarounds to fix this because because yeah we didn't want to wait and it was then the the developers were communicating with the project and so on so that they could go through the process of of uh getting getting a fix but um yeah so this is how a bug in an open source library in a very unknown feature of Ip V4 and IP V6

and yeah we got hit by it and it was uh it it was potentially very bad so the the good thing is because our system was that was using this Library had good logs we could find that no one else except that researcher tried to do this kind of trick so it's so obscure that uh he was actually the first to have this idea so yeah in another case that's becoming a bit old but I think it's uh uh very important example so that's the log for Shell the famous bog for uh log for J so anyone here didn't hear about log for Shell no good so I'm going to go so basically log for Shell allowed you to

send a small string into something and if into a system written in Java if the system use log for J and that string would hit the log forj Library it could allow remote Cod execution so yeah it was pretty bad it was one of the few CDs that got a full 10 in terms of cgss so everybody was freaking out about this um and we looked at it in the beginning and we just thought yeah we don't use Java there is no system done at Cloud flare that that is developed in Java oh we use everything we have python we have go we have whatever you can can think of but no Java so yeah we're safe right so yeah

eventually we started looking a bit more and no we were not so we had a a ton of third-party software that the provider used Java and the provider use log for J we were using they were running in our infrastructure in our servers they had access to a lot of stuff so yeah um we had then to start understanding what is the risk that we're facing and this is one of the most strange incident response experience you may have is when instead of looking into logs you're trying to build an inventory of systems that's not what incident responders usually do that's not what incident responders usually are train to do but yeah that's what need needed to happen in this case

so go through all the inventories find find out all the vendors we are using find it can be commercial software it can be open source software whatever is that we're running that we didn't develop we need to find out if there is any portion of that code that's developed in Java and then after that well let's find out if they use log for J in any portion of that whole thing so it was it was a huge search a lot of people uh uh multiple teams getting involved and it required a lot of communication so we had to reach out to a lot of providers so some are obvious so um if you have atashian tools that

run inhouse they are Java based and this is known this is not no secret so yeah okay we have it let's go so this is an easy one but there are other systems that we had no idea which language they were used uh they uh they were written on so it's it's very hard we and you have to find someone in the company someone that's able to answer this that question and someone that's willing to answer the question so it takes sometimes a lot of negotiation to find find out exactly all the systems that you have it have some piece of a small library that they are using so this was the the communication piece that we had

to go through and and go together with a a lot of other teams to all those those uh pieces of software that could be affected by log forj so that we could fix it and on top of that fixing it is not trivial so you need to do a uh uh uh a fix for each system and it may be different so file names may be different the version of log for J may be different everything you so each system you look at may have a different workaround that's needed so that also complicates the the the the whole response on on how do we respond uh to this kind of incident very WID spread so uh I would say that

probably more than 90% of java software uses log for J or at least at the time it was uh something like that um and on top of that we had the investigation so for each one of those systems we had to go through the logs to understand if has someone exploited this and run something on my server so and and that's different for every single system the logs are different so the the way they they um put the events or the name the of the events are sometimes different so it's it's uh very time consuming and the more systems you have that are affected the more work you're going to have for each one of them um so yeah in terms of

takeaways yeah we we need inventory so that's that's true but but the traditional inventory that most companies have does not list what underlying language is was used for that system so most of the inventory systems that we have uh in in most companies would say oh I have atashian let's say uh Confluence I use Confluence but they will not go to the detail say okay we have conference and it uses Java it has whatever version of java it has all the information I need to know to understand if it's vulnerable by uh log for J uh log for Shell log for J is different story so this requires a different kind of inventory for us to be

able to respond and we need to start adding more information so that we are able to uh to understand what to do uh when it happens yeah the order takeaway yeah we need communication Channels with our providers so when this happens and I need to understand um which software may have been affected uh most of the time I'm going to have to ask the the company so the company can provide me an answer so this is another uh very important Point yeah and open source intelligence so when log forell was was the top trending whatever on Hacker News or Twitter we could see a lot of comment ments and and posts saying oh yeah software X is affected

software Y is affected so this is also input for you so that you can you you need to check those things obviously but uh yeah that's that's a that's a start to know things that may uh may have been affected so those are the uh three incidents with three different kinds of things that uh we had that forced us to do instant response in things that were basically supply chain it's something done by someone else it wasn't our software it was someone else's software that we use some in some uh way that was creating the the problem so what did we learn about it a lot of the things are yeah we already mentioned and it's easy to uh to grasp so yeah or

software as a service have good communication channel so if you're using software as a service make sure you you can talk to their security team when you need because that that's something that happened to us multiple times that you go to support you get first level support and then you go to second level support and then the guy decides oh yeah this is a security issue and he he in puts something that's the first after a while in in our last OCTA incident it took us four hours to be able to start talking to um someone in the security team so we need better channels for response to incidents especially incidents are getting faster and faster so we need faster

response um yeah uh get logs and put them offline so if you have using software as a service if your provider is is attack somehow and that affects you and you can't access the portal and you can't access the logs there is no investigation possible so if you don't have the logs on your own server on your own storage then you you won't be able to do anything where investigate anything that's that's uh simply what's going to happen reduce thrust yeah that's always good uh security practice security architecture uh and yeah monitor for indirect attacks so that's part of the reducing trust is that you need to monitor for certain actions if they happen yeah don't be aware certain

actions if they happen they could be an attack on on trying to proxy through the the provider and yeah be aware of of that and have good detections in your Seam for for this kind of of case um yeah commercial software yeah Communications is key um make sure you understand the logs that's something we suffer a lot of times that yeah we got we get we have logs yes but what are those events what does it mean we don't know we have to go study and we we lose a lot of time uh always reduce trust again and monitor for suspicious activity yeah you're running it in house you should monitor it that's doesn't really matter um with open source coms

are more complicated they have uh certain channels they use depending on the project uh sometimes they they use GitHub they may use Twitter whatever it is so understand how you're going to get the the proper communication collaborate with them uh create trust as much as you can with the the developers that's going to help um again logs have the logs but also understand them reduce trust always and monitor so those are things that need to happen for all those um categories so um how can we be better prepared for security response or instent response in in this kind of environment in a supply chain uh eventuality so yeah we talked about logs uh we talked about

communication channels we talk about inventory so we need to know what's what's in there what we have and what's inside what we are using um monitoring for sure um more be aware of indirect attacks that's uh a big one and finally contracts so if you are in a commercial relationship with a provider make sure your contracts say that they need to answer to your security questions make sure the contract says that they need to inform you when they have a security issue because that's the only thing you're going to be able to to do in terms of forcing them to change the way they approach this uh this relationship so if it's in the contract well if it

doesn't happen at least you have a contract you can try to enforce um so yeah uh anyone wants more information about the three examples I put here uh we have blogs about each one of them so that that's the details that uh we more more much more details than what I share here today uh on how those uh uh incidents happen and how we did to uh respond to them um and yes thank you everybody and you have my uh social media handles here if you want to get in [Applause] touch thank you Lucas any questions we have a few minutes for questions over there a speaker will be with you in a minute hello great talk lucus um so I

would like to it's more of a statement than a question is that as you said inventory of knowing what technologies are being used is very very important and many of us here that work in security have seen the let's say rise of s bombs software bill of materials so um how do what difficulties do you see in like it seems like a no-brainer oh everybody should have an s bomb but what what do you see in terms of difficulties in implementing this thoroughly so that you have like full inventories of every single piece that's used okay uh that's going to be about a previous live that I used to do vulnerability management but but yeah um it's it seems trivial to

generate as bombs for one project when you have a few thousands of them then at scale the problem may be may be harder uh so a few things that I've seen in terms of generating as bombs is is a bit when you have too many projects well volume uh and what do you do with them after you you generate them uh the second issue is uh support we had a lot of um tax stocks that were not fully supported by by the providers we were using so you end up with either ignoring some of your projects because they don't fit what the provider can do or you end up with a patchwork of providers and then you have all the

problems that come with that that you need to start adjusting the outputs and uh things don't really match so um yeah esoms are are a nice idea nice solution but as with any technology at scale it becomes more complicated so uh the the second thing about s bombs now a little bit more into incident response is that especially commercial providers they don't give you anything so in some of those cases log for J for example it's extremely hard to get those companies to tell you if they are using Java and if they they're using log for J in their software they they they just don't tell you so if they and that's the minimum information that's not even a full as

bone so they they don't want to provide this information that's that's one of the the realities um on top of that when you get software as a service they even have another excuse not to provide it because they said you're not going to run the software you don't need to know what we going to take care of it and and uh if you have a a SAS provider that is is affected by log for Shell well yeah it's going to affect you as well so but you don't know and you're never going get an s bomb to be able to find it out so yeah uh the the the problem is how we use a

good tool in in a way that that makes sense in the market and uh that is accepted by by the market so yeah th those are the the the things uh in terms of of S bombs we we nice if we had them but I don't think we're there I don't I think it's very very difficult to get them they'll have a few ways to go until it's widespread thank you anyone else here the front question thank you uh great presentation um I would like to to ask you in the Second Use case you showed um where you you had to kind of develop your own um your own solution your own fix before the goang

developers what kind of things can you do since you probably when you build your software and everything you are going to to collect the the leaves and everything from the the official repositories what what kind of things can you do um in a situ like that well it will depend on the on the library you're using so in that case it was a network thing about uh reading IP addresses and understanding how to parse IP addresses so you can handle this by either not using that for parsing or one thing uh the first thing we did is that we instead of having uh separated searches for IPv6 and ipv4 we just made one big list and we search

everything that that kind of solve this issue but it doesn't solve any every issue you can have on on a library and uh as with software it's going to depend on the case so there may be other bugs that will affect you that um will require another kind of of solution so I've I've seen libraries for example that have um a regular expression Deni of service if you give them a certain regular expression they just run forever they never finish and your software is going to be there and eventually crash so the solution for that is completely different than a solution for something related to networks so uh the the the thing about software is that can do

almost everything but also it's it's complex so there I don't I don't see and I don't know any one solution that you can use in in every case it's it's each case you're going to have to do the one thing you can do in terms of responding when it happens is have good logs make sure your software outside of the library is logging what you're doing and then yes you're going to have information to debug and understand what the problem is that's that's what helped us in this case so we had log saying okay I saw this and then the result was this so it it was trivial for the developers to understand that whatever

they were getting from library was strange I'm not going to say it's completely wrong but it was strange was not what they expected in this case so then they were able to find where the problem was and and that is probably true having good logs or making sure your your own portion of the code have uh provides good logs is going to help you a lot thank you very much anyone else okay thank you Lucas okay thank

you