The monsters in your software supply chain that traditional app sec tools can't find - Matthew Rose

yeah yes I do take things quite literally well thank you everybody I know uh we just all or most people probably had uh Mexican tacos so just for uh housekeeping women's rooms that way Men's Rooms that way uh post your uh taco truck because we know how taco trucks can come back to haunt us in certain ways but I Gress again thank you everybody coming I'm Matt Rose I'm the field ciso at reversing labs and today's talk is entitled as you can see behind me the monsters in your software supply chain that as scanning tools can't find this is something I put together and if anybody seen me present before this is about 40 slides we're going to try and RAM into 20 minutes uh lots of bullets actually it's the exact opposite I like to kind of communicate things in a way with visuals pop culture references analogies um it's kind of like the way my my brain kind of works like family guy so it's always referencing things and uh using strange analogies so let's jump into it I'll give you a little information on who I am exactly so again we're already starting with the uh the movie quotes and everything um in a smaller audience I usually query the audience but I'm hoping at least 50% know this is airplane but my background uh which was mentioned I've been doing this 20 years I was actually started in the ABC space uh in the middle 2000s I was actually one of the first sales Engineers for Fortify software when they came out uh and introduced one of the first uh Enterprise SAS products or static application security testing um it was an interesting time because I would go into to talk to people about scanning their code for vulnerabilities and at that point scanning your code was a weird concept people would look at me kind of funny like a dog watching TV cocking their head going you want me to do what to my code that sounds really perverse you want to do that that you didn't even take it out to dinner first and you want to scan it uh I have an IPS I have an IDs I I have a firewall everything's fine well as we know application security is now a foundational piece of any security program for DOD civilian agencies Enterprise financials so on and so forth uh had pretty much every uh role uh in application security from consultant to sales engineer to sales engineer director done a lot of things I've been quoted in probably over a 100 articles right now and I do a video glassboard series uh called reversing glass a lot of people know me for my glassboard series it kind of came to fruition with the pandemic where I couldn't be in front of people anymore so I bought a tool that was designed for teachers and the rest is history but the big thing just to set the level uh in this movie is airplane and what's the famous line from airplane don't call me Shirley well don't call me Matthew so everyone calls me Matthew is it matter Matthew Matthew is what my mother called me when I was in trouble and I heard it a lot because I was a little bit uh uh Troublesome as a child so if I hear Matthew I kind of think my mom's coming after me so please let's just call me Matt well let's jump into it because we this is a shorter presentation I usually do this for about an hour with a lot of audience participation um definitely going to be around for questions afterwards come by I'm outside with reversing glass with my other friend Matt and his name is Matt Reynolds and I'm Matt Rose so just ask for Matt are and you'll find us uh so with that we'll jump in what is changing in software it's not a presentation at bsides or any conference without some metrics uh I'm not going to read the metrics to you but you know if you don't put metrics or or statistics in your slides you're you're deemed uh insufficient you need to back your your statements up but basically what this is saying is software supply chain security risks are on the rise the reason being is the nefarious dudes as I like to call them have figured out that they could try and hack every one in this room individually through social media or through reverse engineering or doing all these type of things but they found out it's much easier to take the longer con hack the software of the application itself and then let normal distribution processes go out prime example was solar winds they hacked solar winds it was a hack of the MS build environment I'll go into this in a little bit but that was distributed through a normal product update to 18,000 individuals yes it was a little longer con but the benefit was so much more but what we what really has promoted software supply chain security well it's the way software is being developed and again I'm about visuals and references so think about software development 5 10 years ago plotting waterfall development You released every three months it was like The Flintstones I mean it was the if we look back the Stone Age way to do something and Fred and Wilma and Barney and the rest of the family the biggest change I see here is that the dog or for the pet is Dino the dinosaur and now we've moved into modern cicd development uh microservices architecture Cloud development and we've moved to the Jetson this is the same entity the same kind of footprint but much more modernized I'm still really disappointed that the flying cars haven't come to fruition but the flying devops pipelines releasing sometimes a thousand times a day is real and I think that really ties directly into the Jetson as analogy and again for analogies let's jump into top I always like to kind of paint the picture and if anybody yells out anybody know what this movie is anyone you you got it the name of the movie is gung-ho and I think it's applicable to software supply chain security when you hear supply chain your mind a lot of times go to like supply chain in terms of a physical assembly line and that goes to cars and hey there's processes changing well gung-ho it's a great cult movie it's a little bit obscure but that's exactly what happened they were building cars the American way it wasn't going well they got bought by a Japanese uh company and all of a sudden all these new methodologies and processes were introduced so to kind of bridge the gap from an assembly line to Modern software development and software Supply chains I thought this was a great example an obscure example but I think it fits pretty well and by the way it took me a long time to come up with this one I tried so many different analogies so hopefully uh if you have a free moment check out gungho it's probably uh buried in your Netflix somewhere as a ancient classic at this point so as we move forward whoops I went too fast there another analogy and this is where I get into it how many people when they were sick as a kid and I know there's probably some younger people in there there were two things you watched when you were sick as a kid there was The Price is Right with Bob Barker rest in peace and then Days of Our Lives like Sands Through The Hourglass these are the days of our lives but The Hourglass is a great representation of software development and and you're probably saying what the heck are you talking about well if you take this hourglass again I think it's the most famous hourglass out there there's many different movies you could use but again I want it to be obscure and unique we take this on the put put it on its side and you think of everything left of the middle of The Hourglass is pre compilation and deployment activities everything to the right is deployment post compilation so as you're on your left you have your IDE environments you have your code repos get in this example you have your binary repositories and I got off the thing you have your your tooling your CI orchestration you know Circle Ci or Jenkins or whatever running your CI orchestration you have your build environment this is a representation of Ms build and then lastly you have your open Source repositories npm piie whatever it is guess what that's a lot of crap right there that's a lot of things that has to work perfectly every time to ensure that your software supply chain is working correctly so you can use the best tooling out there and I'll get into this in a second it's a question of to eror is human there's so many things happening in modern software development processes in a devops ecosystem I challenge you to find one person that knows everything it's it's a house of cards and that's why offer supply chain security is becoming more and more important because they're the the nefarious dudes are banking on just speed scale and confusion and then what do you do with this after you had hit that inflection point that that uh uh Golden Disc if you will in the middle then it's deployed to a cloud environment you go into a container you go into a data center again all these type of tooling Cloud you know this term posture is floating around the industry Cloud security posture management application security posture man the posture the posture posture the posture how good is it how bad is it again we got more things we got to do when you're talking about supply chain security or software Supply CH Curry there's this inflection point this this significant and very very small moment in time post compilation predeployment you're creating a package you're creating an artifact a dll a war file an ISO whatever that is that is the thing that people you're you're trying to create and deploy that's where you test and I feel is the best way a test for potential risks in your supply chain think of it as the final exam the sand goes through The Hourglass that's the final exam it's easier to look at the grain of sand as it's going through that inflection point than it is when it's actually deployed out to the world for everybody to use or everybody working on it in terms of the source code first-party code third party code costs so on and so forth so as looks or application security testing Technologies I like to use the acronym star a because we have s asked I asked and my favorite one today is yast yet another security tool so we're in a stage where we're looking at the components not the entire entity and you know what all these tools are fantastic uh I've been in the space a long time they all look for a lens of risk that's important but they don't look for a comprehensive list so we're going to use another one of my my crazy analogies we have apis and that the identified risks and with time I'm not going to be able to go through them but these are the things that API scanning Solutions this is a newer technology in the because guess what applications are not unto themselves they're not an island they're an interconnected Network through apis everywhere so that is a lens of risk it's very important the second one is the source code itself this is where my uh career has really taken me is around sast and static application security testing scanning the uncompiled source code for vulnerabilities that kind of line up with the OAS top 10 SQL injection cross- scripting so on and so forth again important then you have open source SCA has become a a big uh solution and very important where you're talking about the open source packages and the thing that I find interesting is you read the industry analysts they say 40 to 80% of all modern software and applications is open source I agree with that but that that is a huge margin of error and even if you're at 80% you're still missing 20% of the first-party code in the other pieces so just thinking that your software supply chain security program can exist just with SCA it's important to identify risks and upgrade and uh licensing issues within uh your open source packages but it is not a comprehensive complete solution we have the running applications when you're testing again for the same type of OAS top 10 top 10 issues but in a running application compared to static code and that's D Dynamic application security testing or the manual pen testers that are trying to use their tricks and their their Tools in their toolbox to kind of break the application of find vulnerabilities you have the QA environment again there's a lot of things happening here QA is leveraging Ias in the functional test to point out a new lens of risk in a running application from the inside out and then if you move into the production environment there's always runtime application self- protection that's a mouthful but basically taking that firewall aspect and putting it in the application or the app server itself but you take all the robots anybody know who the robots are nobody nobody's a Transformers fan trans but okay that's there there's a lot of Transformers we got to be more specific how many people have been anybody been to Universal in uh Florida the Transformers ride well they've really up their game in terms of representing this these are the Constructicons and the Constructicons are you know was kind of like uh Deadpool where it's like five baby five Lions become one super lion while five robots become one giant robot and this is Devastator Devastator is your compiled package and if you look at this all of these things that you're looking for when you look at the complete package the first party code the third party card the open source code uh the the uh Cod space uh components in your application this is where malware behaviors tampering all things that are unique to software supply chain security come into play and if you look at each of the little robot you're never going to find these things because you're only looking at a piece of the puzzle it's only going to provide risk or a lens of risk we have some stickers over there a couple people asked me what they meant and it says vulnerabilities are overrated vulnerabilities are important but if you're looking at software supply chain risk it's malware and the difference between vulnerabilities and malware vulnerabilities is something was written to do something but it does some bad things malware was just written to be a bad dude and that's all there is it never had a good purpose it was just made as malware so thinking about that look at the whole picture Devastator instead of the individual robots so if you don't know what to look for you won't find it this is my friend Wilson the reason why I bring Wilson up here is nobody knew where to find Wilson and Tom Hanks they were crashed on a desert IA they're looking in the wrong spot the thing was soft for supply chain risk is it's typically tied to malware but I'm going to let you in a little secret it's not as easy as grepping the codebase or the application architecture file for malware dll it's not that simple most attacks there is a huge repository and known bad malare out there but most of the attacks uh which I'm going to get into uh solar winds 3cx Circle CI code COV all these things are novel attacks they're new but guess what they do kind of the same things they have the same DNA they have an escalation privilege they have a porter socket opening they have some sort of uh new capability into a file or an API that you've never seen before so you have to not just look for the known which is again malware dll you have to look for what are the breadcrumb trails put on your your Sherlock Holmes hat and do the detective work to say what has changed and why does this not pass the sniff test and when I mean the sniff test when your your husband or your wife or your boyfriend or your girlfriend or your your best friend is like hey does this milk smell good I'm like dude why are you giving that to me if it doesn't smell good I don't want to smell it but that's what you're doing with this type of breadcrumb Trail you're letting you're doing it so you SM pass the sniff test so let's get into the meat of it this is my buddy Sully he's actually a nice dude even though he looks uh a little bit intimidating here but these are the monsters in your software supply chain that all these SCA tools can't find the first one is 3cx people familiar with 3cx anybody raising their hand yeah so 3cx it was interesting because 3cx started to to think they had a problem because a lot of their customers were reaching out to them saying something is off with this latest release we're getting these weird alerts and weird reports that something is happening and they're like huh well they scanned it with one of the um most prominent virus Solutions in and out there which is virus total and it came back clean well it was a very very unique approach where it was a signed package but the signing process itself was compromised and the malware was slipped in at the signing process so it was a signed package it looked legit but it was actually compromised that's something that all those as tools could not find this is a tampering of the signature itself probably the most famous one is Sunburst solar winds again this attack and I'm being very simplistic here was a compromise of the MS build environment so if you think of that sideways hourglass that inflection point it this malware was inserted right at the last point the piece that actually built the software itself so if you scan the open source code the first party code did all your testing on this you would have never found that because it was just basically inserted into the build and was never part of the devops pipeline code COV is what I like if anybody remembers the the show The Americans the 1980 spies that were entrenched in the US this is pretty much the same story they were always trying to put a bug in a phone or a bug on a computer and again the computers were uh not very uh uh modern or up to dat a developer for code cob was compromised where they were able to access their credentials and insert it directly into the code repository and then just build it as part of the package itself so again a different lens of software supply chain risk something that is not a vulnerability it is a compromise of the devops or the software supply chain to insert malware in New in Devious ways and then the last one was around Circle CI Circle CI again is a predominant CI orchestration system where it was compromised and it affected the secrets and if you're not familiar with secrets secrets are the thing that accesses the functionality or the important information it's the key to the lock to access that and a lot of times people have programs or uh policies to update secrets in a in a consistent rotation pattern but just like updating you know to the latest version not everybody does it so again now we're talking about Secrets compromise as part of access to data or access to functionality so software supply chain risks you need to solve there's always the risk and compliance you know the executive order 14028 talking about self-attestation and sbom and then there's the NIS standards you have the software and development release uh process which is associated with the open source packages you know an mpm piie GitHub devops tooling the tooling itself the robots that build the assembly line or the CI orchestration like Circle C that build it uh you also have the It software procurement um in I going to buy this piece of software and this is the story I always like to kind of talk to people if you're going to buy some software you're a consumer of software not a creator of software and you probably have certain checks and balances and things you do here's a security questionnaire fill it out I mean honestly please raise your hand if somebody got the security questionnaire back and it says yeah our software kind of sucks it's riddled with holes please uh you know we promise we'll do better in the future but just buy it right now because we don't have the money to fix it those questionnaires are always going to come back clean we did all the best things you think solar winds or 3cx or any of the other people out there going to say their software is bad you need that final exam that test to prove that the que