BSidesATL 2020 - Connect: Software Supply Chain Threat Detection

hey bees Atlanta this is Mike Doyle I am a principal consultant with synopsis um I guess a little quick introduction of a synopsis synopsis is a big software company they make software for people who make hardware but we are security consultants um they acquired our company's digital about three s years ago so we still do the same work that we did before I guess yep yes hi everybody I'm the Istana Josh I work as a security consultant at synopsis Atlanta and being a technical consultant I worked on various application security projects that include source code review web app and network penetration testing malicious code detection part of which we are going to talk about today so

without further ado let's begin our presentation right so this morning if you were in this track and you saw Eddie Glenn's talk one of the things that he said is that like nearly every business nowadays seems to be a software business [Music] certainly the case already was talking about a rather similar topic he was coming at securing software supply chains from a code signing perspective we're gonna talk about it a little bit different we work with lots of organizations that make software and like Sean Marshall said in just the previous tract here or the previous of the time period almost all of your code was written by somebody else somewhere else so when it comes to the software

that's getting made um how trustworthy are the people in the organizations that make it what we're concerned ourselves with so we're gonna describe the problem a little more depth examine the motives that malicious software developers might have and then we're gonna take you through our process for discovering attack vectors I'll tell you how to take action for these sort of things and then we want to wrap up by talking about who can detect these sort of threats all right so I've got some new stories here on the on the screen if you take a look at and you know but feel free to like you know google these later or something like that but the the one in the upper right

you see from security week is a story about a contractor who developed a set of Excel spreadsheets with like you know fancy macros and stuff for his his employer was Siemens a big manufacturing firm out of Germany so he built these macros in such a way since he was going to be the person responsible for maintaining them on a contract basis he actually put like logic bombs inside of the Excel macros so that they would break themselves every so often just to give him a chance to come in and make some recurring revenue often down in the lower left is a story from the register um about a land of these contractors a based right here in our

city who attacked the United States Army to to the tune of 2.6 million dollars in damages and the story down in the lower right is about a direct employee for you yes he was disgruntled over a lower than expected bonus check he only got thirty six thousand dollars in his bonus so he develops a set of logic bombs that he had deployed to over 2,000 servers he left the company went right to his broker and then shorted their stock so the the gist here is that we we placed a lot of trust in the folks that make our software certainly besides al-qaeda would not be a conference today we wouldn't be conferring without software we're very

dependent on it but the trust we put into the people that make it could very well be misplaced

so what is software supply chain threat detection firstly it is not same as malware detection traditional malware is built by an outside attacker to compromise systems this is an emerging science to identify groups of suspicious functionalities and mal code that looks like regular code but it is built and inserted by developers or an insider who is internal to the surface applied chain we could also take this as a spinoff from regular source code review but the main difference is that the focus is not to just identify implementation bugs and design flaws rather the focus is to analyze the business logic from the perspective that is my production deployed source code secured from an insider threat another main difference

is that we do not just analyze source code files the input we consider for this kind of section is a binaries from the production environment and why do we do that let's take an example there is an acting which is built a source code it has undergone a source code reviewing the development cycle there were bugs that were fixed in the development cycle itself and there was a green signal given for releasing the source code to production from the security team and the production and from the senior management but a couple of weeks after the production release somebody identified a backdoor from a bug bounty program this is not very uncommon and this is because malicious code can be

injected in the executive as late as the final production build so it could be anybody in the internal surface applied chain who could possibly be a threat in this case we are focusing on the software acting the next question is that of mostly for people you know party can app team or an organization already so could reviews our perform vulnerability assessments are being done do we need another shred detection as a part of the pre deployment process and to answer this question we can ask us as a few follow-up questions starting with is the software outsourced if I have acquired the software through a merger and acquisition kind of an activity do I know if my software is healthy or not if

it's outdoors and if the development team on site or is it offshore and what are the rights do I have on the binary packages so all these questions basically give us clues on what could be the motive behind such tricks in the software supply chain so a couple of aspects about about this first when it comes to attribution more times than not when these sort of problem literacy constructs find their way into software into source code it comes from outsourced contractors more typically than disgruntled employees the examples that we gave at the beginning of the presentation you know the three examples two of which were were contractors furthermore the more socially distant those contractors are which is to say

like you know the longer the supply chain the the more likely it is for a for development for a dev to uh I guess go rogue and be compelled to make these sort of problem areas uh something else I want to talk about those that we can't prove intent just by looking at the source code a construct could come to us we can find it it could be it could block accidental it could actually be an accident it might be a lapse of training or it adherence to standards like coding standards a lack of business knowledge that sort of thing [Music] also a developer who wants to evade punishment could set something up to make it look very accidental even though

it was actually malicious now when we get to one of our later slides when we talk about what to do with the results we can talk about additional steps that you can take to sort of who doubt intent but we're typically talking about developers or IT folks or you know now that uh you know like your network is code your network of software on anybody with access to configuration design and build files would would fall in line for something about this analysis so the next section will be taking examples of real-time disassembled packages which we all know show up to the met words and so the source code files so this is an example send email classifying and here

there is a regular email method for sending out an a new PR total is falling out it's calling networks for adding senders for adding the recipients as well as setting the subject and finally sending the email so as I said it looks regular but something is a little suspicious about the construct and the method here and that is a hard-coded email ID is added in the recipients so this leads to few more questions that why is there a hard-coded email ID in the recipe is this an automated email ID or does it belong to you know is it a personal email ID and if it is personal is it somebody internal to the organization or

external to the organization what are the contents of the email that is being sent out and what is the purpose so just by identifying one hard-coded value we can build up a case and do a dataflow analysis to see if there is something for the malicious or North or maybe just bad coding practices there is another example in this particular file it's called normal Java the name itself looks like something is not correct or probably we could sign it off as a dummy files that not called but it's just added in them in the binary package and honestly this kind of finding can be identified in a regular source code review so we have highlighted the

username and password that is hard-coded here but just looking at the source code we could just qualified a qualified is finding as hard-coded credentials in the source code file when we perform something like a malicious code detection on some supply chain thread detection we need to go one step further and investigate the config and the deployment files as well to build up our cases to make it stronger and here we see in the web deployment files web dot XML the method that is used here the class know auth filter it is defined in the deployment file in the bed dot XML file so we know for sure that anybody who is using these credentials username

as user name and the past default password they can bypass the regular authentication method and entrance to an application this next example the malicious construct is right at the top it's actually pretty easy to spot and it is basically a form of command injection we take a get request parameter name folder and then just execute it on like a Windows system this might open up a directory or something like that but of course if you put in a directory name that doesn't actually exist on the directory name that is a operating system command then we'll just run whatever command you want delete all your files copy files crossed with another system you know Bob is your

uncle in this case this is something that definitely would come up through a normal static analysis or just a normal color view or even a penetration test probably even a dynamic scan or something like that but what is of interest here is this based off of the real finding that we had on them assessment was a problem that we found built in into a separate jar file so sometimes you know developers have access to code bases late in the process later than 10 should typically happen in SDLC or you know a developer will include something else from somewhere else like Oh Shawn Marshall said in the last session almost all of your code is written by somebody else so our analyses

will be able to find these sort of mistakes that are basically have propagated through the supply chain to you and we have a last example for code snippet and this is from a manager payment class file this method is for basically processing the pavement for Account Managers who basically one of the senior managers and a simple code again which includes the branch number where the payment should be processed for the account manager and the payment date typically we get a get out payment after every 10 business days and that is what the second last line here set payment date is used for for setting the payment after every 10 days but as I said it looks regular and at first

glance it is not that easy to catch this finding but if we take a closer look something is not correct and a taxi also when we do these kind of stretch attentions and it's on the production packages we may also have a background story on whether something is wrong in the in the payment systems and just looking at the source code here we see that this is not right this is not for 10 business days there's a Tilly in the payment processing so just to explain one day has 86,400 seconds so ideally there should be three zeros for 10 days but there are four zero so it is 8 million six hundred and forty thousand

that's added so there are some suspicious intense traffic and build from here one that it's done intentionally maybe a disgruntled employee who does not want his manager to get his payment on time and it's delayed up 200 days or secondly or it could be an it could be accidental a typo an additional zero and it was not checked correctly and third would be that it is accident it's a combination of accidental and intentional so first glance accidental seconds or manager complained that he does not receive the payment on time and so he gets a payment after every 10 days from the finance system but we using the logic from tow from the from the source code on the

binary package he also receives the payment after 100 days so basically he's not reporting that he's getting paid 2 times for every pay cycle so these are some examples from from your attempts of binary packages that we've seen now let's look at the methodology how do we go about identifying insider trades in the software supply chain just looking at the source code this is more like building up a story we know that subset here is the software acting who has access to the source code file they are building the logic and knowing the thread the first area would be identifying key points of interest which are basically atomic elements that people also seen in the code could be

hard-coded values like a hard-coded date hard-coded email ID a URL or it could be using stealth methods which ideally should not be included in a source code file so we identify those key points of interest next from the point of interest we check if there is obey that we can flow from those elements to a method that is being called for building out some suspicious methods some suspicious constructs so join the dots between the point of interest to the methods that are being called if we can join the dots then let's go one step further and build out it intent that here there is something suspicious it looks suspicious but is this intentional or is this

accidental it could be that the business it that there is a business justification so it's not actually suspicious and it won't qualify to be an internal thread or a malicious code and finally if we know that it is malicious then we take an action which is which she'll be calm except but it's it's the final step in this methodology so this is manual and performing detection would stay man you know doing something manually will always take time so we need to automate and we have automated the first two steps that is by building out signatures to join the dots from point of interest to suspicious construct we can write automation scripts to in terms of regular

expressions and custom rules to refine the dataflow analysis and here we have an example of classic malicious code that is inserted by a development in a backdoor and in simple terms backdoor is nothing but an alternative non-standard way of entering an app or an infrastructure and organization let's take it as somebody entering our house not from the main door but from a window that is kept open so same thing happens in applications and IT infrastructure as well so a developer could use a stealth test utility to enter an application through using a communication channel which gives way to and inversion of control operation it looks like he is that regulation it could be he or she so

basically developers using a functionality to build a command which looks like a which looks like a fan commanded paying bills using static parameters because many sub methods are being called and it looks until most surreptitious because of the way it's written but actually he is building a command dynamically which can be executed on run and that could open up sham axes and we all know if there's a knack to that open any malicious attacker could get complete control on system so what do we do next once we know that there is a malicious intent in the system here are some examples take no action delegated passive monitoring active monitoring and take immediate response we would rather

have this as an interactive session but since we cannot interact right now we can just go over what what should be the steps and there is no wrong answer for this all all these steps to be taken but it depends on the scenario so first thing is that we do not take any steps that is it could be a false positive there could be a business justification as I've mentioned earlier so it's not a malicious code it's not an insider threat in the surface of blanching the next is it it's not a malicious code but because we review source code files we do often find implementation valve is designed close the typical source code will be finding so instead of taking it

further to to the next escalation point we'd redirected so you give it back to the source code review team and say what we have identified the second and third step would be for for finding out for finding out actual malicious intent so let's say that there is a possibility of exfiltrating data to an unwanted system or unauthorized system but the data may not be extremely sensitive or critical so we could have passive production monitoring by implementing some web application firewall rule adding some sensor and production logs to check whether data is actually being exfiltrated or not and if at all to whom it is being sent and accordingly take actions whether to for the block this

exfiltration or not if there's a possibility of sending out or somebody being able to access in some sensitive information then best is to take the active approach and have a compensating library in place or an active bathroom to block the access at the source and final would be to take an immediate response in terms of escalation and getting the legal and the HR team involved that is this we can identify the culprit and it could also and it's genuinely there is an you know there's an internal culprit and disgruntled developer who is who is probably inserting a logic poem or your backdoor it could also lead to his work domination of domination of contract so

we have the results we have gone through the methodology let's see who could perform this kind of detection method on this third detection here we have a developer a hacker and a security engineer again I would have loved to have this as an interactive session but there is no wrong answer again at the end the basic skills needed is you should be able to read source code the programming languages and a developer hacker security engineer would all fit into this bucket but a developer or typical developer would rather spend his time developing his application not too worried about the security inflicted implications while developing this office he has a timeline to complete and have the production have

to app up and running hacker hmm always busy you know adding something malicious or you know an offensive attack so you may or may not want to consider security I am an engineer it is a part of his job to identifying something something malicious but oftentimes security engineers work looking at or you know just just walk out of a checklist and if the complete our checklist we may not consider a lot of custom activities so again as I says no right or wrong answer but we would like somebody like a deaf actor to perform this cancer detection so a combination of a developer and attacker as efficient a smart developer with secure coding skills and mind up an

attacker with this and try to answer some questions [Music] minutes if you see any questions Oh everybody's asking for your slides which that's pretty normal so if you guys want to provide your slides everybody always wants that and then note to all future talks don't meet yourself on June because every time I touch software embrace because something was funky on on said that like only affected me because I put the other way or something it happens like when he says I hate computers all right well thank you we appreciate your presentation today it was great information thanks for supporting besides with Nana and if you guys have if anybody has questions for them they'll be in the connect tract or you

can message them directly and talk more so thank you again all right thanks for that thank you