Supply Chain Dangers and Disruptions

get started so glad to be here good to see everybody here it's good to see lots of faces from st con and other conferences we've seen this year all right i typically start off my meetings at work with some type of joke so here you go what did one dna sequence say to the other dna of sequence do these genes make me look fat so if you got nothing else from today at least you can take something home to get a smile out of it so who am i my name is brandon benson i've been a cispy for a really long time i currently run a sock for one of the silicon slip companies i've been in cyber security longer than

i care to admit so um i definitely am a curious learner i think this is a great field to be in if we want to continue to do to do that so what have i done over the last 15 plus years i've worked with lots of companies to secure sensitive data either encrypting it or auditing it or working with them to make sure their information is secure i've investigated breaches from both internal and external sources i've broken a lot of stuff a lot of stuff and i've fixed a few things too so why this presentation so a lot of our jobs still in the cyber security space it's the cyber footprint for all of our

employers i think in a good many cases as i work with as i've worked with different companies in the world i've noticed that we do a really good job of implementing and coveraging covering risk and mitigating risk for our internal networks we do a fairly good job in most cases of segmenting if you've got to do any type of compliance you do a pretty good job of making sure that patches are somewhat in place you've got a risk management program you've got some of the basics covered you get robust access controls in place you have your active directories and stuff like that unfortunately attacks continue to evolve and we've seen that i think over the

last year and attackers continue to evolve with those attacks to develop new techniques or maybe old techniques we'll talk a little bit about that and so i don't think it'll ever end for blue teamers and i don't think it'll ever end for defenders so this year probably last year last december i think was the biggest one we've seen where you start seeing supply chain attacks in the cyber space not just in the physical space for me a lot of supply chain seemed theoretical until i saw it kind of hit home so supply chain attacks when i thought about it if you'd asked me three years ago i'd say yeah i've done audits on systems or in companies where they take

sensitive hardware and i go through with the company i walk through from when the company receives the hardware into the warehouse how they send it to a secure cage to flash it with the encryption keys and stuff like that how they track it from the secure cage out to the loading dock track it from the loading dock to the end user at which point it's implemented and that could be an hsm it could be a pos device and i would watch all of that and try and determine where people could break in and add code or add their own firmware or hijack a truck to steal it so they could replace it with other things other types

of supply chains i've looked at would be things like okay i'm going to hijack a semi-truck i'm going to put in a bunch of like fake gucci purses so they get the retail stores and get sold for a whole lot of money things like that but i don't think supply chains in the cyberspace is new a few years ago like in the early 2000s western digital which is a hard drive manufacturer ended up with a problem with a whole batch of their hard drives that came out of asia what happened was somebody actually took the hard drive and before as a hard drive was flashed in the bootloader section so the part that doesn't get

re-imaged when you re-image a drive they put in a piece of malware that would end up stealing data western digital found out about that that ended up happening in their factory and they had to recall a whole batch of hard drives and then fix the issue other things i've seen kind of in my career is pos pos devices swap so credit card fraud has been a big deal for a really long time and so we see stores that you'll walk into a store or a 711 and you'll see people put cameras on pos devices or you'll see them swap out pos devices with compromised devices that look the same or you'll see them put little skimmers inside the devices so

they can steal your credit card data and so that type of supply chain attack has not been anything new the pci council came out and put guidance around merchants to say check your devices if they seem like they've been tampered with you should probably get them replaced so shipping industry mercantile none of that should be new as far as supply chain one of the intriguing ones was a few years ago at defcon a security conference somebody actually placed malware inside the cd and one of the programs so when people got home and opened the cd they actually ended up infecting their computers with malware however there's a few that are probably worth talking about that we've seen

emerge over recently and we'll go through two or three key studies to see if we can learn something or see if we can implement controls or recommendations to make our environments a little bit more secure uh before we start though like it's beneficial to understand a typical attacker methodology so most attacks follow four general steps there's typically some type of reconnaissance well let's see if this video will play so it's some type of thieving of data and you should say some type of crazy thieving of data so this is an office i work for on occasion to sideline that person who just walked out had walked into the office um it's a therapy office so the

therapist was there they left their office for a minute that person walked in what they had under their um coat or their sweater if you'll notice is a little bricky looking right because they just took the therapist laptop and her purse and her little booklet and stuff like that i got called in as soon as we found out we identified it on the camera and then we went ahead and contacted law enforcement so this is here in utah then we were able to track they lost their phone as well so then we were able to use some of the tools on the phone and the laptop to find out where this particular individual was met the police officers down at a local

mall as they were trying to spend as much money as they could on the person's credit card and ended up actually getting that person arrested so that's the type of supply chain attack we talk about it was fun it's a little exciting to be able to be part of that but in many cases what attackers do in the cyber world is they'll do reconnaissance just like anybody else you see strange people driving in your neighborhood you see attackers probing your system trying to identify what software what applications you're running they'll look at your website to see what vendors or suppliers you use once they figure that out then they'll try and figure out how to execute or get

access to your system in some way once attackers gain access they'll deliver a payload to try and steal as much data as they can as in as little time as possible without being caught of course they'll use vulnerabilities to use social engineering they'll use suppliers they'll use vendors they'll use a whole bunch of things and then they'll go ahead and steal that data and exploit it and then elevate privileges install persistence and then they'll take whatever they can typically as a last resort if there's nothing interesting on that system it's not uncommon to see people place crypto miners it's also not uncommon and this is why we kind of talk and harp to like our

friends and relatives don't download unlicensed software and stuff like that it's not it's not uncommon to see malware placed in those programs as well that either place crypto miners or credential stealing programs or anything like that in place let's talk about a few different types of attack vectors i think are worth mentioning third party software vulnerabilities so a year ago a huge one came out and i vowed not to use it because it's been talked about a lot so i used a different one so in the last 12 months there was a ticketing system that's used both on internal ticketing systems for companies as well as for companies to use for customers their customers to

issue tickets when there's a problem that tickety system came out with the vulnerab critical vulnerability that allowed an attacker to bypass system authentication through remote code execution vulnerability so the ticketing systems many of those were actually publicly exposed and within a few days and this is one of the points that i'll bring up within a few days of that that vulnerability being released attackers had developed a proof of concept code so they could actually take an attack and exploit that vulnerability much or very much within or very much before the 30 days that you typically see compliance [Music] groups actually say you need to patch right so i was driving to work one morning i heard about this in one of our

news feeds i subscribed to and then based on that as i got into work i worked with my team and said we need to go find every system that is this product that is vulnerable and then we need to make sure that one we haven't been attacked and two we make sure it's patched or mitigated in some way right in our case we had sawed several different probes as we looked at it it was if we looked at the timeline of when the exploit was made publicly available it was probably within two hours that we saw probes against our systems or our environments and then from there we ended up tearing down a couple of those servers

rebuilding them and then for the others that had not been accessed we were able to get mitigations in place to prevent the attack and so i think it's very much the third party software vulnerabilities are all over the place earlier this year apache came out with a remote code execution vulnerability as well last december there was a huge one that came out so that's definitely something as far as that goes but how do we how do we prevent this well many times we can't we rely on these third-party software vulnerabilities or these third-party softwares we need them to be able to get our systems and our environments and our applications to run but there are some recommendations that

we can implement that will actually make us do a little bit better one is know your publicly exposed services i don't know how many companies i've worked with or audited where i'm like did you know this is public they're like no i'm like did you know this other application was public they're like it's not public i'm like well here it is from the internet so understand your publicly exposed services understand the software that is behind those publicly exposed services and then limit lock those down as far as configurations go and limit those exposed services to only what's necessary if you can put them in behind some type of authentication great if you can't then make sure you're up to date subscribe to

the newsfeeds of those vendors so that as critical patches or as critical vulnerabilities are released you can go through and make sure you're aware of those and then work with the teams to make sure those are patched mitigate and patch as soon as possible sometimes you can't patch because there's no patch for it but you can mitigate you can block ports you can block services you can restrict to certain ip ranges and stuff like that 30 30 days these days is way too long i know pci and suk2 and some of these others say yeah patch your critical vulnerabilities within 30 days if you wait 30 days and there's a proof of concept code that's out in the wild

you're too late so we've seen that anywhere from three hours to two days attacks start happening on the internet so that we see people trying to probe and find vulnerabilities so that they can exploit them yeah case one case two third-party access vendors it's not uncommon for companies to hire out for vendors to take a look at their their systems to provide support or other types of access and it's not uncommon for vendor contractors to have system access or support there to support other activities way back in 2014 if anyone followed the target breach that's kind of what happened right the hvac vendor had access to target systems and a hacker compromised through a fishing i

believe it was phishing the hvac vendor from there they were able to grab the credentials of that the vendor used to log into target and then from there they compromised all of target systems which was an awesome hack for a bunch of credit cards to be stolen we continue to see this right i investigated one recently well there's two actually for vendors so i'll talk about both cases um outside of the target one the other one i've seen is um there are times that vendors will use third-party companies to provide certain types of support for their environments as well it could be a call center it could be an e-commerce or e-support desk or anything

like that so the first one i talked about is if you have a vendor contractor and they get compromised then those conditions credentials can be used to then compromise your systems i've investigated probably five or six in the last 10 years or that's actually been the case the other one that i've seen that's kind of interesting as well and we start to see it here and there on the news but not a lot but it's a real threat is when vendors are become insider threats right that's where you see a vendor that's been hired to do certain services for you it could be call center e system support monitoring troubleshooting and you give them unfair to access to those systems

that vendor gets bribed or otherwise contacted by an adversary saying hey if you will i'll pay you x y z number of dollars if you'll give me some credentials or if you'll start to pull some data out for me and depending on where those third-party vendors are that could be really tempting i did an investigation a couple years ago where that happened where i was working there was a company that was providing support for a cell phone provider and one of the two or three of the employees that that company had been bribed externally and they were able to take and kind of make cell phones disappear off the system so they became ghost phones

and they were paid every time they provided that to the attacker they were paid like two or three thousand bucks it was a third world country that's pretty good money right for them and so make sure you keep track of your vendors and also monitor their activity as as far as that goes other things that we've seen as i talk and work with vendors to make sure that they've got good security practices one of the questions i've started to ask is do you use the same access keys and credentials to access our system as you do for other customers and if the answer to that question is yes which sixty percent of the time so far

it's been about uh yes then we require them as part of our contract to use unique credentials to access our system so they're not using the same credentials to access a whole bunch of their other customers um other things if you can get away with it is each of those vendor workers should be able to access your system with unique credentials so you can attribute what they're doing to an actual person instead of the vendor itself it's not uncommon for vendors when they access your system to use some type especially if it's a support desk scenario to use shared credentials so you may have six or seven of people at the vendor company that will actually use the same credential to

access your system the problem with that becomes when there's an event or an incident and now now you have to go and try and attribute and figure out who did the malicious activity and you go back to the vendor and say we need to understand who did the malicious activity and they can't tell you the reason why is because they're using share credentials among themselves so recommendations there and i need to add one here but implement good credential management for your vendors make sure if at all possible each of the vendor people who are going to contact or connect to your system actually have their own unique credentials to log in implement key management requirements

for vendors and i'll give you another story in just a second implement mfa for ssh access don't use passwords and implement key rotation requirements and that's where the other story comes in we're doing some research on one where we were investigating an issue and will come to find out that there was an employee of a vendor who had left like a year and a half earlier but they hadn't gone through and done good kind of off-boarding of their employee and that employee ended up having access to a system of one of the other companies i worked with and so as we were consulting we walked in to do this response we tracked it down to this actual person

and the employee we reached out to the vendor and said okay this is the employee and this is the ssh key that was used to access the system and then they turned around and said yeah but that person hasn't been with us for a year and i said why wasn't it cleaned up now we have a new process to make sure that we do audit of all vendor accounts to make sure that stuff like that doesn't happen and don't allow vendors if at all possible to use the same key for multiple customers i think it talked about that the other thing is if there's a sensitive system or a sensitive process and you need just periodic help support

tag one of your internal people to be on the line or set up whatever web sharing session is there and don't leave them alone monitor them while they're in your system and then revoke access as soon as they're out that's um it's a little more resource intensive as far as you as a company but it does save you from some of these other things happening third scenario popular software library it was talked about or mentioned just briefly earlier a lot of our developers are using get they'll use software libraries that are open source or third party a lot of those open source or third-party softwares sometimes remain teamed well sometimes there's a lot of contributors and this

particular case that happened earlier this year it was the node.js library hackers identified that node.js had millions of downloads a week the attacker actually was able to hijack the account that was used to deploy the malicious and deploy to malicious version of the library for node.js since there's millions of downloads a week that's pretty good for an attacker to have a nice single point of entry as that was deployed the libraries were installed and it wasn't just crypto mining software so i'm going to use your resources on your system to mine cryptocurrency but there's also password stealing trojans that were attempted to be installed in systems as well so i'm going to take your credentials and i'm

going to mine cryptocurrency and you get to pay the power bill and i'm super happy to have your credits right there are a lot of companies that this was reported there were actually three different versions of the node.js that actually ended up being deployed for three different versions of the libraries that were there this goes back into other recommendations as well you have to be able to monitor for critical vulnerabilities and disclosures to even software or libraries of software that you're using to secure your environment implement good monitoring systems in your systems to look for unknown or common indicators of compromise this particular issue was caught because some of the ed or software and some of

the other software noticed that there were strange connections out to command and control servers that were known to be malicious that was identified and then reported back up to the author of the library and from the author library they published an article that was then picked up by the industry so that everybody could look for that mitigate and patch as soon as possible of course and then if you find vulnerable versions of software in your environment make sure that you don't have repositories of those so that in six weeks or a year when kind of the dust is settled you don't accidentally deploy those again so take away these takeaways uh cyber securities will always keep us

pretty pretty active right so every investigation that i've done over the last 15 years uh i've learned new attack techniques um i've used it to kind of improve what i know and how i use it and i think that's kind of what we can do here just take the information we have be able to apply it to a broader spectrum and then from that be able to better secure the environments or the servers of the companies we work with so stop there for any questions okay thank you very much for your time [Applause]