BSides Idaho Falls 2025 - Track 1 Morning

All right. So this is track one and we're going to talk about cyber security considerations for small businesses. And the speaker is Nick Burroughs, the director of operations and chief technology officer at Total Care IT. And remember, small businesses are the backbone of America. They are also the soft underbelly of all of our industries here. And so whether some people believe it or not, your partners are also your your biggest liability and vice versa. So we need to all understand the things that you know. Thank you very much. Thank you. Thanks. >> All right. I uh I just This has been such a fun day. Um I I've been in this room most of the day. I saw the incident

response presentation. If you missed that, you missed something good. Saw regulation bingo where somebody who works with compliance regulations on a daily basis got schooled by children. And um and now I'm here uh to do this present. Oh, the keynote was phenomenal. phenomenal. Um, I missed the lightning talks. Uh, I have a three-year-old who had to go to a birthday party and so I was helping my wife with the baby. Um, but um, I'm here to talk about cyber security considerations for small businesses. As mentioned, my name is Nick Burroughs. Um, I work for Total Care IT and um, before I get too far into it, I just want to give you guys the the frame frame of reference that

I'm coming from. Um so TotalCare IT is a managed service provider and what that means is we um are provide outsourced IT services to small businesses. Um we serve about 60 clients um most of which are in Idaho but we do have clients that are fully remote. We have one client uh that's a cattle processing plant in Nebraska. Um so um we use two different models. Um one is we're the IT department for a small business. They hire us. If they have something breaks, they they call us, they we do projects. We we responsible top to bottom for all of their IT solutions. The other model is co-managed where they may have an in-house IT team. Uh maybe they're doing

the the boots on the ground, the the level one troubleshooting, we get the escalations, we get the projects, we get manage security, um manage their servers, things like that that might be out of their expertise or harder the source. And so, and then we can also backfill. So, you know, IT people working for small businesses can take vacations. I know my experience working for a small business as an IT person. The first time I tried to take a vacation, I got off the plane in Oakland and had to figure out how to fix the power supply of the accounting server from 2,000 miles away. So, um it's not often, but Total Care was founded in

2007, so we're 18 years um just I think this week actually. Um our founder is a gentleman named Aaron Zimmerman. He um is still our president and owner. He came from corporate IT. Um he worked uh as a contractor for HP out in California. He worked for Nuance and he wanted to create a business to bring enterprise level IT solutions to small businesses and that has evolved to also providing enterprise level cyber security solutions to small businesses. Um we manage over 1,700 endpoints. Um I mentioned clients are all over some fully remote and um one of the things that we've recently um started is our own private cloud or colllocated compute that's in a tier three data center in

Boisee. So we're working on our sock 2 certification specifically around that service offering. Um I um my job title's changed in the last six months. I I guess I'm now the principal technical and security architect. Um I got my job split. I I I was like drowning trying to manage operations and manage the technology and security and so I said please and so we have um someone who's doing our day-to-day operations and then I'm doing the engineering and um compliance pieces. Um I have a bachelor's degree in computer science um from the University of Pennsylvania, MBA and masters of accountancy from ISU and uh I've got 15 plus years as of direct IT experience. Um, I spent eight years

managing the IT for a small business, a nonprofit here in Eastern Idaho. I moved to the program side for several years. I have experience I worked for a charter school as their uh chief financial officer for a while and then came back to it when I came to Total Care IT about five and a half years ago. Um, uh, the slide's out of date. I was working on my CISSP, but I've tabled that for a little bit because um CMMC is a huge opportunity that's waiting around the corner. So, becoming a certified CMMC professional seems more pertinent to what I need to do. So, but then I'll jump back into the CISSP stuff. So, I

want to know who's here um just to get a sense. Does anyone here work for a small business? Okay. What industries just >> Okay. >> Okay. >> Okay. What about >> Okay. Uh who here works for like large enterprise IT? >> Not it. Okay. Yeah. Like a large enterprise. You'll find people who are like your C your your network admin and your systems admin and your service support desk and everything. It's very um boxed roles. Um, anyone in government or quasi government? Okay. Um, education. Anyone students? Okay. Did I miss anything? And what do you got? Where? What about you? >> Okay. Independent consultant. Okay. Um, is there anything specifically that brought you here today instead of

learning about activists or enjoying outdoors or going to the villages? >> What was that? >> Okay.

>> That's so familiar. so familiar.

>> Okay. >> Okay. Does is there anything specific anyone was hoping to learn? Okay. Just want to make sure that I' I've I meet the uh expectations. So, what I'm going to do is I chose kind of six general areas of cyber security and when we get into it, we're going to just decide as a group which ones you want to talk about. And this is going to be more of a discussion. I'm going to let you know what I usually encounter when we go get a new client. Um, these can be clients that have had no IT. These can be clients that are startups. These can be clients that have had a managed service provider or have had in-house

IT. And usually it's almost always the same thing. We we almost always experience very very similar things. So I'm going to talk tell you what I I experience and then I want to learn from your guys's expertise and have the room your experiences. What would you do to help that client? And then I'll tell you what we do. Um I'm not going to um I'm not going to talk about any specific products. I'm not going to like tell a specific client that we serve. Um, so I got to keep our solution stack a little secret, but I'll tell talk about the technologies that we deploy. Um, the other thing to note is that we only use

vendor supported solutions. Um, the reason for that, we're not going to deploy a bunch of open source things because as soon as you have turnover, if you've deployed an open source solution, chances are the guy who deployed it is the only one who knows how to use it. and then there's not a community or vendor to help train the next person. And so you kind of we kind of find ourselves in a lurch where we've done that. So vendor supported. And then the other thing is you're not going to find this is the only time I'm going to mention vendors and not because of any bad reason. You're not going to find Crowd Strike or Sentinel One in one of

our environments. Not because they're bad, but because we find that the bigger vendors have bigger targets. they're going to they're more likely to have be subject to zero day vulnerabilities because there's more effort trying to find the exploits. So, we're going to use smaller vendors in our solutions just so that it's a little smaller target for the bad guys. So, before we get into that, I just want to talk a little bit about what makes a small business different um than than um other other businesses and large businesses specifically enterprise. So the biggest difference is that top line, the governance. Uh small businesses are going to be almost always owner operated. Um there are small businesses

that have sea level suites just depending on where they they're coming from, but they're almost always owner operated. Um, and that is a huge difference when you're dealing with decision- making than when you're dealing with people who are at a seauite level that have a board of directors because the people that are owner operated, they're like, "Every dime I spend that is not critical is money I don't bank at the end of the year." Seuite people are like, "Yeah, we want to have maximize revenues. We want to maximize profits, especially if we have investors, but I'm getting my salary regardless. As long as I perform well, I'm getting paid. maybe I'll get a bonus, but it's not nearly as critical,

especially when you're dealing with things that have smaller margins as well. Um, small businesses hardly ever have an IT budget. If they have one, it's very small. Um, in enterprise, it has become a necessity and it's become an investment. Um, so there's usually a higher budget. I know when I was trying to move our nonprofit small business into the new um century uh or millennium 20 years ago, the number was roughly 3% of our budget was supposed to be in IT. We were at 0.5%. So there was a lot of growth that needed to happen. Um IT team, you often don't have one or if you do, it's one person. And I don't know about you, but does

anyone here know everything? Well, if you're the one person, that's all your IT department knows. If you're in a small a larger organization, you have a lot of people to be able so people can learn expertises. And I remember last year actually, and just to as a shout out, I remember Joseph pointed this out from his IT experience that when he started, you had to know everything. And now you have very granular defined roles and you only have to know what's in those boxes. And that's one of the big shifts of the last 20 25 years. Um IT tools typically you're going to have very simple tools um cost-effective tools not that's a bad thing. Um an

enterprise they're usually going to have layered um larger um budgets mean larger tools um and and larger scalability. Uh risk tolerance. Small businesses like I said very low risk tolerance. That's um if they they lose anything they they're they're um they're panicking now. Losing loss is lost. nobody wants to but the risk tolerance for an enterprise is a little bit higher and then um from compliance and I don't not to say that the small business doesn't have compliance the concern over compliance for a small business is very low they don't realize what they don't know um in enterprise depending on what they're where they're going I mean we had a a presentation earlier today if you're doing work in

Europe you have a whole lot of compliance that you have to follow that's different than if you're only doing work in in the United States for example. So there's complex multi-regulatory compliance and that so those are some of the differences and this matters largely because the decisions come from a different place but the concerns are the same to be completely honest a small business has to worry about the exact same things as a large business. The problem is they don't. And so that's where a company like Total Care or an IT professional and proactive person would come in and say, "This is important. If you don't follow this, if you don't pay attention to this, you're either going to be

liable or you're going to to go out of business. The these are realities. Um, no business is too small to target. Um, there are literally services on the dark web where you don't have to know anything but how to find it, where you can buy ransomware kits and for a share of the the profit, you pay like 200 bucks to get the technology and you set up your little server on AWS or Azure or wherever you want to put it and it'll just flood and it'll and it'll start exploiting and uh casting wide nets. Same thing with fishing services. Same I mean there there's it's a commodity that the attacks are coming from a commodity

now. And so $50,000 for no effort is worth just as much to them as $10 million for six months of effort. Um the compliance requirements are growing. Um the just a couple of things I I don't want to go too deep into the compliance, but we have CPA firms for our clients. We don't have car dealerships, but they fall in the same thing. Basically, anyone who trades consumer information as far as like credit ratings, banks, they're all required to comply to what are called FTC safeguards. The FTC safeguards, unlike HIPPA before them, have actually criminalized negligence. If the if there is an uh exfiltration of data and there is data loss that is affect that affects consumers and you

can't prove that you did your due diligence and did your best effort to protect that data, you can not just get fined, but you can get criminal penalties. So that's that's changing. It's something that hasn't always been around. Um and that that's fairly new last two three years. The other thing is that um we have to keep in mind for small businesses especially that one size doesn't fit all. I'm not going to do the same thing for a threeperson law firm as I'm going to do for an 80 person engineering firm. However, I need to listen to them the same way. I need to make sure that they understand the risks and I need to make

sure that they understand how they can be protected economically. So, and then the last thing um that the biggest thing that we're hearing now beyond regulations and being able to maintain contracts is the number of people who are starting to sue. Um if you're part of a breach and you're named in the breach, they're suing the person culpable for the breach or the the company. They're suing the IT providers. Um, and we're also hearing about insurance companies starting to sue their clients for putting in claims when they didn't follow the conditions of the policy that they said they were following. And so now they're they're they're suing them for misrepresentation. So that we are becoming much more

latigious. So for small businesses, this is stuff that people need to start paying attention to. Um, and so that's why it matters. So the six things areas that I I uh thought we could talk about and we can choose any of these but complexity and infrastructure, user behavior and training, access and identity management, security stack, compliance and governance and threat landscape. Who where would you guys want to start? I can start at the top or we can bounce around. Sweet. All right. Yeah. I I saw uh there was a NIST presentation last year where the guy had like three or four different scenarios and I was like that's such a cool idea. I'm going to do it that way. His was much more

graphically appealing because it was like a roleplay game, but like um so what we usually find is sometimes we get strong passwords. Sometimes never ever find password managers. Maybe there's like like that one person that has the Excel spreadsheet or they might have LastPass, which makes me cringe, but that's okay. Um, and um, sometimes they they require MFA sometimes and and that's that's limited to their like 365 or Google Workspace. Um, very rarely unless we find Active Directory is already in place because at some point they felt they needed an on-remise server. They're usually always local um local admin accounts or local accounts. They're usually always local admins. No privilege access management. I know I didn't put that up here. And

then um remote access usually consists of um a clientVPN uh remote desktop and maybe maybe they'll have cloud services. So fragmented identity access management because each cloud service is going to have its own username. They're not doing any type of single sign on or SAML or OOTH or anything like that. Um, and usually they have no centralized control over password complexity, password rotation, password management, things like that. If you guys were walking into this and saying, "Hey, we're your new IT provider." How would you address this?

password training of Yeah.

[Laughter]

>> Oh, my my favorite one that I get that I didn't put on here is >> it it was that it's that except it's usually not the CEO. It's usually the person who processes payroll. They have everyone's password and to their computers, to their emails, and everything. Literally had this conversation this week. I'm like, "We need to enforce security defaults so that we don't have to manually make people sign up for MFA." And he goes, and the person said, "Well, I need all of the MFA to go to my cell phone." And I said, "Why?" And I'm like, "Because I have to log into their email." I'm like, "Why don't we just delegate their mailbox to you then? You don't have to

log in." They haven't answered yet. So, what are you gonna say?

>> Yeah.

>> Yeah. One of the things that I find interesting um when it comes to like this and like password complexity specifically is the conflicting information that you get and I don't know if it's because of like the profit center but like you get like the password managers are like password hygiene you got to rotate your password every 90 days you got to do this this this like strong complex passwords and I agree long strong passwords must have them but then Microsoft stands over here and they're like don't rotate your passwords. Don't turn that feature off. Don't make them. Don't don't require password rotation because then people will do easier passwords, more simple passwords so they don't forget them. And

I'm like, where in the middle am I supposed to meet when I have 90% of my clients are in Microsoft and they're telling me to do this and I have to turn this on by by default, but um all the best practices say, including all the regulations that we learned about say to do password rotation. So MFA's for sure for sure. So um what we'll do um we try to connect devices to Entra to do a formalized centralized uh a formal centralized IM. Um regardless of whether or not they're in a Google Workspace or Microsoft 365 shop, Entra can be free um as long as you um connect it. You don't have to license anything. So, we try to get user

devices connected to Entra if they don't have um Active Directory. As far as the remote workforce, um we I I cringe now when I hear clientVPN. um I can't remember who it was but um they started using um Kali Linux with um a a service offering and I can't remember what it was but they started um it uses an it's an AI service that runs on Kali Linux and it's flooding to identify clientVPN um connections and um trying to just flood and do like brute force attacks to try to get in through clientVPN. Um, so anyone that's like connecting to their gateway, I guess I'm like there's better ways to do this. Um, so one thing that

we'll do if we have people who need to take their device off premise and they need to be able to access centralized servers or centralized resources or even um for conditional access, we'll put um what's called a secure access service edge or Sazzy. We have we put an agent. What that does, and it's really cool technology, is I call it a VPN on steroids, especially when it's done right, because not only does it give you that central connectivity, but it also gives you things like content filtering, secure DNS, um, and all sorts of other services to make sure that their traffic is secure. It's like taking your corporate firewall with you no matter where you go. So, I run Sazzy on this

box. I can go to Starbucks, jump on their public Wi-Fi, and I'm not worried. Now, if I turn my Sazzy agent off, um hopefully I close my bank account. So, um the other thing that we'll do is we do a centrally, u managed remote access solution. Um as an MSP, we use what's called a remote management monitoring system, RMM. And using that, we're able to leverage um a remote access solution that does the authentication at the RMM and then connects via the agent that's already installed. So there's no communication with the gateway. So I don't have to open ports to be able to allow people to access their devices remotely. And by doing those two things,

we've done a lot to um remove security concerns that may have been open when someone thought, hey, let's just open port 3389. Who cares? Are there any questions about um access and identity management? Oh, we also I I don't know if it's in another area, but we do also offer a password manager. From a compliance standpoint, it's very important that they use that everyone uses the same password manager so that you can report on the uh complexity of people's passwords and following those passwords. If you're not using an enterprise central um password manager, you're just trying to you're trusting people and you have no reportability and you're never going to get those check boxes if you have an audit.

So, we will go back. >> Yeah.

We're which >> zero trust. Yeah. >> Yeah. We actually um I love zero trust. Um and I hate it at the same time. Um zero trust is the only way that I know that I can protect a computer and a device from a bad actor. um because of zero days because um even your edr and ngav is only going to know what it knows what what it's been able to learn. It's kind of like that AI presentation. It's only as good as the inputs that that people have been able to put in there. If I do zero trust where I'm only explicitly allowing the things that I say are okay whether it's uh you running this program, you opening

these types of files or you accessing these resources by doing whitelisting instead of deny listing. It's the only way I know that you can have truly a locked down and secure environment that not going to say it's unbreakable, but it's as close to unbreakable as you're going to get in today's landscape. Um, but convincing people that it's worth the the is worth the annoyance is hard. >> Yep. Yeah. We we um had standardized to a zero trust and application whitelisting program and we've loosened a little bit. Um but what's the next area you guys want to talk about? Sweet. All right. So, I've got a little bit more here. Um typically what we're going to find in um small business very flat

network. Um everything's going to be on the same network. Usually one sublet, probably not any VLANs, guest Wi-Fi, if it exists, can talk to the production Wi-Fi usually doesn't exist, though. And they usually have their Wi-Fi password on the reception desk um if they have a Wi-Fi password. And um so, uh typically, um we're going to find the ISP's um gateway device. They're not going to have a um any type of firewall, um outside of that. they're going to use that for routing. Uh there's not a lot of computers, so that's why the flat network probably is okay. Um a lot of times they're going to be cloud first. So they're going to be doing things like

Office 365 or Google Workspace, maybe a couple of cloud providers. Um there's not a there's not a huge client server um market anymore in in a lot of our clients. Um but that is industry specific too. Um and then um usually there is not any type of device policy like and that and what I mean by that is you probably don't have um any type of group policies or in tune deployment or device management and um you may have very little basic vendor relationships um and they probably are just going to the vendor that seemed to have the solution that was cheapest that worked for to them instead of doing any type of vetting as far as where the supply chain

was coming, what they're doing on the back end, how they're using the data, things like that. And I'm not saying I can solve all these problems, by the way. What would you guys do with this type of environment?

>> Yeah. Uh

yeah. Yeah, definitely. Uh definitely secure the Wi-Fi. Um if you are going to have any type of public or shared password, uh make sure that that's on a segregated VLAN or subnet or both. Um and um ideally you also don't let those devices talk to each other and what any other ideas. >> Yeah, Jose

got everybody in the same private space >> but it's still you have like a single district of class Oh yeah. >> Yes.

>> Okay. >> And one of the problems they said they're a large state

infrastructure modify things.

Yeah. >> To do like some incremental changes to their infrastructure. >> Yeah. >> Take everything down. >> 100%. That that's another scenario where what I talked about before with Sazzy with the LAN zero trust you could you could really leverage a tool like that too because um with the Sazzy one thing I didn't talk about that it does is land zero trust where you can actually you authenticate into the Sazzy agent and you can define you're only allowed to talk to these things. So you prevent your lateral movement and you also are able to um through through zero trust define what resources people are able to talk to and that's all done softwarebased and so you don't have to

do a large infrastructure project

>> they really good service.

>> And so they'll actually predict they'll give you a here's the recommended segmentation. >> Yeah.

>> Yeah. >> But yeah, the the varying complexities um everything we we don't I mean we do have multi-sight um clients. We've got a client that has an office here in Idaho Falls and an office in um Austin, Texas. The people in Austin have to access server resources here in Idaho Falls. There's a server down in Austin that people some of the people in Idaho Falls have to access and and making sure that you are securely segmenting those things um and allowing those resources to be connected um efficiently and effectively but also making sure that uh Betty and HR isn't able to access the engineering resources that that's important as well. So, um, so what we'll do, um, is we try

to do as much with network expmentation. Um, in order to do that though, we're going to have to pull the, uh, ISP provided device and put something else in. Um now what we have found especially over the last threeish years um we were putting in um you know big unified threat management systems um with with the subscriptions and everything. Um we were a big Sonic wall shop um and we have found that the attacks are hitting endpoints not gateways anymore. And so by locking down the endpoints with good EDR, good XDR, um good sock, SIM services, uh zero trust or good privileged access management, that's more important than trying to protect the end or the gateway. So we've we have

simplified um and we are we use I think they're very good machines, but they're not necessarily the most expensive. They don't carry large subscriptions and things like that. That's one thing that we will do. Um, and it's very appealing to the small business because for the same price as that annual subscription that you're getting from PaloAlto or Sonic Wall or Sofos or Forinet, you're getting the device once and it'll last for four or five years before it h has to be replaced. Um, so but we um try to make sure that we have the proper threat management capabilities. Um, we're able to use our RMM to enforce uh a lot of that configuration standardization. Um

there are a lot of tools you can use in tune um or you can use your RMM. Um our RMM uh that we use um the particular one they bake in about 300 scripts that you can just be like oh I want to use that one and I can use that to turn off uh USB devices. It just runs the power the script and then USB devices are turned off for example. RMM is remote management monitoring system. Um, in my world, um, where I have 60 clients that are anywhere from five minutes to five hour flight from me, um, being able to gain all of the metric data as far as performance centrally, be able to report on it, be

able to know what devices are online, offline without having to leave and drive or fly, it's critical, and that's what the RMM does for me. So, um, but you can use an RMM even as a smaller IT shop. If you're an enterprise, like I would imagine that the IT people at the INL have some form of RMM that that's inventorying all of the assets, device health, things like that. But, um, and then the other thing that we try to do, now this isn't something that a lot of people want to pay for, especially in the small business space, but we try to do a risk assessment. And at the very least we'll inform them of

risk informally. But we do try to offer a service where early on we will do a top tobottom risk assessment from where they are so that they can then see over the course of 12 months where we get them. So any other but we try but but the the big ones um that we do from the complexity is we try to recognize that even though they're small that doesn't mean that they need to be dumb and have a flat network where lateral movement um can happen. Um one of the ransomware incidents that we remediated was someone who got in through a clientVPN and they just were able to laterally move side to side. it was a seasonal worker whose a client VPN

account should have been disabled. Um wasn't and they just were and so um because of that what that person had going on at home affected the business. So um all right, we'll do one more. Oops. Which one do you want to do next? >> Security stack. Okay. Oops. Uh, we'll usually get see anti virus. Um, a lot of times it's Microsoft Defender and that there's nothing wrong with Microsoft Defender. It's even better if you have like Business Premium or the Defender add-ons, but they don't have any type of EDR or if they do, it might be um a basic EDR, almost no monitoring at all. Um, very very rarely are we going to see anything that looks

like anything like a SIM. Um, individual backup data. said they'll be like, "Oh, I've got stuff in Dropbox or I've got stuff in one drive." Um, and never do we see a disaster recovery or business continuity plan. But I I've never gotten an answer of what's your recovery point objective without having to tell them what that means. >> Yeah. EDR is endpoint detection and response. So to put it this way, your next your antivirus which we now call nextgen antivirus if it's using huristics um and learning models um it's basically looking at behavior instead of file signatures that's looking at um what is actively being done um but it's limited to the activity like I open this

file and the ED or the AV is going to scan it. But there's a lot of there's a whole classification of things that um are that run in memory and they can be malicious and they can have malware. So your EDR is going to have heruristics to be able to monitor scan for that. Um it's similar in in in scope to it's not just responding to what happens. It's looking for behavior that might be happening that's not user interactive. Um they work hand in hand. Um, we actually our our EDR solution is also our AV solution. We we don't we don't say that they're separate. Um, but what you do with that data matters as well. So,

Um typically um as far as a recovery objective depends I mean it really depends um on on how much got hit. Um the the one ransomware I I and I'm I'm thankful and fortunate that I have not had to do a lot with ransomware, but the one ransomware that I did um have to remediate um fortunately they had just been forklifted up into the cloud to a cloud hosting environment. So and all of the data that was excfiltrated was already encrypted. So there was no data loss. Um and there and the servers that were um encrypted all the production data had already been moved. So it was a very easy remediation for us. But I I

know um here in town, Hospice of Eastern Idaho got hit with ransomware and it was about a five week initial re resolve and I think it took six months to actually say that everything was closed out. Um, but they were able to they were able to they worked off of paper and pencil for about five weeks. Um, and that's a nonprofit. That's really heartless. So, um, but, uh, but yeah, so small businesses, it's it's tough. But, uh, as far as what we do, we're definitely going to put in make sure that there's NGA and EDR. Um, we're gonna make sure the EDR, Active Directory, if it's present 365. Okay, cool. Um, excellent. E, AD 365,

Google Workspace. It all feeds into a SIM. We connect our SIM to a Sock team. Um, our specific vendor, their sock team are all retired military people out of Augusta, Georgia. So, they're they're transitioning from the military to cor to business world and civilian life. And so, they hire them. It's pretty cool. Um, >> no, >> they do have a different mindset, but what they're looking at is threats and they're looking at they're looking at the data that I wouldn't want to look at all day. Um, we'll deploy vulnerability management solution. What what one things that's nice about that is you just plug this light this this agent on every device and it's going to track thirdparty

solutions, operating system vulnerabilities, um alert to it, and then about 95% of the time it's a p push of a button on our end to say, "Hey, install that patch. Install that patch." Um and so we are able to monitor manage that um much more efficiently to be able to keep um the vulnerabilities minimized. um they're only going to know vulnerabilities based off of CVEes. So any zero days aren't going to be hit by that or covered by that. Um the other thing that we do um is back up your 365 or Google Workspace environment. Um, what one thing that that and I've gotten into arguments with people about this, but Microsoft and Google, they're going

to protect your data because it's in their best interest, but there is nothing in your agreement with your acceptable use agreement or your service agreement with them that says that they're responsible for maintaining your data. So, if you accidentally delete a file in your 365 and you accidentally empty that, delete the uh the the 30 days and say, "Oh, I don't need that file anymore." And then the next day you're like, "Oh, I need that file." Microsoft's going to be like, "Tough." They have nothing to to recover that. So, we're dealing with humans. So, we back up that environment. So, we can go back 30 days and say, "Oh, yeah, that file was deleted." Well, I can pull it

from this day. So, um and then the other thing that we'll do is we provide assistance in developing a backup um business continuity and disaster recovery plan. Um main thing that that involves is where do you want your backup data to go? We by default will back up data to a local um network share and to AWS in an immutable form. And um how frequently do you want that backup to happen? That is your recovery point objective. How much data can you lose? So if you're like, we can't operate with two hours of lost data, then I have to back up every two hours to make sure that we meet your RPO. And then um how fast do you need to be

back up and running? If your server were if this server were to crash, how fast do you need to be up and running again? That is your recovery time objective, your RTO. And so what we do is we formalize those things so that they know they can communicate that they can manage those expectations. The other questions we ask are like what happens if a computer dies? Do you have a hot spare? What happens if a server dies? Do you have a spare? Do you let's say uh the TTOM dam floods again? Uh what is your plan? And your business is gone. Um you have no facility for people to work from. Do you still plan on operating? If

so, where? How? What is your plan? And those are the questions that we ask. And those are the things that we try to get people, business leaders to think about because they're in denial. They're like, "Oh, that'll never happen. That'll never happen." We got try to get the small business owners to think about those things. So, if it does happen, hopefully it doesn't. But if it does, we at least have something to work with. And so do they. So, do you want to try to hit one more or do you want to ask questions? One more. All right.

>> Okay. So, um did you know that the state of Idaho has a um laws that pro protect your um private information and that if a business acts um negligently and your private information gets exploited, even if they're not like a medical practice or a school with HIPPA or school with FERPA or under any other compliance, there are state statutes that say that that that they're called identity protection statutes, but if a business is negligent and your identity is stolen, then they can be fined like $25,00 000. Um, so most small businesses aren't aware of that. I would venture to bet most people in the room aren't aware of that because I just learned that like a month ago.

So, um, there's also cyber insurance requirements. That's that's the one that we're really trying to push now. We're like, send us your cyber insurance requirements and we're going to tell you all the writers that you said you were doing that you're not so that we can help you not get sued by your cyber your insurance provider or your claim denied. And then that way if something happens. You're okay. >> Yeah. >> So, let's say that um like this is a new thing. This is a new development. I don't think it's happened a lot, but let's say that for years and and in in a parallel, the defense um the defense supply chain is going to be

going through this exact same thing because for nine years, the defense supply chain has been saying we're we're implementing the cyber security and uh maturity m matrix certification, CMMC, and the defense department of defense has been like, okay, whatever. Like, yes, you okay, you said you have. and then they went and audited and they're like, "Nobody did." Um, and so now, uh, it's going to be part of the contract. Well, similarly, the defense department might not even let some companies bid anymore for their contracts because they've were so negligent. Um, but similarly for the insurance, you in order to accept to get the cyber insurance, there's a questionnaire that says, I'm doing this, I'm doing this,

I'm doing this, I'm doing this. If you sign that you said you were doing that and then you come back and you file a claim um based off of the act based off the fact that the actuarial tables that the insurance company is using are based off of the validation or the integ the integrity of what you answered. They set premiums at a certain threshold. If you didn't do the things that they set your premiums based off of, >> they might be suing you for the premiums that you should have been paying if you had answered honestly. It's not happening a lot, but it it it there have been reports of this happening. um PCI compliance. Um if you accept

credit cards at some level, some scope, you have uh PCI compliance. Depending on the number of transactions is the level of compliance that you have to follow. Depending on the level of compliance you have to follow, your requirements change, but apparently that doesn't matter when you're taking a bingo game. No, I'm just kidding. Um you're still getting >> Oh, I am so salty. No. Yeah. >> Yeah. >> Yeah. Your bank is your your bank or your credit card vendor are the people who actually are holding you to it and they're going to require you to fill out these quarterly annual, you know, like I have to we we have some um accounting firms. we have to run quarterly um

firewall tests and and everything and do test stations and stuff. Um it's it's not um the easiest game, but they that that follows anyone who takes credit cards and then industry specifics. Now, I don't know of a single medical practice who doesn't realize that they have to follow HIPPA, but I know of a lot of them who don't care. I' um I ask special questions when I go to a new doctor. Um, >> yeah. Yeah, we we um if you're not willing to follow our recommendations, um we don't need you. I don't want to like like >> Yeah.

>> Yep. Yeah. Yeah, that's the other end of the litigation. Um, I mentioned it. CPA firms have FTC safeguards. Department of Defense contractors um pretty soon are going to find CMMC um language in their contracts um that requires that they have attestation. Um and because these are owner operated businesses, most of them either aren't aware or they don't care until until they do. >> Do you have any federal contract? one. Well, I I I should No, I have three. Um I have one that that's a DO DOE contractor and then um I have two um that are directly um they're they're CMMC level one. They don't have any um CUI. So, they have to we have to do um

audits for them and stuff.

>> 80 of Yeah. >> Yeah.

>> That's that's on Total Care's frame um road map of journeys to take. >> So, but um we're going to just go to the final thoughts. The the main takeaways though um I didn't really talk a much a lot about it. the security can be scalable. Um you we really do need to listen to the business's needs before we say that these are the security things that they have to have. Um but we also try to put in scalable solutions by default from our perspective. Um, the other thing I didn't talk a lot about because we for whatever reason we get didn't get into them, but um, most of our providers or clients, the small

businesses, they don't have policies and procedures. And anyone who's been through any type of security certification, security plus, whatever it is, knows that there's administrative controls, technical controls, and physical controls. And you can't be compliant. You can't be secure without all three. and the administrative controls are severely lacking in small businesses and that's probably the biggest thing that we need to work on but we're so geared as a company towards the technical controls. Um and so that and then um the other thing that we really want to stress especially with small businesses is that security is a defense not an expense. Um and trying to convince them of that is tough. >> So this has been fantastic. I'd like to

have everyone of

the short talks, but they can come and catch you probably around. You'll be able to stay a little bit after. >> Yeah, I'll stay a little bit. Thank you very much. >> Yeah, absolutely. Thanks, sir.

So if I >> Let me have it back real quickly. >> It's on now. >> It's on now. Test test test. >> Test test. Do I Is there a button I need to press? Test. Test. Okay. Sam, >> yeah. >> No button you need to press. >> Oh, I accidentally pressed the button now. Press it again. There we go. Okay. >> Lights is good. >> Good. >> Hello. >> Hello. >> Yeah, no pressure. Hey, >> so I was I was saying >> that's what I hear. We're just gonna >> Yeah. No, she was grilling us during our practice.

Oh, wait a minute. >> Oh, that's fair.

Um I think we are ready when you are. >> So

you know what? Just one thing. So here I'm gonna >> power. >> Yeah, we are ready. >> Taking over the world. >> All right. So this is the uh final talk of the day and talking about understanding tycoon. I don't care what Jason might say. Everybody should understand because it is one cent. It's going to be everything. observations and detection faculties for critical infrastructure defenders. Sam Forn and and J National Laboratories that we call home the best of the labs in case I need the best of the labs. >> That's right. Back here someday. Anyway guys, take it away. >> Hey, good afternoon everyone. Thanks for coming. I'm Sam. This is Jim. And today we're going to talk about uh Volt

Typhoon and kind of maybe some like what it is, how they operate, maybe some things you can do to detect them as a defender. Who here knows what Volt Typhoon is by the way? Just okay, be healthy. Um our agenda, we'll have three real quick main takeaways in case you do want to go to the other talk. Um the background, uh some observed TTPs, uh then detection strategies to go along with them, uh some closing thoughts and some questions. So, we'll try to cram all this in. So, our three main takeaways, um, as far as this threat goes, TTPs and behavioral indicators are going to be a lot better for you than just static IOC's like

hashes and stuff, especially because this actor makes a lot of use of living off the land. Um, making sure everybody knows what living off the land is. Correct. >> Can you say what a TTP is? >> Uh, tactic, technique, and procedure. So in living off the land, it's LOTL, often abbreviated. It's basically using stuff that's on a system for malicious purposes. So you're not bringing in a payload to do something, right? >> Sure. Y >> um Windows, you don't need a lot of that for Windows. It's already bad enough anyways. Um so and then hostbased logs, kind of tailing off that are going to be more useful than your networkbased logs. And we'll go into that a little later um

of why the network is at least for this adversary is a very difficult way to pursue this. And then um vulnerabilities of internetf facing devices like your firewalls and your VPNs are increasingly becoming a problem for defenders. Um, so Volt Typhoon, the first time anyone ever heard this name was May 2023 when Microsoft put out a report that said Bolt Typhoon is using Living Off the Land to persist in critical infrastructure environments. Right. Uh later that year, uh, SISO put out some more detailed reporting from some engagements that they apparently had been on. And the reason why this is important is because they're using very stealthy techniques. And from the former director of the FBI, he said, and I'm

sorry to read this off the slide, China's hackers are positioning on American infrastructure in preparation to wreak havoc and cause real world harm to American citizens and communities in the event of a crisis or a conflict. Okay. So, in the event they're being stealthy, right, you know, that's something you might associate with like cyber espionage, right? So, like stealing stuff and not getting caught. But when they're in local energy, electric uh water wastewater treatment plants, some telecommunications and transportation systems, that's not really a juicy espionage target, is it? Like, not that there's nothing there, but um, they're trying to stay there. They're trying to be unnoticed for as long as possible. Okay. Some observed TTPs. And thank you

for asking me what that meant. If anyone has anything like that, please just stop us, let us know. um they use complex proxy or orb networks which we will get into later in the presentation basically to obuscate their network activity. um their initial access and this is what's you know one of the sticking points of this presentation is believed to be via vulnerable network edge devices right so again your firewalls VPN appliances some routers um and then once they have initial access they have extensive hands-on keyboard using living off the land after initial access so they're using what's on the system to do stuff find things out move around steal stuff and then leave right um this is

interesting because there's almost no use of malware. So again, like EDRs are great, but if somebody's not using customized payload or malware, like you know, those hash signatures aren't really going to get you too far, right? Um the crux of their campaign is generally the compromise of Active Directory credential stores. Um anybody know what Active Directory is? Okay, cool. Um >> is that Windows? >> Yeah, it's like it's like the white pages of Windows, I guess, if I really was looking for a crude metaphor. Um anyways >> for users use that to authenticate and know who they're talking to as well. >> Yeah. Yep. >> Yep. >> Um and then as well they have infrequent

access and actions once they have on target. So they'll they'll be there during normal and abnormal business hours. That's kind of a pain to hunt for, right? Um okay. So detection challenges. Um there's no silver bullet for living off the land. It's very difficult to detect. Um it's very time intensive for somebody to kind of you know figure out like how to tune these alerts and behavior alerts which we'll get into. Um and again because they don't use malware like your your IoC's are probably not going to be all that helpful right you know um >> indicator of compromise. Um and then traditional network-based IoC's so like an IPv4 or a domain are not very useful

because of these complex orb networks which stands for uh operational relay box network which we'll get into later in the presentation. Um behavioral detections are kind of really what you got left and it's really a lot of work to kind of tune those for them to be useful, right? They require not just a lot of data, but the right data and for you to keep it for the right amount of time, which we'll have a meme there in a second. But then the question to ask yourself when you're looking for Volt Typhoon or living off the land, because living off the land is not new, like but they're making extensive use of it and organizations that generally maybe don't

have the best resources or detection. So why would a legitimate admin do this is when you're looking for this stuff? to kind of look at this from another perspective. SANS has a two-stage what they call their ICS killchain. The first one on the yellow is generally like what you would consider an enterprise. So they're going to get in, they're going to move around, they're going to steal some documents or in this case because it's critical infrastructure, you know, operational schematics, stuff like that, right? They're going to try to maintain persistence and then leave, right? All that feeds into the second phase which is basically where they're going to take what they stole develop some sort of

payload to cause a cyber kinetic physical pick your word there effect in a real world environment right so everything that we have seen publicly bolt typhoon is very much stage one it's very window ccentric um so just to kind of manage expectations here um a meme for some fun um so I talked about logs like you need the right data and that is a lot of log that shim we'll go into, but you need the correct things to hunt for this activity, right? Um the Cisipian task of looking for this with either incomplete or in or like non-existent logs, it's a lot like trying to nail jello to the wall. It's just really not going to be a good time.

Um, and then on the flip side of this too, there is somewhat of legitimate business case for not having all these logs because especially with living off the land, 99.99, pick your number, whatever percent is not going to be malicious because these are just normal users and admins doing their day-to-day stuff, right? So, generally, you know, you kind of have to make a business decision where you can keep logs longer, you can have more stuff in the logs, or you can pay less money, right? So pick two because security teams don't have infinite security budgets. >> This is assume that they haven't compromised the machine. >> They do wipe logs. We'll get there. >> Yeah, we'll address that as well.

>> Yeah. >> Okay, Jim. >> Yeah. So, um, right, there's there's lots of different types of logs, um, on any given endpoint, whether it's Windows or Linux. Um they all have different meanings. Um and uh math is a cruel mistress. Uh you know maybe a 100 megabytes a day of of logs from a single endpoint. Well, whatever. You know, we've got terabytes of storage. Well, okay, multiply it times 10,000 and suddenly, you know, um you've got a bigger problem or 30,000, however many endpoints. Um and that's an arbitrary madeup number. Some will generate less, some will generate more. But point is um you can't capture everything and um it's also like with network traffic um it's generally simpler. You just say

I want to you know uh put in a tap and I'm going to capture everything uh across this switch or or a span port or whatever. Right? And um you can ask questions later, right? You don't have to you don't have to make a lot of decisions. You can filter traffic up front, but but generally people just slurp up everything and ask questions later. With Windows uh with with endpoint logs though, uh you do have to make decisions about what do you want to collect and from which hosts and so designing logging policies that make sense um for for different groups of endpoints and so on and and that will scale well. And then how am I going to

get them from here to there? Do I have enough bandwidth? And do I have enough storage over there? And how long do I want to be able to keep them? Um certainly to you you also have to consider that with network traffic can't keep all the pcap forever either. But um anyway there's more to consider here is what I'm getting at. So for this uh for this campaign um the if I had to pick one if I had to pick one type of log uh it would without question be PowerShell script block logs. Um however um the other two are important as well. So um Windows security event logs um you know there are lots and lots of event ids if

you ever uh have gone looking in your event logs um so they're kind of broken into you've got security application system um and there are some others and then within each of those you've got a provider uh one or more providers and uh and then you get into all the different event ids and um anyway so one that is of particular particular relevance here is 4688 by the way I do not have these all memorized because there's too many um but that's that one is process creation and what that can tell you is when a process is launched uh what were the command line arguments that were launched with it by the way uh that part

is not enabled by default so um again there has to be like you have to you have to want to do this um there is an alternative to that one which is sysmon event ID one sysmon is part of the cis internals suite uh very powerful set of tools for all kinds of system administration tasks. Um and uh very has become very popular in uh I guess the security industry uh because a it's free and b it's highly customizable. You can kind of tailor your own config and annotate logs etc. So um we could have a whole other talk on on just sysmon but uh for for now those are some of the ones we're going

to focus on during this talk. Um I'll give you a moment with the meme. So yeah, unfortunately uh no single log is going to tell you a whole story, right? It's a data point. Um and the event ID is not by itself even of one particular significance. The contents of the log can matter depending on and usually does depending on what the event is. Um, so, um, you have to look at the big picture. So, okay, weren't sure exactly where to put this, but this is kind of an overview of the Volt Typhoon intrusion set, um, and the behaviors. So, and I'll give you a moment to digest that. Um, and I think I kick it back

over to Sam here in a minute, but >> so this is we're going to follow this roughly through the presentation. We're going to follow this kill chain from initial access to a lot of the tools and stuff like valid credentials for persistence, all this kind of stuff. So, it's kind of a tricky way to do this order, but we're going to cover as much as we can. >> I've just noticed uh I I had a SISA reference. This is from the SISA um latest SISA paper on the Volt Typhoon. >> Okay. So initial access uh all the public reporting has volt typhoon generally leveraging either zero days or end days against vulnerable network edge devices. So your firewalls, EPAN

appliances, something that basically faces the internet and anyone can do anything or send anything to. >> What was it after zero day? >> Uh end day >> basically there's a vulnerability that's been published but hasn't like the person that has the thing hasn't been patched. So it's been like 247 days or pick your number. Right. So sorry. Thank you though. Um so essentially once they compromise this device they will see what kind of privileges or connections this device has to other servers. So in Volt Typhoon's case it's primarily active directory right um but we did want to throw something up here because it illustrates the point a little bit better or more but the who's heard of

Salt Typhoon? >> Yeah. >> Okay cool. So Salt Typhoon, they were also leveraging vulnerable network edge devices, right? And they were able to um it's a little bit different than what they were doing with active directory, but kind of the overall concept is the same is they were able to sniff vulnerable traffic, plain text traffic and look for credentials, encryption keys, all sorts of stuff, right? Um for follow-on operations, which is why I underlined that. And we don't have a recommendation for this because this is kind of a big problem. Um, everyone's kind of trying to get their hands around it, but essentially these devices, real quick question. Who is who knows their router's firmware

right now? Okay. Well, no, Rita, but like that's kind of what I thought like there these devices are an afterthought, would wouldn't you say? Like, so the implicit trust these things have with like Active Director, other authentication servers, um, these can't be an afterthought anymore, right? And they're everywhere. So, some things to just consider is basically kind of know what you have first of all, but also be ready to rapidly patch or take this offline. Um, watch the Kev, right, to see if that comes up, but as well like know if your vendor requires a proprietary tool to do any investigation with or an integrity checker. Yes, >> I think you should say, >> Kev, uh,

>> known exploited >> exploit. I was going to say key, but known exploited vulnerabilities. I'm gonna hear about that later. Yeah, that's a catalog from SISA that they they've been continuously adding to and is something worth paying attention to. And of course, even more so there if they issue an emergency directive to federal agencies to patch within 24 hours. Uh, everybody else should probably take notice of that too and do the same thing. You're not mandated to, but it would be in be very in your best interest if you have that device or software exposed. >> Okay. Okay. So, uh, getting into the trade craft, we're gonna go a little bit deeper. We're not going to cover

everything in the report. Report's very good. Um, and I highly encourage you to, um, to go to go read it, but we're going to pick out a few examples. Okay. So, um, this is, uh, a PowerShell script that they ran. Um, once they they got in, right, they got initial access, now they're they're looking around. is what we call discovery. Have figuring out where they're at. Um and and they do some of that with simple commands like who am I for instance is one of them. Um which regular users don't typically run a lot. Um so it's kind of weird but but anyway with this script they're going deeper and they're they're trying to figure out pattern of life. You can um

see so 4624. I don't have Windows event IDs memorized. There are a few exceptions, but that is the uh event security event ID for a successful log on. So what they're looking for are successful log on lo on events and they're querying the domain controller for these specifically um and and so they are getting a sense of okay well if I'm going to be stealing a bunch of credentials I want to come back and use later uh or or I already have I am going to um I would like to know when the the owner of each of these accounts is is active and using them so I can blend in um at similar time frames um and and

That's that's really the the gist of this. But again, they they run a lot of things in the context of PowerShell console. Um, which is why the script lock logging is so important um with uh yeah, I won't go any deeper on the PowerShell logging, but um yeah, so uh next up credential access uh and C2 with Whimik. Uh and so who's familiar with WI? So it's called uh Windows stands for Windows management instrumentation and WHIM is the command line version of that. It's actually deprecated. Um it's it's disabled by default in Windows version 10H11 something. I don't remember the exact version but it it continues to be disabled by default in Windows 11 but

it's still there and it can be used. Um but it's not used very often. WI on the other hand is used a lot still and in fact many PowerShell commandlets are based on WI um the functionality. So um so any anyway these screenshots come from the Microsoft report and uh the first one is showing them creating uh manually with this command line uh uh arguments here uh this set of commands they are creating uh remotely um domain controller installation media. So, I know it's probably a little hard to see, but uh on the bottom left of that first screenshot, they're using the NTDS util tool. And then just after that, you see if I mean there's other arguments, right?

But but basically that is telling it to, hey, I want to make a new domain controller without using like domain controller replication. Um and we'll talk later about why they're doing that because why don't they just why don't they just go grab NTDS. Right. Um well, it's not that easy. Um okay, the second example, they're uh setting up uh proxy um to proxy their traffic out to their C2 uh on a particular port. Um they're using the port proxy tool. Again, these are native to tools. Um and uh and then they delete it. So you could argue that's defense evasion as well. But uh but in any case, um so how would you detect this? Well, again, um

if this is happening in the context of PowerShell, you want PowerShell script block logging. I'll say it again again. Uh however, when within uh the context of PowerShell, you run Whimik. Well, that's going to launch a Whimik process as well. And so, you're going to get a process creation log and you would see these details in that um those command line arguments. Um and so, what should you look for? Well, um, in those logs, I mean, first of all, recognize this it ask yourself, is this commonly used in my environment, right? That's why you want to be collecting this data continuously so that you know, have I ever seen this before? And if so, where

and by whom? And, uh, is this this user's first time running it or trying to run it? Um, and so on. Um, maybe there's a little bit of activity and then you see a spike. Okay, that that should get your attention. Um, typos indicate, hey, this isn't like a polished script. This is somebody's typing uh, you know, hands- on keyboard interactively. Um, and and there were typos observed. Now, legitimate admins can also make typos. So, it's just a data point, but it it it's becomes part of the narrative. And, you know, over time, you can you can start to uh that that can help you in your investigation. Um, okay. So um this is kind of a

uh so this slide is about Elsass dumping Elsas. Um who's familiar with LSAs. So it's a a Windows process that basically stores your LO on credentials when you log into the computer. So um obviously very sensitive. those are going to uh be in memory and um and so if somebody is trying to dump specifically the process memory of the LSAs process that's highly suspicious and by the way if you have an EDR an endpoint detection response tool uh it had better catch that if it does nothing else um so this is what like mimi cats is known to do in fact they have been observed using mimic cats in rare circumstances but mostly not Um, they

also were oddly observed using uh the magnet RAM capture tool. Uh, Magnet Forensics is a pretty well-known uh digital forensics company and they have free tools um and paid tools, but this this is a free one and this is something that like the blue team would use when you're investigating. Um, it's pretty unusual though for someone to be dumping memory. Usually have a reason to do it. Um, and so that that should definitely raise red flags as well. like if you're not aware of an investigation going on and you see somebody dumping memory like that's weird. So um anyway uh the recommendations here are you know again pay attention to any alerts related to this type of activity certainly uh and

as always scrutinize your domain controllers uh and um that includes looking for legitimate tools um especially if they're in running from non-nair directories. Um so so yeah um this is not comprehensive but this is I would say uh a good portion of the command line based utilities that uh vault typhoon has been observed using a lot of these are for u discovery um but uh and many of them are executed in the context of PowerShell um but uh again so PowerShell logs and don't just be logging them Yes.

>> Yeah. Absolutely. >> Absolutely. So these right things like you know again who am I ping you know um task list uh net user that sort of thing. Uh so yeah you can look these up. In fact, we'll reference a website called called LOLBAS uh living off the line land binaries and scripts. There's a whole website that's been it's been around for years uh talks about um a lot of the common utilities are can be dual use can can be potentially used um to inform you know thread actor behavior. So um so yeah so don't but don't just have the PowerShell logging. I mean that's step one. Um, it used to be, by the way, script script

lock logging has not been around forever. That was only added in PowerShell version five, which happened, I don't know, several years ago, but um, and even then it wasn't enabled by default. So, there are all these little things. Yes, power, good question. PowerShell is like the, uh, big brother of um, big brother, big sister to the command prompt. It's got tons, super powerful. you can script a lot of things that would otherwise take you a lot of time. So it's very popular for system administrators, people who manage all the computers on a network. Great question. Um so so you want to aggregate these for a couple reasons. Um first of all, as you said, right, if if somebody's gonna uh

somebody could tamper with the logs, they could delete the logs where they're generated. Well, if I'm forwarding them, I I still have a copy over here. Um also it creates the opportunity by colllocating uh disperate uh data sets. Um you know my gosh imagine doing uh trying to do an investigation across you know you've got let's say you've got 5,000 endpoints right and you're not sure how many of them are compromised and anyway you know you need to look at the event logs you're going to go to each individual machine launch event viewer you're gonna have a really bad time. Uh so but by colllocating them in a seam security incident event management system like uh Splunk or

elastic search or arcs site whatever um that gives you the opportunity to correlate those and to do interesting groupings and uh filtering and correlations and visualizations and build a timeline in one place. Right? So you can start to understand the narrative of what a particular user is doing. what machines they're touching and so on. So, um, and and and this kind of gets at what we describe as a behavioral detection, right? It's not an IP address or a file hash or a domain. It's a set of things that somebody did. Um and >> or even one event ID, like that's not going to tell you everything, >> right? Right. Yep. So, they they didn't just um clear the event logs. They

logged in here. they remote desktoppped over there. Uh they explored you know these directories or copied these files and and by the way while this presentation is very host focused um that's not to say don't uh capture network traffic. Absolutely capture network traffic. Use something like Malcolm which by the way shameless plug uh we develop at INL. It's free. Um we could have a I've got a whole other slide deck on that but um but and that basically glues together things like Zeke ora uh etc and aggregates those from one or more sensors um so that you can figure out what's going on on the network and and so in my mind sort of the best detections are those that

aggregate disperate but related data sets to tell to tell a story and that raises the confidence of the analytic that you're creating, the detection that you're crafting. Okay, we're gonna try to keep moving here. Um, remote desktop has been observed. Um, but of course there's many mechanisms for that and and people it is used legitimately all the time. But what constitutes an outlier, right? maybe a user that um uh doesn't they've got the privileges but they never use it and now they're using it or they didn't used to be in this group and now they're in this group and they're they're using it. Okay. Um so things that like just don't feel right. Um and uh yeah

>> compromised password >> uh >> mostly password like yeah like username password >> yep >> from correct what we read >> which by the way and I think >> right so this is where MFA becomes really important right and and um so yeah that's that's an excellent mitigation against this >> MFA applies to a couple slides that that we go over. So, >> yeah. >> And I say disallow PS exec wherever possible. That's kind of uh I'm I don't know why I'm calling out that one in particular. The real idea there is uh try to not use 17 different system administrator tools that could do the same task. Try to limit it down so that somebody uses one

that is not the one that you use. you know, that by itself can can uh throw a flag. PS exec is also part of system internals, by the way. And it's popular because it's been around forever. It's well known. It works for a lot of things, but um it's also kind of clunky anyway. And similar to uh Whimik, a lot of people, I think, would consider it sort of so old it should be deprecated, but it's still around anyway. Um okay, so getting to the crux of this whole campaign, right? uh the the big credential dump going after NTDS. So they're doing this a couple ways um and uh so the first is NTDS util. We saw

the the example of that um to create installation installation media from domain controllers. The other is uh the volume shadow copy service. So there's a command line utility called BSS admin. Um well first of all who's familiar with volume shadow copies? Not sure why you would be unless you're Yeah. So, um they are um it's kind of like they used to call it like a restore point I think in Windows XP. Um basically it's in the background under certain circumstances when maybe nobody's really using the computer. Uh it's uh it's like okay I'm going to create a little restore point so that um the user can uh get back to where they were uh quickly. It includes all of the

files on that volume, including for a domain controller, you guessed it, ntds. Which by the way is kind of shorthand for anytime we say that we really mean also the system registry key, which contains the boot key, which is how you decrypt the ESC database that is NTDS. Um, so they need both of those things. Um, and that's what they they went after. But yeah, so but why are they doing it in this convoluted way? Well, that file is locked pretty much always as long as Active Directory is running just always. So they can't just grab it. So they make a copy that of of something that they know will include a copy of that file and then they can pluck it out

of there where it doesn't have the same protections. It's not locked. Um but yeah, there are I didn't know this to be honest. uh until recently, but uh there are some logs that can tell you about they're not specifically about NTDS. They're they're specifically about ESSE databases of which NTDS.dit is an example. So these fall in the application event logs um and they are specifically from the EENT provider and they will tell you there I took out the event IDs uh you can look them up in the report. Um, but if the database has moved, if it is uh there's a new one in a non-standard directory or if it's m mounted specifically from a volume

shadow copy. So, you know, people at Microsoft, okay, we can gripe about a lot of things they do, but they they they recognize some of these risks and and uh the fact that or I don't know, maybe it's dumb luck and they but um but uh yeah, so the these are meaningful, but only if it's talking about NTDS. I mean, okay, there might be other sensitive databases that you'd care about, but for Volt Typhoon, we care about that. Um, so but understand, of course, credentials don't last forever. Uh, they should expire at least. And so, um, if somebody wants to maintain stealthy uh, persistence with valid credentials, they will have to come back. They will

have to dump it again because those credentials will get stale. They will expire. They will change. And so that's something that you should be looking for all the time and understand may happen more than once if you're >> something else really quick. Yeah. >> Um we don't talk about a lot of OT in this presentation u frankly because there's not a lot of OT stuff in this presentation but active directory lives in both IT and OT. So if you take nothing else from well MFA all the things first but definitely segment active directory you know have separate one for your OT environment versus your IT environment because if the IT gets popped you don't want one being able to

compromise the other right >> OT is operational technology so think of like things that control PLC's programmable logic controllers that essentially enh like oversee or execute a physical process >> yeah and going back to this slide I want to foot stop that because this is commonly misunderstood, right? I think the campaign got a lot of attention because of the um attribution to PRC by SISA and FBI, >> People's Republic of China. >> People's Republic of China. Thank you. And um and the fact that they're, you know, targeting US critical infrastructure. However, um there if you read the report, there's no specific uh observations of them interacting with the operational technology side of the network. They go right up to the edge.

They describe having access to putty sessions. Putty being an SSH client for secure shell access to uh OT systems. Uh but they do not actually log into them. Uh it's not described in the report. So that's a common misconception. Fortunately, they didn't take that extra step. That doesn't mean that they couldn't. Could mean they're waiting or whatever. But go ahead. >> Did the report explicitly say that that never happened or did it not say whether or not? >> Uh it did not say that it happened. >> So, um >> it could have happened and they just chose not to say. >> Yeah. And and this is one of those things to where we're going back to the

geopolitical side is they want to do this in the event of a crisis or a conflict. Taiwan may or may not be in that mix, but you get my point. Like they wouldn't just do this, you know, to do it, right? You know, they want to do this to potentially hold a US response to something in the Indoacific at >> bay. Do it once. >> That's right. Hopefully once. >> Yeah, >> hopefully once. >> Uh well, and hopefully not at all, but you know. Yeah. So, um >> I just wanted to make sure that there isn't that misconception. Um so, it's still very concerning, right? But um at least but but but also again why are we

still talking about this two years later? This these TTPs are you can't patch for them, right? Uh there's they're using native functionality. Um also it's not just one edge device that's been uh compromised. Um, if you look at the KEV, there's a marketked trend of roughly one in five entries to the KEV have been related to VPNs or firewalls or enterprise routers that are public meant to be publicly exposed. So, that's a big deal. I mean, these things are born with a target on their back, right? It's not a secret. They're marketed for facing the, you know, dirty internet, right? And it's not that hard to get your hands on them and reverse engineer them. Um, and

well, won't speak to the level of difficulty. I uh but uh for the reverse engineering side, I'm sure that can be challenging, but the buying it is not so hard. And so expect that these things will continue to be targeted. It's just kind of a fact of life. So, um, all right, getting back to tradecraft. Uh there there's definitely um observations of defense evasion kind of the textbook one is clearing the the security event logs which generates uh event ID 112 uh which says that they've been cleared um but yeah uh there are other examples as well again this is why you you forward your logs and notably the only reported persistence mechanism is valid credentials which is unusual

because if you get access to something that you really really want, what's the worst thing that could happen? You lose access to it, right? So, usually, you know, a thread actor will want to sync their clause in multiple places. So, if they get kicked out from one area, they've, you know, next time this box re reboots, I've got a schedule task that's going to kick off and beacon back home so I can get my tools back on box, right? Um, not the case here, right? They're very dedicated to lay low with valid credentials. So that's notable. >> So I mentioned orb networks which um again are um operational relay box networks, right? So, uh, we mentioned

these earlier and this is in part and we tailed this off of valid credentials to kind of, you know, drive home a point where these things are basically if like botn nets and VPNs kind of like combined and we kind of came something weird, but there are multiple orb networks that Chinese actors Voltyoon and other ones use to do their various things, right? This graphic is from Mandant. Um, I'm not smart enough to make that, but this is a general like what they look like. So they'll come from the back end in China, hop over the great firewall, which I guess just had a big leak, um, bounce through a bunch of different stuff, and that can be least VPS's,

um, end of life routers, IoT devices, right? There's different like as Manny calls them anatomies, but generally this is the concept to just get across. >> So, and then their exit node is generally they will try to get it in geographic proximity of a target, right? So, a popped router say in that city. Okay. So, if you're only using valid credentials for persistence, that compromised router is going to go through NAT and an ISP in normal use by that victim. If you don't have MFA, that is going to be a nightmare to hunt for, right? So, these things are made up of thousands, right? They can they can churn C2, they can they can churn exit

nodes, so they can just change it up, right? This is something to just consider because um network stuff is great. We're not trying to dog it. Like please don't take that away. But for this particular threat doing this particular form of persistence with valid creds and using this like um your time is better spent elsewhere. >> Yeah, this is this is really cool and and also you mentioned the the leak of the great firewall recently. Also it called to mind u cyber contractors in China also had some leaks that kind of just gave some insight into the maturity of that whole market. >> Yeah. over there >> of interest. These are not run by the

orb networks in general and there's a lot of there's since the initial vault typhoon reporting there's been a lot of good reporting coming out on these networks because people were kind of trying were trying to understand how the Chinese went from very very noisy cyber actors to very very stealthy right so they kind of made this jump to you know being very difficult to detect and hunt for but these are maintained by private companies in China they have been listed um in certain indictments by the DOJ um and And then also some of the other like security vendors. Uh Black Lotus Labs uh by Luma Technologies is a great resource for some of these. And the particular

orb network that Bolt Typhoon was attributed to using was called the KV botnet. Right. Um and again just so I don't land in hot water. Mandy made this graphic not me. >> Yeah. And the reporting on the cyber industry in China that was of course all open source. Okay. >> I can find a reference for that. >> We didn't do you guys any favors uh making the reading list super small but you can ask us anytime. But we have the Microsoft and Scissor reports. We have the Lawbass project which is the living off the land binaries and scripts project that has everything you could ever want to know about native Windows and also Unix and Mac binaries that you

could do that are dual purpose. Um I'm representing the lab so I'm not going to say those next two. Um but also Black Lotus Labs and Lumen. Um >> they're really interesting projects though. GTFO bins is the Unix equivalent of the uh uh law boss project and the WTF bins are um lesser this is a lesser known thing maybe a little newer but benign applications that exhibit suspicious behavior. So I thought that I appreciate that that that exists because there are plenty of weird things that they're legit but they they do weird stuff. Hey. So, >> um, and then for listing viewing, um, Dan Gunter from Insane Cyber has a couple really good YouTube videos on how

to detect law bins within kind of the nature of industrial environment. So, that's within the OT environment, right? Uh Joe Slowick from Data Miner has a talk called Typhoon in a teacup and this kind of looks at the reporting of this and basically how there was so much hype against this and it's not because it's wrong but you know they're like get ready to do you know please address these issues but in reality I think everybody knows Volt Typhoon is not the only ones that um look for to compromise active directory or use PowerShell or use PS exec ransomware gangs do this like daily so in one way or another, right? So, that's something to also

consider. And then the orb networks by the principal analyst at Mandant that wrote the initial reporting on this. >> I want to add to your listening list if you want the kind of broader aspect to catch a thief, right? I'm Nicole, whatever name. Um, thank you. That is a fantastic message. Understand how China has kind of progressed in their >> Yeah, I've heard of that. I'll look that up. >> Yeah. >> And then, um, we do the Winning without fighting by Rebecca Patterson who talks about China's question over there. >> Oh gosh. >> Oh sorry. >> Yes.

>> That's a great question. They want to basically the the computers are getting into we'll say are very important to like US national security especially in the Indoacific like the Pacific Ocean right they want the computers aren't in the Pacific Ocean but um >> that was called out careful what you say here >> yeah but and that was called out in the reporting too it did not go unnoticed >> basically they want to cause a disruption uh in the event of a crisis or conflict with the United States for whatever reason. >> So, >> yeah, we can only >> Yeah, >> we can only speculate to their true intent, but we have to assume conservatively that they they mean us

harm. >> We got one more slide. >> Yeah. Yeah. >> Ask a question before I >> Yeah, sure. Uh, okay. So, um, closing thoughts. We've talked about it a couple times, but please maintain a vulnerability management policy for these edge devices that your you know, organizations rely on really. Um, so watch the Kev. So that's that's probably the number one takeaway for that one. And >> and have a plan. And also, you know, deeper in your network, consider having a different firewall. I know that that is painful to hear for uh network administrators who you know the reason they pay big big bucks for Palo Alto or Cisco is so that they can they can

manage everything is everything's the same right they can manage it from one spot but having you know a little bit of diversity of different technologies can be part of a defense and depth strategy so keep that in mind. Um but yeah, in addition to that, right, think about your logging. It should be intentional. You should be aggregating it somewhere. Uh start small. Don't just dump everything in without a thought. Um do the hard work up front and you'll be it'll pay off. Um hostbased data again for this is is going to be more valuable uh just because most of the observed behaviors happened like on on box and wouldn't have shown up in network traffic. uh build behavioral detections.

Again, we talked about what those are. And then um track where your admin credentials are used because anytime somebody logs in, remember those credentials are going to be in memory somewhere. That's another place that they're at risk. And so, uh there there are variety of technologies out there to help mitigate that risk. Um but but yeah, uh finally, public service announcement. Those botn nets are real and they're composed of small office and home office routers. So if you have a really old TPLink or whatever it is uh switch out there, do yourself and the world a favor by replacing it like sometime >> or even just unplugging it >> because the FBI has done unprecedented

things that required extraordinary privileges to be granted to go patch people's routers for them because nobody was doing it. And I mean that that is just kind of crazy that that's where we're at, but I mean um it's a powerful tool that is being used against us in aggregate. So um yeah, keep your stuff patched.

>> Manufacturers will push that out like you have to um I was doing some stuff you have to kind of dig on their website, but you can find it. Um, there might be some other like third parties that that pull from that and there's a database you can look at, but generally the manufacturers I think are required to put that out or at least notify people when end of life versus end of service. So that answer your question, but like [Laughter] >> that's probably time. Yeah. >> Yeah. Other questions? >> Yeah.

Absolutely. Yeah. Yeah. So, uh depending on what CM you're using, uh you know, there's different query languages, different ALA capabilities. Um but but yeah, you want to make sure that they're um uh being parsed consistently and and are um and yeah, there there are tools out there like Sigma um which is kind of meant to be the universal language of seams um that you can kind of trans it's YAML based and you can define rules very easily in those um and they kind of compare it to uh Yara for logs if you're familiar with Yara. Um, >> does anyone know what Yara stands for? >> Yet another >> Yeah. Yet another ridiculous acronym, >> recursive acronym.

>> Yeah. >> Yeah. Um, Yep. And it's it's basically pattern matching for files. Um, so um, so yeah, there's a lot of good resources out there. Um, the, um, Splunk surge team publishes awesome research on threat hunting. Um, and yeah, there's all kinds of people writing great stuff out there on like medium.com and stuff like that. So, um, but yeah, always keep in mind that, you know, strive to correlate and, uh, build robust detections. >> Yeah. >> All right. >> Well, I'd like to thank uh Jim and Samim and Sam, please give them a warm round of applause. [Applause]

Yeah, thank you all. >> Happy with that.

>> Oh, thank you. >> Oh, hey. Yeah. Hey. Thanks. You bet. >> I work for >> Yeah. Yeah. I mean, it's the reason we're all using Signal, right? >> We've been really fast. >> Yeah. Yeah. Um it's >> put in some temporary mitigations. >> Yeah. Yor >> I think they're gonna >> I can give you my email a lot of stuff like

>> Wow. Yeah. >> Yeah. That's pretty >> rigorous. Do you want to turn off your microphone? >> Hold on. I'm >> Oh gosh, I forgot about the microphone. Thank you. >> Yes, please do.

>> Oh, thank you. Appreciate it. I don't know what he's not there. Okay. >> Well, this is nice. It's got so magnetic lavalier. See this?

>> It doesn't work. It was crap about it. I was trying to wear it up here.

>> How late?

starts at

>> actually.

Got to be

>> terribly.

We're just curious.