10 - Defending Linux Against Real Attackers

All right, good afternoon everyone. I'm Anthony Navarro. I think there's some feedback there. I'll stay back here. Okay. I'm a Linux systems administrator. I've been doing it privately for about 10 years and that plus 5 years with my current employer, total of 15. And I've almost almost I say almost because I can't go as far back as 30 years Debian, but um I've seen a lot of different configurations of Linux throughout my time and I'm going to specifically just to just for the simplicity for me anyway cuz I work with it every day. I'm going to particularly focus on Rocky Linux because it is close to the Red Hat family and cuz that's what I work with every

day and I'm going to put a particular focus a little bit on SE Linux. I'm going to discuss how to defend against attackers and what to expect in today's landscape. So, let's do a little bit of thought experiment. Let's assume they got in somehow. Maybe you have a default configuration setup of Linux. You just install the ISO and you didn't bother to do any security adjustments to it at all. So, then if that's the case, then one needs maybe one needs to review how exactly do attackers do this on a day-to-day basis anyway. Like I mean, there's a whole industry around it, right? They're called pen testers, penetration testers. Of course. They always focus around how do I get

initial access? Right? For example, is there a regular user that's got a really easy password that I can brute force? That's just an example of many. Then there's privilege escalation after that. How do I go from being just a you know, UID 1000 user, just an everyday user with no access, how do I go from there to a higher privilege, such as UID zero, root? Then, the next question after that, once they somehow achieve that, how do I persist? Because that's the real power is enumerating, knowing your environment, and then staying in that environment, cuz it's kind of useless if you can't come back to it later or stay under the radar. So, of course, then

after that, how do I, uh, from the attacker's perspective, how do I prevent myself from being noticed? Right? I mean, how do I give myself the equivalent of Harry Potter's invisibility cloak or as close to it as possible? And then, how do I prevent the blue team's recovery? So, that's kind of where That's So, there's the thought experiment of a penetration tester, an attacker. So, then now, let's look at from I guess, technically, my perspective, since I'm employed as a Linux systems administrator, from my perspective, job security for me means I got to make sure that my Linux systems are pristine and clean and not being attacked 24/7 or not have anything messing with them.

So, from my perspective at or maybe from a blue team perspective, knowing what the attacker is going to do or wants to do, how do I counterbalance that? And that's where defense in depth comes in. Is my network set up properly? Do I actually have a proper firewall, not just from the operating system perspective, but from above? So, from the cloud's perspective, if I'm using a cloud provider, or on an [snorts] on-prem situation, do I actually have a proper is my third party actually doing things right? I should probably check in with them and make sure. Or from the OS perspective, am I using firewall D? Am I using NFtables? Am I using IPtables cuz I have an older system?

We'll see. So, that's one thing I'd have to make sure of. What services am I running? I guess that's the next question. I mean, that's kind of obvious, right? Is is my Linux system specifically meant to be an HAProxy server where I'm only allowing traffic in for certain applications? Because I have a Kubernetes cluster back there that is serving the traffic or serving the application. Then I need to understand identity. A lot of companies usually have either a freeIPA configuration if you're more open-sourced in that respect or you use that as your preferred method of centralized identity. Or maybe you're in most companies that are using Active Directory. Maybe you all have your names and usually it is

the first initial of your first name, last name is kind of the typical setup. And that's your username. And then as a result, that's how you are identified as you when you log in to each of these individual systems. Then of course, OS. I'm taking more of a of a Red Hat family approach, but there's others obviously. There's Ubuntu Linux which is going to use the more Debian family. And there's much more other diverse operating systems out there that do it differently. And of course, which application? Like I've probably you probably heard me say a moment ago, is it a Kubernetes cluster I'm protecting? Is it just individual servers I'm protecting? Am I defending a Docker swarm? Is it

just a bunch of virtual machines and then within those virtual machines are a bunch of Docker containers or container D. And of course, how am I monitoring all this? Cuz this, you know, at some point you must be thinking, "My god, you're probably running in some kind of You You must be at Pearson Airport cuz you constantly have things going in and out of the time. How do you watch all that?" So yes, those are all layers of defense that I day-to-day, 24/7, because I literally have a cell phone in my bag right now that could go off at any time for a pager duty. So I have to be ready. So and do I have these systems patched?

Are they patched up-to-date to the best of my ability, to the best of the company's ability, or in which the company has allowed me? Then that's a question I have to be able to answer. The next question I have to be able to answer is is SSH configured properly? Because believe it or not, that's kind of one of the biggest deals today is configuring your secure shell. Because of its popularity, it is a huge target. So you have to make sure that's up-to-date and configured properly. And there is countless different standards you could follow. I'm not going to bore you with those checklists. But one of them that's very popular is the Center of Internet

Security Policies, or their checklists on how you should configure your Linux systems. And they specifically say, "Make sure you are not allowing things that to me are obvious, but I want to make sure that I review that and say it out loud. Don't allow root log in just like that. Make sure your SSH keys are specific to you, which you can create when you're configuring these files. Make sure that it isn't easily sniffed out. For example, if you need a jump box, they call it, meaning you're going from one network, you're on a VPN, you're logging into a server, and then from there is where you do your administration tasks. If it's very specific spot you're always

going to use in the IP is consistent, then tell your systems to use only allowing that IP to connect to them via SSH, so that if there's another third-party person who happens to break into your network and uses an Nmap to check that Linux server that they want to look at and say, "Ooh, that looks like lunch for me." They'll scan it, they'll see nothing. Because it's only your IP. So, unless there's an insider that's telling the outside what's going on, at least that will give them grief. It'll give them a harder time to understand or to try to break brute force your SSH, because they actually have to be your IP to do it. So, it's

you're just adding you're just giving them a hard time. And that's good. Services. I don't know how much I can repeat this. If you don't need it, don't use it. And this is something I've seen in a lot of places. Sometimes, some administrators can be pack rats, I like to say nicely. They like to just keep old software around because you never know when I need it. The problem is when you keep this old software around, especially if it's opening a socket, one, it can easily be read by Nmap scans, very easily. And the second, sometimes they're so vulnerable, in fact, that there's a well-known public exploit for it, either through LOLBins or through just common

knowledge because of their patching that's being done. Or it's already been said by maintainers not to use the software for X 1 2 3 reasons, you should probably not use it. So, I cannot stress that enough. If you don't need it, you know, mask it with, you know, system CTL mask whatever that service is, if you really need it around, but don't expose yourself. Or just uninstall it. Just DNF uninstall it. DNF remove, all right? So then back to another thought experiment. What if the attacker had shell on one of your servers? Hmm, let's see. Most will say that once they get root, you're absolutely screwed. Maybe. But it also really depends on a lot of

factors. And if you're in my shoes as a Linux systems administrator, being stubborn and being persistent and consistent are always in your favor. You have no idea. Because it's not as simple as just getting root and you win. No. They have to know when to use sudo and when not to use it. Cuz even as root, believe it or not, you can still stop them. Then you also have to understand permissions. So what if their root user? Maybe a particular file you have chattr and it's immutable with a plus I and they have no idea. Cuz how would they know unless they asked you, all right? So you have to make it always try to

make it difficult as much as possible, even if it's little funny things like that. Credentials. Where did they find the credentials in the first place? Right? When you're in my shoes as a Linux systems administrator, you need to know where those credentials are and you have to be aware of where not to put them. You know, using a tiny bit of common sense goes a long way. Hmm, I probably should not put on a public-facing web service, I probably should not put the MariaDB back-end credentials there, cuz that wouldn't be good. I should probably put them in something like an, you know, a NordPass equivalent or some other place where it'll be encrypted or out of the hands of someone who could

easily see it from the outside looking in. And of course, there's privileged paths. Such as are your permissions set up in such a way or or you know, is it a little sloppy? If it is, you might want to clean it up. You don't want your files to just every time a file is generated, it gets 777 on permissions where root or owner can see it where your groups can see it and just anybody. You want to kind of close it up. If the service has to have seven it most likely doesn't. Six could probably be enough, meaning it just needs to be able to read and write, nothing else. Then you could probably configure it in

such a way where it's only that owner or that user, six and then only needs to only needs to write or only needs to read it. And then for example, four and then no one else from the outside, zero. So there's you have to also know which permissions make sense in what situation. And like I mentioned earlier, avoid horrible services where humanly possible. Make sure it's in writing if you're working for a company that is telling you that they need that service. Make sure you understand and that they understand the risks of keeping something like that around. And what they plan to do to secure it up in other ways. Because that's where boundaries are

going to protect not just you and the company you work for but it's just best practice in general. Right? I mean, we're going to look at discretionary versus mandatory. Which access control is going to save you today if there were some attacker who got shell and got onto your server. Everyone always talks about discretionary because it's the most talked about and it would and it's in my certification because I am a Linux Foundation Certified Systems Administrator outside of work. And that is one of the things they will definitely teach you in that course to go for that credential. However, once you start working with Linux every day, you start realizing, "Hey, I don't have to always use dis-

discretionary. I can add layers. And one of those layers is mandatory access controls, MAC. MAC is when you use a software such as AppArmor or in my example here, SE Linux, that comes packaged already with a bunch of contexts. These contexts are assumptions that the software makes to secure your system. So, for example, a web server. We'll just use a easy example just in the interest of time. An Apache HTTPD server. When you first install SE Linux, it's usually installed by default on a lot of Rocky Linux when you first install them. But, you can configure it further and say "Yes I want HTTPD running. I want it enabled. And I also want to make sure that it's specifically

using only sharing publicly outside of the server that HTML file, that directory. So, in this case, var www html. And I want to use this specific context. The context that it comes bundled with already is called HTTPD syscontent_t, which is to them means that the policy is going to assume that only outsiders are going to see what's in that directory. And that's it. They're not going to be looking at your passwords. They're not going to know or care what users are on that server, and they definitely don't care about any other ports unless you tell me otherwise. Because by default, it's assuming port 80 or 443 when you first install it. But I've seen some places and I've seen

some companies where they use different ports. An example would be that they have an Apache server, but it is being used to as a proxy for a different for example for for like a monitoring system such as CheckMK or for Grafana, or they're a proxy for them, and they'll use a different port in the back end. So, instead of 80 to 443, it's going to something like 8443 or 8080. That's when you have to go tell SELinux through their series of commands, "Hey, I want you to allow 8080 and 8443 for HTTPD {underscore} T to access." Because the same thing applies there. As long as the policy knows, as long as it's on the same as long as it knows the

game plan as you do, it will follow it and will not allow anything else to be accessed from that service. And then of course, there's attacker persistence. They're in the shell and they realize there's way too much here. I'm a little overwhelmed. I'm running out time because it's really time-based when these attackers can stay persistent. They want to make sure they persist. But now the question is uh as the defender, how long are you going to allow them to persist? Or are you going to even allow let them at all? And there are ways to sniff them out. For example, you got to look through your If you are familiar enough with the Linux server,

you'll know how to check the baseline of a Linux server. You'll know that certain names don't make sense when you're looking through the systems when you just type system CTL status or just system CTL in general, it'll show you every single you know, system or name of a process or service that you've configured or was configured. You can look through that name or list and say, "That doesn't look like something that should be there." And you can do a system CTL status of that service and you can read the details and you'll notice, "Yeah, that shouldn't be running from there." So, that's one way. Obviously, there's cron tab from all those users. You can go through each of those users and check

the cron tab if if you've noticed something strange. That's another way to sniff them out. Another way to sniff them out is there's something in my SSH authorized keys that I didn't put there. I should probably delete that or comment it out or take it out and save it somewhere else. Make sure that if if it is legitimately supposed to be there, I'll find out. For now, I'm going to disable it. Unexpected users and sudoers is another one. Sometimes, they'll place something in the ETC sudoers.d directory and you'll notice a random gibberish or a user that you did not put there or was not supposed to be there. I would probably delete it immediately. And then of course, there's startup

scripts. Is it in your Is it in the usual suspect places like user local bin or user or usr local sbin? If it's not and if it's in {slash} root or in the user that you did not create and you did not put there, that's pretty suspicious. I would probably report that. If not if you're the administrator, just go ahead and delete it. Get rid of it. And if the attacker is inside already as the example that I posed to you a few minutes ago, where they have shell and they're looking around, they're trying to enumerate, they're trying to figure out where they can stick around, well, there has to be a way to just

straight up put them on blast, right? And there is. You could be the best pen tester in the world, but if you can put them on blast and sniff them out, well, sorry, man. You're going to have to go somewhere else. So, these are I put some examples up here, and they're pretty good examples, I like to believe, of some places you can use and be proactive. Such as in in the Red Hat family, it's var log secure. Var log secure keeps track of every SSH communication, whether it fails or not, it takes a note of it there. And it even tells you how many attempts. In case it doesn't tell you when you log

in, it'll still tell you this user from this IP tried to connect to your server X number of times. Then that gives you an idea that maybe I should install an additional software to prevent this from happening again. Then there's the SS command that will check anything on the network stack. So, is there anything reaching out to the internet or coming in with a connection established? I should probably find out what that PID is, because every process has an ID, and so then that helps you sniff out that specific process, and then knowing the specific process and service, there's your then there's your smoking gun. Am I using SE Linux properly? Let me use the command to see if there's

any denials. And if the denials look way too sus, then then I should report it and or make sure that I stop it before it gets out of hand. Then there's other tools like LD, which are used for something like if your files have been tampered with. If there's a Linux server you're barely logging into, but there's a lot of changes in files, that's pretty sus. So again, that's another one that you can use to sniff out your attacker. How about the the amount of services they're using? Or the amount of new services that you didn't add there? That's another way to sniff them out, too. Then of course, system-wide, like let's just say you

have a Linux fleet. You have like 50, 100, 1,000 servers. Another way to sniff out your attacker is by keeping track of all of these things all at once in one place. Are you using Elasticsearch? Are you using a mixture of Elasticsearch database with Grafana, for example? Are you using Fluentd, which is only a few megabytes when it's running, but it can save you by keeping track of every log? It takes a copy and sniffs every log and puts a copy of it there at that database or on that other location, such as OpenSearch. So that even if the attacker came in, did some shenanigans, and then tried to change the logs, well, unless they can do it in less than

5 seconds flat, good luck, because a copy of it has already been sent out. So those are ways you can Those are things to get familiar with and things that you can use to sniff them out before they cause any more harm than good right? So, in summary, how do we recover from something like this? I mean, of course it depends on the company. That would be an easy answer. But the not so uh straightforward answer would be try to contain it. So find out where and how, preserve. So take all the evidence you can from everywhere you've looked, assuming you've checked those places that I've suggested, cuz those are good starting points. Find the entry point. How the heck did

they get in the first place? Well, if you look thoroughly enough through those logs and through everything, it'll be kind of obvious where they came in from. And of course, if it was through a script or if they made something, delete it, get rid of it ASAP. Cut it off. Make sure they don't have VPN or make sure they can't come back in. Find the root cause. Once you know what the root cause is, put in the fix. Is it some Is it because you're using an unpatched or using a really old kernel? Maybe consider doing a DNF update. Just make sure you're up on a newer system. And of course, rebuild and restore is your next approach after that. Now,

of course, depends on the company because some companies would just rather trash the virtual machine and just recover from backup. And that's a legitimate idea as well. Any questions?

>> One of warnings or announcements that with Frontier AI is that um breaches instead of hap- happening in months or weeks are now happening in hours or minutes. When you have this tsunami of um vulnerability notices and patches coming out, as a Linux administrator, how do you balance between bl- between patching and just blindly patching versus putting in sufficient um quality testing to see that whatever patches you're putting in isn't generating adverse side effects? >> All right. Thank you for that very good question. This comes down to two things. The business's appetite for risk and experience. And at least I can speak for my experience. If the company is aware you've made the company aware

that this is a serious situation and the business decides that they're willing to take the risk then you do the best you can to reduce it as much as you can and at least the business is on the same page with you. Most of the time the business is very receptive and open to hearing the risks from their professional that they've hired and they will look into the appropriate measures to either patch the like to help you with their dev team or with their third party to find a method that patches it as best as possible. Because unfortunately 100% is not possible in today's in today because of AI like you've mentioned. However even though 100% is not possible you

definitely want to be able to make the attacker you want to make it so less and less worth it for them to the point it slow them down so much to the point where they're like, "Oh god, forget it. Even with the best AI, do I want to spend tokens on this and get nothing out of it?" That's kind of where you want to be at. You want to find the balance where the business is willing to understand the risks of what they're doing and at the same time you're providing the best possible solution you can give them. >> Thank you very much Anthony for your talk today. I think you've highlighted how important the human is. Thank you.

>> Oh, yes. Thank you.