BSidesBUD2021: Sudo For Defense: How Can New/Lesser-Known Features Help You? - Péter Czanik // BONUS

Hi, I'm Peter Czanyk, open source evangelist at OneIdentity. Today I will talk about sudo for defense, how can new or lesser known features help you. First, let me give you a quick overview what I will be talking about today. I will introduce to sudo, then I will talk about some lesser known features of sudo, introduce you to alerting with syslogng and finally show you what is new in sudo 1.9. So what is sudo? In the past few years I talked to many different people at conferences and other events about sudo and I often asked people how do they define pseudo? The answer depended mostly on the experience of the person and the size of the environment where they were

working. One of the most common answers was that it's a tool to complicate life. Which is not so nice, but if you think about that Many people login as root, so as administrator and do everything as that. It's no wonder that they don't really like sudo. But even seasoned administrators say that "well, sudo is just a prefix for administrative commands". Only very few people mention me some advanced features of sudo, like session recording or seeing the logs who did what on a system. So, what is sudo according to the sudo website? sudo allows a system administrator to delegate authority by giving certain users the ability to run some comments as root or another user, while providing an audit trail of the comments and their

arguments. So, as you can see, it's a lot more than just a prefix. But let's see what xkcd says about it. that sudo can also help you to get a sandwich. Just make sure that you use the sudo prefix when you are asking for a sandwich. So let's jump into configuring sudo. The main configuration is the sudoers file and it consists of rules. similar to the one on screen, who can do things on your system, which hosts, which users and which commands are allowed. You can see that this rule above grants practically all privileges to members of the wheel group, but even this This rule is quite useful as when you run commands through sudo, you can see in the logs

who did what on your system. As soon as it's not just you and your best friend who is administrating a system, you will start to create more strict rules. Any of the fields in the above rule can be replaced by a list of values, a list of users, a list of hostnames, a list of commands, and soon you will start copying and pasting long, long lists around in your configuration. That's quite difficult to maintain. And it's easy to make mistakes like leaving a username when you don't want to grant privileges anymore. That's when Aliases can come handy. Aliases can greatly simplify configuration as you define an alias and write long long lists. But you write

this list only once and you can use it in your configuration many times. Which makes your configuration not just more simple but also less error prone. Here are a couple of examples: a host alias for web servers, a user alias for administrators, a command alias for commands to reboot a system. sudo comes with a huge set of defaults. While most of them are perfect for your system, there are situations when you want to override these. You can use the defaults statement in your sudoers file to override defaults. Here are some examples: override which path is considered to be secure by sudo, which environment variables to keep, and turning of insults for your users. These defaults can also be host or user specific

as well. For example, the line on the screen turns on insults for members of the real group. So, what are insults? These are fun, but not always politically correct error messages when a user mistypes a password. It's sysadmin humor. Many people like it, but some people feel offended, so in recent versions of sudo, it's disabled by default. Another lesser-known feature of sudo is digest verification. you can store the digest of applications in your sudoers file and this way you can prevent modified binaries from running. This is quite painful to maintain, but if you have users you don't really trust, it can give you an additional layer of protection. Another useful feature is session recording. You can record anything what is

happening on the terminal and play it back just like a movie. It's especially useful when you need to give shell access to your users, as then you only see that bash is started in the logs, but actually users can do anything on your host. These recordings are quite difficult to modify as they are not in clear text. On the other hand, easy to delete as they are saved locally and with unlimited access users can also delete the files. But stay tuned! SUDU has a plugin based architecture. It started with version 1.8. Most features of sudo are now implemented as plugins and you can easily replace or extend functionality by writing your own plugins for sudo. There are both

open source and commercial plugins available. One of my favorite plugins is sudo pair. It's open source. and it makes sure that no user can enter comments on their own, but there needs to be an approval. The other administrator can also terminate the session if there is something suspicious on the screen. The downside of this project is that it's developed in Rust. which means that it's quite difficult to compile it and even more difficult to distribute these plugins to your hosts. Let me show you how it works. You can see two terminal windows on screen. The right hand side is where the user wants to have a studio session and the left hand side can approve or

deny it. So Let's try to have a session here. I type in the password, but I don't get the shell access immediately, but I need an approval from an administrator. So there are two values printed on screen. User ID and the process ID and these needs to be typed by the other administrator. But first let's deny it. And you can see even if I know the password for sudo the session can be denied. Let's try it again. And this time approve it. As you can see immediately what happens on the right hand side terminal is immediately mirrored to the left hand side. So Let's issue a couple of commands on the right hand side. Let's change to

the directory of the root user. Okay, and then let's delete everything. But luckily on the left hand side the other administrator is watching carefully what is happening and can terminate the session immediately when this nasty thing is happening on the right hand side. As you can see session is terminated. On the right hand side when I hit enter nothing happens. My files are saved from deleting. So I can continue my talk as it's not deleted. Let's get back to the configuration of sudo. The configuration file itself is a single text file. On the other hand, it's recommended to use vi-sudo to edit the configuration file as it not just edits the file, but also can do syntax check for you. So there is a smaller chance

that you close out from your system. If you don't like vi, use the Editor Environment variable to set it to any other text editor you like to use. If you are experimenting with sudo, then make sure that you know the root password. Yes, even on Ubuntu. A syntactically correct configuration doesn't mean that you can actually do anything on your system if you must do the configuration.

The sudoers file is read from top to the bottom, and always the last setting wins. Which means that you should start your configuration with some generic settings, and add an exception at the end. Here is a sample configuration, it's taken from CentOS, and I added a few rules at the end. As you can see, it starts with a huge set of defaults, Then there are two rules to grant root and members of the wheel group all possible permissions. And finally I added a couple of rules. The first one enables insults for the members of the wheel group, the next one disables insults for the rest of the users, and finally logoutput means that we are doing session recording. Well, there was a common problem

in the previous configuration. If we change back to this slide, let's focus on the last couple of lines. What did I say about pseudo rules? always the last setting wins. Which means that even if we enabled insults for members of the wheel group, on the next line we disabled insults for everyone. So if you want to insult your wheel group, then let's reverse the two lines.

Once you have more than a couple of systems to manage, you definitely want to do some kind of central management on your systems. You can manage sudo with the usual central management systems like Puppet, Ansible, Chef, Salt and so on. The downside of using these applications is that they don't update the sudoers file in real time and users can modify the configuration files locally. There is another solution to store your sudo configuration in LDAP. The advantage is that settings propagate in real time and they cannot be modified locally. On the other hand there are quite a few limitations, like you cannot use aliases in your configuration and if you don't have network access, you cannot access LDAP,

then you cannot run sudo idr. One of the most important feature of sudo is that it logs everything what is happening. You can even create alerts. These are good old-fashioned email alerts sent to the configured email address or to root by default. Everything happening through sudo is logged to syslog. Just make sure that your log messages are centralized, otherwise it's easy to delete log messages. If you use syslogng for collecting sudo logs, they are automatically parsed by syslogng, and syslogng can easily send alerts about specified events to Slack, Sprung, Telegram, wherever you want. If you are lucky, then you never have to use debug logs. These are used to debug rules when you feel completely lost, why something

doesn't work, and it's also used to report problems which you do to maintainers. Let's talk a bit about syslogng and how you can create alerts from pseudo-logs. But first, what is syslogng? syslogng is an enhanced logging daemon with a strong focus on portability and high-performance central log collection. Originally it's developed in C, but now you can extend it in Python and Java as well. When it comes to configuring syslogng, my initial advice is don't panic. The SyslogNG configuration is simple and logical, even if it looks difficult at first sight and often also at the second sight. It's using a pipeline model. There are many different building blocks, sources, destinations, filters and so on. And they

are connected together into pipelines using log statements. Here is a typical SyslogNG configuration. which is typical for the var log messages file. You start all syslogng configurations with a version number declaration and then include scl.conf which pulls in a number of different configurations from the syslogng configuration library. You can define some generic options at the beginning and then you can see a number of building blocks: a source for local log messages, a file destination for war log messages and the filter typical for war log messages. And finally there is a log statement which connects these three building blocks together: the source, the filter and the destination. On the next few slides I will show you how you

can send your sudo log messages to Slack and get an alert when a given user is running commands through sudo. First of all we have a filter here for sudo log messages. Then we have a file destination, it's JSON formatted, it's used only for testing, so you can see the name value pairs parsed by syslogng from sudo log messages. And then we have the slack destination where we send the alerts. You might notice that there is no parser for sudo log messages as it is parsed automatically by syslogng, you don't have to define it separately. So we have here a log statement which connects all of the building blocks together. We have the source, the very same source we used previously for var log

messages can be used here. We use local log messages, filter them with the sudo filter. And here we have the most interesting part of this log statement, another filter which is checking a given name value pair from the part sudo logs and if it equals my username, then the alert is sent to Slack. And here is a screenshot from Slack. You can see all the different commands executed by me through sudo.

So, what is new in sudo 1.9? First of all, we have no recording service, which can collect I/O logs centrally. There is also an audit plugin, an approver plugin framework, Python support for plugins, and you can define change root and change working directory from sudo. So, what is the recording service? It allows you to collect IOLOGs or session recordings centrally. Anything what is done by the user through sudo, it's streamed real-time and securely to a central repository. Why is it important? First of all it's convenient. You don't have to login to each of your hosts separately, but there is a central location where you can search through your sudo recordings. It's also availability, even when your clients are

down, you can check the recordings. And also security, the first thing when someone breaks a host is to delete traces, but if those traces are only at the central location, those are much more difficult to be deleted by the hacker.

The next new feature is the audit plugin. It's not user-visible, it's just an API to access all kinds of sudo logs through plugins. It's especially useful from Python, as you can do logging and alerting to Elasticsearch's different cloud providers, without using any external tools like the previously mentioned syslogng. The approval plugin framework lets you make session approval without using any third-party binary plugins. We don't have to replace the policy plugin, but you can extend it with the approval plugin framework. Using Python you can for example easily connect sudo to a ticketing system and allow a session only when there is an open ticket for it. Or limit access to working days, working hours. And I already mentioned Python support a couple of times. You can

extend sudo using Python scripts. It's using the very same API as the C-based plugins. On the other hand, there is no need for a dedicated development environment or compiling anything, so it's much easier to distribute these scripts. You can do it even with your configuration management system.

The policy plugin API decides who can do what on your system. Only one of these is allowed. So it can be enabled in sudo.conf. And obviously you can also implement it in Python, but it's just a nice exercise. I don't see it really practical. but you might have a situation where it's useful. In the next slide you will see a short Python code which allows you to run the command id, but nothing else. Here it is, as you can see the checkPolicy method checks the command name and if it's not id then it returns with an error. Otherwise it continues to set up the environment and allows you to run the command. Here you can see a screenshot.

When the user tries to run ls, then it's rejected by sudo. But if you run id, then it is run with correct permissions. The Iolox API is a lot more fun. You can access input and output from user sessions from Python code. So I have a couple of examples on the next few slides. Break connection if a given text appears on the screen. Break the session if someone tries to do something nasty from the command line. and ask for a reason for the session. Who is the first Python code? It's really very simple. If "my secret" appears on the terminal, then there is an error message that "don't look at my secret" and the session is rejected. Here you can see a screenshot.

There is a file called "myscret" in the "do not enter" directory. So you can see that the user is starting a shell through sudo, changes to the root directory, lists it. Oh, there is a directory called "do not enter". It's a very nice name, let's enter it. So changes the directory, tries to list it, but before anything could be shown on the screen, the session is terminated. It's a bit more difficult to work with the command line, as there each keystroke is sent separately. We collect keystrokes in a buffer and check if rm-fr appears among the keystrokes and we break the session if it happens. Note that this is an overly simplistic example, it's not for production, it's just to

demonstrate that it is possible. If you change the order of options, it will happily let the comment through. It's just for demo. Here you can see the user starts a session, lists a directory and then starts typing "rm -fr" to delete everything, but as soon as "r" is pressed, then the user is kicked out from the session. The third example shows that you can ask for a reason why the session is started. so you can interact with the user before the command is run. And there are multiple ways. The first one echoes on the screen, the second one masks the response with stars. Here you can see the user is starting a share session through sudo. There is a

public reason and the second one is masked with stars. Using the group plugin API allows you non-unique group lookups. It also helps you to extend your policies without replacing the policy plugin. For example, you can use it to check if an administrator is on duty. Or I have a quick Python example. that no password is used if the user is part of a given group name. As it's part of the policy, it's configured from the etc_sudoers file. You call the group plugin API from Python and then there is the actual check that if the user is part of my group then there is no password asked. And here is the Python code for it. It has some hardcoded groups

and my group has my username and if it's matching then I can execute commands without entering a password. One of the latest features of sudo is change root and change-word-hint-directly support. Before this, giving change root access to your users required to give out root access, which is not a good thing as that user can do anything. Now you can easily trace what your users are doing through sudo. And you can also set the working directory through sudo. So if one of your scripts is checking the working directory, sudo can easily place the user in the given directory before executing it. Both of these features need to be enabled explicitly in the sudoers file. You can set a fixed directory in

the sudoers file or make it configurable by the user. As you could see from my talk, sudo is not just a prefix, but many more things. Even the previous version of sudo had many interesting features. LLS's digest verification, session recording, configuration stored in LDAP, and 1.9 introduced a lot more things. You can extend sudo now with Python plugins, with new APIs, and can collect your session recordings centrally.

Thank you for your attention and if you have any questions about sudo, you can reach me on the email address below or on Twitter.