What You Most Likely Did Not Know About Sudo - Peter Czanik

Our company was acquired by one identity and this way Todd Mueller, the developer of Pseudo became my coworker, colleague, and I became suddenly very interested in Pseudo. I read about it and it turned out that Pseudo is a lot more than I previously thought. As I learned quite a lot of things, I put together a talk about it. So, quick overview. What is sudo? And then I'll go from aliases to plugins. And finally, as a syslogng guy, I will also show you a quick syslogng configuration, how to do alerting on sudo logs. So, what is sudo? Well,

I asked these questions to many people. I didn't count, but probably 300 or 400 people altogether. And I get very diverse answers. The primary answer was a tool could complicate life. That is already the su command. We also have the root user. Then why to use yet another command?

But even Sys and SysAdmins told me that it's a prefix for administrative commands. And only a handful of them told me, well, if you use sudo on a system, administrated by multiple people, then you can see who did what. So what is sudo according to the sudo website? sudo allows a sysadmin to delegate authority by giving certain users the ability to run some comments as root or another user, by providing an audit trail of the comments and their arguments. So, as we can see, it's a lot more than just a prefix. If you take a look at the configuration of a freshly installed sudo, you will see a similar rule in it. A user or a group of

users can do practically everything on your system. The columns here are who, where, as which user, and which commands. But by default it's all, all, all. And before you think that... Before you say that it's completely useless, it's already useful stuff even with this very permissive configuration. You can see in your logs who did what. As your organization grows and it's not just your best friend who is doing administration with you, you will definitely extend this configuration and fine tune it and start to use lists. You can replace any of the fields in the previous configuration with lists. So you can have a list of users, list of host names, and so on. As soon as

you have one sysermin quitting from you, you want to remove the permissions from sudoers. And if you have many lists, you will definitely forget to remove the user from one or even from multiple permissions. That's where aliases come handy. This way you can really simplify your configuration and also make it a lot less error-prone. Here are a few aliases. A host aliase for web servers, a user aliase for admins, and a command aliase for rebooting a system. And at the bottom of the screen you can see how to use it. So once you have user araces used in many places, you don't have to change all of them, just a single place and the user is removed. Pseudo

comes with a huge set of defaults, which are quite okay in most cases, but of course not always. The defaults, Keyword in the configuration can change this default setting to something else, like changing here the path where commands are considered to be secure, the environment variables kept by sudo, or if you want to insult your users. Well, insulting, you don't have to think about anything serious, just some funny messages.

And these defaults can be also hosts or user-specific. For example, here, instead of everybody, at the bottom of the screen the setting is valid only for the wheel group. So what are the results? A

fun, but not always politically correct message is printed by sudo if you mistype a password. C-SAT means in my age, I love it, but recently the politically correct standard error message became standard, but not everyone loved these.

There are of course many more useful features in Pseudo, for example, digest verification. In this case, you can store a digest of an application in the sudoers file, and if the given binary is modified in any way, then sudo can prevent it from being executed.

Which is quite useful, on the other hand, it's not so easy to maintain. especially if you have many different kind of systems.

Session recording is another possibility in sudo. This way you can record anything happening on the terminal to a file and play it back. For example, if you

It can come handy if you want to know what your administrators are actually doing instead of what they are supposed to be doing. Or it's not really useful if you have only bash listed in your syslog messages. But if you have session recording, then you can see what comments were actually executed under the

These recordings are difficult to modify as these are not clear text. On the other hand, if someone has too much access to your system, then can easily delete it as these files are stored locally.

And here comes a quick demo of session recording.

and then we enter the command on the right hand side, very security sensitive LS. And on the left hand side, first we list the sessions recorded and then we play it back. The last one was this line and the session ID, what we want to play back is actually this one. so we give it to CDReplay and here you are let's do something more serious a shell

whatever, that's enough

and then we can play back even the content of a shell session.

So the user cannot hide what he or she did on the system as it's recorded.

Starting with version 1.8, sudo has a plugin based architecture. What it means, that even basic functionality of sudo is implemented as plugin and you can easily replace it or extend the functionality of sudo using plugins. There are quite a few open source and commercial plugins for sudo as well. I want to demonstrate Sudutare, which is an open source plugin available on GitHub. It can be used to make sure that no user can enter comments on their own. So there needs to be someone else who approves the comments and sessions can be even terminated if the approval side can see some suspicious activity. The plugin itself is developed in Rust, which is a kind of bad news as it's

not so straightforward to compile it and even less straightforward to package it for any operating system. On the other hand, once it is done, it's a quite useful plugin for sudo. So, Let's see how it works. One more.

Yet again, we enter a sudo on the right hand side.

And this time the command I entered is not executed immediately, but it prints that we need sudo approve and prints two numbers. The first number is a process ID what we want to approve and the second one is a user ID who is asking for the permission. So

here on the left hand side I'm running the approving application

and I don't approve it. So this time the application was not executed. Let's try it again.

Okay. So this time it was approved and we could see the output from the session on the left hand side. Let's do something more complicated. Runoff shell. This is where things are getting more interesting.

Let's approve it.

Okay, so I'm working on the right hand side and we can follow everything on the left hand side in real time. So, okay.

Ok, let's do something nasty. Ok, luckily the left hand side guy, the approval guy can see what's going on the screen. And quick enough, pushing Ctrl D, session terminated. So, coming back to my right hand side screen, I'm a slow typer, so let's try to push enter. What happens?

session, uh... Common metal authorized hang up. Meaning that my laptop survived, nothing is wiped.

Well, yeah. So, uh... Let's go back to my slides. Uh... A bit about configuration. Uh...

The configuration of sudo is stored in the EPC sudoers file. You can edit this file directly, but it's not really recommended as

it's easy to mess it up and then sudo will not work. So you should rather use vi sudo as it also does some syntax tracking on the configuration before saving you. If it finds any problems, then it will let you know. If you don't like VI, just change the editor environment variable and another text editor will be used.

Even if a configuration is syntactically correct, it doesn't mean that you can execute anything. Which brings to our next problem. Even if you are a Ubuntu user, where there is no password for the root by default, when you are experimenting with sudo, you should give a root password. Otherwise, you can learn how to break your phone system or reinstall it. So don't forget to have a root password when you are experimenting with sudo.

So, how does the configuration of sudo works? It's read from the top to the bottom, which means that anything you enter later overrides the previous setting. Which means that you should start with some generic settings and add the exceptions to the end of the configuration. Here is an example configuration. It's based on the send to s... ...stidors file. Lots of different defaults set. Then... ...settings for...

...rules for root and the wheel group. They got... ...all possible permissions. Then... ...the wheel group receives inserts.

Nobody else receives insults. Logout means session recording. And we have the aliases from the previous slide. Just fill the screen. So, there was a very common mistake on the screen. Logical mistake. Can you spot it?

It helps if I come back to this slide. You can start with generic and add exceptions at the end. Just look at these two lines. First we allow insults for the real group, but then we disable insults globally, which overrides the field is setting. So it's very common, even I did it a couple of times. It's something to keep in mind when you are editing your CWS file.

When you have more than just a single machine, then you will most likely want to do some central management of your settings. Puppet, Ensaver, Chef, and so on all have modules for configuring sudo.

On the other hand, there are a number of shortcomings when coming to these central management applications. When you change a setting centrally, these are not pushed real time, but there is some delay, sometimes even half an hour, which is a long enough time if you deleted a sysadmin

and the guy still has some access to your machine. Also, if you store configuration locally, if you have too much access, like you give shell access, then they can override the settings locally. And it's also error-cone. these tools don't do much syntax tracking. There is other possibilities for central configuration management for Sulu. It can read configuration settings from LDAP, which has the advantage that settings are propagated in real-time. And they cannot even modify it locally, so even if you give shell access, the sudo settings cannot be changed. On the other hand, there are quite a few limitations, like you cannot use aliases, which makes LDAP a bit more cumbersome to the computer, and also if LDAP is not accessible for some reason, then you cannot use sudo. So

both have some advantages and disadvantages as well.

I mentioned logging and alerting already at the beginning. There is no reason to use sudo if you don't take care of your logs. Sudo itself supports alerting through emails. You can write your own rules when to send emails. All events are logged to syslog. Just as with session recordings, a local log message is not something you can rely on.

Any user can delete those if they have enough permissions for it. If you, and I mentioned sysorienting, If you use Sysogenti for collecting your pseudo log messages, those logs are automatically parsed and you can easily do alerting Slack, Splunk, Telegram, Elasticsearch and so on. We will see it in a minute. I also have to mention debug logs. If you are lucky, you never have to use it. It is used for debugging rules if for some reason they don't work as you expect them or you are often requested to turn on debug logs when you report programs to pseudo developers. Let's talk a bit about Cisco Genji. It's a syscog implementation implemented in C, focusing on portability

and high performance central log collection. When it comes to configuring syscog ng, my initial advice is don't panic. The configuration is simple and logical, even if it doesn't look so at first sight, and often not even at the second sight. It has a pipeline model with many different building blocks, sources, destinations, filters, and so on. And these building blocks are connected together into a pipeline using block statements. Here is a very simple configuration.

It starts with a version number, then

you can include external configurations. SCR.conf stands for Cisco Genji Configuration Library, which is a collection of Cisco Genji configurations. For example, this is which implements student message parsing. Then, here close to the bottom of the screen, we can see a number of building blocks, a source, a destination, filter, and finally a log path, which connects all of these together. Now let's go on to configuring syslogan v4 sudo. Here is a filter, which is matching on the sudo application. Here is a destination where we store all of the log messages in JSON formatting. We use here JSON because this way we can store name value pairs

in our log messages. So the min-value pairs passed from the sudo messages. And finally, in this case, I used a Slack destination. So anything interesting from my sudo logs is sent to Slack in real time, buzzing on my phone or in my web browser.

And here is the most important part of the configuration, the log pair. We use the very same source at the beginning of the configuration, the source for local log messages. Then we have a filter for sudo. And this is the part where we use the log pair. name value pairs passed from the cdlogs, so we can easily match on the usernames passed from cdlog messages, in this case my username. We save it to a file and we also send it to Slack. And

here you can see a screenshot from Slack. with all the very interesting applications I ran through sudo ls in our variations.

Coming up in sudo 1.9, I cannot say dates, but hopefully this year, end of this year, there will be some quite interesting new features. I mentioned when it came to session recording that session recordings are stored locally, which means that users with too much permissions can easily delete it. Not anymore with the upcoming recording service. You can stream session recordings in real time from sudo to the recording service, which means that Your users cannot delete traces anymore as it's not on the machine.

Also coming is an audit plugin. It's not something you can directly use from your servers. It's there for developing plugins which need log messages from sudo so it's particularly an API you can use it to collect any kind of log messages from sudo

you have seen the sudo pairs plugin which was quite nice but it's an external application and not easy to compile or distribute

There is an approval plugin framework in the works, which means that with sudo 1.9 you don't need an external plugin to approve sessions in sudo. And my personal favorite is Python support for plugins, which means that sudo is developed in C,

But using the Python plugin, you will be able to expand it

easily using Python. Which means that, for example, you can write a plugin which streams log messages directly to Elasticsearch or whatever you want.

As long as something is supported by Python, you can do it. This was also one of the reasons for the audit plugin.

So, in the past half an hour, we learned that sudo is not just a prefix, but also file permissions. You can change those using aliases, defaults, secure your setup using digest verification, do session recording, store your configuration in LDAP, do plugins, and do very diverse logging and alerting.

Do you have any questions?

Yes, I have a question. First, thank you for the very interesting talk about the pseudo here. My question is, are there any plans to integrate all the pseudo login with the auditing daemon? Because I think auditing is a very robust logging platform already present on most Linux source systems. For example, the mandatory access controls like Slinux are logging into the So I think from my point of view this would be really interesting to have those sudo logs also integrated into the... As far as I know they are already sent there. Okay. So because then you can also have the initial user ID as far as I've seen this what sudo logs there is only the effective user ID. Am I correct with this? So

the user ID which the user currently has. So not ID which the user logged into initially.

There are quite a lot of blocks sent to... I would say I don't know by heart all of the content but there is extensive of it as well. Okay. Thanks.

Any other questions about the thing we don't know about sudo?

Hello, thank you for your talk. I was wondering if we can use your Slack web to contact you in the future.

Actually, I'm not sure if it's still online, but probably I I use these slides a number of times, so I might receive messages from all the world. I didn't check recently.

Thanks. I learned now that sudo can be used with the pair plugin for ransomware. Seems like it.

I think the pseudo plugin, the pair plugin approval could be used as a ransomware.

I guess... Probably, yes, but as soon as you can extend something, usually it depends on you if it's for good or bad. It was more of a joke, but... It's terrible. I have a question for the pseudo-replay. Can you also go forward backward like a video recorder or is it only playing in one direction? As far as I can remember, there are some keys to change the speed and go back, but it was a long time ago that I tested those parts.

Yes. Okay, it's...

A problem just forward. I'm not 100% sure. I will check it later in that. I don't see it right now in manual, but I have some memories that it can rewind it as well. So...

could you please switch back to the slides? You start again, display the main page please, because I so quickly dash F to filter something. What's the meaning of this feature? I don't know by heart.

Okay. Do you know if there could be some possibility to apply some regex and to use these commands as some kind of DLP system? So for example, if you see that the user does a CD to a sensitive directory, tries to copy files, to SCP, and just alert on this type of regular expressions? Not yet, but one of the reasons for the plugins is to make it easier possible. Thank you.

Are we done with questions?

So I guess yes. Then thanks a lot Peter.