Out With the Old, In With the GNU: Command-Line Habits and Tool Discovery

all right good morning everybody we're gonna get started in a moment my name is Doug I work on a red team for some large financial firm and today just want to introduce you to Lesley Adams she gave this talk at a derbycon this year and she's added some additional content and obviously her talk out with the old in with the new and without further ado let's yeah hi oh thanks for coming to this talk this talk is specifically meant to open a larger conversation between us all and command-line habits and I am going to explain that further in the talk so thank you I am a former sysadmin for Linux specifically did Red Hat syncs Red

Hat six and five and unfortunately for and CentOS and unfortunately also been to which I have opinions about I love L cars as you can see I really wish L cars were real I am currently an IT pentesting consultant for a civil infrastructure defense contractor etc etc I love them I don't evangelize it I started with nano nobody's perfect xfce is great and that's me yelling also I recently became a cyborg I now have an NFC implant in this hand and RFID in the sand so if you want to come say hi after the talk and get my business card from this hand you're welcome to it so why why am I here talking about this

this happens command-line usage habits affect us all the time when we're on site when we're at home when we're teaching other people and when we're learning from other people it can happen on site in front of clients what I mean by that you can have very sloppy command-line habits in front of clients if you hit up and enter and try to edit your code on the line up edit your code enter see if you get a different result in front of a client you don't look really very sure of yourself and hopefully it will improve you I really hope so but seriously why am I doing this talk you can fingerprint CLI habits just like you can code if you

use the same variables and your one-liner loops over and over again people people be like oh this is the same person I recognized this or if you only use a certain tool set and you don't expand your tool usage you get fingerprinted and if you're trying to be as quiet as possible if you're red teaming you want to get as much chaos going as possible precision there are so many tools that are written that we never use and when we forget about how to use them we often slow down ourselves down in the process in slow what we do also tool exhaustion exhaustion I can speak so we have a few amount of tools that we'll use all the time everyone

uses grep you know a lot of people use cat and then we'll use less and then we'll use like a few other things and we'll combine them all into one one-liner and we don't really use anything else that's out our disposal so if we do that what's the impetus towards making something new and improved what's the point why do we write new tools at that point so this tree is old the eniac tree this is the 1970 version and I know some of you might have seen it but some of you might not have this is a lot of branches of distributions of in any akan later eunuchs and what it turns into for BSD so here's what it

gets a little complicated I went and counted all 26 public distros that listed their package totals so just built-in commands what have you an averaged out to about 18,000 commands that gets installed every time you use Linux give or take it's too much and we've used maybe six commands hilarious 29,000 FreeBSD a lot and you see where I'm getting at and we use not a lot of those so there's going to be a lot of redundancies we use grep a lot when there's P grep or egress we don't use certain commands when other people do why is that a lot of that comes from tribal knowledge so wherever we learn it's from somebody else generally I I know that there's the

phrase RTFM how many people actually read the man page now and see what it does does the man page tell you how to use it with other commands though no not really so media books videos whenever somebody writes a book or makes a video they're using their own experience and their own command-line habits in that book so when you learn you're learning the way they do things on-the-job training same thing if you're sitting behind someone and shadowing them there you're gonna learn what they do you're gonna pick up some of their habits classes mentors the same thing social media people will put their one-liners up and Twitter and you'll see what they do then and they're kind of

fingerprinting themselves like hey this is how I type github same thing and we'll get into github later and IRC IRC back in the day people will just say again RTFM but a lot of the people in our industry are new to the industry they're not going to have the time and they're not it's really discouraging to tell someone Daugherty you have them now so you know venture them as best as you can with commands for message boards a little Stack Exchange please don't run the things on there without reading them first so for classes are they formal informal a lot of formal classes we'll look at the most common tools that are used for the specific job that you're

doing how often does that get updated though and how often are your trainers updated on new commands so what are we doing on the command line lips mm-hmm so I love this tweet how many people here use awk okay few of you how many of you using awk is it to do anything other than printing columns out of a line or printing data yes one one person thank you so awk is a-- is a data-driven language just like Perl but when we use it on the command line we use it to print where did you learn it you probably learned it from somebody else like say hey I want to print this oh well you can use awk here you go and so

now you're printing columns and you're using awk only very superficially when it's actually pretty powerful another thing we pipes there are pipes everywhere every one liner I've looked at as an admin and even now as a pen tester I see tons of pipes I see commands repeated and we're gonna go through some of these and I have anonymized the sources of some of them so for example we have printf and then sorting and then cutting it and then using a find again with type 4 awk and then regex but he's also using dot star start so it's extremely greedy regex we've got a cat to see how it's just there's a lot of unnecessary commands here even more we have a grep

going into another grep going into a knock to print out non brace expansion columns in the what is the one of the longest prints I've ever seen we've got I know usage which a lot of that is printing stuff for you who are you copying and pasting it to it is you know if you know what the command does you don't need to print it it actually adds i/o more we have print sort unique sort head awk it you know we just use the commands we know and iterate them over and over again maybe using a different flag but we don't use all the flags we don't use all the commands you know we're getting very little narrow-minded

here as as a command-line folks and I will say nobody gave me powershell examples and I'm kind of curious why that is one of my colleagues gave me one and there are pipes but it made sense it's like ok there it's redirecting output to the next line ok but so if anyone has PowerShell examples like this please let me know and I'll add them to the talk but everyone who volunteered kind of just like shied away and didn't give me any after bit but you know it kind of begs the question is it because redirect errs and unix are just more prevalent and that's why we don't see it in powershell but don't have examples

can't tell you so just more examples here this is one of my favorites for I and automatically your fingerprinting somebody if they're variable is I all the time for I in sequence one day and a hundred they are checking all non defunct PHP procs and if it's over sixty it's going to kill them but it's also going to cat the load average to show you how effective this is every time between one to eight hundred and then it's going to sleep for one but then it's going to kill them so that the gist of that is we are doing that on a really busy server that's about to die why do you need to know the load average

and how effective it is every time you kill a proc you are just causing more problems on the server as security people we run commands and we don't necessarily think about how it affects the server itself we're just like oh this is a target this is whitelisted this isn't scope doesn't matter it's not blacklisted I can topple it over it doesn't matter what I run so when you're trying to be silent on an engagement if you're red teaming what you type matters the way that you type it matters a lot of us use tools like Metasploit that's fine that's not going to get you everywhere you do use command line at times what we type affects the server so

you're not going to look up exploit information on the same network that you're trying to attack why are you also going to leave a noisy CLI footprint yeah you can clean it up but there's going to be other indicators to show that you've been on the server and I'll go through this now so if you're running one-liner and you're not quite sure how it's affecting the server time is a really good indicator so we've got an example I'm just using grep for the word and out of a Doc's file and I'm counting how many lines are in it piping it to WC techo same amount both times using grep with the c flag it's slightly faster so

there's a lot of flags that we don't use and i'll go through a couple of examples later on but a lot of our commands have redundancies that we just just don't use and every time you pipe it adds a little more time to execute a little more time so the benefit here we don't want to topple over a production box even if it's in scope because we're gonna be the bad guy if we're red teaming and if we're if you're blue teaming you got it just like be careful you know it's situation specific but next we have each top so each top is really nice please don't run it on a server that is really low

powered it actually does use up quite a bit of resources from here you can actually see if the commands are nice on the third column so nice seeing makes them higher a lower priority for the server to to actually execute and to use resources for so your negative numbers are gonna be lower priority sorry sorry highest priority and positive numbers are lower priority so here default to zero it's just gonna do everything at the same time so if you want to run something in the background go ahead and nice it and the problem is that when you start a process you can't nice it the same way you have to use a command really nice but you can just type nice

before your command and it all iced it for you all of that command it's so nice so also this is true I've done that when a manager's are walking by sometimes at my old job because top is basically the hacker typer but with slightly more relevant information for people instead of just looking through a log file like I normally would do and hit down down down yeah it's way more fun a VM step so this was vmstat when I was running MSF console and you can see the number of procs running there one and just taking a system snapshot and you can see how much CPU i/o is done just by one command so what I used to do is run this in

screen and screen is great for running in the background and just see how what I'm doing affects the server because like I said a lot of us Ron Metasploit and then wait for shells and finally one of my favorites s trace or D trusty trace on BSD so this shows all system calls for anything that is running on the server you can attach it to a process if it it's a little more handy to me than looking at peak apps sure peak apps are you know all network driven this is going to show you what what it's connecting to and when and exactly what file is using the call to connect what the bad thing is it's

extremely verbose please do not just leave this running in your terminal just just pipe it to standard out or pipe it to less I usually pipe it to less so the other good thing about s trace you can use it to detect botnets that love to fork procs and this happened a lot when I was coming across Zeus so it would make a proc and then fork and so the product number changed so my EPS trace just didn't work anymore but there are flags that will actually follow each forked process for you so really we want to start reviewing the commands we use all the time our tool set a lot of people I know I need to do this just put

all our one-liners in a text file just all of them and then we could just copy/paste as we need them that's good but you got to check on what you're typing maybe there's a better way to type it so review any of those ways to consolidate them you can use github gist's I actually really liked those please be smart about that please don't put sensitive information in your github just please please please and know what your command line inputs do so how can we do better at this I have a little acronym I made fubar different so you can prioritize your flags so for example sort a Q is the exact same as sort type to unique but it's going to be

slightly faster easier load on the server and these things really add up utility usage make sure if you're using aliases be really careful aliases also need to be updated just like one-liners you're going to use on a system so brace expansion or aka curly boys please it's a lot easier to read for yourself and it's a lot easier for the next person on the command line to see what you did it's just please please please don't put variable one variable two variable three it's so much easier to use brace expansion account for server resources is your box of Conroe just like a two core boxer is it beefy doesn't have like you know are you going against a

cracking rig do you have anything else running against it in the background is there anyone else on the server really need to take these into account it might seem like a very quick server but just check memory check this face maybe you're outputting something to a log and they were about to fill out the partition just be mindful that and finally review any new peer tools that are out I know that Twitter specifically people will update github and they'll say hey check out this tool I did on github go look at it just to see what they use for commands especially if they're doing maybe inputting bash into it go check it out there's the SH see exactly what's

being tight how are they doing it and that's kind of going back to the tribal knowledge tribal knowledge isn't bad but it can be harmful if we don't improve how we use what's available to us for example I had a colleague that didn't know that P grep existed they just use grep with a you know capital P flag what have you or Ygritte and or they would instead use grep tack oh there's a lot of commands now that take over for flags and older commands and one of the big problems with all the commands that we get is that the one that we don't absolutely sorry there's no obsolescence of commands we don't take them out once something is better

that's written we just leave them in there and that's why we get up to the thirteen thousand sixteen thousand knot of tools that we saw earlier so some of the information that we can get I personally like to look at anything that's changed I'll do at sea time check in user bin or either Espen specifically after a kernel update just to see what tools were updated if you look in there some of them haven't been updated in years years and so we have to ask ourselves why are we still using tools that haven't been updated in years has there actually been anything that's been written that's better that's more efficient and a lot of times there isn't

and some of the problems with that are we don't have granular searches for a lot of tools that we have on the server we you can install apropos which is up here and say apropos firewall and it will look for tools that effect the firewall in some way based on the description but if you don't have it installed you're not going to know about it unless you start searching repo this becomes a bigger problem because you can't look for hey what what new tools came in the repo recently it doesn't there's not a good source for that man resource searches for BSD and Linux don't provide complementary information for other tools so man pages tell you

how to use your tool that you're using right now but they don't tell you how that tool we're in conjunction with another tool they don't tell you that hey maybe don't use this on this kind of server it's very there's no context so they'll give you examples but not not in a greater sense so there are a lot of the problems that we have here in law I'm giving this talk is that we have too many tools we don't use them and when we use them we're using them in a way that affects the server that we don't really take into account so going back to Metasploit you can actually see how it affects the server I'm trying to be a

patron for the server here like please don't hurt the server the server needs to give you information don't kill it you know be gentle be night I sit so eventually we go back to fubar yes all the over there you go pardon me so that's a lot that I have I really wanted to open this up to questions and discussion because I've looked for a long time and there is no great way right now to say hey these new tools have been updated let hey there's a newsletter like there's no newsletter or digests that comes out you just update your kernel and use the same tool that you're using where somebody will write a tool but they won't use new commands in

it I'm saying so thanks y'all and anonymous donors of your one-liners also but uh any questions so far yep I gotta say do you have like any stories of we're using the wrong commands as like made a big difference yeah though you said the wrong commands or like non optimal commands okay I had when I was in admin there was another admin onst on staff that was working graveyard they couldn't get the control panel to work so instead of fixing PHP they rpm removed PHP they removed PHP from the whole server and then left a ticket note with the zero byte stats of user bin PHP and left for the day and we didn't know until the client called and

said hey why can't I access my control panel and we look and there's a ticket and the dude was gone just gone for the day no PHP on the server so that's one egregious thing like don't use your tools this way but I've also seen one-liners where people will use find commands ice I love fine command it's great using fine command and piping not to RM I get scared a lot of times because they don't check and see what it's going to delete first there's a print command for that you can use the print flag and it will say everything it selects and then you can use the delete flag which will prevent it from deleting

directories say so there's really good ways to use tools and there's really unsafe ways to use like really common tools also okay I mean I have another one yeah well unrelated you said we should kind of avoid using the same like basically changed up our command-line habits so that we don't get fingerprinted have you been fingerprinted by that or do you have like techniques for fingerprinting people based on their command line habits I have been fingerprinted by it specifically bash scripts the variables I used to use were very static I didn't change them and then I would just use the one-liner over and over so when people got on the server they're like hey Lesley did this because I can see

this they didn't even have to look at batch history to see who logged in or what have you so it becomes a problem in that sense I haven't seen it a lot with my colleagues when we're doing pen tests but a lot of times when we're on the command line it's to grab information from logs and at that point I see other people's habits I see the you know did I type it right no I didn't I'm gonna hit up arrow you know edit it a little bit and type it again and so that every time we do that you're just running it on the server and sometimes it really affects it depending on the server resources

nobody else fine thank you hey okay so I define a lot of aliases and functions in my bash RC and it's great for my convenience but I worry that it's causing me to forget how to do things that I commonly do so we have a comment on that do you recommend to not do that and type it in manually or what should I do here so I do use alias but I kind of refresh it sometimes sorry I cover we're talking about LS shortcuts before you know everyone wants the human readable flag on LS so a lot of people use alias to do that cubed readable flag but they also add maybe tack a to show hidden

files or they'll add another another flag they'll have preferred hey this is the LS I use the same as process checking people will have their you know their own BSD or Linux based Flags they'll use to show the process tree and they don't do anything else there's gonna be some reason you need to use it differently you might need to reverse it and then you have to go back and you can't use your alias anymore because now that's set to something else so I I like having flags at times aliased but they're not out for every use case there's no way they can be so just go I would just check it like every now and

then I'm not gonna give you a time frame to be like check it every week check it every day but just go back and refresh yourself on it because maybe you learned something new and you want to actually update it and our questions are there any resources you would recommend to a new user for Linux so they learn the command-line properly this one is really hard because a lot of Linux tutorials online are very similar and they don't they're you're getting the user experience of whoever wrote it right if they're learning it from someone else online they're gonna use the same thing and this is one of the bigger problems a lot of Linux resources online just don't

update ever they'll write it and then they'll set it and you'll see a page from like 2003 with the commands and it's still up and nobody touches it and everyone still goes to it and it's still getting site traffic and it's still up there yeah it's relevant but we can make it better we can do a little better on it so yeah I mean you're welcome to ask me please ask me all your Linux flag questions if not I don't mind at all but yeah I'm honestly online resources they're good they're good starting place and this is part of the problem there isn't another resource hub and maybe I should make this and then cry every

night for ways to improve your your bash view your your zsh foo just to make it more efficient on the server in you maybe I need to make those resource any other questions

all right everybody give Leslie