← All talks

Protecting the Penguin: Linux Security Fundamentals

BSides Dallas/Fort Worth · 202155:2934 viewsPublished 2021-11Watch on YouTube ↗
Speakers
Tags
CategoryTechnical
TeamBlue
StyleTalk
About this talk
John Robertson walks through a comprehensive approach to securing Linux systems, from distribution selection and hardening tools to vulnerability scanning with OpenSCAP and long-term maintenance. The talk covers practical controls—partitions, mount options, SELinux—alongside the administrative and vendor-relationship practices needed to sustain security posture over time.
Show original YouTube description
BSidesDFW 2021 Track 2 Session 1 - 06 Nov 2021 (wait for it... first 90 seconds the audio was not working.) Protecting the penguin! Linux security as amour! In this talk, the speaker will go thru securing a linux system. This talk will start where all good security should start - at the beginning! The talk will describe the how to develop good security posture. Does the selection of an linux distribution matter? Yes, actually it does! We'll look at the security controls and how they need to be applied to the development of a server. We then look at the tools to build boxes - namely by establishing a vm environment on which to develop linux servers and the scripts / techniques to secure them. We then use this tool to build boxes. Dealing with the box after the build is like traveling to Mars, once you get there, take a picture of your feet! Take your favorite control checking software (OSCAP?) then get a good measurement on your vulnerabilities. Then the fun really begins - what to configure first, what to throw out, and what to add to it. We then talk about ownership - how do you keep up your investment. Madrob is a 20 year veteran of all things infosec with regards to linux and solaris systems. In this time, John has been hacked, smacked, fired, and spent a lovely morning discussing the finer points of system hacking with our favorite three letter agency.
Show transcript [en]

give away all the all the beans here so i'm just going to stick with i gotta start over neat all right good all right good morning b-sides dfw i'm john robertson i'm your track to nine o'clock speaker we are here in the greater metro dallas area and i'm going to be talking about linux security this morning i'm going to back up you know what we got to do that let's go back here because i apparently was doing a mime impression and it didn't work out so well so this is links between a rock and a hard place dealing with security audits this is my own opinion these are just some guy who has worked on some linux

boxes here and there and is going to be sharing you some of some of what i have learned these are not necessarily technical controls or some administrative controls or some other things that i want to talk about more specifically why this talk i love a good linux security talk these are just something that's really near and dear to my heart a bunch of years ago i was involved in you know being somebody's lunch i hadn't taken care of my security like i should have and i got got hacked and i think a lot of us in this business have probably been hacked and it's like yeah that's not cool we're not doing that again so i get under this business and it's a

really it's just very important to me it's very near and dear to my heart so additionally when you are in charge of linux security securing a system to the point where it'll pass an audit and by passing an audit i mean whatever entity is certifying you saying you're good saying you can be on the network or just generally your own peace of mind being able to sleep at night that's where i'm trying to get us today and these aren't always necessary technical controls some of these are administrative some of these are just again your own peace of mind and things are going to help you be a better security guy also this is going to help you better

understand your vulnerabilities and how they are fixed and the linux and linux like things are becoming more and more common i know microsoft is with their newest version they're doing some linux things i see different linux boxes i get phone calls on occasion hey i got this weird appliance and it's broken can you help and crack under crack open the hood and voila there's a linux operating system staring back at you this awesome linux has all kinds of neat features so every day is halloween links all right moving on there we go uh here's some resources i'm going to pause for a second these are not real great for the online but this my contact info is at the end of the slide

if you want into the deck and if you want to reach out to me i'll gladly share some of these these are mainly vulnerability finding help you figure out how to manage your vulnerabilities openscap if you're not familiar with that it's an awesome project for linux i think they're getting into the iot business as well and they are i don't think they do a lot of windows right now but i think they're getting there the uh stig viewers unifying compliance hub the cis security cis security guys those are kind of not anonymous access then some of them might be paywall but they're decent resources to look at i would if you've not been involved in

this field terribly long time i would really think twice about taking my vulnerability list and handing it to somebody more resources kind of footstop for you these things here and really celebrate my ability to put black backgrounds with blue fonts again the slides will be available later if you want to partake of that usenix that is a older organization that does they cover all the what's the unix and the linux variants uh they're really if you want to really want to get deep into this field using usenix guys and girls are hard to get involved with ssh security can't put something enough there's a lot of that out there and i've seen a lot of

folks try to do the google admin thing on it you need that subject is deep enough where you're going to want to really do the research into it and then some notes on using partitions again this point i've seen a lot of folks who hey i'm just going to throw up a box real quick and we all know happens when we throw up something real quick to demo something it becomes dev it becomes test becomes staging poof we got a weird box in production and it has no partitions in it so bad things can happen we'll talk more about that in a moment bottom line up front if you are in the beginnings or you're kind of

dabbling that way or you're one of the this is your other duties as assigned good system admin work it's a graces for basis for great information security work a lot of things you would do as a linux admin if you do them well like you should or you are have the opportunity to do them well i should say that that will help the ia guys out that i ask guys out later and hopefully slow down the hacking type so just a little bit i'm not sure what kind of internal program you have for developing your security developing your boxes developing your mindset virtual virtual test beds are a really awesome place to start unfortunately i can't a little foggy

this morning i can't so i can't do my old man voice but back in my day we'd go to the to the bridge over on 9th street we'd buy our servers and expired hardware out from under the the deal it was uh first friday sale and unfortunately a lot of that's gone away but it's still kind of a relevant thing virtual badge is a better way to go though so there's a whole other talk if you want to get into figuring out how to make your laptop good for doing virtual test bags softwares and hardware and some good things and some not so good things and we get to probably the one statement that's going to help

and it's a little narrower than i'd like i mean buy me schools buy me books send me to school and i still can't edit my slides understanding where the areas where you need to harden i want to broaden that a little bit know what's going on with your stuff that's why you're developing these things that's why you're going through your checklist manually that's why you're developing these things manually so you know what's going on probably one of the wisest things anyone's ever said to me in the security business and the admin business is know what's going on and be able to explain it that piece of advice advice right there i've seen people make careers just on

that one piece of advice and that is where i will continue so we'll go on uh tools there are a number of tools you can use in this field i have often marbled when i was first getting into it how do you guys know these tools how do they know where to find this that or the other thing well twitter uh other admins dealing with user groups that's how you learn a lot of these and these are some of the tools that i use off and on and a couple i want to foot stomp here script we've had that uh i've seen on various platforms i think the rocky os guys have it i think

the uh solaris guys might have and there's a couple other platforms that have it basically this not only captures it says bash history but then some it captures what you're typing in and then it types out the response it captures a response you're basically getting a recording session really awesome tool to use and you can also use that to kind of develop your own remediation scripts if you choose to do that this one i kind of lightly recommend excel slash numbers if you're in the mac world great for tracking thing great for doing quick lists and sorts and things of that nature uh death you've heard of death by powerpoint well it's cousin is death by

spreadsheet and it's a very real things it's a tool you can use it i've done some great things with it but be wary really if once you get too far down the road with this you want to get into a database light tool something uh obsidian some other lightweight database you can put on a laptop keep with you track things it's going to be a lot faster and a lot easier for you producing reports and getting results out of those i did i don't have a good database recommended because there are a number of them out there and it's like religion i'm going to tell you what's the best out there openscap we've talked about that

there's you don't want to be running all your checks alone this will go out and do scanning for you openscap.org great resource something that's got a bit of a learning curve to it frankly there are profiles there are what do you want to put into it how strict do you want to be how far checks do you want to do openscap is one of the tools you can use there are other tools out there that again your mileage may vary using those a couple of a couple others here i want to foot stop for a minute putty if you're using it it's an awesome tool it is a great tool uh back in the six eight six nine days i think is when

they added these cereal components very useful in the seven seven one days i started doing uh whatever kind of memory cards that you use for identification again that's very useful very good information out there but please please please for the love of all that is holy when you bring putty in have a good program for it some of the areas i've done some consulting things in putty was that dirty little secret that oh we have this really neat tool and it does some things and guess what that's awesome that you're able to get in you do able to do this thing real quick or you know one of the appliance guys needed a tool real quick just to get in and do a

console well it sits out there on the network and somebody will find it two and a half years later hey i found this really neat tool so guess what i think it was putty six late six nine early 7-0 we had some really nasty vulnerability and some i mean literally the entire internet just had it's like okay everybody stop go abduct your putties delete your old ones do that they're still old putties out there and they're still problematic and please just have an agreement with whoever does your software or whoever you know if it's that if this is your own personal stuff update your putty on occasion please don't leave the old versions out there

it very very analogous to the christmas computers hey everybody got a christmas computer for christmas hey i put it on the internet and nobody thought to patch it and well it is literally christmas for you and it's christmas for other folks so update your stuff regular expressions yes they're hard yes they can be difficult it can be very trying and it is it's one of those that you just need to do it okay read i've got a couple of books recommendations and that's on that's one of the books i carry around that is a good recommendation please just get into regex i know of so many folks who have uh it's portable absolutely thank you uh

it's absolutely portable it goes from perl it goes python it goes i'm sorry can we we can stick that in excel can't we that's awesome is there anything you can't do with regex just an awesome tool that you just again buy the book start reading it it's good uh these next two guys screen and t-mux they are kind of cousins to script screen is a really awesome tool uh if you have you know i'm typing along i'm doing something and all of a sudden my session goes away i'm doing remote stuff and it goes away well screen kind of helps you with that you can go if you've started it correctly you can go back and rejoin your session and go

right back to typing or if you had a long script running that script should hopefully not notice when your session drops you go back and rejoin the session life goes on that is not 100 guaranteed again reminder this is my opinion my views so your mileage will definitely vary definitely definitely minus my very now there's a new guy in town and he's at least a year if not two years old i first caught my attention about two years ago i saw some talks on talks on tmux how you can divide up screens you can do all kinds of neat things with them and yeah that's a some distributions have decided that's going to be a mainstay if you want bash

on this system you're getting t-mux and that's awesome if you're an os guy and you're used to it and you know what's going on it's not so awesome if you're a scanner or you're another guy who still thinks the 2x kernels are just all that i know you need to you need a 3x for it you know getting into the older guys that's the newer guys excuse me that's awesome tmx is something again it's not as bad a learning curve but it is something you need to read the man pages uh i'm so sorry i don't have my uh i don't have my uh uh really neat slide for uh neat slide need me for this

if uh there are two books on my bookshelf one is the t-mux man page and the other is the kern and richie's uh c manual okay moving on there we go we didn't jump out okay so those are tools again your mileage will definitely vary uh chat if you want to talk chat up tools that's an awesome thing successful also starts with a great install a couple of things where you get ready for install time first of all know your audience what are you going to do with this box if you're going to go do some big application do they live and slash off do they live in user do they live on their own partition

those are things you're going to need to kind of know also you're logging your logging is going to be very important i cannot foot stop that enough logs huh yes definitely you want you want you want to be tracking it you want to be in your logs and you want to get the locks off the box yeah come on leave your black box just somehow we got to have that absolutely you put it anywhere else outside the box exactly

get it off the box as quick as possible hourly daily you know and gosh hourly if possible i've had things that were great at 8 8 a.m and come midnight they are dead and i'd really like to know what happened because i got a thousand more of them just like it and i yeah i've done that bad thanksgiving trip thank you no never want to do that again so now that your audience uh partition's important minimal installation please for level oh it is holy don't do the uh just install everything yes you've got the disk just what's the just because you have the disk just because your memory doesn't mean you should use it all right now

use that for later for your logging or team accessions or whatever you need to be doing that minimal installations and then know how to add on stuff later okay yes love love those packages love those install groups those are awesome now one thing that you do have to really get involved with what you're doing on your os you really need to know that once you do this often enough you will develop a hey i don't have to have it all but i know i need extra lives for this i know i need you know my x11 i don't need a full-blown graphical install i just need enough to spend on the next window on occasion

you will have to know how that all that's going to work watch your mapping no again know thy audience talk to the backup folks oh goodness so especially in the case that you're doing like a brand new release hey uh rocky os 9 is out we're going to go do it that's awesome we're going to go build some test test boxes have a conversation with your backup guys because their bonuses their continued employment is probably based on your bosses performance indicators as well as based on being able to back up your box and restore it when a booboo happens and a booboo will happen so if you get one of those brand new boxes and i say hey i'm brand new i've

got to have brand new butter fs i've got to go brand new ext9 that's awesome but if your backup guys can't understand it and it's not guaranteed they will yeah guess what we can't restore life life is not fun at that point there are ways that your backup vendors get around it and yeah but you don't want to be doing that at sunday morning at two o'clock in the morning and my prod whatever box is down and i've got you know event horizons coming guys we got to get this going and yeah talk to your backup folks one of the later slides i want to talk about and you're taking care of the personnel backup folks are one people that you

definitely want to be buying the drinks for and buying lunch they are critical to your success they show up on your audits that show up on all your stuff keep these folks near and dear to your heart if you want to it's great job security if you do work in the backup area god love you there's a pretty quick exit though you lose enough data or you lose data that's a problem finally get an os for which you can purchase support this is predominantly again goes to the mindset of how you're securing your stuff how your auditing needs to work i've seen a lot of shops we don't want to buy support for our dev and our tests and

our stage we just want to buy it for prod which if you're cutting something and you're down you're way way down the list i i would see that as reasonable uh you you got to practice the way you play though folks i'd at least keep my test or my stage or something like that yeah keep spent a little bit of money there uh i know red hat's got a new program out what was that march april they did a new program yeah so be doing that sort of thing and the trick is i want to go figure out that tmux problem in stage or test somewhere other than prod again sunday morning at 2 am i prefer to be asleep not working

on something monday morning at 8 am i don't like the brick wall that i don't know is coming i don't cringe when i walk into my office get an os where you can buy support because this leads to some other really good things so we're going to divert slightly go over to partitions not spend too long here break out your partitions please i've seen i don't know how many folks that say hey i'm just going to do one big glob and we're going to send it on to happy little way it's yeah just swallow it all in a few cases that's okay if i'm doing some micro install on a raspberry pi raspbian whatever okay maybe

but by and large if you're doing something that you're going to care and feed for for a long time you need to break out your partitions especially your logs partitions yeah definitely your partitions absolutely absolutely controller you must be uh you must be a dba sir no just just oh that's all good uh yeah don't put all your stuff on one deal the particular gotcha is slash boot i have seen i don't know how many installs that they didn't put reasonable stuff on slash boot they back in the day i think centos was doing like half a gigabyte kernels aren't big that's why we'll just use a little space here uh again know thy audience if i'm in a

situation where i've got to be having multiple kernels available to me for boot time yeah that's going to bite in the day and age of you know terabytes on your linux systems come on a couple of gigabytes for the couple of gigabytes for the boot partition is not unreasonable just let him have what he needs to have okay why we fill it up we crash it life gets very unpleasant virtual environment concerns so here's the neat trick in the hardware day age let's say the hardware age we were concerned about okay i got these discs i need to glue them together with raid fives i need to do all these other different things well these days we're doing cloud we're

doing things like uh home nasa's home raids uh uh free nas and doing hong kong so a lot of that is taken care of somewhere else so on the os level i don't worry as much about that you do need to be mindful though when i'm putting a system together do i want to have just a couple of partition slices i'm going to call this disk do i how do i want to do that up for the level i i would not do more than a couple of disc or what we're going to call disk virtual disk in this area maybe one for the what i call the head i'm going to put say my boot

my kernel this stuff i'm going to take opt var and some other guys put them somewhere else and i'm going to take the whatever application this thing is i'm going to take him and stick him somewhere else maybe on separate disk the awesome part about that is when you have to go like you got to leave a system and it's time to copy the you know you you park the park your v environment i've got to take all my sys my system components somewhere else so i'm going to copy these three big files that represent my three big drives well that's awesome but if i got one big system i'm gonna be one big drive i'm

gonna be sitting there waiting wait but if i can take the application partition and just send him somewhere else and marry him up to another and i suppose i've got another box and i've got another hat what i'll call ahead and it's where my slash boot my opt my user all those guys are parked somewhere else or i've got another copy of them i can marry those up to that other guy and i'm good to go you're not sitting there waiting the point is if you have one one great big thing one great big box um you will it'll take it may take you a while to get up get out of the environment if

you're trying to get out in a hurry and that can be a problem mount options this is another situation where you are very concerned about mount options because okay if i've got say no exec there's half a dozen yeah mountain yeah fs you got a bunch of those things that you're worried about and somebody in your application just feels like he should be able to exec his stuff out of slash temp well yeah no we don't allow that and uh then when his boss and your boss finish their conversation at lunch yeah you can go ahead and yeah no remount is an awesome option just be mindful of those are a control those are things you

need to be mindful of and just again know know what's going on your stuff i will tell you i've i've spent a lot of time troubleshooting a no exec set in a slash temp or var rar temp option i've spent a lot of time troubleshooting that and i wasn't alone because i and my application counterparts have no problem engaging our vendor i mentioned buying the support this is part of why you buy that support first of all you should have no problems talking to that vendor even if you just go to red hat and buy the 335 option that still gives you rights to email them i would try and talk them into a phone

call i think you get what eight to five support on the no cost thing you'll always talk to you on the phone don't have a problem talking to the vendor they can do a lot of things for you i will also tell you i have taken control issues i can take controls i mean vulnerability controls i have emailed them descriptions snippets all kinds of weird stuff like hey this control doesn't match in your os what do we do about it usually i get a oh so terrible tier one uh we'll typically hand you off to the security team hopefully those guys have seen it before and they they can help you get around a problem

quickly at a minimum if i'm staring down an inspector and i say hey you're right i got a high level vulnerability i've got it it's what is it low medium high and whatever the other one is extremely bad extremely bad vulnerability here but here's my email trail i've talked to the vendor on this and this is a problem nationwide worldwide whatever wide you know system-wide if i'm showing the auditor that i've been talking to this vendor that's there's going to be some points in your favor it's like hey they're trying to fix it they know it's an issue there's been progress absolutely and you show your iais team hey we're working with the vendor one of the sites i worked in

you did not leave on a certain level of box if this was considered like primary you didn't leave unless you had a ticket number and that's part of my was part of my checking out of the building it's like okay uh yeah i am so-and-so i've got this system down and oh here's the the ticket with canonical here's the uh the other folks over at bmc have given me these ticket numbers so this is this is how we're making progress we're showing and you know they have your phone numbers and get a hold of you if they need to that's very important also hesitate on that last line they're paid to be your friends that makes them sound really bad i've

i've worked with a lot of really cool vendors folks that and i've even had them say hey your support died a month ago really can't spend a lot of time on this but you need to check this note or that note be engaging with these folks and please also in all fairness the vendors please understand that is how they make their living that's how they do things and it is important to respect that and just have have an open conversation saying okay what do you guys need to do for us to all be cool here what do you guys need from us to show your boss that you're doing some good things and that way you keep them engaged and

they can also hook you up with other resources if you're looking at a problem or you're looking at how to develop a system to do something you talk to your vendors like hey man i don't need names or stuff like that but or is anybody else doing this i'm trying to do thing x talk with those vendors again this is probably old hat for some of you but if it's say something maybe you haven't done or maybe you don't think you want to do reconsider that and doing the lunch and learns that's another thing that i've benefited from greatly one one last thought on the tier one guys i've had them pass me off to folks that

ended up being the security writer like one of the head shed security guys for a company that was a very cool individual to talk to and i got to the point i could email him directly saying hey this is broken or we noticed this or we you know hey we saw this on the news or better yet my boss my boss's boss saw this in his copy of pc weekly and he wants to know what's going on able you're able to explain that as i mentioned before know what's going on and be able to explain it all right moving on software selection again do the minimal install it lessens your attack surface i know thy audience know what they what

their needs are going to be are we dealing with bmc am i dealing with what other other vendor that's going to have products on there you know big hint third party guys don't always know that 777 is a bad thing and for some people that's an airplane other people just like yes [Laughter] yeah that's uh my my personal favorite is that you know it helps us get christmas faster because uh yeah no we're not going to do that keeping your systems patched up for you secure your systems once you get into the software business you know you've got the install done you got the configuration done then you secure it then you go back and patch it

secure it you know double check it to my horror once i was patching up a big really big box went through all the patching got the box brought back up it's off serving the customer and security comes along they do their thing and it's like i thought you were working on databases well i am why the hell are you running apache like oh mother oh that is holy uh yeah that prompts another conversation with the vendor it's like why are we running apache guys yeah and that became part of my checklist is like check for apache why do you have apache or just do a quick search for ssl.conf where is it right in there we

removed you go away yeah well that's daily daily getting your system patched up for your security system and fewer lines just for your files to scan configs to check and that will speed up your scat scans uh again i i've don't know how many checklists i've been working through it's like oh goody i've got to go secure this graphical environment here's a graphical environment it is real quick to say are you using this graphical login well no see ya other times it's not you're not so lucky but by and large that is a much better thing all right ssh security we're coming up we are at it's 9 35 now and i think you're going

to cut me off at what tin till okay or just whatever yeah i'll just keep going and we'll get there when we get there all right ssh security this is something in my opinion that you should really spend the time on i mentioned you know on your bookshelf you should have the what do i tell you the t-mux man page and the current richie's c manual well if you had the third book this is probably the ssh security uh some this is really kind of cool a tattooed tattoo lon you know i practice that every time and i never get it i'll just take that off the slide next time author and continued researcher i mean the gentleman's still

out there doing some things and being awesome to just follow him see what he's thinking get on his ssh.org believe is the site he actually owns so that is very cool you wrote it you know the stuff on it and you're pretty much everywhere even though yeah even though people don't know it uh yeah yes yeah that's and you know that's a whole nother answer hold on a slide see let's you go start going through this stuff and you learn more uh respect the subject read he especially the security intel uh i've been i don't again i don't want to use my own land voice but i've been dealing with ssh a while and to this day i'm still getting new things

like have you considered this and no i had not i just assumed i set this flag and it's like nope there's what it wants to do there's the suggestions you make and the configuration files and then there's what you tell it not to do and those are three areas where are your keys kept no really user.ssh not everybody does that and that's getting to be kind of a thing when you talk to new boxes when you go talk to strangers they're telling you hey this send me a key because i'm you i don't leave it with you i don't trust you that's part of your ssh security i have my own special place i stick that

only me oh and continuing on there's a lot of awesome stuff in that configuration file uh respect that configuration file this is one of the ones that will shoot you in the foot and you don't play anymore if you get this wrong terribly wrong uh what part are you listening on everybody seems 22. so does the bad guy that's not always the answer i mentioned when you get new [Music] creds on a box i starting to see more and more where people are saying yeah we're on this port remember i told you to go have lunch with your network backup with your backup guys have lunch with your network guys too make sure they're taken care of because

if they don't know you're on 22 thanks thanks for playing we uh we and there's there's a whole thing on that so we can talk that in the uh yeah the next the next slide in the next on chat uh listening address so here's one of those nice what i call ankle biters yeah if it says zero zero zero zero you're listening to everything and that's awesome you really wanna have your bosses mind blown just have your pen tester come in and ssh to your website and it responds to you no you know what at a minimum turn that one off make sure you you know you may die on other hills but not on that hill

syslog log whatever info whatever you feel you need just don't turn off logging so many of my problems have been solved being able to look at that a lot of what you will see is the permissions are not correct when you do ssh you use a strict configuration mode yep that is your friend i've yeah it's one of those you let the juniors bang on their head on that one for a while then you just go do a quick change on your you're awesome x forwarding s11 x i just thought there's just not a great x-manual out there right now and again maybe in the chat somebody can recommend a really good x-manual if you don't need it turn it off do what

you need wolverine never heard of it there we go ah okay sorry shifted i didn't shift gears fast enough anyways ssh that is something you want to care deeply about and especially x11 because x windows security yeah one of the stars no no get out just no boo uh we have so no x windows how much do you need really how much do you need do you need a graphical login do you really or do you just need an application that needs to spit out x lock or x eyes or whatever x yeah kudos you can get x bill running uh do you know the whole thing what well a portion of it dude i just need the

lives just enough to spit out the wind or somewhere in between know what you can what you're able to do and just appreciate again the previous slide which i'm not going to go back to talks about x windows and ssh there's nothing quite like having one of your security guys show you that you turned off x forwarding in your own ssh and that's why your x windows aren't working yeah then your customer gets to the point where they recognize that as well so just appreciate those who have a relationship and be respectful of it firewalls at a minimum have them turned on and just logging everything and runs through it yes we can go ahead go find you a log vendor

and they can talk about logging yes matt rob just is that bad on these log files at a minimum turn it on know what's going on i found so many problems that way in my opinion they should be built from scripts and those should match up to your documentation uh my doc if with me it's kind of like everybody hates mad rob because he builds his firewalls with scripts and which is not that bad but he also uses that as his documentation and yeah i got you know they'll hear you for a while but after a while they're like no no i don't think so and uh yeah just there needs to be a relationship

there really there does because later on after the hack you get to say hey we did this or we didn't do that or here's what's going on uh that documentation is super important need to include references to any agreement you have one customer i worked with they had literally signed pieces of paper saying we will move your data or we won't do this or that references in there because there's nothing quite as fun as figuring out why am i actually moving this data hmm could somebody be exfiltrating some data yeah gdpr absolutely absolutely and that's really important there made the matches up with your system diagrams again i'm that really awful guy who does pictures

because where i came from in school i do them in crayon and graphing paper and there are others like me log file conversations you have a vendor for that get them off the box as quick as you can you want the black box info as quick as you can get it if a boo boo happens se linux that's another thing i have two stages on this se linux that is a security thing that is a getting to a point where if you want to be get a certified box you will have it turned on and enabled watching knowing how to use those sc linux tools is super important and that's something that you run into

constantly biting you there's the sc linux project that's an awesome page there's a sh two stages here the short learning curve you can get it functional you can get not killing your stuff but you always need to be mindful it's like hey sc linux is out there running i wonder if that's why that app is acting weird i wonder if that's why that install won't complete be mindful of it know how to use the tools to look at your se linux stuff again that should be part of your [Music] whatever log file reviews you do however you do that seaweed under the surface there are some gotchas there are some things hurt yay 42 my favorite number

uh seaweed under the surface you can really muck up some things there are some security controls that you do that are if you do them wrong that is the great way to exit that box again you'll be talking to your backup guys or you'll just be doing a new install or clone however that goes getting into how you do your security mappings things of that nature that is very much a vendor conversation and i really would be talking to those guys on it that's why you pay for support oh vulnerability scanners scanners we're getting into religions here there are different scanners that do different things for you some log into your box look around some of you log

into your box and look around and report results back to them however that needs to work whatever mechanism is governing you pay attention to what they're telling you to go do openscap is a great candidate if you don't have anything else to use there's a oh there's a scaff workbench some really awesome tools in here you do your whatever what you call protection profiles how locked down does the box need to be does it needs to breathe do i need to track every process that gets started or not up to you however you need to do that whatever level you need to be stuck at customization files if we know we've got to scan every box

because i've got to have a report for every box but i know this class does databases this does my application x this doesn't do anything but support that's awesome you can do profiles that say okay we know we're going to see a lot more world writable files on sport boxes because they go deal with other interfaces so we know we need to be more mindful of that this database server over here though yeah not so much guys log in they do their patching they log back out i got a bunch of whatever database port you're using running back and forth so that's what we need to be mindful of and you don't want to spend all your time

looking for graphical login problems if you're totally a command line situation if there's no x on it great there's a bunch of checks i get to skip figure out what you're scanning however you're scanning it do you log onto the box and scan it does something else log on you get the picture openscap kind of getting to the parting's thought there what kind of reports do you need out i have seen and used both the xml and the html and just be sure to save your copies because yeah that's that's part of my that's part of my uh how i get my bonuses is hey look we started out really dirty you had a thousand of these things

we got halfway through and you know look you're down to 750 and you're moving the right direction but then you come back the next day there's unfortunately you get into politics in that sort of situation and it kind of bites and these are just things you need to be able to document your work is all i'm saying again know what's going on be able to explain it all right this is probably the the best slide in here because so we've talked a lot of technical stuff partitions scanners configurations all that other good stuff you need to get to a point where you take care of the nut behind the wheel taking care of you there are a lot

of areas where you need to be concerned about linux boxes are 24 by 7 by 365 and there can be a lot of them and they can have lots of problems and you can have lots of pressures because somebody's spending a lot of money to take care of this one box and it does this one function and you have one customer for whatever reason his profiles never work out right his patch profile is completely wrong every time he logs in he doesn't understand it his he doesn't like that flavor of ssh client whatever burnout is a very real thing and it can really be a problem i have to be honest with you i really haven't

done a case study on where these where folks burn out where they die off i really haven't don't know that but you need to know you again know what's going on and be able to explain it are you burned out have you worked an 80 hour a week have you have you been doing that week in week out are the vulnerabilities killing you how what's going on there because as much as a problem as it is for you to be gone for an afternoon or a day or a week it's much worse when you're gone all together as in you go find another job that really doesn't help those on the job site and it doesn't help you

eventually it's all about you're being able to take care of yourself and that way you can contribute to your team and to the field that you're working in so please take care of you take time if if you don't feel good if things aren't going right you know please have those relationships i mentioned going out to lunch with your network and your backup guys no really do go to lunch with these people and that way you can lean on each other it's this day and age it's really important to be able to do that especially in the age when we're all working you know far away from each other you need to be able to lean on each

other be able to call somebody up i'm super fortunate i can talk to my management it's like hey i'm not feeling right things just aren't going well and we can work through some things uh we're coming up and the man with the clock has just come back and he's doing he's so he's doing the route i'm going to guess that's a root dance he's doing that's awesome and see he's taking care of himself he's not he's avoiding burnout because it's the beginning of the day and he's got a long day in front of him i know all right so quickly education and certifications if your boss will pay for certifications or education you need to be please take

advantage of that those don't always come along and it will help you later on it yeah if you've got however that works for you uh in all honesty that is that should be part of your conversation when you're applying for a job when you're looking at going to work for a place what do you guys do on education certifications and uh for whom do you want to work you do this long enough and someone will come to you and say you know what you're doing an awesome job you're managing everything you're organizing your grade organizer we'd like to make you the lead or we'd like to make you the boss for some people that is a no way never

ain't going to do it conversation just consider one thing for whom do you want to work the guy who's going to get the job if you don't take it yeah my my memo has always been yes i will take the lead i will be the boss but i'm always going to have the captain's yacht available to me if i i want my logins on the box i will take patr patch night responsibilities that's you know one month one quarter whatever yeah just keep that in mind read only fridays you can thank a guy named peter coffey and he actually did come from pc weekly i think he's long since retired he's a columnist a bunch of years ago

and it's not necessarily friday but it is whatever last day you were working please don't go in and decide you're going to go patch up a box we're pleased to se you know hey we need to update usb drivers in the kernel oh it'll just be a quick reboot no problem at all that is awesome i will i'm going to steal that from you it is read only friday and it's really funny that is spread through my entire office and everybody except for one individual the guy who signs my paychecks decided that that was a probably an awesome idea and yeah if we go out it's because management needs us to go out on something

be mindful of this and also keep in mind who you take with you so if you go patch a kernel and you blow it you're going to be talking to backups chances are pretty good if you're reloading a kernel unless you're just that magic and you're working on a node cluster you're taking databases or application team with you and if you're in a big enough organization you're taking cm and qa and all those other guys with you um you're taking customer remote customer chats you're doing all those guys with you so when you blow it on a friday a lot of people are going with you and you're gonna be buying a lot of drinks so whatever the last working day

is be mindful of that and yeah taking care of yourself i also want to do a pitch out here being able to explain things kind of the theme here being explained things and know what's going on great organization called toastmasters that's helped me be able to explain technical things to non-technical folks or to bring it down a level so that other people understand it great organization and that's part of the reason i'm able to come here and do these talks quickly we're going to go off read because i'm over and you're not throwing things at me up uh scap scap generates uh remediation scripts if you so choose or you can go out and get your own

remediations going on ansible does remediations for you there are custom scripts out of github vendor related scripts and processes or you can go roll your own python anyone books paper or electronic there are certain books that i do in paper uh anything with python and hackers in it is usually actually a pretty good book the art of war what business are you in you have to have that if you're going to be in this business a bash reference electronics fine on that but the classic shell scripting book and i did not list the regex book those are two paper copies 101 excellent site sir those are good things that uh books that i have i carry them around

with me i've got a stack of books i take with me another a couple other good books deep work that's the cal newson and i may have misspelled that last name great read talking about how you organize organize your day to be more effective and how to really get good things done he's got some really good warnings in the book this day and age a lot of people don't have a long attention span i'm surprised i'm able to stay you can i'm sitting you're fidgeting around so uh being able to focus for deep focus on long range things it's very important and that work that book talks about it atomic habits making very minor incremental changes

again great for scripting great for learning reacts those other books are very good parting thoughts uh securing a box is not necessarily a hurry up last minute process if you gotta hurry up and remediate a box please don't please don't because you're yeah unless you're just that good and this subject can well you buy me books send me school and i set up slides like that uh this this can be a pretty deep subject hurry up is just well just hurry up as best you can

lack of planning on your your end then make an emergency on my end uh and last slide uh you saw several cat videos there's there's cat commands there's man commands there are no dog commands at one time i actually did have a dog command some got some script from my own uh that's who i am reach out to me i'll be on discord for a while uh yum yellow yeah that's kinda yeah yellow dog modifier yellow yeah anyways however that works out so uh thank you very much that's it for me that's 55 minutes worth of goodness i think they got this on tape and that's how you find me on twitter reach out to me

love to hear folks love to hear feedback on this it's stunk isn't it is a good introduction line but tell me why it stunk always looking to improve these things so with that i'm done i don't know i'm it