PG - The Struggles of Teaching Automation - Joe O'Connell

we want to wish a very good morning and welcome to b-sides to the proving ground this talk is the struggles of teaching automation by Joe O'Connell and just a few announcements before we begin we'd like to thank our sponsors especially our inner circle sponsors critical stack and volley mail and our stellar sponsors blackberry blackberry secure code warrior and paranoids it's their support along with our other sponsors donors and volunteers that make this event possible as I said the camera is live all mics are hot please mute your cellphone's as a courtesy to the speakers and those out on YouTube after the talk there'll be time for Q&A hopefully I'll wander around with the microphone raise your

hand I'll bring it over to you just to make sure that YouTube has the benefit of hearing your question as well as the response thanks guys without so hey what's up everybody just kind of do Mic Check just before you and everyone can hear me OK I don't to be talking too loud usually people yell at me for that so I'm Joe O'Connell if you have any questions about this at all at yoga fed on Twitter and Joe at yoga fed if you want the slides for later I'll read I'll rebreathe that up after the after everything so let's kind of get going

so a little bit about myself so I'm from st. Louis Missouri I currently work at Raley Health I do security analysts work and it's really fun I'm also a Bolton Wanderers fan and if you know what that means yeah we're very bad but I still I still support them and please come say hello at any point if you ever see me at any conference ever I love talking to people I love meeting new people it's really one of my favorite things to do and the security community community has really helped like teach me how awesome everyone is so please come say hello alright so a little bit like about what's what this kind of talk is more

about it's kind of about how I built this study group and how we kind of taught each other how to learn how to write Python and it's it was a really really fun project to do so one of the big things that we had to convince or kind of talk about was buy-in for management so one of the problems that we kind of started to notice in our security operations was that we do both access requests so like identity and access management as well as we do security analyst work so anytime that there's some type of alert that goes off so our company started growing a double the rate that we were expecting so when

I started about a year ago we were about 800 people and now we're about 1,600 and that really put a burden on us in terms of like focusing on more on identity and access management over the security analyst work which both are very important so we needed a solution to figure this out and one of the solutions was to teach as many people as possible how to write Python to automate a lot of the access requests that we're doing so at the beginning of this the first thing that we kind of decided was the biggest thing to get passed was to get time for everyone to learn I'm in order to get time you have to talk to management and

figure out how to get time in order to actually learn on the job which is very hard because a lot of people are like go learn on your own and that's very very difficult and it's just you know a lot of people are burned out in this industry so it's something that we were trying not to do and the other thing that we kind of wanted to do was a repeatable process so that we can redo this every time that someone's on board it and go through this again which that's very difficult and we're still kind of trying to figure that out just because of all of the things that we learned but it was

it's been a very fun project like I said so some of the things that we were kind of trying to do was align everything up together so that everyone was on the same page and everyone was kind of going for the exact same goal so we said a yearly goal for our entire camp for our entire team to all of us learn Python so there's about 12 of us on the team that wanted to learn Python of those 12 three of them had some type of training in some other language or at least had some understanding of code the other nine did not so I'm included in the other nine so it was a lot to go go

learn but we try to do it together and it was a lot of just pushing each other as we're kind of going through this process so with aligning goals as well we did individual cordial core illegals of like we want to get this far as well as kind of converting those quarterly goals out to yearly goals which we're still kind of going on today and it's been kind of really fun too because we didn't just say like only people from my security team can join this actually we allowed anyone to join so we had some people from IT join us we had some people from like coaching which was really really fun and actually the one

of the coaches that here today so she someone fur coat will be presented so there's a lot of fun so the goals of our program kind of like I said was to teach everyone Python and then converting old batch scripts that we had that were written by someone who had left the team to Python and then to create a sustainable learning program so the bash scripts that we had were really good because they succeeded but there was no documentation on it we had no idea what was actually happening and we kind of had no idea to edit them if anything changed in our environment which our environment changes all of the time so we we were focused on that at the

beginning and it kind of helped create a convertible way for us to take something that we learned in learn Python the hard way if the our program that we're building and converted into something that we are doing every day and actually making it feel like we were successful and doing some and not just like wasting anyone's time and this right here was actually like an interesting problem to face just because at the beginning we're like oh sure like we can just easily fix all of those batch scripts like that doesn't look difficult and we're six months and I guess and we've converted about 3/4 of them but I feel like they kind of need to get farther along so hopefully we can

finish that by the end of the year so how the program worked was a weekly study group during the day about an hour long and basically what we tried to do at the beginning was just go through all of learn Python the hard way and kind of go through that together and continuously push forward we were kind of strolling with motivation with that and then also just making sure that everyone knew what was going on and how to do it so we kind of wanted to do more like problem problem solving and figure it out that way so we started focusing more on lika we code is a website that gives you problems that you can test out

or try and kind of solve on the website and then it gives you it gives you the solution at the end or you can look at the solution if you get stuck we obviously were trying our hardest to not look at the solution but we we had each week individually we would have a leak code example that we would send out to everyone that they would try to solve and then come back and and show us up show off what they did so that was kind of one of the things that we did the other things that we kind of focused on with with both learn Python the hard way as well as just overall this the

structure was kind of something called show-and-tell she went oh it was really really fun and I'll probably get into that a little bit more at the end just because that's kind of more of like a success that really happened and then another thing about applying knowledge in real life it was actually very difficult for us to find simple enough problems for us to continuously grow as like two contenders to grow as like software is engine you know software engineers I guess that's not really were going for it but continuous grow our skill set of writing Python and code so some of the short terms stuff that we kind of encountered so the way that I'm

describing short term is something that we deal with on a day a weekly basis so something that like it was something that was very difficult but it was an ongoing thing that was just very difficult and trying to do first thing that was very hard was keeping people to attend so at the beginning kind of talking about aligning goals a little bit are one of the the goals for a lot of people was just to attend this study group and I continuously grow as I can tell it's grow throughout the year so we kind of kept track of who is attending and then like how often they're attending and then kind of showing like where their

progress is and that was working for a while but eventually it kind of petered out when we started getting more and more busy finding time during the day was also very very difficult so we were going through this at the beginning we were we just had an hour a day or sorry an hour a week to teach automation and then kind of go from there but then what we figured out was that that was not enough time to learn at all so what I did next was grabbed two more hours during the week so that we at least had three hours during the week to actually sit down and try to write code which was

really great and it hoped it helped a fair amount more as in as three times more time to do anything you'll learn much faster so it was definitely a lot of it was definitely worth it participation was actually very difficult one of the things that I really focused on in this study group was to have everyone involved and everyone kind of talking so that with show-and-tell kind of the thing that we I've mostly focused on was whatever you do just write something to throughout that week bring it to show-and-tell and do it and just kind of show us and talk through what's going on in this in the script and act like no one has any idea

what anything is that actually helps a lot so that helps in terms of actually understanding what's being written as well as making sure that people didn't just go to a stackoverflow copy paste and print something because a lot of the time people will do that and have no idea what the sec overflow does the code from the stack overflow does and it makes it really difficult to learn actually how to write and program but it was really really fun to see everyone participate and like grow into participating because everyone at the beginning was really shy and very quiet eventually more and more people started participating and it made it a lot more fun and then partner

coding was actually one of the more heart one of the harder parts so our office is currently there's five we're across the nation so we have DC Minneapolis Chicago Las Vegas San Francisco and LA and we were really struggling with trying to teach everyone across across the states I guess and one of the things that we noticed was that Visual Studio code has a application built in where you can do partner coding basically through it it's Visual Studio live and that helped a lot but actually what helped even more was doing the Visual Studio code live as well as with a Google hangout so that you could do both the live coding and kind of talking to each other at the

same time so that was a super helpful and would definitely recommend doing something like that so more like solving attendance like I said one of the things that we kind of focused on and solving the attendance problem was like I was trying we were trying our hardest to make sure that it was worth going to which a lot of the times that was that was very difficult but one of the things that we wanted to do was like just kind of do a weekly like event and kind of try to push and make it more fun and make it more of like a less structured thing so I actually never we never had like managers or anything go just so it

was more like a calm relaxing like we're all here together learning situation and then the other things like I kind of had mentioned before was like creating an environment of participation even if like sometimes that meant talking to someone beforehand and just being like hey I'm going to call on you make sure you kind of have something to do just so that everyone kind of understood that like everyone should try to be beer participating and trying to kind of push it push it through so some of the long-term problems that we we kind of faced and like I said the longer problems were something that we're more just difficult things to deal with throughout the entire process as well

was motivation so at the beginning we try to tie motivation to our quarterly and yearly goals which tied to bonuses eventually which was very at the beginning like I said also kind of helped and worked but right now kind of midway through the year it's kind of stopped slowed down a little bit because I think everyone's kind of forgot about the yearly goals so we're kind of trying to refigure out the motivation right now and that's been very very difficult and something that I'm still trying to figure out itself if anyone knows how to solve motivation please let me know because I'd love to know and then like it kind of mention before like finding problems to solve

which I thought was I thought that was gonna be the easiest thing to do just because there are so many different problems so you can look any back oh you can you can do that but then like breaking that down and kind of understanding how to break those problems down as like writing programming is very difficult which I had no idea that going into this that looking at a problem and being like oh this needs to be broken down all the way down to like a little baby can understand basically and that was just very very difficult in something that yet again we're still kind of learning and I think we've gotten a lot better at

but it's something that we're kind of working on as well and then finally like the different scale levels were very difficult so like I said we kind of had three or so people who were very very good and had like learn how to write some type of program language so we leaned on them a lot to kind of help us throughout this process but we did have some people who had never touched any programming language ever and some people who had never really been an IT or had messed around with any type of caen deep more like deeper computer like learning I guess in terms of like outside of the PowerPoint and generic things so the skill levels were very

fast and that's I think what made it a lot of fun as well just because everyone's kind of talking and figuring out different problems that they're facing and then working together to figure it out so so some of the other things that we kind of did too like solve those problems we're tying rotations goals like I mentioned and then using the code was actually like I said before it's really really helpful on lis code has a easy medium and hard mode and what we tried to do is we tried to find four or five that people would start out on so something like but something like really easy something just like a loop or something along

those lines or just kind of showing like oh this is kind of how you do the first original program that you're writing so like a little bit a little bit more than hello world but something like less than a massive program now and that's kind of what Lee code did and then there's like the medium and the hard sections which also were very difficult as well but the medium section I think was kind of where you finally understood how to go from just writing small scripts to something a little bit bigger and having an understanding of how data flows a little bit and then lastly being encouraging to beginners was actually a very very important thing because writing code is

very intimidating especially if you have no idea where to start or yeah and not only that but so many people have so many different answers on where to start and where to begin even from the language that you start with to what books to use to everything so what we were just trying to do and try to like push across was to understand that everyone learns a lot differently everyone in this study group actually learned a different way so what worked best for me was actually a udemy course at the end that's kind of the thing that worked for me for some people the learn Python the hard way we're it's a lot better and that's kind of

something that kind of solved that problem some people did automate the boring things and I think the biggest thing that I that I noticed was that at the beginning I thought oh we're gonna have this very defined study group we're going to know these all these things we're gonna do this this and this and then I realized that everyone is different and the only way to solve that difference problem is to let them figure out how to solve it and that also kind of leads back into how to like motivate these people how to motivate anyone and I think a lot of the learning how to write code is a you just have to sit

down and do it and you have to motivate yourself regardless of what that is so that was a very very fun and interesting problem to solve I guess it's not solved but interesting thing to like deal with and it's something that we pushed through so a little bit of like the Show and Tell that we did I'll show a little bit of the code that we were some of the code that we wrote but so I'm gonna show and tell that we like focus on did was the was more on like their specific job set so for instance NIT one of the problems that one of the people were facing was that people would come

up to them and just be like hey I need you to do this thing and he would be like oh go make it to your ticket but the person would never make a Jiro ticket so if he had to figure out a way to solve like that small problem so what he did was he decided to write a small little python script that basically came up and popped up like a little like text box and he wrote down like two two words that said like make a ticket for this person about this thing or something and basically just kind of anytime that happened he did it made a note on to his desktop and that's kind of how he solved

the the whole Adira issue so we did that show-and-tell and that was really really awesome thing to like show one of the things that I did for show-and-tell was an EI so I'm a big FIFA soccer fan so I really like FIFA the video game and actually FIFA has an API that's kind of hidden I guess I do like search and figure out where it is it's not on github or anything so I play this game called pro clubs and they have data online so what I started to do was grab that data online kind of convert it into like a dictionary so I could use it and then display it to my friends who were

playing really I just kind of wanted to show that I scored more goals than all of them but it was enough motivation for me to like you know do it so I had a lot of fun and so kind of going through some of the scripts that we build so the one on the left here is this script for the for my front from my friend Doug who is doing the JIRA ticket issue and then the one on the right here is the it's mine I do want to kind of point out that it says FIFA 17 but I'm not playing fifa 17 just to clarify they just haven't updated their API since FIFA 17 but it's the same data or

to the newest data so if you're an EA that'd be great if I could get your API let me know and then some of the other code that was written actually by front by Mia who's actually here at here today so she wrote a little port scanner and she is actually currently looking for her first security job she was one of the coaches that was here so if you have a entry level security job you should go talk to her and she really worked hard and learned a lot so one of the so please go say hi to her that'd be great so what should you take away from this and one of the things I kind of want to

take like let you guys know is I think that it's easy to start a program if you kind of just say like I don't know what I'm doing and that's fine but let's try to learn this thing together because that's kind of what I did and it seemed to work somewhat somewhat well and a couple other things I kind of want you to learn is like how would you get buy-in I think the best way to get buy-in from any business section was just to talk about return on investment right when when you're sitting down you're talking to someone who maybe isn't thinking about security all the time like you have to talk to them more

about business and talk to them about how did like what return on investment you're getting from using this time which was a very easy thing when you're saying well we're hiring 800 people in a year wouldn't you like that to be automated because then we can actually focus on more security things and we can get more money from not just giving you access to things but focusing on the security aspects which is what you more care about I think that was kind of like the big sell and I kind of what I would focus on when you're trying to start some type of program and then aligning goals Eleni goals was very important I know it seems like the aligning goal was

kind of stopped midway through but I think that we're gonna try to do another push and just kind of remind everyone with like what's going on and it also hopes that everyone's on the same page and trying to go for the same goal anytime that you have two goals that are trying to be accomplished and they're maybe not competing but they're the same goal they usually fail because there's not enough resources going to one of those goals to make it make it succeed so focus thing kind of like on aligning goals and how to get buy-in from from upper management it's probably the best way to go so I have about five minutes left or so so I'm gonna take questions

and say hey Joe did you consider any other tools like papad around similar chef as opposed to Python when you were making your choice were you considering any automation tools like puppet or ansible when you started in this journey no we did not so the big one that we were focused on was was Python just because one a lot of our company already used Python so it was something to kind of keep us all like maintained together and make sure that we're all on the same page as well and then also a lot of the the systems that we were giving access to where you needed to use either like a web GUI or something along those lines

before hand so we hadn't looked into ansible or Scheffer puffit so almost as important as the language itself is how did you teach debugging hmm I didn't so I guess I'm going to say that I did not teach debugging cuz I have no idea hey one of the things that we so yeah I don't have an answer for that and I'm sorry yes nest project learn how to do that thank you were you able to find any mentors from like the software engineering side of the house to kind of help you get over some of those early humps and sort of train and coach the team yeah I did so actually luckily like I said the three

people who kind of had some background and understanding they really helped to mentor and helped teach at the beginning and then they what was actually really awesome as I kind of like took a step back and only came in when they felt like they were needed which is really great because it kind of kept this nice like we're all learning and like no one knows what they're doing until we're like we're really stuck and we have no idea oh yeah I have two questions one for Python the hard way what was your what did your like syllabus look like was it like a week we would do two chapters and then second question is did

you guys consider any other resources like codecademy or any of that other stuff yeah so at the beginning we kind of kept it like go at your own pace type of situation which was a little difficult because at the beginning of learned Python the hard way anyone use it you can run through like the first 15 and a a week because they're all very like learning how to print and you're just kind of like going through the motions which is really good fundamentally and it really needs to be done but it's pretty quick to be done and then in terms of like looking at other things we didn't look at other things until we started to notice that not everyone when

was like loving learned Python the hard way and we try to solve that problem I know a couple people did use code Academy and are using it now to like learn so I think all resources should be looked at expecially for individual people just because whatever that person likes to learn I think is the best way what steps we take and make sure that the code that you were putting out there was kind of adhering to security best practices and make sure that you know those common pitfalls like committing code with API keys and stuff wasn't happening yeah so actually one of the one of the other teams is we actually worked with a little bit of our

application security team to make sure that our code was actually fine that was actually a very difficult thing to learn at the beginning because I had no idea how to actually do that so learning how to actually have an API key like hidden so that you could actually like reach out to who you need to was it's definitely a difficult process but really fun to learn so but fancier question we work with Hopkinton security alright guys that's all the time that we have for Joe here I'm sure you'd be happy to take questions out in the hallway if you see him around but super one more round of applause [Applause]