BSidesPhilly cg07 Smarter ways to gain skills or as the DoD puts it Dr P Shane Gallagher Institute f

I could have tuna everybody thanks for bearing with us through the fire drill our well it wasn't really a fire drill the good thing was it wasn't us somebody burnt some corn or popcorn be invited back next year hopefully so the next gentleman it's about to speak he's a co-founder of 0.3 they're giving or they're actually hosting a CTF here today one of the sponsors and one of the co-founders so please help me welcome Eric born bush

hey thanks everyone so feel free to heckle and if this doesn't work pull a fire alarm will be will be totally fine so this is actually an encore presentation that we did for the National Institute for cybersecurity education the National Institute for cybersecurity education runs an annual conference their conference is by data scientists with PhDs and data modeling and academic theory for data scientists with PhDs in education theory and and learning models I am NOT a data scientist I don't have a PhD in either education theory or learning models I actually am NOT a deep science or have a PhD anything so I'm just a guy I ran color commentary for the first run of

this talk to dr. Gallagher's play-by-play and so as you can see dr. Gallagher isn't here we wish Shane a speedy recovery he had a medical emergency this morning so hopefully I can fill his shoes in today and be both colorful and informative so the talk is actually called after read this one leveraging the right learning model and embedded analytics for outstanding results in cyber operator training that's a terse a little academia focused and also indicative of the next 67 slides that follow this one so stay with me in the spirit of besides we'll be winging it we'll be doing it live so for now I'd like to call the talk smarter ways to get skills or as

the US Department of Defense calls it leveraging models and analytics and right Ladas you know stuff that comes tough right so so a little bit of background right to the Internet's a thing right it's not a fad it's not going away we probably need to pay attention to it right this is DoD policy right they've said that right from a DoD perspective Internet's not going away so we should pay attention to it so my company was called in to meet with the director of force readiness and training for the u.s. department fence and he's like guys we are not prepared to fight a cyber war [ __ ] all right he got my attention or hi like what does

that mean right so cyber warfare is many in the audience probably know is a series of vulnerability research exploit development effects usually you don't want to get caught while you're in the process of doing all those things most of the time you can't buy your targets your target hardware software like at the local Best Buy so a horde right you you often can't copy/paste your way to success right not all of our targets are running Windows XP or some misconfigured version of PHP 5 you know you can't always just throw stuff in a Metasploit and and have success against a nation-state so we need to train professionals and so the director of course read instances

hey like 0.3 you guys in [ __ ] yeah like we're in right like how do we do this so first thing we do is say hey what do you what are you doing now and the way that they're training now in the military is using something called the Victorian model which any American citizens audience is familiar with the Victorian model right rows and rows of desks and chairs and students can lectures boring PowerPoint multiple-choice tests everybody memorizes the definition of the think ramps for the test forgets the next day like nobody wins it's an awful way of learning particularly this craft and so they have a schoolhouse that they call Jake hack I wish Shane was here cuz I don't know

what that doesn't know that stands for I know a good awesome you went through it did you go through Jake ACK okay yeah is it awesome so yes yes we got winners so anyway for those who have gone to the Joey something something something dude my condolences right that sounds awful right that's not how we learn so the Pentagon came up with something called the cyber operations Academy course so my understanding is that they wanted to call it the cyber operations course and then they just didn't like the acronym so we got the cyber operations Academy course so we throw away the canned lectures we give everyone nothing but keyboard experience the entire time the

very first challenge when you walk in our door is use after free bug that relies on lock chain to gain execution right that's the very first thing and then you have to the next six months to learn how to do it over and over again right we focus on the hard stuff like we're vulnerable research zero-day exploit development reverse engineering malware analysis in memory forensics we also emphasize soft skills so we divide up the students into teams everything is team-based because that's how you work that's how you fight the students are given hard problems long times to solve them we don't go through the whole like alright it's lunchtime like if you can get the answer it was 17 we come back

let's talk about cryptography right that doesn't work so we give students weeks and weeks to solve some of these challenges and as a result they have to learn things like time and resource allocation for some of the projects we actually give the students a dollar budget they have to go and buy things they got to determine like what's the best use of those funds division of labor how to conduct briefings that's all emphasized in the in the co act and so the course is again it's all hands-on so we meet at 8 a.m. we meet at 12:30 p.m. for a big you know big family hug right like who is stuck on a thing and literally has no idea how to

research the way out of it all right you're stuck like let's all nudge that team along class let's kind of debate this out but for the most part that's it then you know go back in your homework right and so the intent is a lot of these challenges are hard but the may think it makes them hard is time right we know what we ultimately need to do we just don't necessarily know how to get all the steps through so great give you time and resources and a team to go solve those problems so it turns out that this actually isn't new it's it's old it's like Dark Ages and caveman old so probably ever on the

audience's aware like ours is a craft and so the best way to learn is by doing right it doesn't make sense to read a book about it it doesn't make sense to watch a video or even to attend some of the talks and hear how somebody else did it the best way to master a skill like this is to go and do it so it's craft that's all hands-on and we we provide mentorship using something that is ages old again and called the journeyman apprenticeship model and the rest of the time again just everyone is learning and so I've skipped about 15 slides here again data scientists from the Institute for Defense analyses from the advanced distributed learning

initiative and some experts from Booz Allen Hamilton came and evaluated everything they evaluated the students they evaluated the mentors they valued the curriculum the facilities like everything and so I'm not a data scientist so we'll casually skip a dozen or so more slides but know that there's all sorts of stuff about second order derivatives and taxonomy buried within here if you are interested I can give you a copy of the slides later but this is ultimately how cognitive apprenticeships are modeled the intent is at begins again mixed sense right at the very beginning we're gonna hold some hands gather around class let me show you how I do a thing all right now repeat what I just did now I'm gonna

step back now you guys do it now I'm gonna step back to the doorway now I'm gonna go down the hallway and you guys just keep doing it for six months that's that's the way that the co act works and so we've now gone through three pilots or the DoD has gone through three pilots of coax and each of them has had a different objective so the first objective is like does this work right intuitively it makes sense we learn by doing I don't you know I can't take a multiple-choice test to show that I understand how to program I have to program but we don't date it actually back that up so we need to have a pilot

and collect the data and show that it works and and how how well does that work the second pilot that we did was to create parody so again the existing schoolhouse JK it stands for something it's awful it's it's gross but that is the program of record that is how the military is conducting operations training now and so we need to make sure that the co act is in line with J CAC so if ever there was a replacement no one you know knowledge is is is lost and then arguably the most powerful was the most recent pilot we just wrapped up does this apply to civilians right or is this inherently a military thing and that's really

powerful right so does this apply to those who can't afford college right those that don't have a litany of certificates that's really important right you can help break stereotypes you can knock up knock down gates that are set up by HR departments that are wiping out whole populations of qualified candidates from consideration in the work force the second reason is a little more inherently military as you all know if there is a cyber conflict in the future it will not be in the public space it will be in the private domain right like it's one thing to go like capture that hill and plant a flag you know here's my territory but you know the target space is banks right

utilities places that civilians have inherent ownership of at least in the United States and many other governments across across the oil and so we also need to make sure that we're not only training the DoD gene pool that we're also making sure that US civilians have a knowledge base of how attackers are breaking into your network getting past all of your hardened antivirus and firewalls and intrusion detection staying there without you having any clue and then doing something that really gives you a bad day that awareness of that is inherently important and that knowledge should be shared is the vision of our customer and so real quick on data there were two research questions on the first pilot so

again I have to read this one because this is this is kind of important right so question one is as words of the course right did-did-did learning happen it's important to capture and the second thing is how can we benchmark this style of learning against other forms of training so the government the military in particular is interested in this right so doing the thing by awareness and concepts doing the thing with code and then knowing who is your enemy so that roughly is six months of curriculum so the data scientists came in and they created these things called mind maps this is an example of a mind map so it turns out that like to write your own

exploit you may need to know a thing or two about pointer arithmetic which also means you may need to know a little bit about pointers and arithmetic so this is a mind map of all the things that have to happen in terms of cognitive awareness before you can actually complete the challenge there are pages and pages of these things for all of the exercises we run through from key loggers that don't get caught by antivirus in memory forensics detecting things that are normally not detected and all this skill sets that go into it here's another example I just went for the key logger [Music] and so who came to this couch is I think one of the more interesting things so

that's a lot of words summarize a few people with formal experience right a few people with college degrees or college experience one of the more shining stars of the course was a with the air force called cable dog so a cable dog is the individual that goes to new facilities and runs low voltage Ethernet cable through drywall so that is his job never touched a computer like literally on the very first day of class we had to teach his kid right click from left click he is now reverse-engineer malware by the way he's getting out of the services pretty soon he's looking for the right employer so if anyone is hiring I'm more than happy to matchmake but you don't

necessarily need experience to go into a curriculum like this and walk out with skills that matter in terms of benchmarking we have the students run through a number of independent assessments in addition to the ones that we as instructors created for the students so the students went to the Department of Energy's annual cyber fire exercise and placed extremely well against professionals we ran them through the O SCP curriculum ever knows osep and when shaking out ok so who is 70 percent pass rate 7-0 on the first year again most of the students came in with no they weren't they weren't operators they weren't red teamers they weren't analysts we had linguists we had all source analysts we had a procurement

specialists like his job was to buy things and and now he's passed he passes osep we ran a heads up DEFCON style capture-the-flag event called the a3 cyber audience all our students against six months of instruction going against fully operational teams professional military outfits that have gone through all the training have their shirts to operate and be effective NSA came first place but our team came in second Coast Guard came in third and our second student team came in fourth I might be mixed up there were six teams we captured six fourth and second fourth and six not bad for six months of training against fully operational teams that have been complain and and and working

together for years we also had an Tod brought in some evaluation exercise sponsored by a third party and our students benchmarked well against just a global pool so all of that lends itself to this so I don't really know what's going on at a slide but I know that there's a lot more green than Purple's so something obviously happened something very important happened and that's the premium Post gains that you'll see throughout this style of learning all what's effective and consistent so the 2016 pilot so second iteration goals did we get lucky do we have just this magical student and this magical set of students that really had experience and just lied on the application test or can we do it again

right and again can we make this course as similar to Jacob as possible so we were given more students roughly speaking they were similar in nature to the first batch of students we evolved our curriculum to match the 180 learning objectives of Jake ak so let me explain that so Jake ak is a hundred eighty days and so his co-ack 180 things you got to learn the way that Jake ak teaches is very very linear right today is cryptography today is Windows today is networking like all right the end of the day you guys are now crypto experts windows experts throwing experts think very very structured co-ack because it's challenge based we get to weave in all

of those themes into an underlying set of challenges and so again going back to that key logger exercise that we have right students have to develop a key logger that does not get picked up by an averse it's assumed in that process you have to learn a little bit of crypto and a little bit of Windows and a little bit of networking so rather than having like today's crypto day windows day networked day we it's all a story that we can tell over six months so we matched all 100 almost all 180 learning objectives apparently Jake heck does some classified stuff that we know our class is entirely unclassified so we there were some things that we couldn't get

we're not pretty darn close the assessment vehicles were the same so they went to do E's cyber fire size they had the a3 they had a bunch of different industry events that they went to I think they ended up going to be sides DC or something and passing through a lot of like Raytheon has their game of ponies challenge I mean all the vendors have their own our students were washing them osep this year we had an 80 percent pass rate again similar sets of students linguists procurement specialists all sorts not cyber people so this year on the a3 we were we had 8 teams and the coax students actually captured first and second place so that means that the best

teams that the military put up to us there were two Marines CPT's that cyber protection teams the Coast Guard had their CPT there was actually an industry team a a private company of professional pen testers the coax students with six months of experience came in first and second place on that event and the other teams filter down towards the bottom part another bar graph looks like some awesomeness is happening any questions on this cool data right data it's exciting so this is the one that this is the part where I get excited on 2017 so how many times have we heard like oh you want to do security well start as helpdesk all right then you're a

technician then you're sysadmin then you're programmer and then you're ready to be security like who's the security special to all these crazy things right or how many how many jobs do you see on on wherever you you know look online for job postings and it's like I need a PhD in this and these twelve certifications and you have to speak French Italian and Latin like and none of those are even applicable to the job description because there's no other way of benchmarking and and it's it's frankly it's it's it's it's it's not good for a big percentage of our nation that does not have college degrees all right so we looked at our pilot demographics I'm kind of putting a

couple horse when we look back at the individuals that went through the course in years one in years two these were a lot of junior enlisted men and women no or little college experience we now have two years of data on these individuals they've not gone back into the workforce and they are winning awards they are getting citations from their employers doing some really cool work and then again they're going placing at sans net Wars again raytheon's competition some others so the students are the alumni now are just rocking it in the real world as well they're engaged they're staying active in the community really neat stuff right very inspiring stuff and so we now can say hey maybe

the style of training can be used to address an underserved population so DoD forged a partnership with the city of Chicago and this summer at the community the city's Community College the City College of Chicago in Chicago this program was brought to approximately 25 students roughly speaking one-third were military and two-thirds were from Chicago on many from the south side you can see some of the examples of the students that we were given we had a peep hot driver a bank teller one of the students literally was homeless and had a lot going on in his personal life right and like that's a lot to deal with by itself now you're tacking on 40 hours a week of

like reverse engineering and their analysis I think the guy had a lot of bad days but he went through the course and graduated in terms of results so the course has ended we're still looking at data analytics again that's the chains imaginary shame like that change job but anecdotally it does look like people are happy so the measures of success are now real-world we're not necessarily caring about who placed one at cyber fire or whatnot but to the civilians the number one interest for going through a course like this was jobs it looks like we have somewhere between 85 and 100% job placement many of those jobs paying pretty decent wages some of the candidates some of the

alumni have declined offers they're waiting for the right employer so again if anyone's hiring I am more than happy to matchmake you with some pretty solid talents today on the military side the individuals already employed so getting a job wasn't really a measure of success for them the measure of success was that OS CP cert this opens up very specific career enhancements I'm told I'm not in the military but though if he was like the reason why many of those individuals went to went to cost so we just found out yesterday that of the we had eight military students that sat for the exam in all eight pass the exam on their first or second attempts so that makes

them eligible for promotion eligible for certain job roles that they weren't eligible for before having that cert very cool so the data science aspect of this presentation is on metrics particularly with X API so X API is an open spec the DoD apparently created it many many years ago and opened open sourced it we we use it in our class I'll explain in a second but what's nice about X API is it enables us to hook at different positions of the challenges and that lets us measure how far people are advancing how quickly they're advancing we can in some cases determine levels of frustration right like it's been a while since you've gotten to the

next step what you continue to be logging in you're engaged which is good but you're not progressing which is I mean you just need a little bit of love right and that's the whole point of the course so our platform that we use is called escalate we are talking about like shameless plug right you can absolutely try it for free at provost got me right profess got me a room go to profess about me that's the learning platform that we use in the course and so with with escalate being x api enabled again it it allows not only access to the challenges but also the instructors and all of those analytic capabilities on the backend so we can

see things like who's logging in and when when folks are stuck when and where individuals are going when they want help and then we can determine are those resources useful all right if everyone goes to a particular web site and then comes back with the answer that's a that's a good website we should keep track of that right or archive it or something people are going to watch this video and it doesn't help that's not a good resource we can also measure things like learning pathways this is the next generation of academic modeling I can't speak authoritative Lee - and I wish Shane was here but the the escalate model is very Netflix oriented so you

can login and the instructor can say hey class like we're doing this then we're doing this then we're doing this but it's it's a choose-your-own-adventure and individuals can kind of do whatever they want on anytime nights and weekends or within the classroom and so by measuring who is going to what challenge in what order as things are completed and attempt or attempted and then completed that's a really neat way of finding out what that person's interests are where their aptitudes the strengths and weaknesses are what categories appear more interesting and relevant to the general population and so that information is is interesting to academia and I guess to us as well so in terms of next steps really just more of

this like this works so we've completed the pilot of the instruction phase of the pilot and now we're going back through all the data to try and mine out and tease out all the interesting things I'm under saying is next year there will be a formal comparison of the new model again hands-on you know no powerpoints against the existing model a lots of powerpoints lots of multiple-choice tests and the way we're doing it is through another course so the fourth course will be in the spring of 2018 and that will be in in Maryland so that is my talk if there are any questions I'd love to open up the floor anything at all and again feel

free to pull the fire alarm or throw something at me we're pretty flexible up here yes sir

I might need to hear that again and I really wish Shane was here I was gonna say what kind of learning models did you use during your three or four different tests that you tried and which were the most successful one more time which learning models do we use like Bloom's taxonomy this kind of learning strategies and learning models so Bloom's taxonomy is my understanding of Bloom's taxonomy and the new one not the old one is what are the learning objectives and to what degree right so can students identify something or can they develop a solution or can they raise that as it correct so we do a lot of that we do a lot of the higher

cognitive pieces so by the end of the course it's not enough to say hey I know what exploitation is the students are offering exploits and because of all the challenges were written by us the instructors inherently they're zero-day right so there were they weren't able to just like find a resource and copy paste it into Metasploit and go for go for victory they had to go through things I didn't talk a lot about the defensive side but that's that's really big for DoD because a bigger function like offensive is a little more sexy but the defensive sides were all the the manpower personnel is on a defensive side DoD is very aware that like checklists are bad right it's it's a

really bad day if like you do this then you do this alright everything is green we're safe we're secure I like duty recognizes that that doesn't work and so we through a lot of stuff with the students and made them really um recognize like oh crap like everything is implanted right there hardware's implanted or software's implanted like what do we do like how do you trust a platform that is inherently untrustable because it was built in a foreign country all right but but when you mix enough of that together and have good monitoring in place and good processes in place that the students develop that seems to be more effective than reading about the definition of encryption and

then taking a test that says I need to know is encryption like really important kind of important or none of the above right like hands on it's just the way to go that is your question cool yes sir

yes sue thank you that's that's all I as a salesman like this that's a great question so two ways you can get three ways you can get involved so if you are a business you can sponsor a student to go through the curriculum and get involved that way if you're an individual that wants to go through the curriculum there too a so again we've got co-ack I'm sure there's a silo you can go through the course the course is tough right it is six months it's 40 hours a week and my understanding is that there's no stipend so you got to figure out how can you commit six months of your life without income right

there's a tuition associated with it I'm not quite sure what that is just yet but it'll be roughly about 15 $20,000 per students before scholarships are applied it will be taught at a community court a community college course so it will be what's called a cap Rove's Maryland Higher Education Commission so there should be some college credit equivalencies so that's one way it I'll say the shortest path is if you want to take the course see me after the class or email me my email is up on the screen that's Evan at point three net the easier way is escalates so we recognize that not everyone can put their life on hold for six months not every manager is

gonna let you go for six months and not everybody just wants to be constantly be lodged and immersed in this for six months so escalate for us is a way it's much cheaper we have two packages one is roughly $1,500 for three months the other is $3,000 roughly for a year that subscription gains you access not only to the challenges but to the instructors so if you get stuck you can like digitally raise your hands and week either open up an IRC session or a webinar we have office hours and everything and that's again that's Netflix model so login username password and you have free range to the to all the challenges in the system that's

called escalate and our booth is like right on the side of the wall and we've got tons of like literature for that so you mentioned non-security people getting involved in cyber whether it's federal or private one barrier I've found is if you're in private and security and you don't have a degree and you want to go federal or something any thoughts on that yeah it sucks it does right and so like we recognized right I think this is a friendly audience right we get it like if we've got skills that we can demonstrate like there's value there and a small like my company employs folks that don't have college degrees because they've gone through escalate challenges and they've proven

to me that they can do the job that I'm hiring them to do larger employers are working towards that space so I IBM now has something called gray collar so you've got your blue collar jobs your white collar jobs I'm sure IBM is trademarked the hell out of this they call it grey collar and and gray collar is again designed to be a diversity initiative right so I don't need the kid from Ivy League school with the certs right like that's one person but maybe I can get the individual that can get through some hands-on and that's the new gray collar economy so if IBM is talking about it that's going to make waves down

the road I know that there are three Congress people on now that recently issued a letter to OPM and said hey OPM this job description says you need this degree why like prove it to me because unless there's something degree bearing in this job open it up to all the community college kids right or anyone that has skills so people are starting to think this why but I mean your specific question of how does this apply to federal government like that's that's that's a slow slow race you'll get an IBM before you get into like the federal government cool any other questions

via hi my name's mark Rivas I think there this might be the perfect time to get this going in the city they've just started a new computer science initiative in the public schools and the people who are backing it I think can really make it happen so I'll talk to you afterwards yeah I love the CES for Philly just just launched on Wednesday yeah that's great again we've now gone from like professional military down to the community college level we have a pilot coming up hopefully in the near future at the high school level you know of course the intent is to push down and like stem is kind of a thing to write down that third grade and third to 12

level so we don't have data to back that up that this is effective to a third grader we don't curriculum that's really tailored towards a third grader but we do have data on the college level so it has been shown to be decently effective any other questions yes sir

so you mention the oven Oh s PF tis or other certifications they help you to pass it to G X or Cisco I sorry it's really hard to hear sorry you mentioned the OSPF there are other certifications that your training helps you to pass like the GX and the Cisco certs so I've very limited data set on that so we only included personally speaking I do not like the OSC P what I find wrong with the OSC so it's it is the it is the best that is out there right now right I get that but there are two issues I have loves to be number one is the targets are not modern right everything is

Windows 95 or there's not the machine is so crippled down that a legitimate user couldn't use that target and the other thing that I don't like about the the OSTP curriculum is like the whole concept of try harder is just really nasty and egocentric and so like you haven't figured out this thing that we've like purposely made asinine ly hard gotcha try harder right as opposed to like hey like you're interested in this let's nurture that let's help you up so our CP was included because the DoD specifically said hey I need I need a benchmark and it is the benchmark that's out there right now because it's hands-on it's not knowledge-based like the CISSP or whatnot

we had one student the second year who went on straight from our course to take both the CISSP and the ceh and he passed both of those on his first try we don't we don't train for that or teach that and so it's that's not a good benchmark for us yeah I think you have a knowledge of of both I think it sounds like that this would be an amazing foundation and depending on the jack certification which is more specialized depending on the subject matter it would give them a great head start in some of the areas yeah like the CCNA is like Cisco vendor-specific so it's also proud not a good model for this particular type of

training any other questions alright guys had one thing that actually so I I work on a red team for a large financial institution and I get asked this question a lot so I'm actually a little bit for you bringing this question a lot though so how do I get started or what do i do how do i how do I get a job in the fields right my answer has always tried to encourage folks to get some hands-on right through CTFs or an osep if you have no experience at least if you do like three months in the lab or you know it's your least getting to hands-on even and his comments on the osc Peaks it drives me

nuts too cuz I've you know I've gone through it spot-on it's just older stuff but still the stuff still relevant and it's a great foundation for moving forward plus a lot of employers still benchmark you know the certification as well so also he mentioned how much it is for six months if you look at the per cost of a sans course for a six day course that's a bit they're like up seven thousand make sure you know dollars for a six day class this is six months forty hours a week for fifteen to twenty thousand hours that's actually a pretty good deal so but the hands-on methods I definitely you know wholly support that so yeah I

appreciate it and so for us what the way we say in coming up with analogies which is ironic cuz I'm like the least fit person but when we say his Escalade is a Jim Wright like everybody wants to go to the gym and have the muscles but very few people invest the time to lift the heavy thing up and down and get the muscles and so you're not gonna learn in a one-week boot camp right you'll feel good because you just like bask in the presence of a luminary who shared some wisdom then you're gonna go back to your office on Monday you can't apply any of it because you didn't really get it and

you didn't really get it because you didn't do it and so the reason why this training is so effective is that it sucks like it's hard you're gonna hate it because you're struggling and that's that is the investment in the gym right that's the heavy lifting up and down and so you'll have your aha moment you solve the challenge you look back like holy [ __ ] like I spent like how many weeks learning it but now I get it as opposed to like let me just look up the answer in the back of the book and move on right so that's kind of the model that we've come up with I was so like how you said you guys work in a

team environment so you know consider support there's built-in support and last minute questions one more question no great thank you please give our website is it takes a few minutes