Joe Kuemerle - Everyone Can Play! Building CTFs To Teach Non-Security Folks

awesome thank you so much i really appreciate it uh my name is joe kemmerly we're going to be talking today about building capture the flag in use for security training for non-security people i am a product security engineer at salesforce i am not speaking on behalf of salesforce uh don't take anything that i say as investment recommendations we are a publicly traded company i'm not talking anything about financials don't base your retirement on me uh i will be talking about things that salesforce has published on the engineering blog and a few other places because i have been a participant of actually building ctfs for doing security training for both salesforce customers as well as salesforce engineers

there will be an interactive component to this talk because i'm working on getting ways of the audience involved so i will be asking various times if you want to type things into the discord chat i've got it up on my other screen i'll do my best to keep up with that while still talking and with that we can go ahead and jump right into it there are three main things that i want to cover today for the purposes of this talk we're going to talk about the why the what and the how the idea is to give you enough of a framework to take this back to your companies back to your own personal lives however

you want to use the uh ideas that we've come up with to enhance security training make it a little bit more memorable make it a little more interactive and really gets more benefit because honestly eyes forward slideshows by security people until your eyeballs fall out doesn't seem to be an effective way of informing especially developers on security topics and getting that information into their heads so that they use it again in the future so this is an idea that a couple of us had at salesforce uh ctf competitions have been a staple of security event for years i love it they're fun they're competitive ways of showing off your skills i personally find it useful because i

i use them as a learning tool i've learned about vulnerabilities i've learned about techniques it's given me incentive to find things out uh we find it very useful in general in the application security field and there's been a lot of talk about gamification and you and how it's useful to attract and retain attention how we've been struggling to get all these apps that concepts to be retained and brought forth again when the developers actually writing code so we decided to build some games around caption and use capture the flag as the game model for doing security training this is not just something that was decided on a whim and said hey capture the flags are fun let's do it we

actually did do our best to put science and research behind us i'm sure many of you have seen the learning pyramid or the cone of experience out there this is widely circulated and also broadly debunked set of research one thing that you'll notice is looking at those pyramids is those percentages are awfully suspiciously round numbers and you know reading is 10 you only retain 10 of what you read that's arguably not true at all broad base of our learning comes from reading and if we're only recalling 10 it's really at odds with our day-to-day experiences so a few years ago cisco put together this survey multimodal learning through media this is an actual scientifically based

survey they did deep research worked with education experts performed experiments and have a quite a bit of resources put into this it's an awesome awesome paper i strongly recommend taking a look at it reading through it it was formed the basis of a lot of our thoughts around shaping our programs uh as well as if you really want to get a chuckle there's some really fun educational specialists that deconstruct the learning pyramid in hilarious ways what we picked up from the uh the research in the cisco papers is how to build ctfs and use them as a learning tool we're focusing specifically on two different technical audiences and for the purposes of this talk i'll

talk mostly about that the first audience are salesforce customers salesforce is software as a service we actually build a platform it has a shared security model our customers can actually write code that executes on our systems in our environment so they are our customers are partly responsible for the security of their systems if they do something bad programmatically they can expose holes into their and expose their data through no fault of salesforce the other audience that we tend to focus on are internal developers salesforce is a product company we build complex complicated app mostly web-based applications we have thousands and thousands and thousands of engineers writing more and more stuff every day and we want them to

write in the most secure way possible we want them to take these learnings and really incorporate them a third audience that i'll touch on a little bit through this talk is some of the work i've done with teenagers and lesser experienced people who are interested in technology built and using capture the flags in order to get uh teams really more interested in technology in general and security specifically the next generation of security engineers has to come from somewhere right the main thing we wanted to take into account is this uh square here of the different types of learning and what skills they're most relevant to coding and security topics in coding are complex skills it's not something

that's a basic skill so we're more import we're more worried about the second column there the higher order skills and how to get the most bang for a buck in getting the concepts taught retained and then used in practice and we found that the interactive model of multimodal learning is our highest bang for the buckets our highest return on investment it gives us our students the best chance for success multimodal learning consists of many different principles there are eight of them there on the right hand side we'll talk about some of the other ones that we used as well but for now we're going to talk specifically about the direct manipulation principle something that's hands-on something that's meaningful

we're going to give people something that they can do and have that actual benefit of doing something as a part of learning it that seems to be a fairly effective way for this uh we wanted to build something that our players can actually build upon and again we have two separate audiences so what do we do for the customer facing audience at salesforce where most of our customers are sales force administrators and developers we built a capture the flag specifically around salesforce technologies and tooling where we would give someone a deliberately insecure sales force environment and issue them a set of challenges that they had to go through to make that environment more secure for our internal developers we needed

something that was uh a little more developer focused that we could build on uh while we do incorporate some of the sales salesforce as a platform challenges in there we wanted something that was more technically relevant to their day-to-day work again it should be relevant to them to their work so we started with using oas juice shop as our basis for our internal development challenges partly because it's a fun little application it already has a capture the flag mode and supports a lot of interesting uh concepts that we wanted our developers to learn so by giving our players the ability to directly manipulate a real world experience that is performing all the steps required to secure something that is not

secure it's going to reinforce that behavior and ensures that all of our players are going to be learning in the way that's got the highest measured success rate as i said we have three different types of audiences with various levels of security experience we built internally this is a plan that we put in place uh before covered 19 hit there is has been a sales force for many years a onboarding process for new engineers where they get two weeks of boot camp of coming on board learning salesforce tooling and technologies there was always a significant amount of security training built into that uh traditionally it had been a product security engineer going and lecturing and then having a handful of scattered

exercises for the pla for the participants to practice on when we rebuilt the program we incorporated capture the flag as a central part of it so now our new employees come on board they get their training and they are immediately challenged with here you're now competing against the rest of the of the students in your training cohort for fun for prizes for bragging rights by performing new security actions and it's really brought engagement up and again on the customer facing side we are at salesforce we were putting on a number of customer facing conferences and wanted to target those audiences with the uh platform specific challenges our goal is using that gamification in order to

increase knowledge we want to increase knowledge by increasing interaction in order to do this we wanted to have something that's approachable and easy to use we know that we're competing for limited time and attention no one has enough time today to do everything they need uh we can't just give them you know some fun picture hey come do security stuff it's fun it's not always going to catch attention we want the entire experience to be smooth and engaging we want to keep them involved in us the longer we can keep them involved the more topics they're going to be going into and the more that they're actually going to be learning and using to that extent we integrated everything

as much as possible so that everyone could have a smooth experience and minimize the amount of flipping between windows changing contexts measuring their learning measuring their successes minimizing the amount of changes they have to do lets us maximize the time we have to teach them things one reason for keeping it as integrated and immersive as possible is at the customer specific ones this is what we are competing against we are trying to do security training while there are thousands and thousands of distractions that our players could be looking at and doing other things so we have to keep them involved to keep them interested this is where we got to draw on our research once again

using the uh multimodal uh configurations uh looked at some of the other principles so keeping things together in space keeping things together in time uh keeping things uh represented in a single way so not making them do multiple things to accomplish the same goal we are building all of these principles into how we are developing and targeting our capture the flag challenges to make them relevant to keep them together and to keep our players involved with the minimum of overhead i won't go into a lot of detail on the overall architecture of some of the things we built uh i did write some blog blog posts on the salesforce engineering blog uh they're out there and available if

you want to see some more in-depth writing about the actual technologies that we used sufficient to say we did build of course the salesforce focused challenges on the salesforce platform i have some nice apis that we wrote running in heroku we built on ctfd open source uh capture the bike scoring system as our platform and again more details are available in the blog post but to keep things as easy to use as possible and is integrated i will walk you through what we call our integrated challenges this is how our customers and anyone else can interact with our sales force based challenges for uh for finding and catching flags so it starts with registration you register yourself with ctfd our

first integration is that then behind the scenes we create a trial instance for that user and they get sent an email where they have to verify their account log in they bang they have a sales force what we call a sales force organization it's just an instance of sales force that's used for them that they have full access to they're presented with a series of challenges there's a set of trivia challenges as well as hands-on interactive challenges because there is some period of time that it takes for the salesforce backend to spin up so this is one example of one of our administrative level challenges we're giving the user the information that they need that their

sales force organization is probably vulnerable to cross-site scripting attacks they need to go and figure out the right way to make it secure they go into the platform do the right thing by clicking the check box and they don't have to do anything other than be in the capture the flag system and say verify there's api calls that happen behind the scene we inspect and make sure they did the right thing if they did do the right thing and they solved that challenge we give them their flag they submit it and they get their points they go on the leaderboard easy peasy they have a maximum of two browser tabs that they have to work

in and the same workflow takes place no matter what challenges they're doing with that we had some really good participation uh last year at our dream force conference we had hundreds and hundreds of players come by and actually sit and play in my opinion the sheer numbers of players is not necessarily as relevant as this other fun measure which is the amount of time that our players spent involved playing capture the flag either median or average time looking at these numbers we're getting tens of minutes and sometimes an hour on average of our players interacting with capture the flag think about the last time you went to an in-person conference with thousands and thousands of your

closest friends and you spent 10 minutes 20 minutes an hour involved with any one thing it's probably very unlikely unless you were working in another capture the flag again the average number of questions and engagement that we got from our players was very significant they are um while we did have a few players just come in answer one or two and go off and do other things that a large number of our players actually did participate to a great depth and and made this something that they were very interested in we have different categories of challenges and all this is out in the blog so i'm not going to cover a lot of it because we have a lot of other things

to cover but we do try and make it as broad based and as interesting as possible which brings me to audience considerations you need to when you're building these capture the flags one thing that you'll need to unlearn if you've built ctfs for security people is that you need to take into account your audience what their level of comfort is with technology what their learning goals are to get out of this experience this is not just put the most brain teasers that you can out there that will frustrate a lot of people i've learned that many in the general population are not as crazy as a security folks when it comes to i want to find out the most mind-bending

way possible to do something you know make it so obscure so difficult that the fun is in the hunt it's not necessarily the case for a lot of other people take into account your audience as you're targeting your capture the flag and if you're going to do something for developers versus business people versus as i said i've done some work with teenagers before it really is a way of piquing their interest but it can't be too frustrating step one is building our challenges how we're going to determine what set of challenges are appropriate to put into a ctf there are a number of considerations that you need to keep in mind and again this is where it differs from

a security player ctf we are not making this obscure we are not building this as a tool for i know the most obscure set of vulnerabilities that i can chain together to break into this particular thing this is a learning and teaching tool so we want to keep a number of things in mind especially the appropriateness the difficulty and the playability of our challenges i have a set of five criteria to keep in mind uh that i recommend that we work on as we are building our challenges uh just ways of making sure that what we are building is going to be useful and that it will captivate and engage people versus presenting them something

that's difficult to understand and it's going to put them off speaking of putting them off there are some red flags that you need to be aware of when building out your challenges three major ones that i've come up with are listed here on the slide things that you want to tend to avoid unless you have a really good reason for putting that sort of a challenge in and especially the relevancy one we are building tooling that's going to teach people things that we want them to take back in their day to day day work so while stenographic challenges may be interesting and fun unless your developers are doing something like that as part of their day-to-day work

it's probably not relevant things that you want to focus on more probably injection attacks cross-site scripting any of those categories of vulnerabilities where you've noticed your developers need to have a little bit of remedial education and to bring that into their coding challenge building doesn't work well in isolation collaboration is a great way to build challenges collaborating within your team it helps you to stay focused it helps you to think of what your most prevalent security issues are in your organization and then you target the challenges that way so you plan out what you're going to build you work with others you can actually work across teams across groups to see what else uh what other

areas the company may want to contribute to this which is great because then you get extra challenges with no additional work you really want to stay focused you want to limit your challenges to the small number of specific names and if you are building capture the flags i'll talk about some tooling later on uh that makes it much easier to build consistent repeatable capture the flag environments and tune them appropriately so you can collaborate and have a library of different systems of different uh focused areas that are appropriate for different audiences and use them when you need to and where you need to i came from the developer world i am a huge proponent of agile techniques

so collaboration collaborative building of things testing reflection on that having regular retrospectives and then iterating on your content to hone your challenges is extremely effective it it helps you to to build and focus much better when you're working with others and to have that feedback cycle and to have that continual improvement process especially because you then your challenges don't get stale and you keep them relevant so this is not just a one-and-done process this is going to be evolving over time uh late and again later on we talk about the how we'll talk about some of the technology that's out there that can really help you to to be effective with this this is the point in the talk where if

we were all in the room together i would have you stand up move around get into groups and do a little bit of interactive and collaborative challenge design unfortunately we can't do that but this is where the discord channel is going to come in handy what i would like you to do if you'd like is to we're going to take a few minutes and i want you to think about what might be a good challenge for your particular audience you have in mind whether you're thinking of developers at your company uh teenagers that you want to teach topics to business people your company come up with a challenge idea something that you think meets the criteria of relevancy that's

appropriate it's interesting it's solvable that has concepts that you want to reinforce while avoiding your red flags go ahead and type something into track one on the discord channel so that everyone can see what those ideas are this is also nice because it gives me a short break from

talking

all right i see a couple of people typing so that's awesome

nice as i said i'm happy to take questions uh at the end of this i'm also available on twitter i'm out there on linkedin and everywhere else so this is this is a topic that's near and dear to my heart i'm always happy to talk about it so we had someone who did some work with high school kids little to no network experience network focused challenges that would be a really interesting one especially thinking about how to incorporate flags into that another idea i have three similar leads with keywords as a clue absolutely uh picking out keywords is interesting and making things as hands-on and relevant as possible is what we found is a way of building success

the last time i did this in person at a salesforce focus conference i had one guy who spent most of the day on the ctf and he was so excited so happy about it he actually ended up uh winning it but he came up to me near the end and he's like this has been great i've learned so much today although it's going to be a little hard to explain to my boss why i spent all day at one spot instead of going around the rest of the conference to which of course i replied well just tell your boss you didn't go around to all the other areas of the conference but you did get

a great tour of all of the security feats many of the security features that are available to you so so it wasn't just doing one very focused thing it was kind of broad based but very specific on the security features that they needed to know for day-to-day work swift coding using it to keep engaged how do we track and manage challenges that as they're being developed uh i'll talk a little bit more about that and how but basically what i have and what's out there that i open sourced are some tools uh we use yaml files for doing our collaboration so we've got git repositories that have the various challenges in them we use ammo files

people collaborate on writing challenges and then i have a set of tooling that pre-processes those gmo files converts them into ctfd format and some other tooling that cftfd is spins up and automatically imports them so we can i've got a continuous integration continuous deployment process set so i can push a button and kick out a fully functional ctf so those those links will be in the slides and elsewhere as well awesome as i said making it easy to participate is key we want to be able to make the enrollment as friction free as possible you want to make it as easy so uh in terms of customer focused at conferences it's a very easy registration

process other audience types like the internal security training we have at salesforce we've got a little bit more leverage over the people in that this is now part of your job so your job requirements are to sit here and participate in the ctf for the next couple hours planning out how you're going to get people on boarded getting their tooling together is is a critical part of success and honestly now that burp has the integrated browser newer versions of burp has made our job so much easier uh because traditionally that was one of the hardest things of teaching anyone in these these capture the flags and security trainings is getting developers to set their proxy right and getting the

uh the proxy certs trusted on their system so i'm so happy that burps got that built in and that's really helped with our onboarding and enrollment integration is another pillar making it very easy to participate by making a smooth seamless set of steps and as few steps as possible for people to become successful anything that has an api is an ideal candidate for integration where you can have back-end services that can use metadata to examine whatever environment you're in and determine whether the player has met the goals of the challenge or not this is something we're doing for salesforce we don't just have flag values hidden within the salesforce instance because the player could just

go and find them and not actually have to do anything to solve the challenge the actual flag values are stored in our back end apis that validate that the player did the right thing this is something that you can take to cloud providers you can take the aws azure gcp and if you're running stacks on docker kubernetes other software as a service platforms that have sets of apis where you can inspect metadata you can make focused challenges that have this integrated verification step so that your players have an easier time of getting their flag values and again that is something that is open source and repository at the top of the screen there was a

little ctfd plugin that we wrote and released

to make it engaging for people to participate you have to have good scoring it has to be understandable it has been meaningful you want your scoring to reinforce the goals that you're looking to accomplish you also want to maximize interaction it's not just shouldn't just be winner take all and you want to make it so that players at the range of technical levels that you're targeting are participating so someone who's less familiar with the tooling technology and someone who's a little more familiar should both have ways that they can engage they can get on a scoreboard and honestly have a potential to win whether it's you know the top x players get this nicer prize and we'll pick random uh

participants that meet a minimum bar of scoring that's a great way of encouraging people to have that set of participation and that they don't have to be the top scorer in order to get some reinforcement and be able to have that recognition of playing this is another thing that i'd like to ask if anyone has ideas i'm not going to pause but if you do have some ideas of interesting things that you can think of for reinforcing this type of behavior and scoring feel free to put it in the discord channel so that everyone else can see it but definitely avoid a one winner take all mentality it does not help encourage other players uh to maximize their participation

you need to build interest for this not just from your target audience and from your own team but across the board you want to get buy-in and sponsorship from your leadership this is going to be something that takes time and effort to develop this is not something that you can just get off the shelf especially if you're targeting a certain audience there is an roi behind this and if you can demonstrate that to good management they will absolutely be on board your return on investment is this ability to show you know we are teaching people security topics and you can pull metrics out of just about any ctf scoring system and be able to come up with numbers for

you we are getting some good engagement we are getting some interest as well as getting you'll always get those good anecdotal data points as well every time we run a capture the flag which is every few weeks we're on boarding new engineers we usually get feedback from at least one or more of that group saying this was so much fun i stayed up until 5am because i was so involved in it so we have all those anecdotal data points as well you will need to have some resources you'll need time you'll need money you'll need other people's time so you want to work with other groups you want to have play testers involved so either within your

team some of the leads maybe your security champions to help play tests you want them to have the ability to have that exclusive time to focus on play testing feedback and improvement now with the entire covid situation uh this is absolutely great because no one can be in the same room to as much of an extent as possible so this works well remotely players can play from anywhere they are in the world this is great for virtual conferences as well we're playing capture the flag here most other events do and you don't need to be anywhere other than on the internet you want to have this great discussion of how to build these ideas out work with other

groups and communicate communicate communicate communicate up the chain with management what you're doing why you're doing it how you're progressing communicate across your level maybe look at working with other teams you can engage other teams in your company uh for building challenges if you have something like a gr a risk management group a grc group infrastructure networking anyone can contribute to this and what you'll get out of it are a set of challenges that you can have in a single ctf or across multiple ctfs that cover a broad range of interest at the company and that's also a way to build that collaboration think about how you can publicize this this is something you really want to

draw attention to you want to get people interested about it you want them to be aware of it you want to bring them into play and then you also want to communicate after the fact of recognizing those people that have participated recognizing people that have done well as well as doing your best to recognize across the board be open about what your expectations and results are this is not a cure-all nothing is ever perfect some of the measurements may be slightly fuzzy you know engagement time is one thing but that doesn't necessarily always track with they're learning everything but you should you'll normally get that level of interest from your players and they'll they'll be interested and

involved in it it is something that not a lot of other people have done so it's usually something that's unique and interesting having the right incentives is key uh competition is an incentive um but other things can help you want to make your incentivization as inclusive as possible so you're not discouraging people think about how you can have multiple levels of reward so that you're encouraging everyone to participate to at least a minimal level so maybe have some sort of staggered sets of levels of if you can manage to get so many thousands of points you'll get virtual swag you'll get a you know badge on the employee leaderboard you get so many thousand more you get something else

this is really an area that is specific to your particular time your particular circumstances and what your leadership is comfortable with doing i don't know that it always i don't wouldn't say that it should be money it should be something uh it's not quite as transactional as that one thing that i've seen is fun is issuing challenge coins so people that make a certain level on the ctf are eligible to challenge coin something that's not really commonplace across developers development organizations something that we're familiar with in the security space and it can be an interesting way to get people a little bit more involved because they're like what's this challenge coin they get this little shiny piece of bling and it makes

them want to go and get more and learn more as i said competition is good to a certain extent we want to make it fun but not bloodthirsty keep in mind that people are different different people have different motivations there are some people that actually seek out public recognition other people want to shy back a little bit and not really be called out individually participating in the ctf usually requires some amount of competitiveness we want to make it friendly this is i love this gift this is a belt sander racing it's an event i think it's in vermont people put it on every year and you have to sit on a belt sander and race it's a

fun race there's a little bit of competition but it's mostly about having fun you can also encourage teamwork have collaborative challenges there is a technical conference that happens every january except for this one in sandusky ohio at a water park called codemash every year they put on a capture the flag and there are collaborative challenges there's a whole slack group dedicated to it players help each other out with the challenges there's quite a few it's a general developer conference some of the challenges involve cryptography various encodings and that and there's an open channel those of us who are in the security space do our best to collaborate with developers give them hints point newbies in the right direction for

solving their problem do i encourage players with more skill to play a mentorship role to new players absolutely that is an outstanding idea internally when we're doing this capture the flag training there is always a security person who leads the training and that person is available for consultation they do to do their best to work with the developers if there is anyone who gets stuck on a particular thing we give them hints and ideas as to how to solve the right challenge

on that note making your challenge is the appropriate level of difficulty interesting and engaging yes impossible to solve probably not unless you've got one bonus question that you want to make nearly impossible to solve just for those people who may really want to stretch themselves focus on your specific audience even if it means cutting down a number of challenges making things too broad making them too easy too complicated you're going to undermine your ctf success maximize your end goal of internalized knowledge not frustration

keeping your scoring difficulty versus reward you want to have appropriate motivation for people on getting points on a board tends to internally motivate quite a number of people make the points reflective of the difficulty of challenge of the challenge uh make sure that you do have a good distribution of challenge difficulty something for the newer less experienced people to participate easily in and a set of things that are a little bit higher value that maybe are a stretch for them but really can give them that that drive to look into it and do some research this is also where your challenges can branch out beyond something that's just a technical challenge something involves collaboration with

another team with groups with departments one of our processes internally is there is a secure software development life cycle at salesforce like i'm sure there is isn't most of your other companies there is an onboarding process where changes that the development teams are developing they have to submit for a security assessment in the onboarding and this dev boot camp we lead the entire group in completing a challenge to go through the process of creating this security assessment and submitting it so that there is this group learning's taking place and it's something that everyone participates in and at the end of it everyone gets a flag so it's a nice way to get their points on the

board it introduces them to other people in the group and can really help build that collaboration for these uh new engineers that don't really may not know a lot of other people it's a great introduction for them to meet other people on the call because they're now not in the same room with them you want to follow up on this uh this is a follow-up for communication make it uh do a lot of follow-ups after the event poll people find out how you can measure their interest measure their success this is not just helpful to bring that feedback in for the next time around you can take this and again use the anecdotes use the data show that

this is uh interesting engaging you've got that return on investment that is justifying the use of security budget on this think about how you can use the buzz from this to build on it as a feedback mechanism have goals to have multiple ctfs with increasing levels of difficulty maybe have elimination rounds there's all sorts of different things you can do to uh use this feedback to build excitement for the next round and doing this over and over again using tools is actually pretty straightforward this is also a really fun way because there are never enough people in security think about it you know who the people who are most involved in this is who are the most

interested and engaged they're ideal candidates if you have a security champion program you've just now got a really great way of identifying security champions more than that you've probably got a way of poaching engineers that are really good at security to come over to the dark side and actually get paid full time to do security get people who are that interested you know do a little internal recruiting with it doesn't hurt making it easy to run so this is where it gets uh where the rubber meets around it's not just fun and games it takes a lot of effort to build a ctf we've already got so much work how are we going to be able to fit this in

uh we fit this in by minimizing the amount of effort and friction ever in any place possible we want to make this smooth and we want to make this easy and of course the first thing is authoring challenges i talked a little bit about this earlier we threw some ideas out in in the channel but we want to have a way of building collaborating like i said there are in the resources there are some examples of how we're building them internally using yaml uh you can use markdown you can use whatever but we have a structured set and then a set of tools for converting that easier to collaborate on artifacts into a ctf that's fully functional with the minimum

amount of human interaction required and playing the same thing over and over again it's going to get boring so you do want to be able to plan for your ctf to grow change and evolve over time so it should be something you can easily update keeping that integration in place is great keeping that code updated using tooling that already supports what you need to do whether it's a set of apis various ctf platforms have different things that they can do for automatically creating environments just starting off g shop is in ctf mode is a great easy to start tool for getting a very low friction and relatively easy to integrate ctf up and off the ground

hosting it is always going to be fun there's a lot of platforms out there and so we picked ctfd mostly because it fit in with what we were comfortable with and what we're building and has that open source and especially the plug-in ecosystem uh there are a lot of other frameworks out there use what works best for you that github repo on the screen is a list of ctf resources really nice to look through and figure things out highly recommended that you check it out and deployment minimizing that friction means minimizing the repetitive tasks we i've set up continuous integration continuous development i can spin up a targeted ctf at the touch of a button

and because salesforce owns heroku i've got a great environment that i can immediately push it out to there's also a free tier of heroku free tier of so many other cloud providers so you can very easily and cheaply spin up proof of concept environments but if you're going to invest the time in building a ctf i strongly recommend investing the time making a continuous integration delivery process some sort of automation tooling to make it so there's no human interaction going from to go from collaborating on challenges to having a ctf system pop out the other end the more hands-on work you have to do for that the less frequently you're going to be able to do this

and the harder it's going to be and it's just going to get to be too difficult for everyone to do so having repeatable ctfs and stable environments lets you spend the minimum amount of effort to get the maximum amount of success we've i have this set up right now we can spin up these environments make sure that your environment is aligned with policies and requirements if you're building challenges that have sensitive uh properties of them if you're working with internal systems if you're working with anything that may have particular privacy ramifications don't expose your ctf out to the internet at large make sure you set the appropriate boundaries for accessing your ctf system whatever works for that particular

scenario uh keep in mind remember you're exposing information in the ctf so you want to make sure the access is appropriately controlled some of the tools that i use for developing this tool sets are built on core ctfd which is built on python node and various apis is the some of the platform for uh for the api section use a lot of testing on docker and heroku everything out there is free nearly free or really easy to use for just about any environment that you can think of i did a hands-on lab version of this talk this past winter at rsa that github repos out there it's a set of utilities and some documentation for

them that you can use as a starting point if you want to use ctfd and if you want to use the data representations that i've settled on or you can take it as inspiration and go off your own way it's just some basic node scripts basic uh transformations and then it's all hosted on heroku if you want or docker if you want to just run it locally so now we have our easy to collaborate easy to run environment we want to make sure it's effectively used again this is not just having fun we want to reinforce those learning principles teach those skills reinforce it having the appropriate metrics and measurements it's going to help us to demonstrate our

success and justify why we have that money this is going to be something that's ongoing when you wrap up one you can go ahead and start on the next one i'm going to build on that communication when you run one successful event you might you know it may encourage you to run even more usually if you start playing ctfs you go out and you start looking for even more this is something that i found applies to me for building them as well i built one and then i wanted to go and build more so make sure that your you have those processes in place to accept that feedback to incorporate it follow up with players build and

update challenges and again going from the finish line of one right into the start of your next one keep that momentum going so we have all this we want to be able to communicate effectively how do we convince management that what we're doing has that value what else what more to managers like than dashboards shiny graphs bits of information statistics most of the ctf platforms you can very easily extract that information out of use it to build graphs and charts to demonstrate whatever measures of success you're using to reinforce these concepts i'm approaching the end of my time so we did cover the why the what and the how we're incorporating formalized learning principles uh

we are building on successes we are defining our metrics we know that we want to increase security knowledge we want to keep it top of mind we want to improve the retention of their concepts we're doing that by keeping the audience in mind building relevant appropriate interesting engaging solvable challenges we are avoiding obscurity we're avoiding non-relevant uh challenges we're avoiding something that's too open-ended and has too many potential solutions we're collaborating on our challenge building we're reaching out across teams we're making our capture the flag easy to use we're making easy to get into we're making it straightforward minimal effort minimal contact switches making it easy to run for ourselves so that we ourselves as executors of it

have minimal amount of overhead in creating these environments and running them we want reusable pipeline we want consistent deployments we want to make it effective we want to have our measurements we want good metrics we want to have an iterative process so that we can take what we've learned and build it back we want to be able to clearly communicate our results and we've covered the how how we can build challenges there's tooling that's out there that we can use there are blog posts that i've written there will be more coming on how we're doing this we have this automatic ability to automate our build and deployment we have our hosting we want to make sure

access control is in place if we want to enhance our experience there are ways of customizing all of those ctf platforms for doing making things easier so many different dashboard toolings you can use your existing bi tools and what is one ultimate goal having a self-provisioning process where you can have a team that wants to do a team building exercise be able to come in push a button and be issued okay here is your ctf environment with yours you know what they set up challenges so they can have inner team challenges across team challenges something that lets people very easily have a fully provisioned environment if you like this and you want to take it

back with you there are three things that you can do for next week three months from now and within six months i'm not gonna go through these but these are steps to success that i feel are appropriate and achievable within those time spans uh gives you a great set of tooling to have as a part of your security curriculum with that i am done talking there's a set of resources here like i said there's some open source work that the salesforce has contributed out to the universe i'm very grateful to them for letting me write that and then releasing it out as open source the learning resources are there blog posts there ctfd resources and then a few of my own

personal scripts that i open source myself that you can use as a basis for building ctfs with that i will go ahead and wrap up if there are any other questions please feel free to type them out and i'll be happy to answer them thank you for attending besides boston and i'm available anytime i'll have the slides updated and posted in the slide channel after this