CTF@Work, School, or Anywhere

Thanks so I have a story to tell about how we were able to use games of a couple of different kinds to help raise the level of awareness and with a security within our organization and also just try to give people a chance to brush up on their skills in ways that may not come up and as part of their everyday career got the obligatory about me slide my the best qualifications I have our number one I'm a big geek and number two I love playing games that's pretty much it of course if you're a an old-school hardcore geek this will tell you everything you need to know about me probably more than you want to know

about me and of course I have to start off by saying that these are my thoughts my ideas my story my mistakes and anyone that I work with or work for may not necessarily agree with endorse or certainly wouldn't consider me representing them up here doing that they have they're starting to hell they can come up here and do this so that's kind of how I feel about that so enough of that let's talk about games now if you let the historians or the anthropologist tell the story they'll probably tell you that humans have been playing games as long as there's been humans and it seems like as soon as we figure out what to eat and make sure

nothing else has us on its list of things to eat we want to spend a lot of the intervening time frittering away playing games and wasting time but it's not so much a waste of time I think there's a a method to this madness when you think about it where do we get our initial download of operational data for how to be humans we start off interacting directly with the world through games through play we learn how to figure out how to make our bodies do complex maneuvers without falling over that's a pretty good one basic life skills we first encounter and learn these things through games through playing now as we go on of course the

games get more sophisticated we learn things about language and the nuances of communication we learn about strategy now opponent against you know someone else is trying to out think us we learn about resource management we learn how to build a castle on the frontier of a French Empire as you do and sometimes we play games that banish - the weird table at lunch but even in a situation like this we learn practical things right what does this game teach us well we learn that not every encounter with another person is a zero-sum game we don't have to make someone else lose for us to win we learn how to band together as a group as a

team and accomplish a common goal we learn engineering design we learn imagination creative writing I think one of the most important things we learn is that despite our best intentions and careful planning things don't always go right and can you think on your feet to come up with a plan B when you discover something now there's a lot of this information that we try to still convey in more traditional settings someone will stand in a room in front of you and tell you facts and that's part of the learning process - but it really shouldn't be the only part of it otherwise you get audiences that look like this okay yes but and that leads to

a lot of frustration when you go to apply things because you know hey look I've been told I need to implement this circuit or design or reverse engineer or debug something and I've tried it three times already and I'm just not getting it and this is too hard and I just wasn't cut out for this I quit but you put in a game scenario what happens the 37th time in a row we've crashed our mario card into a tree or we've fallen off a level in some game our reaction is okay I'll get it this time set it back up again especially if there's a little bit of competition with the person sitting next to you with

their controller and you want to just get one more second off your time now the idea of using games as a teaching vehicle is something that isn't new I'm not the first person to think of it by far it's been in the news lately it's also been the subject of a number of academic papers and conference presentations and even a master's thesis here and there so people are talking about it they're looking at it I didn't start my journey by looking at the material or any of the research or reading about in the news I started by having the leader of our red team work come to me with a question I never thought I would actually hear at work

asked of me but his thought was how we approach this sort of adversarial war game kind of scenario how do we learn to think like our enemy how do we learn how to defend against them says that's a lot like a game let's make a game out of it instead of talking theory let's roleplay it let's simulate it and one of his points was whenever we do it the other way it's a real pain to get people to actually show up and participate but you say we're gonna do a tabletop exercise D&D for systems and mins or whatever people show up and they get into it it gets fun and they still learn the strategies and work out how to do the

the actual work it helps you get into the mindset of who you're going up against and to think in terms of not only theory but scenarios and what happens next after this event and what do I do and I respond this way oh and that was countered by the other player doing something what next my point of view was not so much the red team effort but just in general thinking about how to help my colleagues build up their acumen for different kinds of security work and we do a lot of different kinds of work if you look across our whole team but using games a lot of the similar way think in terms of scenarios

what happens next think about problem-solving and above all make the whole thing fun enough that people want to stay engaged and you yeah even here you can use competition to keep it fun and keep people motivated and that actually turned out to be a lot more of a advantage than I originally would have thought now one thing that's important if you're thinking about doing something like this is to know who you're working with now the obvious angle of that of course is that everybody has a different level of experience you have your your hardcore red team that can tear apart hardware and know what's going on you have people that live in word processors all day writing policy documents and

everything in between but everybody has something that they can gain by getting into this and you can use the mechanics of the game you set up to helped overcome that obstacle and also even leverage leverage it to your advantage give someone you'd give everyone another a feeling like they have a reason to be there and something they can accomplish the other angle of it is not as obvious and I think of this as the system indents because there was one time I was at a conference years ago I think it was a UNIX Lisa conference and unfortunately the only thing that I remember now of that whole conference was one of the evening activities the

organisers had set up a dance in one of the grand ballrooms of the hotel for systems administrators now I'm not saying sis admins can't dance but all I'm saying is I was in this ballroom with a couple hundred of my colleagues and they were variously standing and sitting on the dance floor talking about network topologies and how to set up file servers while the most confused band in the world played on wondering what in the world was going on in this venue it's really important to understand the expectations of the people you're working with we made a mistake with this in terms of this talk I'll share that we had a class it was for intermediate and advanced

application developers it was an application security class and we taught them how to look for exploits how to how to exploit them how to prevent them it was great we had an in-class lab which gave them the opportunity here's some intentionally vulnerable apps let's walk through the process of breaking them works great as things went on we decided to move it to a self-service online kind of class nothing wrong with that so the labs went along we hosted them in their own little server that's fine somewhere along the road we came up with a new class that was an introductory level a security awareness class also a great idea somehow we stuck those labs with

that class and we've expected we might have some problems with the the ability level so we addressed that we solved it if you ever played the the old infocomm games back in the 80s or 90s and bruh how they had the hint book where you could reveal clues one at a time the one starts with maybe go over there and look at things and at the end you get down to the clue that says go take the thing put it in the other thing push the blue button and we did that with these labs so that they would go through the experience of doing it even if they had to be led step-by-step and it was even things that said right here

this text right here that's the that's the thing and people some people rebelled against this because we were so mismatched with their expectations not the ability level but they came into it saying I don't understand why you're having me do this this doesn't compute in what I thought I was doing by taking this class and there we missed it so just to underscore that that's actually a surprise for us anyway now where I came into this was actually very casual never intended to start building any kind of a system or a game framework I simply said hey you know be a great idea if we had some kind of little puzzle some kind of challenges we

could throw out to the security team and this started off in an email and sometimes in meetings would say hey for this week why don't you try solving this it could be something simple like this cryptogram and then we'll come back next we can talk about the answer sometimes the challenge was figuring out what the question was in the first place and maybe maybe getting past the point of wondering if the challenge is whether I should scan this thing into my phone in the first place or not fair enough and as time went on we started getting more sophisticated with it and one of the things I developed that I wanted to try and do more of was to guide people's

thinking out of the normal avenues they were used to in their jobs and so it have a challenge this is an example of one of them later on in the game I said consider a hypothetical padlock digital padlock has four digits you'd type in on a keypad and if you get the right digits it'll open if not a little flash of LED or something telling you what's wrong pretty simple concept but in setting this up I wanted to make sure we had several steps that the first one or two were really easy for people that haven't thought about this kind of security before and could still have an aha moment and then we get deeper from

that but I wanted to make sure we get out of the the the initial reaction so the first question is how secure does that sound now we think about it four digits if I had to brute forces to have to type in all the digits zero zero zero zero nope zero zero zero one no that's not it either oh that sounds complicated and when you think about it a mechanical padlock that's really complicated it's got all the different positions those the different bidding patterns you know I don't know how people pick locks but it sounds magic makes you have to try it maybe all of the teeth are all the way down maybe one up maybe one up a little

more until you finally get the one that makes the lock open I mean that would take forever I don't know how people can pick locks and this is electronic and I don't even begin to understand how that's approached so it's got to be secure that's the initial mindset of someone that doesn't think about secure years and familiar with the kind of devices and so the first step was I said just quantify it how many times how many different numbers would you have to try before you had tried them all and get them to say okay well there's four digits so there's 10,000 possible combinations what I want to get here is four especially the beginners to get out

of the mindset of it's complicated so it's secure okay if you could figure out well exactly how complicated is it that's your first step in challenging your assumptions and this is a phrase I said so much in the application security classes I had a co-worker wanted me to put on a t-shirt and just wear it around all the time but this is probably the biggest thing that we want to try to accomplish people is to get them to challenge the assumption that they make when they first look at a problem and so said okay having done that now the next part of this challenge was a researcher has come to you and she claims that she can use a

technique where she's still brute-forcing she's trying each combination till one opens she's not got some way to look inside the lock but she can do it worst case in 40 instead of 10,000 whoa so the first question though is this even credible is it plausible could possibly happen and I said she says that this was caused by a common software security flaw so what I wanted to do here is to get them thinking all right is it possible what is the relationship between ten thousand and forty is there something there that leads me to what you might be doing which in this case would be identifying each individual number separately and not all in combination and also the fact that if

you're familiar with common software design flaws you might are to be thinking about side-channel timing attack something like that but if not you have something to Google for and if you look at the top 15 or 20 security flaws something that will lead you to some ideas especially if you figured out maybe it was trying each digit at a time it's kind of like the whole idea of the the physical padlock people don't pick locks by trying every combination together they they exploit the fact that we don't manufacture things perfectly every time well people sometimes forget is software works the same way so then is that having done that question three and this is where I really wanted them to think

beyond oh here here's how we can break the lock open okay what would you tell the developers to prevent this from being a vulnerability in the future because ultimately we want to make sure that we not only identified vulnerabilities but close them and so by getting them to challenged our assumptions getting to think step by step a lot of people came out of this one saying well I hadn't actually thought all the way through a problem like that before overall we also had some that were there they actually had a stand up some software that they would they would communicate with so we say okay telnet to port 1970 tell that or whatever somehow you connect to the port and it

sends back something like this and we explained what it was we didn't like Surprise them with hey here's this thing this turns out if you're also a geek like me use is a game of life board this is a perennial favorite just shows up in puzzles and computer challenges all the time it's in several CTF is probably all the CTF but the idea is it's a very nice simple problem that you can then base these exercises on because it simulates life in a petri dish you have very simple rules that determine generation generation where cells will survive where new ones be born and where old ins will die and you can code that pretty simply and then we

can do some things around it I'm enough of the geeky I actually will spend time in years past in a boring meeting or a class or something doing it on paper but that's are you afraid back to slide one I'm a geek I just have to live with that so people connect to the port and it spits us out and so you go aha it's a game of life board I know the drill I look at this I can even in my head go through in a piece of paper and write down what the next generation after this would be put it in a file netcat it back to the port I'm done right nope because

when I connect back to it it says ok here's the answer it's generating that randomly every time ok no problem I'll write the code which is obvious what I was supposed to do in the first place write the code that will read the board calculate the next generation send the answer back and I get my congratulatory note actually I don't I send that back and it says ok here's another one I wasn't expecting that ok no problem though I have all the code I'll just put it in the loop and what I'm trying to get people to do here is to and so they challenge their assumption about knowing when they first start analyzing the protocol here of

what the next step is going to be and you can sit and watch your program solve one after another and the thing will just keep going back and forth playing with your script and all of a sudden it throws one of these at you and then one of those at you now am I just good to to sleep being mean to these poor people yeah no no because not only been trying to get them to challenge their assumptions but this is key to the whole idea behind what ultimately are we going to do if you're debugging some kind of a system you're you're reverse engineering whatever it is you're doing when you first look at the behavior

of the system you're working with it will occur to you oh I see what's going on and you make some assumptions but you'll probably be wrong the more you interact with that these things are constructed like odors in all kinds of layers you peel a layer back there's all the more complexity that gets revealed the more you go and so if you go into it expecting that expecting surprises and to be able to respond to them so much the better so we put together challenges like these on all kinds of different topics that are relevant to the kind of work we do in security and the response to that overall was pretty good people said hey you know I realized this is

somewhat Sall selecting cuz people that didn't think it was cool just didn't play but the feedback we got was that people liked it they said hey you know it was a fun thing to do something that I don't normally do in my job and I learned from it but a few bits of feedback that actually made me stop and think how can we address this first of all there was the issue of the wide range of skills whatever we do with this it needs to be approachable no matter what your background level is the idea is to to try and increase that for everybody not just to cut out people who either have already been there before or

aren't up to the level where the challenge is written to other people said that was fun and I thought it was at a level it was okay but I was kind of intimidated by the idea of coming to the next week's meeting and not having solved it like some of the other people did or people talking about it in the break room later I want a safe place where I can sort of privately work on these on my own and then when I solved them then I can go hey okay solve the thing and the other thing was they felt like because it was coming out an email if they weren't there to start or if they needed some

more time they're like well if I you know next week I'm gonna hear the answer and I feel like you know I'm just gonna not play this because it would have been fun but that got in the way the other thing that occurred to me is I'm writing a lot of these challenges and it's fun and all but that's not what I'm getting paid for I want to if I'm going to keep this going in a an ongoing self-sustaining way I need to incur people to contribute puzzles to the whole thing and make it more collaborative and I think that's a great part of the learning process too is to get people to contribute more content to

what you're doing so I thought about it and it's like well what kind of a game framework would makes sense we've looked at role-playing kinds of scenarios the red team is doing that's great for them it seemed obvious to me that a capture-the-flag kind of framework made the most sense there's a lot of kinds of capture-the-flag games in particular I'm talking about the with the so-called Jeopardy style we have a whole collection of different kinds of challenges and different point values for the difficulty levels and the person who wants to play can come in and say I want to do that challenge and they go off and do it and they get points for it

so it's very self driven which was important to what we were doing and this kind of a CTF it's a little different than a more traditional capture the flag event or a contest where everyone comes somewhere on a Saturday and they play capture the flag and usually people who do this a lot they go and see who's the top gun and yet our team won the flag and it was great and then you go home at the end of the day and you're done that's a very different dynamic than saying we're gonna set this up and it's gonna be running for 10 years and people can come play when they want and I'll be talking not only about what that was

like for us to navigate but also some of the technical details so we'll talk about the people stuff and the technical stuff here but that was definitely something we had to to look at and along with that people have to be able to play at their own pace at their own interest and and to be able to join at any time without that being a disadvantage saying well you know you guys have been doing this for a year and I'm only starting doesn't matter it's okay we also wanted to address that whole intimidation factor by not making a leaderboard that showed everybody and what their position was and you're being put up against people that are way more

advanced than you so we went with a a tiered kind of leaderboard that took care of that and I'll show you an example of in a second and there's even how you set up a game like this is a little more challenging than an event where you can come set it up do it and go home so there's some things that that make the game continue to be viable that needed to be changed from what the traditional kind of method was like this is an example of what our leaderboard looks like our challenge board look like looks a lot like a kind of like the game show last night we have a category in each column and so they

can pick something they want to try want to try a coding challenge you go in that first column the point values are putted indication of the difficulty level so you can kind of pick ones that are our you know you can go down the list and realize that they'll get harder as you go but there's also a always a 10-point sort of a hello world level challenge in every single category that just confirms that you are tooled up for the rest and this is a category you'd probably successful for it so that quick little check like for instance in the code challenge the holo world thing just says telnet to this particular port and so

you do that you're telling it to the port and it gives you a greeting tells you what to do in this case just send back a string hello server and the opre but in an inhumanly short length of time it's not a challenge to see if you can type it's a challenge to see if you can code so what we actually want people to do is write a little script that will interact with a tcp port a few lines of code but you're going to need it for the rest of the category so we have them go through that so we'll run my script it connects to the port I can see the conversation going back and forth it

sees the greeting it sends the expected response and the system says congratulations that was right and here's a flag which I can then go back to the game and claim for my points and when I do that I get put on the leaderboard and this is what I was talking about before where we did we did two things we made it a tiered leaderboard so you actually have a beginner intermediate and advanced level and people are only competing score wise against people in that same category so you don't have a beginner going I am three hundredths on the list I'm never going to get up to where these guys are now they're just competing against for

the top positions for people that are also beginners it's also not only tier it's a top-5 and so people that feel like they just want that safe space to practice for a while can do that they know where they are on the list because it says at the top you know you are number whatever but only when they reach the top five in their category doesn't show up so they can get some recognition going hey I made the top five but they don't have the intimidation of hey I'm way below here and I can see how I'm doing and so can everybody else now as far as how this works inside I'll give you a few a few details of how that

would actually function we have it the game server setup which has its own database that keeps the game information and it hosts the challenge module so every challenge is written into its own separate module so that again we want to make it so that people can collaborate with us and say hey I thought I'd have idea for a puzzle all right write a little module for it we just plug it in there's also a web server that handles the the user experience for everything but the challenges what would typically happen is you come into web browser go to the challenge board pick a challenge there's also an authentication step that happens in the background this is as simple as

having the game maintain a list of users and passwords you'd register and then you could login or in our organization we tied it into our single sign-on service so that we can just go to the website and says hi Steve let's play a game but once you've done that then you go off and interact with that challenge module which is usually on its own TCP port like the example I showed with the game of life port it's like those so for instance to use the life as an example you'd select it on the board it would give you in a description of what to do and it tells you to start at port 1970 so you go off and do what we just saw

and as you do that you're interacting directly with that challenge module and it's giving you feedback along the feedback along the way say nope that's not quite right here's a new twist but eventually you hit on the answer and it says congratulations you finally got it and here's a flag and you then go back to the website with that flag and you post it and claim it now in a an event-based system it's pretty typical to make these flags some kind of static string but in a game that could last a long time and people can play it and then three years later one of their colleagues sitting next to them is getting frustrated I can't quite solve

it and have you know the person next you go oh yeah well the flag for that is Twilight Sparkle just type that in so we need to make it so that the they were dynamically generated Flags what we did is made them just a random 63 bit value which is pretty hard to guess usually when someone solves a challenge that's generated stored in a database so the system now knows to expect it to come in and if if a module can authenticate the person which a few of them can do then it can assign it directly to the person but normally it's the first person that goes and claims that can get it so a

64-bit number is a little inconvenient and we also for various reasons had to go with the design decision that says we can't assume the copy and paste will always work because there might even be on a different computer doing the challenges versus a tablet or something that could be interacting with the website on so thinking of all the different ways you could represent that I decided that the easiest thing if you wanted to have to be able to look at a flag and retype it was this is this is the system used by open BSD use s key system where to take 64 bit number and gives you a short phrase of words it's basically representing the number in

base 2048 so you go back to the website you type in the flag value and it says congratulations something else we decided to do to give a little bit of a recognition of people who can solve things quickly every challenge is worth a number of points this is worth 300 points but this is also worth an additional 300 bonus points if you can solve it within a particular length of time so as soon as you pull up the description that bonus point number goes to 300 and starts ticking down to 99 to 98 over the course of I think in this case a couple of hours and if you can solve it in within that time then you

get some bonus points too if not it's always worth at least the base point value we never wanted to make it so that if you weren't fast enough it wasn't worth doing anymore and when you go back to the challenge board the having solved the challenge the score you actually got including your bonus points is now shown so you can see at a glance where your score came from so a lot of this hinges around talking to these modules and here's where we hit another challenge we wanted to make them so you could plug them into the system we wanted to make it so that you didn't have to trust the code in them very much

and especially because again I don't get paid to do this and so I need to make sure that I can collaborate easily with other people we need to make it so that it's really easy for somebody to write a challenge module without having to know the CTF framework at all and so to add a module it shouldn't have to know how to generate flags or anything like that so very simply what what it is is when a player connects to that TCP port that host challenge the system will launch part of the CTF framework called the generic module launcher it launches the challenge module and arranges for the players connection to be on it's standard input and standard output so

from the point of view of the person writing the module for a challenge it's really easy you write a piece of code that just talks to the user in the normal way on standard and standard out and that's it this six line bash script is a perfectly valid challenge module not a very interesting one but it works it all it has to do is interact with the user when it exits back to the launcher the launcher says oh the module exited so I guess they didn't get the challenge right and it bothers or it goes through the trouble of telling the user that they didn't get it right sorry breaks the connection off if they solve

the challenge all the module has to do is just exit with a particular return value back to the caller that's all it needs to know it then says ah good it congratulate user generates a flag hands it to them and then they can go claim it the other thing is we also thought you know for a couple of different reasons we make sure that people can't connect to a port and then just sit there partly for reasons of keeping the challenge level right and partly for reasons of not wanting to tie up resources on system but again we don't want to make the module author have to deal with that stuff so the launcher will wait a

little bit and then start sending signals to the to the module code now if there's a particular module that knows it may be in for a long haul but it can judge whether the person's making adequate progress it can count those signals coming in or decide when to call it quits but if the module has absolutely no code to handle it whatsoever the default action is for it to simply die and so the module exits and again the launcher says oh I guess you didn't do it in time and it hangs up and that's all Thor really was - you know architectural II - making a module so I mean it made it really easy to plug

these in now in terms of running the game you do have to have a little bit of expectation setting for the players but it's surprising how far the golden rule will get you on this and our so our first rule was just don't be evil we did have to add a little bit of detail to that over time everybody always wants to say it's a hacking game so obviously if i hack into it to get a flag for free that proves I'm clever and I win right know me so we said that's AB that's explicitly not allowed and the whole point of this is not I'm not here talking about setting up a capture the

flag event at a hacker conference on an isolated network this is something I set up at work you might set up at school or in any kind of organization where you want your people in your organization who still want to have jobs later to interact with the system so if you launch real attack tools against our system we have a whole department of people with probably not enough very much of a sense of humor that will come talk to you I was just playing the game is probably not a defense that's gonna work so warning any of them of that first and the the most important rule was just relax don't get too stressed about it this is not you know an

assignment it's a fun activity you can learn from and so just have fun with it so having built all this up and run it through the reaction was again really pretty positive the I think one of the one of the things that the signals the success of a of an application is if it's put to some purpose that the programmer never thought of when they wrote it and this was the framework we used I mentioned the the in-class labs that we had for our app set class we ported all those in-person labs to this framework as modules in the capture of the fly game so through that we had three web applications that were written

to be intentionally vulnerable that people could attack and do things to now that presents an interesting challenge to for a persistent game people break stuff so they could part of those challenges this is actually one of the models from the CTF board but it's representative simple database lookup app that if you type in something that it was an expecting boom we could trace back that looks like a sequel injection my goodness it is vulnerable the sequel injections and the next thing people try is dropping the tables and whatever now when we were doing this in the live lab environment we had one of the teachers sit there babysitting the lab server because people were going to cross write

script each other and lock up browsers or trash the database we had to keep fixing it if we're gonna make this be something we can set and forget and let run for a couple of years that can't happen so there's a couple of approaches that we took for that and and they both worked in pretty much a way that's appropriate to each of that trying to think of a good way to put this but basically for each of the modules we use one of these two approaches one is to simulate a vulnerable app and one is to actually sandbox it the goal of that is to keep one player from interfering with someone else everybody when they come

into one of these apps should always see a pristine app that's working that doesn't know that it was just broken by somebody else playing the game if you broke it though you should come back and see the still broken version just like you had it so one way of handling that is to sandbox it so let's say you come into the app you get your own copy of the database your own infrastructure some might even build a virtual machine around it that's more than we did but I've seen it done successfully there's but to give you a virtualized view of the app that only you see and you can break it and everyone else has

their own virtual version the other tactic we took was simply to simulate the vulnerability the app wasn't vulnerable at all but it said hey you just posted this input to me hmm let's see that matches these five regular expressions that looks like a sequel injection yeah so congratulations you broke me and it gives you an appropriate result that a broken app would do and you get the credit and yet the app is still there running just fine for everybody else now I also want to give you a little view of how its put together behind the scenes to give an idea of how all the moving pieces fit together this is every box is a specific

part of the system that can be identified we have if you're coming through the web interface and that's to look at the leaderboard the challenge list or to interact with some of the backend services that make all that work you're coming through a particular a couple of interfaces or some static content but you come through the service module into the API this is the core as it's got developed the core of it settled into this one library that we linked a few of the core programs with and so there's a path where the web stuff can get through it when you go to the modules they have their own path by the so ultimately go through that one

point that arbitrates what you can do to get to the game system local administrators have their own stuff but again it all comes through the API library and what that allows us to do is to consolidate all the pieces that have game privileges that is can do anything to manipulate the game to generate flags or change scores that kind of thing not all none of the modules have that ability and very few of the web pieces do it's able to let us administrate this and keep it sane pretty well and there's only two pieces that have any OS privileges at all and these are wrappers that then launch all the other pieces so you have one that's set up for modules

and web interface access and the other is run through pseudo which allows us to use sudu configuration files to say which admin is allowed to do what commands and what way to the system and we're it that way and speaking of administration one of the things that was really important to us was to make it so that we could we could collaborate on that too so you don't have one person whose whole job is to make this thing keep working and so we wanted to make it so that adding a challenge that the game wasn't oh yeah you edit these five sequel tables by hand but made it so that anybody with that was trusted to do it could just do

it easily so they're all done with CLI commands which means it's scriptable so things like you'd get a report that says show me what the state of the the system is we have these challenges the scoring information how many times people have tried solving them you can get a list that shows you what flags have been issued which also tells you which of your challenges people are solving which ones they're not it's a good idea of maybe what things are working for people and even making changes is all through pretty simple CLI tools so you know adding a category to the game adding a challenge to a category adding a new player changing their score because

reasons cleaning up flags that were issued but never claimed by anybody stuff like that made it really easy to do and so what we ended up with was a persistence framework where people can add new challenges of their own the availability of it is suited to people's own schedule so no one's under pressure to have to get through at a particular speed as a variety of different levels for what people do for a living and where their acumen is is centered and a tiered scoring system so that they weren't didn't have that intimidation factor but still got recognition and above all we want to encourage people to think about new ways of solving problems or dealing with security issues you know

if you have somebody they spend all day writing policy say they don't do technical work and they go through a couple these exercises and it occurs to them you know what we should make sure our policy covers this this and this thing that I've never thought of before it gets wheels turning in people's heads that maybe hadn't hadn't occurred to them and so now they're more interested in things they're there doing things of more value to organization we solved the solution replay problem with random Flags we also want to make sure that administrating this was simple and automatable talked by the sandboxing of all the things though that we learn and did the most important one was this make

learning fun because it doesn't matter what you're doing but particularly in security or any really high technical topic it's really easy to be really boring and and have people just not really want to learn a new skill but if you make it a fun game suddenly I can take on that challenge I can do that we had one might have noticed on on the challenge board there was a 1,000 point crypto question and I sit next to somebody that was spending many lunch times trying to solve that and I heard the cries of anguish as she was doing this and then one day yes like okay I think she solved that one and just just

so probably if it weren't in a game format might have gone it's not worth it it's not part of my job but she you know persevered and they got it and it was like that was really cool I'm gonna go talk five more people into doing that because I went through all that pain they can do it too but the satisfaction really helped her of it so that's my story and I'm sticking with that question

so multiple choice versus a set answer we've done both and we found the multiple choice one worked really well for the very top hello world kind of things but as we got on we found more that that getting them to come up with an answer we've actually found most of them were more dynamic where we're having you use a tool or write a script that would interact with something and actually diagnose it down to a specific answer you could then type in but would make sense once you'd gone through that process

ooh very good question we kept the for the bonus points we kept track of time absolute wall clock time no matter how many times they connected so if it was two hours they could you know connect forty seven times and get cutoff for time or whatever and that wouldn't that would still be accounted for um so my company has a process for releasing code as open source I'm going through that so hopefully event you know soon I'll tweet something out when when it's available so cuz I'd love for people to play with it and do something better with it than I did all right thanks very much [Applause]

you