A Winning Competition

The track name for this is the round truth. Um, this talk is once again a winning competition and it's going to be given by my dear friend Wasabi and uh, so I just have a few announcements that I just need to go over really quickly. Uh, number one, shout out to our sponsors. We'd like to thank especially our diamond sponsors Adobe and Aikido and gold sponsors [laughter] um Run Zero Drop Zone. Um [clears throat] it's their support along with other sponsors, donors, volunteers like yours truly that make this event possible. Um so this is a talk about cell phones. These talks are being streamed live. Um, and as a courtesy to our speakers and audience, we ask that

you that you check to make sure your cell phones are on silent. I don't want to give anybody a dirty look in the meantime. So, I'm not trying to be your mom. I promise. [gasps] And, uh, just to let you know as well, if you have a question, use the audience. Um, we're going to do this at the end, just a question and answer. So, um, if you have any questions, you know, feel free to ask. Wasabi. I'm sure you you'll be cool with that. But at the end I'm gonna go around and with a microphone so you don't have to get up. So um so far as last thing as a reminder uh besides um Las Vegas photo policy

prohibits taking pictures without explicit permission of everyone in frame. Um these talks are all being recorded and will be available on YouTube in the future and I just wanted to let you guys know that as well. So, just um without further ado, uh let's go ahead and get started. Please welcome Wasabi. >> Thank you. [applause] I'm going to turn that off because I have a a attached microphone today which may or may not be working. Okay, there we go. So, thank you for coming to a winning competition. I um did this uh talk as an effort to just challenge myself. I normally talk too much. So instead of getting too much talking this time, you're going to be

getting too much text. So please ask questions as you get it. Um this talk is mostly about how to improve competitions. I myself and a lot of the audience here are volunteers for different competitions and the this talk is a speedrun on how you take that competition from something that people's just participate in to something that people really like. And uh there's a lot of different components of this and I'm only covering one area. So please afterward if you have questions please actually talk to the audience because a lot of fellow peers and co-volunteers are are here today. So with that I am Wasabi. I am a security researcher, cloud engineer and competition creator. Uh I decided to do

this as a challenge to myself. So I'm sorry in advance as there's too much text. Um, people may be wondering, what types of competitions are you going to be covering? Well, there's a lot and we're only going to be doing roughly three, but when people think of competitions, they probably are thinking of CTFs, capture the flag competitions. Those are great. Those are really cool. There's a bunch going on right now, but they're a very specific type of content. And that content makes things very limiting to a larger scope. For example, if you have a capture the flag competition, you're not going to see people trying to actively defend a system. That's just not the way CTFs

work. And so then you also have attacking defense competitions like CCDC and things like that where you are doing attack or sorry, you're doing defense and sometimes attacking as well where it becomes a purple team competition. And then there's of course red team and forensics competitions like CPTC and circus. These are a lot of fun because they are a very different type of competition. In the case of circus, it took us almost four years to get the formula. I don't know that we got it 100% right yet, but we've turned it into something where students actually perform the work, make a report, and they're graded on their work, their report, what they turn in, and instead

of being live and dynamic, they can do research. They can adjust what they found instead of the heat of the moment. But overall, that's not the point of this talk. What we are talking about today is we want I want to share how competitions can get made improved and what goes well and terribly and as you grow what those things do to the competition. So what's the big deal about competitions? Well um interestingly it got cut off. That's interesting. Um anyway, uh the uh so I'll just add that part, but um why do competitions? Well, simply put, the long story short of it is that a competition is meant to simulate a large variety of real world

in something that is easily digestible and performable. So for example, um you can build new themes as things change. For example, social being a social media network, being election systems, being a bank with ATMs, industrial infrastructure for uh you know baking and cooking. Um and then simulating the whole business infrastructure to that simulated business so that everything functions. So is there HR? Why is HR doing what they're doing? Is HR making things harder for your um your people? Is there a corporate espionage going on? These are things that you can put into scenarios when you build competitions that are just not available if someone just read a book or I mean I guess if they ask chat GPT there's you know

there's there's a ton of stuff that just doesn't get seen. It also gives students a chance to see the emerging technologies that they don't normally see in their courses. Uh academic competitions are force students to deal with new things in the past couple years. One of the big ones is LLMs. We've put them in, we've used them in variety of different ways. And then also the other thing that students don't ever get to see real uh is industrial control systems. And I can't say competitions are real because they're not. But it gives them a simulation that actually lets them use and challenge themselves. Are the protocols dynamic and uh proprietary? Yes. So, do they have to

figure out how this thing works live? Yes. And that's where competitions come in. You don't get to see that. I can talk to you for hours about Modbus and it really won't matter because you're never going to see that until you actually have to get your hands on it. So, uh, it also builds teamwork. And teamwork is something that competition teams are really, really good at. There are teams playing right now in CTFs and pros versus Joe's that may have never met before, but now they're a team. And when they finish that competition, they're going to still be a team afterward. Well, unless there's some dramatic drama that goes on, but overall they're going to be a team and they're

going to talk to each other after the event. This networking builds and shares the technical skills as well as all of that communication where people become a network. And it also forces students finally to just work in a high stress environment. Whether it's any type of competition, it's really stressful. And uh when you have something high stress, it lets students feel that stress in a real way. And the that's what I love about competitions the most is that real stress. I mean, nobody else likes it. When I when I was a competitor, I got super stressed myself, but then when I got to working, I wasn't as stressed about those scary situations because I was prepared. And

students come back and tell me, "Oh yeah, that situation where we had to do all these incident reports, well, we did it. You know, it wasn't bad when we we worked in it. It was super calm. I did an interview and they said, "How would you deal with this?" and it was oh yeah just I repeated the stuff we did for CCDC and so those things are where competitions really win and also competition alumni are everywhere I think over half this room is um former or current competitors so it's it's really good to see I mean it may be a bit of a niche topic since there's so many of our lap there but it's a really cool environment and so

you may [clears throat] be wondering what about that AI thing and I it's it's cut off but I can summarize AI is really helpful. AI is extremely powerful. Students are able to utilize it in in most of the competitions I help organize but they it cannot solve for things that you do not know how to ask for. So as students work on things and they try to do things we've seen them write reports where AI wrote the report and it did not understand the context that the report was being written. though it failed and they of course failed the task or they're trying to fix systems or solve challenges that we've given them and they don't understand the context. So AI

helps them and solves that but they uh they just didn't get it. So it it it it helps but you still have to have that hands-on experience to ask the right questions. So um I've already been given the notice that I'm over halfway through and uh that's exciting. My clock is 7 minutes in. though. But uh >> sorry, [gasps] >> it's okay. I I I did this as a challenge myself, so I'm sorry for all of you. Um but about the CCDC thing, um I I really wanted to just do a a background of what CCDC is. Basically, it started way back in 2005. The region I'm a part of, Western region, started in 2008.

It is congressionally recognized as a framework for building out cyber security talent to meet demand. Um, and I know I say that almost ironically because there's not a lot of job demand right now for cyber security in some cases, but um, it is a pipeline into the industry. Um, students are in teams of 8 to 12 and they are attacked while having to deal with all the stuff that we throw at them. And so it's a lot of fun. Last season we were the Federal Bureau of Control. It was a um SCP like secure, contain, protect aliens um scenario. And so we had all these signal generators, jammers and things that they were having to maintain. It was all simulated, but

they had to understand these systems and how they were intercorrected, interconnected. They had classified systems and they had to do data classification and these were things that students had never seen before. So it was a lot of fun and teams learned a lot. So then there's the circus thing. This is new. It's one of the newest cyber competitions that I'm aware of. Um, this was mostly put together by um, a trio of us. We've been wanting to put this on forever. Um, this was made out of Coastline Community College and it originally started for community colleges, but we're expanding it. And basically we give students a company that's been hacked and the red team scenario is the

red teamer breaks in or the nation state breaks in and we actually have either a real red teamer going through the network and then we capture the artifacts. Sometimes good where we give them full disk images, sometimes really bad where we just copy files from their desktop and say here's your artifacts and they have to build a forensics case and find out what happened. Um, and then they have to present it and they have to prove their uh chain of evidence and the handling. So, this was really cool and I have to give a um shout out to uh Dr. Brown who's in the audience and he and really gave this the um academic spin that was needed to

allow students to actually have this and make it into something that can be like courses almost. It made the competition a lot of fun and again we're expanding it because there was so much demand. So how do you design a competition? Um well there's a lot of things. Uh we'll go back to little story mode but originally CCDC was the same schools. You had eight schools. They always were the same schools. You pretty much knew which school was going to win one or two schools and it was very repetitive. It wasn't exciting. Um, but that gets boring because you you're doing the same type of stuff. But even with that, you had all these things. You had to bring

laptops in. You had to bring equipment. But that that keeps things very static cuz where do you live offseason? And that's the biggest uh point when you're building a competition is where do you live? There's a lot of logistics that go into running a company. They're not as many for running a competition, but you're still an organization. How do people log in? How do people see information about the events? How do people do anything to log in? Do they need a VPN? Do they need credentials? How do those credentials work? What does it give them access to? These are just things that you're probably like, "Oh, yeah, that makes sense." But you have to think about those. And if you're

constantly tearing down your environment and bring them back up, you don't get very good resources because you're constantly just trying to keep the things working that are working. And that's also a cost. If you want to keep stuff up 24/7, you have a couple of options. You either have to have stuff hosted in a data center or you have to have a VPS VPS or some combination of all of that. And then when you get into in person, what about logistics? Where do you host your volunteers? Where do you host your competitors? What do you feed them? Do you feed them anything? Maybe not. Do you have internet? How much does internet cost the room? I know

probably here in Bides, they spent a long time on that question because that question is never simple when you go to venues. Um, and then how long is the competition? Because that increased costs. How do you store stuff? What does the additional gain days gain versus lose? So, if you have a 5-day competition, what does that benefit? Or if it's a one-day competition, what do you lose by having it all packed into one day? Um, and what most of all asking all these questions, what do students get out of it? If they have to deal with the all the logistics and all of the processes that you're trying to get working and they can't actually use it,

it's not great. However, if they have a lot of opportunity to learn and gain and have workshops, is it memorable? Do they learn something? A lot of what competitions come out to after the fact is the memories. It's, you know, you come back and you're like, "Oh, you remember that time when I was in and we were competing and we did this thing and we found that sol that solution." That's what people will come back to. So the final thing is when you're making a competition, you're trying to build one out, you have to ask the question for everyone, what will bring me back for next year, both as volunteers and students. So competitors are sometimes organizers,

but organizers are not competitors. And I I know that's literally written there and I'm not trying to read from my slides directly, but this is a very true exact statement because you don't have overlap or at least shouldn't because a competitor can play in your competition but and then they can become volunteers but you really shouldn't have a a volunteer also be a competitor because you get this weird state where do I know too much about the competition to be able to play fairly? Um, and then also you get ideas because as a competitor you start thinking like how would I play this? But as an organizer you're disconnected from that and you are asking sometimes the wrong

questions because what's funny to you and what's fun and makes it enjoyable for you that keeps you motivated may not be the same thing. So suddenly you're making things too hard and too niche essentially for the main audience. Um, we also one thing that we learned that was very popular was sharing resources. We were the first uh group uh competition group to share all the resources out at this point. Um I'm pretty sure we are the largest uh we have something like 10 to 12 terabytes of previous competitions, PECAPS. They've been used by researchers um PhD students and all sorts of things. They've been turned into workshops and training material and that training material we bring back and share with um

the students again. The final thing on this is though um if the word gets out that your competition is too hard, you get really dedicated players, but really dedicated players are not the majority of what keeps the competition going because remember that cost balance thing. What we've learned is if you make something really hard, you get good players and you can't keep it. There's no pipeline. So, you have to balance it. You have to create tiers. Something that we're doing this uh past year is um trying to make it so that um we have something called invitationals which are like pickup games. We've always had them but we have more now. So the pipeline is in to get students

interested and then they can go into the competition. Any competition can have this and I really encourage it because otherwise if you make things too difficult you just lose all those things. And then um one other thing about the realism and asking the right questions. Sometimes making the competition, this is especially true for CTFs. You make a challenge that exploits something really interesting, but you miss how it relates to the real world. And that's fine for CTFs. It's really good for CTFs cuz you're going to have those niche skills, but as a broader scope, not as it's not approachable and it doesn't always apply directly to finding a new job. Um I have only a few minutes left but um

this is something to think about with resources when I first start uh and that's yes that is smoke um uh but um uh when I first started taking over for WRCCDC um the we had an we had to move our resources from a data center at hosted at uh the university was hosted out of into um a dedicated hosting provider that worked. But the first competition we ran, we got emergency maintenance on our systems. So I was sitting there my first time running an event and I got an email saying we are shutting down your servers immediately and then everything went down. There was no that was all of our resources. That was all of our ops. Everything was gone.

So the question that comes in is what type of equipment do you do? And I'm not going to repeat all of this, but the the summary of this is how do you handle those logistics? What happens when systems go down? Do you have backups? You don't have to have mission critical level of stability, but you do need some plan, especially when you do loaners or rentals like that, you have to have that plan in place for what happens if 50 of your 100 systems are broken. What are the resources? Are they too old? What do they run on them? Uh fun thing with loaners is they'll sometimes have operating systems still installed and so you can't actually get

like they'll be locked down in like uh MDM mode and managed devices and then you can't actually unlock them. So you have a just basically a paper weight and you've just paid for that paper weight. So one thing about for competitors and students uh students [laughter] generally uh winning teams generally specialize whether it's CTFs, CCDC, CPTC, they develop their own process for making the competition work. Um, one really cool example was Blue Spawn where they built their own EDR as students and it worked really well and it caused the red team attacking team a lot of challenges and teams will practice more that are good but it doesn't mean perfect. So they will constantly rotate

out cuz it's four years usually, right? So they will constantly have new teams and then the skills change and so they're constantly having to redevelop. If you are a a competition team, the one thing I would suggest for you is to prepare notes and take notes. Even if it's 10-year-old notes, you want a tree of all the effort that you put in so that the new people coming in can work off where you went. Um, final thing, um, uh, the some of these changes that I've been talking about. When we first started this, CCDC never had a winning team at nationals. Um the highest we got was in 2011 we had third place. After all of these changes, and this took many

years, we started making it to um nationals. And not only did we make it to nationals, we've had two winning teams, two winning teams at once um four times in the top three in the last 5 years. And so we went from being the school the team that was, you know, the never actually won to winning. And it isn't to say that we as organizers did all that. That is most of the students. But we made the changes to make it easier for students to succeed. And as organizers hopefully watching this talk, make those changes so your students can succeed too because that will really bring the make the difference between teams. Um, finally, here's our uh

winning team for this year. I just wanted to show them off cuz that was pretty exciting. Um, but with that I am at time. >> Yeah. So, uh, with that I hope do I have a couple minutes to get questions if there are any. Okay. >> Any question? Oh, that's easy. You want microphone? >> I want the microphone so that the video can come back. >> Yes, sir. >> Thanks. I appreciate the talk. Um, you mentioned discouragement kills competitions. You had the vi picture of students in hazmat suits. What have you learned about discouragement in western region CCDC over the years? >> Uh those are not students. That was the scenario. Uh there was a containment

breach in the scenario. So we all went in in full hazmat suits and we removed equipment as a but for discouragement. Some of the things that really discouraged them uh were things like that we didn't even think were big deal like putting like mines in like bash shell like when they did ls it makes the slow locomotive go across the screen just these little frustrating things that we didn't think were a big deal that really discouraged teams or not getting feedback that's a big one as well if teams don't get feedback they don't feel like they're doing anything cuz they don't know what they don't know and so they just give up and so that is the biggest one I is if I

I were to think about it, it's really getting good feedback was the biggest discouragement if well not getting feedback is the biggest discouragement. Sorry. >> Any other questions? >> Or is there another >> I don't see any other hands, so feel free. >> So, you didn't talk much about scoring. Um, what makes for a good scoring system, good scoring architecture? How do you balance it? How do you keep the game competitive but not so close that you don't have a winner? >> I honestly probably would hand the microphone behind you and answer cuz this is this is Dr. Brown and >> hi, I'm Pro aka Dr. Brown. >> He has helped so much with scoring. I

just make the pages technical. >> I I have to give a shout out to my chief judge in WRCC DC Alchemy. uh because between he and I and Joe and um our red team leads um uh they they have helped try to level the playing field so to speak. Um students will try stupid stuff and we will catch them and we will tell them don't do that. And it's kind of like um you know college teams that cheat when we catch them then we have to penalize them. I've penalized teams and said hey okay great you did stupid stuff see you in 3 years um on the scoring engine side for their services. I wish basic was

here because he's well more qualified to answer that question. >> But we've gone through three revisions. Yeah probably give me a bit. Yeah, >> I I would boil it down to honesty, transparency, and taking accountability, right? You will >> you will have issues during your competition. You will have a time that a um sponsor goes in and trips over the power cord and takes down a team. That happened to me and Joe in nationals in what 2012. >> Yeah. >> Um >> unforgettable. >> You have to figure out how to make that fair. You have to figure out how you're being honest with the teams on what you're doing. The most frustrating thing for me as a competitor was how is how is

my score calculated? What's going on here? And how can I learn more? And you you have to figure out how to be transparent. >> Yep. >> Well, thank you. Well said. And I I handed it over because it is a team effort. There's literally no way to make all of these scoring things one person. So, I know I'm at time. I will be wandering over there for a few minutes afterward. But thank you all and I hope you got something out of this um and learned some cool new things. So with that, thank you and I will >> [applause]