BSides Portland 101

hello hello welcome to welcome to besides there would be an introduction to the speaker but we screwed around and ate up that time so we're gonna introduce ourselves I'm Michael Lewis I'm Ricky oh this is Topher and then Joe is mint mentioned there but didn't appear that's fine we'll just continue without so firstly we we have badges oh my god they [ __ ] work that's amazing I really didn't think that was gonna happen you can direct most of your complaints to me so what is on on the badge so this is a on the back if for those of you who have a badge and they're playing along at home on the back is a NRF 52 a 32 module it's

arigato BMD 300 these are really neat these are really neat tiny little ble modules that they have a cortex aren't armed for their cortex m4 inside and they're really low-power really neat and they're so small that you can actually embed them inside of an RSA SecurID token there's a that's a different talk there's a neat little LED screen d-pads that most we probably work you know they all flash but QA we didn't get the chance to actually to do QA so the buttons may be work and you can like flip them through into a little see if my button works maybe you can flip them through into a cute little like sasquatch hunts cameras games it's got

two two oxen or an oxen docking connector and a wagon hitch and a power switch that is a little bit flaky so if you flick it on and flick it off and it's still on move your battery around a little bit or accept the fact that it's always on so how did this happen so if you went to Vegas and you were part of the DC 503 party then this is sort of familiar to you this looks very similar to the bet that Joe primarily designed for the 503 Trail which was amazing and we had this great idea that we were going to just recycle that design and then this wouldn't be such an epic

[ __ ] like it always is and then we kind of had our Vegas hangover then we said well you know wouldn't it be great if we had these like actually produced because it's really hard to solder these by hand and we knew that we were gonna have a lot of people so we really need to make this you know to do design for manufacturer right to design this to be producible by machine so the DC 5 or 3-bedroom made a hundred of them and we had the really tiny parts professionally assembled and then the other parts we drank a lot of caffeinated beverages and hung out of control H and just soldered them until

we couldn't see straight and we knew that we have way too many we have way too many people to do that and that we really would benefit from having professional assembly so that took a little longer than I thought it would and then there was a there was a lot of chaos panic and ultimately despair I won't go into the chaos and panic but you can be sure that you know most of it was probably my fault and it actually all came together on Wednesday I had sent the email to the assembly house saying we have to come to reality here and that you know this just isn't gonna happen and we this is all my fault and I

accept this and you know we'll we'll try to get badges out to people you know maybe a month after a con and they sent me back email saying we're rolling we're gonna make deadline so an emotional rollercoaster and then they called me a little bit later and said hey you know the battery is backwards and you can't insert or remove the battery without ripping components off the board and and I asked them well you know how many have you assembled where we've identified this problem and and they wrote back 300 and they said well you know I I think we can I think we can hand rework all these it'll it'll it'll just it'll just be a little bit of

money and and overtime and and I I was unsure but Dean said yeah let's do it so unbelievably amongst overcoming all obstacles they assembled it all we picked it up on West on Thursday afternoon we assembled it all at control you know got the wheels on the batteries programming a controller tested them and oh my [ __ ] god they work and we had a hundred percent yield this has never ever happened to me before if someone told me that they did their badges and they got a hundred percent yield I would think that they like really planned everything really well and everything went according to plan and I know that that's totally not true in this case

lots and lots of people lots and lots of people helped do this and and they're way too many to think there was an entire crew of people there was entire crew people who truly test night working feverishly hard and it's amazing to see everyone come together and I'm tremendously thankful for everyone to help make this happen I mostly won't I most of all want to I want to thank our sponsors so ash Park donated the PCBs which as per usual are perfect purple PCBs Ruggero donated the radial modules and cascade systems heroically assembled this and donates the assembly and I really just again want to say thank you to everyone oh and there's some sort

of hacking you can do with the badge I think there is a badge hacking contest you should hack your badge and there will be some sort of judging ceremony later to be announced because I didn't plan ahead and with that so also like Joe finally came because the speaker's wrangled the guy that's wrangling speakers so yay Joe I was gonna say something I was gonna be relevant but I don't know what it was so yeah it's been pretty cool to have badges every year so ash Park has donated the PCBs that's four years now this is the first year we had ragazzo support us by contributing the little modules those tiny little modules in the back and CST did the

assembly so it's really great to get all these different sources helping us to make this happen they're all nice local companies that we really like and it's also really nice that Mike keeps doing it because every single year he says he's not allowed to do badge again and he does and I don't actually you know he's not allowed to bad again let's take let's revisit this in a few months maybe he'll forget or maybe everyone also forgets but so yeah that's it or maybe there's someone in the audience who's designed a really awesome badge in the past year and would like to you know volunteer to be the new badge person and hint so yeah thank you

and thanks Mike and here we go Topher speaking of doing stuff for some reason Joe keeps asking me to do things and I say yes without actually knowing what the role entails one of which was leading all of the events and contests and building the CTF again this year so I this is besides it's a security conference so a big portion of what people come for all that are the talks but for people that you know don't want to go to every single talk and every single track there's typically you know other events and happenings going on so you can interact with other people or you know try out your hacking skills or whatever you want to participate in so

this year we actually have a really awesome contest and event set up and is you know is common in a lot of hacker conferences these days we have a really awesome CTF which we'll talk about so as far as the contests and events that are going to be happening for the duration of 'besides we have info set quiz show so this has been going on for a couple of years now last year was a huge success how many people actually participated or watch this last year yeah a couple of people great so Steve's doing a really great job with this and pretty much what it is is you've all seen this show Jeopardy so think of this as hacker jeopardy it's a

series of questions and and whatnot to the to the competitors but they all have you know some sort of computer or security theme so some of the questions are you know what's the best text editor and the only answer of course is why so so it's fun it's you know it's got some comedy aspect to it and it's great there aren't that many people currently signed up to be actual competitors for it so I encourage you to go to the b-sides website and sign up to compete it's a lot of fun and there's really awesome prizes that will be delivered during the closing ceremony when is what oh the quiz show yeah so the quiz show is

happening tonight at 5:00 it's like 5:15 to 6:00 or whenever it ends at UM it's on the schedule as well there's also this thing called Carmageddon that Dean's been running for the last two years how many people like reddit here how many people like shitposting you should play this game so the intention of this is essentially shitpost as much as you can on reddit and collect all of the karma and the person with the most karma wins and is the you know the best [ __ ] poster it besides PDX I'm looking I'm looking at one person in particular that should do pretty well at this I'm saying a sign up there is also prizes for this and

then coming back this year and this is an event that hasn't really been widely publicized so it hasn't really taken off but we as the organizers feel that there is merit to it is this notion of a technical book swap so like you know like all of you I assume I have books on my shelf that I've read that I will never pick up again because I've either gone through the material or it's no longer relevant for what I care about and it would be cool from a community standpoint to be able to give that away to people or exchange it for another book that I want so this this you know informal event in the event room there's

a table next to all of the lanyards which is also a cool thing that's like looking for work and looking to hire particularly for Portland hires so it's you know community driven go go through your resume on there get hired but also at the same time there's the book swap happening on top of it to learn more technical skills or to give those skills to the community so I encourage you and I'm it's likely that you didn't know about this today but if you know going home tonight you look at your shelf and there's some books that you'd want to contribute to this event I encourage you to do so and put them on that event

table um I don't know maybe maybe don't have stuff about like you know X I net D can be whatever if somebody doesn't take it it's you know take it back whatever that sounds fun do you have books should bring those books in and see if somebody wants to learn that crap also for the first time this year we have this thing called engineering stories so an Meixner is going to be around Friday today from 1 to 5 and Saturday from 1 to 4 and essentially what she's doing is she's taking peoples engineering stories how they built something what they're trying to build and curating it on her blog Engineering's daughter with the intention of this being you know

community involvement and community feedback on solving actual engineering problems or the struggles that one faces while trying to solve those problems she's trying to you know get get questions curated to pertain to certain communities one of which of course is security since she's donating her time for b-sides and we'll take your stories and post them on her blog and in such a way that you know reflects the struggles or the challenges that you've overcame in some of these engineering tasks all of that information is going to be in the event room and on the website and there will be a sign-up sheet so she's going to be doing 20 minutes lots of interviews and we'll take you know a set

amount of stories and actually post those on the blog and we'll get the access tomb as well and we'll put them on the b-sides website so this is something that we've never tried before it has promise but it's going to take people to actually go out and give those interviews and see how it goes with your story that you have to tell because all of us have something to say there's also coming back again this year lock-picking village so this is the thing that it wasn't necessarily known if it was going to happen and then Kenny miraculously sent out to the mailing list I think Thursday afternoon that hey I'm gonna be at b-sides so this has been

happening for you know several years and it the majority of hacker conferences there is you know a lockpick village happening um so this is really really excellent to have and kudos to Kenny in the tool chapter of Portland for putting this on again this year so those are the majority of the contests and events happening in the room and now really and this is biased for me as somebody that was a part of the organization of the CTF and somebody that is really into hands-on skills and I feel that CTF is a really excellent learning platform both to practice offensive security skills and defensive security skills which I'll talk about in my talk tomorrow but if

you're not familiar capture-the-flag is essentially a computer security competition or is if you watch mr. robot how he put it hacker jeopardy or or the hacker Olympic so it's essentially you'll you'll be presented with you know a series of challenges whether their web binary forensics reverse engineering and the intent is to solve the challenge and such the in such a way that at the end you get a flag out and then you cat that flag to a scoreboard and you get points and the person with the most points is you know they win the CTF and they're that they won the hacker Olympics the unfortunate part about CTF is at a lot of conferences especially save bigger

ones like Def Con for example the CTF s have a really really difficult barrier to entry and people feel discouraged from participating because they don't think that they have the skills in order to solve even the you know the basic 100 level challenges because at these conferences they're not basic the whole idea behind b-sides which was reflected by Joe during the opening ceremony is this as an event for people that are new they're incoming it gives them an opportunity to voice their ideas and get started so we kind of take that notion and put it into the CTF so these challenges are you know amateur to intermediate there's nothing that should be that tricky or surprising to somebody

that's new and I say this not as you know if you're new and you can't solve these I'm not saying that's bad I'm saying it's a way that you know we have things like buffer overflows that have been existing since 1983 or whatever and they're not things that are meant to discourage you from playing and it's a really good way to gain skills that can take you further in your exploitation career so this year we developed 16 challenges across four domains so it's a four by four scoreboard we have web binary exploitation reverse engineering and shellcode challenges speaking of you know useless books that you might not want to read does anybody know how to

how to write MIPS you should play CTF because one of the shell coding challenges is MIPS you're welcome

so one of the hardest parts about CTF isn't necessarily the challenge writing it's how you're going to host the challenges so in the past at some of the CTF events that have been ran at besides one year in 2015 we actually had you know a dedicated server sitting in the event room it was all a local subnet and you just connected to the server with you know the local network and there was no internet access to it which is cool but then you know you can't go on Google and try to get help from things and read you know the manual for poem tools or whatever so last year what we wound up doing is Jesse and I at our old

organization wrote a series of challenges for the CW top 25 which are the top 25 most pertinent vulnerabilities across the software domain including you know binary and web and we took those and we're like okay let's make the b-sides PDX CTF out of this because this is very much driven to the this is intermediate and amateur and most people from reading these articles about these vulnerabilities that have been existing for years can solve them but then it became the question of how do we actually host this infrastructure in such a way that it's it's secure from when somebody gets you know root shell in a box or a shell on a box they're limited to their executives and they

don't tear down the infrastructure or we're giving them something unintended so I reached out to my friend Andrew who at the time was building a platform and they agreed to let us use it and sponsor the event so we wound up using if you played CTF last year it was this really awesome platform from Symantec where it was fully isolated every player was competing in their own you know little sandbox and it worked out really really well so going into this year I you know DEFCON time knew I was gonna run the CTF had no idea what the infrastructure was gonna look like and of all things we're sitting at the Flamingo in the Cabana

that Mozilla was hosting and you know I'm talking about how last year's CTF was a huge success and I didn't know what to do this year but besides San Francisco did a really cool thing where kubernetes implementation and ran their whole entire challenge in kubernetes and the guys sitting next to me looked over I'm the one that wrote that so I'm like oh wait what this is amazing we should connect and talk and he's like yeah you know I kind of copy it from this semantic platform and Andrea is like I wrote that so it became this whole little circle of you know it's a small world and people that are designing and hosting CTF somehow all wound up at the

same Cabana at the Flamingo so we all coordinated with each other and wound up with this really awesome infrastructure platform this year it's all deployed in AWS in Mozilla's instances actually and then being one of our platinum sponsors is totally wicked that we're able to host in them and it's it's kubernetes everything is built with docker which we'll get into you know don't don't fret it's actually not that bad how we're doing it because you know docker security oh my god don't do that so yeah I'm gonna invite Youssef up on stage from Mozilla I mean he's actually visiting from out of the country they flew him here from London just to give just to build our infrastructure for us

and to give this little section of the talk so welcome Youssef hi everybody I only have a few minutes so this is going to be a very very very quick run-through of a lot of complex things so take this more as a go forth and look at these after after this talk but so yeah running a CTF inside docker doesn't sound like a great idea so these are some of the things that we've worked on too so sort of harden up the kubernetes cluster in case the docker security fails so one of the first things and sort of the key the key security mechanism right now for kubernetes is robeast access control and so essentially that's what it says on

the tin it's you define roles and you combined it to various resources inside of kubernetes and it allows controls the controls access to the API and so by by default in earlier versions of kubernetes role-based access control isn't enabled by default so anything running inside the cursor has completely administrator access to the API and also by default in kubernetes is anytime a pod so a container a set of containers is deployed in kubernetes a service account token is mounted inside the pod that gives access to the API so by default every container inside the curse that has cluster level cluster level administrative privileges which isn't great so from Cuban se is 1.6 and a later it's enabled by default but a lot

of people end up disabling it because there are some services that don't support car back but like helm for instance the package manager or they just created their cluster before it was enabled by default so if there's anything to take from take from my section it's just enable are back on your cluster or create it when you're creating your cursor the next big thing I'm going to talk about is container networking and the new network policy resource in in kubernetes by default kubernetes uses cube net as its container networking overlay which is sort of really simple it does the basics there is no sort of cross node networking or network policy implementation so for any production

kubernetes cluster I just recommend using one of the other networking plugins like calico weave that kind of thing if you are using calico on AWS just remember to change the disable source destination checks otherwise nothing will work and you'll spend a half a day try figure out why yeah and so another thing the besides infrastructure was created using K ops which made this thing super easy so as you can see there there's a simple four to just enable character networking you don't have to worry about anything inside the curse the same with our back there's authentication equals our back super easy to do with K ops we love it happens illa so and it covers a lot of

use cases for most people who want to use kubernetes and so yeah cocoa enable implements the network policy and so by default there are no network policies defines that all traffic can run inside of the cluster which doesn't really help us much so this is sort of a sample configuration Network policy that will just deny all ingress traffic to all pods and so that gives you a good starting point from there you can whitelist whitelist the networking you want to do so this is on the left here a super simple nginx pod that exposes port 80 and on the right here we have a network policy that simply put allows all traffic into port AC on that on

their nginx pod and so when you get in you can do more complex things with this as well so for instance here we'd have a a memcache D pod and this network policy will allow it to allow the web pods to communicate with that so instantly right there we're separating out isolating each of these services from each other so in case one gets pwned people won't be able to move across the applications inside the core so some other things that I'm going to briefly touch on not go into too much detail on these is the pod security policy and mission controller this lets you control what kind of dock and containers can be run the main use case for that is to

disable host networking if if somebody is able to deploy pods inside of the cluster with hosts networking completely bypasses all of that network policy stuff so that all becomes a moot point the node restriction admissions controller so one thing in coop Nettie's is that cubits the nodes that run the containers by default have access to all the sort of the secrets every service in the cursor needs to run the node restriction emission controller makes it so that only the the queue only the secrets that the cube Allah is actually going to run it has access to so it can see everything inside of the cluster so it's at least privileged using certificate authentication for cubits and Etsy D you can enable our

back but it doesn't really help much if people can just write and read the state of Etsy D directory you can just bypassed all of that role based access and specifically for AWS your procure your containers are probably going to interact with the other AWS services and the buy in AWS you can only assign I am permissions to an AWS instance not by container and the metadata proxy is able to sort of transparently assume different AWS roles for different containers so you don't have to give a broad set of AWS permissions to all of the services running in the kubernetes cluster so yeah I know that was a really big information dump but I'd recommend

going off searching all of this stuff if you're running or want to run a kubernetes cluster then you have come talk to me if you have questions about this I believe now I'm going to hand over to Andrew who's going to talk about the fun prizes you can get if you manage to rig the cluster thank you [Applause] so there's a kind of one more thing about the CTF that we were able to add at the last minute so mo Zilla we have this checklist of security principles before we put something into our production environment there's 12 and 4 categories I think and once one of those things reaches that level of maturity we put in what's called our core bug bounty

program who knows what a bug bounty is who likes cash so we've decided for b-sides since this is our reference implementation of kubernetes running these challenges in what we believe is our most secure configuration that we will consider this cluster in scope for our core bug bounty program so if you can pivot out of one of the b-sides challenges and you can hack the cluster itself please find me and I will connect you with our bug bounty people and you will get a substantial pile of cash so you have a my boss over there Jeff Reiner to thank for putting this in scope for the bug bounty program so thanks Jeff and I'm so glad that we

could be here and do the infrastructure for this event and I mean as far as CTFs are concerned right I mean you're giving people intentionally vulnerable challenges and in some cases especially for Epona Buhl's category of challenges the whole intent is to cap that flag when you're running commands as you know a user on a container so there's been a lot of really fascinating talks about kubernetes security particularly stuff from Dino capsulate he did a talk recently at one of his security meetup groups in New York City where he showed in about eight minutes how if you have code exec on a on a kubernetes pod like a docker container basically in five commands you own the whole entire

environment so the whole notion of this particularly around the bug bounty is that should not be possible in this and if you get code exec and one of the challenges which is intentional you shouldn't be able to do anything more than that so we're really stuck tobacco structure this year we believe it's a secures can be for a CTF where you know you don't necessarily have to be a good developer to write CTF challenges because the whole intent is they are hackable so we'll see how it goes with that said though there are a lot of things as a challenge organizer that we could have done a lot better so one thing in particular is you'll notice

and I'll mention this in a little bit but we're going to open source all of the challenges including the infrastructure code and you'll notice that some of the code is you know slop together which you know whatever like I said you don't have to be a good developer in order to write CTF which is great but one of the things we're doing is like using X I net D which of course we all know is old and deprecated because system D ate everything but at the same time but it's system D so we didn't use that this year but we can in the future and you know if you look at house ETF's are kind of modeled a lot of

challenges that people write will be modeled after something else that they've attacked or exploited whether or not you're you know in a pen test role or a red team role a lot of the ideas and inspiration for challenges you as a Content developer come up with are things that you've previously done which makes it really interesting and you know if you go on the internet a lot of stuff is still running X inot D so this is totally legit another thing is starting earlier in the year so like I said this a lot of the inspiration for this really came from randomly meeting people at Def Con at the Cabana especially the infrastructure side and we hadn't necessarily planned

from an organizational perspective how many challenges we wanted to do and who was gonna write them and I actually got an email Eric wound up talking to Joe I think or Dean and was I want to help out with the CTF of besides this year who do I talk to and I got you know an email in my inbox took about a month to actually meet up and go over what the intent was which is going into another thing to do better which is more organizers and organization so other than the infrastructure side which we were pretty hands-off with it was two people that wrote all 16 challenges which if you look at say other CTF teams there's a lot more than

two people doing the doing the work and in a lot of cases particularly say what legit BS was doing at DEFCON they treated it as a second full-time job which of course we didn't necessarily have the resources nor manpower to do so and having lives is kind of cool it can be there is a comment that it's kind of overrated but yeah I mean what's life right but there's you know certainly room to improve for organizers and organization of the event before it actually happens at b-sides so if you want to you know get involved with organization or you feel that you have a really unique idea for a challenge and you want to contribute come talk to me

and we can make that happen for next year I'll also probably do a better job and actually have you know comms reached out whether or not it be on Twitter or the besides PDX Google group to get some more assistance with this and then of course continue to improve our kubernetes cluster and our AWS infra I anticipate this going really well this weekend and I hope that we can utilize Mozilla for next year from the infrastructural standpoint also you'll notice and this is sad because we had this epiphany this morning as we were setting up there's no live scoreboard how did we not think of this so I want to get Eric up here just so people can

see see him because he put in a substantial amount of work with me this year and he reached out to me I didn't reach out to him and without his help I don't think I would have been able to do 16 challenges and get all of the infrastructure and organization done by myself so Eric wanted to say something really quickly about that yeah just a few quick comments like Topher said in the beginning a lot of CTF challenges have a really high barrier to entry with this one in particular we really went for breadth over depth so if you're a person who's maybe heard of sequel injection or heard of a buffer overflow or wants to do things that normally

might get you a CFAA investigation this is the time and place to do it if you're new we really encourage you reach out to one of us will probably be at the CTF table somewhere around here and and we can help you get started we're friendly we don't bite and really encourage you if you've never done a CTF before this is a great contest to start so that's all thanks and then of course big thanks again to the Mozilla team and them hosting the infrastructure in getting our kubernetes cluster for us and the surprise bug bounty program for their core bug bounty which is totally awesome so one thing that doesn't necessarily happen after a CTF event is organizers

will you know they'll make they'll they'll put their events on a repository and have people contribute write-ups for how people solve challenges but nobody necessarily published a source for how those challenges were created and I think that's a shame I think that sharing as an organizer the the challenges that were created showing showcasing you know how I thought a particular vulnerability worked and how I wanted it to be exploited as valuable to bring the community forward both from learning how not to write code or if you are an organizer learning how to write vulnerable code intentionally and then also showing you know if somebody is completely stuck and they're trying to solve challenges afterwards which with

this you'll be able to do everything is going to be a single docker compost command to bring everything up locally on your box after the event it's really useful to have source and I want to you know try to get a movement started within the CTF organization space of people publishing source after their events also of course there's awesome prizes this year for the CTF so be sure to come to closing ceremonies I'll do statistics of how many teams played how many solves there were there and that sort of thing also like I said if you want to be involved next year whether or not it be with planning challenge writing or helping out with

infrastructure please reach out to me I would love help and I would love to get this going is a bigger event at besides every year so yeah come play CTF it's in the event room we have dedicated Wi-Fi besides PDX CTF party there's also a series of evening activities happening throughout this throughout this conference so last night as you've heard there were a bunch of hackers meeting up a control age where we assembled all the badges I play tested some of cetf to make sure the environment was still stable after we made our kubernetes cluster a little bit better last night but with that said they're one of our sponsors Thursday nights are always open house Wednesday nights Deane

leads a thing called exploit workshop and it's usually a pretty good turnout of people just hacking on stuff and exploiting things like you know if you're familiar with the crack vulnerability that just came out play people were just playing around with that willy-nilly and having fun and seeing how vulnerable their phones were that'll never get patched also tonight technology diversified is hosting an after-party at Pascal this is a new hacker space in the area it's fairly recent and they're going to have things like retro games lock-picking badge hacking music hacker bingo and then they're also gonna have a couple of tables to further play the CTF event there's there's a meet-up page for it it's linked to you on the website it you

know the events page for contests and really the event begins at 6:30 so you know go get your drink on tomorrow night there's also going to be another after party of course everybody's going to be really tired especially those that organize the event but you know there's always room for drinking so there'll be another event at control H between 8:00 and 10:00 over which isn't that far from here you can literally just hop on the max yellow line it drops you off right at the front door and that's the end of the slide deck I don't know if people wanted to ask questions totally open for that in the spirit of this being a 1 a

101 talk and nothing you know being rehearsed and do it live mentality we did it live yeah so the closing ceremonies are at 4:00 it will be done around 4:30 and then I will get push so tonight or tomorrow night it'll be pretty instant everything's already ready we had it on bit bitbucket because you know they support private repos for free because github you know for some reason likes to charge people lots of money to have open source things so yeah it'll be available tomorrow yeah yes soon I will I will clean up all the swear words in the schematics and then I will push them to get and I'll tweet it out I should do that soon

like maybe after this also also we'll have the schematics the source file the source for flashing lists and like a rough series of instructions basically you need the NR the NRF SDK to blow the bootloader but that if you wedge your bootloader but that's already done but this is an RF Arduino project so if you just download the Arduino NRF 50 X because it's 51 50 and 52 SDK well done the Arduino runtime for it then you can flash it directly through the UART connector of the back

I'll publish a bomb cool well thanks everybody for coming to b-sides enjoy all the presentations enjoy the events and contests enjoy the parties and let's hack the planet together [Applause]

you