2016 BSidesLV - Common Ground - Day Two

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e e

you know the mic is working properly gu it's already on yeah I think you need to put it got just do a this you can sing more one two one two start singing like whistling whistling CLE the

[Applause] wind e

k

okay welcome for the second day of pide so we have to first talk today about Mo incident response and let's warmly welcome Kua

sendor hi hello uh my name is I'm working as a software engineer in security team at and I'm here to uh tell you today about how we automated our Mal response at Yelp so first couple words about me I started at uh Yelp around two years ago and since then I've been mostly involved in our malware incident response process and also in the meantime automating and uh basically um working on our security processes uh before that I used to work at sap in Sophia antipolis in France in a security and Trust research group and prior to that I was studying in kco and also doing joint degree uh with the University of telom parch is urom also

in Sophia antipolis so just sort of a quick recap yelp's mission is to connect people with great local businesses and this led over the past 12 years since uh the company started to over uh2 million reviews as you can see on this slide 90 million of the users are coming from the mobile websites and applications around 70% of the setes are also from mobile and and web apps uh mobile apps and um basically our mobile app uh website and right now yel Yelp is present in 32 countries worldwide so what I'm trying to say by showing you all the stats is basically that we have more than 4,000 employees by now and most of them are actually using uh

MacBooks to do their daily job so sort of a queen interaction to our malware incident response process uh this sort of starts when one of our employees gets an alert either like some detection by our endpoint monitoring software or some network monitoring stuff uh that detect some suspicious binaries on the users machine or some suspicious Network traffic coming from from a particular machine uh but just let me first introduce to sort of to uh people involved in all this process so F will have a Yelp employee who typically uses MacBook to perform his daily duties and from time to time they will wander to some gray sides of the internet right for instance getler to download uh latest update to

Adobe Flash Player or some free video converter app which comes packaged with malware also for free we have H desk Engineers so they are kind of serving as a interface between users and us the security team um hellis Engineers are the best people to perform all this task they have best Outreach to the users in terms of both different time zones and different uh locations because Yelp is also present in uh various locations around the globe our offices are in various locations around the goob and we have also security team uh that is also consisting of malware analyst so the people who are basically in charge of analyzing the uh alerts about malware on Mal infections

on someone machines so job of Mal analyst is typically to answer this three questions how the Mal got there in the first place is the machine even infected or not and how we can prevent and detect further infections to sort of uh stop spreading malware all over our uh company in infrastructure so there used to be lots of false positive in our approach in the alerts were we were receiving so we started involving malware analyst as early in the process as possible so we are doing this initial 3ish and this is basically to establish whether this is real or alert or a false positive whether it's something like uh Windows threat on mic OS machines that

we don't really care about and this is sort of to save us time from all these other tasks like forensics collection and forensics analysis so we can weave out as filter out sort of as quickly as possible whether this is a real threat or not there's always saving us some time um but our traditional sort of approach involved later on after this initial 3 Edge uh basically collecting forensics from the from the machine I'll tell about this a bit more in in the next slides this task used to be performed by hesk ninja so they're like as I mentioned the best people to do it because they are really leing close to the users they can just go grab

someone's machines take it up the network uh run the necessary collection scripts get the output back and then the mod analyst can start uh start analyzing the output so they can assess the risk related to the infection so when it comes to different kind of tools we have available for digital forensics collection on Mac OS there is OSX editor so this is script that more or less uh inactive for the past few few months uh it's open source and GitHub it lets you collect different properties from the from ma OS machines there is osquery which also uh is open source project Open Source by Facebook it allows you to query different system properties like you will be quaring SQL

database knock knock is also quite useful tool that lets you figure out which processes are running on your mosos machines so this may give you more insights into whether there are some known processes like system processes or something that was actually installed by the user potentially packaged with malware uh there is Google rapid response framework this is a bit more interive they let you for instance collect file samples so it also gives you more insights into in in in terms of whether the machine is in fact with something or not because they they give you possibility to collect the samples of the files from the machines as well I'd like to also mention book that was recently released is OS 10

incident response scripting analysis by jiren Bradley this books also comes with kind of ideas for scripts that you can uh basically have to collect various forensics from from maos machines and also gives you some ideas about how to analyze them at Yelp we use uh OS X collector which is a tool based on OSX editor it's also open source on a GitHub uh so this is a forensic evidence collection analysis toolkit for OS 10 uh we open sources it some two years ago it was actually first project I worked on when I joined YP so I'm pretty proud of it that it's still up there and still used by uh people actually let me get a quick

show of hands how many of you are familiar with Ox collector in the audience couple of hands okay cool so uh basically whatx sctor is is a simple python script that you run on potentially infected machine it will uh collect various system properties uh you see it here on this slide and then output them as a Json file so that analyst can take this file and try to figure out all this properties whether machine is infected or not so uh the way OSX collector runs is it collects gathers all this different information from uh pist which are kind of like you will say Windows registry things on M OS machines various sqi databases that also

mic OS uses to store system properties as well as some other local file system information for for instance for applications install in the system or uh browser history browser extension things like that so here is an example of how uh such a Json entry collected by OSX collector looks like it comes with some common Keys like file paths file hatches time stamps uh there are also signature chains for instance for for binaries that might be useful to figure out whether this is something we expect on the system or not so what we used to do later on after collecting all this files uh from the potentially infected machine is that M analyst would basically sit down and

with some simple tools like grab or JQ which is actually quite cool thing for uh Json visualization uh they will go through the scripts basically trying to find some u in events that were happening around a certain time frame or it also allow you to with help of JQ uh filter and show only URLs related to users activity around the certain time frame and basically based on that anals will try to figure out the where where for instance the file was downloaded from or when the file was installed by the user trying basically to figure out on to the questions I mentioned earlier so this also work pretty well but if you have 30,000 lines of

Json output it may come as a really job of basically looking at a lot of Json and don't get me wrong like I like Json Json is very pretty it is simple uh but it's also very easy to read and process by by code from code so actually why don't we let code process the Json output and this is what we sort of done as the next step so uh early on WE automated the Json analysis process with what we called OS excal output filters and they basically what they do is they augment the initial Json with uh all different properties for instance uh the information from our internal blacklists or information from some external threat intelligence apis they

will also try to construct the list of related files to a files that are potentially infected and then produce some sort of summary of findings so this is are really cool it automates the whole kind of analysis process but the the tool itself OSX collector output filters it was quite tedious to maintain because smaller analysts had to basically get the install installation of the tool on their machine uh when they started the analysis they had to also basically sit there watching it running they if the machine went to sleep or if they decided to like close the machine or for whatever reason they lost internet connections all this sort of process will kind of halt and then they will

have to restart it because it connects to various external thre inal apis uh via htcp so basically the tool was not written in a way that allowed them to kind of like pause it and resume it at a certain point of time also not to mention that basically when Whenever there was like new version of this OSX collector output filters malware ANS were in charge of themselves updating the source code getting all of the dependencies so it was really really uh TD's task and not something we were looking uh to do uh with the process that we were actually trying to automate so we thought we can do better and we turned the ox collector output

filters into a service and we called the service Amira automated malware incident Response and Analysis so right now with Amir what analyst uh is doing is just dropping the OSX collector output file to the S3 bucket and a miror will automatically trigger uh the analysis of the new object in the backet this is based on uh thing called S3 even notifications so we have configured this S3 bucket basically to send a notification to an sqsq Whenever there is new object in this buet this sqsq is called here on the slide am three even notifications so whenever there is uh a new object created a notification will be sent to this Q Amir will periodically check for the new messages in the

sqsq and upon receiving a new one it will fetch the related OSX collector output file from the S3 buet and you can see actually the output of The os6 Collector is packaged in tgz file file to save some space because it's like lots of Json so we want to compress it as much as possible so it will Amir will also extract the first decompress the file and then extract the proper Json file from uh from the archive and then it will run execute all this different analysis filters on the OSX collector output file and after all this process it will uh send the results of the analysis for instance to another S3 bucket uh so the

Mal analyst can patch the results from uh from the bucket and see whether the machine was infected basically read the whole summary of the of the analysis process uh here are the examples of the analysis results that Amir produces so for instance we'll see some domains and hashes that were find from the black on The Blacklist that we are creating it'll also give you an idea about information found by contacting the external threet intelligence apis and that will also provide you suggestions so basically for the things that were found on the external uh Fred intelligence apis but are not listed on your Blacklist it will suggest uh you to you to to add them to

the to The Blacklist Amira doesn't require too much configuration to run basically all you need to do is to figure out on your own this S3 notifications thing it is well documented uh on the AWS document M mentation so it's not really uh something difficult and then just to run it you uh basically need to specify sqs Q name and the AWS region where the queue was configured to run there is also a possibility to specify this uh results of bler so there there is for instance possibility to add also some other uh results of loader so results of loader are basically a way uh for you you to tell Amir what to do with the

analysis results so you may want to add some other way of Distributing the results for instance you may think about uh sending the results of the analys via email or some similar um or some basically attach them to your inent response platform if you have some more uh Advanced system to to Three Edge this alerts and that's sort of why I was mentioning that kind of this S3 the the end results of the S3 braet is is um is is optional instead are there any questions so far related to Amir are you using it as lamb function you running it as your so the question is whether we are using it as a Lambda function or running it as our uh

internal instance so we are running it as our internal instance um yeah there are several factors it Lambda faes are really cool but I found them quite Eed when it comes to importing some external dependencies like basically this whole OSX output filters but our first idea was actually to think about Lambda functions uh there were some other questions okay if not let me continue so actually you can go even a step further with all this forensics and also automate the forensics collections tab so basically uh what you can do is instead of just getting the machine and running the ox collector script on the machine uh you could think about basically having some script that

will run OSX collector on the machine and for instance upload the results to an bucket if you have large installation in your in your company so like we IDE have around 4,000 employees with with mic OS systems you probably use something like your inventory management system you could think of basically just dropping the script to run OSX collector collection and then uploading the results to S3 bucket and that will trigger uh the Amira to to uh run the collection run the analysis on the on the results here is an example of s script it's very basic I actually stole it from someone else it's the only thing it does it's just uh calculating this signature

for AWS S three so it can um send a file there and then trigger the whole analysis process uh so the whole analysis sort of uh saved us a lot of time in certain cases it saved us up to like hours from several days um we it used to be when we were involving also help desk into the whole collection process uh we have to wait for them they were in different time zones uh sometimes they had to chase the user which was also in the other time zone uh so the whole process could easily take up to several days and then the whole analysis as I mentioned when it was interrupted uh I mean takes basically

all this effort from you you don't have to sit there watch how it collects all the all the uh information from various uh threed intelligence apis it also reuses lots of caching mechanisms so all this OSX collector output filters it comes packaged with some basic cache that will not issue the same queries for instance if the user visited the same uh websites I mean most of the users are actually visiting probably like 80 90% of the websites they're visiting are the same uh so when we're running the process on each by each individual malware analyst uh basically all all all of them ahead to pretty much get the same information all over again and with

airror we're just able to have this information at once from the apis and cach it so for instance saves us lots of quot from the external ipis and it basically makes the whole process even faster uh it also cut all this interaction between the malware analysts and help desk so right now Amira is taking care of forensics collection also the analysis obviously there is a need for analysts to review the whole result summary and actually provide remediations that are then executed by the it engineer uh help desk but still like there is less sort of human interaction uh less errors possibly that could come all over along the way also there is no need for this

physical collection so even less uh uh problems with basic yeah chasing users down the corridors and taking their machine off the networks we can just remotely run the script collect all the forensics get the analysis done and then basically malware analyst can sit down and look at the analysis and figure out whether there are any false positives or weer down any sort of uh other uh problems and it also allowed us to uh do more proac active forensic collection so right now even on the machines that we are not uh really sure that they're infected or not we we haven't even received any alerts but potentially there is some suspicious network activity from our DNS resolvers

uh we could practically run a mirror and get all of the forensics analyze them and then figure out whether uh the machine is impacted or not and the whole thing is open sourced so go try it out uh I'm really looking forward for any questions related to the project any issues that you've spot uh if you have any suggestions also don't be shy create some PO requests send them my way I'll try to uh review them and on that note I'll like to thank you for coming and I'm open to take any

questions what kind percentage I'd say it's probably way more than 80% like around this like 80 20% rule right so I guess some of them will be clearly false positive some of them will be like oh it is a threat but it's not applicable for this particular machine and yeah Amira basically helps us to to figure this out because even for certain press that are for instance our endpoint monitoring say are windows only we'll still get the alert and then we'll have to figure out is it seriously Windows only for instance some browser extensions there don't really uh are related to to any systems so that's how we can um also analyze it uh are you planning any Integrations

with like sandboxing Technologies uh cuckoo or anything like that yeah so regarding sandboxing uh so this is purely sort of for forensics collection so there is no sample collection obviously it would be uh very interesting to connect it also with something that could process the sample but then we are sort of approaching this problem of how we are transferring the samples along the way so for instance Google uh rapid response framework they do something that allows you to pick a sample and I guess at that point of time we would be able also to uh have some more reliable analysis when it comes to the file so so far we are operating basically on file hashes I guess file

names things like that uh I mean URLs are pretty okay like sometimes we are basically collecting a sample from the original URL rather than from the uh machine uh because also maybe sort of to give you a a heads up on all this remediation process apart from just getting rid of the threat on particular instance uh what we are trying to do as well is block domains block IP addresses serving malware so at this point of time actually it is more important for us where this threat got from and if we are able to pinpoint it to particular domain URL and get a sample there then we know we have to block it right so it's it's

actually more related to what we are doing later in the step uh so now that you're now that you're able to collect a lot more and analyze a lot more at scale is there have you found particular things that you've uh particular indicators or particular types of data you've collected now that were not worth collecting before uh that were not worth collecting colleting they were too expensive yeah so there are particular parts of forensics collection done by OSX collector it tries to get as much information as possible that's why this 30k lines for machine that was running I guess for several months uh and then the whole analysis process is also longer because of that um yeah

we decided for instance not to look too much into cookies collecting the browser there were several issues actually with that also that collecting cookies value from someone's machines is a security uh issue in the first place because you're collecting lots of information that should not leave machine or there is assumption is not leaving a machine sometimes we get some U noise related to to one of the filters so there are filters that try to extract domains from particular URLs uh there are also uh filters that try to create kind of a network graph of related files so for instance if you have some related files to the one that's infected maybe it's potentially interesting to look at

them as well and these are most of the part too noisy to be actually taken seriously we we very seldom look at them on the latest Mac hacks that are out there using GPU Graphics attacks are you looking at those uh not really I'm not too familiar with I was wondering if you um keep your OSX collector files and periodically rerun them you know as your threat feed updates and if that's proven any of of any value to you yeah so so far what we are doing we are keeping the uh malware forensics uh that we collected from the machines uh we are not Amir yet is not uh to stateful apart from this cach that

I've mentioned earlier it's not uh creating any state there was actually a project uh that was presented at scon in Utah last year uh where people were trying basically to also put all this information uh I think it was mongod DV but you may think of okay let's put it in something like elastic search cluster or splank and let's query it so these are potential next steps for this project basically apart from just taking as like one machine individually let's try to see how this machine differs from all the other machines in the network this is actually something that osquery is able to do for instance like having the whole Fleet of machines and trying

to compare one machine against all others in the in the in the same um infrastructure what's the runtime of uh collection and airror I mean how long does it take on each laptop and how long does it take to process so the collection process so just purely OSX collector script it does rely on how basically for how long time the machine was used if the machine was used for several years you will and for instance like browser history is not puge like you have lots of apps installed it it may take quite some time uh then the whole anal so like worst case like you know up to a day let's say yeah yeah we

had the cases when it was like running for a day like it was still in the good old days of how this Engineers trying to uh get the machine right now as we are running it by our inventory management system we to be honest we don't collect too much insight about when the collection was started and then when the collection was finished we only know this time when the collection was finished so you don't know if the collection was took so long because user for instance was not in the office for like a day and his machine didn't be up to the central inventory management system in the best case I had actually on my uh When I Was preparing the the

presentation I had a case when the whole collection and Analysis process also running anal which which which can also may take up to yeah another several hours it took eight minutes basically from collection to to having the results uh available for for the Mal analyst to to look at you mentioned started doing proactive collection um have you looked at collecting that data and trending on the data over time in something like elastic search or Cabana yeah so this is something we are not doing yet but definitely uh something I'd like to work as a as a next steps basically after we automated all of this uh kind of similar with respect to active defense do you guys have a

particular threshold that you have to meet um in order before you uh deploy the script or do you just kind of have the script you know already available to run on machines and you run it across all your machines or do you again or do you have to reach a threshold so uh yes we we do have something like a threshold it's basically this alerts we get from endpoint monitoring or network monitoring so this is our initial kind of trigger for the whole Mal malware incident response process but then we had some cases in the past when we were suspecting part of the fleet to be infected with something and then with just one click we were able to deploy

the script to like hundred of machines get the analysis done by the next day and look at this so it's way more scalable than our previous approach in this matter what kind of file sizes does OSX collector like put out like have you seen uh file size in terms of the The Collection yeah so not sure it was actually on the slide but uh the compressed tgz file it's usually several megabytes if you decompress it it's like I know 60 megabytes 80 megabytes something like that okay any other questions still okay then no thank you Kuba let's hear it from [Applause] him

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e e

um my name is Alan fredman I'm from the US Department of Commerce and I'm lucky in that I don't actually have to do much talking in this I'm just going to tell you a little bit about what Commerce has been doing and the answer is we've been having really smart people who are dedicated about the security Community uh do a lot of hard work to make progress on a fairly old problem infosec vulnerability disclosure uh so my corner of the Department of Commerce the national telecommunications and Information Administration has convened an open and consensus driven multi-stakeholder process focusing on vulnerabil disclosure um and it's been about running for about nine months and we've had brilliant folks from the security

research Community from the vendor Community from the intermediaries who help promote that who've done a lot of work uh initially and we're going to hear from the three working groups that have been very active over the past nine months one on awareness and adoption another on safety and disclosure and finally on multi-party disclosure um so this work that you're about to see is not from the Department of Commerce it is from the stakeholders who believe that this is an important issue uh at the end of this talk we're going to have a fairly brief talk and the goal really is to have some discussion to hear feedback so that you can share your perspectives on what

we're missing and what we can do better to bring about some positive change so very briefly Department of Commerce likes it when markets work uh online trust is a huge priority for our secretary because without trust in the systems that we use uh there's not going to be Innovation there's not going to be adoption markets will fail and sometimes to fix a market failure you need active regulation you need to weigh in with the big stick of government uh we believe that there are actually often lighter touches so we want to bring together those who care about this issue and say how can we have collaboration around vulnerability disclosure now this isn't a new debate

we don't want to reinvent the wheel we want to find some way of saying there are standards out there people have been thinking about this issue for a long time how can we actually uh make some progress so rather than trying to write new standards or even produce best practices we're trying to come up with some principles of what we can do to of what we can do so that researchers who want to continue to engage know how to engage organizations who are new to this issue understand what's at stake and what the path forward is and the underlying approach is there is no one siiz fits all uh that every organization is going to have something that's unique to them

and different researchers each bug is at the end of the day going to be unique and need to be handled differently so what are the broad issues that we can do so the process has basically involved a lot of talking it has involved some very tedious meetings and even more tedious phone calls so the people who've been engaged uh we really have to thank them for all the hard work you're going to see the working group chairs but behind them as they will say a lot of very smart people were dedicated to this so first we will talk about the awareness and adoption group and what's noteworthy about this work is that I think everyone

in this process has said at the end of the day raising awareness and driving adoption of existing good practices really is the most important thing and relates to all the other work that's going on in vulnerability disclosure Jen sorry Jen Ellis from Rapid 7 and Amanda Craig from Microsoft thanks alen uh so the big learning there was that Alan does all the slides from now on um okay so uh as Alan said I'm Jen Ellis um I uh run community and public affairs at rapid 7 so uh I head up public engagement and also think about how we can support the security Community you guys a little more uh which is where this Falls in I'm Amanda

Craig from Microsoft and I work on cyber security policy issues Round of Applause for Mr Josh [Applause] G okay so um why why this why did we do all of this um and and is it just because we wanted to sit around and come by our um I think you know ultimately as Alan said there's been a lot of work that's gone into talking about vulnerability disclosure and handling in the past 20 years um lots of incredibly smart people have worked on this problem and there are very well established uh best practices for both sides of the equation for for researchers and for um for for vendors uh in fact there are isos there are two of them however

there's not very much adoption and so the the real realityy is that the problem continues and that if we can't solve adoption everything else that we do is just academic it's just um a lot of sitting around talking to people to death as Alan said so for us that was why we really wanted to focus on the adoption and awareness problem um is to make the theoretical um applied and actually try and see some change occur um that was that was a big kind of focus for us um so what we're going to talk you about today is surveys ooh service so sexy there'd be even less of you in the room if you'd known that

this is what we were talking about um so we uh we it's kind of funny we we were talking at the beginning and there were lots of people in the rooms oh you see people even surve um so at the beginning we were talking there were lots of different voices in the room from all different sort of sides of the the uh conversation and there were lots of people using the kind of dialogue you often he in this discussion lots of assumptions being made and we had this sort of like very side conversation Amanda and I One day about how you know if we're going to get meaningful about coming up with ideas for adoption which is what we were

focused on then it would be really good if we could get to the bottom of what was really going on with the surveys and like really understand the truth behind it and so the idea the surveys initially was something that we were going to just basically put together in a week and get out and it was going to be like a really quick thing that's not really what happened um and so they became this like huge thing and it ended up not being one survey it was multiple surveys you talk about the who we surveyed and why yeah so you know Jen has mentioned some one of the reasons why we initially came up upon this idea of a survey was to

challenge our assumptions and sort of um you know challenge everyone that's been part of this conversation for a decade or more to to rethink you know have the Norms changed but there are also you know new players in this space there the the iot folks that are newly technology providers automakers Aviation companies medical device manufacturers and so on that are newly dealing with this issue of vulnerability disclosure um and so we wanted to also capture you know what what's going on in their world um and and we did we were hoping that we would get some real data from the these surveys that would help us identify ways to drive greater awareness and adoption and so you we thought about different

communities that we would survey very obvious to survey the technology providers and operators and the security researchers as they two very Central players to this we also did consider doing a consumer survey the reason behind that was that you know if consumers care about security that of course helps to drive adoption of security practices like having a vulnerability disclosure policy in place um for vendors so we were interested in knowing you know to what extent this is a consumer issue or a user issue we ultimately decided to not do a user survey um Jen's going to talk in a few minutes about all of the sort of method methodology issues that we faced and and

disseminating the survey and and the bias that we likely captured um and surveying the the internet the way that we did um but all of those issues were really really exacerbated in the case of a user survey um because is it was just going to be really difficult we're this survey the and the dissemination of the surveys was just done by the awareness and adoption working group as part of this NTA process we didn't really have any funding or any expertise and and surveying and so you know we were we were just going to be doing uh what we could to promote the surveys and so we felt that the responses that we would get would just have incredible bias it

would either be totally random from users or it would be people that were already really really interested in this topic and that would that would be the reason why they would notice this in respond to to the survey so we ultimately didn't do that what did we do M we did do the the technology wrer and operator or vendor and the security researcher survey we tried to make these surveys really short really simple I think they were both like nine questions um you can see you know in general we were just looking to figure out what what are you what are you doing and why what is your general expectation H what is your behavior what is your rationale

for that behavior um in the process of researching in the process of disclosing or receiving and handling a vulnerability just want to highlight that you know these questions the fact that they were so simple we think helped us get gather a lot of responses but the the the flip side of that is that you know there were some limitations and the questions so for instance you know if we were trying to um understand that there would be a vast array of experiences that researchers would have and we want to capture all of them a lot of times we uh enabled multiple responses to any single question which then made the data hard to interpret for instance we asked

a question about how interaction with a vendor was and we had like 55% of researchers say that it they had really frustrating experiences in Comm communicating with a vendor and we had 60% say that they had you know a really productive conversation with a vendor so you can see those numbers don't quite work out and and we but we took what we could from from the responses that we received so as I'm going to say said we surveyed the internet um and we have been mocked for a huge amount uh the nice thing is when you survey the internet the entire internet mocks you for surveying the internet um and they also helpfully provide feedback on your

survey uh after it's already out in the public domain and I mean it's fine because data scientists tell me that there is no validity to the survey anyway so we could have just started changing it um but we chose not to do that because it seemed kind of skey um so we had a lot of learnings in the process you know the the reality is like as aanda said we were kind of Highly limited uh you are all playing spot the meme now aren't you um so yeah so uh we were sort of Highly Limited in that this is um you know it's a volunteer organization it's not owned by anyone sort of entity and so we were

limited in kind of what we could do on the surveying we wanted something that was free and open and easy so uh we did go the sort of Survey Monkey route and I think if I was advising somebody wanted to do this as more of an academic exercise I would tell them not to do that uh but you would have to pour a lot of money into it honestly um and we had a huge number of learnings along the way one of the things that we found out is the vendor survey didn't really work for the open source community and that was a little heartbreaking because we really wanted to capture them um we also had

used some of uh survey monkey's kind of standard um like yeah demographic splits and for the uh verticals for the vendor one that was really dumb because one of the categories was technology and uh obviously a lot of people identified that way interestingly not everyone did um so there were definitely some some pretty some pretty huge learnings for us in this process but we did try to um I would say like push Beyond just putting it out there and dumping it and leaving it and we knew that that was never going to be effective as a thing to do anyway so we got really active with trying to get media to promote it trying to get

people we knew that had um wide followers to promote it we reached out to isacs Amanda did a phenomenal job of talking to a lot of vertical alliances a lot of vertical isacs and getting their members to do it and we could see that coming through as the results were coming through oh is this me again um so how do we do um so on the research Serv we had 414 responses now again we surveyed the internet so 414 may not sound like a particularly High number but when we went into it we were very hesitant about what we would really get in terms of response and the stretch goal that we had set ourselves was 250 for each survey so we

were really delighted by 414 responses and the data actually did show kind of a a pretty decent Variety in who we got in terms of um whether they were people who would uh doing research as part of their job and that kind of stuff um so the the largest um the largest percentage of geographic uh split for both surveys was definitely the US which was not super surprising since the NTI is a US Government um entity however we did see some other uh governments pay some interest in it and and some people kind of promoting it internationally so for the researchers we got 210 us responses and then you'll see um the splits here so half of the people who

responded uh listed themselves as sort of quote unquote independent researchers um it was nice the thing that I thought was awesome was that we actually got people who are accidental finders responding those were the people we thought would be super hard to reach we didn't think that like the way that we were disseminating it we would easily find those people so it was good that like people who don't consider themselves to be professional researchers took the survey and that data is I think very valuable um and then in terms of the tools we had asked this question about what kinds of um tools people use I'm not sure that tools is really the right word for some of these things um and

this question again was one that I think kind of became a little crazy and Bloated over time uh when we were preparing it and so ended up covering a sort of weird mismatch of things um but you can see I'm not going to talk through all the stats cuz they're right on the screen in front of you so so you can read them and thumbs up okay so yeah now getting into the actual uh data that we got from the researcher survey we asked a a question about how researchers disclose um what's their first action in disclosure and then also a separate question about what their expectations are in disclosing and how that influences their behavior so with

the for the first question the response we got back is what's captured in the graph which is like 67% dis disclosed to the vendor um something like 10 and 13ish perc uh disclosed to a bug Bounty provider or to a coordinating organization like C CC and then just 4% approximately either just don't disclose or um go public in full disclosure um so we were you know excited to see that what we kind of consider coordin vulnerability disclosure with captured in that Circle was the sort of predominant Norm but then um we sorry we did look at you know what how you know expectations change behavior and we saw that when um a researcher submits a a vulnerability to a vendor

and then and doing so they also provide a time frame and then that time frame is not met by the vendor uh then then they will go public so 24ish percent of researchers said that they had they had done that around 7% said that they um the vendor provided a timeline when they disclosed and then the vendor didn't meet that time frame and so then they went public but about 8% said that the vendor provided a time frame when they disclosed the vendor did not meet that time frame um and they considered But ultimately did not go public and I think it's just coincidence that it looks as if the researchers were giving us the

finger um so unsurprisingly um well I thought it was unsurprising uh but then I work in Communications people said that communication is valued shockingly people like to know what's going on and feel like they've been heard um so we had 95% said that they expected notification when the issue was resolved which seems kind of reasonable um 68% said that they would really kind of like regular updates that seems like a pretty decent reasonable thing so that they know that something's happening um and uh 57% actually kind of went further and said they'd actually like to be involved in testing the fix to make sure that it really is is kosher and works um and and that was kind of interesting in

that there were researchers who were like we're super happy to stay involved in the process we want to partner with the vendor we're not just like hey this shit's on fire and then running away screaming um which is cool like we appreciate that about researchers and then um 84% in a similar vein said yeah totally happy to answer question for V from fendors want to stay engaged want to stay part of the process um and then trying to look at the rest of oh never mind I can't get out there um so yeah so communication was really important people really valued that they expected the vendor to be open and transparent with them and apparently I've now done

something to the slides there we go yes I don't computer thanks Josh oh is this me again God damn it how did I get so many slides um so this is yeah this is actually the there's a reason this one's me because I kind of talk about this issue all the time um so we hear a lot about the chilling effect of um specifically the cfaa and the dmca and specifically with both of those the fact that they have civil action in them which means that a lot of vendors use them to threaten researchers when they're afraid of disclosures so we hear a lot about that but it's always a little hard to know whether that's

something that is a little bit sort of fuddy and overhyped um it's not 64% said that fear of legal repercussions is a serious serious issue for them and it it seems as if it is something that um makes them question whether or not to disclose to the vendor do the last 26% say oh now I need to be the last 26% responded from prison it's not true by the way for the recording it's not true um uh so yeah um and then we also had 24% said that they are afraid of stumbling into confidential information um only 24% the others are like whatever sounds great let's see what I can find um it's why I do this

uh 31% said they are afraid that exploits will be used nefariously I don't if we had the word nefariously in the survey but I hope we did I think we did uh another finding um in asking what does a forging what a a a researcher expects and return for disclosing a vulnerability was that it's not just about the money you know consistent with what Jen was saying about communication we had 70% say that they expected um to hear from the vendor after they submitted a report 53% said that they they expected um to have some recognition uh only I think 20% said they expected nothing uh and then 15% said they expected a monetary reward and

you can kind of see by those numbers and how they add up that this was one of the the answers that you could check many boxes so you weren't Limited in saying you know if you wanted one thing you couldn't want another and still only 15% said they wanted they expected a monetary reward one it is a different story probably but expected does that interest how many people are surprised by that show hands like this was one of the biggest surprises for me I was really surprised that because we hear so much dialogue in the community about you know how people are selling vulnerabilities because there's a market for it now and if we don't have bug

bounties then we're not going to be comparable and all that kind of stuff and the reality is only 15% care about that which is kind of Awesome by the way like that means that this community is doing this stuff for much higher purpose and that is fantastic I love you guys you are amazing fortunately amand's talking to the next so this we're now moving to the the survey of the technology providers and operators or the vendors a a demographic are we no no it's good okay uh demographic information as you can see the there was the issue that of the Survey Monkey self- populated and Technology being um a huge uh that percentage of our respondents so that

may be slightly skewed um because we did try to reach these newer kind of Technology providers in there but I will say a quick anecdote on this um one of the meetings that we had um I had I asked a question about technology providers in the room and their attitude on something and ask for show up hands as I am want to do and there were a whole bunch of people who didn't put their hands up and I knew that they were from what I would count as technology providers and so doing the thing that Alan tells us not to do I put them on the spot and was like did you not answer that because you don't you can't like

publicly answer it or do you just not identify as a technology provider and they said we don't identify as technology providers and that was to me super interesting these were automakers and they like totally had this this that we're carmers that's what we do that's that's how we've self-identified for a long time so even though now we're dealing with things that have millions of lines of code in them we don't consider ourselves to be te quote unquote technology providers so this this actually as much as it seems kind of like really obvious we we get this result it was also slightly surprising to us so just like the the researcher survey predominantly our responses were

coming from the US we did get some you know 10 to 15ish responses from UK Germany uh Japan Canada I think um and then the other thing to highlight here is the Divide between large and small organizations um we had if if you can imagine a thousand uh employees as the dividing Mark um there are I think what is it 160 respondents were smaller than that 125 were larger but the our respondents that is heavily weighted towards having um you being really large or really small so having more than 10,000 employees or less than fewer than 100 employees um and Alan sure just very briefly on the the math side of things uh one of the ways broke down uh the

respondents uh using clustering analysis due to the wonderful Bob rudist uh is to say Hey listen we noticed that roughly half of the participants seem to be uh mature they've thought about vulnerability disclosure and roughly and and have implemented a number of different practices for different reasons and then roughly half were not mature they didn't have many practices so we can learn something about you know what does a mature organization look like versus an immature organization when it comes to vulnerability disclosure thank you Alan and and one other thing to highlight there is that we didn't necessarily in the respondents to our survey see a split among large or small organizations being over overwhelmingly mature or immature recognizing that

likely outside of the context of our survey and what we captured there might be more of a difference but there were a lot of small organizations that still kind of met this maturity model criteria so one of the the things that we captured is what are mature organizations doing so we were had a kind of a bar of 75% or higher for for what we were wanting to highlight having a dedic ated monitor monitored path um for investigating triaging and resolving vulnerabilities having uh a process for providing end users with alerts uh providing researchers recognition um and vulnerabil and having vulnerability reports inform form your secur security development life cycle were all really um heav lots of

organizations um were acted consistent with those best practices versus for the immature organizations or the less mature organizations for each of these they were between uh 8 to 12% following these behaviors okay so uh why are they doing this stuff the ones that are mature why um so excuse me sorry um so basically uh the biggest thing is always going to be that your customers tell you to uh you know people people vote with their feet and if you feel like your customers are not going to buy you are going to change your behavior so um in the more mature bucket uh we had 79% say that their customers cared about this issue whereas in the less mature

9% they were kind of like ah whatevers um and we could you know theorize Who falls into which bucket but we won't that's for in the bar later um the other the other big ones were I mean certainly like Corporate social responsibility there was it was a little woer on that but not not super low um so the more mature was 65% and I actually can't see the number for the lower one um and for the uh for the the it reduced the cost which we thought would be a really big thing um the more mature it was just a little over half 5 54% so we asked them which best practices they were looking at um and we

gave them three options so one they were deriving their best practices internally two uh we actually gave them multiple options but these were the top ones um so one they were deriving them internally two they were looking at the isos and three they were looking at what other companies are doing does anyone want to guess which stat goes with which thing so other companies anyone want to shout out stat wow uh no the one that none of you get 59% said they were looking at other companies um isos 49 absolutely which was a little surprising to us um honestly and says a lot about the availability in the and the awareness around the isos and then

76% were deriving their practices just looking internally at what made sense for their business which is actually kind of awesome um and I always have an Archer slide uh so we have this it's actually we're going to be doing this bit after you've heard from the others but this is where you kind of are going to help us brainstorm ideas for driving adoption thank you very much sorry this was a little and thank you Amanda and Jen and now Josh Corman who shouldn't be a stranger to many of you uh has uh offered us that slide's coming later um some the work that's been going on on the safety and disclosure working group and thank you Josh for coming and

running an entire track on your own but still coming up here to share what your working group has been focusing on okay so I'm going to go fast but is not to discourage B which I'll probably have to get later um one of the things uh it was very clear to me is while disclosure is not new to us I think one of the missing pieces and a lot of the the fear and trepidation of a public coordinated vulnerability disclosure multistakeholder consensus based process was that didn't want to relitigate the old vulnerability disclosure Wars we didn't want to reopen Old Wounds we didn't want to get into responsible disclosure fights so there was a lot of

uh polarization coming in but what one of the things that I realized through the work we've been doing with I am the Cavalry is that a lot of these safety critical Industries they're at year zero of their journey and 15 16 17 years ago Microsoft was sending Sease and assist letters to our friends and now they're giving them six figure cash prizes and celebrating them at Blu hat and it's a vitally necessary part of rech development and their product and their customer satisfaction and whatnot so I call that meantime Enlightenment about 12 to 15 years and we're going to have a similar Journey for these safety CRS but they are at year zero or year one and I

don't expect that we can afford 15 years for safety critical Industries but maybe it's going to take one or two so what I tried to put on the table early is that this is where bits and bites me flesh and blood and where the consequences of failure will be measured recards or records loss but in human lives in national GDP in crisis of confidence in key markets that are necessary for our way of life it may even lead to a compromise of our civil liberties and our values as a nation or as a global community so we tried to assert and people seem to go with it that safety critical Industries could be a superet

of the requirements and constraints on designing a journey to go from crawl to walk to run if you're playing Josh Corman Bingo that's the Center Square um but the crawl walk run idea is what we decided to do and the C has worked with Tesla with GM with others to get them on their Journey towards coordinated vulnerability disclosure it's our belief that if you can create a high trust High collaboration Zone then you'll have better outcomes and essentially the simplest way I put this is do you have a beware of dog sign for the researchers or do you have a uh a welcome map and the crawl idea was what's a minimum viable product that fits on one page

that can be used as a template so you can sell it to your general counsel without a whole lot of attached surface and red lining and entropy such that maybe you could get started on your journey and one of the things we've learned from these folks is that you can never retract you can always expand but you can't ever retract so if you offer cash you can't get rid of it if you have a narrow scope you can't make it more narrow um not really so the safety critical working group so people seem to agree that maybe it's not a beautiful in Snowflake but if you can Sol solve for forever days an unpatchable vulnerability and a piece of industrial

controls equipment if you can solve for that then it may also be useful for non-safety critical Industries and we've had some consternation over this we've done some duplicative work with ar who hates us for it I'm just kidding but I I think that Jen and U Amanda's survey work has been very Cooperative to some of our assumptions very much so so um this is a little bit frustrating for me to do a collaborative process because I have this Burning platform idea not that I can't collaborate but um one of the things I tried to remind the group and I will remind this group is whether you have a coordinate vulnerability disclosure program or not thanks to the

DMC exceptions dmca exceptions coming in October which is two months from now uh you're going to get a surge or a title wave of vulnerabilities and and the data in the survey shows that what was it 64% yeah are afraid of legal reprisal when that fear is reduced you will see submissions and without betraying any confidences the rumor is that in the first 48 Hours of GM's program with no with no vulnerability Prize or not they got over 100 submissions and I asked them you know if you got a lot of submissions I won't say how many how many of those do you think were found in the last 48 hours and that was really

the key question because you can't find an automotive bug in 440 hours these were known but people were afraid to share so um the same is true for a lot of these folks that simply reduced the barrier so what we wanted to do was show some sort sort of um again minimum viable product and this is a template that we've used with others outside of the collaborative process and I was really trying to add some urgency that even though this process may take a while and it will take some time to get the survey back we wanted to cause some sort of action prior to October 1st because if you want to build capacity and muscles Before the Flood this is the

time to do something so um we basically made a really really ugly word dock that got a lot of fighting but I essentially said they should fit on one page and there's there originally were four blocks do a brand promise do an initial program scope keyword initial that we will not take legal action in unambiguous terms and then the mechanisms uh for submission and ongoing communication and then we've had to add this last one more recently because some things have changed uh submission preferences so it doesn't affect your legal posture but kind of bugs you will and won't prioritize or accept um so really quickly um the brand promise idea is safety is super important to us and in

addition to all the wonderful things we already do with our own staff and with third party contractors we want to cast as wide in it as possible and invite the participation of willing allies in a whole of community approach to try to um find bugs and be safer sooner together that kind of thing so this is a marketing thing that can get you a lot of people look at this as a legal thing or an engineering contract this is very much a brand a reinforcement thing people were really pleased to see GM's program people were very pleased to see Johnson and Johnson's program so uh this reassures to your customers that you care about this and take it seriously

the initial program scope um because these are saf to critical Industries who have no idea how many bugs are going to get submitted to them and they don't know if they can internally triage them we've encouraged an explicit scope reduction don't ask for bugs on all the things you've ever made maybe pick the most recent model here maybe just pick one making model of your car uh that way it becomes a throttling mechanism which you can expand after your pilot no one judges the Pentagon for having a 20day pilot they did a pilot and based on what they learned they're going to do another phase in all likely so the idea is start narrow start on something that's newer

and more modern perhaps something that you feel the engineering teams cont triage and based on the received bugs and the quality of those received bugs you can always expand scope over time and by stating initial you won't be judged for this as a finished product against something like a Microsoft you'll be judged as we're beginning our journey um there's also implicit scope which I will skip for now there's the most one of the most important parts is we will not take legal action if and while I don't have Archer slides apparently I have my cousin B slides um so if you uh if you there's some pretty good exemplars here but basically we've seen people have as as few as four

bullets of as long as you follow these three things these four things these six things I think GM had eight things as long as they're affirmative unambiguous um clear people you're basically having a covenant with researchers that if I'm willing to admit to these things then I know I won't hear any fear reprisal and these ones what we're finding should be fairly immutable and Evergreen if they change frequently people won't trust the program researchers won't trust the program expectation management is 90% of any human thing you ever do so this is really how do you submit what are the minimum expectations for submission and what are the expectations for initial uh acknowledgement of receipt and ongoing

communication Cadence and rather than prescribing this we've outlined that the negotiation of the ongoing Cadence should be on a per buug per relationship basis because not all are created equal but um you know the iso Stander does believe say within seven business days knowled re see and then submissions and preferences and priorities we wanted to separate separate uh something that's added some confusion some of these programs on day one only had four sections and then as their Engineers got flooded they started adding exception after exception after exception and the researchers just stopped de in their tracks because they didn't know were these exceptions to the legal posture or were these things that we just don't want to hear about cross a

scripting and there's a bunch of things we're tackling as well about how do you do change management and Version Control because these things should change and grow over time as you learn more but we also don't want a moving goal post so we're you're talking about something like a legal posture and we've had a lot of discussion uh and we haven't come up with Perfect Solutions but we have seen how folks like Microsoft and others have tried to dampen the risk So to that end I think many of those things uh were alluded to without slides so scope one of the less obvious let me just do 30 seconds on this we cut it from the

document because we had bait over it but it's actually proving to be very necessary after the survey I basically say white hats have five key motivations uh and they all start with a peak there's protectors want to make a really safer place there's puzzlers who do it for Challenge and curiosity there's Prestige who do it to win a white jacket or to give a keynote at death crme there's profit who do it for money and there's protest who do it for or against some political cause or ideological cause if you fail to include a cash reward it's not a failure at all I I'm deliberately and overtly encouraging that the first for disclosure program does not have a bounty attached to it

and one of the reasons for that is you will only attract the subset of researchers who are protectors and puzzlers they won't give you arbitrary conference deadlines they won't quibble over how big or small the cash prize is and when you're ready to you can always layer them in so we've also encouraged the lack of U monetary reward in fact one of the coolest rewards that anybody gets is things like this challenge coin from Tesla or things like a t-shirt from the Netherland that says I hacked my government and all I got was this lousy t-shirt people prize that t-shirt more than they would prize a small cash reward that's well below their day rate

for the kind of work they put into it so at least for these safet critical Industries we've considered um we've strongly encouraged the lack of one and you'll notice that GM was not criticized for not having a bug ding program but FCA did get a little bit of criticism for having too small of one uh and we're in this learning curve together and we're trying to figure it out but I think this working group has been very intense very candid and and data you're seeing from Jen and Amanda's work is cooperative so with that I'm going to run away right now um but if what we really want is more feedback we've had some researcher feedback we've had some

safety critical industry feedback and we've actually had people already publish theirs even though it's not done um but for right now I have to run so find me um join our working group calls we have a draft it can always be better and the problems we need the most help on are how do you change management thanks thank you Josh [Applause] and now we from art Manion from C who has literally been at the center of this debate for uh 15 years now personally 15 years organizationally 28 I think we count it's a hard problem I mean come on it's a the problem is it's a people problem not a technology problem so you can't solve it with a technology or a

protocol so uh uh art Manion SE coordination Center I'm going to represent the work of a bunch of people um who are part of the multi-party disclosure subgroup of the Nia process um this got merged with uh first form of incident response and secure whatever it says up here form of incident response in security teams has a special interest group with a very similar bit of work going on a lot of the same people doing two things people were not happy with that merged reduce duplication increase efficiency um the work continues under the uh under the special interest group within first um we get some nice administrative support from first like web XE stuff and and uh

we actually have a person taking notes so that's very nice um why why look at coordinated vulnerability disclosure uh for multi-party um why coordinated disclosure at all vulnerabilities exist attackers attack with them against them uh we're kind of sort of sure think we're sure that disclosure is a good idea and and is is an effective defense that's something we might be challenging coming up um there's some general agreement on the model you find a bug you report it you wait uh and the vendor fixes it and you disclose and after that um people disagree pretty quickly on how long do you wait do you disclose when do you disclose how much information so framework might be four boxes but the

details go all over the place so why coordinat disclosure why multi-party disclosure um more vendors means more complexity uh I asked my more math informed colleagues they said it was not exponential growth but it is greater than linear growth when you add people to the to the mix keeping secrets gets very hard after the second person finds out about something when it's the 73rd or 125th vendor that just gets harder and harder managing comms gets very difficult uh and uh expectations and policies get start to start to get uh become in Conflict after had a lot of people um we're seeing a lot of more shared code format protocols these days different supply chain relationships

than you know Unix and windows used to have years back Josh was up talking about safety Industries um the way cars are made in the US is a supply chain well was new to me uh and does not fit existing models of um how software gets into things so that's interesting uh this was also covered there are there are vendors as I still refer to them who make things they think they're car manufacturers they are technology manufacturers but they don't think of themselves as that yet so GM is not new to the planet GM is new to disclosure um the process we went through to try to come up with something useful was not a survey uh but sort of a

uh conceptually structured thinking process uh we are going to try to derive something useful from what what our real the group's real world experiences are so uh different types of coordination and disclosure cases um variants that come up from those cases because the exception is the rule in lots of places especially here uh you are always going to have a variant it's very rare you get a coordin coordination disclosure case that's goes smoothly and perfectly um for each variant what caused it how could you prevent it in the future how do you respond to it uh lists and lists of things Cluster those lists see what sort of things arise to the top and that from

there you you get your practices and potentially further from there we're going to get principles out of this thing um here's a very simple example so a a very clean relatively clean case multi-party coordinated disclosure is going on uh variant four a vendor leaks early um what's the cause they send something in plain text prevention is encryption response is cats out of the bag everybody run so uh guess what someone just Josh just said this right it's this is a human sort of organizational problem M expectations huge huge thing um publish and follow your policy at least tell other people what's going on uh doesn't mean you agree it means other people know what to expect from your what you

expect from them which is at least a starting point um some common terms or common reference might help but we're not asking everyone to agree necessarily on one cvd policy that works um we're getting an interesting example coming out here an emerging thing open SSL is a good example of this it is so difficult to pick who to tell first they basically have an announcement saying there's something coming next week and then on next week on whatever day everyone finds out at the same time I think uh communication is huge uh frequent is probably better um people feeling they know they're being talked to they're part of the thing keeps them involved and happy uh less likely to to

leak something um if the finder does not reach the vendor you have no coordination going on if the fix information does not reach the users you have no fixing going on and the whole thing is a waste of time if those things don't happen so comms is clear you can get help coordination centers can help sometimes uh we actually don't want to be involved if we're not needed because we have other things to do but um uh multi-party is a case where often having a neutral somewhat more objective third party who's actually paid to do all that Communications mess um can be useful more practices uh have have relationships know your supply chain up and down know your peers horizontally

know other stakeholders this means Outreach to researchers your suppliers you want to know when somebody Upstream put a new version of lib PNG in the thing that's in your dashboard um you want them to tell you that so you know what you need to do or not uh incentive incentives we've already covered this in previous slides but chilling effects are a thing uh be cognizant of those I'm not telling you what to do but be aware that exists they exist um you can do you can you can do various incentivization rewards reward structures um an idea the groups thrown around is to exclude repeat offenders if you always leak if you always uh are not

playing with the social group common social creature behavior is you're you're out of the group we don't tell you next time uh keep clicking here uh whatever you do during a multiparty thing is going to be multiplied be careful with what you do especially disclosing sorts of things provide good information we like cve or something to tie things together we like machine readable information and lastly I'm going to throw up we have uh the group has not worked through these principles slides um so take them with a they're very very much in draft state but uh harm reduction being prepared uh you have some responsibility to inform others if you buy into this whole defensive disclosure is a a good thing

in the first place you may not agree in the first place in which case these principles won't apply but if you follow that these are the kinds of things we think are sort of overw overarching sort of human goals that you might want to uh um uh consider feedback is happening now feedback happens is ongoing we expect to have a draft of the MPD multiparty Disclosure document out for public comment uh you can join the first Sig you don't have to be a member of the first to join and be part of our uh I think twice a month calls at this point and contribute that way obviously there's open comment period you can contribute that way that's my email

address if you have other questions and thank

you so I want to thank art and you can stay up here because now is the time where there were some questions earlier um but we can do some quick takeaways from the survey earlier uh and the awareness and adoption group and really one of the reasons that we wanted to come out here is because while there has been active research or participation of people who care about security not just because they make stuff but because they're genuinely interested in security we wanted to make sure that we heard from the security Community uh and so uh there the three working groups that you've heard from and here's a refresher of the takeaways on the uh the awareness

and adoption group of what they found it's behind us uh and so if you have thoughts and we can revisit some of the specific questions of how can we reach safety organizations and organizations that are new to it and build templates and get them aware of the crawl stage uh how can we coordinate across uh multiple parties and and what are some of the basic Frameworks for that and and of course uh how do we just raise overall awareness uh so I'm going to stand here and try to moderate and Jen's going to swear at the end uh I mean not at the end all the way through an accent that makes it f like nice it's

fun it makes it nice with an accent it makes it nice there's a tweet for you right there uh so please uh so I know there was a question earlier but I'm going to return the mic uh you've got that mic okay Alan so first of all I have to do the legal disclaimer my opinion is not my companies um mostly because my regulator is sitting two seats down for me so I work for a very large Pharma company that I am trying to convince to get involved in this so I have some feedback and I have a challenge for Alan and I think for Jen as well so is it not swearing because that's

impossible yeah um so I know you guys are doing great work okay so I've been following what you're doing and it's fantastic and amazing however you guys need more of an ego you need to publicize what you're doing yeah yeah well it it it may not be a problem for some individuals but for as as a working group genuinely you know serious feedback you're not selling it enough okay you guys need a website you need to do more engagement I know that ntia doesn't have money to put behind it but you need to find some way for me to be able to go to my VPS and my business side people and say I want us as a very

large corporation that could support this to get involved in this so so the challenge for you particularly um I do a lot of work with Ian as well which is also an npia thing you know my boss then yeah I'm good friends Larry um so they do amazing stuff on multi-stakeholder engagement and I think there's really learnings that could come from that side over into what you guys are doing and then just from the community involvement just looking at the slides and on the feedback and the people that you got I think we need more people in my position so more Healthcare vendors more Automotive manufacturers more people that are part of that whole mix

of companies that were manufacturing or were Healthcare but we are now technology providers which I totally agree with yeah we need to work out as a group how do we get those people involved and that comes back then to the first thing is how do we sell this as something not just as a this impacts the security and Technology people within a company but this is something that the business needs to work on and I want to just call out as well cuz I didn't get a chance at the the the calvary thing yesterday what Suzanne has done with the stuff with the FDA has quite literally revolutionized how security is being spoken about within my industry here so

I I have I have a 30 second anecdote that I want to say I've spent 10 years in many Pharma companies trying to push this discussion that we're having here six weeks ago I got an email from my vice president of reg Affairs and our director of QA asking me what we were doing in this place not the other way around that's how much it's changed in the last say six months to a year yeah so this is where we are now and I also think that the interactions between these two groups are how we're going to move forward as a security community and that's really important that we take this and grab it now and set the path

for the next particularly for my industry we work slowly the next 10 years and that's starting now I really appreciate that is fantastic feedback thank you um I'm going let the the experts respond uh but I'm just going to say that the final step as we draft uh this process is to say okay how do we take the document that has been built with the input of those of you who really know what you're talking about and how do we get broader adoption now in some cases we're lucky because we are the Department of Commerce and because it is a voluntary approach we can get industry to wave it at their Regulators who aren't as friendly as Suzanne and

and and the Chamber of Commerce for example hates regulation loves this approach um and any advice that you guys have in terms of how we can take this and and build it so that it's something that is not just uh in the Tech Community but outside and Global because it is always hard for especially in the US government to remember that we are not the entire world uh so uh so you know how do we take this uh Global varful but um suzan was your session recorded yesterday did you know it was so I would just say for any of you who were not able to catch suzan session that you should check out the video it

was really really awesome and um she has done incredible work at the FDA and um and they are certainly to my knowledge the regulatory body that is most ahead with um adopting cyber security norms for their their vertical um and it is super inspiring to see honestly so you should definitely check out her talk um in terms of what you're saying one of the things that Amanda and I have talked about a fair amount is at some point ntia has to move on um you know that for them to continue forever is not the best use of taxpayers dollars and as the uh example of represent of Taxation with no representation I I want Alan to be doing

other things um so I I think you know at some point there has has to be a moment where we decide that this can stand on its own two feet without Nia and so you know something that Amanda and I have talked about is for adoption you know as I said at the beginning when we first started talking about the surveys our whole thing was about coming up with ways to drive adoption it wasn't about coming up with surveys the surveys were a means to an end and they ended up taking so much time that now we're at this point where we're kind of like we hope that this doesn't lose momentum we hope that people don't feel like the

surveys were it and it's done and then they're just like well someone figure out adoption and move on like this is what it's all for so now we have to solve that problem and we have to think about how do the people who are interested in this topic continue to work together and I agree with you that there needs to be some kind of centralizing point that people can focus on I think the challenge is when you have shared ownership how you do that and how you pull that together particularly when you don't have a lot of resources behind it so it's definitely something that we have to think about I I agree with you

completely on that front but it's it is going to be a challenge and I have a question back you know in a way I'm guessing that it's probably all of the above but you mentioned that your VP of Regulatory Affairs is someone who is coming to you but I ideally it would be from a act like the a different business perspective coming to you and saying for our reputation or for the value of our product or so on we are also concerned and engaged so my question back is you know what is the way to reach that VP is it you going and and talking about others in your industry like peers that are doing this

well and what they're gaining from it is it just more publicity like what are what are the mechanisms that are going to be meaningful for for those BPS okay so two points we're gonna we're gonna mik you for those watching at home thank you so two points for my industry we are pretty much all year zero companies on this um so we're probably not the best example of that I will say one thing though my VP of Regulatory Affairs would not be talking about this if our CEO and board and everything were not asking questions about it it's just not on her radar it's just not there so it is being talked about at that level

and for us that's because of FDA but from AC from across the industry um this comes back to I think we're still in the security researchers and technologists talking about this and the com's piece that we need to move on be it when we take ownership of it ourselves is how are we showing the business value in this and not in you know return on investments or anything else it's we need to keep really simple messaging to you know sea level people that here's the impact that this will have and like a company like Microsoft is probably a great example of that that it has really been ingrained into company culture and you know you guys probably have

experiences from 10 15 years ago when you started that process and how you sold that and that's something that we need to then take over and say okay this community thing that started with the ntia but is now evolving into a a a community-based set of resources or working group that's going on how do you sell that to your company how do how do I as a security person within a large corporation say no longer do I want to try and get involved in this as just me I want my company to support this I want my company to send me to the meetings I want them to put 105% of my time to that

and to that I have to sell a business case first not a technologist case and that's the messaging that would need to change the work can stay the same but the messaging would have to tweak a little yeah just I'll throw in real quickly uh since the car guys is not around um uh in the US do and Nitsa are there's there's signs of progress there as well and not as far along I'd say as the FDA if I had to somehow measure them but there's some you know the the vehicle safety Regulators are looking into it in the US so that's a good sign as well and I will also add that you know

art has been doing this for a very long time and if we are busy at the Department of Commerce on cyber security C is really busy right now and the fact that they're engaging in this I think is for me a signal that says that uh at very least they want to make sure that we don't ruin it but I think that it's going to be uh something that they can then take to the broader Community uh together with you know organ like the Chamber of Commerce and and we're also talking to you know the auto trade groups uh who like to communicate and be able to speak as one voice to their regulator to say hey hey we're moving

forward on this so think about what we're already doing before you go and impose something that may be quite dangerous because early discussion that we saw coming out of the auto sector um you know there was never anything very official but there were some signs that said that they were proposing you know treating security flaws the same way you treat uh some of the existing safety flaws um which are huge events for a company uh and they're they make a company not want want to learn about it and not want to affirmatively

engage so there was discussion about uh the disagreement between the industry on when you report something to the vendor or company and then when you disclose it and we've seen a lot of back and forth between Google and Microsoft on this very matter so Google has that policy of you have exactly 90 days and then Microsoft has the policy of we only patch on patch Tuesdays do you think there should be more give between one or the other such as Microsoft shouldn't be so specific on patching or Google shouldn't be so stringent on 90 days or do you think that they need to find some sort of better Middle Ground what SE policy on this

I so it's a fun debate certainly um in a sense I don't know how super useful it is so so actually so here's here's actually to take your example I'm pretty sure if I remember after that whole thing happened Google modified their policy so they now have an extra to get a twoe kick at the end so that's an example of we have a policy we have a policy you each knew what it was we disagreed a premature disclosure happened and then uh somebody adjusted their policy so things things shifted to work a little better uh together um there are so many people in organizations there so many ideas different business practices different release Cycles we're not going to get

agreement I'm not even going to try to get you know what's the exact number of days uh we we surveyed a little bit and the average was in in the 60 department but that's that's just what's actually happening and it's not you know that's not a good a good number um yeah c c WR we write we write 45 days we almost never almost never do 45 days that's our we couldn't get back we didn't hear from the vendor in 45 so we'll drop 45 on you instead of zero um so it's a fun debate I'm not sure how super useful it is to try to figure out what that embargo period should be I think you're not

going to get agreement there's a lot a lot a lot of vulnerabilities there are too many to have to really almost care about individual ones it's very fun too don't get me wrong I love it sort of but there are so many we need to just it needs to be all this be part of process not hey that specific thing that Tavis was yelling at Microsoft about on this day was like everybody stop and pay attention to that it's a there there's more more scale than that so yeah I'll just add that I think um I I'll just said that I think that um re the reality is that a disclosure is a collaboration and in any

collaboration there has to be some give and take there has to be willingness to find common ground and so that works Prov provided that both sides are collaborating right so I think the reason that you set a timeline is exactly as art said for the situations where people are not collaborating when you're not getting a response and nothing's happening but I think if you are getting a response if you are finding that there is engagement then the reality is to hold on to a deadline for just like because this is what our deadline is and we're going to do it is actually not putting the users first it's not really sort of thinking about how you it's not really thinking about

the risk model in the big picture sense and it also is not going to create um the the trust and the benefit of the doubt between the parties that is going to lead to the best possible outcome so so we like just I'll just give you ours and we work very closely with s we push all our disclosures through sir we do a lot of disclosure either for our own research or for members of the M community so our published policy is we go to the vendor we give them 15 days and then we go to se and we follow CS 45 days they get 60 days in total but what typically happens is if there is

engagement then we talk to sir we talk to the vendor and we agree mind that makes sense and it's normally less than 90 days depending on the sector and how complex it is but we are all agreed that we'll be flexible because at the end of the day that's the best thing for the users and that's what we care about so would you say the compliation would you say the specific complication there was just then Google and Microsoft not collaborating well together yeah ABS 100% it was two big companies not willing to bow to each other sorry and I I think the people yeah if you start with the framing that we all care

about g go back to it now so who here oh go ahead I don't actually particularly have a question I wanted to get Jen to expand a little bit more on theer because it does Archer obviously that goes without saying um but the the sort of change that the researchers side needs to be cognizant of because when you're talking about Automobiles and installations in oil refineries those are not things that fit a hard and fast 90day completely agree and even once you've gotten past the disclosure you look at car I mean I can go buy a car today that has not had its airbag replaced or does not have all the modern safety stuff so from the researcher side

we're in a totally new yeah area where we can inadvertently put people at risk yeah while following all the ethical rules I I agree like I think it's the wild west all over again and I think what's interesting is that we've entered a sort of new um point in the culture of the community I kind of think of it as um like sort of uh the security Community 2.0 which is terrible but it's almost like you know we had our generation Alpha and now we're in a in the sort of Next Generation and we we've we've evolved or we've matured out from being from skites who were just curious and wanted to know how worked and test

limits to suddenly being like people who are buying cars and putting our kids in them and going holy like there's actual implications with this stuff and so I think you know a lot of the researchers I talk to feel the weight of that they actually you know going back to that 15% caring about the money a lot of the researchers I talked to are doing this because they want to save the world like that's what it comes down to they actually really care about the impact protect consumers yeah and so oh thanks um ker is not a good good example of this he do not want to save the world um and so I think um I think that you know

that is difficult like we are sort of making up as we go as Josh said like we have to find the new paradigms of what that looks like and even at rapid 7 that's a process we're going through right now is as I said we have this disclosure policy it's been a disclosure policy for the five years I've worked at the company and right now we are in internally looking at that going we need to now have a category of this is this except when it comes to things that affect human life and then it's going to be like what does that look like how do we set timelines when you're dealing with cars right it's a totally different

Paradigm we have no clue what that looks like but I think the only way to get there is to work together and when I say work together I don't just mean within the research Community although that for sure but we have to work with the automakers we have to get to understand what their process is we know what software development looks like a lot of us have worked in software development we don't know what autom looks like have no clue so and and like and the same for medical devices you know we we need to understand what that involves and the only way to do that is to work with them and have empathy for what they're going

through and if you approach that as you know you're the bad guys and I'm the good guys you're never going to achieve anything I said that the right way around right yeah um sometimes a little selfing the um yeah so we like we really need to find that Common Ground I think that's the first

step I think I do I get an award for that that's impressive so uh one thing that that art did not mention that that uh some of the the the smart people in his group I've been thinking about is there will be instances when you will will disagree on how to handle things and that doesn't that still doesn't mean that it go back goes back to the wild west and we should have Anarchy there still are uh some things that we can do to still have good practice in that case or do you can I put you on the spot to share some of the thoughts that you guys have had uh so this is this is disagree the the

agree to disagree how do you make that constructive um well you may not I mean it may come down to not agree I mean simply not agreeing so the the Google Microsoft example Google Google win right that happens um sometimes one of the parties involves holds their ground is tired of waiting uh values puts different value on the impact of the vulnerability or the the safety impact or the whatever it might be uh you know chances of it being actually attacked in the wild chances of an attack being successful ful time to deploy fixes uh are attackers going to bother with a very complicated attack when fishing emails are still working um what kind of attack are you looking at Broad

opportunistic attackers or nation state targeted attacks people have different opinions on all of those things and and that can come down to how bad this they think something is which causes them to say it's fine to dis disclose now or it must wait um to to some extent the dumb answer is you have to accept you're going to lose out sometimes your opinion is not going to carry the day uh and be able to deal with that you know it's it's it's largely if you have the information about the vulnerability you generally legally have the ability to publish it uh at least in the US I'm not sure that's quite the same thing in in the EU

but um you know publishing details of of a bug is is Free Speech in the US attacking someone with it is not is is typically uh illegal um so I don't know what to say really except I mean it's certaint I'm trying to think of our cases we accept that um we will ask people and we will ask nicely and repeatedly and make good arguments to do certain things delay or extend or hurry up and publish or hurry up and get your fix out uh and we try to not surprise people you know if we're if we're going to be the ones pulling the trigger we will say look we're publishing in 24 hours you know we're

not done done negotiating that's that's going to be our date um so you can not surprise people and you can accept that you are not going to win the negotiation every time um you can negotiate of course and try to achieve consensus I don't know that I've got any any magic that was it was the the not surprising and the try to negotiate with things that I was sorry about that have to take notes for me now um along the lines of how to speed adoption or make it uh more companies adopt especially those companies outside of the technology industry you need to actually take a step back and take one of those companies and have them look at

this this is terrifying to them they've been making you know widgets for all of these years and all of a sudden you've got a bunch of crazy black hats running at them regulation coming everywhere and actually making them understand that the security research Community we're not coming to get them and obviously as your survey shows we're really not coming to get them we're really trying to make a better a better world they're seeing the guy hacking an airplane they're seeing you know these crazy people you know trying to free the world's information they think skates and yeah yeah making them understand that this this room this is actually who's out there trying to help them not hurt

yeah thank you any media story with the picture of the scary looking right it's gone past I think I saw a mask recently yeah I mean how many people type wearing a b lava I always do I I actually can't see the keyboard properly without one so um yeah I I I think that there's fear on both sides and the problem is fear makes people behave in um irrational and defensive and awful ways suffering something right and empathy just completely breaks down I agree so who thought that what we said today was fairly common sense so that's actually this is this is this is Ries I think that it means that this is not a contentious issue uh who

who has worked with a uh vendor before too to actually say hey I found something got a couple hands has anyone been on the receiving side more uh I won't put anyone on the spot but is there anyone who's been on the receiving side who wants to share some experiences or React to what we've said

today yes and it's hard without a process at the at the simplest level because there are people out there that want to do things and there are so many Industries not just my own but others as well that if there's not a process there when you get a vulnerability in report into your customer service email address or something and you have no idea what to do with it and the company has no idea what to do with it I worked in a company a number of years ago where I got an email about it because in my internal company profile thing I had interested in vulnerability disclosure on it and somebody had literally Googled on our internal portal and it came to me

so that's where so many companies are so without the work that you guys are doing it's hard yeah and we need to make it easier because then it becomes easier for everybody I mean the reality is that like people are super busy they have processes of how they like to do things like generally people are not bad people they just they have things that they've been told these are your priorities these are your goals and you're happily working away as an engineer and you've got a delivery deadline and you know that if that delivery deadline slips then that has a high impact for the business and that's a problem and it has a high impact for the customer as well

and so when somebody comes to you and says hey we're going to blow up your schedule we're going to divert you onto this other task and this thing that you thought you would dealt with ages ago and it's going to completely impact all your other work like that's frustrating at best and at worst it's like it's it makes people insecure it makes them anxious it's difficult there are no like people doing this stuff or typically there are not a lot of people doing this stuff who are just about it and we're the same get vulnerability disclosures and you know there are times when I have to have a conversation with the software guys where I'm like okay you've done the

patch have we put it out yet and they're kind of like oh we have a release going out in a week and I'm like you get that we're a security company right and it's not like they don't care they do it's just that people have this whole thing of like this is the process that we go through we always do a release on a specific date Microsoft Patch Tuesday like it's the same mentality and so you you just that's how you think and people don't like change very much so when you're like Hey we're going to do this completely different thing that you've not done normally done they kind of sit there scratching their heads so for those of you who would like

to engage further uh there's still a lot of work to be done uh please talk with art talk with Amanda and Jen talk to me uh tomorrow the eff is hosting an event to continue this discussion uh I'm particularly proud of the fact that the eff and the Department of Commerce have said listen this is something that we need to actually work together on uh and and and you know find some solutions that that work well for all involved uh and I'm going to give a very quick plug if you really like the idea of you know talking with other security people about processes uh our assistant secretary just announced our newest uh multi-stakeholder process on iot

security uh and it's closely related to this uh so the challenge is there's all sorts of smart consumer stuff out there uh and not all of it is built for security so how do we promote consumer awareness of security and how do we reward manufacturer attention to security we're going to be starting with aftermarket security taking a page from our colleagues at the FDA and saying listen patch ability is important but there's no real understanding of what it means for something to be patchable so let's have a discussion around the technical definitions of patch ability uh and then figure out a way to condense that into a way that it can be easily communicated to the consumer the

consumer has something quickly to look for here are some small words that I can understand and look for in a package but they are supported by a well-defined technical description that can be used to build standards or built to spec on so trying to derive it from both the demand side and the supply side uh and if you're interested in that I'm very happy to talk further so please engage uh thank you for your time and attention I'm guessing that you have a last word because you always do I know I'm sorry I really I I wanted to not in this case but I just want to thank everyone who's participated in the process thank you

very much and I want to thank the three people up here who have given a lot of work they have well their day jobs are are far beyond what they are already have time for and this is something that is on top of that uh and it works because people who care passionate about these issues get engaged so please I invite you to also get engaged thank

you

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e e

want to thank our sponsors especially vers Sprite productivity tenable Amazon source of knowledge uh who amongst other things uh they do to keep the uh uh make the conference work uh have donated us uh a number of books on devop SEC and Docker security which we will be happy to give out to people who ask uh excellent questions uh if you enjoyed today's panel or want to leave comments you can leave speaker feedback on the .org site and with that please welcome our [Applause] panel this is exactly how bide [Laughter] started then don't give anybody a chance to can I let me introduce folks um we are the board of directors of bides um Global it's sort of the parent Corp

bsides events are run by the people that run the events we don't run them from the board but you have oh yeah that's that's right we need we need the mic so the so the stream works oh no no I'm already freaked out because I got a wireless mic I need wires I was just in focus focus no cat squirrel um this is what our medians are actually like they did it is uh except we don't have eye contact so it's even worse here so the board um consists of Mike Dawn David mortman Michelle Clinger and myself I'm Jack Daniel you guys want to say a couple of words and then I'll kick into my bit so why don't you guys say a

little bit and then we'll roll good me yeah always me um so is this on is this thing on okay um uh yeah so Michelle Clinger my role uh I'm I think the second to last person to start to to join Dave you you were the last uh and my role is actually the event coordinator liaison so I'm the one that manages for the most part the info uh email address and then when you reach out to us I'm the one that uh you speak to uh I coordinate um I'm the person you first Contact for conflict resolution um and then these guys we just manage uh the bigger picture but that's my role I'm David mortman I'm the I'm the

newest member of the board I've been here with the board about two years uh well there's there's a mythical fifth member of the board uh who mostly has an autoresponder email saying he'll get back to us eventually uh I was mostly added to make sure there was an odd number of people on the board and I'm really odd so it works out well uh my name is Mike Don I saw a long time ago uh there's a Twitter conversation that went on about uh uh putting on events and someone said hey should put on an event I said like putting on an event to hear yourself speak is ridiculous so uh over the weekend we got together an idea of

putting together uh a framework for security friend conferences following uh kind of the bar Camp style approach and uh originally it was called security Fringe Based On The Fringe Festival in Edinburgh Scotland but apparently many people are not aware of this and people someone slapped me and said people know music more so than uh uh you know obscure although not so obscure uh uh massive events that occur in the world uh and then uh so that was Saturday and then Sunday we rewrote everything as security bsides uh and uh and got things going uh got things going from there so I'm I'm Jack and um I get way too much credit for what happens in

bsides because it takes thousands of people including every one of you here to make bsides happen particularly these folks um and so what we're going to do is those of us on the board are going to chat a bit talk about what we do a little bit of history and then we have a couple of uh organizers of large events that are going to join us and this we want this to be a conversation with the bsides community particularly those of you who organize events or want to organize events but we want to hear from everybody because we're trying to figure out the direction a couple of years ago three four years ago now we had a big

discussion about whether the organization should have a big role or a small role and uh the decision was largely small role um but dealing with some things and we'll we'll talk about that but let's start out with um uh banshe jev Southwick will be will'll be hopping up later as will Thomas Fischer um Thomas runs besides London which is not small uh and Banshee sort of runs this giant not sort of Banshee does an awesome job that's why we're here because she makes bides LV happen along with Kelly and 200 other people um so past present future um you are here you could help so I have a question here how do we go from this um was anybody else at the

first event you were so uh the 303 crew led by Chris Nickerson rented a house uh of like six miles that way um and the air conditioning didn't work so well and so we ended up taping garbage bags over the windows and doors to control heat and putting uh standup uh air conditioning things um and then we pulled this off we had some great content and conversation some amazing things happened um and one of the keys was this chill out space also known as the kitchen and living room and and some great panels uh the first one was not without incident um the squirrel is too long a story to tell sober uh but we actually had a fire and

we did almost burn down we left one person in the house the last thing that was said is just don't burn the house down um it took about a full day to get the smoke out uh turns out if you have a big spool of electrical stuff with power for something like an air conditioner running through it you shouldn't take a door off a heavy wooden door set it on top of it to compress the field down and then let the door burst into flames turns out that's bad so anyway um whoa at least not intentionally so this is us um this is event number 271 globally since uh July of 2009 today is um event 272 is happening in

Prince Edward Island in the Canadian maritimes they didn't think that people were you know if they were still in there anyway 272 events um as of today besides events have been held in 94 unique cities in 25 countries on every continent except Antarctica uh the growth rate is amazing the first year they were like five or six um in 2014 there were 60 events globally uh last year we closed out at 62 U bare minimum of 64 this year but as a lot of them stand up quickly um I wouldn't be surprised if we hit 70 events this year uh there will be at least 12 new bside cities in 2016 uh the green dots are cities that

haven't held events before um a couple of these are very recent because I did this and then drove here Tel Aviv Athens uh Kiev this is an older screenshot um Cusco Peru medine um a handful but that's um you know it something happened as I say to my H yeah Belfast is uh is is not on there yeah this is an older screenshot sorry this to give you an idea this is like a five week old screenshot to give you an idea of the rate of change so the global organization the mothership who we are and what we do um so I'll start with what I do which is I kind of play back stop to Michelle she

does the the bulk of it and she's the one you want to talk to you need to keep her in the loop if you're running an event even if it's your you know fifth or sixth year give Michelle a heads up you know dates and things and between us we make sure that um that we add it uh you know to the Google Calendar and she keeps the the front page of the wiki up so that people can know where to find things um we do coaching and mentoring I'll let Michelle talk about that uh because she's amazing at that um and then we do um uh conflict resolution Big C uh and the conflict resolution varies

dramatically sometimes it's just helping people figure out how to do a transition uh sometimes um I don't know if you've noticed this but sometimes there are people in the hacker Community who um are kind of interesting to work with and every now and then little bits of drama happen we're trying um well is on tape so so yeah I was going to be more graphic in my description but uh so stuff happens and sometimes these conflicts are you know people two people that want to run an event in a town and or you know a community and one sort of drags their feet the other one's gung-ho or they have different differing opinions uh some sometimes it's

transition sometimes it's we started out as a bit of a um bratty startup thing and thumbed our nose at um Global corporations that happen to run other events that might be happening uh down the street um and at a few other conferences and uh we've got fantastic working relationships after Mike and I and and others um I landed that with a lot of other conferences so you know we've we've done things like work with shukan to make sure they we don't damage other community conferences we encourage people to work with them uh we bridge the gap with RSA we bridged the Gap with infosec Europe um we have connected with other things we bridged the Gap and

worked well with black hat trainings um and we've gone from a hostile relationship with black hat and ubm to um they're still not cool with the fact that you can get what you can get here for free um or very low cost if you're a don um and then they do their thing but they have they they run a fantastic event and it has its place and people that think it's failed to not good at math because when you get 14,000 you know when you get 12,000 people one year and 14,000 the next year and I haven't that's not failure growth at that rate is not failure so it's a great event um it's not for everyone um and we have a we

have a truce with them basically I would like it to be a better relationship but you know they're making money it's cool so we put out fires some of them are big and some are little and and um what else yeah oh so there are a couple of resources we have uh currently there are only two events that use the kit sofos has donated tens of thousands of dollars worth of networking gear over the years um the equipment that you see around here is all sofos wireless access points SOS Gateway controllers that do the wireless and firewall and they also do qos and little application identification we don't block anything outbound uh except SMTP if you could cuz

believe it or not yes people have spam bots on their laptops still um so we just do that to be nice uh but I run I Warehouse that equipment and transport it currently besides DC and uh Vegas are the only ones that use it so of course they're 3,000 miles apart and Jack has to drive them back and forth and back and forth but I'm okay with that and I'm gonna shut up now believe it or not somebody else say something okay all right so um not before everybody showed up I asked the question um I didn't think this many people would show up um because I who cares right what do we do you throw an event you show up it's fine

so I'm I'm actually glad to see a lot of people and I wanted to know why you guys uh wanted to be here and so it was some of you are coordinators and some of you are um thinking about throwing an event uh so here's a couple of things that uh I do when somebody steps up um and decides expresses interest in throwing an event um so this is going to be a level set so all coordinators have been told this with the exception of one or two that may fall through the cracks and go directly to Jack and bypass the governance process but um so here's where everyone is supposed to begin and then events tend to customize their own

they may do things a little differently but here's what happens you send an email to me uh and you say I want to have a have an event I get together with you for an hour call and I go through uh depending on how much experience you've had with throwing an event I'll I'll get into specifics about event uh planning or I'll just go high Lev level and it's really the dos and don'ts for bsides um expectations from you as the coordinator from us as the mothership and if you have any questions and so the topics that we usually go through are picking a venue um guidelines around that picking a date uh because there are guidelines

around that you don't want to step on somebody within a certain geographic area um another thing that we conflict Resolute often uh is putting two groups together saying figure it out um you really you got to you got to you got to care for each other um so it's the venue it's picking a date whether to charge what are the guidelines for charging um if and if you have any questions about that just throw that up um uh uh guidelines for cfp guidelines for getting sponsors um talk about budget a little bit What's Your sponsorship kit the different options people have used um what else uh uh uh and collecting the money right that's usually about 50% of the reasons why

events don't actually happen uh is because the collection of money is the hardest part and so you step up say you want to uh throw an event I check to make sure nobody else has expressed interest if they have I'll put those people together and say go go have a talk maybe help them out um so it is kind of a first come first serve basis um so whoever's expressed interest they're kind of like the lead um and if somebody else expresses interest and that lead decides for whatever reason life gets in the way they don't throw the event then that person kind of takes over so those are the topics so we're all on the same page um now here are the

dos and don'ts uh first don't um the C well first must have the cfp process the call for papers must be open right so that's the first beginning you are not you should not invite the entire invite your entire speaker lineup the whole in initially of bsides was firsttime speakers uh people who can't fly out to to other conferences to submit right so this is really was to to help the local community have a voice so that's the first requirement if you want to you know invite a keynote that's fine but the cfp process should be open how you vote for it is a different process we don't care uh online Survey Monkey me I

was a dictator and bid DFW if I wanted to pick everything that was up to you that part um and again it can be AED of course but that's the first rule um second rule is speakers uh sorry sponsors um sponsors cannot be guaranteed a speaking spot that is an absolute no no um so they can submit to the process the cfp process but they should not be guaranteed as being a a sponsor so that's another item um third item and the the most egregious uh uh offense you could do is provide the registration or the participant list to the speakers that is absolutely not allowed sponsor sponsor sorry um yeah we do not we do not give

them that information we should not be giving them giving them that information um and if we find out then we kind of have to again big see conflict resolution find out why that's being done because again everybody is on the same page when they start go ahead yep yeah this is this is an open conversation by the way but not not preaching to you so if this derails to do something else let's do that so I'm uh an organizer with besides Rochester and we ran into that exact problem where we had a sponsor like well we want your list and we're like well no but then we had this kind of big internal conversation like what if they

voluntarily wanted to give it and we'd let the sponsor know that right is that is that acceptable under your terms opting in opting inar and that's what we figured and we didn't actually go the whole route anyways because we ended up telling the sponsor no but we wondered if that was actually yeah as long as it's very clear absolutely nothing wrong with that there's depending on the level of organization too um for those of us that are in organizations that are 501 c3s or registered Charities or the equivalent in your country um you run into a set of regulations where um I would be hardpressed to defend um effectively selling an email list of our

participants um under uh unrelated business income tax so but that that's a whole another conversation yes let me let me let Thomas so I've had that discussion with sponsors number of times I have a bigger problem is if I give out the list without permission from the participants I could actually go to jail and get fined because but so if I go back to the sponsor problem right what I explain to the sponsors is that that's the whole point is you're coming you're going to be sit setting up may you know you have your table it's like talk to the people the people will come and talk to you if you present yourself correctly and you ask them and you can get their

names and and email addresses that way it's there's a lot of other SP don't get it one of the one of the things that's worth pointing out with the sponsors is um if they're not familiar with bsides they don't always understand this one of the things that sometimes helps is to point out that it's not like um some of the commercial conferences where as a as a sponsor or vendor um all they want is your money they don't care you can't go to the talks you can't go to the parties you can't do anything right we just give us money stand in your booth um until feet ache and then keep standing that's all we want from you it's like if you stress

no no you're part of the community we want you to submit talks but probably ought to have your engineers submit talks we want you to be engaged in the conversation if something looks interesting join because we want you to be part of the community and sometimes that makes a difference and other times it it just doesn't get through to them but but so um I I I uh help organize La besides and my and and also I do uh Isa conference and my experience is that you know there are two types of people on the sponsor side there's like the marketing people and they have their metrics and their metrics are leads and somehow they think a name is a lead

which it isn't it's just it's just a name um but but that's their metric right um and then the the other people that really engaged with the community like you know the engineers maybe their sales Engineers are really interested in security they kind of get it but my experience doing this for the past few years is that the people that are trying to go Shake You Down to get the contact list and maybe they'll sponsor they're not going to be the right fit they're not going to engage they're not going to participate they're going to send people that show up and just are on their phone reading email the whole time um so for me I I just go to another person that

organization that will support support me or just skip them because at the end of the day we don't need a lot of sponsors for a really good con

he just wanted to find out if it was allowed yes get let's get jev's perspective okay yeah so one of the things that we do here in Vegas is we go after the recruiting budgets instead of the marketing budgets if we have a contact in marketing we'll say yeah this is really isn't to kind of markting kind of conference because you're not going to be allowed to sell anything you're not even allowed to present that way because we're a 501 c three think of it as a CPR or an N or uh uh NPR or a PBS ad right you can say what your mission is but you can't talk about your product or your

sales so put us what yeah yeah yeah we get the HR budget or the recruiting budget because they want to come meet the engineers that are looking for work and maybe poach somebody from tenable because they know that Jack's the best at what he does I mean but they can't pach jack obviously but we found that that's a better approach than going to say some of the other companies that say well so when do I get your email list which is almost anybody that we deal with in marketing so yeah just consider going after HR and recruiting budget instead that has been super successful the past two years um going after the like tech systems and

XYZ name the list that is I I can't believe how they just want to throw money at you just to just to be there I want you to keep going because you're doing I just want to point if you're not aware of this there's a two-day track here Higher Ground which is a career development track it's not just about recruiting pop your head in it's it's over there right straight across um and see how much activity is driven and the companies names that are in there that are interested in spending money here for recruiting and think about that for your event and it it does depend on your community right uh it's easy to Target

San Francisco or uh here when everybody comes here it's not going to work in every Community but it's it's worth an angle uh it's worth it's absolutely we couldn't have we couldn't do what we do if it weren't for recruiters I'll point out that uh that at least one of the vendors here on the floor yeah at least one of the uh sponsors on the floor over here uh has stopped recruiting at the other large conferences here in town this week and they're recruiting here only because this is where they're finding the talent so word is getting out at least about some of the bsides events that if you want to recruit you go to you come to

bsides not the local other events that it's often paired with because the people who they want are coming here how does that change

dnam what do you mean

to speak to speak to the the recruiting um and and make it personal for the people that give me a paycheck twice a month our vice president of HR was here yesterday our director of talent acquisition was here yesterday we've had four recruiters here including our Le lead we also have had some of our developer team come in but they're not working the event um we've had a couple of our sales Engineers come through one of our strategists um and so like a lot of other companies that are here that you've seen the names at bsides all over the place I work someplace that gets it um but yeah you do see recruiters you know and um if

they understand what's going on and you tell them you actually want them to be part of something if there's a talk they want to hear go see it um so it it's the conversation you have to have with them right I mean just look around the floor out there and you'll see there's a bunch of folks out there who are not who don't sell security products right I mean they are here purely to recruit and they are they have HR folks they have hiring managers they have recruiters out there because that's what they hear

vet so yeah you get very different people than if than even are showing up down the street so this is a good segue into setting um uh expectations um so for those that don't know or expressing interest know nothing about how bide starts the mothership does not determine what events take place so we don't look at the map and say hey we're missing an event here it's whoever decides to step up and say I'm crazy and I'm G to plan this event that's literally what it is um though we have said we need to have one anarctica volunteer please so so some people think some people email the mother ship and say hey actually we just

got one yesterday and says hey you know there's not an event here maybe you guys can consider it and like no that's not how it works if somebody steps up to do it then it will be done and Michelle what what was that City I don't know wait the respon that's a great idea you should go do that so um so now to that point because everyone with th with that base guideline I gave earlier everyone sets up their event their way right so they should know what the local community needs who to reach out to um we don't mandate any of that we don't vet somebody to determine if they're good enough to throw an event right they

don't need to have uh security you know uh experience they they're just the one the group or the people or the person that have decided to take on the initiative to throw the event

so money oh yeah the people yeah know I was going to get to that so so to this point um it depends so how that that whole situation set up how do you what this what the agreement is with the vend with the that's all dependent on how they do it yep no yeah yeah um and it also depends on you know how big your team is uh D DFW we basically were just a two-man team for several years and um we did everything so we may not have had the time to you know walk hold the hand of those sponsors the recruiters other than tell them there's a ton of people here here that you can hire come

and they they've done pretty good um so uh the mothership really does nothing um we just provide guidance some collateral when you first come on I send an email with a slew of links and um a huge attachment um and you're off on your you're off on your own um we don't collect the money either that's what I uh alluded to earlier you have to figure out I'm saying you you know Royal you have to come up with a way to collect the money uh Bank account U you they either give the money to you directly there's tax implications and I go through all that uh uh in that in that first hour talk but that is all on the

people who step up to do the event that's it yeah the only thing we provide is the wiki the wiki oh and you know what the Wiki burn the wiki down and then there are people who complain about the wiki cuz one of the requ cuz one of the requirements is you have to have a page on the wiki you don't have to do duplicate you know if you create an your own domain but you have to have one because when that domain goes away we still use that think of the wiki as a way for people who are thinking about sponsoring to look at it and go wow look at all these events that have occurred

it is it it's essentially an archived marketing list right of bsid so as painful as it is you have to create a Wiki page because that's but we've Jack and I Big C conflict have to address that particular issue and it it needs just some basic information you would not believe uh how often I mean it's not like daily but uh every few weeks somebody reaches out to us and says I would like to give this event money and I can't figure out how to give them money all the time all the time all the time contact information should be at the top that should be wa when where yeah uh you know a couple of sentences

at most uh a link to website and contact information there's a template that you can just fill out it doesn't have to be much but yes we continuously have to help people find a way to give organizers money that's kind of problematic you should have learned that in your lemonade stand when somebody wants to give you money make it easy um so other than that so people um then I keep a running list of all the events and contact information and and run the wiki and so one of the things that's and now I'm going to toss over to Mike but a lot of the things that have come up um hold on a second uh regarding

the contact list this potential agreement that we're thinking about um the wiki all of these things that come up that seem a little bit boxy or or legitimate or or actually have our together um those come up because we had to address the conflict so anything we talk about perfect example um somebody steps up says Hey I want to throw an event EV and after a couple of months if I don't hear from you I'm going to reach out and say hey no pressure you still still planning the event no I'm not oh okay nobody somebody else stepped up they're interested but that first person created a ton of accounts Twitter accounts domain name right just because they were preparing

now okay it's time to hand those off they don't belong to you right it's the communities argument that's mine those are my accounts I'm going to do it I don't want somebody else to do it and now Jack and I have to uh mediate that so one of the things in the in the convers ation that we have is hey guys understand that if you choose to no longer do this event it's not yours you got to hand it off um so a lot of the things that we talk about and that Mike's going to uh bring up is because we've had to address it um it's been a problem a lot of infighting um stuff

like that fighting between bordering cities so we're trying to figure out a way to be small business or small government but then people are asking us to be big government and so this is this is where we're at and this is kind why we decided to do this and so I'll throw it to Mike just just I I I do want to stress um conflicts are actually fairly rare but we're at 272 events and we're all volunteers and uh so if we get a conflict a month and some are big and some are little that's a substantial burden on us and we try to keep it structured in a way that we don't have community

policing in public uh because you know what happens when hackers have a public argument it's not pretty yeah I I think that's a really really important Point um so like when we think of we we talking right before we got started uh about what bides is and it really is a community it's a philosophy and it's a brand right and these have evolved and changed over time and when we got started the very first thing we did was we put together Wiki which we now wish we never did the kiss of death right this terrible terrible Wiki that we're tied to but it's we all started with the idea that none of us know how to put on

an event so let's collaboratively add content someone says hey I know a couple companies that make t-shirts hey I know I've written an event agreement before hey you know I I've copied something from something else I've done somewhere else and we kind of collaboratively add that together so that the total population of people have a great set of resources to get started to the point where now Michelle is of coaching people through the process not just her it's other people as well and so another thing to consider is like what is a bsides event there was a a bsides event in Gold Coast Australia which was people saying Hey I want to get together for

two hours at a bar and put something on and people like oh that's not a bsid event like why not right and there have other events that have started and the next year they're like yeah no more of this bside stuff we're going to call it something else we're like that's great so we do is we see the community and the philosophy and the brand evolving and ultimately I think we'd like to just continue to see that evolve right so the only reason why I think we all got together was because a lot of people started looking and saying hey uh you know we're looking for guidance or or whatever it might be and and we we

it's I don't even think we provide that guidance I mean we just basically provide you know call people up and say hey sounds like you two have a problem or something like that a number of things have come up over the years that really have potentially put bsides in a position where it could have burned down altogether as a as a brand or a philosophy you know people saying hey I didn't like the way this sponsor treated me so I'm going to go out and publicly out them uh and then maybe they never want to contribute to other events again right there's a ton of event things that have occurred so one of the things that

we want to raise as as a possibility is that maybe besides as a philosophy and a brand has outgrown you know the or from this perspective right and and maybe you know we it becomes the community that uh that that handles conflict mitigation and the becomes the community that addresses and triages issues that arise and becomes the community that addresses these types of things and we just continue to add that shared set of knowledge to the wiki that if we're looking at something like a sponsorship kit it's just as much as important as maybe an event agreement as just as important as something else that that someone might contribute so we want to kind of start that conversation going

about how the organization kind of continues Beyond uh uh a set of people that being said there's implications of that and I think Jack keyed off to one of them right um if uh if Michelle wins the lotto and goes off to Tahiti you know who's going to be calling the events to to help walk them through these things I think it can be the community but I think what we need is for that to occur we really need people to stand up and say I'm going to help out in these different areas and that's occurred considerably in the past so going to kind of start that conversation going and and look for anyone that has

thoughts or ideas so if anyone does you know we're looking for you to raise your hand and and and contribute ideas because that's how that's how the whole philosophy of besides continues yeah I mean I would say that the conflict resolution piece really falls into into two large categories one is just you know what's the wisdom like what have you learned from having other organizations other conferences that you've helped to organize like you like the question about uh how do we communicate well with sponsors to understand about sharing you know attendee lists like this is a known unnown problem space we've dealt with it before we can help out the other one is really being and this is where I tend to

have most of my involvement these days is as being the four of us being somewhere between disinterested and uninterested parties uh in the conflict at all uh you know there have been a couple instances where there was two different groups in the same city who are fighting over who gets to put on the event neither of whom necessar we making any progress um we don't care I don't care who you are all I care about is you know it's really plain Dad no I have a six-year-old and a 14-year-old and half of the conflict resolution is just saying really talk it out yes there's a lot of um a lot of times when we get pulled

into the big ones it's it's the lack of adulting um there's a lot of ego unfortunately that's a shock In This Crowd yeah um there's a lot of um I I there's something there's something that happens when one becomes a coordinator um I man I I manage my team uh really good with um lots of Weaponry but um what ha there's a power that gets bestowed on someone they they they feel this influx of power and um internally it starts to fracture uh and and and that's again something I bring up be aware that you may not agree so put a good team together or don't put a team at all um and so we're at the point

where we're all volunteers we all have personal lives we're all burnt out and the idea of trying to become more structured we're not going to please everybody there's always going to be a set of coordinators or people who have what they think bside should be it's it's going to get ugly um as we try to quote unquote fix things so we're looking to walk away and let the community commute yeah one thing I have to say is that even when we've gotten pulled into these you know adulting situations it is because the people are really excited and passionate about what about doing an event um and it really turns into really they just want someone

else to help them you know resolution and then they walk away like adults I mean that's the great thing I want to stress one of the reasons uh that we try to solve these problems quietly um it is because uh thousands at this point it's safe to say tens of thousands of people around the planet are part of this community and bad press travels a whole lot better we have built communities we have built friendships we have brands people that are that are close you know my bid's family is closer than my real family except for my wife and kids right and that's not that's absolutely true now when something gets a little ugly and starts

to get uglier and goes public that damages the work of thousands and thousands of organizers and sponsor of volunteers it makes it harder to get sponsors makes it harder to get speakers and so we've been walking this tight RPP of how we control this without being too controlling and you know one of the things we've we've done trademark Mike has gone to the trademark thing and that that's contentious that means lawyers um the draft for those of you who've seen the draft that's with us working for what seven months taking the lawyer words out and it's still a lawyer document so you know that's the real challenge we could throw it to the community for those that

have been part of this for long enough you might remember a couple of incidents that happened where the community policed things it over 100 124 emails in 24 hours on one of the mail lists so we're kind of hoping and there might have been a few raised tempers as I recall I was measured the entire time yes well you always I want to go back to the first question over here I was wondering I was wondering if anyone on the panel could contrast and compare with the organiz organizational structure of white club I'm sorry we can't talk about yeah rule rules number one and two uh I mean you know cannot emphasize cannot emphasize the what Jack said uh is the

is this idea that um that in the in the early days it was dark and maybe now we've evolved past the early days but it's really important that whatever path goes forward uh we all learn a bit will adopt a greater sense of of empathy and humility and collaborate on helping solve problems internally noilly I really want he's been asking for like the half hour I really want to reiterate that I mean the reason a lot of these conflicts come up is because everyone involved is so passionate about doing this well and one of the things that I've learned over the years with various organizations is that it matters so much because there's literally so little actually at stake um

which is another reason to keep the conflicts on the on the quieter side because in the end inevitably the all the parties walk away maybe not happy but at least accepting of the situation and then they may come back two months later and work together really well we don't want to uh exacerbate any problems or broadcast those unnecessarily yeah good all right so I have two questions um one is you mentioned um mediating uh same city conflicts yep um what are kind of the um the guidelines around uh or or or what loose rules do you use for um same city is it based on like demographics or is it based on you're talking about uh geography or

someone wants to throw up a city EV a bsid event in X City yeah so and maybe someone else wants to run like a neighborh like a county next door yeah um what we've the guide there is actually a guideline for this which is I give in the first uh um so there's two there's two pieces to that there's one um starting a bsides and picking a city um after you reach out to me I'll see uh if one somebody's already expressed interest and then if it's in a a state that I've that's has a city find out how close they are and unless it's several hours away we recommend that they don't do the

event um because or or explain to me why that why the new city that you're looking to do like a town over why um that would be beneficial what what is wrong with the first one that's not providing the value that you need another one maybe 30 minutes away or adjusted by six months or adjust it Opposite Polar Opposites of the calendar so those that's guidelines uh again we we really don't have enforcement so we're not going to say no we're just going to say here are the guidelines work it out then the other piece is when you hold the event uh so like like he said uh opposites of of of the calendar and you don't want to pick a

date where a surrounding State not just city state is throwing an event um that's basically the loose guideline okay well there's or the other issue is that comes up a fair amount is someone says I want to run a bsides I want to run bsides Podunk and then three and they say I'm going to do it in April and then someone else comes along and says well I want want to run a bides Podunk in April and you know how do we resolve that issue of who gets to do the event and the general rule is that it's first come first serve and occasionally it's been interesting with some larger cities actually this comes up hard this is a

bigger problem in the really large cities it's it's really hard in San Francisco Chicago New York Boston to find a facility that is not stupidly expensive and so we got people show up and say I want to run an event and it can take them two years to get an event together and then someone else comes along a year later or in some I just want to bring it to scale for a second because we're talking about cities but there are also individuals that step forth that say I want to run besides State and besides country right and there are guidelines around that as well um we we we are in the beginning we really didn't think

about it right because it was small but now because so many are standing up we we tell people don't do don't do country don't do State uh stick to city or county or what that town is known for um because if somebody else wants to do something in that country or that state well you just you just taken the name and now they're kind of lost in the in an umbrella so that's another guideline that we give I think we'd make an exception for besides Antarctica that have the con but it's it's worth noting again this is why we get uh very wound up about not keeping us you know not keeping that wiki page and not telling

us as soon as you have a date for the next year cuz you've been working on this event for eight months and you don't tell us and somebody says oh I want to do one and it's it's you know 150 miles away and they pick a date you're like well haven't heard from the guys in wherever haven't heard from the woman that runs this one and uh so we say yeah cool that sounds good good luck good luck uh we haven't heard from and so you shift yours two months from when it has been the past two years and then you're like hey what do you let people drump on our date for like cuz we're not

psychic or your parents so I I'm I'm curious I think that a lot of people here represent people that have have put on events or are are interested in putting on events and there's hundreds more that that really reflect that as well and I think it'd be good to get to get the word out but um you know who here is interested in participating with uh with Jack and Michelle and others in in triaging these things exactly because when you when you raise your hand we are going to Mark you for death and uh you had question yes so make sure that you come up afterwards I think this is critically important because in order

for the community to to to to to to support itself we need many many many more people like that okay okay all right let me stress this this is the beginning of a conversation we would like for those of you who are organizers to join us in the Google group you know mail list continue the conversation there uh if you'd like we can do it do some of it in the larger thing and email us and we'll continue the conversation in whatever form factor I move that we actually set this up in peer list and take the conversation online info at security bites.com what L but no I was joking right right on the slide and it's.org oh well it

also works. or at least the website does we all both oh male doesn't Okay it go ahead go ahead last and then speak yes

CL think consider

that's sales we can't do that here we're a 501c3 maybe other bsides can if they if they're not under 501c3 but we cannot that would be really well I'm I'm assuming you're speaking of this bsides here I can't speak to other bsides we don't have the room for that here so uh what we do for that in London is we actually have what we call call community space and we have tables and we consider startups like that but are trying to get kick off as a community partner and they get a table and they can sit and do what they want um well we try to avoid doing product sales same thing but they get they get to show

their brand and explain who they are right it it depends it's like it's it's like it's like she said it really depends on on where you are and what you can do but if you if you if you offer it as like well you know you're coming in as a partner you know I mean you can do that and you can get the people in in for us it works I can tell you that the uh for for a lot of smaller events I mean the bides Las Vegas is stratospherically larger than any other event uh but the budget for a lot of these events is like $500 $1,500 like it's you know for all in of

all their sponsors I think so not probably what you were expecting this talk to be um no real future defined other than us walking away um if this is if this was your baby what would you want to do with it because it is your baby yep that's what we're going to do that's what we're that's where we're trying to go um wolf man oh W it it's it's so and you know there there are two things that I say to Mike that I've said to Mike since the beginning all right since after the be I've I've said um over and over hey Mike I think we might be on to something but um the other one if you

have sensitive Years cover them up uh the chant that I have had since before the first event is don't it up um I'm going to go back to your point Claus about the Arsenal you think that's a good idea yeah do if you think it's a good idea do it yep bring it in make it happen this is a docracy anybody suggests anything to me about oh we think you should be great you should have a cfp or a CTF great you can run our CTF oh we think it would be great if you had a lockpicking village great you can run a lockpicking village if you don't want to step up and run the

lockpicking village don't tell me you think it's be a good idea step up and do it right and that's the way besides works so that's so that's kind of what the future going to look like um as we start pulling away um any I'd be interested to hear of concerns fears no please don't how dare you so I have a concern yep but and I've seen this before like the Belfast guys reached out to me asked me like you know can you give us some suggestions and things like that they were at London and said you know give give us a hand sure they never reached out afterwards and that's that's a fear that you need to you come over

it's like i' I mean that you need if you're going to do it reach out to the community and talk to the community right that that bite that group that Google group because if we don't do it we're basically doing it in our own little corner and You've Lost That Outreach and you've lost that experience that pool of experience that you can get from the rest of us so it's really important to actually participate in that discussion group because if it doesn't if you don't you you're you're doing it by yourself and you're probably Reinventing the wheel for a lot of things so it's it's really important that you try to reach out and keep that

Community discussion there was a question in the back there can you jice hello some of you might know me I run bides Delaware I named it before we decided not to do the state thing um it's a small state state there's three counties in the state I would love it if somebody else wanted to run one in Dover which is the only other city that anybody cares about um when there are conflicts where people cannot adult effectively that may risk brand damage yep with your stepping back how might you recommend they be resolved to avoid brand damage excellent question don't know we don't we don't know I'll I'll tell you what's not a sustainable model

is counting on us to do it um Michelle is like everyone here has a life and real jobs except me and I work at a company that gives me a stunning amount of indulgence and that's the only reason that um you know Michelle has given up a stunning am amount of time we haven't even gone into the millions of hours it took Mike to correct an accounting error that a CPA made for us that set us back years on our 501c3 um relying on us having people having the spare time now that's not to say that I'm not going to jump in and try to adult but listen if you people I know I'm old but if you look to me to be

the responsible adult in the room all the time you're in trouble right you know it's hacker summer camp I have gotten 10 hours of sleep since Friday night uh I I can adult when I have to especially if it lets friends have fun but um we need more people stepping in to your point um I think the realization is going to have to be made everyone's going to have to understand that it may madmax it out there it may become postapo apocalyptic and besides may burn um and then from there Rises a new Falcon um Phoenix Phoenix oh f i i I think that's a really important point though is that is that you know we all right that's our time

we'll be right out there if you want to have more conversations [Applause]

yes yes yes perfect sir Tim yes

y

e

e

e

e

e

e

e

e

e

e

e

e e

bar toop video game cabinets uh actually just curious curiosity of me I want to know how many of you know what that is awesome awesome I guess you probably wouldn't be here otherwise uh favorite game erotic Photo Hunt everyone yeah okay good so uh just a little background about me uh I'm an offensive security consultant uh by day I work as a contractor now my favorite things to work on there are probably net pen and physical penetration tests uh my last job is with acuant Labs but uh now I'm Indie so if you want to hire me let me know in my spare time uh I love to hack Hardware uh I found it like very

fun and rewarding uh to learn about how these things work under the hood uh last year I did a presentation at black hat uh about the b key which is a device that uh uh my co-creator and I uh created to interface Bluetooth low energy with wegan which is the access control protocol used to uh let people into most commercial buildings nowadays and and office spaces and basically with that device you can uh use your cell phone to open doors and get get the data needed to clone cards uh so I found and and and I I mention that because that's also uh technology that's from like the Apollo moonlanding era and uh the these

Mega touch bar units are not that old I will switch out my mic one

sec okay can you hear me okay great uh but they are from around 1997 I found that this kind of stuff I really like hacking on it because it's uh a lot simpler than the technology today less complex The Core Concepts are the same uh so it's pretty easy to understand and then apply to Technologies you know that are in use today so this is what my desk looks like most of the time and I feature this photo because the S logic probe there which is uh this guy obviously it's there and then the bus pirate is used prominently in this work uh so one night I was at the bar with a friend playing some photo hunt

and the Machine had to be rebooted and I saw this screen which uh I was really excited about everyone knows that's Linux right right away uh so at the time those units were still like widely in bars and I couldn't afford one but fast forward a couple years to a couple months ago and I'm trolling Kajiji uh and the rest is history I have two of them in my basement now uh the one I got running was a DOS based one but uh functionally it's very similar they just kind of at one point decided to use Linux cuz they realized that Doss was probably uh going to go the way of the dodo so uh I I

found the best way for me to get engaged with the project if you're getting into Hardware hacking is anyone new to it or never done it cool is to find something like this that you kind of know and love and uh you'll care about it when you play with it you'll you'll feel it's it's like more rewarding so for those that that haven't seen it this is like the typical screen of a of the the video game once it boots up and you can choose your game and there's probably a hundred different games or so uh on on the device so as I said I'm into cheesy games I once had a cat themed birthday

party uh I grew up in the 80s and my first PC was a commodor 64 uh first game system Sega Master System and my favorite game of all time is Galaga so I'm a bit fascinated with 8bit stuff not uh that's what this that's not what this is but anything that's uh kind of generally obsolete gives me a little bit of nostalgia so I didn't really think this would be a good talk when I started because I was like this is just something I'm interested in um but then I got going along and realizing what I was doing I'm like hey this this would be kind of cool to describe to someone to that really wants to get into

Hardware hacking because it's really as I said accessible stuff that uh can help you understand the basics so I'm going to go over today basic uh poking and frotting of the megga touch Hardware uh in which we'll find the custom Isa IO card that uh they designed or they had designed for them and then there's a Hardware Key inside each megga touch that uh when a new version of the software came out they would they would replace they would send it out with this Hardware Key and you needed that to make the software run uh and so that's going to be part of what we're talking about of course because I wanted to to to copy those

keys essentially because the the systems are no longer supported but I'll get into that later so from there for those keys we go into protocol sniffing with the logic analyzer that I showed you and uh then looking at the stream of bytes from the protocol and reversing that to find the password that we password that we need to copy the keys and then we do some development of custom python code for interfacing with the bus pirate that I mentioned which is a universal serial device what kind of universal it's kind of like a Swiss Army knife for dumping the keys and potentially then uh writing them to new keys if we wanted to do that

and this all seemed appropriate because uh this is Vegas kind of land of cheesy games so I was going doing some research for this and I came up with this quote on Arcade like game cabinet for sad lonely men who don't have iPhones and this guy goes on to say the good news is Mega touch went out of business in January the bad news is that doesn't mean they cease to exist so this guy's a comedian I think he's like tongue and cheek with this stuff but this guy I looked him up he was born in 2005 he spent two years in the bars without an iPhone which means he was drunk because those are the first two years you spent

really drunk in bars and the rest of the time so he doesn't understand what the the novelty of this was uh and I certainly do because I was way too scared to approach you know women in bars so I hid behind these screens playing these stupid games so uh anyway he on again it says the machines offer a bizarre mix of smartphone game ripoffs which I don't understand because these were created way before smartphones um and uh bizarrely outdated laggy interfaces which great of course they are because they're from 97 uh so about two years ago Meritt Bart toop business ceased production uh putting these cabinets firmly in the class of uh Antiquated video games and I

guess I looked it up they're called it's called abandoned wear so they just kind of said hey we're not supporting it anymore do what you will but you're you're on your own so that's what really pequ my interest here I should clarify what I'm doing here is an effort to preserve these games uh not to skirt copyright law U there's a provision that was actually added to the dmca a little while ago you probably heard of it for for online games that required a licensed server to run that they would and and they made a provision for preservationists to allow to circumvent those measures I think this is kind of in the same vein at least I hope there's

no one in here that thinks otherwise so uh when I first got the game of course I went right into the there's a little button on the inside of the unit that uh allows you to go into the menu because I've never seen that before and there was some pretty funky funny fe uh features in there but that that's not what we're we're looking uh at today I just thought I'd show you that uh here's the inside of the unit it's uh turns out it's just a big 150 lb uh computer maybe more I didn't actually weigh it but I I carried it up the stairs from where I bought it and I almost killed myself as you can see it's

pretty thick uh heavy gauge metal so most of the space is taken up by the old CRT so be careful if you try to probe around these things high voltage uh in In Those Old CRTs but the rest of it is a run-of-the-mill Pentium along with the proprietary Isa card that I mentioned that handles the io uh and there's some funky connectors and stuff like this centronics connector I believe that's what it's called that was used for updates so you plug into cdrom easier and the iMac came out a year after this which makes this thing look pretty funny if you think about that so uh smoking is bad I learned this was in a bar for like

10 years and it stank really badly so I had to give it a good a good cleaning uh I had some acetone lying around for when I thought I was going to etge pcvs on my own and it came in handy for cleaning uh the crap from like the contacts from The Edge connectors and stuff on this on this unit so this is really kind of the centerpiece of of the only custom Hardware in in the whole Megatouch uh uh unit and uh it's pretty simple actually uh how did I figure out this was the an IO card uh well first the internet told me because I Googled it but then I also took a closer look myself at uh the

components on the board to see what they were so uh this guy right here is a PC card connector and I wasn't sure what it was at the time but you can very clean clearly see the Cirrus logic badge on it and then and the etching on the chips so pretty simple just Google and uh there we go we have the data sheet It's usually the first uh hit when when you're Googling things like that if not maybe go on to the second line uh these parts are very commonly used they're not going to invent their own part so chances are it's on the internet already and pretty easy to understand given its age uh so next found a sound blaster on

board by doing the same thing and next to it uh a little amplifier because the speaker was built in so they didn't want to have a external amp so they put an amp on board as well cool we're getting through this board pretty quickly uh I was confused by this one at first because I didn't know where all this was going thought maybe it was a expansion but this is actually just a is a bus debug from old Pentium days and I I found out uh about that by turning the board right over and you can see all the traces go to the PCB Edge connector which uh it's just common sense from there uh the board looks like it was

probably used in like all of their machines these guys do more than just bar toop gaming so there's so many unpopulated uh headers and whatnot on this board that it could have been used for all kinds of things uh okay and this is the I button key the focus on most of this research and uh what was required as I said to make each version of the software work uh so the keys are somewhat broken already I should mention that one of my kind of Heroes uh Joe Grand did a talk on them at I believe it was Defcon or black ad a long time ago and uh Joe found a password uh guessing a a

dictionary attack against these things because when uh you supply I should mention actually first they have encrypted information on them and when you supply them with a password they're supposed to spit out the encrypted information if the password is not correct they send out random data so thing is the data is not random it's calculated with a mathematical equation so uh known you can you know the output based on the input so you can basically tell if it's the wrong password or not using one guess but I wasn't really interested in doing that because there's like a trillion possible combinations uh the only feasible attack against that is kind of a dictionary attack and if it's

not a dictionary word you're out of luck so let's move on uh what else can we do with it well we look up the data sheet again and uh we see that it uses the one wire protocol so they have a secure uh ROM for their key but a completely clear text protocol communicating with the ROM I'm not quite sure why they used the secure ROM but uh they did so cool it stores uh 1,152 bits of data in three separate storage areas that are called sub Keys uh the secure memory cannot be deciphered without the matching 64-bit password passwords can be different for each area and um there's also a 512-bit scratch Pad which is stored in the clear

on this ROM and uh accessible without any key and the IDS of each of the sub keys they they can give them names are also accessible so we need all that we need a lot of that information when we're doing the duplication so I figured I'd mention it there's only two contacts to hook up one of these guys and I'll pull it out right now cuz I have it here bottom of my

bag sorry about that so it's pretty simple simple you've probably seen these things before they're used for other purposes as well but yeah two contacts parasitically powered which means that there's ground and power and data in the same line so when we boot up we uh see some information about the key uh the mega touch I bought had intermittent problems reading the key uh even after I cleaned it with the acetone I it boots sometimes it says the key is fine and other times it just goes in a continual Loop which is one of the reasons uh why I wanted to do this work because we need to preserve these things well that's arguable but I

think we need to preserve these things so the lifespan of this uh I button or DS 1991 uh was reported to be over 10 years so right now this one's 20 years old and it's still working which is pretty cool um but in order to get if we want these things to remain around working we probably either need to hack the software which is kind of complex and messy because there's so many versions or just come up with a way to circumvent this key so as you see there's there's even still a market on eBay for these things lots of them end up being about 30 bucks Canadian shipped um and uh hopefully the aim that I wanted to to do here was

build a repository of them so if you legitimately have one of these things and your your machine is broken you can just go and download it and uh write it to a new I button and you're good to go so uh this is how I started with uh sniffing this bus and right here you know here's the eye button I just kind of jammed a a jumper in in between jumper cable in between both contacts and then I used is anyone familiar with logic analyzers cool okay so I use S which is a really cool little device 100 bucks and it does it it decipher all kinds of protocols and what it does is it will

plot voltage over time in uh like ones and Zer cuz it's not analog so it doesn't have a curve but as you can see up there uh that's what we have and if you tell it this on this line it's uh I squ C or or spy or uh one wire it will try to decipher based on the data that it's got on that line so I want to show you actually the logic capture now so this is the oh and need to turn this off all right awesome so this is what it looks like and the I believe it's the the long pulses where there's it's it's high it's a high signal is a one is turns out to

be there's eight bits here in each one of these turns out to be a one and the uh the short pulses here the the low the mostly lows is zero so then you have like you have like five ones and two zeros there that make up the the hex uh over there and I'm cut off but anyway as you can see we have a ton of data from that I that I read when I turned on the device here it's not super useful in this format but we've properly decoded it now and we can save it and do whatever cool thing is is we can export it uh to excel which is what I did next

so let me get back to if you have any questions feel free to jump in while I'm uh doing this here please cool okay so on the other side that was cut off what you didn't see there's also like a display of commands because there's built-in commands in the language that are standard so it'll say like oh this is a read ROM command which which it it knows and uh but but the proprietary commands of the 1991 we had to uh figure out ourselves so again we have to go back to the data sheet and that's what a lot of uh you know this Hardware hacking stuff I found is it's just like pretty Common Sense go to the

Internet read the data sheet a lot read the data sheet again because it's confusing and so on and repeat so here's the memory map uh I said it has three regions uh in the N DS 1991 obviously there's a password ID and unsecured data that's this they call that the scrap uh this data sheet is actually quite easy to read it's a pleasure most of them kind of suck and for the more modern Technologies are really difficult to understand I'm not an engineer so uh I have to lean on my friends who are this is a walk in the park by comparison uh so what now we have a visual representation we know we want to dump those areas we have the

data now let's get familiar with the command set so because we need to know that in order to decipher the data so they've provided a really detailed flowchart for how to use this thing uh and let's look at the first example Mega touch does in the the the command capture that I took uh if you were play paying close attention when I showed you the logic dump uh you saw that the first real commands that were sent to the I button uh from the machine was hex 33 which is up there 33h that's what that means you send there's a master in a slave the the um mega touch AES the Master this little guy acts as the slave

so uh the master sends the read Rob command the DS1 1991 sends the family code back and then six bytes for the serial number and then it sends a CRC okay so I want wanted to go through the Excel first of all to make sure that things were saying I got a good data capture and that um and that I understood this properly and turns out I actually did so here on the left you see this the the data sheet on the right you can see my Excel that I was just marking up so we got a TX transmit from the master and then we've got the RX so we know that the serial number is that we know that

the the family code is is correct and they probably do this on the mega touch side because they want to make sure that the you know things are saying on the bus and they want to see that you're using a ROM with the correct serial number so uh what's next we know the comms are captured correctly well we have to get familiar with more of the commands and commands uh are pretty easy to understand actually so if you wanted to read the scratch Pad you send 69 and hex and then you send um one one plus the start address so if you want to start at the beginning of the scratch Pad you send zero if you want to start

one bit in you start 0 one right pretty cool and then one's complement of of that guy which is the opposite in binary of the zeros and ones okay uh so at this point uh we know kind of roughly what the command structure is so I wanted to show you the one wire the actual analysis excuse me I got to switch back again to my

spreadsheet awesome okay so we got a bunch of Rand this is this is the whole dump of the communications so we got a bunch of random reset conditions at the beginning because you're just powering on and then sure enough we see the read ROM it reads the family and then it goes and reads the scratch Pad so it's checking the I button to make sure that the data on the scratch Pad is as expected so we can see the address here one's compliment the re and then it just sends back a bunch of zeros because there's nothing on this scratch Pad so okay great go down all right now here's where it gets more interesting they're starting to read the

encrypted memory now and we've deciphered because now we know the read sub key command is 66 they start with the first sub key do the ones compliment and then they the the slave I button sends back the sub Keys ID which is actually the software version and then we get the password in the clear pretty cool so that's the password right there that we need to read the the whole thing we just sniffed it right off the bus and then obviously then now we have key data which is there and then it goes on it reads the second one and surprise surprise they re use the same password and the second ID is the date

published 99 and then it goes and it reads the third one and again we use the same password and we have some like clear Text data that was on this boot up screen Canadian version two so we're in Canada all right so we understand now that we understand the the protocol and we have the actual password we can do some fun

things awesome so I have all this information and I'm like where do I go next I I obviously can't do this with Excel every time and you can't send this out to the world and make it easy for people with Excel so um I have a ton of Tools around my house got an Arduino got a bus pirate like what do I use so the bus pirate I Googled it bus pirate I button of course the first image that comes up is uh someone's already done this for me awesome so it's really easy to wire up here it is on my breadboard uh and it's only two wires so the bus part getting into a little bit

more it's a Swiss army knife for talking to things via serial protocols and you'll find in anything any H Hardware hacking you do you're going to use this a lot yeah everything speaks some like most chips probably speak spy or uh which is serial peripheral interface or i squ c nowadays but one wire is used for like temperature sensors and stuff like that as well uh and on their website they call it an open source hacker multitool that talks to electronic stuff you can do a lot of cool things with it like you can if you're into cplds complex logic you can program those you can shift data out to a shift register and light up a bunch of

LEDs or whatever you want to do uh so here's a movie I actually didn't want to do live demos and tempt the demo Gods uh so here's a just a movie of uh this this the bus pirate in action uh reading the scratch pad and then writing to the scratch Pad because I wanted to make sure okay does communication actually work in the real world now that I have all this all these uh passwords and stuff so it's first in disconnected mode that's what high Z means then you switch the mode over to one wire and so it's ready and it's got a bunch of sub commands in the one wire mode so we want to give it power over to

the I button so we tell it give it five volts and then uh all these commands are documented as well you can check them out on the on the website pull up resistors because this device is parasitically powered and uh we only want to use two cables to hook it up and we turn the power on and we look at the Save shortcut commands in the in the bus pirate and one of them happens to be search for a ROM so I issue the search for a ROM command and sure enough it finds it on the bus cool and uh after that we want to read the scratch Pad so if you notice in the Excel you

always have to reset the bus before reading so that that curly brace resets the bus then we tell it there's only one device on the bus so I'm talking to you that's that skip ROM command then we give the actual hex 69 start address one's compliment and then read back 64 bytes so it read I had another eye button in there that I wrote Dead uh that I wrote dead beef to so that's cool it's working and then just to prove that it's working we uh we can write some more data to the scratch Pad so I'll I'll write a little bit more

here just change the command to 96 and we change the start address that we want to write to to the end of the Dead beef and then we put in the

data which is bad

food everything looks okay but we go recheck and sure enough it's written properly all right we have we're making real progress now in well in my mind anyway I'm really excited by this time because everything's working I'm able to Comm unicate with these ey buttons I know the password cool so the next step is to make it even easier we put the bus Pirate into bitbang mode which then we can write a python script and instead of doing all of this with type by typing and commands we can make it reproducible use Pi serial and just you know there's all these commands here you you want to do a search macro you send this these

bites you want to send data you send these bytes preceded by the number of bytes of data you want to send and so on and so forth so it makes it a lot easier easier all right so I also wrote a python script to dump the key and this is the key it's mega dump uh and this will actually automatically dump the key this password is stored in the in the script now so you don't have to um you don't have to actually put type it in but there's there's an option to add your own password uh turns out that the passwords are different for each version of uh the actual Mega touch so um that that presents a complication

that we'll talk about later but we have all the correct data here and if you looked at the spreadsheet you know that you know can version 2 is in there and this is not random data that it's sending back to us so we've essentially owned their their ey button at this point uh so future uh at this point as I said it's thoroughly owned uh we can make copies of them uh we can uh back them up so the bits required to uh to to run these games don't go away forever and I prefer this method to the hardware method rather than you know editing or or um hacking the software so you know you skip the test because there's like I

said there's so many versions of the software out there that uh it would get messy after a while but there is one problem and that is apparently uh the megga touch units check the ey button serial number uh to make sure that it's in the right range uh or and then it won't work if it's not in the right range because they the Merit the company bought like I don't know a million eye buttons at once to themselves and they said you know if it's not within this range then you could probably patch that in software too but again getting messy so I didn't bother implementing the right function because once I talked to people

and learned about this I'm like well it's not really uh worth it to write this but it would be pretty simple to just Implement those extra commands in there uh and when I say simple I mean like there's only probably uh three bytes of commands that it would take to re rewrite this to the to the actual ey button also they're out of production these things are out of production now that that pauses uh that that's another problem uh so it would be easier probably just to emulate them them uh because one wire is a really well-known protocol so and as you saw there's only a few bytes in a command so you just have to teach whatever you're

whatever you're using to emulate it those few Comm commands respond to reset anything any um Library that's available will have the one wire protocol stuff in there already like presence condition and and all of that so you just have to em emulate the actual extra proprietary stuff that's in there uh so Arduino actually has a one wire slave Library so what I thought was all right we'll get a teen CC 10 bucks or maybe an adif fruit huzzah which is based B on the 8266 little micro that's a $2 Wireless micro and you can maybe even upload buttons wirelessly that would be pretty cool or just buy USB even that would still be okay and uh and then you're you're good

to go you can up upload new keys maybe you could put in a web monitor for uh the one wire bus to see to debug in case you know in later versions of the software they changed how things worked and I know they did when they got to like the 2005 versions they used a different ey button completely so that would take a little bit more time to um to implement but also uh you could easily use you could easily add to this a sniffer function where if you just flick a switch on the board hook it up in between the probes like I did uh it would just copy the current key then you

can take the key out switch it back into the other mode and it's got its personality there that's what it's going to that's what it's going to emulate clone mode so to speak so I didn't get to this point yet because all this other stuff took up a lot of my time and I also a real job and my wife would be mad uh if I didn't get paid so I'm I'm moving there but all of the software you saw is on my GitHub already uh so you can take a look at at what I did uh I am going to do this because uh I've actually had a few fiction Autos reach out to me and say

this is cool stuff I my mega touches were broken I always thought that uh you know they they said that there's no way to copy these things they're encrypted and it's like okay well let's let's fix that uh so I did do a little bit of work on the software side too because of course when when you're doing this Hardware stuff it's really hard to work kind of in isolation because these things interact really closely so sometimes you need to answer questions about what the heck's going on and uh for that my buddy Jer here helped me out a little bit on the software side I'm not so hot with Ida so uh I got him to

load it up and take a look at stuff but what I did do was I can use GP that's for sure and I looked for the key inside of all of the Mega touch files and there was like 2,000 references to the key in clear text in uh in the hard drive so that's pretty cool and I was like why are they putting the key in in there and the actual payload like the encrypted stuff was in the dlls as well uh so we found and if you can't that that function there calls fudge security it's Linux based but their games are written there's two versions there's Doss and that's the one we were playing with and then they moved over

Linux uh 90 maybe 2000 at some point so we were looking at the do stuff because that's the the box I had and the hardware is kind of flaky and I didn't want to screw around with trying to up update it and it all takes a lot of time so uh yeah we found this and we realized I think um they didn't want to go back and do a key read uh or they didn't know what what key it was so they they stored the key for when they're at storing I think high scores on this guy and so they just fake security then so in this particular version you could you could pretty easily circumvent the copy

protection by just reading the key out of the files but that's changed in in later versions they don't fudge Security in every file I don't think but uh that was kind of funny and I really I want to do some more research into the software side as I said but uh I haven't just haven't got there yet so that's all I have uh for today on hacking the mega touch uh I want to thank all of you for your attention and for coming and the stuff is on GitHub my GitHub is there uh but it's under heavy construction but if you have any questions like file a a ticket or whatever and we'll we'll chat about it

uh I'll put the ey button cloning stuff up there too when I'm done and uh Happy hacking everyone [Applause] does anyone have any questions all right thanks very much got one oh

wait thanks for the talk um did you do like with your own device that you bought did you have a look at any uh sort of hardware bypasses for the coin mechanism to so you don't have to constantly dropping coins in your own machine I didn't play around with that I think I think it was broke actually so yeah I was really hoping they had a bill collector on it so I could take a look at the bill collectors to and how that thing worked but uh yeah I haven't got there yet yep you just put a button in but at least the later firm where have a setting for free play in the menu so no

hacking required yeah so you know them pretty well eh nice any anything else all right thanks again everyone it was a [Applause]

blast so did you attack the hard drive serial numbers at all did they the older firm did um the unit you're looking at I think all of them so there's

actually

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e e

phones and you know just be respectful in general and uh fill out the feedback forms they're all on shed scared um okay our speaker this evening is Andrea Scarfo security Analyst at Open DNS 12 years ago she cut her teeth as a CIS admin and now she is a malware Hunter so please give a warm welcome to

Andrea than you guys hear me okay all right yeah so the first uh security conference that I ever attended was actually a bsides in San Francisco and now this is my first time presenting in at bsides again so go bides so the title of the talk uh an evolving evolving era of botnet Empires um just a little bit about myself again I'm an analyst on the security research team in Open DNS in San Francisco um before that I was ass sist admin for 12 years and then um in my free time I enjoy rock climbing uh playing video games and surfing in the ocean not online but I like that too so the agenda for today we're going to

start with the history of botn Nets just a overview the birth of botn Nets uh move on to the life cycle of a bot uh highlight some botnet uh network architectures uh then move on to what a bot can do botn net functions uh highlight some different malware that uh Bots usually drop uh highlight some famous botn Nets uh some detection methods so uh ones that uh we use at Open DNS and then uh Manual analys of botnet domains uh from DNS traffic okay so to begin with uh what is aot what's aot net uh it's a network of computers uh infected with malware they're controlled as a group uh a bot will perform uh automated commands and

uh they're controlled by uh command and control servers uh controlled by a a bot master and then in this talk we're going to focus on uh domains that are uh associated with botn Nets so to begin with uh Bots were were not always malicious uh started out uh with uh an IRC channels here's an example of a non-malicious bot uh that would you could just play a nice little trivia game with uh in an IRC Channel and it would even serve you a a cold beer um so when uh Bots first started to mve around on in IRC in 1999 a notable one is called egg drop uh this one uh was designed to control uh user band

lists and uh help prevent uh Channel floods from happening and it actually had a feature that was called botnet and it would um which linked the Bots together between the rooms and you know we see that features evolved today with linking Bots together uh pretty Park uh was able to download files and execute them uh in different IRC channels and then one of the most famous agoot uh which was one of the first customized Bots to order so uh from the coder you could get a bot designed with uh specific exploits that you wanted one of the most famous was the Elsas exploit uh and uh which actually got the attention on Microsoft and law

enforcement and uh to find the the coders so they could shut that down this is a example of a part of the code that was in uh aabot when it was first seen and the source code was actually uh released on the internet which led to a lot of uh um Bots uh borrowing that making them their own uh so it started out with just Bots uh participating in a targeted IRC attacks so uh denial service attacks and channel floods um in 2003 2004 there was a a boost in malicious bots on on the net um and they started to move away from just uh communicating through uh IRC to uh HTTP uh just because with firewalls uh

you could just block uh the IRC traffic but with HTTP it's usually always allowed through a firewall um so the life cycle of aot uh it'll start with the uh infection and and then uh spreading so uh an infection uh the what they use for infection um they use Spam so they'll uh initiate a Spam campaign and send uh uh malicious attachments to to lead to the download of the bot um or through uh injected code on a compromised website uh which would lead to uh to an exploit kit and also through uh malvertising so uh with malicious ads um and then from there a bot needs to make uh contact with the command and control server so there's

different uh rallying techniques uh that they can use um less common is a a a static IP list and a config file in the bot uh just because it's uh easy to figure that out so they've moved away more from that to uh using uh domain flux so um through a a DGA a domain generation algorithm it will um produce thousands of domains uh based off of those djas and then um only one out of those thousands will be uh will be used actually as the command and control server um and then they'll use IP flux so they'll be constantly changing uh the IP uh that the domain is resolving to um then from here once it makes cont contact it's

ready to accept commands and uh report back so from here it can be used uh as part of a Dos attempt it can be used to uh spread uh as a Spam bot to spread more spam to further the uh infection and make the botet grow larger um or it can just become an info stealer to steal from uh information from the machine um and then the goal is to maintain that uh that bot keep keep it there evade detection um and they'll use the same rallying techniques uh with the domain flux and the IP flux um some of the network architectures that are used uh the first was a uh centralized uh command and

control topology here so also through IRC HTTP there's just one um command and control server um controlled by the botmaster and so just that one server would send out commands um and uh store the stolen info there uh next is uh a peer-to-peer infrastructure so the infected Bots uh can share in sending the commands and the data to and from the commanding control server and then it's using um uh file sharing protocols TCP UDP icmp um and then there's the hybrid one which is the uh the most resilient and the most popular one nowadays uh so just combined centralized with peer-to-peer uh and there will be multiple command control servers uh and then one control panel

that's controlled through um and because of the multiple command and control uh servers can act as uh they act as proxies so it uh makes it harder to detect so botnet functions what can a bot do uh so I've said before you can it can act as a spam bot to uh further infect uh more computers uh it can be a act as a form Grabber on a computer so in the config there would be a hardcoded URLs that it's going to uh to search for uh traffic going there and then it will grab what's entered into uh forms on on those sites uh take screenshots of a machine just to grab personal data uh

redirect HTTP traffic you're trying to go to uh legitimate site and you get redirected to a fake fishing site where it can steal your credentials um key loggers to steal passwords uh they have VNC uh modules where an attacker can uh remotely control your PC as if they're physically there and then uh stealing cookies so participating in a session hijacking uh to gain access uh as an authorized user to a site uh and then the BS will uh after they get infe start an infection on a machine they can also drop different types of malware so um for again a a form graer form Grabber uh this is example of an old older version of uh dryex with um

preconfigured uh URLs that it's going to spy on um you can be used as a troan back door to intercept uh uh any network traffic um the most famous right now is the with crypto ransomware um which encrypts the file system and then um holds the uh decryption key uh for a ransom um banking chions like dryex uh just spies on banking sessions um a a quick fraud bot so like the uh beep troan uh just to generate Revenue by clicking on ads and then of course as a a a spot um so one technique that um the malware will use uh once it's on a system is a code injection um this uh screen capture here is uh the angler

exploit kit um using Code injection on a a compromised website um and we've seen the angler uh exploit kit um as part of the ners botnet uh so here this this screenshot capture is showing a a dll file that gets injected into the uh browser process and the the OS sees it as as a safe process so it allows it to run uh and then at that point the exploit kit can uh take advantage of any plug-in exploits uh that the machine has so through a silver light flash or Java um once uh once it executes that code then it's going to download the nier's bot on the machine and it will be enlisted into

the botn net some famous Bots um dryex it uses the uh hybrid uh peer-to-peer Network architecture um first spotted in 2014 uh mostly U stealing credentials to targeted banking sites um today it's mostly spread through spam and um U embedded uh Word documents with malicious macros embedded um originally it had a centralized uh uh Network architecture it switched to the more resilient uh hybrid peer-to-peer um there's a lot of numbers out there for the estimated amounts stolen but uh 30 million is one of them and it was the target of an attempted uh takeown operation in 2015 which wasn't very successful um so here's a an example of the the different layers ERS that are uh

seen in the hybrid ptip here uh layout here um the first layer is uh comprised of infected users computers so this is where they'll accept and carry out the commands the second layer uh we'll call nodes that uh is more infected users uh but these users can act as HTTP proxies and that's between the uh the Bots and then the frontend uh command and control servers which are next um and then the proxy layer is made up of compromised servers um and that's going to act as another proxy between the back end where the control panel is where um all of the stolen data will be held uh so I have a a visualization uh a

movie here um using um open graffiti it's an open source tool um my coworker tibo can't really pronounce his last name so I won't try uh this Tool uh TBO made this tool it's a 3D data visualization um so we'll visualize graph data sets uh so what you can do is uh take a Json file um that has uh the relationships of the IPS hashes URLs domains um uh scene and uh put it into open graffiti um and then we'll get a a visualization here so this is showing um all the relationships um what the information I gave it here was uh C2 servers at a time between um 2015 and 2016 of uh the dryex

spot net um so the blue the blue nodes are uh the IP addresses of of the command and control servers scene and then um showing how they're related to uh URLs that were seing and then um in the orange color and then uh on the purple color it's a hashes so a lot of these mostly uh just the dedex rosion and what this does it helps us um see the patterns in in their in their uh infrastructure and you can see how some of the IP addresses are related to each other through through the hashes seen and others are uh just out there on their own and other uh the ones related have some pretty uh large

clusters okay another famous botnet ners uh it's also a hybrid uh peer-to-peer botnet so they use um they also will use uh djas um that they can push uh update through the peer-to-peer Network so they can um have the DJ's uh sign different CNC servers uh out through the botn net and they they'll generate a lot of these just to create noise so um there'll be thousands of uh of domains out there and they'll only register uh actually one of the domains the rest of them would just be NX um they ners has been seen uh spreading uh locky and dyex through spam it's also been related to the uh angler exploit kit and just recently uh it's been seen

uh as part of of hosting the lurt command and control servers uh this is a new uh banking troan it's estimated to have stolen $45 million already so here's another visualization using open graffiti of the ners botn net uh so these are the uh uh command and control servers that we saw and um how they're uh related to uh the hashes the hashes I got from virus total and these particular ones here are uh showing that they were seing associated with lurk cber and beep so both of these bot Nets have been the focus of some takeown uh attempts uh ners had a bunch of their Community Control servers uh sink cold um on June

1st they actually uh went completely silent for about 24 hours uh during that period uh 1 million hosts were seen trying to attempt to connect to the uh to the command control servers um they've resumed full activity uh already by the end of June and are taking part in more spam campaigns uh with dryex so they're takeown attemp Happ happened in October 2015 um really they resumed activity a week later um and it's due to its hybrid uh uh Network there um so it's still spreading the triex uh troan and it's actually been uh doing something new lately where it's uh participating in spreading uh the locky ransomware through uh through spam and it's usually uh through uh JavaScript

exploits in the email so some detection methods um these are through uh analyzing DNS traffic uh so at at open DNS we see uh 80 plus billion DNS requests a day uh from the customers that are using our DNS resolvers um so then since we get all those requests we're able to see the uh traffic patterns of queries going to the domains uh and then we have a the history of a domain so we'll uh be able to see the history of the IPS the domains been Associated to the name servers the asns um we have a we have a tool um that I use and it's called investigate uh so we'll use it in when

we're investigating domains to see the DNS requests and then pivot off of the uh the different data that we have that's a screen capture of it um so one way uh one detection method is through uh detecting uh domain generation algorithms so again that's where it will use random characters and combine it with some type of dynamic component to generate just random domain names um it might be a number combined with the current time and just symbols uh it makes a new domain each iteration and then the purpose is so that the uh the bot has a a way to contact the command and control server um so very few of the domains are actually registered out of

the thousands that it will generate uh it just creates noise makes it hard for the analyst to uh to know uh which one's actually being used but it's easy to detect a DJ um domain name just by using lexical filters uh here's an example of some angler exploit kit uh DJ so here uh I guess what they're doing there there's a a letter a number and then an animal name in these and they're all registered at on the uh dot top TD um these are an example of uh the tinba banking chosan djas so it is possible to uh reverse engineer the algorithm so to find out uh what the what what the next domain is going to be

that's registered uh and then you can we can pre-block all of the all of the domains uh and then this is an example of uh some BP djas so another way is uh by detecting traffic spikes in the DNS requests that we get uh so if there's a sudden surge of queries uh from the clients uh we'll see that sudden surge in uh query requests but um you know since a sudden surge in traffic it doesn't always mean that something malicious is happening we'll also run that uh through other filters so looking at the domain history of it so the IP history like if there's been abrupt changes the domain has changed uh IPS uh just numerous times um the query

volumes that are actually coming in and then the geolocation like the distribution of the queries and uh then if this Spike has happened before like is it a normal thing for this domain to suddenly have a spike in traffic or not so so this will lead us to domains that are associated with exploit kits uh fishing campaigns and then help helps us uh find the DGA that are associated with the botn Nets um so once these automated uh tasks run than uh the analyst we can do a manual analysis on the domains uh and then we can find additional bad domains off of the IPS Associated or the name servers or the hashes that the malicious hashes that

have seen making a call out to those domains and then we're able to give them categorization by using all of these different uh indicators this is an example from investigate of just showing just a sudden spike in traffic whereas there was no traffic before and it's a sudden spike in the queries so uh manual analysis so here is uh this domain uh spotted on July 18th so all this data here uh is uh what was seen on July 18th I think it's since been uh sink cold after that date so s we see a sudden spike