LT - Calling All Researchers: A Discussion on Building a Security Research Framework - Michael “DrBe

Hey, there we go. All right. Hi everyone. Thank you all for coming today. My name is Michael Smith. Also come call if I can speak. Also go by Dr. Berk. Uh today we're going to be doing a discussion called calling all researchers. A discussion on building a security research framework. Uh this is going to be a quick presentation and then followed by hopefully lots of audience participation and dialogue. It's kind of the goal here tonight. So it's going to be as long as you guys talk and if you don't talk then it's going to be very short and well not fun. So a little bit about me. I'm a senior security engineer at E+ security. I'm

also a doctoral candidate at Capital College working on data mining heterogeneous data types. When you if you ever want to fall asleep very easily, come talk to me about that and I'll uh give you the rundown. Uh, I got my Alphabet certifications, my CISSP, my OD CP, my GPIN, my lowly security plus, all that fun crap. Uh, if you want to get a hold of me, Twitter at drbersc or you can email me at drbears atgmail.com. Please feel free to spam me anytime you want, that's cool, too. So, why do we research? How many out of you guys out there feel that you're independent researchers? Feel that you're researchers. Done any kind of research? I got one there. Who's made

it? All right. So, let me ask another question. Who's ever released a a tool, security tool, script, anything like that? All right. Who's ever dropped a zero day? Who's ever written a paper, even a blog article? Hey, guess what? You're a researcher. Congratulations. Welcome. Welcome to the club. So, why do we research? Well, we research to contribute to the existing knowledge in the field. Our main goal as researchers is to build up our field. It's there to take the existing literature and it's there to build upon it, improve upon it and create and expand that knowledge. Uh we want to improve the field. What's one of the greatest improvement we've ever seen in pin testing? Meditate.

Yep. I mean, think about how that changed pin testing, how that changed what we do for a living. Completely changed. Brought it to the brought those really expensive tools down to the masses. Perfect example of research to help improve the vehicle. Um informed policy issues. This is research such as building uh policy issues. Uh think about some of the privacy stuff that's going on right now. Stuff like EFF does, ACLU does. Uh think about some of the stuff we've seen recently even um Aaron Schwarz we uh some of the very other legal uh challenges that we're seeing right now. Those kind of research can help drive those policy issues because what hasn't done what what have those

issues happen? Well, they've helped help really push that discussion forward. Made us as a community aware, hey, this might be a problem. We really need to get off really need to get out there ahead of this and help build up that uh build that up in public policy and skill and career building. It's one of the main thing why why are sometimes that we work on some of these projects? Well, we wanted resume building. We wanted as uh ways to increase our skills and make ourselves better. So, what kind of security researchers might we find out there? Well, we have academic researchers. I I I myself as an am academic research because I'm a doctoral student. If you've done a

master's thesis, um you work at a university, you can be considered an academic researcher. Usually very rigorous type research, you follow a lot of very stringent rules. There's a lot of ethical considerations, things like that. Government researchers, kind of similar thing doing it for the government. Uh corporate researchers, I'm working for a company like Tenle, like Rapid 7, like Immunity, maybe finding zero days, maybe just uh building tools, things like that. Uh profit research groups. Uh for those I'm talking groups like uh the ones that um uh uh like um uh name uh kind of like Vpensia those kind of groups groups that might uh go and find zero days sell them out to the

highest biders. So those are I'm calling profit research group and kind of the people that we're going to talk about today you guys. Those of you out there doing out the blogs doing your tools dropping your zero days doing that kind of research the independent researchers. Why are independent researchers important? Well, again, we want to grow the field for multiple angles. You know, we don't just want to grow security from a standpoint of, oh, hey, let's find the newest cool zero day. And that's the only way we're going to grow. We don't want to just grow it from dropping new tools. Those are very important, but we also want to look at different research. Want to look at

research from a policy standpoint. We want to look at research from a um, you know, from new technology standpoints. Uh, Brentface right over there, he's working on a brand new tool. um again something that's going to help propel the industry forward. Whether it's a minor contribution or bigger contribution, it's still a contribution to the field and that's where independent research is important. Uh what are some of the other reasons that we do independent research? Well, users workers may lack the necessary skills. Some of you guys that have written scripts, what are the main reasons that you wrote? No one else. There you go. Exactly. And it might be that you work at a place or you're using a tool where

the tool doesn't do what you want it to do. So you need to find a way to do that or else no one else there knows how to do it. You got to find a way to do it. That's what you're doing. Someone may lack the necessary skills. You learned it. Now you're propelling the field forward. Some vendors need public motivation. Let's be honest. How many times have we heard of people that have announced the zero day and done the responsible thing and formed the company. The company says, "Yeah, I don't care. Not going to do anything about it." and then it gets dropped and then all of a sudden, oh, hey, this might be a big deal and the PR looks

bad, so let's go fix this right now. Public motivation. Some vendors just don't care. And that's a good point, too. That's another reason why we drop it. Why why have we seen vulnerabilities drop? Well, company just may say, "Hey, I don't care. I'm not going to fix it." So, you know, if no one knows, yeah, what's what's it off my back? So, that's another reason why we do independent researches. Uh, we talked a little bit researching the customer overlap. Again, you might own a tool that need that you need to figure out a way to make it work better or make it work in a different way than what it was originally intend to intended to. Uh, improved tools can

lead to better security. Again, metastoid, uh, Nessus, um, uh, many of the other various open source tools that are out there, bro, perfect example of something starts out as public or as open source, seeing it go into maybe a commercial, uh, viewpoint too, something that helps them better the field. Um, uh, Snork, another perfect example. Snort went to Sourcefire, which is now obviously Cisco. Um, serves the public interest. And this to me is one of the most important things outside of, uh, growing the field of multiple languages. Research serves the public interest. Why? Because better security is better for the public. Everyone uses the internet. Internet's maybe one of the most important inventions. I I in my

feelings of the 20th century. You know, some of you might disagree, but to me, think about what the internet has done. Think about how it has changed our lives. not only giving us all jobs but also you know literally changing the way people interact, changing the way people research, changing the way things do. The internet has literally taken businesses and taken traditional businesses and completely shifted the paradigm and made it so that those businesses don't exist and then uh knowledge itself is power point. So what are some of the motivations that we have for independent research? Well again we want to grow the security field. when improve the industry, improve the world. There's altruists out there that basically drop

this stuff, make these things to make the world better, a better place in their opinions. Uh build a reputation within the community. Hey, you know, we all like to be loved. So, that's a good way to get your name out there is always to go out there and uh help contribute to the community that you're a part of. Uh build a career or build a business. Again, I throw out Snort, Nessus, um Bro, you know, something that's growing up now. uh Medicate which you know wasn't grown as a business but became you know when it was sold to Rapid 7 HD went from someone developing it to you know basically the cso of rapid 7 um

knowledge money and sometimes just for the laws so the biggest problem that we see with independent researchers some of the pitfalls you may fall into uh you know these are some of the things we're going to go a little bit more into detail but uh the and again I want to point out this is just this is just some examples. This is not a comprehensive list. This is not the only list. These are my these are some of my opinions on it. Others of you may have other re other um other opinions and other pitfalls that you've either experienced or you've witnessed or you just think might be a pitfall. Uh so some of the ones I'm looking at are

legal issues, failure to site previous research, plagiarism, which kind of goes hand in hand with that, ethical issues, reproducibility, and prejudices. some of the legal issues. This is a big concern for everyone uh everyone else community in the community. Anyone doing vulnerability research, I can tell you right now probably has some concern about whether the uh they're going to end up in prison. And I'll be the first to say it's full. It's it's either over prosecution, it's poorly written laws that have not kept up with the time. Unfortunately, that's what's out there and that is one of the pitfalls that we face. Um you know, let's look at some of the legal issues. These are just some of

the the heavy, you know, heavy stuff we've seen. Um some of you might be aware of the uh talk that was supposed to happen at black hat with that Volkswagen uh basic or not black I'm sorry it was academic paper pap pap pap pap pap pap pap pap pap pap pap pap pap pap pap pap pap pap pap papers there's so many talks on hacking cars a week ago so researchers in Europe were going to drop a paper on the Volkswagen cryptography that they utilized figure out way to crack got sued block them from publishing that paper instead of doing something that actually would be beneficial for the community Volkswagen doesn't want to let their information

out and I have a feeling doesn't want to do the whole recall thing that they'd probably have to do if that is uh broken. um you know basically blocked the sue the block the sue blocked this academic research paper and deprivives the industry of growth uh weave in AT&T I know all of you followed that case tragic case in my opinion just absolutely horrific something you should have never gotten in my opinion shouldn't even have been found guilty on but you know I'm not a lawyer so I can't speak to the expertise on the legal issues there but obviously my opinion someone who feels very strongly about that I know most of you I that I know in

this crowd have also have similar feelings to Um, Sky Skitech forces Al Kapazad to sign an NDA and it was basically uh a talk that he was going to drop and there's some more information on all these on the attrition legal threats, but a talk he was going to drop and uh basically they forced him to sign an NDA so that he would not um disclose any of that information. They threatened to sue him. You know, corporate bullying. Basically, they have the big bully stack. They have all the money. Golden rule. You all know the golden rule. Those who have the gold make the rules. and uh two skater talks at ICS got cancelled due to legal threats. threats

from the skate companies. Things that again absolutely important because what is skate for those of you do you all know what skate is? Most of you all right cool. Those of you that don't skate is basically what runs our you know most of our utilities, power grids, that kind of stuff. Very important also very riddled with holes and you know again something that propels the industry something that for the public good is out there and got sued from being able to mention or talk about citations and plagiarism. So, who's familiar with attrition.org? All right, most of you. How many of you ever want to end up on attrition.org? And not in a good way. That's good. I didn't want to see anyone

raised their hand there. So, plagiarism is in research is a killer. You never want to do you never want to plagiarize someone's work. Why? Because it it hurts research. You might say, well, how does it harm research? It's just, you know, copying off the television writers do it all the time. Movie writers do it all the time. ruins research because it keeps people from actually innovating. People don't want their work stolen. They may not go out there and they may not publish it then because of that. Um and also honestly they did the hard work. Whole point of research again is building upon that field. Want to build upon that existing research. So that's what we do. We go out there we get that

information. We cite the people that we had it from before. We let people know hey this is the original work. You can go check out this original work. Now look at my work that builds upon their original work. We both had contributions to the field. This is what we're doing. this is what we're doing there. Um, so you always want to credit your preceding work. I don't care if it's research paper. I don't care if it's a simple script. If you take a project and you fork it, get out there and make sure that you cite the original project. Just, you know, put a little board up on the top in the rim state and just

saying, "Hey, based on, you know, script XYZ." Boom. That's all you got to do. And for those of you that get mad if someone forks your project, get over it in my opinion. Now, that's again might be controversial. just my opinion. But the whole point of that stuff, you want people to build off of your work. That's why you that's why we fork things. Let them take your open source work and let them improve upon it. They might come up with an idea that you never thought of that makes it better. And think about some of the stuff where if we hadn't opened that up to the community, if things like that hadn't been out there,

think about some of the innovations we may not have ever seen. Um plagiarism citation. What can it be? It could be a major embarrassment for you. Can't be a career kill. uh certainly, you know, in academic circles, you get caught plagiarizing, that can kill your career. You know, me personally, you know, again, doctoral student, if I play, if I got caught plagiarizing something, I'm kicked out of the program. I know people in my program that have been kicked out. And we're not talking like a I failed to site one thing. We're talking people that copy page after page. You know, we're talking like the Gregory Evans special. That's what we're talking That's what we're talking about.

Allegedly Gregory Evans special. It's pretentious. Um that's what that's what we're going to talk about. those kind of things, but we also don't want to fail the site. So, that's something that you want to definitely make sure of. That's usually more of an embarrassment, not something that someone's going to kill you over. But that being said, you know, it's still not fair to the original author. And again, like we said, academic saner uh some good examples uh attrition.org, their plagiarism site can give some good uh good examples of people that have uh participated in plagiarism. And uh you know, basically whether you agree with what Jericho and attrition does or not, doesn't matter. They're going to do it. you do it,

there's a good chance you might end up on there. So, last thing you want to do is end up on there because it can be a, you know, major embarrassment. Um, also link to a favorite site of mine I use in academic uh my academic writing is uh produce uh online writing work uh language workshop. Um some great stuff on plagiarism on there too. Ethical issues. Tons of ethical issues out there. These are just some of my favorites. Um you know, ethical issues that independent researchers might face. Should companies be forced to patch on an arbitrary timeline? So in other words, you know, there's companies out there that say, "Hey, who are you as a researcher dropping the zero date to

force me to patch my software?" Who are you out there to force me to do this? And you and there's there's a point to that. Now, I disagree with them because some of them want to take, you know, all the time in the world, but you know, there is there is a question that's out there, an ethical question of how long should you give a company uh before you actually drop that information? Uh does exposing a vulnerability decrease security? Uh, one of my favorite articles that actually kind of uh, spawned a little bit of this was uh, uh, we've run a great article in white and cited some other research that basically was talking about where why

you should sell your zero days instead of doing online vulnerability culture. Now, I don't necessarily agree with him on this, but his point of it was uh, the research basically, I'm sorry, I can't remember who did the actual original research, but if you look in the wire article, you can find it and if you ping me, I'll be able to tell you. Um, in this original research though, they looked at malware and found that a lot of the malware out there was based was built off of public disclosed vulnerabilities. So, yeah, there's there's a point in there. Does it decrease security? Is creating a tool that make hacking easier? Okay. You know, again, we talked going back to

metas-ploit. Metas-ploit brought that kind of exploit framework to the masses at a at a, you know, cheap price. Free can't beat free. Was that a good thing for our industry? I feel it was but again it's my industry. Other people might disagree because it made you know made it easier for certain people to be able to exploit the attack boxes. So there's an ethical consideration. Again I have my opinion but it's not necessarily the right answer. It's just again my opinion. Different answers different uh opinions. Should vulnerabilities be sold, bounty or publicly released? This is a big hot one right now. You know and honestly you know me personally I feel I should feel they shouldn't be. That being said, if I

had a vulnerability, someone's offered me 50, 100 grand to sell it to them. I get it. I mean, honestly, I get it, man. I I don't I don't I don't actually fault them. That's a lot of money. I mean, that's a career. That's a year's salary for something more for other people. So, I mean, that's, you know, it's it's an ethical question to ask. Should you sell it? Should that be something there? Should there be some kind of stipulation you sell saying, "Hey, after x amount of time, if you want me to sell it, you have to publicly release it." Again, no right or wrong answer, but something to keep in consideration. Reproducibility. This is a favorite part

of mine with research. So just because we find a vulnerability, can we reproduce that vulnerability? Uh being able to reproduce the vulnerability is critical. Just because you find it, it's like a lockpicking. But you know what lockpicking? If you pick the lock once, have you pick the lock? That's correct. No, for those of you shaking your head, you have to pick it at leaking is you got to be able to pick it twice to be able to really say that you're able to pick that lock. Same with the vulnerability. You want to make sure that you can reproduce that vulnerability. Keep doing that over and over. Make sure that you can say without a doubt document and say, "Hey, if I

give this vulnerability to Caspian, Caspian can go and he can reproduce it on his own." Um, need to be the need to prove that the outcome is consistently predictable. Again, same things. Is every time we run this vulnerability, it going to cause the same uh buffer overflow? Is it going to cause the same uh, you know, the same process to stop? We want to be able to show that it's constantly predictable. uh if the tool release makes things safer, then people need to know how to test with it. You know, how many of you have gone out there seen tools out there and can't find any kind of documentation or anything on how to use the tools? Now,

some of you are smarter than I am, so you might be able to figure it out easier. Me, I'm an idiot, so it bugs the hell out of me when I can't figure out how to use the tool. I'm spending half my day trying to figure that out. Half the time I'll abandon it and just do my own part. But again, something out there. Hey, we documentation. Again, we're trying to help the industry. We're trying to move the industry forward. We want to make sure that we make it so that people can actually use what we're producing. And uh research should be independently reducible like we talked about. I give it to Caspian. Caspian reproduces

it. Prejudice. We've all faced this. How many of you say that you tell people you're a hacker? And how many of you have they said, "Oh, you're a bad person." Or they give you that like look like, "What are you up to? Can I trust you?" Pisses me off. Hate that. Drives me out of my mind. I saw something the other day and I can't remember now what it was on, but it was basically a Facebook posting to a response to an article and it was basically like, "Oh, all hackers should go to hell. They should be in prison." Blah, blah, blah. And seriously, I I I had to fight not to respond to it. I I

like literally I had a response written and deleted it cuz I'm just like, "You know what? I'm not going to get into a fight with an idiot." But that's a prejudice that we all face. The general public view unfortunately because basically media and other people just don't understand what we do is that hacking equals evil criminal. You know, which is not the case. You know, most of us out there, hey, we do this for a living. We do this as a job. We do it because we love it. That's why we do it. You know, most of you that got into this industry, I guarantee you got in there because you have an interest in it and

you love it. You enjoy doing it. You don't just do it for the money. You do it because you like it. or you wouldn't be here when you come do events like this. Uh academics take issue with the lack of formal processes and methods. You know, independent researchers, I can tell you academia looks down upon homeless. You know, you don't follow their processes. You don't follow their procedures and there is a gulf there. And that's a damn shame. Now, there's a paper I wrote on academic tools that could actually be used for and not published yet, but hopefully we'll be at a talk soon on academic tools that can be utilized for pentest. Some great

stuff. um leaves at Derbycon. Uh I know uh Lubix and Wick are going to drop a uh basically a data mining tool on Git that um goes out and builds password lists. Um honestly, I found an academic tool some does something very similar. Goes out, pulls down repositories, and then does full data mining looking for a lot more than just the password list, but gathering information from there about the companies and such that are there. a great tool that goes out and it pulls down all the job ads and it does data mining on those job ads to find hey well if this company's looking for people that know Cisco know checkpoint know this this and this all you've done OSN

hey we love monster.com because you're telling us what kind of equipment that you might have without me having to do any kind of scanning of your network at all great gave it served it up to a server great academic tool because there's a distrust between the communities at time and I say it more bears on the academics than it does on us but you know there's a hatred among our community of formal rules I get that too. Um, you know, there's great research that could be grouped together, that could be utilized together in a joint tandem that really could help change the world. And sometimes the examples that sometimes the opportunities are just lost because of

that. And people take issues with they're building on their work. We talked about it a little earlier. Again, you don't like someone forking your work. Again, get over it. To me, bad thing. You want people to fork your work because you want them to build upon your work. Nothing wrong with that. You want to keep a fork for yourself to be able to keep building what you want to do. Nothing wrong with that either. Completely respect that. Understand that. But give other people the opportunity to build. One of my favorite things on Reconng when that when that got uh dropped is literally said, "Hey, this is my framework. Feel free to take it, fork it, build it up, and do

whatever you want. Utmost respect for those guys. All right. So these are some of the uh these are some of the resources that we have um you know for independent researchers that we might have formal research frameworks APA MLA who's familiar with APA MLA all of you who hates APA and MLA not more than you I'm surprised not more than you because I despise it hate doing it I have to do it on a regular basis EFF coders rights project great project awesome out there great stuff to go to to help give you some legal guidance if you're dropping zero days or dropping other kind of tools uh Wikipedia citation just some way to site

some things. Uh, easy, not quite ALA or APA or MLA and uh, a great vulnerability disclosure, the correlated vulnerability disclosure policy. Kind of what those guys follow when they do a vulnerable, which brings us to our problem. I just listed four sites and that's four sites. There's a ton of them out there. It's a disparate amount of resources and we have to go out and we have to find exactly what we're looking at. It's not always easy. Um, oh, some other good stuff someone mentioned to me that I was talking to earlier. Uh, Google, go look and see what people have done before you. Anyways, back to the spirit resources. So, you may not know where to

go. You How many of you are familiar with the EFF coders project? Right, a couple of you. All right. So, again, some of you may not know of that and have looking at this like, hey, how do I how do I do this? What's the responsible way to uh what's my legal liability? How many of you you were familiar with Corland's vulnerability disclosure? I see a couple people. Again, a great resource. So, we have a dispirit amount of resources. That's a problem. Not all resources meet our needs. APA does not really meet the needs of the hacking community. It might be you're writing a white paper. Sure, I can see where it might meet your needs. You're dropping a

zero day, you're dropping a script, you're dropping a tool, doesn't meet your needs. Um, some systems are clunky or painful to use. Anyone that's citing MP APA, it's clunky and painful to use and some systems are too rigorous for adoption. Again, APA MLA rigorous for adoption, tough to use. So my proposal, my proposal is we need to build an independent security research framework. So the goal here isn't something that's mandatory, isn't something that's required. The goal here is to build a resource and guidelines that people can help follow, help keep them out of trouble. you know, good re not only a resource for people to go to to be able to um you know, find what they're

looking for, uh, you know, as far as citations, find what they're looking for as far as disclosure, as far as drop stuff, but, uh, something out there just kind of gives them guidance on, hey, this is a good idea. This is a great resource for you. This is what you should look at first. And, you know, then check out some of the other resources out there. Use what you feel is right, but this is a good starting point. Uh, something similar to me in scope to the PTE. How many of you familiar with the PTS pit testing execution standard? Fantastic framework. Uh Dave Kennedy, those guys that are responsible for that went out there built this framework on how pin testing

should be. No more, you know, you shouldn't go out there and just hand in a nest report and say, "Hey, I pin tested your network." No, you didn't. These give a great thing. I love this site. I still go to this site on a weekly basis. You know, some of my methodologies I I use for pentesting, I built off of this standard. It's a great site. So something like something similar on those lines. Um built by those in the industry. Who are the best people to uh create a framework for use in the industry? Well, there are people in the industry. Uh shows researchers expectations in terms of disclosures and citing the previous works. I've actually

come across people that don't realize they need to site previous work. They just think it's okay. And again, it's a naive thing. It may sound weird to you, but again, you know, a lot of them are new. They just haven't done it before. They're not out there doing it like, you know, they're not out there doing it maliciously and they want to be part of the community. They just didn't realize that's what they were supposed to do and that's what it was expecting them. I respect that. You know, we want to correct that action. We don't want to crucify those people. We just want to correct that. We want to make them better members of our community. And we

want to help new researchers avoid those troubles and pitfalls. We don't want you to end up in prison. We don't want you to end up on attrition. We don't want to you to end up ostracized from the community. Bad things involved. So, what I'd like to do now is just do a discussion and uh just kind of talk amongst ourselves and basically kind of get some ideas on uh what you guys would like to see in uh your just thoughts on the topic in general and what you guys would like to see in some kind of a framework uh framework for this. Um so, uh open the floor. The University of Detroit Mercy uh a few years back was

trying to build something like this, but it was really only for the academic circle. So you know I mean it was you know supposed to be you know central repository for information but it just completely died because there was not there wasn't enough a there wasn't enough involvement with other universities b I don't think there was enough involvement in just the hacker community in general but c you know coding so I mean how can we get you know the hacker community involved in building something along with academia because you know from what I've seen they're pretty much at odds and and what Mark asked was that um you know basically was referencing a uh Detroit

project that had uh was building tried to build something similar that died and because it was more meant for academia and uh it died out because there wasn't the participation. So how can we go ahead and how can we build that and that's a good question. My opinion is I want to definitely get people that bridge both gaps. You know we have people out there that have done their masters. We have people out there that have done their doctorates. We have people out there like myself that are working on those kind of things. Uh you know those are great people to have in there. But you know it's not just those people. We don't just want those people.

We want a wide we want a wide variety. You know, the goal is and the other thing is to get people that are actually passionate about it, not someone just like, yeah, whatever. Get people that actually care. It start, you know, hey, I what I like to say, yeah, we got 50 people that care. You know, I'd love to see that. I don't think it's going to happen, but you know what? PTS didn't start with that. Other frameworks, other other research that we've done in the industry didn't start with that. Might start with one, two, three people and then it grows. You get out there, you get the passion, you get passionate people out there to help promote it. And

as you help grow it, it should help getting bigger and bigger. And so that's kind of, you know, again, my opinion where you might go. Yeah, go. Well, one thing I've noticed with academics research systems is they have a especially medical and other scientific disciplines where you're actually basically have to prove things through reproduction and other types of facilities. Some of those would be extremely useful and other of them would be way too honorous and burdensome because in in technology circus the results are pretty evident. You can see what the output from this exploit was or you can see what the output was from this tool will happen every time somebody runs it in the appropriate

circumstance. With those you have the longer term where people have to basically go and replicate things inside of a very specific circumstance and observe data over a long period of time. But I wouldn't dismiss those things because of that because there are a lot of things you'll find especially when you're doing large statistical data sets they are very very good at. So pulling in components from those guys especially in documentation sense having a very standard way of doing an abstract having a very standard way of citing people in in using an existing consistency you want to rebuild it and then taking those uh burdens of proof and pre-existing requirements and adapting them specifically to the type of research in

security that we're doing. I think it's a great point taking some of those academic to research um you know resources and adapting them to our groups and you know goes kind of goes back to what Mark was saying where we need to find people that know those systems and help bring them into the project or at least are interested in learning those systems. Um you know me I I I can tell you some of what she said I I know some stuff I don't know. So, you know, me, I'm not a great resource for that. But obviously, this is a this is a project that I'd like to see done by the community, not me, not our other, not

just me. I want to be a part of it, but I want other people. And that that's that's, you know, kind of the end goal, but definitely having people in there that will help bring those resources. I agree with you 100%. We definitely bringing taking some of those resources from academia, putting them into our into our system, and adapting them to the way that we work. That to me, you know, that to me should be a very important goal. Casp. Um, so this was brought up at Schmukcon and I I don't think you were actually at the Schmukcon talk that kind of I mean I'm not going to insult them, but it went a little off

its rails. There's a lot of talk about academia versus hackerom. Uh, and one of the things that kept coming up and they kept circling back to was publishing and very specifically publishing and peer reviewing. And one of the single biggest problems I find with exploit releases and a lot of the other stuff that we're doing in terms of the research we're publishing is that there is no formalized peerreview method. There is no way of someone turning around and saying, "Yeah, this is reduc reproducible. No, it's not." Uh, in a way that's actually consistent. So what we've got instead is we've got these like little groups kind of going, "Yeah, so and so is a jackass cuz he can't do

x, y, or zed. You know, his work also sucks because of this. I'm not even looking at it." And you've got other people kind of going, "Well, yeah, you know, this this one particular piece of, you know, code or something that he wrote actually sort of works, but we're not going to talk about that because we're, you know, busy doing something else." So, I mean, my question basically is if you're trying to create a framework for this type of research, where do you start with the peerreview process? How do you get a group of people together who are actually involved in that? And this kind of ties to the thing that uh that's a good that's a good question. That's a

problem that needs to be solved on there. Uh my p I again I just have personal opinions on this. My personal opinion is I would rather start walking than running. So my thing is get the ideas in place first and then work on building that and building that out. But that is a great point you had comment. The biggest advantage and I go back to medical field because it is one of the biggest bodies of research that they have and it's also a disadvantage for the same reason is they have a journal system. Yeah. in in journals is how and the more reputable journal typically the ones that are better established they're the ones that are willing to publish the

peer review alongside the original data. So they they that is how that networks and the people the payment system is broken in those systems but agreed agreed but we don't really have anything short of a few online websites. I was going to say what like 2600 hack he hack 9 those kind of yeah I agree with you those and that's a problem too. I mean, I'd vote to see some of the uh the some things that I the one stuff I can see that may work for that are some of the item mags. Yeah. Don't do that aren't peer reviewed that you know they limit you to 15 sources, that kind of stuff. But honestly, that's that to me is

something I'd love to see is us getting some kind of our own academic thing, whether it's SANS doing it, whether and that's something uh you know, I mean, I'm on a couple SNS email lists, uh, you know, advisory board list because I know some of you who have scored high on the test, have been on those things. Um, you know, it's something I pushed for on there and said, look, this is something that would be beneficial. They have the Sand Reading room, great room, and that that's that's an act that's a resource, too. But to me, I agree with you 100%. We need to have that kind of stuff whether it's sp whe whether it's

something that starts out of this project or if it's something that someone else that uh you know like EFF or another group helps sponsor. I mean that's a that's a great point and I mean it is something where having a um actually having a resource for people to publish to and whether it is peer review or not and I don't know I don't know what the right answer on that is and that's a great that's a great question to bring up to a group when building this out of frame. Well, I mean, the thing is that whether we like it or not, we're going to see a peer review process that occurs. Uh, one situation, yeah,

you know, in the medical and actually I was going to say physics, bio, and a couple of the other uh sort of scientific bodies where they're actually doing rigorous academic research, the peerreview process isn't ad hoc. It's basically just it's always going to get you send in an article, it's going to get reviewed by somebody before it gets in a magazine. um we have something that's a lot more ad hoc and kind of random if you want where yeah you know maybe that vulnerability that someone's disclosed actually really matters but if you've been on you know full disclosure for example there are always going to be those ones where you're just kind of like why is this guy putting up a

cross-ite scripting thing that everyone already knows about what you know what the hell is the point of this and full disc actually kind of is almost like a peer review process except that it's a peer review process it's a goat rodeo you know you've got five or six people insulting each another one guy saying, "Well, actually, this is valid." And like five or six other people going, "Who cares?" Reddit's about to be the only thing that's little above that. Yeah. Well, I you know, Reddit's Reddit's another good one, you know, and and I mean, I look at this from this point of view. On the one hand, we've got I mean, it's effectively the equivalent of crowdsourcing your peer

review process, which means that you're going to get a bunch of people insulting whatever the work is, and it's kind of a crucible. On the other hand, it sort of works in some senses because you're going to get stuff like attrition is another good example. you're going to get stuff that comes out that's actually pretty solid. Here's a question though now and and I agree with you on the crowd sourcing of people sitting and selling the work. Is that question question become does people you know by putting it out there crowdsourced and people actually sitting out there insulting the work? Does that actually discourage people from people building? Absolutely. I mean, I think of uh you

know, I don't you know, I hate to use Georgia as an example, but those of you that know Georgia know when she released her uh smartphone uh you know, whether it SATF was it um smartphone testing her or testing framework for a smartphone, she got crucified and I'm and I I thought it was the biggest bunch of crap that people were doing. It's like, you know, people were like, "Oh, your coding sucks. You're doing this." It's like, then make it better. Don't just complain about it. Make it better. She put a tool out there that was designed to help make things better for the community. She's doing what we all should be doing, building our community, and people

wanted to crucify her for it. And it's just like, are you out of your minds? I mean, this kind of goes back to something you were talking about, you know, with the cute clip art and the and the and the cute borders. Uh, the thing is that, you know, they're always going to be these squeaky wheels, you know, grumpy people who basically just want to insult everything that's going on. Some of them are actually probably good at what they do, but if they're it doesn't matter. That wasn't a shot at Boris, by the way. He's not here. I love Boris. He's a good guy. Um, but I mean, you're going to get these in every

community. You know, you get them in the physics, you get them in the biology community. I mean, I've seen them in medical IT pretty much non-stop. And it's it's a common thing to find people who are like, "Well, this is new. I don't like it." Or alternately, this person's trying to make money off this. So what? You know, so again, we go back in my opinion to this kind of peerreview thing where what do you say about this? You know, how do you turn it into something that's at least a little bit less subjective and a little bit more open? I don't know. There was a and I hate to use this example, but I you guys

familiar with rate my professor, rate my teacher.comites. Oh, those professor rate my teacher. Yeah. A while back, you know, I used I used to be um fairly um fairly active moderate. What they would do is they'd have a pool of, you know, stuff that you would go through and you would moderate and there would be a set of trusted people. So, I'm not saying, you know, at some point, you know, it's going to be, you know, subjective, you know, in choosing who goes in stuff like that, but in the interest of meeting halfway with academia. So because there's no actual um you know there's no actual peerreview process in the hacker community and there's a very stringent review process

in the uh academia community. If we could kind of meet halfway you know agreed somewhere it might help you know push things forward. being halfway would be a good thing. And I mean I I'd like you know you say people that go through what I you know me personally what I'd love to envision in a system like that is people that go through and the people that care you know people they might not necessarily you know and those that can do certain things they do it they can test certain they have certain aspect but the point is that they care. I mean it needs to start out with people that actually give it. That's what you need

to start with. Any you guys have uh anything or any questions or add to the conversation? Yeah. Let us keep talking please. Yeah. again you know everyone you're part of uh you know part of the community we'd love to uh love to hear everything that you guys can uh you know anyone can contribute thinking on the topic of a peerreview group a couple years ago I joined hackers in my home city and it's amazing where is that area Winnipeg okay hacker space it's a beautiful family of people who come from various different backgrounds and technology various different industries and we can look for each other's work say something's been present at example was recently at Fest we want to do a run

through of the presentation. We'll give them some honest feedback. Um we're all friends enough to know where the lines are but how to give meaningful criticism that before they take a talk to some conference it's testing for it. So same with dropping uh information have it among the Google uh community of hyperspaces and say here are people who show commitment and enthusiasm for making it better. That's a great idea and that might be a great place to start. Um might be a great place to start. Um thank you. uh what's it called? Uh recruiting people. That's a I mean that's a great that's a that's a great point. I never I never would have thought of that with the hacker space.

That's a great idea. Really awesome. Yes sir. Yeah. And to go along with that uh I was on the board for issa Ohio. They uh we always come up try to bring people that are new to speaking or just new to presenting a topic and bring them into it works pretty well. And yeah, and that's the great thing about groups like ISSA. I even think that's the great thing about groups like Defcon groups, you know, again, it's just it's groups of hackers that, you know, again, they care about they care about the industry. They care about it might be a little section of it, but they at least care. They get out they they care enough to get off the

sidelines and do something. And you may not agree with everything that they do, but they definitely they definitely care and they definitely get out there. So, you know, groups like that are, you know, per that's, you know, great example. I mean, there's some those are some awesome things. And again, this is one reason why something like this I want to open up to the community is because those are ideas I never would have thought I'll be the first one to tell you I never would have thought of thought of those and that's a great even the peer review system that wasn't even something I thought in scope of this and but it's it's an excellent point. really

is to kind of riff off what he was saying, you know, in terms of, you know, like kind of and what Katherine was saying, you know, in terms of, you know, distributed um kind of an ad hoc pure review process, you know, what if, you know, we got like an organization of maker spaces. So that way it's there's still some object um t or subjectivity in it, you know, anyone who joins the maker space can be a member, you know, and can, you know, look at this. But these papers are distributed to different hacker spaces and you know they look at so like my set your hacker space you know I don't know you know

burp set you know all these different you know collections of people they can take a look at this and then you know say hey you know this is what we think about it and you know get some good feedback and and the goal is not to rip people on it the goal is to give constructive feedback and I mean again that's a skill to learn that's a skill you know unfortunately people some people in every industry but since we're talking about our industry people in industry lack at times I mean, I've, you know, I've seen people say, "Hey, that's the stupidest thing I've ever looked." But I've had other people say, "Hey, look, you have an error in your code.

Here's the way to fix it." Or, "Here's a good idea that might make it more efficient." I mean, God, if you've ever looked at any bash scripts I've written, it's the the most asinine just awkward kind of things that I can give to somebody else, they go, "Oh, yeah. If you just do this, this, and this, you're golden." Well, hey, it worked for me, but that's a way better way of doing it. So, kick ass. Awesome. Well, that's that's what we want to that's the kind of thing we want to see. The other thing to consider about that criticism is that, you know, the people receiving on the receiving end of the criticism also have to be able to take it. That's a

great point, too. Something that can be kind of problematic if you haven't been exposed to that, you know, that the kind of academic crucible, if you want, uh peer reviews, for example. I mean, I've seen people come home crying from from doing dissertations because it was just like, "Yeah, I had three errors." And you're like, "Well, it was only three." You know, it's not like they kicked you out of the program. Don't get so hurt by it. You know, you're doing work. your work is not your baby, it's your work. And we, you know, I've done, you know, to talk about the dissertation and stuff. I've sit in on uh sessions where and again, it's not actually the the the

defense, but it's defending what we've written so far. Yeah. And and and what you do, you get three or four people that'll just sit there. It's non-stop peppering you question. And honestly, by the end of it, you just feel like shell shocked. But the thing is, it really does once you get over that feeling, it makes it better. And it's, you know, and it's a great great point. So people, you know, learning criticism and, you know, something in this maybe making it clear, hey, you can't take this personally. Well, it's been a great discussion, guys. I really appreciate it and uh really appreciate it. So, what I'm issuing is a call to action. As you

probably guessed, what I need is volunteers to help design and build this framework. So, this is a project that I am starting and you know, again, it's not me creating this framework. I need people in the community to do it because there's stuff I don't know. There's stuff that other viewpoints of people from various experiences in life and all of that helps again build grow make it better. So right now uh right now these are kind of the uh working uh working stuff. This is the stuff we're working on. Uh website is infoscresearch.net. Um and just put it up this morning. It's really ugly and uh basically it's just kind of a placeholder kind of thing. So, actually,

if someone knows how to web design, you might be the first person I would love to hear from uh to help with this because my web design skills are um clip art. I know what's worse than bad. Clip art. More clip art. Pretty Hey, you know what? Clip on the site. So, it's amazing, dude. I didn't take care of email uh email infosc researchframework@gmail.com. It's a mouthful, but then Twitter infoscrf uh as well. You must a follow obviously dr.com. So, thank you all for attending today. I really appreciate I appreciate the interaction and uh you know so long. Thank you for all the patience.

Should drop the uh cable. You dropped the cable.