Grappling Hooks on the Ivory Tower: This Year in Practical Academic Research

giant reduce Jeffrey of course for speaker request for this talk we've requested a top secret clearance shaving yak and the reason for this is that essentially it's our goal here to take all of the mysteries of a bald papers and show you how both to break through that pay wall and also how to avoid reading papers that feel like you're shaving a yak you'll notice of the yak is unshaven we're both academics we refused to shave yaks and if you ever find yourself reading a paper that feels like it's yak shaving just stop and find a nicer scarf and carry on so yeah we're here to talk about grappling hooks on the ivory tower this year in practical

academic research and instead of this being a title and subtitle sort of thing we've actually broken this talk into two sections so these are sort of our two titles are working with today so the two sections will kind of be going through our context and information flow and practical papers and examples the contacts and information flow is really what is an academic environment what is happening in it how do you optimize it and how do you really like break in and get the information out that information flow part is exactly that you want to talk about how to get information from academia into industry and back and forth then we're going to follow this up

with the practical examples exactly what is happening happening in academia that you might not have heard about yet so as we go throughout this talk please do ask questions the entire time because I the story I like to tell is part of why I wanted to news talk is I was talking to one of my friends they said I [ __ ] hate academics and I'm like hi I'm an academic and they said well but how are you an academic and I'm like well I published papers I do the research I apply for grants I connect with people in industry I do all of that work that that an academic does and he said but

you don't act like an academic and we kept talking and talking about this the point where you know they finally said well I still don't understand what an academic is now because I think you break the mold and through that time yeah he has so many questions that we really want to answer here today so if anything we're saying doesn't make sense or you need clarification put your hand up immediately we want to answer everything as we go through so our first thing that's important when you're talking about contacts and information flow is why do academics do what they do now most of you probably know citations everything is always about citations for an academic citations are what get you

grants they also get you any promotions they get you your funding they get you any sort of thank you salary increase as well the interesting thing about citations is that you can see you have a whole range here you have some people that 599 sounds pretty good you've got the 1500 2500 and then you have some researchers that have 13,000 and what these numbers can sort of indicate to you is that these people are collaborating in the community they've probably got lots of students they are reaching out to other universities other groups and they probably got lots of funding because each one of these things you have to pay for the student you have to pay for the materials you have to pay

for any of the resources so that's also things so the citations indicate those things as well um I I want to add to that too as an academic that currently doesn't work in academia I'm a graduate student but I also have a pentesting job on one of the biggest things if what you're trying to do is set up some Industrial Research if you want to grant for this really cool tool that you're going to develop or something and you're trying to get it from DARPA or you're trying to get it from NSF if you find that you're eligible for the best way to establish the kind of cred and a name that can really get people interested in

like reading your proposal and you know establishing that you really do know what you're doing sure the merits of the proposal are what is evaluated but if you have if you know what to cite if you can say the important papers in your field to show that you've done your research in that proposal and if you yourselves have been cited in this area it really helps you to establish credibility so even if tenure isn't what you're shooting for this still helps yeah it's basically like the fake internet points of being a researcher and the interesting thing here is people think academia is so different from industry but you know what we have this exact same workflow

starting from choosing a topic like what are you here for what interests you and then moving into reading materials this is where academia is a little different from industry is we spend four months on that or six or more depending what you're doing for myself I've been doing lit review for about three to four maybe five months now just to make sure you have that base for what you're talking about but then you move into developing your plan like how are you going to tackle this how are you gonna make sure your time is best spent then you move into actually like experimenting making your tool interacting with the people getting involved and usually like if you

haven't [ __ ] up your experiment you [ __ ] up your experiment there's no way you're doing this right the first time people will be like yeah I have a gender-balanced experiment it's all like everything else is counterbalance it's fantastic and it's like how can you only have one female then out of 15 participants you can't even balance across an odd number in when you're doing a binary gender balance and they're like oh chips and you realize that after you've completed all your work so what you do then you expand or contract your scope it's like okay well we already done the research maybe I went to the gender balancing part anymore or you know I've seen that

happen a good number of puns but then you do you expand or contract your scope but then from there when you go through the loop again you're not choosing your topic you're modifying your topic and we've all been there um doing your research or your think or doing your research think you want to do one thing and then you realize there's something else like when you want to look for something in your house but then you realize your lightbulbs burned out it's like oh I need to find lightbulb then you go to the hallway and you're like oh this door is broken I need to fix the door and there's like all these steps it's kind

of what happens when you're doing this part but then sometimes you end up going through this again reading your materials developing your plan experimenting and then you get to the point where you're reviewing your shortcomings you're like well I'm pretty sure I'm doing this right I'm pretty sure the informations correct I don't understand what is is different so you go back to reading your materials and it's like then you realize the paper you've decided you look at all the other citations that have come after and people are sitting there and being like no this papers actually [ __ ] that paper is actually wrong and that's the thing that happens as well that a lot of

people in industry don't understand is or don't necessarily taking is that papers can be wrong too but what happens there is it's not a that paper is completely wrong it's that paper needs to be improved and so you look at what sites that paper since it's been published and you can find the next piece that works on that or in the next piece that works on that and as sort of this layering effect of knowledge not a concrete this is it forever that's not actually what happens here so even in academia we're still always improving still going through this iterative process but sometimes some other people pick this up for you and then even from there you might keep on

going and then at some point you're going to develop a plan and then realize as you're developing your plan the whole idea [ __ ] you can you can actually create it you can actually get things done and we've all been there right it's like I want to do this amazing thing and you realize wait I don't actually have a quantum computer yet so then you go straight to expanding contracting scope then some do you sometimes you get really enterprising grad students and what happens is you go straight from reading your materials to experimenting [ __ ] the plan we're doing this live [Music] and we've all been there so yeah this is this is totally a process that we go

through differently as well through different phases of our career as researchers and one of the things that that we find that we do a lot here is that process of narrowing and contracting our scope to match up with what we find is actually possible and so this is what happens when I'm writing a tool and I really want my tool to be a perfect disassembler for this strange architecture I want my tool to be the universal arm hacking tool and I'm going to put it in Metasploit and see that it works what you do too is you start to do it and then you probably will decide like maybe I can actually just solve a

limited version of this and test the waters and see how it goes we do that too and so a lot of the time you'll find some very small studies and they look like they've just shaven a yak it's this procedure that only works in this one narrow little tiny circumstance that could never possibly happen in reality and what this is is they've said well I want to actually publish something before the end of my career and so I'm going to make this proof about a very specific part of this and then later on I or someone else will generalize this and that's the important thing is you know when you're trying to write a thesis it has when you're in academia

trying to write a thesis for either your masters or your PhD you definitely want to make sure you can actually leave school eventually I kind of like being a career academic and is constantly learning but at the same time it's like I really want to get a new project one day and I'm sick of it and so yeah that scope has to be an appropriate size for you to be able to write about one topic like one very specific topic in its entirety which is why academics seem to put themself in a niche like I mean I do social engineering robots like who's gonna hire me for that right but you know so what does this all lead to in

industry you know hopefully it's always leads to I'm being able to release the tool helping other people question

yeah yeah sorry okay so help so how so how about actually tearing apart experiment and to experiment and test so from academic side or industry so from academic side it's part of your plan so if you go back to this here yeah that's no you know in academia they are so completely tied together when you were developing your plan and your experiment you were deciding right then how you're testing you are making your rubric for how this passes for what happens for what you're actually looking for because that's part of being a thorough researcher is you can't just really go in blind it's all about like I said before you might want to do an

experiment that is gender-balanced to figure out for myself for example how some of these robots whether I have a bigger effect on persuading people using a robot if they're female or male so I know right then are there I have to split it into female male I have to know I'm testing it on both groups I have to know exactly what my experiment is I have to know exactly every single step and then I have to put out every single piece of a survey every single piece of what I'm doing and then I have to submit that to an ethics committee before I even get to start so you're sitting there yeah and and you're sitting there

and you're doing this and it's like yeah I've worked on this like beautiful shiny ethics and it's gonna be accepted and they're gonna love it and they send back a 10-page thing saying this sucks and so that's it it is that the testing is always part of that because that's more the the important thing is your experiment is your set of tests so does that answer your question if you've done in experiments in academia as with here and you haven't tested it and documented your results it is as though you have not done it at all I that's one of the reasons that we write the papers even artifact of what we've done and justify our existence it's to say this

is my past year of work um we don't often include the failures in this that we're aware of because there are page limits of course you wouldn't know it but there are and there there's also this other process that we'll get into later of like the review and then revise and resubmit and shepherding it's it's a very different process that's much more artifact focused than what maybe industrial resources are used to cuz like as a grad student I stay home all the time um the only way my adviser knows what I'm doing is I write down every single single little thing I do it's like installed this took me four hours that eats beyond there it's like that's half a day

installing these very specific academic tools that only fit on this operating system and this set of requirements and you find that out as you [ __ ] it up so it's like constantly you document your fuck-ups only for your visor no one outside if your little tiny community ever sees it and remember how I talked about narrow tools are done tools yeah and so again what is this sort of lead to an industry hopefully when you do your research or you're trying to do a pen test or you're doing whatever you do every day it'll hopefully lead to a promotion for you or it will lead to a tool or it will make somebody's life in

your company easier or happy client there are all these different metrics and in academia we have won conferences and journals this is like the top metric everything else sort of falls underneath because what conferences and journals are is for us is again that like permanent artifact that gets put up there your name is tied to it like forever I mean I I've read turns paper how many times because that's just one thing that you kind of just get to learn how to read and all these papers from years and years and years ago you can still get them you can still read them there are people there are librarians that are they're digitizing old copies of papers

so this is the thing is that there's this is how we make our lasting mark we put these particular journals on the slide because they're particularly poignant ones for our particular segment of the industry when I go and I'm like I really like to do this cool project or I think that this new technique would be really useful for my clients these are usually the places that I look when I do my literature review every year I do it I read all of the papers that came out and like sift through and pick which ones are useful and which ones are not um I I go through Oakland musics and CCS first and depending on what I'm looking

for I'll go through some of these other conferences also and I even make a point of attending I Tripoli Oakland every single year in order to make sure that just these peoples work is right in my face to make sure that I'm aware of it because awareness of the work is kind of a pillar of academia it is as though it is as though someone is going around in a CTF challenge and they they've never even heard of smashing the stack for fun and profit in a big part of my world this is the same as admitting that you've never read Claude Shannon's entropy paper from the thirties and the other thing too is I also listed chi

here because there are some venues that are outside security that still talk about security and privacy chi for example is on human-computer interaction it's about usability it's about how people interact with their machines but they've had actually a fairly large showing for security papers lately and that's not something that people in this community might be aware of is that this issue is not only being tackled in our conferences but almost every other groups conferences as well because it does affect everyone so fully especially with my I'm doing some technology privacy and sexuality research and that's all in CAI that's not anywhere else here and so I really found that interesting is that if I want to talk

about like how people can use teledildonics to put it like make a fingerprint of people using these devices I don't go to security conferences I go to Chi and this is all interesting because conferences and journals are also slightly different like Falcom is saying conferences have page limits most of these ones are I like them because the pages are usually limited to eight you have eight pages to get everything you've worked on for six months out my entire taxonomy of language theoretic security flaws that I published in November was eight pages long and woefully incomplete but it's that way so that you can actually get through it and so this going back to that cycle of how academics do things

and that narrow focus one more time is this idea that you have eight paper eight pages everything needs to go in there so it's all bite-sized chunks and that really is a reflection sort of what we do here because we have the hour-long talks we have the half-hour long talks and that's sort of our sound bite as well so as you hear stuff like this I'm sure many of you approach the speaker's after and that's usually what happens with these papers as well is you want people to approach you after to collaborate and get those citations and so this process works kind of like this you've got the author and they get the fancy hat and they submit to a

conference and the conference will look at this paper and be like okay what does this actually contain we need three experts to look at this and rip it apart it is not a nice process they are trying to make sure that the science is solid and that the information is there and good if it's a good conference so it goes up to your reviewers and there's always reviewer number two that is the person that gives you usually the hardest feedback the one that's like I think everything is broken and this paper sucks you should read this other paper that you then find out it's actually not anything like your paper but you know you have to take that

criticism and these reviewers might be your friends they might be some you've collaborated with there might be that one [ __ ] that you see that conference and want to like hit every time you see them but they're still the ones reviewing your work there still might be the ones that you're constantly interacting with so you know you want to always make sure your work can get then like if they hate you and they're looking for flaws your paper still needs to get through them and then you know they do that listen you back ten pages of things you need to fix and you do it because this is your work you need your you need it

out you need to get into that top to your conference so you to your changes put it back to conference and this is where some changes happen is for some competence there's three things it's either completely accepted holy [ __ ] that's amazing both blew our minds yes please there's also the stage of we'd like this but it really needs polishing and a third one which is like oh [ __ ] you know and then some conferences are starting to do shepherding and shepherding is when they take you through for a while so they like the paper but these changes need to happen so you make the changes the Shepards like you didn't really dress these

things can you try this again and they'll do it again so not only have you spent like these conferences we talked about our once a year you will only have one time every year to get your stuff in it's like the worst thing ever because if it doesn't get in you have to wait an entire like entire year to submit again someone's a submitted you get to the conference and then they start reviewing that takes three months dot three months leads into another three months of you like or maybe even a month just changing things back and forth doing that shepherding process and then six months later the conference happens and then the paper get puck gets published but if

it's I Triple E or ACM then no one else gets access it for a year so you do this and then your work doesn't actually get out to the public for like three years and so that's part of what we're gonna keep on talking about here will help you get around that a little bit but the purpose of this process even though it may seem obtuse is to ensure that this work is correct and stands on its own merits one of the things that happens a lot on our side in conferences like this is the reviewer I've done this only sees the abstract and I'm scratching my head and I'm thinking who are you even do you

know anything about this topic what is going to be in this paper what is the conclusion whereas with this process they get the entire paper as it is going to be in the template that it's going to be in the format that it's going to be in and then they get to make that decision and oftentimes those reviewers and I've actually had the opportunity to review from that side as well will be people that have been pulled in from you know industry like people that the reviewer has known that are qualified to do the review but aren't a professor anymore they they will say um you're really aware of working this area because you work in

this area does the thing that they're saying about the industry that you work in actually makes sense because the last thing in the world the Journal wants is to be the Journal of either badly described research or the Journal of research that is just outright wrong and they keep needing to retract and so the other thing here too is that as researchers we [ __ ] hate reading bad papers like the thing is is if we're one of these reviewers like how often you get see papers and it's like all formulas or all difficult language and you see people postseason it's like oh I don't want to read that yeah we get that too we don't want to be sitting there and

doing all this so when you hear other people saying like oh academic academic papers are difficult to get through what end up hearing is what have the academics ever done for us and so yeah to just really drive this point home is that the idea of papers is to get people to want to work with you to get your work out there to make sure you're changing the world how you can and having that little bit of an effect on science so when you do a paper wrong you're actually shooting yourself in the foot if you're making a paper that no one wants to read no one's going to read it and this is a perfect example you

can't say what the paper is I'm sure Twitter like that and no one will ever find out because it's the it's in the Journal of hadley described research that no one reads so the question was is why do people think they can get away with not stating their contributions it's sometimes people are just really bad at summarizing yeah so it's some conclusions when you read them is just like and then we finished like Thanks yeah it's like it's more almost like a to be continued and so that's really hard because we'll get it to this in a few more slides but how to read a paper one of the first things you do is read the introduction read the

conclusion and so if they're not meshing and you can't go from one point to the other in your head and think like yeah there's I definitely want to see how this middle part happens like [ __ ] you're writing a bad paper there's another point that he makes the sides not stating your own contributions and that point is misstates related work which I yes on you can't get away with misstating you're related work there are certain fields that have certain bodies of work that established that field and so if I read a paper about leg sack and I don't cite any of sorry about this this work um people are going to look at my paper and like hmm do you know what

other people are doing here are you reinventing the wheel have you checked this for correctness against other things if you find one that's a seminal paper in your in your field and you find that it's wrong you still have to cite it to say and this one was garbage engineers why you can't just ignore hits if your paper will be rejected for it yeah and sometimes the conclusions part of that too is that people once you've worked on this project for a year and a [ __ ] or however long you've been doing it sometimes you get so sick of it and it's like I really just want to throw up on this paper at this point

and it's so you get to the point where you're like I just I can't look at this I've read this one paragraph 50 times and that's the one like the point I am in one of my papers right now and it's just like I'm so happy to be here so I don't have to work on it for a week and you know but I love this work I think it's amazing and I will talk your ear off if you let me but it's at the point where I've read my own paper so many times and each parts individually I'll never read it from front to end because it's just I know it all so well so it gets that

point where you know you you think everybody else around you knows what you know because you know it so well and that's what happens with the conclusions yeah and there's other things here too is like is the paper hard to read and unfortunately one of things is checking for poor English there's a lot of people that are international I'm using English as a second language and some of these places pay five thousand dollars per paper to get somebody to a grammar check it and check it for language and all this sort of stuff and it's something who doesn't speak the native language natively so it's still wrong and this is something that's really bad because if

you get into a certain group that publishes 20 papers here that's a lot of money coming out of the school to pay for these grammar checks which aren't even full so it's a little unfortunate and it's something we judge people on but it's something we're trying not to as much as well is is the content there is it explained well and that's what it should really matter here and then the last point like incomprehensible graphs I've seen so many graphs and papers that don't have their axes labeled and like so you're showing me a picture of bars it literally doesn't mean anything yeah or you or you find like all these pictures in papers and are not labeled

they're never referenced then it doesn't seem none of the numbers that did it the axes are labeled make any sense it's it's weird yeah I it's steganography so next how does information flow from academia to industry does anybody have any questions before we start

so the first or the second slide had citation count but it also had two other fields I'm assuming those are metrics derived from whose sites who like second degree 3 degree orders can you explain what those two are oh I was just kind of curious so I don't actually know and this is one of the great things I loved doing as an academic I loved saying I don't know but you know what I can find out for you within 24 hours and let you know because [ __ ] yeah I have a question to answer you know who does don't you the librarian and go chat about that in just a moment

Thanks um have you guys found that when you submit to a conference with a very specific focus and perhaps you're submitting interdisciplinary work that those reviewers tend to project their own expertise onto your work yeah you maybe speak to that so one thing I so I do crazy and interdisciplinary work I'm taking robotics I'm taking sociology I'm taking psychology and I'm taking security and privacy and shoving it into one paper I have close to 85 references on one of my papers the normal is about what 30 so you end up doing that and you have to actually approve points that are so simple in other fields like robots with faces make people want to talk to

their face like you think that's kind of straightforward like for me coming from Robotics but that might not be something a lot of people in this room knew or I have to explain that you know when a robot acts like an animal people treat it like an animal like gasp for me but that doesn't mean it for anybody else here so that's another thing too is something that you do not necessarily have to cite in another field because it is a no one trusted fact that everybody cited for 10 years all of a sudden you have to find those references for that paper and you have to re like create that knowledge and juxtapose it into a different field and

so that's exactly it is these people like well in I feel this [ __ ] you're trying to pull doesn't make any sense and it's like but in that field that's standard so they definitely try and say I'm the expert here [ __ ] you and you're like no [ __ ] you at risk of or any other time here I want to talk about this undercurrent just a little more um so one of the things that we do when were writing the papers is we cut out anything that we think that the reader couldn't do it in order to keep it brief and fit within the page limit and not keep restating things on this we call an intuition

required from the reader or how hard is it to read this paper um there is a body of knowledge that everyone in a particular sub field is expected to have and that body of knowledge is usually the most cited paper in that field so if you feel that you're lost and you don't understand the terms that people are using the best place to start is to either ask a librarian hey like what are the important papers in this subfield or if you can use the Citation Index yourself and see what does every paper that I'm trying to read a not understanding cite let's read that yeah and the other thing too is on like I

mean come approach one of us we are academics we'd love to help you find this because this is how we support our community as well as our community is there's two that we're trying to balance but we want this information flow to happen and like this is what is next is this idea of you know this might be you this might be somebody trying to find that piece of work and I Triple E ACM there's like there's decry de Gruyter there's what are the other journals journals publishers of journals Springer yeah yeah no that one doesn't exist it's just forget about fine they do so to force people to go to the conference with their research grant

budget it has the effect of making things somewhat inaccessible but we we do have some ways around that for you don't you love to share and like number one thing I use as a grad student is Google where yeah so Google Scholar like scholar.google.com um [ __ ] this guy like right now mint they will find the abstract from the impact site with the paywall on it in another PDF and then that's the two of them together for you and then you get this a hundred and eighteen versions of the one paper I found and like honestly it the first one might be two ACM the second one I might also be two ACM it might be to another

ACM place but then you have so many other versions that you can find this webs are find this PDF and that's the great thing as Google Scholar actually helps you get around these um I hope they don't stop it now because we're bringing attention to it but it's fantastic and honestly the thing a few years ago used to be library search tools like going into your school library and finding that information but honestly they don't do the keyword search nearly as well as Google but the other fun thing here is you get a filter bubble on your research and this is extremely bad because my one friend and I she does threshold cryptography again I do robots and we're trying to do

mobile research and I would only get like stuff vaguely related to robots she'd only get stuff vaguely where they related to public key infrastructure but it wasn't the mobile permissions that we were looking for and we do the same keyword searches and get different papers coming up and the citation numbers would be different it would just it was really interesting that that was a problem so we had to go throw things so I mean you go in incognito mode but I mean as a researcher that's really [ __ ] irritating because again I might have used 80 papers in mind that I did by guarantee you are probably read another half of that and thought they

were [ __ ] and I'm not including them so you end up doing all it look throw these papers by the time are done some academics are amazing at naming papers some are just like generic word generic word generic word and then another paper is that but like in a different permutation and it's really hard to differentiate so I mean I really like the purple links for that one reason as well with these two organizations membership in them it's not closed and membership in them is not expensive I'm a member of both it costs me less than two hundred dollars a year for this I can access so many papers an ACM every month so many and I Triple E

from some of your larger employers are likely subscribers to one of these two as an institution and so you may find that magically if you try to access the paywall paper from work it might just not be pay weld anymore and access to these is usually mediated behind proxies and network access controls and things like this um another thing with them too is that with ACN you can subscribe to a magazine such as communications of the ACM that will distill what's going on in the field for you and the purpose of this is to help you keep up to date so I mean I don't mean to plug for them I'm not really affiliated with them in any

way beyond being a member but I personally find them useful and in fact one of the papers that we have at the end of this talk in our reading list showed up in my mailbox from one such description as I was packing to leave for this conference and the other thing with these two is if you've ever gone to school and you still have your school email address you can usually still sign up for an account and at the student price which is like 10 bucks so kind of nice see if your home institutional walk and it still works is all my party that's very true - nope they don't start that the other thing here too is that

you can also go straight to the offers so the thing with authors is that their CDs again are based on citations but other things they can do is if they use seminars or presentations that's also how you can get grants that's also how you can get people to pay attention to you also your press coverage if you get press coverage for your university yeah they're gonna want to fund you like talk about the people who did car hacking right and they were working with University of Washington for some of that the University of Washington Scott much attention from other schools for that like I still have that mention in my school occasionally too so doing that

sort of thing for an author and using their work for that they will probably hand you their paper because there's nothing stopping them from giving you a PDF emailing it so when you're going through those Google Scholar searches look at who the author is and find their email it's usually posted and just say hey I'm currently doing this I'm going to be you know using our I'm looking for research that will support this this this and I really like this paper can you give me a copy and yeah they handed out like candy just ask so if you're frustrated by these paywalls just approach people talk to them same thing you do at this conference to get information just be a

friend and that's how you can get things from academics as well so again you can go through these groups or you can go straight through the academics the other thing we already discussed was you know walking into the library seeing if the computers unlocked printing a paper not a can't really you know support that but you know it's a thing that can happen walks you just walked in there I mean use your phone take pictures a screen whatever some institutions will will restrict access to library I'm Stanford generally I think will allow you like three days per year to go in for free the place where I'm at school University of Lethbridge literally if it is open

you can just walk in all of the computers with access to the journals are unlocked and you can just go through them on University of Calgary is the same way for most the most journals not like legal ones but the rest you can just use them um so the guys things in the front hi so some some librarians are very supportive of getting information out there some librarians will actually give you this information themselves if you walk up to librarian and ask and say I need access this paper for this research I'm doing some of them will say yes not as many but there's an opportunity and I mean come on how many of you social engineer in here right you

student forgot the card forgot their number I don't know there's yeah those ways to do this people this about copyrights they're not trying to deny you access mmm-hmm microphone just as another point a lot of the journals will contact institutions about IP ranges they have leased and as long as you pull an IP from that institution you'll get access to the journals so sometimes if you can sneak on to the Wi-Fi or they have guest Wi-Fi it's still good enough to get to the journals you can involve yourself in any way with your local university or get to know some students or academics that is our point how you get the paper is that way exactly and again like

that's the thing is talk to authors talk to other people because even if you're like a proton academic and say hey I want to do this project they might be willing to help you there might be like yeah of course I want to get on this too can we write a paper on this like together and bring people from the industry into academia that's one thing I've been trying to do as well is when there's people in industry who have been doing amazing work and aren't getting the you know a reputation they need or they're not getting enough notice I'm like how would you like to write a paper on this and help them form that idea

into a format that will work for a conference because that's the thing too is what academics do is a lot of fitting things into very specific framework very specific ideas and like having to word things in certain ways and that's something we're experts at let us help you so if you have a project that you want to get published approach an academic we want to be your friends we're here to get that information back and forth because science is the most important thing and if writing a paper is not your jam I totally understand there are also a whole bunch of grant programs one of the larger ones is cetera STTR that are specifically targeted as a qualification criteria to

pair off a small business and an academic research laboratory to write a paper and also need the product it's a thing mm-hmm next thing is how do you revive abandon academic tools so as I said earlier this is my week off from my research and it's something that I love my research and I'll probably keep going with it but there are many people that as soon as they're done the like just but that I'm out and they will drop whatever tools they've made whatever they've done to make their thesis work because they are so sick of the topic you spend five years on one thing and one thing that's so so narrowly focused you sometimes

just want to like leave it and run away open a rope ramen shop in Chicago sell some waffles on the side like you're just out and you're cleaning up that technical debts once that project is over and you've got just enough of it done to finish your thesis and drop it on the floor that's to an academic who's trying for tenure or something like this is totally yak shaving it is a task with no reward mm-hmm so this is one of the things if you want to revive academic tools offer to take over the project when the person is done because if you touch their tools or their research or anything while they're writing a thesis

they can't use it they have to remove it from their research and they have to do that much more to finish even if it's just like a little things like fixing that bug got something that could get them in trouble and they might not be able to use that tool as part of their research anymore so it is a they've got a lock on that and until after you can't do anything off microphone we're not all students either we're not all subject to this restriction all the time so check if the person is a student as you get to know them and if there's student this might apply if they're a professor who's done it probably doesn't go ahead could

you elaborate and exactly why a person would ya be considered too contaminated if somebody else worked on it because when you're writing your thesis or dissertation it has to be your work and your work alone it's a proving thing it's how you get the dexstar clean because like the point of science and when you're a researcher there's this bubble and this is your research bubble and when you're doing when you do your research and you finish your thesis your bubble gets this much bigger all of science just like one little divot on it and that's what you've contributed yeah so satisfying but that little bubble has to be yours that is your masters you're the master

of that topic or you are like that's how you earn your doctorate in a topic is being the one person who added that pin to the map and so like that's just it's how you prove yourself and it's trial by fire and tears and blood and sweat but that's the thing is just no touchy so it ends up being a bit of a relay race so as soon as a person is done be there to pick up that baton be there to be the one that can hand it off to or like we said before offer to collaborate because if they're not a student and they're just a researcher [ __ ] yes please help

me there are times where I've had to take a robot and build a graphics pipeline for it so I can do some graphics work for bounding boxes on it to see a bowl on a field of white lines and there's a couple people in the audience that know exactly what that's like and you have to do this all from scratch you can't like use some of the other tools sometimes if it's your research that you're doing so it's nice when there are people there to collaborate on if you're not writing your thesis specifically um and some of these things you can even offer to give in-kind gifts to the researchers of whether like if anybody has new robots

or IOT devices they want to drop off please give them to me but the in-kind gifts are huge for researchers as well because the grant you only have so much grant money so offers to take it over offer to be there to work through it with them at the very end so they can give you a description before they run away to their ramen shop the other thing here too is talking with a technical debt be neighborly realize that yes they are probably done with this they probably haven't a job fix any of the code it's probably written worse than a first-year writing their first hello world program like it's not meant to be seen by people

that one researcher or thirteen thousand citations does not release any of their work because this so badly written it is just so disgusting the like no it would actually destroy my reputation to put this out there and so that's another thing too is if you're taking over this project this is something you need to be aware of is how much these researchers probably hate their own code and it makes them scared every day when they look at it so offering to help them with that technical debt when they passed off is very important I to a person at Copeland who had written a disassembler and they found that it actually performed better in the usability test that was standard

than Ida Pro and got the reversing done just as well and I asked them so when do I get to use this disassembler and he said oh I'm never publishing it I would be embarrassed if my students saw this code so please offer to take it over just yourself sometimes they will be more apt for that than posting it on public AB and so how does information flow from industry to academia well popular media researchers get our inspiration from some things you all do if you have a crazy project that ends up getting inish to any attention like that's inspiration for us because it's like oh they did that I wonder if they could do this this this this this you

get out your notepad you start making your plans you start thinking of your literature review and popular media is one of the great inspirations for us and so our articles on sometimes it is a like in a journal that we don't normally read and it might not be written by an academic it could just be like an editorial journal with an article and we subscribe to those to get inspiration as well it's absolutely fantastic or even like I don't have it on here but watching videos from these cons really like inspires me for sure and this is our favorite open-source tools it would love when there are open source tools that will do exactly what I need

available on github it's the biggest support I can get yeah there's two tools actually that I wanted to call it as an intersectionality sort of topic here the first one is called Tamron and you've probably you've heard of Tamron Oh your favorite audience member oh you what exactly and that's what it's for he said whiter gotta believe in good light oh yes okay wire wire the wire app is it's protocol is proved with Tamron so is TLS 1.3 large parts of it whenever you write a paper with TLS someplace really you could pretty much put Tamara into Google Scholar and find all the papers on TLS 1.3 this is an example of a tool is written by academics that's

used sometimes as we just found out today in industry and by academia it's kind of hard to use it requires a little bit of a learning curve I mean I'm super stoked to see someone else who knows anything about it but it really it's a tool that effectively does a lot of the symbolic analysis or sort of heavy lifting for you if that sounds familiar symbolic execution if you've ever used Coverity you use symbolic execution and this is a tool that uses the same techniques from academia to help you with that another tool that I wanted to talk about is called on ostinato and this tool is really cool if you're doing any kind of network

analysis every single paper that has to do with like the performance of something I I put a whole bunch of them there over like particular types of physical links we use ostinato to generate traffic at wire speed to test whether or not it works as its intended to so you can use these tools today oh so now it was a tool that was actually not developed by academia but it came from industry and it was just so usable that people carried on the work that people did to develop ostinato in order to try and and and do their own kind of extension to it and it's an example of how if you do your thing right and

explain what you did and leave some artifacts like publishing your source code and making an application that's kind of sort of maintain dish then academics will happily take it up and do all of the heavy lifting work and the work that may be hard for you to do if you don't have access to off-the-shelf 60-hertz why wireless hardware you may not be able to do that study on the bottom there but because you're an ostinato someone else was able to do it now we know and the other thing about academia is like after one tool is brought in and you hear somebody talk about it in a presentation it proliferate it's fast and everybody is

using it all at once all of a sudden you hear it in every single paper the next year because everybody is solving the same problem at the same time with their own tools that they're creating from scratch and it gets really overwhelming so yeah we're actually like almost over time yeah almost out of time so are there any questions anybody has right now Mike from so you brought up the eight page length issue one thing that's been brought up by a couple different academics it said is the oversimplification of complex issues is leading to a greater problem than we've ever thought it would do you think that forcing or pigeonholing these complex issues into an eight-page

like anybody's won't be tweet sort of it has to do again with that old process of of narrowing and broadening and making sure that you can actually do the thing within your allotted time you have like a publication cycle you should publish a paper every year or two if you can and so sometimes when there's a complex that you like that you have to analyze the entire body of work multiple papers by the same authors in the same lab to really get a full picture of what's going on you have to read systemization of knowledge type papers to figure out what's happening and journal articles rarely have a page limit that's that intense they're often 40 or 50 pages long if

you're interested in that much we want you can totally pick those up and read them so yeah the answer is you can there are some that are longer but for conference journals especially that short page length I've run into that problem a few times is reading some of them and it's just like okay but what about this what about that what about this and it's like but you're not thinking this edge Hayes or that edge case and then you find then you see what side of that paper sends and there another paper that covers one edge case and another paper that's written done another edge case but they redo about a third of the paper every time because

you need that background it's not that is usually what ends up happening is it's like you know paper part 1 paper part 2 paper part 3 or paper part 4 and you just follow up person's body of research and you can get that but in the meantime it's just like get that soundbite out so it is a bit like a tweet and to get the full picture you that's why we spend months and months and months on a literature review is to make sure you get all the surrounding papers and not just that one and when I see that paper that's incomplete and I can't find that work there's my opportunity if I was looking

for something to do and it's usually something the great thing is is you if you ever have a research paper you really like and you're reading it there's usually a future work section of everything they wish they could do and if you want to make that one thing happen that's a great opportunity to do that collaboration any other questions all right so being as we are almost out of time we won't be able to get through all of the papers that we put on our reading list and I'm going to see if it's possible to publish these slides and maybe we can get that out there but I want to talk about how to read papers

instead because it's kind of fun they can be a slog and it's as you might have noted it's a slog so what we do is we make multi pass sort of reading of them and straights actually pointed out a paper to me that's kind of Derringer that describes this process which we identify on the bottom left you can visit that link it'll go to the paper this is the thing that we call a locator so anyway what you do is you start by reading the abstract which says like why they wrote this at all what the point of it is you can then skim it skim it by reading the headings the headings concatenated together how'd it make a

short-form version of what the paper is all about then read the introduction of course I mean the introduction is supposed to give you the background knowledge necessary to understand like why they're even doing this study and what kinds of things are the problems that they're trying to solve or what their aspirations from this research are then skip the whole thing and read the conclusion and see what did they find was it important do I care about this at all because the whole middle of the paper is what are we doing and how did we get to this conclusion what are the parameters of our study it's a waste of time to read it the conclusion is not going to

help you then when you're done with this process you get to make a decision would you like to throw it away and find anyone or would you like to pour over it and figure out exactly how they did it and what all the limitations on their findings are read all the graphs and see exactly what it is that they're trying to document here and this is the thing it's like I had to read about 12 papers before I didn't want to scratch my eyes out doing it it did take a bit of skill to understand how to get go through these and how to do those multipaths because I thought I had to read it like

a fantasy book I had to read it in order step by step and like stop and like actually pour over that one point until I understood it but doing this process you really get to understand where you want to go with it whether it's useful and exactly like how you can contribute to what that work does after any last questions I think we have one minute to do a critical reading of this paper and [Laughter] with that I think that we're just about out of time unless this includes the question no it's five to so we'll be around if you have any questions for us later on we're both on Twitter I can get it okay you get it we're both on Twitter

and of course always available in the community if there's some sudden thing after you've read our papers that you're wondering why we chose them or if you're wondering how to talk to an academic or if you have a difficult research problem that you're just not finding a solution to or if you're wondering like I have this tool and I'd really like to get into this area of research but I just can't get anyone to talk to me um we are your friends we are your neighbors come say hi I don't want to touch your ass that's right I'll get my robot to do it we'd like to thank Falcon and Brittany for an excellent talk

[Applause]