BSidesAugusta 2017 - Track4: The Next Big Idea for Cyber Innovations by Ernest Wong

very happy to be here besides Augusta my clock is mostly about cyber innovations or innovations in general very quick pulling of the audience I actually like to just get to know my is where I can do these thoughts folks you are cyber professionals cyber security professionals any hackers in silence okay yeah okay I didn't see as many died ere this conversation to me that's the indicator of I'm more of a hacker conference or more well what about first time attendees first time first time these days yeah great yeah I know again we have KITT all right sänger tenants right now intense from Bullock out of Fort Worden it's great high school I mean high school elementary school

middle school explain any other yeah okay great that is hopefully this presentation doesn't blow your mind too much way cuz I I geared this presentation for second graders and I found that this presentation weren't better for second graders and I didn't originally and might know castles made verify this you're gonna be my kid because he actually moderated the first time I presented the very first version of this talk that was for 15 minutes that was a very hasty presentation personally I actually think this this room is the best room of all the tracks right this is the whisper and mother room right sneakers I know I know I know someone who's not seeing speakers so this is

finished first majors I tell all the content I teach at West Point I teach this image sharing I'm giving them all silent project so I not 5 ft 8 and what fun don't give us today's so meet you for free we get paid grade so we know how many TAS that's what I do that's now to work them all my side projects then I'll bring back up a little bit later but on the four movies I tell a woman like it has to watch sneakers the one war games as the other so you've already seen war games real genius and matrix you've seen the matrix as well so real genius and stingers that's yes your homework assignment I

said you're taking my class that's your homework sighs don't worry eight I just watch the movies you'll see why very quickly I work at this organization called the army cyber Institute we're located at West Point New York I told you I was an instructor there professor and IQ system you doing but this break a quick video from the scene yet they did a nice job talking about you'll need exit out of here and then close that one out so you can see the video yeah it's complete that here we go so seeing it there's this really nice video you call it propaganda if you want but they actually doesn't nice jumps throw and half explaining right

everyone ask me what

seeking just volume as well I think I might have just as my volume of mine as well okay yeah there we go here we go yeah there we go feel the quality control my equipment government equipments what it is I'm planing on the government all right let's see if this works third time what is the future of cyberattacks and what's the military doing about it we visited West Portland the Army's prestigious 200 year old military academy to watch the annual cyber defense exercise dubbed at CES a yearly competition among service academies to see which teams can protect their servers from NSA hackers people are starting to realize them we built all these things that were not relying

on but we sort of built it out in front of the idea of security and so things are kind of getting away from us and now I'm going to take some time to kind of wrap in an iconic figure in a man who is really complex cool there you go the West Point team spending months creating a server they could handle email file transfers and other services for thousands of simulated users for several days the group of two dozen cadets huddle together in a computer lab to guard their systems begin to exploits from the NSA so the kids are part of an extracurricular activity it's not a client's evolving new things and being able to protect something that's so

important in our everyday life and stuff virtual a scoreboard was projected in the front showing how well our meat was faring against competing teams with grades based on how well each link maintains its network naturally the our team activities on this year though it's really our efforts to predict post often ways the country goes back a concept called threat casting now that internet connections are just about everywhere with threat casters you're trying to put yourself into what the attack surface would look like that's my boss because we know the attack surface is going to be huge he went on to describe a series of potential cyber threats we may face in the future such as a personal care robot

being act and harming its owner and we already know about that right saw a couple of additional mass traffic disaster pulse think tank is also considering the potential of some pretty sci-fi concepts like augmented soldiers and autonomous drugs we've been doing is trying to figure out how the army will bring cyber insider effects to the bathroom I don't know if it's terrifying or comforting that high-level army officials are thinking about robot attacks and Bionic soldiers ensure cyber attacks may worsen but efforts like these should help the military prepare a little more for what's coming [Music] there's supposed to be a part 2 coming out in the next month or so so I get to go this link or just go to

Yahoo and do it do a see next search for a West Point sci-fi army should pop up but really what I'm talking about here today is just fall really thinking about innovation and really I think innovation you read the abstract in the book is disagree I think innovation is really driving really are us invent really talk about it how many folks actually study we actually know how to make better innovations that's what I'll talk about here I don't think of innovations as a monolithic entity I break it down I break it down into two four quadrants any business majors in the audience this is yes so your business manager you know you can break up any right you break it

off the world in four plugins and that's what I've done so let's hug it out baseline two criteria what is the level of complexity or a sophistication with the technology the other is a target market or what users want to use right whether it's new user or existing users and we're on this lower left-hand quadrant what I call that right low complexity but existing users I'm calling on sustaining types of innovation okay on the high side well think if I compress see I'm commissioner I think what about heist like new markets new new users I call that breakthrough that's why a little who's my Air Force yeah there we go therefore it's done that really well I'm

going to show you going full later but I'm actually focusing on this one went quadrant right here right when it's low technological complexity but very simple things but new users I'm calling that revolutionary innovation right I actually want to think of revolutions that's the analogy on these right revolution in warfare there are a couple hard business professors that coined this term and disruptive innovations that's why we need it why can't you like revolutionary disruptive innovations are things that are low tech but target new users okay so where's this mate right so I'm gonna help hopefully try to explain what each of these quadrants are in a more simplistic manner when I think of right this notion of breakthrough or really

client app innovation a lot of folks think that's the only types of innovation they're out there I can guarantee some folks in this room think that way especially like 9-3 try the folks outside the vendors they probably think innovation can only come from hi-tech I'm very expensive warranty okay but I'm gonna argue with that I think right unfortunately that's the portion apart right that's the way a lot of people think about us I'm gonna argue right and unfortunately right in the military right in the military longer leaders only think that when we breakthrough innovation right I took in Air Force does that really well the breakthrough stuff that's the only type of innovation that exists I'm going

to try economist with this with this presentation here I'm thinking revolutionary sustaining innovation they're just as powerful okay we're going to power our nation against cyber threats how am i thinking about this so game I'm not gonna introduce myself on my slides I'm new to myself as this Lanesville one I'm a military intelligence officer so I think generally in terms of threats that's what the twos do right the a to the N twos the s to s G when I think of sustaining innovation is the one character that comes immediately to my mind is Spock right star trek fans yeah good spot right he's very logical he's gonna give Captain Kirk the courses of action that had the highest probably

of success right even it means in women's that's right sacrificing himself right that was Star Trek to sacrifice himself evolutionary innovation any remembers fly boys to spy that's what I'm thinking of evolutionary right the black sky is trying to kill the white spy and every episode of rank one wins or another just by doing something that upwards the other spy what about breakthrough high-tech stuff my fix in the Air Force but right from the movies the character that comes immediately to my mind is James Bond James Bond gets all the high-tech gizmos filming for masters division other than my sense they have resources all Great Britain to give all these cool points and I don't

think Vickie pierce brosnan right I'm thinking Pierce Brosnan the he's controlling the end of you he's got the watch that cuts through whatever it is that means to save the day he's gone behind that points what about revolutionary innovations does that even exist revolutionary innovation we have any characters that come to mind for me now I know I'm dating myself here MacGyver right what is he now yes we're salami knife yeah the dummy was actually gonna use chewing right that's some paper with the white carries in his pocket but right he gots that pencil that graphite Linda uses right whoever that is disposable now I know that we have seen don't die deliver now they have their

happy make of it I actually have seen it I saw one half enough so I saw word hold on okay that's good I closed up by an oldie I'd say you think yeah that's not good so if you don't know the guy you don't want to want you to remake I know you've seen this right you've seen Jason Bourne so Jason Bourne is disruptive and he's revolutionary he's using whatever tools are disposal to save his own life Lord his friend's life now one of these characters really doesn't belong here right told you I'm an Emily officer military intelligence threat I'm huge Star Trek fan really spot doesn't belong here so sorry spot on that I send you away now the

character I'm thinking is mission impossible I'm not thinking I am not thinking is breakthrough he's going I think that was from rogue nation where he's playing a five-pound bag club playing spider-man of course it's cool but I'm thinking for instead of Spock for sustaining I am thinking mission impossible at the television series remembers Leonard Nimoy played Paris in Mission Impossible right he spoke like 10 different languages it was just a variation of English just smoking different dialects I remember watching when I was actually playing Japanese ambassador so he was portraying a Japanese accent it sounded very funny to me I'm Chinese but it's not me but I think was for the benefit of the audience understand well but

really the entire my Mission Possible forces their sustaining right you remember the Peter Phelps character he's picking the best human assets he can to calm these government's out of their secrets or doing bad things we might need that for North Korea but again going to me Tom Cruise man that sort of means I'm Tom Cruise any topics fans in here know Tom Cruise bands no one went no one saw Jason this presentation on on presentations human human persuasion yeah I saw his presentation had changed mine so the 15 minutes I had I had concluded right revolutionaries Tom Cruise getting hot God right so if you if your big fat Top Gun now I can't sing as well as Jason so

I have to use this so if you haven't seen Top Gun well I have a different take on it slightly different taping with Jason yeah well I'm sure glad sound is working in here buying all the other I need a pipe that sound over the next Billy's of man coming over the phone alright so if you can read this right amazing bring this down let me see if I can bring that down so you read this on March 3rd 1969 the US Navy established this program called Top Gun in the Vietnam War the US pilots both Air Force and a name which suffers some academies been about a thousand have disappeared that he's claimed to be shot down from

1965 9 16 that they suspended air operatives for we're Department of Defense said we can't sustain this right the Russians just a Sony so I mean we'll probably so many more aircraft we couldn't sustain that war need a fight with them so the department suspended all air operations no more bombings some more dock lights in the air what the Navy did right what the Navy did was they created Top Gun you creep this right down to San Diego Miramar you create this school so man fighting right your silence our planet is tactics of the Russian MIG 21s then B 1735 they were doing pretty good to read a 110 calories right for every one Polly

we should sent the spies we shut down three Russian pilots but that was not good enough the Russians just had so much more what's the Air Force always like right we need better weapon systems range systems on our f4 and we need to build bigger plus so the air force wing technology high-tech low-tech revolutionary interesting thing here there's a book I had I gave me a citation up to my cadets I can't remember what the name of it but you actually Google this right Google Top Gun real Top Gun not the movie amazing outcomes for the Air Force right when they got their new f4 phantoms upgraded their kill ratio state about 3:1 Navy the Navy pilots who went

through Top Gun and actually that spread from the wind was about 8 to 10 week course so again not every pilot went through it but do you know what the stats work for the Navy pilots 3 2 1 was original stat right this shut up 1969 17 Russian kills four wannabe maybe problem huge difference right that's right now you don't need to be a mathematician the lastest history so this right where the three two one two 1701 just by doing disruptive Meishan okay that's my take on Top Gun so the alternate talk a title of his talk was and what I learned about innovation from watching Top Gun but the b-sides folks didn't like that so I

stuck with my original title let's see again this is not from a mathematicians perspective right this whole notion of even carve out the world it's for plugins yeah I kind of like more numbers right I like numbers do I tell my crew what is this researching where are the numbers okay so a still not for mathematicians perspective but if I add one more quadrant right or one more access to my boss complexity and then target market target users and you know awesome potential how can this innovation drastically alter the environment well what are those innovations that alter the environment it's the ones in the top two rights revolutionary breakthrough so when the business books are telling you to take

risks to start to write through make mistakes they are visionary space they're taking risks in revolutionary and breakthrough innovations those are the innovations we want if you really want to risk because when I first set one so again it's still not a map to something but this is probably more represent writes Moore's game what it would look like the class answered not my change this third axis right I gave it a little hint to it already still not mathlete crap but I might change it now and call it all bit of success I gave it away right which ones have a high spot expects the ones that are blue and red ones are green the ones are greeting

right we have greater higher probabilities assessment and sustaining innovations it's because your your targeted users your target market already exists you have you can do market research you can get the data when you're doing things for new markets we have no idea what's gonna happen right it's called an experiment we've heard a little bit about that from some of the talks today if you remember Colonel Stanton this opening address at the very end he said be brave all right what else do you say be brave be be creative yeah be brave be creative that's what this talk is about I'm hoping to encourage creativity but in the right ways now hopefully if you understand what quadrants of innovation that you're

targeting either as individually or as a group as a company you might have creative success that's again still not so mathematicians perspective but this is probably what these quadrants would look like more representative skill wise to find that through so let's take a look now again I tell my cadets right prove to me in numbers I looked at not too far right using history this notion of the green revolt happy in 1910 1911 it's not too far away right five six years ago a little bit history lesson remember how many countries revolted

yeah WikiLeaks when they released our cables their State Department cables US State Department cables I said a lot more allies in this country aren't in this region our attention video pretty corrupt

[Music] quickly spread to 13 other countries so began small data size 14 countries and was working but this time interesting how many of these countries actually successfully revolted were in their king or their leader was deposed immediately I'm not saying years later immediately deposed well first one it also happened in Yemen they're still having problems era but that the King black team flew out to England somewhere I'm sorry Egypt Yemen and the fourth was Libyan right Gadhafi Gadhafi film writing was killed two years later so 4 out of 14 all right so you if you want to do the math there you can I've done the parsley right I don't like having general scheme math and

focus I do the math for them but I don't really consider Egypt a revolutionary success what happen in Egypt military takeover right so that's not really a revolution that's cool a military coup and being in the military myself I just know how dangerous militaries all right the military is too which is why from US perspective all the military folks in here know that's when they move us like every possibility right we sort of learn from the Romans read funny philosopher figure what what was bad with Roman armies Roman legions it was very dangerous when they closed the Roman legions I'll come back to the Rome bad things happen remember Julius Caesar it's your military coup so I don't tell feature so

here are the stats I was actually surprised by this how high this one I'm using this as an analogy for describing at least mathematically even witnesses what I'll probably success for rubbish 21% in this case I know we have still a question mark in Syria Syria is still debatable right unfortunately with my studies and Conor insurgency operations and insurgent forces the fact that Russian has stepped in against the US and the fact that Assad is fighting those forces aggressively it is not looking good for the insurgency's yeah don't believe the hype this has long wars favor in certain long wars you're not favoring the surgeons it only favors them if they have the willpower if they

run out of us support they run out logistics certainties failed very quickly and they can get rid of leadership very quickly that's not the number one guy definitely kill the number one guy and a number two guy my folks missed that yeah okay so getting back to this notion of okay so this makes it a little bit more realistic right maybe 20% maybe single bit of scream like for single digits but I'd like to think of it as majorly batting averages right okay that's the easier for me to remember major bebang averages the problem is Hollywood once again Hollywood gives us the impression that revolutions are easy watch all these movies in the problems we're so easy the king

Star Trek fan I'm a Star Wars fan okay so instead fathers tough business right it was I'll cut this out I'll cut this out yeah sounds like that yeah cuz really the one empire figures it out right the empire figures up these rebels are causing a problem right in this evolutionary space where Darth Vader come up with to take care of the problems with this breakthrough the Death Star the Death Star is a breakthrough innovation might taste a long time to bill it's cuz the role the rebels destroy too long right right one week time lots Rd lots of Matt hardened cream topping well we just needed to do with you the force but let's draw a

favor figure out a very bad news for revolutionaries right that's me what we don't see impression that thing that's why for me Star Wars Episode five was a better episode as well you like the baby on that okay let's see another way of looking at instead of Lee added a quadrant you find okay this came really I'm going to show the site the author's again but this notion of these green innovation for a sustaining evolutionary it's really it's not like there's no improvement over time it does improve over time but it's incremental small changes there's breakthrough innovations if you look at a graph here is that it jumps the curb that's like I'll break through the in parentheses jumping the

curb right you actually going beyond what the market needs that's innovation it's changing the environment has potential this successful what about revolutionary where did they start you don't have the same resources of folks in green I ask my lovely buttons in terms of military revolution you are starting off at the bottom okay and not only are you worse off like if you're working and your dress-up box on the revolutionary scale you are initially far worse that were pretty mission before were somewhere to mainstream earnings so how these things succeed right this notion of disruptive innovations came from to Harvard Business School professors Joseph power when Christensen it's amazing they say this this what they thought was

seemingly contradictory how can folks that create products there are far worse than what the market already has succeed I told you the percentage are small right major league bakeries well let's think about some of these xerox vs. Canon I'm thinking about the older folks search does anyone remember what the print divisions right your companies even like an academic institution what how big was those print divisions remember print divisions come up a copy room division could be the core of size of this room it's because Xerox kept focusing on the best customer we teach to our customers the IBM's the big companies that can afford and what the latest features what can I do hard canon

disrupt Xerox yeah they went for me what was that was their market right new market was no longer big companies their new market was home-based business was the inferior to us Xerox last I can write when they came out with that especially agents they laughing yeah but what was the what was the okay what was the value that made it smell appeal from the market but really what was the one

you can't grow a business in the lifetime o FF I mean couple hundred dollars still expensive we're in both we'd have to see today but compared to a Xerox printer holy smokes you can afford that same thing with right IBM Apple what Apple focus on cross that mention here that Apple 2e right green screens cheaper focus on out consumers IBM focused on mainframes right why would we have these great matrix right 10 times the speed computationally computationally they're on the Green Line superior to the Apple 2e but IBM left that out Apple appeal to the home based market and again it was but weird vibe is better performance in those pcs to where point intersects we passed that

point now right of course Mike going back when we follow another exam hobby how the tech business remembers right all these folks grew up thinking those are good vehicles right back my seventies couthon into is a great vehicle back descendants they're small right they don't have a lot acceleration but if the attributes for the American my space la space we like power right we like that people go fast very quickly what about this change this wasn't really a market consumer market external what was it I don't bargain 1979 right 1979 oil embargo so what became the the future that's one feature folks one new market one fuel efficiency right so was this external shock that causes

to market behavior shift ah more larvae example more larvae example this actually came from an audience member asking questions this is we are back right we can't be promised the government they give us blackberries it still have security there's the last but their market shares one percent now at the height when President Obama when he was doing called the CrackBerry Blackbird a grim had about 21 percent market share about 1 percent now they do have their own smartphone down but to compete with Google LG Samsung if you if you attended Jason's talk on presentation styles I've given you right I'm giving you the concrete examples here right there's no way to assume I'm gonna give you a

little more flexible you get it right when you think about other examples today it exists all over the place so if you know the question of rather revolutionary innovations exist think about sports does it revolutionary innovation taking place sports everyone read Michael loses Moneyball sabermetrics the notion that you can be or get these gecko the Yankees or Red Sox when in the Oakland A's but just by using statistics is readily available we've talked about a lot a lot of talks today right that's revolutionary if you leverage that and you go against their conventional grain which was right go against the grain of the traditional sports scouts do statistics TV shows aren't we in and dated with reality TV

game shows those are revolutionary innovation very cheap doesn't cost much to write script right for a reality TV there is no script there's just anything write and edit in less than five seconds that's what I learned also induces movies do we have four examples of revolutionaries very cheap movies that did very well there whoops that's working my mind yeah of course there's also examples like to find the breakthrough ones to write Transformers a lot of high-tech CGI news we're not using a 24 hour cycle I I report er right anyone can report on the news now and CNN will take it up that's why HLN is no longer region called headline news right it's like the

mystery channel now holy smokes yeah even CNN is being disrupted by disruptive games restaurants every actually almost every single restaurant that starts up unless you're a chain like franchisee you're if you're franchisees you're in this green space right any new restaurant really is a disruptive innovation revolution you think your food is better than everyone else's that's being start with do you have experience yet they've learned I know you're trying to hear new markets technology right it's Saturday this goes on and on forever so again disruptive revolutionary relations taking place every day every muck shark tank that is all whole revolution innovations right so if you understand the space right revolutionary innovations you cannot get money from

banks banks give money to folks in the screen space do you need venture capitalists angel investors bridge investors this red space what about blue space if you're looking for money who funds blue space right government's funny if you're smart chemical so you'll be doing it if you're a smart guy that's not really don't know I don't disparage breakthrough innovations I'll just try and promote more revolutionary innovations let's see now yeah so really this notion of I sort of gave it away right how can we best protect cyberspace if you're both you've been there right everyone this room I'm trying to do that even if you're a pen tester you're not you trying to make defense better so

what are the innovations that we need well yeah I focus this is pop on revolutionary innovations we do need them but we also need breakthrough innovations we also need to sustain an evolutionary but really we need all types of the nation's but what this framework tells you though is if you really want to take risks if you want to change the way they do want to go for the top - if you want to go for the low hanging fruit and the immediate victories go for the bottom - right so gives you a framework for how you can do things if you're an individual company right with less than 50 people you're probably gonna be in the

revolutionary space if you're a vendor that's outside getting government contracts yeah okay so it gives you a framework for what types of innovation are most appropriate and for your success and that's what we want right successful innovations here's the problem if you do not consider disruptive revolutionary innovations how I get this is coming from a military perspective don't officer words that we didn't win okay so what we didn't win sorry the Korean War what happened the Korean War yeah yes the North Koreans right the North Korea did not have more MacArthur push them all back North Koreans were decimated yeah no military China came in estimates were anywhere from 200,000 to 300,000 initially cost me Alvin

from October of fifteen overall million Chinese volunteers right but what wasn't the Chinese did well they asked at this point differently what MacArthur rely on this was again 1950 coming out of World War two what was his intelligence asset that he liked what just came into arrow forget airplanes had pictures what film right so it's what film so what film became the new breakthrough innovation for the US military yeah we have a nice reconnaissance aircraft what's wrong don't think you need darkness the Chinese military whether or not they knew or not when they were their soldiers to conduct movements huge maneuvers strictly at night by punishment of death right the one Chinese captain that had the pistol

they were moving at night gentlemen Carthage Jake to consistently underestimate military you can underestimate by three you underestimate my tank you're gonna lose a foot you're in time that's your what about being honor we talk about me and I'm earlier were that was successfully with the US Navy not so much human trail the fact that we could bomb the Ho Chi Minh Trail and just obliterate the belong at North Vietnamese supply lines what's the problem about notion if your supply lines aren't tied to this supply trail it's not tied to infrastructure this modern operation that you're learning from bullying this mind is right this notion of nolo train doesn't exist for dis my operations that's what happened

right we bombed heck out of ho chi minh killa usually didn't know a factory yes supply lines continue on now overtime right the Vietcong and North Vietnamese they develop the road networks because it was so successful again we stopped our bombing campaign from fall of 1969 as well when that happened yeah build road networks there's my stuff um Iraq Afghanistan what was the number one killer number one neighbor on the bathroom and where's my TV staff are in the very name of it is telling you this is a revolutionary destructive pranks it relaxes my even no weapon systems an improvised weapon okay so that's a system that it's causing they thought sure over time it improves right all the

time it went sustaining over time right if they started using copper right it was done in weapons that ammunition is found long battlefield cyber warfare my argument is that the first instance of most malware is disruptive and revolutionary anyone want to counter me on that there are some examples of breakthrough I'm thinking Stuxnet but again this form I can I can't confirm or deny that the US or Israel had any involvement when stuck now but if you watch the movie zero days right that's why as an academic right I'm silent zero days if you watch movie zero days it says the US and Israel had snow I can't confirm it viable let's see now for cyber office so this

notion of where cyber offense resides nice flight I'm thinking really it's not just it's not just rubbish right it does exist and sustained it doesn't history it also exists in breakthrough but a very small amount writing stuff right very especially stuff at one point I'm not like most X 10.9 Stuxnet 1.0 especially for why do there's not what I'm talking about now if you can trust this with cyber defense right cyber defense yeah we're doing a lot of breakthrough right the US government this love breakthrough contract a lot of that evolutionary exists sustained I don't think we're doing as much revolutionary cyber defense now I'm excluding folks from this room investing you guys are I think you're

doing revolutionary defense but my argument is if you look at these two graphs side-by-side again as a military intelligence officer yeah breakthrough innovations or offense we didn't need to exist right there's so many backdoors that you can find because you can stay in the revolutionary space if you're on the defensive side however if we give up on the revolutionary defensive measures we're giving the attacker too much time right let me talk to you characteristics of breakthrough nation that's it lots of lead time again both revolutionary and breakthrough in this scene tall big success so if you're leaving that gap you're exposing yourself to a lot of time where the adversary has the ability not only to

revolutionary but go down to singing and evolutionary as well and that is makes it tough for the defensive side that's my my argument this presentation and here really this is my second last slide this is where I get my text right Jason and that previous talk my presentations he said yeah stories yeah this is really my my well my final thoughts so I will go through this slowly just as Jason recommended okay most revolutions fail just remember them don't believe star wars right don't believe starboard most revolutions fail but that's where the greatest change had so if you're looking to make experiments if you're looking to do innovation that's a groundbreaking and causes lots of change look in the

revolutionary breakthrough space because it's tough because once right Darth Vader figures all you're doing that if they're smart they'll laugh at you they're smart they're gonna try and take that space with me they're gonna control market share rather than the innovation zone Green is again the sustaining and incremental nations now if you want to encourage revolutionary the best ways last night you want to be a brilliant doctor experimentation preferably cheap fast and many many iterations you want many iterations so we learned that in elementary school when you're doing experiments don't walk your first one you do it many times sample size n you want bigger bigger than 30 sample size the hand bigger than 30 so fast cheap though that's what's

got to be I wasn't I was a NASA fellow so I saw a lot of what NASA was trying to do trying to be disruptive it was just monitor DNA that's why I say from the Air Force Air Force will not give an Air Force contract for them to be disruptive they are so ingrained in that blue space for breakthrough on keeping their right they like satellites they like high-tech stuff I'm gonna keep you in there from the military who do I think is great for destructive but more specific Navy did some things right I tell you Top Gun seals right seals they have to rely on themselves small man teams club man teams that most shields

are great Special Forces for the army hopefully our cyber teams hopefully our cyber teams are thinking this way disruptive invasions experiment I also like Coast Guard I like the breeze like Marines because again they are at the Coast Guard and the Marines get the leftovers whoo Navy in the army so they have to make it work yeah wow that's insane wrong but they make it work well that's for me a revolutionary sign now attributes of when you do it right when you do experimentation right right be bold be creative - outside the box think hopefully what it does is it creates this notion of being more creative and being able to take risks okay that's

really the bottom line I think really what the colonel Stan was talking about earlier here yeah here here is my last time it really think the last thing I really wanted to get out here is like just like our mill turning our failures right in the Korean War Vietnam War even the failures in Iraq with gas tank failures right now father or not war alright sheep the South Army says we're not cyber war we're getting we're gonna attack that right everyone is taking our if we don't do experiments though what we're left with is we're left with the evolutionary sustaining innovations right we might have some breakthrough right the Air Force I know loves doing

that DARPA is doing a lot of that NASA is doing a lot of that yeah now if you don't do it revolutionary innovation the problem is we get we don't do these failures we don't get to learn from this space that's the problem not learning all right we keep saying we're learning organization we said that a lot through the wars of against and in Iraq right we say we're learning organization that's let's experiment because I don't want to be a military that's just a non zero defects mentality does anyone remember that terminology we said that sounds like a great thing right cool Powell used to say that he's pop by his mistakes I don't want to just be non

zero defects mentality army it's great that we're doing that right yeah we don't fly our folks right especially for you tenants yeah we don't fire flicks nearly from make mistakes right but I want to go beyond just the non Judy fix mentality I want us to be with an experimenting frame line now we want them especially right in the spread space I want us to be revolutionary experimenters low-cost cheap many experiments yeah let the Air Force they were doing experiments to in the breakthrough side but they have a better DNA with maybe success experiments those are expensive now if you fail in that area it's time again seems right at best maybe bad manners I think with that

here's a citation of all beer I think all the movies and all the quotes I used Barry built and the B size team everyone in green here thank you our Moustakas - we host our own conference again my very first iteration of this talk that Mike Bukowski duck Mycoskie moderated was at the last us cyclone that took place in October last year our very next one is six to seven that's right seven to eight November in Washington DC so if you're interested in attending we cap our tents at 500 but we still already think accept not open call for papers is done not just to go I got rejected from round two so again log great speakers should be at

this conference that's why I know how much effort goes into the production of commerce like this I wanna thank everyone in the green shirt everyone in the gray shirts and Phil and everyone blue shirts are putting on this great great coffees with that I conclude the presentation and welcome any question that you might have