How to Lie with Statistics, Information Security Edition

Dan Castell here as the room host for this morning um first I'd like to start with with a thanking for our sponsors and especially Open DNS for this fantastic have and if you could just

quick all right we just warmed up the room for our first Speaker which is Tony Martin beggy um he has uh for a large Global retailer uh leading the firm's cyber crime program uh including risk analys uh Enterprise risk and security analysis 20 years of technical expertise and network operations cryptography and system administration he a Bachelor of Science in business economics from University of San Francisco and certifications such as uh Cy cism and C uh Tony martiny thank you

thank you thank you for the introduction I am so excited to be here and I really want to say thank you to open DNS for such a great space here was um yeah was anybody at the DNA Lounge last year for bides I was too the first thing I noticed when I came here is wow the floors aren't sticky so it's a it's a great um great space this is my fourth year at be bsides and I've listened to a ton of great talks over the years and yesterday I've made some great friends made some really good relationships here and let me tell you you all have the great quality of questioning everything I mean you people

question everything you question pentest results you question authority you question vendor claims and it was really that quality of of questioning that led to the Genesis of this presentation idea that I have so let me ask the room a question really quick have you ever uh read a risk assessment or a vendor security report of the latest Boogeyman security threat and just felt like something didn't add up you felt like just the numbers just didn't make sense to you has anybody felt that I know I I feel that all the time but why question it right of course these are the experts they wouldn't be trying to mislead us would they they might be they

might be we're going to dig into that today so before we get too deep just a quick introduction about myself um my name's Tony Martin VY I've been in the it and infosec world for about 20 years now at this point um the usual disclaimers apply I'm here representing myself and all opinions belong to me feel free to reach out to me over Twitter or email so how to lie with Statistics the 1954 Edition this was written by Daryl Huff Huff was the editor of Better Homes and Gardens in the 1940s and 50s and he had a lifelong passion for statistics ironically enough he wasn't a statistician he was just a um a DIY type

guy he wrote a lot of books and a lot of pamphlets on gardening and um home repair that type of thing but in his personal life he just he loved numbers and through this book in the 50s he introduced the general public to a lot of Concepts such as um correlation does not equal causation how surveys are used to manipulate people that type of thing and um I would imagine that everybody has seen the the show madman it was the admin of Madison Avenue that really Drew Huff's eye I mean this guy really had it out for him if you read his book read his book he's constantly hammering away at them he saw some of the claims they made um like

nine out of 10 doctors think that Newport cigarettes are good for your health that type of thing and he just really exposed a ton of that so what I've done here in this presentation is really taken the foundation that Huff built out 60 years ago and I've tried to update it for the information security field um it's a little bit modernized um and I'm I'm trying to um expose some definite manipulation in our field but I think that you'll agree with me that um the foundation that he laid really is the same so we're going to go over um some classic ways that numbers and statistics are used to bend the truth we're going to hit four main areas today

so number one we're going to take a look at how surveys are used or more often misused number two we're going to look at visuals such as pie charts line graphs and bar graphs and those are used to subtly steer the reader to a particular conclusion number three we're going to look at a um a term called the semi-attached figure and this is a phrase used to describe a statistical bait and switch it's really sneaky it's also my my favorite one and last we're going to take a look at the post Hawk fallacy and this is also known as Cor coration does not imply causation so let's jump right into surveys nine out of 10 households agree

that surveys are bad that's a that's a good statistic right which households what are we actually talking about here surveys are extremely common in the information security space and after all who doesn't love infographics I I like looking at them they're a very effective way to convey little or no information in in just a a pretty little graphic it's also the easiest way to steer somebody toward a particular conclusion so let's back up a moment and talk about surveys um a survey is a poll basically what you do is you ask a small group of people a particular question and then from there you take that small group and you extrapolate the results to apply to the general

population so let me give you an example um just last week I was reading a survey that um that said that 59% of cisos experienced a Cyber attack last year in which the attackers were more sophisticated than the defenses so the survey takers didn't go out and pull all cesos that wouldn't be possible or even desirable they pulled a small group of cesos and then they extrapolated that to the general population they said all cesos think this this is a a pretty common pretty common tactic um surveys are very common in Vendor sponsored security reports in fact I have seen many annual security reports that are based solely on surveys um I can only think of one off

the top of my head the Verizon data breach investigation report that is not survey based the rest of them are are survey based as far as I know and um all surveys including the ones um that are quoted in annual security reports s have some sort of built-in flaw bias or error that's just normal that's to be expected and that's okay as long as the survey takers are aware of them and they try to reduce those problems as much as they can as much as humanly possible there are many many annual security reports academic journals um risk analysis um articles blogs tweets everything that rely on survey data as Security Professionals we take those blogs articles journals annual security

reports we read them we learn from them we quote them in steering committee meetings we use the data for risk analysis um when when your manager comes up to you and he or she asks you um you know what's our exposure to this threat a lot of people go to annual security reports which are based on survey data so I can I I I can imagine that all of you can see how important it is to get this right we have to use good data and um stat surveys that are statistically sound we rely on this data for our firms to make good risk aware business decisions so let's take a look at what surveys are and what makes up a good

survey so surveys such as Gallop poles they seem really simple on the surface Gallop poles are um most often seen in presidential elections you know who were you most likely to vote for if the election was held today but the science behind surveys are actually rooted in math and statistics as a matter of fact there are a lot of top us universities that offer undergrad and graduate degree degrees in survey science so that's how scientific and mathematical surveys are so there's three main components of a statistically sound survey three components so the first one is population the survey taker has to know what they're studying how big is the group um and what is the group a lot of

people even skip that next you have your sample size this is the size of the group you're studying so remember you can't study the entire group you study a smaller portion of it a good survey taker will do all they can to make sure that small group is as representative as the big population is possible and next we have a really important concept called the confidence interval and this is your margin of error it's usually articulated in forms of um plus or minus five points that type of thing you might have seen that in fine print below a Gallup pole um basically how that works is the larger the sample size the lower your margin of error is so those are the

components of a statistically sound survey what happens when a survey is not statistically sound so you get you get [Laughter] biased so just a um a definition of bias when you're referring to a survey one would say that a statistic itself is biased when that statistic is systematically different from the population that's being studied so your sample doesn't match up with the general population there are many many forms of bias that are found in statistics and by extension those surveys that um that use that statistical data there's probably the one most common form of bias that I see in annual security reports and and other um statistics use in our field and that's a bias called selection

bias that happens when individuals are more likely to be selected to participate in your survey than um than than uh than the general population and that just builds a whole bunch of bias into your report so I'll give you an example of this um let's just say you want to do a study on the impact of Dos attacks at at uh at companies so you as the survey taker you go out and you buy an email list let's just say you um you you buy a million emails so you construct your email and you tell the people in the email that you must be an information security professional in order to fill out the survey you say you

have to you have to be a pro and second if you do fill out the survey I'll enter you in a drawing to win a $50 Amazon gift card so I I got this email last week I'm sure you guys see see these all the time so the survey comes back you compile the results and this is what you find out 89% of Information Security Professionals think that dos attacks are the biggest threat to their company so does this sound right is that that doesn't make sense does it it's not 89% of Information Security Professionals that think this there's selection bias here it's 89% of people that actually clicked on your spammy email and filled out your your

survey and more often than not they just wanted to get that $50 Amazon gift card so how many of those people went through and probably just clicked on anything they possibly could just to get to the end and we're also um have the problem of those survey takers self-identifying themselves as Information Security Professionals there's no way to check that so A good survey would do as much as they can to reduce all of those problems I just mentioned this is exactly what selection bias is so I don't actually want to call anybody out because I I do believe that most people act with the best of intentions but I really do think I would be remiss if I didn't show you all a

real world example of how bias occurs in our field so has anybody heard of us uh a report called the cost of a data breach um I won't mention the name of the research SE Arch institution but I will say it rhymes with Pokemon and the reason why this report bugs me is because it's the most quoted report of all time I mean I hear it everywhere I hear it at work at least once a week I see it as data for risk analysis which is dangerous I see it in blogs I see it on the news I see it in the newspaper I see it in tweet I see it everywhere I have the report here and um

on the very last page they disclose their limitations they disclose their bias so right off the bat they disclose non-response and sampling frame bias which is a form of um selection bias but there's also an interesting disclosure that I I want to read to you and um and I am quoting here statistical inferences margin of error and confidence interval cannot be applied to these data given that our sampling methods are not scientific our sampling methods are not scientific I can't use this I can't use this in a risk analysis I can't quote it I I think that I feel like I would lose credibility so we've gone over some problems with surveys and let's give

some quick tips of how to spot problems and surveys when when you're reading reports or um or articles so the biggest red flag for me is when a survey methodology isn't listed at all so I can't say that about the Pokemon Institute at least they they disclose their flaws a lot of surveys don't even do that that's a big red flag second no margin of error if the survey taker isn't disclosing what their confidence interval is their margin of error that's another big red flag that the survey itself probably isn't statistically sound so you might want to run away from that one and last vendor sponsored and let me say that this isn't always the

case there's some great great security reports that are issued by vendors um especially the aformentioned Verizon data breach investigation report that's just a solid report that I reference all the time but when you're looking at a vendor that's paying for something you might just want to take a look at motivations and just try to figure out are they trying to sell me something um that might skew the results a little bit just pay extra close attention to that if something doesn't seem right investigate a little bit further so let's move on to the next topic this is probably the most common way statistics are used to mislead people graphs and the reason why it's just so

easy to create a graph nowadays in Excel just select some columns insert a chart so this is through data visualization and data visualization is basically taking numbers that might not mean so much to the naked eye such as um you know a lot of data in an Excel table or a SQL database and you basically represent the data visually to help you tell the story there are many many ways to visualize data we have bar pies lines area radar scatter there's a ton of them but in this section I'm only going to focus on three main areas of data visualization primarily because they're the three most common that we see in a business setting um we're going to look

at first pie charts which is probably at the same time the most hated way to visualize data and also the most common way the most loved way next we'll look at line graphs and then lastly uh we'll look at bars so the problem with pies what's wrong with pies everybody uses them pie charts are extremely common they're probably the most common form of data visualization as I mentioned but the problem with them is that they distort reality they distort reality that you see the most important thing to remember about pie charts is that it represents 100% it's a whole piece um if you don't know what that 100% is if you don't know what makes up the hole use something

else use a bar so here's a data set that I created it's pretty simple it's um security incidents in 2014 it's a table view it's very straightforward we have five data points to visualize so let's throw this in Excel create a pie chart and see what we come up with so again pretty straight forward um it's the same data we just saw in the table but here it's articulated as percentages of a whole so the whole part the 100% that I'm talking about is all security incidents in 2014 and every incident type is a little slice so some information Security Professionals um myself included might want to see the data labels change the actual number of

incidents instead of percentages so right off the bat that's one way to um start masking some data there um I personally would use a bar graph if I was trying to articulate the actual number of incidents instead of a pie chart but I think that this type of chart is okay um so how could we manipulate this pi to tell a different story let's take a look here so I did some very subtle sneaky things here um I didn't change the underlined data at all the numbers are exactly the same underneath but let's just say that I want to underemphasize hacking incidents let's just say that I don't want to talk about hacking at all

and I'd rather shift the conversation over to Lost stolen laptops and lost stolen mobile devices so I did three things here first thing I did is I converted the 2D pi to a 3D Pi this right off the bat is the best way to manipulate data because as you can see um the slices in the front have larger 3D shading it's much bigger I moved um I moved the um the x axis I basically spun the pie I spun the purple slice the one I don't want to talk about all all the way to the back it doesn't have 3D shading so visually it just looks smaller and last this is really sneaky here and I'm wondering how many of you

caught this just on first glance I remove the data labels there's no percentages there's no number of incidents here so this right off the bat removes your frame of reference and removes your scale so you solely have to rely on your eyes to try to figure out how big each slice is and how much it makes of the whole um so let's take this a little bit further let's say that I I really really really yes

question 47 oh no problem so let's just say that I I really don't want to talk about hacking I mean let's just say that I'd rather have a root canal How can I manipul it further so if a little bit of line is good then a lot must be better so what I did here is I moved the Y AIS this basically rotated the pi like this so you're looking at the pie almost straight on the purple slice again is way in the back and I didn't change any of the underlying data underling data is exactly the same but this is what I mean by um distorting reality so this looks ridiculous right would everybody agree with me how many of you

has have seen this Ridiculousness in a in a business setting yeah a lot of hands a lot of hands don't do it what about the TAC s oh I'm I'm gonna do that you're one step ahead of me yes okay one last pie so let's just say that I changed my mind I want to talk about hacking now let's just say this is all I want to talk about and I want to want to underemphasize other things so remember from the previous slide that hacking only makes up 3% of total incidents but I want to overemphasize hacking so I did a couple things here I converted the 3D pi to a 3D EXP loed pie

I changed the Y AIS I basically spun it so I moved the purple slice back up front like it used to be um and I modified the 3D perspective this is one Mouse click in Excel that moves the 3D perspective and what I did is I just played with it I just kept clicking until visually the purple slice was as large as possible completely distorted reality I completely changed the story that we're talking about without modifying any underlying data at all so we've spent a lot of time talking about improper usage of pie charts um a couple quick tips on proper usage of pie charts if if you do want to use them um I think

one of the first questions you really have to ask yourself is are pie charts ever appropriate to use at all there's some really strong opinions on that um let's look at a couple of those

Nickelback um Walter hickey's a journalist he works for um Nate Silvers 538 great website if you're interested in survey statistics visualization check it out if you have a chance there's another opinion um that's a pretty strong one um that's Edward tufty he's a visualization Pioneer and statistician both him and Walter hickey think that pie chart should never be used I wouldn't necessarily personally go that far I think that there's very limited um circumstances to use a pie chart but if you're going to use them just try to keep a couple things in mind as I mentioned earlier pie charts represent the whole it represents 100% of something so if you don't know what that hole is don't use a pie chart use

something else number two pie charts represent a snapshot in time that's you know today or you know or yesterday it's a snapshot 2014 if your data doesn't support um being visualized as a snapshot in time use something else um number three your percentages have to add up to 100% I know that seems Seems stupid right so I have no idea what happened here my my guess is this was a survey um a hated survey and respondents were probably allowed to select more than one answer um I I can only guess I don't know but they really should have used a bar chart um number four you really want to limit the number of data sets that

you're comparing to three or four five maximum anything more than five really starts to give the reader a really hard time in trying to discern the the slices so you really want to use as few data sets as possible and lastly there is no professional reason to ever use a 3D piie chart in in my opinion or for that matter an exploded Pi um I just think that exploded pies are the worst it just distorts reality if nickel back if uh if Nickelback is um synonymous with pie charts then 3D exploded CH pies must be Millie vanila because it's just um it's a lie you're lying and it's just not

accurate so let's shift gears for a moment and take a look at line graphs I personally really like line graphs here um because it's um it's a little bit harder to manipulate the reader but it's still possible so let's take a look at how this happens so um just to set a baseline for everybody again here's a table um this is a table of malware infections it's a fictitious company um this is data from 2014 so taking that data on the next slide I created two line charts um so let's explain this data really quick what we're seeing here is that a fairly steady rise in malware infections from January to December um so you can see in

January that's 419 December it's 510 over the entire year there's a 21% rise in malware infections so just keep that in mind 21% let's take a look at our line chart our line graphs so the first graph on the left is a pretty standard line chart that represents the data well it represents a slope but steady rise in malware infection so if you took a look at this you would think yeah that you know that that percentage makes sense now the second graph on the right tells a completely different story I didn't change the data this is the exact same data in both of the charts but if you look at the graph on the right you would

think boy we have a huge problem here I mean some might think that we have a malware epidemic at our company that we have to do something now all I did was change the vertical axis scale that's

this on the left it starts at zero which is the lowest possible range because zero is always a possibility right and the upper scale is 600 that's a little bit higher than the highest observed range on the graph on the right is a totally different story I changed the lower scale from 0 to 4 100 this makes every change seem much more dramatic um so I think that um that's a a really big way to mislead people just through simply using line graphs so couple of things to keep in mind um really just always look at the scale that's being used a good line graph starts at zero if it doesn't start at zero ask yourself why if they're using a

different scale than the standard scale what are they trying to get me to interpret the data apps what are they trying to mislead me toward um it really Alters your perception of data when when the vertical scales change like that um and as I mentioned line graphs represent changes over time it's not a snapshot as opposed to pies so here's a really misleading line graph this is um a filled line this is from a presentation that Tim Cook did back in 2013 and this illustrates the sheer number of iPhones that his firm has sold so there's a couple problems with this so first of all what are we even looking at on on my line chart I I manipulated the

vertical scale in order to manipulate you guys there's no vertical scale here at all it's completely removed how many units has Apple sold is it Millions is it we we just don't have any

idea many number number

number right as far as this one is yeah okay showing the

[Music] I there there is a punch line I do disagree with you but um but I I'll get to that in just a moment

um okay there there's a couple different ways to visualize the data here um so what we're looking at here is a steep Blue Mountain of goodness that um there there sure are a ton of units that are being sold um I think that the problem I have with this graph is using the concept of cumulative scale uh cumulative sales I feel like it's misleading this leads the reader to believe that this is how many iPhones are out there and that's really not the case because iPhones break they're thrown away they're traded in um you know they they just uh fall into disrepair um I do think it's okay to tell people number of units sold

cumulative sales I think that that's okay but you have to give the reader context you can't just throw up a a a a line graph like this without any context how do you add context this is how I think the chart should have looked this is from a blogger at Quartz qz.com he overlaid a bar chart on top of the cumulative of line chart that really adds it adds a vertical scale it adds labels and it adds context to the concept of cumulative sales so I hope that that answers your your concern a little bit so bar graphs to what could possibly be wrong with bar graphs bar graphs don't have some of the same inherent errors that pie charts

have um but they still can manipulated bars are really good for comparing two items two or more items and they're really good when um when you're trying to show how something changes over time so just want to spend a quick moment and look at a a bar graph so here's two of them the underling data is identical but does anybody know what I did here it's the vertical scale again it's that pesky vertical scale on the right I changed the scale to start at 200 instead of zero and it makes the increase in Social Engineering attack seem much more drastic um I think that the one on the left is a little more visually reasonable so let's move on to

um yes is there any context where having a

nonzero there are um there's a a great book which I referen at the end from Edward tufty that shows all the different ways to statistically use um bars and line graphs um so next topic um the semi-attached figure and this has to be my personal favorite because it is it's so pervasive everywhere it's also really hard to spot unless you're specifically looking for it so a semi-attached figure is when a proof is given for a claim so you have your proof here and you have your claim here the reader looks at it and upon closer inspection these two items are not related to to each other at all um the reason why it's called semi-attached is because the proof seems

attached to the claim but it's um more often than not it has has nothing to do with it um marketing types and advertising professionals are absolute masters of the semi-attached figure um two common examples um I know that uh semi-attached figure is kind of a a hard hard concept to grasp without good example so I want to give you two quick ones that it's not necessarily security related but it it helps um understanding so does anybody remember the CZ commercial um that ends with now with redson that's the semi-attached figure what the redson right I mean it it sounds good it sounds like it will make stinky breath less stinky but what exactly does redson

have to do with cs I mean it's it's a chemical inside of the breath mint but in the advertising campaign they're not attaching the two together another example is when a marketing claim says 25% better 25% better than what you have a statistic 25% being offered as proof for a claim you know you should buy this great product but they're not attached to each other semi-attached figure so this marketing claim I don't know if anyone remembers this um I first saw the phrase unbreakable back in 2006 and I was driving up the 101 and I saw a huge billboard that just said unbreakable Oracle and I just I just kind of went like this I didn't know what to even think of this

um of this claim so for those of you that don't remember it this is referring to Oracle Linux which is um uh basically Red Hat Linux it's a it's a derivative this is classic bait and switch the vendor making a statement unbreakable and they're trying to lead the reader to associate that with L Oracle Linux the two are not attached to each other um of course of course Linux is not unbreakable there's no such thing nothing is um Oracle Linux has been subject to the same vulnerabilities that Red Hat Linux has been to uh you know over the years it just reminds me of so much of now with redson it's it's the same thing

unbreakable So when you buy buy software that's unbreakable what exactly are you buying are you buying something that can't be hacked something that can't experience downtime something that can be patched without having to uh experience a reboot are we referring to high availability here we don't know we don't know there's no connection there so Oracle almost immediately started backing away from this claim they still use this as a marketing piece but now they say that refers to a process not a product so I know yeah exactly so another example of the semi-attached figure this is a true story I was sitting in a meeting previous employers quite some time back and a vendor put this graph up on the

screen this is um the number of cyber security incidents reported to federal agencies from 2006 to 2012 now just looking at this this is a fine graph I don't have any problems with it the vendor was selling us next Generation firewall technology the people in the room fell silent as the vendor started their lecturing pitch he started in on us look at this graph from 2006 to today there's a 10-fold increase in cyber attacks we're at War we're at War here and you he was pointing to us in the room you are not adequately protecting protecting yourself in this cyber War the current current equipment you have cannot protect you against this type of unrelenting attacks and the salesman

went on and on and on and on and on does anyone see a semi-attached figure here that that's one of them yeah yeah what are we talking about right

right they're making a ton of presumptions about data that we don't even really know what's being represented here um so the vendor was really trying to lead us to believe that the sky was falling and who knows maybe it is I attended some talks yesterday that certainly would lead me to believe that there's Doom and Gloom on the horizon but this graph has nothing to do with that so just a a there's some great comments here but um a a couple of probing questions I was thinking when the vendor was giving us this pitch um so let's just assume that cyber attacks have increased 10-fold since 2006 let's just say that the graph is right why why

have they increased are there more computers in 2012 than there were in 2006 there might be are there more websites to attack that's also a possibility too you're not hearing the full story here what the the the number that I want to see is the ratio of attack Surface versus attack is detection of attacks better in 2012 than they were in 2006 are we seen more are we responding to more so there's just a ton of questions that you would ask when somebody throws this graph up and tries to give you a um a doom and gloom marketing Spiel um so there's a lot of things that somebody can do with data like this and

um in the next and final section here let's see what happens when somebody takes two data points something um you know one of the data points might be this and they claim that one causes the other but when you inspect it more closely you realize that there's there's no causation there at all and this is correlation does not imply causation so this is also known as the post Haw fallacy and the explanation is simple enough just because two data points correlate with each other it doesn't necessarily mean that one causes the other this is a logical fallacy called aerious correlation it was coined by Carl Pearson it's a great statistician in the um in the 1800s so

let's take a look at a couple of spirous correlations this is a great one um it's not security related but um I think

it's it's great I know so this is from the website of Tyler Vian he's analyzed hundreds of data sets and he's found really weird amazing correlations um this one here for those of you that can't read it this is the number of people who drowned by falling into a swimming pool that correlates with the number of films that Nicholas Cage has appeared in so um it's the website's really worth checking out if um if you're interested in logical fallacies and data so just a really quick sidebar go off topic for a moment does anyone find anything interesting or strange with this graph I I copied it directly from his website the vertical access the vertical access was changed

in order to make the two data sets correlate so I put the same data in Excel just to see what would happen if I started the vertical axis at zero and there's no correlation that the data doesn't flow together but it's just an example of how you can manipulate graphs to in order to to read the lead the reader to a certain conclusion um it's just another example of why you you you know you really need to question everything so let's switch gears and look at another example here [Music] um this is a chart um reported lost or stolen mobile devices correlates with the number of users that completed security awareness trading so I was in a room a while back

with a few Auditors and some other security people that I worked with and the lead auditor that um I was working with was very very concerned about a rash of BYOD phones that were um being reported lost or stolen um the auditor was absolutely POS positive he knew the reason why we've had you know we had so many stolen devices and he took the liberty of taking data and throwing it in a chart very similar to this I've admitted the company name of course to protect the guilty um the auditor looked very grimly at me he looked directly at me because I rolled out security awareness training at the company and he said you're causing

this

I didn't know what his angle was I thought how could I be causing this did I steal all of those phones he said you are teaching people that confidential information's on their phones you're basically telling people the value of sensitive information and they're stealing it it's true so his reasoning was that prior to the security awareness training people were blissfully ignorant about the Treasure Trove of information that was on their iPhones and now they knew that they had valuable data so they were stealing their own phones in order to exfiltrate data out of the company um but if you if you look at this combo chart if you look at this it certainly supports his hypothesis

doesn't it I mean it makes it makes total sense would think that there's a direct correlation between the roll out of security awareness training and loster Stone mobile devices does that make sense it's a spirous correlation so I am actually seeing more of this lately not less and I think that it's a disturbing Trend I think that the cause of this is the availability of Big Data um people have a ton of data at their fingertips and and it's very easy for the the the Casual reader with a little bit of excel to um draw casual relationships and immediately jump to conclusions has anybody heard the phrase Let the data do the talking yeah a lot

of people I run when I hear that post ha talk fallacies occur when people aren't careful enough when they reason data is not an end to itself data is used to construct a hypothesis and test it even just a casual investigation into the underlying data is enough to avoid committing these types of logical fallaces a majority of the time so just um in conclusion to wrap up this graph of course security awareness training didn't cause people to steal their own phones I performed just a um a really quick investigation I called some of the people that reported their lost stolen mobile devices and just asked them a couple questions about the circumstance es around the incident um to make a long

story short what I found out is there is a correlation between the two data points but it's exactly the opposite of what the lead auditor alleged people just simply weren't aware of the requirement to report their lost or stolen mobile devices they didn't know that they had to report it to our sock um so the security awareness training educated people to that fact they got wise and they just started reporting their um their mobile devices so this is actually really good numbers not bad um the uptick that we're seeing here is um caused by the training not really um an effect of it so just um to conclude here um a couple tips and resources Daryl Huff called the

manipulation of Statistics statistic cation it's um a combination of manipulation and statistics so there's some uh hopefully you all have additional Tools in your toolbox um I really do encourage all of you to always try to assume good intentions most people aren't set out to manipulate people um not Mo you know not all the time at least um I really don't believe that most people or even vendors are out there to deceive people they just don't understand some of the underlying concepts of Statistics logical fallacies um data manipulation um another tip I think that I try to try to employ all the time is looking at the source of the data whether it's a survey or a a graph or a

pie chart and just checking it out yourself if that's even possible if there isn't a source given um or if you check that source and there isn't a methodology behind that data collection don't trust it just be a little bit more suspicious of it finally don't believe everything you read just because it seems sciency or because it has a lot of data attached to it Steven coar has a great word for this he calls it truthiness a truthiness is a truth that is a truth because it just simply Feels Right without any regard to evidence facts or logic so um lastly here's some additional resources of course there's Daryl Huff's book um the Edward tufty

book is a great resource if you're interested in data visualization and how to measure everything by Douglas Hub this is um this has become my Bible when trying to measure things or um or do data collection um just really recommend that book so um thank you everybody for coming today are there any questions yes so the cation correlation U what you showed was awesome example I guess I feel like there are times when I want to I'm a professional like I I actually think these are related going and actually proving causation is take way more time than you know so I feel like I want you to tell me show not ID I think that um even just a a casual

investigation into something is enough to eliminate that typological fallacy a majority of the time even just a little bit of Investigation um something likeing would enough right something like this had no investigation whatsoever it was just the guy just pulled it out of his ass [Music] yeah

yeah

yeah yeah thank you for that yeah any yes over there if you go back to all theart examples couple slides I think the last show had the um the so it had like the the stolen pH the hacking inci and that security incidence was there a reason why you added that six field that security incidence in 2014 was it just to like illustrate a badart is there reason for that first entry right there oh that's a typo

yeah see don't trust pie charts thank you anything else yes

besides the question was besides post talk fallacy are there any other common bias or fallacies yes confirmation bias um wanted to cover that here it's basically um you are more likely to um believe an assertion or believe data if it confirms your pre-existing belief um I think that that's probably the biggest problem in common risk management methodology today is the confirmation bias

yes absolutely yeah we're about out of time thank you everybody for coming really appreciate it