The Numbers Game: Lies, Damn Lies, And Statistics

statistically there probably are lies in this presentation so be ready for that uh my name's James I do security stuff and I get irrationally angry about the misuse of numbers so youve probably heard this one 70% of cyber attacks Target small businesses it's been going around for years you know it's it's a well-known one marketers love it you've got people who will offer their products to solve this problem to small and medium businesses by the way only the actual main post there is one I mocked up the rest of them are taken off LinkedIn Twitter various places so they are genuine claims so cyber attacks happen every day but large corporations are not the only target More than 70% of

the attacks are targeted at small to midsize businesses 70% small and midsize businesses 70% according to the National cyber security Alliance we've got Source this time it's great small businesses small businesses small businesses 70% and More than 70% of compromised small businesses go out of business for anyone wondering by the way the statistic is about 90% of small businesses go out of business in the first year of operation so that happens a lot so what's missing from all of this well sources said lies right when you see these posts when you see this marketing the thing that's almost always missing is a source now when there is a source can you trust it most people will say oh there's a source

that's great I've done my due diligence I've checked there was one about the national cyber security Alliance saying 70% of small businesses the National Security Alliance have this statement an increase in the sharing and usage of this third party 2011 statistic 60% of businesses Clos within 6 months of a Cyber attack that is the only place they address anything even remotely like the claims made turns out it was in a report that they published and then disavowed because they'd taken it from a third party and decided it wasn't reliable was not generated from their research it wasn't 70% it was 60 and it was about businesses closing not where the attacks are aimed you might be able to tell I get

irritated about things I probably shouldn't who likes maths yeah we've got some saying yes so interaction time who knows how big a small to medium business can be up to 250 employees brilliant so large businesses are over 250 employees who knows how many small to medium businesses there are in the UK many a lot 5 and a half million who knows how many large businesses are in the UK not many less 2,000 it is more than 2,000 no 36,000 or 37 if you round up told you lies brilliant we've got those numbers now so we've got 5 and a half million small to medium businesses including micros I mean 72% of UK businesses don't have any employees possibly because

they're money laundering shell companies but that's a different issue I've got another talk on that sorry 8,000 large businesses 36,000 small to medium or medium businesses 99.86% of UK businesses remember the Precision there because that gets pointed out later as a way to spot false numbers uh so over 99% are small to medium 0.14% are large Enterprises what does this mean well how many attacks were there and attacks can mean anything fishing emails are included in attacks whether or not they succeed so how many attacks do you think there were in let's say 2024 to date Millions 7.8 million remember we are including fishing emails we're including WhatsApp messages we're including online graffiti everything counts as an attack these days usually

sophisticated if it makes it to the Press so 30% of that is 2.3 million so 2.3 million attacks against 8,000 companies 70% is 5.5 million so 5.5 million attacks against 5.5 million smmes so statistically if you're anme you're doing pretty well you've got one attack a year to worry about meanwhile if you're a large Enterprise you've got a different problem that's the statistic is worse than useless because it completely misleads on the scale of the problem now I'm not saying small and median businesses don't need to worry about security they do everyone does to a point I am saying this is a load of unethical marketers or in ignorant marketers who've picked up on a

statistic and gone that will help us sell our Silver Bullet to people who don't have the budget to actually understand what we're [Music] doing or knowledge there is a skills Gap it's not at the entry level so common statistical mistakes here's one lack of proportionality which yellow segment represents a bigger [Music] number it's the smaller one really common you will see charts all the time sometimes they'll have labels written really small I always love that they at least tried to do it right 70% of small to medium Enterprises or sorry 70% of tax will affect small to medium Enterprises that's exactly what that problem is if we'd had the source if they'd included the actual data that

would completely change the claim I'm sorry to the salese in the room but marketers lie learn maths uh correlation is not causation now for anyone who doesn't know the little symbol there I do love my formal logic symbols that is does not imply because it's not that correlation is not equal to causation that's a mistake people often make causation always involves correlation correlation does not always imply causation and much of the time it doesn't such as when we're talking about the number of Brewer Ries in the United States and solar power generated in Peru instantly remember that URL it's brilliant it has thousands of these and they've even added chat GPT to explain why each one is a valid

correlation so you've got you have spurious correlations and for added fun AI hallucinations to explain them this is a gift for misinformation anyone angry about misinformation this is really the site you want to on

hand who here did GCSE or a level statistics anyone remember the big bag of marbles with black and white marbles in yeah how much do you have to draw out to make guess a useful guess about What proportion of marbles are in there there is so unrepresentative samples one of the most popular things for companies to do in their marketing is say we did a customer survey and 80% of our customers are satisfied with what we did brilliant um what about everyone who's no longer a customer I I really want to know how many people aren't your customers anymore because if you've got 20% unhappy customers and they're just two tied in to leave I question how good

your product is or your service another one the office of national statistics they put a lot of work into representative surveys however saying that they will survey a few thousand businesses each year to get a representative sample of the entire UK business landscape there's a massive margin for error in that especially when you consider cyber security companies there are 55,000 estimated cyber Security Professionals in the UK now 5 a half million businesses 6 million if we're including the large ones and everything else 55,000 professionals if you go to the large Enterprise every single one of them has at least one cyber security professional at least someone who's labeled that whether they are or not is

a different question the smaller businesses don't which means that if a company which is say a cyber security consultany makes it into that sample and it's a small or micro one and this has happened I can voucher it they sent me a survey and offered fine if I didn't do it then you've just skewed that completely because you're now suddenly saying yeah one in a thousand of these small businesses we're surveying has I'd say reasonable cyber security knowledge now from my own experience I've only spoken to dozens it's not that they've not got concern it's not that they're not worried about it is that they have absolutely no way to address the problem so when you start throwing stats at them

like 70% of attacks are targeted at you yes you fear uncertainty doubt the way that we sell worse much much worse and where I think the problem gets really bad learned helplessness we are saying this is too big if you don't buy our 50,000 solution then you're doomed like all small businesses well yeah but I run a little Bakery out of my home kitchen I don't have 50,000 to spend on an IT solution I'm really worried about cyber security threats because everyone says this is the big threat but I can't do anything about it I can't for the big Solutions so all I can do is sit there with constant anxiety that I'm going to lose

my livelihood and my business now In fairness only 62% of the UK working population work in those small businesses so 38% are in the large Enterprises and don't have to worry about it they have a security department for that the disservice that we are doing with the way that we do marketing to those small businesses is to individuals to small family businesses small group businesses by saying You must invest this much you must do this much you must understand everything is horrendous this is part of why we have all of the problems around fraud in the UK largest crime 40% of crime is fraud the most common theft you will get is through fraud and over 70% of that is

online fraud in some way that doesn't include by phone that doesn't include by text message over 70% is online and then we go to people oh yeah but it was obviously fraud how did you fall for it you're really stupid you fell for it people who have been caught by fraud are scared to talk about it because of the condemnation because of the way that we play with Statistics and because of the way that we do our messaging around dealing with it yeah this isn't just about statistics by the way it's a whole messaging piece includes lies but we really need to fix this as an industry Cherry pecking data when you really can't get the

numbers from your customer survey to go the way you want just you know tighten it up a little say well we're picking our biggest customers or we're only picking our most satisfied customers because they responded soonest we set the cut off at 30 days originally now we're cutting it down to 15 and we're sending all of those a gift

basket absence of evidence if we do not have data to prove something it does not prove it doesn't exist again going back to fraud there's a huge issue so the estimate is that it's 40% of crime in the UK I think it's gone up to 50 now so there's one of those lies there's another estimate that it is massively unre under reported so about one in 10 cases are assumed to not be reported now how do we know that well they've done surveys and people have said yeah I was affected by fraud but didn't bother reporting it because action fraud were found unfit for purpose nearly a decade ago and they've not been fixed yet but people who don't want to feel oh

I was stupid oh I was caught because that's the message they've been sold aren't going to suddenly answer oh yes on that Anonymous survey so it's assumed to be massively under reported we have some evidence that says it is but anyone saying n out of 10 cases aren't reported we do not have the data to back up that claim we suspect it's a huge problem that's a useful thing to do to say nine out of 10 cases are not reported that is far too conclusive we do not have the research we do not have the data and that will apply to so many things that statistic 70% of small businesses go out of business within 6 months of a Cyber

attack well yeah 70% of small businesses go out of business within 6 months I mean it's hardly surprising going back to those nice formal logic symbols that one is therefore one of my favorites it looks really cool bit Predator but never ever trust marketing research the company sent you a useful white paper drawing all sorts of conclusions the response should be what are your sources what was your methodology what was your sample size if you can't tell me all of this then it's not just going in the bin I'm going to send it to James who will rip it apart post it on LinkedIn with a very angry rant about how you got everything wrong because he

gets unreasonably fixated on these things those are just a few of the statistical mistakes you will find there's another one that often comes up spotting fake numbers when you really can't be bothered to manipulate your research or cherry pick or just draw the wrong conclusions when you really can't be bothered you can just make up a number usefully humans are really really bad at random numbers and I do mean really bad now when you see people quoting statistics on LinkedIn or making up statistics on LinkedIn there's a few things that you can watch for we don't like zero psychologically it seems that we do not feel zero is a real number so if we're inventing a number out of nowhere you're

less likely to see a zero in it than you would otherwise being too precise 99.78%

nearly 100% over 90% over 95% when it's a real number we get incredibly casual about it when it's a fake number boy do we love our digits there was a certain post by someone who I will not name DNS some of you got that uh which said there were a certain number of vulnerable DNS servers in the world and it was something like 2,471 1972 that was what inspired this particular bit on the spotting fake numbers they went over precise anyone here know how many DNS servers there are in the world yeah neither do I no one does no one cares uh clusters we we repeat things and this is really weird when you go on to the next

one so once we've put a sequence of numbers in one of our massively over precise sequences to prove people it's real we're not good at varying it so you will find the same sequence of two or three numbers used multiple times in a madeup number really really often oddly it's quite rare in genuine natural numbers and by natural I don't mean we're talking e i pi and so on I'm talking about statistically generated numbers ones that come from real research it's quite rare to have repeated sequences like this but doubles triples even are actually quite common in genuinely researched numbers psychologically though we feel oh that's not random enough so I won't have the doubles together I'll space

them out a bit this by the way is great if anyone here does want to make up random numbers I'm telling you exactly what to do to make it convincing don't do it so doubles triples they are quite common in nature in actual research they are very rare in numbers that people have just invented wholesale my favorite benford's law this is about the first number of any number so about 30% of the time any number derived from research will be led with a one less than 5% it will be led with a nine it will never be led with a zero but that's not useful but when you look at these numbers again we tend to psychologically

when we're generating them lead them with something significant or important to us sevens nines fours threes fives various different ones the way that we come up with randomly convincing numbers is very different to the way that reality does now all of these are just ways to spot madeup numbers and it doesn't mean that if it doesn't match any of these rules someone didn't just invent it then you go back to the source and you start digging through and say where did it really come from how was your research actually done I've yet to see a marketing research report which includes their methodology ever does anyone here come across a vendor white paper which includes the actual method they

use to generate those conclusions genuinely interested has anyone 90% of the time 70% of Statistics are made up on the spot not including that one I heard that from someone else there you go you've got a source it's true they're never going to do it it's not favorable for them to do yeah they've got an agenda they've got an agenda and yes it makes sense that they have an agenda but again we're not selling chocolate or something like that where yes it has an impact on people's lives but all we're saying is oh we're preferred to this other brand or we're better we're more ethical none of them are ethical we are doing marketing we are selling to people

where this makes a real difference to people's lives and their companies majority of those people are small to medium despite the fact I'm willing to bet most people here work for large Enterprises anyone here work for or who here works for a company under 50

employees now fortunately we know the statistical likelihood of that so it should be about 99% of you who work for small to mediums under 50 are you about to be awkward at me have you thought about the statistical variation of the people in the room I'm I'm not going into the Dei piece and that I'm not qualified to talk about that and other people are yes we have a problem in this industry around it no I don't know the solution where was I thrown me off completely now right so we are in an industry that makes a real difference to people in that when we get it wrong we scare people a lot we also use technobabble a lot which

is kind of like saying a wizard did it and ran away so that's brilliant for the average person because I don't even understand why I should be scared but I'm terrified know we have world-ending disaster scenarios that get talked up on TV a lot yes I'm sorry about the crow strike thing I did my best to be reasonable and moderate around it and then other people took the quotes but we really need to do better not just at communicating ourselves but at calling out these unethical practices abusive statistics yes because that's a pet peeve but also outright lies nonsense threats and I completely understand why people don't want to call it out I have had a lawsuit threatened

against me for calling out that was expensive and I had Insurance again that's a whole other talk but for anyone who was there or wasn't get professional Indemnity and get uh personal injury Insurance one of those covers you in your profession against people trying to sue you for defamation the other one covers you in personal circumstances against people trying to sue you for defamation this will change the costs of a defamation lawsuit against you from about £3 to 5,000 to send a letter and make it go away to maybe about 500 you want this insurance if you're going to call people out because the people who do this stuff get litigious really quickly right I hate to do the any

questions thing and I do have a bit more but any questions how many out of 10 dentists prove this [Music] message employed by me or in general uh if we're talking about the ones employed by me then it's a statistically significant number which is currently undefined because that's a division by zero do you see AI feeding into these or false because AI is Reed for making statistics from other sources were you in Slovenia last year I did a talk there about the dangers of delusional AI um I see AI as a huge problem I've already seen Security Consultants generating reports for customers using Ai and not properly checking them now these are people who are meant to know what they're doing you

let marketers get their hands on that and it's going to get bad really fast so I see AI making it much much easier for these lies to come out and these lies to spread everyone know the lawsuit incident in the US one of my favorite examples lawyer goes to court and presents his brief presents his case to the judge and judge looks at it and says this precedent doesn't exist the lawyer goes oh I use chat GPT to generate it now I want to be completely fair to him at no point had anyone explained to him what an AI Hallucination is I teach cyber security on some boot camps I have had students come back to

me to argue against things that I know now anyone associated with those boot camps will know that one of my favorite sayings is there are no right answers only ones that are less wrong there are definitely wrong answers and they came from Ai and if you are using chat GPT to argue against someone with I hate to do this because it's appeal to Authority but with over 20 years experience then something's gone wrong their language models not information models so I see it making it much much worse because I've seen the lack of restraint among people who should know better and for people who are just told this will tell the truth and answer your problems

for you that's going to be horrendous do

no you theoretically yes well it's more complex because often people are doing this in social media posts and they're doing it in sort of their personal capacity even if they're putting it under a company if it's not paid advertising the advertising standards agency don't really get much to do with it there are some exceptions around Health claims and similar cybercity claims they on and on top of that if you're going to the advertising standards agency they need to prove that it's inaccurate that it's wrong we do not have the research to do that in most of these cases because we have very little data to go on So in theory you can and if enough people

complained they might do an investigation and someone might get a slap on the RIS but most likely because this is what I would do if I was completely unethical and worked in sales I don't but I would turn around and say oh well we had this survey n out of 10 dentists agree because I've got the data to back that up that means it's a true claim now if they're completely inventing it out of nowhere yes there is something that you could do but even then there's a reason that certain people DNS are still so active and prolific at spreading this misinformation or disinformation because it's deliberate so you talked about reports needing methodology and never including them

which is absolutely true but what other things should people say if there was a marketer in the room God forbid um what should we actually be looking for in those papers to lend themselves to legitimacy what should we be looking for sources if they're not including a source there's a reason why methodology and independent research that comes to the same conclusion now the problem is a lot of the time we will find one company does a research report and everyone else quotes it like the number of companies there saying 70% of small businesses well it's reliable because I saw it somewhere else and they saw it somewhere else and they saw it somewhere else I've got another talk

based on something similar which is more about the whole web of trust thing we tend to trust people that we like and that means we will trust the numbers that they give us and it's not that people spreading these numbers are necessarily bad with some exceptions it's that they're they're not thinking it through they're not considering it carefully so yeah the answer is it's really difficult similar you might heard women will only apply skills and that's been proven that that s is completely made up the report supposed to have said it was me extension didn't say that

um you see it that's everywhere that's yeah you know the cost of cyber crime worldwide is in the in the trillions what's the source for that number

anyone that's the profit

[Music] from last 15 years oh it's gone up a few times they you know add a few digits each year I remember 10 years ago it was in the billions is it CSO probably no they quote other people they don't do research themselves they just the inflation

that's so just just that somebody who works with marketers sales forun I can gu that even there is a source it's skew so you can go in and chck the source all make sense but it's cute like yeah a common example hospitals are the most dangerous public spes people shocking isn't it you give it to a marketer so don't trust market research even resources Statics in question statistics I mean importantly remember there's context there's propor to be considered now sadly as humans we're also really bad at big numbers so the moment it hits the millions it stops making sense even the thousands are a struggle a thousand and a million and a billion and a trillion they're roughly

the same thing right I know mathematically everyone here is going to say no that's ridiculous psychologically all of us if we just see that we're going to go oh yeah big number it's just big it's many can't remember who said many but yeah

[Music]

at some point you have [Music]

[Music] what are or

are make your best effort to make sure those numbers are reliable and include the caveats I mean I I ran a vulnerability scanning program for a national infrastructure company for about 2 years years and in the first two days after we finally got the tooling set up it was brilliant we went from over a quar million vulnerabilities detected to under 100,000 in a few days yeah that would have been a great number to report to Senior Management and told them yes give this guy a raise lacked context I could have just taken it and reported it I looked at it and it was because there were three vulnerabilities that we had marked down as false positive with good

reason so it's do your due diligence be transparent be ethical do your best and be aware you can fool yourself you're the easiest person to lie to not for me to lie to but for you to lighting uh this is my get in touch you can go there because it's obligatory now I did a whole book thing but mostly if you want to get in touch do any ask questions oh that was a phone not a hand right well in that case really really important takeaway here if someone is using numbers to make a point this includes me then ask questions about it they're using it for an agenda doesn't matter who they are think tanks

in America funded by certain companies Lobby against minimum wage because it will cause inflation and cost of living increases now those companies shockingly employ a huge number of people at minimum wage there's always an agenda there's always something behind it even if someone's agenda is benign be aware that that may skew things ask about motivations ask about methods ask about what their selling or what they're lobbying for someone's using numbers to try and make a point including me ask questions thank you