Let That Think In: Thought Experiments And Their Application To Cyber Security

thanks very much Craig hi everyone uh so this has lit that thinking thought experiments in our application to cyber security so my name is Matt wixey I'm a threat researcher and writer at Sophos I've been in infosec for about 12 years so I work with John who did the keynote just now um although this particular talk is in a personal capacity so the views and opinions aren't necessarily those of my employer uh before doing threat research I worked in pen testing research and development and threat intelligence at PWC and before that worked in law enforcement I'm also a PhD student at UCL and usually if you've seen me talk before I usually talk about very Niche weird and wonderful things like light and sound attribution worms criminal marketplaces that kind of thing this is quite a different talk um and also as you can tell from the opening slide my ability to keep up with memes ended in about 2017 so there's going to be some really crappy memes uh in this presentation so what I'm going to try and do in this talk is explain what a thought experiment is and how it works try and apply it to cyber security and then give you a guide to creating and using thought experiments as well so let's start with a brief primer on thoughts experiments so it's actually quite hard to Define and there's no consensus in literature as to what a thought experiment is it's probably easier to say what they're not so here are some examples of things that people are mostly thought leaders have said our thought experiments mostly on LinkedIn so a hypothetical scenario based in reality what if we had a billion pounds to spend on security a tabletop role-playing exercise so you know the kind of thing you're the head of a sock and you've been hit by ransomware or think like an attacker that kind of thing or analogy so the one you hear a lot is leaving RDP exposed to the internet is like having your your front door open that kind of thing uh that's not to apply that these can't be helpful um but they're not thought experiments in the way that they're commonly understood in other disciplines I even read when I was kind of compiling this list I read about one thought leader in cyber who talked about doing Hands-On thought experiments whatever they are so why are these things not thought experiments well as I mentioned there's no consensus on definition um but there are some common elements to the proposed definitions so it's an imaginary scenario which for some reason is unrealizable in reality either ethically it would be unethical or illegal to actually do it or it's just completely impractical to do it and crucially they have a goal uh whether that's to destroy an existing Paradigm or support an emerging one it's not just exploring possibilities and sort of wondering what if and what might happen um and it's not something that's done mentally just because it's more convenient to do it mentally there's also an element of rigor in the construction of a thought experiment and in the answering of it so I thought experiments typically come in two parts you have the experiment itself the scenario and then you have the argument that follows it but it's all worth also mentioning this is a controversial Topic in science and philosophy so some people disagree thought experiments can actually be useful some say that empirical experiments are more valuable and provide more uh more use so a really brief overview of thought experiments they began probably with Socrates and Plato City uh allegory of the cave is probably the most famous One although lots of others um anyone familiar with the allegory of the cave Plato's allegory of the cave so it kind of um uh a number of later thought experiments built on that one so you had things like uh they cause demon brain in the vat if you've heard of that and then ultimately things like the Matrix and stuff as well uh they weren't really formalized until 1897 by a German scientist called Ernst muck who at the time said they were a necessary prerequisite to doing empirical experiments although that's no longer considered true um they've obviously been used extensively in philosophy and also in physics so in physics uh they uh there are some really famous examples there's Galileo Newton Schrodinger Einstein and you might be familiar with some of those more recently in Information Systems research and in politics and in law as well and then even more recently our Ai and privacy so probably starting in 1940 with the Turing test uh with regards to Ai and then a lot of others since then as well thank you so I mentioned empirical experiments and people sometimes think that these two things empirical experiments and thought experiments are in opposition to each other um they're not really or they shouldn't be um both of them are manipulating independent variables applying some sort of control and then measuring the outcome on a dependent variable in order to test some sort of hypothesis or to test assumptions or to make inferences about the world or all of them uh there are people that say that thought experiments are more of a rhetorical tool um so they're an argument um or a paradox or whatever but ultimately it's really just the data and then the interpretation on the discussion of that data is then used to support a refuted position um there is a little bit of a difference between philosophy and physics in that regard um so with philosophical thought experiments the philosophers do tend to have an argument in mind because philosophers love to argue whereas with physicists it's more of a model like a mental model that they're using to explore something so let's um talk about a few examples of thought experiments these are all in philosophy uh the first one is Mary's room anyone familiar with this one before I never heard of this one so I'll just kind of read out the thought experiment and then I'll talk about the argument that underpins it so Mary is a scientist and she has lived her entire life in a black and white room so everything in the room is black and white the furniture her clothes all the equipment in it whatever uh there are no windows um there's a door but she can't see through it uh the only contact she has with the outside world is with a monitor uh attached to a camera but it's a black and white monitor so Mary has never seen color but I mentioned she's a scientist and she specializes in neurophysiology so she knows everything there is to know about color she knows what happens in the brain when we see a red tomato or a blue sky she knows the specific wavelength combinations that stimulate the retina for specific colors and how this leads to us associating that stimulation with a word like red or blue so one day Mary is released from the room and sees color for the first time does she learn anything that's the thought experiment so uh quick show of hands who thinks that Mary does learn something when she goes out the room for the first time yeah it's quite intuitive thing isn't it so um this thought experiment is an argument against physicalism which is the argument that everything in the universe including Consciousness is physical it has some sort of physical form and it basically argues that there are non-physical things that can only be obtained through experience so when Mary is released from the room she uh experiences color for the first time as opposed to just knowing everything about it so it's a really long debate I'm not going to go into physicalism and sort of the ins and outs of this one um but this is very famous Lord experiment that um uh stimulated a lot of debate so we can see something interesting about the structure of thought experiments from Mary's room this isn't applicable to all but it is to many so we have kind of four steps we have an element of World building so Mary is a scientist in a black and white room we have some rules uh so Mary can't just walk out of the room there are no windows and so on we then have a challenge or a conflict so one day Mary is released and then finally we have a question does Mary learn anything uh let's talk about another one day cars demon which I mentioned earlier uh anyone familiar with this one no okay uh so um it's quite simple just imagine that there's an all-powerful demon which presents a complete illusion of the physical world so that the sky air Earth colors shapes sounds all external things are merely delusions which this demon has created so you don't actually have hands eyes Flesh Blood or senses but you believe you do because of the demon so this is this was descar's way of exploring uh what he called systematic doubt how do I know that anything in front of me is actually real This was later expanded on with the brain in a bath thought experiment in 1968 how do we know that we're not just brains floating in a vat somewhere and everything we see is a hallucination so again similar structure to Mary's room this one doesn't have a particular challenge or conflict but other than that it's the same and then the experience machine uh by nozick in 1974 so this one is um based on a premise that there's a machine which can give you any pleasurable experience you want so if you're plugged into it it's indistinguishable from reality but you'll stay there until you die and you can choose the pleasurable experience you have you'll be attached to an intravenous drip which provides you with nutrition do you have any reason not to plug into that machine so this was nozick's argument against Hedonism um he was basically saying that pleasure is not the only valuable thing in life that most people wouldn't choose to be plugged into this machine we want to do certain things not just have the experience of doing them we want to be a certain sort of person uh if we did plug into this machine we never see our real friends and family ever again and so on so Brown in 1991 suggested three types of thought experiment uh destructive thought experiments which destroy or challenge an existing Theory constructive thought experiments which support a theory and there are three subtypes of those mediative where you facilitate a conclusion from a theory conjectural where you just have an explanation but not necessarily a theory and then direct where you don't start with a theory but you end up with one and then platonic thought experiments as well which are both destructive and constructive uh some other formats um that are very common is something called a reductio ad absurdum argument this is basically where you establish a claim by showing that the opposite position leads to absurdity so uh you'll be familiar with this if you had an upbringing like I had if you do something bad and you tell your mum I did it because my mate did it and your mum says well if your friend jumped off a cliff would you do that as well that's a reductive ad absurdum argument um along the same lines like if you ever got told off of picking flowers or something and then someone said to you if everyone picked flowers there'd be no flowers left that's that's a reductor I had absurdum argument uh Sometimes they come in the form of paradoxes as well so uh there are lots of examples of these but like Achilles and a tortoise is a really famous One by Zeno uh Zeno's Arrow as well um and sometimes really strange forms as well so uh fiction has been argued as being a thought experiment or at least some examples of fiction so is the Matrix thought experiment is a good example of that and then particularly in cyber security there's a whole load of um fiction that could be really interesting inspiration for Thought experiments things like cyberpunk sci-fi Tech horror so things like Brave New World 1984 Black Mirror that kind of thing uh and then co-ons are um a really fascinating one anyone familiar with coans ever heard of the term before so they originate in uh Zen Buddhism um and they're like sort of riddles or puzzles that are given to Zen novices by Zen Masters um and sometimes the novices had to think about these for for years in in silent meditation um Carl Jung said that they aim for the complete destruction of the rational intellect so they run completely counter to Western dualism and rationalism and that kind of thing um they're intended to like provoke what in Buddhism is called the Great doubt so um and then eventually lead to Enlightenment but basically they disintegrate logic or they're intended to so there's lots of examples one you've probably heard before is what is the sound of one hand clapping that's a Cohen another one is does a dog have Buddha nature um so a student asks this to a Master a lot of Koons have an answer which involves a concept of moo mu which is really hard to Define but it basically means like nothingness or unasking the question or just rejecting the premise of the question entirely um I won't go on about cars for too long um but they're really interesting so about 2 000 of them survived um from when they were first built there have been no new ones written for centuries but they're online if you want to have a look I wouldn't binge them because your mind will melt but uh yeah um definitely have a look at a few um and one really interesting application of thought experiments is to address something called a coonian crisis named after Thomas um so Coon's Theory or a coonian crisis is um when you have a paradigm or a theory that explains a certain phenomenon over time that theory will result in more and more anomalies which at first are either ignored or attributed to measurement error or that sort of thing but at some point at some point you just can't ignore them anymore and at that point you have a coonian crisis and thought experiments have in various Fields destroyed that existing Paradigm completely kind of torn it down and then constructed a new one which both explains the pre-existing phenomena and also the anomalies so that's kind of one really interesting application of thought experiments some criticisms against thought experiments um so some people have argued that the conclusions from them are sometimes due to biases so if we take the example of the experience machine which I talked about um nozick said that most people would say no to being plugged into the experience machine but a lot of critics have argued that people wouldn't say no because they're rejecting Hedonism they're saying no because of status quo bias so they just prefer things the way they are um so some critics proposed a reversal test so if you knew that you were already plugged into the machine would you choose to disconnect from it for example Norton who's a very vocal critic of thought experiments has said that they are just picturesque arguments that they don't provide us with any new data that they simply reorganize what we already know and just make it explicit um and then there are epistemological problems associated with thought experiments as well potentially so the question that gets asked is can we genuinely learn new things about reality just by thinking and if so where does that knowledge come from which in itself is a kind of thought experiment I think the Counterpoint to it is a thought experiments aren't necessarily aimed at generating new knowledge but just to evaluate and justify theories um some of the scenarios in thought experiments may be uh too far removed from reality to actually answer the question or to focus on it so for example there's a thought experiment by a guy called Derek parfit um called amoebas which involves humans who split down the middle and that's all he says about it so you're trying to kind of conceptualize where they split at the waist or are they split in half and can they still function as humans and does it hurt and that kind of thing um so you just really kind of end up being distracted from the main point at hand they almost may also just have really bad logical flaws in them so lucretius is Javelin is a good example of that lucretius's Javelin is a reducto ad absurdum argument that the universe is infinite so he asked us to imagine a man at the edge of the universe throwing a javelin um and he says that the javelin can't go forward because the man's at the edge of the universe and it can't rebound because what is it rebounding off of that's absurd therefore the universe is infinite but he kind of doesn't take into account the fact that a surface can be finite and have no Edge so a sphere for example and then finally thought experiments can involve idealization so we say what we think an ideally rational calm person would say in response to a thought experiment and therefore may not get uh realistic answers okay that's the boring stuff out the way uh we'll talk about thought experiments and cyber security so why do we need them in cyber security um will they allow us to test assumptions from practical scenarios um as I mentioned they can address cunying crises which I think we have a number of in cyber security so an example would be humans are the weakest link right which um personally I think is quite an outdated way of thinking um and you could argue that our anomalies associated with that belief um and we also need more thinking in cyber security we often prioritize the accumulation of knowledge not unreasonably but um kind of thinking about issues in cyber security can also be really helpful and finally they can be an alternative to or supplement uh throw away hot takes which you've probably seen a lot of on social media um that's not to say that hot takes are bad they often contain really interesting fresh insights but too often they're just posted without much thought really quickly and just as quickly forgotten whereas thought experiments could put some rigor and some consideration around them as well so I'll talk about some thought experiments in in related fields starting with AI um the first one is the kitten and the Cockroach um this uh I've tried to look for I've read this once on the internet and I've never been able to find it again so if you know who did this let me know and I will credit them um but it's basically this so imagine I have a box on either side of me one on my left one on my right cardboard boxes and the one on the left I open it up and there's a kitten inside so it's about three months old really fluffy really friendly um who would like to hold the kitten quick show of hands okay cool and then uh on the I'll take the kitten back from you I put it back in the Box on the Box on my right I open that and I have a glass jar and inside the jar is a cockroach and the offer is that I will unscrew the jar and I'll tip the Cockroach into your hands so you can give it a cuddle uh who would like to hold the cockroach okay now imagine that I have the ability with the click of my fingers to transpose the brains of these two creatures so I can put the cockroach's brain inside the kitten and the kitten's brain inside the Cockroach so I snap my fingers and that's done I now open the box on my left again and I take out the kitten so the kitten is no longer thinking cat thoughts it's thinking cockroach thoughts it wants to find somewhere dark and warm to hide away it wants to lay eggs it wants to feed it thinks it's got six legs instead of four uh and if you held it it would probably try and wriggle up your sleeve who wants to hold the kitten now some strange people over there um so that's the thought experiment and the idea behind this is a thought experiment which you well it's anyone about Hazard a guess or what that thought experiment is about you want to shout out kind of yeah so it's basically about um why we shouldn't trust AI um it is saying that no matter how friendly and anthropomorphic and cute and cuddly uh some form of AI might appear so whether it's a chatbot whether it's like Sophie the robot or it's dressed up to look like you know Megan uh in the film the recent film something like that the brain that's inside that