GT - Magical Thinking... and how to thwart it - Mara Tam

let's get this started please welcome Mara Tom hello so somewhat unusually this is a kind of policy talk in ground truth which is you know a little bit of a mismatch in terms of the track but I I have made it a sort of minor personal crusade to develop talks that are not hated by by practitioners but that speak to policy issues so we're gonna give that a go this is a very condensed and somewhat updated version of the keynote that I gave at troopers in Heidelberg earlier this year and the topic is magical thinking and how to forfeit my name is mauro i am a you senior fellow at the Center for studies on terrorism

that is my current vanity title Washington is full of vanity titles and that happens to be mine this is about what DC looks like right now I I actually live in Northern Virginia I try not to spend too much time downtown because this is what do you see looks like art you know it however bad do you think it is I promise it's worse these are some of the government agencies that I work with I work as an adviser to executive agencies on policy issues relevant to information security really trying to get at really trying to chase the dream of evidence-based policymaking my my centers right now a subcontractor to the Office of the Director at DARPA

so we're digging around in the cypress I could have order and trying to figure out what what DARPA should be doing or can be doing I in in support of those goals so our very very rapid agenda today is basically that I see unknown cause a with Gaza why can't we have nice things the best hariom vocabulary which is you examples of metrical thinking in the wild and then porting which mostly we're going to be talking about translation layers and why we don't have them why we need them so we're gonna do a short literature review this talk is substantially inspired by a couple of pieces of political satire specifically the one on the left and the one on the

left is from studies and intelligence which is the CIA's internal journal for intelligence professionals and it's called the best theory of intelligence writing that was written in 1982 this is part of the Declassified collection of of articles from this journal that the CIA started posting I think back in the late 90s and the best area of intelligence writing was in turn inspired by the political bestiary published in 1979 and you can see they the political best Gerry treats such topics as viable alternatives impressive mandates and other fables and this funny little I guess anteater Hydra thing in the best theory that is multidisciplinary analysis so the the general idea is that there is an imagined universe behind every poor

policy or procurement decision it's not like law makers wake up in the morning and say hey I'm gonna make some horrible legislation today and it's not like and it's not like any of us wake up and say hey you know it would be great I'm gonna waste all of our IT budget on something useless so despite the fact that it's sometimes convenient or satisfying to believe that that these poor decisions are a result of stupidity or ignorance or malice most often they are not so I'm going to abuse my speaker privilege and actually give you a a curmudgeonly intro rent or things that I hate about normal policy talks very quickly first we had a lot of policy hobbyists in the

information security space and this works both ways you get people who have worked in in policy who decide that they're gonna get a piece of that cyber cutter and they decide that cyber policy is like every other policy and it really isn't and it doesn't work out very well on the flip side we sometimes get practitioners that dabble in policy like it's a weekend hobby governance is actually a profession it's hard people spend lifetimes learning how to do this and it's not unusual in my universe for you to have to work in the field for 10 to 15 years before you really start to get good at it it's security things like like Export Control Foreign Affairs type stuff when

you get into the realm of international law treaty formation you're really talking about a decade plus before you start to be really good another thing I really object to is perpetual 101 ism or or oversimplification it might feel nice to have talks that are accessible but we're really not having a conversation at conferences like this usually about policy we're not actually doing deep dives into specific issues that they're sort of awareness-raising which I would say that at this point the information security community is is adequately aware that policy is a thing that we need to worry about and then there's you know of course the standard doomsaying or hand-wringing i it's not terribly useful to my mind to spend all of our

time talking about how the sky is falling we get crisis fatigue in policy and I think we're starting to get some of that in in information security so that's that is Mike remotely grant so here are very quickly a few examples of magical thinking in the wild so we've got going dark crypto jihadis intrusion software which is a term of art from the boss at our arrangement which Sergei bravas has very eloquently written about the standard XP policy of the standard execution path and then my my last one one of my favorite ones is but the technology moves so fast so going dark how dark is dark and I am NOT a lawyer but I'm going to use the lawyer the

answer which is it depends and the slide is is one that I pulled from a presentation that I gave to some congressional staffers on on this topic and one of the questions that comes up a lot in for the congressional settings is ok well what do you mean by going dark and so what what I tried to do was I tried to give them a model of what is what is you know former director Comey's worst nightmare what are all the things that you could be doing in your digital life that would keep him awake at night and so you know running beta versions of software or using end-to-end encrypted messaging or you know basically having a

sound and robust communications ecosystem that is hardened against interception and you know that's somewhat worrying to law enforcement but in reality what we have is generally closer to this and you may recognize the Shadow Broker from Mass Effect I and I don't know how many people picked up on this when the shadow brokers announced themselves and became the the periodic presence in our news cycle that we know and love but the shadow brokers line is I know your every secret while you fumble in the dark and it's difficult for policymakers to hear that the reality is not safe and tight and locked down and in curved and impenetrable to interception that we we actually are sort of teetering closer to

this model another favorite and and a really good way to a really good way to to grind any conversation in Washington to a screeching halt is to talk about crypto jihadi I and this is this is an interesting one because the magical thinking here is that encryption somehow uniquely enables terrorism but there is a particular nexus between non-state actors and this technology which produces bad results and in in reality I'd like I honestly wish it was that simple because what you have here is a quote I I think this was after the I want to say that's after the battle on attacks so every time there is an attack we discovered that the perpetrators were

known to the authorities and what this shows is that our intelligence is actually pretty good but our ability to act on it is limited by sheer numbers and in democratic societies where we have rule of law and you need a warrant to to perform search or to surveil you need court orders and approval and things like that physical surveillance is actually really expensive and what European intelligence agencies have been saying for years is that it's not that they don't know it's that they lack the capacity to follow every good lead and they lack the capacity to actually track each and every each and every person that they reasonably consider to be a threat I and surveillance takes manpower

it takes cars it takes people working in shifts and it is you know the the problem here is not really the technology the problem here is resource constraints another example and when the is my favorite one I so this is from the the Belgian federal and they have been very open about the fact that quote when it comes to internet communications we generally have to enlist the help of our American friends managing information sharing between an intelligence service of one country and a police service of another can be challenging on several fronts including from a legal dimension and so what that tells us that what that tells me is that this is not so much

about we need we need help cracking encryption this is actually about information sharing and this is not something that we hear spoken about in conferences like this all that much but I kind of hope that that changes because information sharing is it's not just important it's critical and it's also very difficult right now and in this case what you have is a mismatch in the the legal regimes between an intelligence service at the federal level and then a foreign law enforcement agency and it turns out that sharing sharing to similar peers is a whole lot easier than sharing from an intelligence agency to a law enforcement agency when you start crossing like that it becomes

incredibly challenging more recent examples of things like this Australia has been making some policy moves to try to compel companies to be more responsive to requests for information that they may hold on their servers and that is largely driven not by any desire to break encryption but by the fact that when the Australian government requests information from Facebook it might take two years for that information to appear this moves at a snail's pace and that's even in that is in cases where where they may have a counterterrorism Nexus so that's even in critical cases it moves that slowly so information sharing hideously unsexy i its unpop people hate you for talking about it and trying to get them to do it but it's

it's something that we need to get much better at this is one of my favorite n o isms complexity kills briefly going to talk about submarines uh for reasons that I hope I'm clear I very quickly since I only have ten minutes left so ballistic missile submarines are he asleep on Plex they are expensive they can kill you and me and everyone they have a lot in common with complex network systems in a lot of respects so much like computing systems that discrete components of an SS BN can function perfectly and perfectly within their design specifications but when assembled they can still produce catastrophic failure as a result of either machine human or blended interactions so each piece can be

perfect but if the assemblage is flawed the entire system can be can either sink or be compromised so this submarine in particular contains a 13 by 17 meter pressurized water nuclear reactor it has I think 20 for trying to warheads on it and it operates at a depth of roughly 240 meters so not very much needs to go wrong for this to go very wrong they're designed to run for 30-plus years they they also can go through an engineering overhaul which increases their lifespan and that undertaking takes between 30 and 40 months in drydock combination and takes both of the permanent crews assigned to the boat that's pretty intense as far as life extension goes but it is an example

of what I'm going to talk about next which is the difference between operations and maintenance spending and development modernization and enhancement spending and this is one of the sort of magical thinking issues that affects both the practitioner and and the policy side so when you have hideously complex systems like a ballistic missile submarine or like say the State Department's internal networks you have to spend money to keep the thing running right you you have an operations and maintenance budget which maintains steady state it maintains serviceability but then you also theoretically have a development modernization and enhancement budget which can improve your capability or performance it's what you tap when there's any regulatory requirement that you have to comply with and this

includes capital expenditures is anybody in the room in charge of budget does anybody in the room get to spend money on IT stuff for an organization okay so how much easier is it to get money for blu-tack and duct tape than it is to get money for honest-to-god new stuff so this is a breakdown of federal government IT spending and this is one of those charts that kind of makes me want to cry because this is a breakdown between the total spending on non major investments and the total spending on major investments major investments in this case can major investments are things that require budget justification you you have to specify the money that you're spending and that's like only

half so there's not much like you know that that those orange parts of the bars is kind of like that's just money that we're throwing it's tough to keep it going and who knows who knows how again this is another chart that also should make you want to cry that blue segment is is your development modernization and enhancement portion of the budget and is your capital expenditure and compare that to operations and maintenance and honestly I think those ratios basically need to be inverted this is true at most large organizations and it's definitely true inside the federal government um I guess somewhat happy note this needs to be worse this is actually an improvement over previous years so a 30 to 40 month

overhaul is actually operations and maintenance spending not DME and this is an extreme example of how it is easier to get money to keep something limping along and tacked together than it is to actually get money to say develop a classic submarine that you know runs for six years from inception now we're very quickly if in the last five minutes going to talk about wang this is the network corollary to the SSB an example i don't know how many of you are aware of this story but back in the 1980s the state department became the single largest customer of Wang laboratories the 840 one point three million dollar contract that the State Department awarded away in 1990 saved the company

from bankruptcy that is about 1.5 billion dollars in today's money on on Wang and associated products and what you see here are two headlines the first one is State Department contract keeps Wang a boost and then five years later the State Department a snail in the age of email describing how in the mornings it's not so bad but from noon to 3:00 when the email traffic picks up it can take two to three hours and this is one reason why DME is so hard to get because institutions have a long memory for money spent like this where you spend 1.5 billion dollars over five years on brand-new systems and they are obsolete on delivery and you are

already having to come up with a Wang replacements a fine technology replacement schedule as you know before before you even accepted delivery of all of your machines also briefly I wanted to point out the single greatest acronym in the entire history of the federal government this describes the Wang one-way interface or Wowi force the transmission of unclassified information to classified networks and just in case this is difficult to see I also have the text here and that was actually from a Foreign Affairs manual from 1996 some of these protocols had to be active into the 90s into the late 90s because the wiring replacement program took so long and went so or over-budget and this is

the sort of result you've got you're limping complex system I on its way to the scrap heap and the story behind this submarine is particularly apt because this submarine actually sank on its way to being decommissioned anyway a quick last example of magical thinking is related to intrusion software this is the lovely magical thinking that if only we can squash the bugs one by one fast enough eventually we will be more secure and the magical thinking behind this is I have squashed one bug therefore I am one bug more one bug less insecure or I am one more secure we all know it doesn't work like that however still remains many people's favorite windmill to fill that challenges for adoption for

structural for structural fixes remain things like you know unwillingness to rewrite your entire code base and a memory safe language and the the fact that you know hard things are hard especially doing them well this is just a further illustration here this is from Sergei Golub honest hacking team and get gamma international business government malware goal Ivana got really excited when he was digging through hacking teams hacking teams awareness about all the efforts that he was going to find and guess what did not find any and now that we have seen large-scale ransomware campaigns spreading through SMB this should not be surprising to any of us now the thing that we are missing our translation Larry ders we cannot

throw more engineers at this problem and expect to solve it there is a new general ISM in this field that we have not yet defined and have not yet embraced the need for this is you know the obligatory dent den year slide every speaker writer practitioner in the field of cybersecurity who has wished its topic and us with it we're taken seriously has gotten their wish we in the security issue have never been more at the forefront policy and you ain't seen nothing yet however we are failing we are failing incredibly badly this is number of popular press citations by discipline in the American Academy of Arts and Sciences governance dual use technologies edited volume from 1.4% nuclear 22.4% in

bio two twenty four point five percent of all citations in dual use for cyber in this book came from popular press and I cannot impress upon you or how I really catastrophic that is in practice this is one of the reasons why we get bad policy and it's one of its one of the few things that I can actually say we can all do something about so it research is if the only citations in your research are wired articles and and stuff from from hacker news like we need to do better than that so we need to develop those translation layers to communicate fluently between technical and mission space we need to better the standard of documentation and technical

and policy research and we generally just need to be more accepting of the inter linkage between science and politics and with that I think I am down to like maybe four seconds so if there are any questions now would be a good time do we have any Commission's so what do you think behaviorally causes the kind of blind spots in people's thinking and it was something that I really like this because it's very somewhat about to be hit on the first novel track and so what do you think cognitively causes this dissident so to speak so why why do why are the imagined universe is so divergent from the reality Jeff I think there's a number of

reasons for that I think one of the biggest ones is just I actually Corti doctoral wrote something really intelligent about this about maybe 10 or 15 years ago and it was about how policy formation that impacts general-purpose computing is fundamentally different from policy formation that has impacted other types of Technology in the past and policymakers are used to not being technologists they're used to not being nuclear physicists they're used to not being you know biologists or or you know any other scientific practitioner but still being able to come out with relatively coherent and functional policy and that breaks on general-purpose computing for the simple reason that it is general-purpose and when you have a policy cadre that is

used to policy formation for mono purposed technologies and then they suddenly find themselves attempting to regulate or shape individual functions of a general purpose technology that's when this all starts to fall apart the wheels start to fall off when you have a ubiquitous general purpose technology where only only its applications are are any question one more question just to clarify did you say that did years slide say that there were no exploits done that need them no there were there were no exploits and there was actually a flash of a if that was found in there no there was not that was that was a sample uploaded to virustotal it was never observed in the

wild the speculation is that the flash oday that hacking team had was used basically for marketing purposes so that was never operational however it was a really good a really good sales tool we would like to thank Mark Tom for her presentation if you would like to continue the conversation [Applause]