BSidesMCR 2019: Predicting The Unpredictable By Thinking Like An Attacker - Nick Dunn

okay hi everyone that's Azure probably expecting as the title of the talk if that's not a talk you're expecting you could be in the wrong room then I'm done I just explained how I originally got into this I was originally a secure software developer who spent quite a bit of time working on online banking and eventually found out sir I could have ferb far more fun and more interesting time of my life by breaking things instead of building them and criticizing or the bubbles work and I moved into this security consultancy area of the Gansu work on penetration testing what a threat modeling can't review so a lot of this knowledge it's being coming out and

how is pop comes from nuts original work and design and developments and things I've learned over the years a threat modeling and general security consultants it's what we're going to be talking today is served a quick review of conventional threat modeling moving on to then when that sort of threat modeling isn't enough and more needs to be done and a bit of a process Georg checklist to recognize when you should move into that area and when to try something difference a number work through an example and this question is obviously at the end before I starts how many people have done threat modeling before and how many people are new to it okay so this first

part then it'll be a review for some of you and an introduction for the rest of you and before I start although I'm calling this the conventional way you're a standard method there is there's no accepted definite right or wrong way to do it there's a few different approaches but what we'll discuss first is the the more conventional way that it's usually done and you can all read a lot faster than I can talk so I'm not going to read out every line on every slide but essentially threat modeling so it's conceptually simple it's gaining some general idea of what can go wrong with the application and then moving on to look at how you can fix or further

verify that's and hopefully reviewing what you've done in improving for next time and the way we're going to talk about this the more conventional way of doing is is to sit with some subject matter experts and do what we call decomposing the application so generally start with their view of how the system works and there are architectural diagrams and logical diagrams before moving on to doing your own diagrams to model how data flows through the application which usually highlights where things can go horribly wrong that that will lead to you coming up with your threats which you will them ranked or quantify and our data flow diagrams are builds of these components again it's def very different from the

conventional sort of architecture diagram it's quite user friendly easier someone who understands and as you model the data flow you should end up with something some ranking of threats on these lines where we have the stride method that's conventionally used will be spoofing whether you can impersonate another user or another system tampering for anyone has done that a lot of penetration testing is one of the principal areas we find a lot of fun modifying the parameters attempting to cause unexpected behavior and by unexpected I do mean things going wrong repudiation generally we'd look at falsifying an audit trail making it look like something's happened at us and satins or concealing previous activities what a good example of that would be

making a cash withdrawal appear not to have happens information disclosure is fee of the excessive exposure or exposure of information you might not have wanted end-users to see of sometimes it can be details or sensitive data about a person or you could apply the same principle equally to technical error messages that reveal data about buckin systems denial service is going to be something that you I'm sure all familiar with and finally the e for elevation of privilege where an attacker would attempt to get higher levels than they should be permitted to have so carrying out activities normally for an administrator but without sir obviously without pre possessing the admin login so that's all a bit abstract

I appreciate her for anyone who hasn't done it before that sir probably needs an example to make things a bit more concrete so this is going to be far simpler than something you would do in real life was if you consider earth a completely fictitious online the service that sells things to people I'm sure you can imagine the sort of thing we might speak to the developers and come up with some high-level diagram like this that shows how data flows through the system and through different trust boundaries so for the user interacting any data that comes from them would be untrusted data moving from their across and to what should you do militarize zone within where the application sits would

that interrupt them further with stuff on the corporate network again data flowing both ways and you'd have different levels of trust they're applied to these this data that flows through and you would go on to use those stride techniques that I've just been talking about to build up this ranking of threats that could come for a system like that is all be because sort of things they come up with if you are threat modeling any web application although if you'd had two days to do it when you just handy then that you'd obviously have questions asked of yourself now there's a lot of advantages and disadvantages to this approaches it works very well for conventional web applications networks

applications and mobile applications because really that's what the process was originally designed for that's because the process has been designed for those sorts of applications it generally gives very good results in terms of coming up with a test plan or coming up with design issues as you can see I'm saying it works well most of the time and that implies obviously this times it doesn't work so well and for certain types of non-standard applications it can cause you to go through this process just assessing it as if it was a standard web application you you can introduce blind spots and realize if you do it that way you may have missed issues so if we take a look

at nonce some non-standard sort of compromises that have happened over the years don't know if anyone's familiar with the story of Barings Bank and Nick Leeson he at some point in his life got used his special privileges to set up something on the production system called error accounts eight eight eight eight which was used to hide away an error by one of his stuff that of course the bank to lose several million pounds from someone choosing the wrong option [Music] Potiphar betrayed I should explain bearings Bank were at this time one of the oldest merchant banks and the Wills and had the Queen as one of their clients Percy started to take riskier and riskier trades to truck what like a

gun boy trying to pay off his debts with one big sort of when he gradually built up bigger and bigger losses that we're all being moved to this accounts which was basically the results of giving too many permission the wrong sort of permissions to a single user and a sort of luck of auditing of on that's lack of correct auditing not sure trucky of the trades correctly and you could argue a minor bit of social engineering because he did could convince a couple of IT people that this was a training accounts upon the live system it does beg the question of why the people being trained would that account were gradually losing millions and millions over a period of a

couple of weeks but that results eventually caused the banks who announced losses of eight hundred and fifty million pounds in the 90s when that was a lot of money and it was the closing of the bank I mean more than exists now obviously it's still a lot in the case of Stuxnet much more recent I'm sure more of you have heard of that it appeared from deconstructing the exploits that had been deployed afterwards some what appeared to have used publicity photos of the president's walking around the nuclear facilities to identify which versions of Windows were being run in the facility which had then led to a situation but you could argue a lack of good desktop policy and locking

down had resulted in USB keys being used to transfer exploits so to those systems which were then used to attack centrifuges used for the uranium refinements on those systems at that time the word the whole idea of exploiting the hardware to refine the uranium was a bit of a new thing it was the first sort of exploit I think anyone had developed for these Siemens uranium refining devices in the case of the Estonian online services those immersive DDoS against government's systems and banking systems there where in their case it caused a major problem because the majority of the population overwhelming majority did all their interaction with government services online and almost the entirety of their

banking online but there was a further unforeseen consequence of that but some of the banks shut down their system thus in order to reconstruct things during the DOS attack which cut off their access to ATM machines which left the majority of the population of Estonia unable to use an eight no more rules which sure it's easy to be wise after the event but I think there's a few things there that wouldn't have been cool to in the conventional threat modeling diagram and on stride process with just looters and I'd started thinking about this over the last 18 months or so I first looked at been asked to develop a threat model for a mainframe and then been taken down this

whole routes inside my heads okay if you got fire extinguishers in here are they the sort that would damage the hardware or your login visitors what's the temperature control system is the temperature control system accessed online and I found myself getting down this route more and more as we started solid goods medical devices and machine learning systems where we find yourself looking at some quite specialized or unconventional bespoke systems that were then interacting over a perfectly normal HTTP channel or similar with regular conventional systems and I sorta wanted to think about how we could do this in threat modeling process there's a guy called Nicholas Nassim Taleb who wrote a book called the Black Swan

that's talks about the some of the other unexpected events that can have certain catastrophic or serious consequences that were unforeseen at the time and he gives a sort of simplistic example that a turkey that's being fed every day by humans might think hypothetically because it can't think really butter while these humans are amazing they feed me every day this is a really good process things are working out well and the turkey proceeds along those lines until Thanksgiving arrives and suddenly sees yeah there was an unforeseen event there and so I started to look at these sort of set of steps we could use to decide when we needed to use not just the normal strike enumeration what maybe

bolts on a few other sorts of things that we could live gotten I will be putting Jeff with some wiring for all of this later but in general I decided if the target utilizes particularly new concepts or non-traditional technologies when some of the stuff we've just spoken about medical devices machine learning or quantum computing thats feeding its data into a conventional system a system that uses specialized hardware like a mainframe or a quantum computer or you prefer nin uranium something that depends heavily on processes as well as technology then you've probably got a reasonable case for stepping out for doing destryed process in the normal way but then stepping outside and taking a look at what else could go wrong the

reason of I've got more detailed checklists that I'm going to put online together a clearer more detailed way to go through this book we'll use a high level version for now to keep things short and simple but the reason for using checklists pilots generally have a checklist before they take off because it's a nice simple way of getting competent skilled people to to make sure that nothing's missed and that a process runs smoothly for trivia fans the World Health Organization introduced surgery checklists surgeons in 2009 which reduced death rates from noir point eight percent to one point five percents um the other sort of scary part of that is as well as reducing death rates there

were other sorts of things that you'd like to avoid as a surgeon like swabs being left inside people the wrong limb being amputate seeds that was the general aim of the checklist so for death sorts of process I thoughts of things like you should be looking whether it has a physical attack surface we've already mentioned medical devices earlier and one of the reasons I had to step outside the process there was not only because these devices are plugged into a human at one ends on the data flow diagram which is quite sobering when you're assessing it's there's also the whole thing of is logging being stored on removable media could you deliberately mistreat someone and then

remove the logs can a laptop be plugged into it in order to services and is that going to cause issues or I'll have someone to operate the machine in a way that it normally wouldn't be we've spoken earlier about sir new experimental technology centers of quantum computing or machine learning how sure are you that the data you beam that's coming out of that system is correct how sure are you that the data going into that system is correct do you even know that it's been built and programmed properly and of course as we've said earlier there's always with these systems a channel running between the convention system but then the new sort of super Withey system that's going

to save the world what Choppers further the chances of compromise algorithm opacity was of course a thing I've thought of it into when we've been looking at machine learning systems do you even know that the result is corrects all those results correctable can they be correct seeds is there even any way of fixing system afterwards for banking like what I've talked about bribe blackmail here it's it's sometimes seen in the financial sectors an additional risk that someone could hire some temporary stuff and just bribe them to take photos or pass on details misbehave generally if it's often a lot cheaper than going through the firewall Lords doing a generally technical methods we've already spoken about

Stuxnet mainframes and quantum computing or other things that depend on some place trust heavily and some control of their environment with some external piece of hardware so that's an additional threat belugas the cost of running a quantum computer is enormous so if someone is able to compromise that yeah the environmental temperature controls that can cause huge cost overruns for someone they can cause the other things we've spoken about about incorrect data that you may not even be certain as in corrects both cops were another sort of thing that we'd consider it in an entity in terms of the mainframe issue there's been times in the past where people have backed up every night's or every week

and had plenty of backups ready to restore the system but then it's been burned down in the same fire that burned down the rest of the building they were destroyed the system that was protected [Music] and there's always as we've said with with the point of view of hiring malicious insider or bribing someone it's important Seleucus the path of least resistance how would I do this if I was an attacker there's a common thing and threat modeling and pen testing where you're told to think like an attacker which is possibly not the most helpful thing gift you've got to imagine how an attacker thinks I think it's far more helpful to look at that an

attackers objectives and go through some checklist of what data do I have what's the value to that of that being sold on what's the value to someone of that being corrupt seeds can the DOS attack benefits someone else as in online gambling where I used to work for a company that did online gambling and for them they would have a job attempts roughly once it's why someone's along with a ransom demands so if we look at this ways of putting this into practice we'll look good sir it songs us to model a machine learning system we can see fall sense of those criteria whether earth there are some processes and procedures taking place in the

machine learning system that effects the behavior of external systems holding important data could be medical data financial data that's us those critical things of algorithm opacity we really don't know whether or not it's generally working as it should do and so in that situation if we were attempting to look at something like that we would go through this process of following the stride model and then looking for whatever else we could do afterwards that falls outside the nonce the regular strides thickness I appreciate that may look a little bit intimidating at first sites but the real things to be aware of here as well what we've thought about about boundaries funds separation of different systems that are interacting

as part of a very critical process for a supervised learning machine learning system that be some training data that's fed in initially to help them make its decisions and produce some models you-you that could and then that model will be used to secure actions against live data which will then result in some control of some external system which could be for detection systems face recognition but generally the real point is that this external system has been controlled by separated on little-understood processes that frequently fall out sent to the remit of other companies or third parties or some weird process of data gathering that you may have had little control over or may not have thought

through completely so in the case of machine learning systems as I said there's a few things that would lie outside the typical stride method of doing it's which could expose features in inverted commas that an attacker might want to make use of things we typically look at from an adversarial point of view of ML is bias that could come from bad data about processing of good data and unintended features that could assist in attacker I appreciate some of that's a bit abstract if you imagine a company this has so many sort of feedback messages coming in that there's a machine learning system set to answer them and to send make an appropriate response to a person if someone found out what the

right words to use to cause that system to automatically give refunds that then that's the way this sort of thing could cause a financial loss from interfering with the model or taking advantage of features of the model the threats sort of specific to the training data it's huh sir how's the training dates have been set up correctly has it causing the model to behave as you would expect it feeling as it should do there's a a well-known example the US military is thought they devised a system that could recognize tanks from aerial photographs which worked sir with hundred percent accuracy on the training data but sir around two or three percent accuracy in real life it had been

trained entirely with to recognize tanks in photos that had no clouds and so it was just saying tank whenever it saw a photo with no clouds and no tank when it saw a photo with clouds so was it good thing to be aware of a few looking at this thing in terms of the production data thus there are known instances of people looking at anonymize data and [Music] being able to identify real people from its own personal data under a KC surfer like or the certainly the possibility of bad day to deliberately introduce to a training process not function correctly I realize I'm running over Beth so I'll just quickly run through that certain all of these

using the checklist can be applied to other sorts of systems and the mainframe there's the issuer on the communication channels the specialized technologies and as for quantum computing and medical devices similar as viruses issues basically there's different ways of extending strides I'll put the slides online so you can go to those links quick summary threat modeling is great

don't assume it's fallible just because it misses attacks just look for where you shouldn't reiterate to do things differently next time definitely do not focus on technical attacks to the other solution of deciding whether you choose online but the world to decide to allow many people to interfere with things in other ways always a dunya methods to what you're looking at any questions