Joe Gray - Social Engineering Your Metrics: Using Data Science to Provide Value in Reporting

all right well yet without further ado Joe gray social engineering your metrics very cool I think under represented one I remember telling Joe and I was looking at an early version of this I've seen a lot of metric talks and a lot of them just don't get into enough detail enough examples and stuff like that so it's good to have one of these and metrics in general is just one of those things that doesn't sound very sexy to talk about but you get in a situation where you need them and and boy is it important so I'll leave it to you Joe take it away thank you very much Adrian so um here's me my office

so I'm Joe gray you all didn't take time out of your day to look at my ugly face so we're gonna go ahead and get into the slides so welcome to social engineering your metrics basically this is kind of a derivative of one of the chapters of my upcoming book when I was ready the defensive portion of the book talking about ways to integrate with Incident Response to find ways to have better awareness programs I was brainstorming and it came to me you know we should talk about some metrics and I didn't want to reinvent the wheel but at the same time I wanted to provide something that would kind of disrupt things and actually have it change for the good

so about me by day I'm a senior ascent specialist at complex I served in the Navy on submarines not gonna read everything on this of course generalist promotion for the book it's due out October 13th it is available for pre-order on Amazon that is a redirect link that will take you there I used to write for Forbes now I write for whomever I feel like it right now the competitive side of me from ascent and social engineering kind of where I got my basis from I'm the 2017 Derby social engineering capture-the-flag winner the runner-up of the 2019 hackfest in Quebec City social engineering capture the flag I can't stress how important the phrase Polly vous Anglais was that day and then I've

competed in quite a few ascents ETFs and missing persons CTFs notably getting fourth place at the derby condo since ETF last year fifth place in the most recent tres labs missing persons CTF and second place in the tres labs missing persons CTF at hackfest the same one I was competing in the social engineering stuff with I'm not a lawyer but here's the disclaimer so the thoughts and opinions are mine mine alone not those of any of my employers past present or future so to break into this and get in line with the objectives I want to talk about current general metrics and social engineering specific metrics the problem with that I'm going to introduce some

new metrics a little bit about analysis models and some potential for future integration so with that being said current InfoSec metrics that we have in general we have metrics about passwords things like these are the most common passwords these here's the common time to crack this list of passwords here's the complexity of such-and-such password on average X percentage of people within an organization have their email address password phone number or something in a data breach and then what getting into the vendor world we have response metrics so that comes into things like service level agreements between organizations and manage security providers but then it also gets into other other terms of responses such as internal things that could be used in

performance evaluations and what-have-you then we have what I call the vendors favorite metric and that's dwell time especially when we're talking about vendor selling malware protection or something along those lines so dwell time basically the time from compromise until they actually decide to do something and while it's not specifically a metric per se return-on-investment is a pretty common theme that is metric ish or metric adjacent if you will so going into social engineering especially like fishing we have opens clicks credentials stolen and then we might have averages or a ratio of opens clicks stolen credentials so how many people open the email how many people clicked on the email how many people input stuff on a form you can convert this over to

vishen by how many people kept you on a call for a certain amount of time how much time was the average phone call based on the predefined things you were trying to collect how many of those did you collect on average and then regarding ratios it could be the number of opens as opposed to the number of people of ascent to the number of opens in contrast to how many people clicked on it or how many people opened it and you stole creds from a lot of different ways you can go about that but I mean getting into the meat and potatoes of it we have to ask what's the problem with this and I'll just summarize this on a

single slide and that's these metrics really don't measure much basically they're just measuring something so that we can say we did better on this metric that has no application than we did the last time so we must be doing better that's one of those things that I like to call digital melatonin it's one of those things that helps execs sleep at night and I mean there's nothing wrong with some level of digital melatonin but at the same time we don't want to get too wrapped up on in that if I'm if I'm providing someone metrics I want to provide them something that is actionable something that they can work with I really I hate using the smart

acronym but specific measurable attainable realistic and timely that's that's something I would look for in a metric so the logic behind that statement that they really don't measure much let's face it emails are meant to be open and we have a lot of trading platforms and they really have anything for like link detonation you might be able to input your own links in there but it's not really going to test anything and the links are usually really strange and then you know a lot of non tech folks there they use the same password everywhere or they may use some variant of that for free for example a friend of mine reached out they had received one of those emails

claiming that it had turned their webcam on and observed them doing explicit stuff and they were gonna release the video unless they paid him $200 in Bitcoin the password they cited had not been used in about five years the wind of a cited in the email but it was literally the person's child's name and the year that they were born so that's a very common thing and it ended up being an eight character password so in some compliance schemes that's fully compliant even though it was all lowercase and just numbers two of the four typefaces the point with that being that what if someone required a nine character password well that person probably would have added an exclamation point to it or

something to that effect but let's let's dig into these bullets again so regarding emails and the the fact they're meant to be open a good fish should not set off any alarms with the subject line or the sender of the email the text of a well-constructed fish should raise some eyebrows but not immediately set off alarms and that's assuming you're using something like Outlook that has a preview pane and then the other thing is the the training platforms a lot of that stuff is predicated on the assumption that the organization's going to have a training program that actually employees that has something that actually engage the users and they actually care about oftentimes it's like oh gosh Security's in my inbox

again I've got to do this stupid training I'm gonna have to work extra today to finish my to find workload because I've got this deadline or something and I mean conventional wisdom says oh if it's HTTP it's safe if it has a green padlock it's safe that's not the case anymore and then of course we have human nature that deals with various types of curiosity no matter what you do no matter how you train people there's always going to be someone that's going to either fall for it because they're naive or they are gonna fall for it because they're curious or because they don't care so and if you want any evidence regarding things about

non tech people using the same password or variants you can't get that from have I been pwned but what you can find out from have I been poned would be a bunch of the data breaches that particular email address has been in there are other resources like D hashed or Scylla Scylla is not commercially online right now so you have to know somebody to get into that but basically you could query an email address in one of those platforms and there there are others out there there used to be one called the leak source but every time I knew when a new leak source pops up the feds shut it down in a relative short amount of time but

anyway you can query an email address and see that variation so what am I getting at well I mean what let's let's deconstruct how these things that we care about affect us so if we care about opens what we're telling users is that we are going to punish them for opening emails and if we've got someone who's naive or we've got someone who's problematic maybe not very productive or something to that effect they may use that to avoid work oh I'm sorry that email look like a fish so I didn't I didn't open it or if you have an email address that you ask your users to send emails to you may get flood with legitimate emails and in the

initial learning phases that time but when you when you've got an employee who's been with your organization for months or years who has been in the training program for weeks months or years that are still sending emails that they know to be authentic as phishing emails then obviously there's a problem there and opens yes I understand in some cases there are some types of malware that when you open it it will execute I understand that and I'll address that in a moment but Oh caring about how many people open an email that doesn't accurately measure risk it measures some level of risk but I've got more on that a little bit later so worrying about

clicks I will admit that clicks are more relevant than opens uh it does test the human but at the same time this is the point I was getting it from the previous slide building defense-in-depth should be built in such a way that it allows for users to make a mistake without causing a fatal error for your entire organization if if your security architecture is built in such a way that one single user clicking on one single fish that has one single piece of malware can take the entire organization down completely you need to rethink your security priorities that being said if you have the defense in depth and you have mitigating factors because at the

end of the day I like to reference wink or toes book time-based security and basically that the TLDR of that book is that we don't need to build security that's 100 percent absolute we don't want to build things that's going to protect us for everyone because as we saw in Alyssa's presentation before this it's a constant moving target it's a continuous game of cat and mouse so that being said we need to build our security defenses up in such a way that they will defend long enough for us to detect that something is wrong and take from steps and actions to mitigate the problem and the same should exist for clicking on on links there should be

anytime that you're doing things like this and you're running with malware or malware simulation or something to this effect there should be a test of your technical controls in addition to the people and honestly a lot of managers miss that point a lot of people get really passionate about pointing the finger at users calling them stupid punishing them with more training and don't get me wrong if a user falls victim yes they need to be retrained they need some sort of remediation but humiliation and hardcore punitive actions threatening their job or their livelihood that's not the way to go about it and that falls under the theory of bees with honey instead of vinegar so looking

at stolen credentials you know this is also this fails to face the reality that is predicated on not on having a password policy and what I mean by that is if you're requiring people to change their passwords depending on which compliance framework you're in or whatever your defined interval is based on your organization it could be 30 60 90 180 days could be 365 days you may not ever have to change the password policy but what I'm getting at with that is okay if my password was NIC 1999 that's only good for up to 89 days if all passwords are changed every 90 days but because password reuse is so prevalent then that causes issues

because even if I captured NIC 1999 your next password might be NIC 1999 exclamation point and then the next one might be with the @ symbol and and there are ways to go about this and that's why I really try to hammer home with that final bullet it's mitigated with things like multi-factor authentication and password managers of course someone to know the password Twitter password manager they have to have access to whatever method of alabang communication you are using for multi-factor authentication but the thing about it is if you're using a password manager it's a lot easier to advocate to the users to use a unique password for every single website because it's absolutely transparent to them they don't care I

mean so what if you enforce a password change every 30 days even if you wanted to go that far I think that's a little extreme but if you wanted to go that far so what it's literally been going into a forum clicking a button in their browser updating their database and going on about their day and then if they have their password manager on their phone it seems there too so it's not the end of the world so with that being said you can care a lot less about stolen credentials with the use of password managers assuming and this is based on a large assumption that users are creating and using unique passwords that are

complex for every single resource that they use so the TLDR of that is organizations do try to put the burden entirely on non-technical employees I don't expect Betty in HR to be a security guru just as she should not expect me to know the laws around working in human resources in the state of Tennessee I don't have an MBA because I'm no good at accounting so therefore Melissa in accounting cannot expect me to do her job nor should I expect her to understand everything about security but just as I have to adhere to the processes that we use to submit our expenses and things for repayment and I have to have that limited working knowledge of the accounting process

Melissa should also have that limited working knowledge of my discipline of security to understand hey you know this looks a little fishy I have some questions about this who do I reach out to and then of course some managers executives and professionals within is within the industry they fail to see that technology is Dalby all solution it can help mitigate the problem but it's not going to completely solve the problem as long as you don't care about availability the only way to solve the problem was to completely disconnect all power sources but then why are we doing this and with with companies they they fail to admit their own shortcomings they're like well you know we paid for this random

training that was made six years ago that still doesn't address all these other things well that you can't a Betty and HR can't make that happen that's going that's got to be a decision from the top and a lot of not saying all but a lot of organizations fail to pick up on that and get meaningful use of it I mean one of the one of the hardest things about being a consultant back when I was in consulting was basically telling companies that their baby was ugly and what I remember I got was you think your security posture is great well you didn't think about all of these things and here are some additional things to think about I mean I didn't

legitimately go and show people with their babies were ugly but proverbially speaking yeah so getting into the metrics that matter so we've got open to click ratio that's that's kind of one that we've looked at before as a traditional metric but it actually has some some teeth to it open to action that accident could be forwarding that could be inputting data that could be downloading a file with a macro in it that could be responding whatever the action is you can define that for each individual part of the engagement the third one is the one that is the most important in my opinion I don't care as a security practitioner I don't care if someone within my organization clicks an

email and the email has an exploit kit that's trying to drop ransomware on my network I care that the exploit gets there what I care most about is that the employee reports it to me so I have ample time to take action there's nothing else create a rule within the routing equipment that would vent that host from communicating with any other her so basically putting it in a rubber room disconnecting power removing it from the network restoring from a known good media all those things that we always hear about but if it's not reported to me I don't get that head start I have no opportunity to take action and then the same things for reporting so or

I'm sorry for clicking so click to action so how long from the time you click an email until you input something in the form from the time someone clicks an email how what is the time between their first click of an email and the time it is reported and then you can even go and say on a per user basis what is the average time between the time someone clicks on an email and reports it and then oh man I didn't mean to do that I totally input information on this form that I shouldn't have I need to report it that's a very valuable metric as well and then we duplicated that slide so the

click metrics again click to action we went through that the click to reporting at the end of the day I want to know hey you know I did something wrong ok cool you did the right thing you're gonna you're gonna have to do some extra training but all is good in the world trigger incident response do whatever we need to do so the here's the other one that's very important these are the distance measures and I got ahead of myself and kind of talked about them before but open to action that is the time and I'm working on a script right now they're will automate this it's not done I can't show you anything because

it's that it's not even close yet but basically measuring the difference in time using epoch time so measuring the time in seconds from the time a user opens an email until they perform an action with click and again we can do that on a per user basis or as an organization as a whole the time from the first click until their report something to that effect and then the same thing for we have it for the actions we have it for reporting and then the action to report what is the distance within that time you know how many people going back to the ratio measurement with action to reporting we had 12 people I'm sorry we

had 10 people click on it but only one reported so we would in that case we would say we have a 10 percent reporting ratio very simple math with that and I try to keep the math as simple as possible with us we're not going to be looking at equations don't worry we're not gonna have any anything from any of those movies but the the distance measurement basically you know we ultimately do want to avoid an incident altogether but we also want to be notified you know i we don't want to be notified by Brian Krebs the FBI or you know whatever else there's a reference to a winch for toes book again focus on

the defenses and then how much how much lead time do you have to enact incident responds that's that's really the absolute meat and potatoes of what I'm going for with this so from that lens I alluded to this before but what if you what if you are a consultant well I mean you can provide even more value I mean how do you think organizations like Symantec and Verizon come up with and and the anti fish workgroup come up with things like the is tr the DB IR and the fishing report of the year well that's because they have member organizations that collect metrics and submit them but if you if you work as a consultant and

you have several clients you can do this on a per engagement basis and a population basis the population basis is going to be all clients the per engagement can be curve facing the engagement or per client so nevertheless so the sliced metric that would be all fishing again engagements across all clients client only of course you're gonna have to anonymize some data you don't want to release anything about an organization but the ratios themselves you can say on aggregate Mott our customers tend to have a 42 percent reporting rate as it relates to clicks they have a 19 percent reporting rate for email opens and a 63 percent reporting rate for having input information into forms so that's

something you can you can provide and say okay based on this information your team is really good at reporting after they've done something wrong their marginal at reporting something after they've clicked on it and they've realized it's bad they're not doing so good at identifying these emails as they open them so with that one being the subject line there's not a whole lot you can do with that but you can there are ways you can go about things things like Proofpoint for example that will prepend or upend the word external or whatever you choose before or after the subject of an email that's always an option as well but anyway you have access to this data and

of course you would have to look at your your consulting contracts with clients and verify the verbage with that but if you're using an autumn eyes data then you can say hey here's some value we did 319 fishing engagements last year here are the statistics you can provide it to the client exclusively for client only across all of our fishing engagements here's what we have and then you want to include fishing here's those as well so and as we get into that we're gonna deal with a little bit of the statistic analysis and this is probably the part that's going to make some eyes glaze over I'm not gonna get too deep into it but basically we have the mean which is

the average that's basically the central tendency and the equation for that is there are some basically every value added together divided by the number of values we have the median that's going to be the center most value in your data set and then we have standard deviation and that's the amount of variance between data points within the data set regarding that a very high standard deviation is going to show a lot of variation between your data points lower standard deviation will indicate that they're grouped very similar and whenever you're assessing what is high and what is low that's going to be based on what values your data has so with that being said when you're dealing with

the statistic methods depending on how many data points you have like for example if you're a consultant and you're doing efficient engagement against an organization that only has 8 employees you may be able to do quantitative and qualitative so the qualitative and the quantitative quantitative is assigning a firm numeric value to each each level qualitative is like high medium low good average bad something to that and in some cases there are some things like for example Beijing if you're talking about activities on phone calls that could actually call for a mixed method and the reason I say that is mixed method is you could measure the overall sentiment of the phone calls a lot of them were positive couple were

neutral and one was negative okay that those quantity those qualitative properties would be qualitative but then you can still measure okay this is the number of times I called bidding or here's the number of times I called here's the amount of time I urged on each call here's the time I was on each call did you receive any reports and that does require the client security team to provide you some forms of information but with that that would be mixed method and that's just kind of getting into the data science part but that the challenges that rely on this if you want to do like if we wanted to apply like big data data science

basically the only thing I thought on this slide is regression which is similar to standard deviation it takes a very large data set and it compares each value to each other value and mass that's how organizations find out who the best person to market to this that's that's a very common use case for regression and regarding regression you have to have a very large data set for it when I was taking a data science course in college I one of my assignments required me to do regression and actually figure out who the best person to market to was I had 10,000 unique rows and I think it was a hundred and fifty columns I don't remember the

exact number but nevertheless and each of each thing had been quantified so that was a data set that was good enough to get something from if you only have eight people you're not going to be able to perform that it's just not possible another thing would be cluster analysis which is basically doing comparative regression that is something that is very similar to what we might call a B testing so getting into the practical application of this well how would how would we do this well like I said I'm working on a script that would actually assist with parts of this and we'd have to integrate it with your same or syslog or Windows logs or whatever

you might have to have some integration with the ticketing system if that's if that's your main method of reporting things and then of course measure distance and difference between things using Heacock time like I said I'm working on it but I've now got it finished the other thing you could do with this is actually input a list and say here are the internal non-routable IP addresses and the users associated with them so that we can measure based on the tracking pixel if you decide to use that in to fish or any other quantitative method to see who actually input information and who clicked the window just a side epical unethical sidebar associated with it would be I

wouldn't share the information that was input by the users just because I can't guarantee that it wouldn't be misused I would just give a yes/no they input they didn't so if we want to integrate this with data science I talked about cluster analysis we could do some level of a be testing with this which would be oh you potentially send two different types of phishing emails to to define groups to test how they respond to things so that you can adjust your training based on the differences between those two emails could be differences in the links differences and the verbage differences in the landing page the delivery it could be time of day so many variables

with that and then we can bring in other things like we're current neural networks and long short-term memory models which are even more advanced data science concepts so in conclusion you don't have to have a doctorate in data science to be able to meddle with this to be like really really really good to the point where you feel comfortable selling it yeah probably I'm not trying to sell anything with this I just like the idea of the metrics I guess in hindsight I might be trying to sell the book but that's not the ultimate end game that also being said though ultimately using these data points and metrics that allows us to actually provide value to our

organization and adjacent organizations whether it be our customers partners or whomever via the additional security so that's that's always a part of it as well and then also as with anything whether it be malware protection act sec whatever there's snake oil especially when you're dealing with sales and product teams and then just keep in mind and this is a very important concept statistics can be manipulated in various ways for ill-gained so depending on how you frame the information depending on how you normalize and stem the data set that will that can affect the way that it is perceived as well as the way you visualize it it can have that same impact so for example putting something

on a scale from zero to a hundred when the highest value is we'll say 14 that's gonna make things look very small and that's just part of the human nature but if we put it on a scale from zero to twenty or zero to fifty it's going to look more significant for that reason so keeping in mind the way that we visualize the data that's important as well but visualization is not a necessary component of that so we'll go ahead and start taking questions in the meantime I'm going to share some additional slides with you as well I've got some upcoming training events online they're all pure virtual there's short links to get to all of

them I've got coupon codes the 15% off coupon codes that also gets a 15% donation to the organization associated with it so trace labs women women of security or operation safe escape I also created a coupon code for 25% off for besides Knoxville so there's those coupon codes I've got the basic OSINT course coming up and the regular expressions course the basic ascent course is business oriented though regular expressions course deals with Python based regular expressions and some decent size data sets I've also got another missing persons assent course coming up as well as an opposite for host and privacy course same thing with all those coupon codes in cooperation with workshop Khan as part of the layer a conference I will be

giving a full-day person crash course there's the registration link for that there's a coupon code in the works it's just not happened yet there's the short link to it and I did not set the price on that so I don't have very much control over it you may or may not be aware but no search has donated to book vouchers to give away today they've also given us a coupon code for 20% all books there are the instructions to do so I've also posted the instructions in the discord server and it's in a couple of slides I am going to be sharing my slides so this will be on schedule if you want to attend a free virtual

conference it was going to be in Atlanta but it's not anymore it's virtual October 22nd 23rd there's hacker halted so information with that that being said we've got a few minutes so any questions you you let's see don't have any in the GoToWebinar let's check track one here

we were sharing some memes in the in track one the Orange County Choppers one where he's like what do you mean I didn't pass the fish the fishing test haha it's always a good one yeah it's like why is security saying why are you clicking this stuff you know and the dad says if you're so smart why don't you block those emails in the first place I mean that's my favorite I mean being honest I I submitted that meme to the Maury Povich Show and his test came back that that is not a lie nice so tips tips for thinking about useful metrics is a question that we've got so you've got to think about in regards to what it is

you're trying to achieve what it is you're trying to learn from it and how you're going to go about it so basically is the juice worth the squeeze is the reward worth the lift so if you're coming at it from a consulting cent since you need to collaborate with your client and say hey if you're trying to get distance metrics especially as it relates to reporting I need to know the exact time that someone reported this because you're going to know the exact time you sent the email because obviously you're the one doing the fishing so that you can convert both of those times to epoch time and measure it in seconds and then if you want to

compare it and you can move that to minutes and hours that's a matter of dividing by 60 or 3600 if you want to go directly to hours regarding two ratios same thing you will be able to tell partially a ratio of how many people clicked it versus how many people input information if that's the route you went the problem with that is unless you host the fishing server internally you were only going to see the organization's not address network address translation so that's going to be their public address so that's another opportunity that you're going to have to collaborate with the internal security team to be able to get access to those logs and parse it

and break it down that way does that answer your question I was on the question was on discord so be a good place to jump in later on after after we wrap up here um need yeah it sounds like a you know common problem is you know like people are expected to record metrics but just so many of them are just worthless you know like they don't really mean much of anything right and I mean I look at this from two perspectives one I want to collect all the metrics because there may be a way that we can eventually figure out how to I'm just gonna use the example of cure world hunger with this right you might

be able to figure that part out but at the same time even if we can't solve that problem we may be able to find some trends that we can build better defenses with but at the same time I don't want to spend all of the time creating these metrics doing the analysis doing all this and that if nothing's ever going to change and I kind of think that's where a lot of the the world is today because people we we have a negative unemployment rate well prior to covet and we have people who are significantly overworked that don't have the time to implement a lot of things and that plays into it as well and that leads to

general frustration about anything though and yet we get feedback that you did answer that first question perfect did you want to talk about no starch yeah I put the slide up here I'll bring it back in there's the new search stuff so again 20% off their books there's the redemption instructions it's a valid until May 15th show no starch lots of love because full disclosure they are my publisher

so the secure code warrior championship is done and thank you very much Joe for that that was a great presentation again lots of good conversation in the in the channels