BSides Edmonton 2019 - Matthew Maglieri, Chief Security Officer, Ruby Life

nothing's working today all right good morning Kinnear there we go okay good morning first off I'd like to just thank all the organizers for putting this on and if I'd invited me to be here with you today it's truly an honor to come here and and to speak with you all and I'm really looking forward to the conversation so this is Ashley Madison cybersecurity in a world of discretion or at least that's the title that the PR team let me put on here but in the hotel last night I decided to change it to the non-working Windows laptop No all right there goes the big reveal

[Music] okay well there's the new title anyway so we're stuck on that slide everybody has a plan until they get punched in the mouth it's a Mike Tyson quote and that appropriately describes what happened to us so my name is Matthew Michael area and the chief security officer at Ruby we are the parent company to Ashley Madison amongst several other dating websites I've been with Ruby for about three years now prior to that I served in the lead role helping build mandaeans Canadian practice primarily focused on offensive red team operations as well as strategic Sauk development and design engagements been with Ruby for about three years we're based in Toronto we're Canadian and I'm here with you today to

share what I believe to be is a rarely discussed perspective that of an organization who has gone through the worst case scenario a headline-grabbing breach and the share of the lessons learned from that incident with you so we can begin to tackle the fundamental question of how do we achieve breach prevention or perhaps more appropriately is breach prevention even possible so I've got my coordinates up here please feel free to reach out and connect or come up and find me after I'd love to connect with as many of you as possible so please don't be shy

all right sorry about this guys just give me a moment there we go all right as with all good stories we need to start at the beginning and the beginning for the context of our discussion today starts with the 2015 hack of Ashley Madison this incident saw over 30 gigabytes of corporate and customer data stolen from the company and leaked out to the public Internet the impact and follow to both the business and consumers was profound there were multiple class-action lawsuits there were three different regulatory enforcement actions one from the office of the Privacy Commissioner here in Canada another from the FTC in the United States and a third from the Australian Information Commissioner there was a tremendous loss of consumer

trust there was extended and continued negative media exposure and most importantly and most regrettably there was tremendous impact to the personal lives of some of our users and so the company knew that if it was going to survive the incident and indeed survive as a business that we would need to become leaders in the privacy and information security space because of who we are were because of the nature of the business it would not be sufficient to do anything less and so my general counsel and privacy officer and myself who she joined the same the company around the same time as me we set ourselves a pretty simple and direct mandate and that was to build leading

privacy and information security programs and so we got to work in the immediate aftermath after the incident response engagement ran down we worked with Deloitte's Canadian team to perform an ongoing and continuous compromised assessment of the environment to identify any potential continuing evidence of attacker activity we also deployed a full suite of Beck latest-generation Network and endpoint security controls and then fed all those into a new hybrid security operation center staff both with internal ruby resources and Deloitte Cyber Intelligence Center in in Montreal we also completed a full security program redesign completely overhauling the governance program from the ground up next under under PCI it's a little known fact that if you are subject to PCI

compliance and you do suffer a data breach you are automatically considered a level 1 merchant from that point forward regardless of your transaction volume which would typically determine your level of compliance under the DSS standard so because of our breach we needed to implement and maintain level 1 PCI compliance which we've done so for a number of years now next after some of this foundational work was done we brought in our friends at fire i''m and ian to complete a series of proactive assessments red team engagements penetration tasks web application assessments to assess the overall effectiveness of this foundational work and with that it began it gave us a foundation that we needed to begin to tackle some of the concerns

of the regulators so first off we needed to satisfy the concerns of the office of the Privacy Commissioner in Canada and the OPC understandably took a very privacy centric approach in their enforcement action against the company and specifically they wanted to see us augment the privacy protections that we afforded to our customers so we brought on chantel Bernier who is the former interim Privacy Commissioner to come on board is Ruby's special privacy advisor for a number of months we also worked with dr. Anne Kevorkian who was formerly on Ryerson's privacy by design Institute to augment our privacy program and through a series of policy process security and many months of work we work to align ourselves with the privacy by

deciding standard we then brought in the privacy by Design Institute to complete an independent mature assessment and indeed I'm proud to report that Ruby and Ashley Madison is to be first in today only dating website certified under the privacy by design standard next we needed to address the concerns of the FTC and so the FTC took a much more information security centric approach in their enforcement action and they wanted to see us implement a recognized information security standard align our program with that standard and then have that implementation assessed for design and effectiveness every two years for the next 20 years so knowing that we were going to be saddled with this framework whatever we chose for

quite some time we obviously took our time evaluating the different options you know as those of you are familiar with there are a million different security frameworks probably way too many and we ultimately landed on the NIST cybersecurity framework and so the CSF is unique in some ways is that it was developed by a consortium of academia government and private sector experts and so the result is a framework that is both thorough and comprehensive but also tactical and pragmatic enough incorporating many of the key controls that we felt we wanted to implement any way to ensure breaches ilion's so it took us about six months to do the implementation to align our program into a completely overhaul what we already

had and to align with the framework we then brought in Ernst and Young to complete an independent maturity assessment which we then submitted to the FTC for full compliance so this work gave us the foundational air support if you will to allow the business to begin to resume normal operations so in 2017 they're around 15,000 fine of 500 unique sign ups to Ashley Madison every month or every day sorry works out to be about 500,000 per month in order to derive at these numbers we actually brought in and a ey assurance team to perform an independent review on the sign up numbers the ratio of male to female users and indeed to certify that certain problematic business practices

such as the use of chat BOTS and engage our profiles had been shut down several years prior should you be interested the full report is available there probably caution against you visiting that link on your corporate devices or workstations though however but should you be interested there now that reality was for for myself and my GC and privacy officer as we looked at these numbers which now today in 2019 are closer to thirty thousand signups a day and we realized that while people are going to continue to use this service and people are going to continue to sign up in droves and they're going to entrust us with their personal information and so we felt a

responsibility to stay with stick with those people and to fight for them and to build and can you continue to build and expand our privacy and security programs and to take things to the next level and specifically around security we knew that the company was operating in a threat landscape and was going to remain a target that it wasn't just sufficient to meet the benchmarks set to us by the regulator's we need to go further and we needed to take it to the next level so we set ourselves a pretty lofty goal and that was to build and maintain a leading threat based intelligence-led program capable of defending the company against even the most advanced threats now

there's a lot in that and some of you might accuse me of playing buzzword bingo here and but really the reality is what we were trying to do is to understand the threat landscape that we operated in understand the unique adversaries that we faced as a dating website as a controversial service offering understand the tools tactics and procedures of those adversaries how do they do what they do how do they compromise organizations and then to build a program capable of defending against them but you know as I'm sure all of us in this room we are all security leaders or managers or practitioners or students and we all have the same goal right nobody wants a breach so it's one thing

to set yourself a goal to not be breached and defend against advanced threats but how do we actually do that and the unfortunate reality is that every week multiple breaches are publicly exposed new ones it seems like every day we're hearing about something new I can tell you from my time working at mandiant one of the leading Incident Response firms in the world that many more breaches are happening every day and every week that are not disclosed and indeed mandiant publishes a report every year called M trends which is sort of a state of the cybersecurity Union that's great I highly recommended you check it out it's free you can download it but one of the key metrics that

mandiant measures in the m trends report is the global median dwell time so for every investigation that mandiant has done the year prior the dwell time measures what was the earliest evidence of attacker activity in that breach - when that breach was ultimately first discovered now when I worked at Mandy and a number of years ago this number was north of a year it was between three and four hundred days which meant that for the investigations that we were doing the attacker was typically active for over a year before somebody found out they were there now the great thing is this number has been steadily improving in 2016 and 2017 it was around a hundred days and then last year it

dropped down to 78 days so it's clear that we are improving as an industry we are getting better we are moving the needle and that's really exciting the problem is is that it often takes a skilled adversary a matter of hours often just days sometimes a couple weeks to compromise your organization and complete their objective so it's great that we've improved and it's great that we've moved the needle but if it's still taking us close to three months to find targeted attackers in our organizations well then it's rather moved them and so I think it's clear that our current approach as defenders as an industry is failing we're not keeping up with a threat and with that I'd like to present

the argument that breach prevention in its truest form likely is impossible but achieving breach resilience is but before we dive into that I think we need to explore well why is our current approach as an industry failing you know never before have we been spending more time money and effort at every level of organ our organization's on cybersecurity and yet the torrent of breaches shows no sign of slowing down so John Lambert who's the general manager of the Microsoft threat intelligence Center published a piece a couple years ago now trying to answer the same question and what John came up with was that defenders think unless attackers thinking graphs as long as this is true attackers win and what John

goes on to describe is a bit of a situation of two realities the defender reality unfortunately being a little bit like the fog of war where we can see ahead of us and we can see behind us but we don't know what we don't know and what's more we have all these obstacles in our way in our organisations that slow us down and prevent us from achieving the agility and the dynamicism required to combat today's adversary we have technical debt and legacy systems we've got a global talent shortage we've got challenges around burnout and retention of our existing staff we've got political eyes organization silos poor information sharing within large organizations we've got an ever

seemingly ever-expanding list of compliance and regulatory obligations that we have to comply with but unfortunately don't do much to actually improve the security of our organisations and all of this creates an tremendous disadvantage for us as defenders

and apologize again we've got some laptop problems so all right on the other hand the attacker reality is very simple it's find and exploit the weakest link in your organization that one employee that didn't attend anti-phishing training the one laptop that you forgot to patch the one API server with the Miss configuration and poor authorization or authentication on it gained some level of access to your organization pivot escalate privileges and repeat until they fully compromised your organization and for anybody that's ever spent any time an incident response or red teaming or are familiar with how these attacks often go the method that they're the route that the attacker will take through your organization is extremely nonlinear it is very difficult

if not impossible to model as defenders in our traditional ways of thinking and seeing the world and firewall rule sets in risk control matrices and patch reports these are all linear representations of information but to understand the attackers reality it means you need to understand the hacker mindset which is this is my current position in a system I'm going to look around me and explore all the different relationships that other systems nearby have with me I'm gonna move to those systems no matter how unlikely they might be and in terms of achieving my objective and I'm gonna see if there's any way I can escalate my privileges and I'm gonna expand out that way and so if

we try to combat this kind of nonlinear thinking or this these offensive tactics with linear controls then it's gonna look a little bit like this graphic here on the right with that gate being the perfect representation of the latest next-generation firewall or expensive blinky security appliance if we deployed and that a path on the Left showing us exactly what the attacker is going to do if we deploy these technologies in a linear fashion without fully understanding the unique vulnerable pathways that exists within our individual environments and so I think it's clear that we need to evolve we have to figure out Blue team 2.0 so to speak the old approach isn't working and the threat

landscape that we're all facing whether you're in oil and gas technology financial services insurance and just starting out it doesn't matter we're all facing a threat landscape that has never been more hostile that has never been more persistent and the reality is the unfortunate reality is the cavalry isn't coming us in this room and other rooms like it in the in around the country around the world were the only ones responsible for figuring out this problem and solving it and so the way that we've tried to tackle this at ruby is through what we call offensive driven risk management and so really all it is is a fundamental need to understand the adversaries and their tools tactics and

procedures their TTP's what are the adversaries that we're facing as our unique individual organizations who might want to target us and then what is the up-to-date and latest and greatest information about how those adversaries do what they do how do they target organizations what tools do they use what tactics do they use and then we can begin to model this behavior pretty accurately actually this graphic here is the mandiant targeted attack lifecycle it's very much based on the Lockheed Martin cyber kill chain there's lots of life cycles like it but they all basically try to model the different phases that a targeted attacker moves through when compromising an organization and so we can use something

like this in our case we do use the mandiant life cycle to break out the individual phases that an attacker will move through and then to model the attackers activity in each of those phases to simulate it and emulate it and assess our performance against that and what this looks like is a continuous process of emulation of proactive testing and adversarial simulation emulating the tools tactics and procedures of the adversary at each individual phase of that lifecycle analyzing our results performing gap analysis measuring key metrics such as what was our mean time to detect this simulated incident what was mean time to respond did we improve since the last time we did this a few

months ago how does the staff feel about it how does the sock feel about it did we improve did we not improve what did we do right what did we do wrong and then adapting feeding the results of this into control and process enhancement securing our environment mitigating vulnerabilities enhancing our response processes and then repeating the cycle over and over again continuously and the key thing here and the way this differs a little bit is that some part of your organization should be under friendly attack at all times so it's no longer sufficient to just go get a pen test done once a year and you have a couple consultants come in and plug in that a user segment and

you know exactly where they are and your network guys are watching them the whole time and you know you get a report at the end of the day we need to move to continuous assessment so we need to be continuously staying up-to-date with the tools tactics and procedures of the adversary we need to be emulating those in our own environment assessing our results and improving so well okay great well how do we actually do this so in the last three years or so we've seen a huge surge and Canadian companies building their own red team's this is something we always advocated for number of years ago it's finally starting to happen which is very exciting so these

are red teams within the organization that has a mandate and the resources and the authorisation to continuously test the organization continuously attack the organization and try to find vulnerabilities you don't have the resources available to you to do that you can bring in outside cult consultants to do this for you there's a lot of amazing talented firms that do this and then next if you have any kind of internet facing presence or application whatsoever I cannot say enough good things about a bug bounty program so we manage ARS through hacker one it's one of the providers that offer these programs there's a number of them but essentially for those of you who aren't familiar a bug bounty program is

basically a framework by which you invite researchers around the world to continuously test your sites and services to try to find vulnerabilities responsibly disclose those to you in exchange for monetary reward so in our case we've been at this for about two and a half years now we've found in fixed 500-pounder abilities in our sights and services on the low end we'll pay about two hundred fifty dollars for cross-site scripting on the high end for a remote code execution we'll pay around fifteen thousand but the key wonderful advantage of this is that it's crowd-sourced security even as I'm standing here talking to you guys there are a couple thousand maybe a few hundred researchers around the world testing my sites and

services reporting vulnerabilities trying to find issues before the bad guys find them and exploit them and then of course facilitated simulations so tabletop exercises if you have an incident response plan that's eighty pages long that you haven't tested and you've never opened it and very few people know it exists it's pretty much worthless going back to our impromptu title here everybody has a plan to get punched in the mouth if you haven't tested that plan at a technical level at a managerial level in a board level it's worthless so bringing in third party consultants or facilitators to help you run TT X's to test those plans and ensure that we're prepared for a breach and then of course specialized testing

as you need it if you know ot or SCADA environments or mobile app reverse engineering whatever you might have bringing those folks in who have that specialized knowledge to your environment and then what we can do is we can actually feed the output of these offensive risk management processes into a traditional risk control matrix type of model so in our case we're using the mandiant life cycle we've got the phase on the Left we've got the common phase activities in the first column and then some example controls on the right and I'll run through this pretty quickly but basically the first phase initial recon so this is when the attacker is going to try to build a profile the target

profile on your organization essentially they're going to case the joint and try to find out you know what does my target look like what does your company look like this can involve open source intelligence gathering checking out all your employees LinkedIn seeing if they're sharing information on there that might help them are your employees going on LinkedIn putting specific technology versions in there LinkedIn profiles and sharing that for example they're going to perform passive and active port and service scanning so services like showed in or nmap scans to try to map out your network and so we're offensive testing can help us here is it can find many of these issues before the bad guys do a bug bounty program finding

vulnerabilities open source intelligence simulation or gathering if you pay a couple consultants to do a no sand exercise for you and find out what exists and what information is out there on your in your company before the bad guys find it next comes the initial compromise so this is very often done with social engineering email phishing this kind of thing occasionally you'll see vulnerability exploitation or internet miss facing miss configurations again where offensive testing can help us here bug bounty programs finding those misconfigurations phishing exercises red team exercises testing how many employees actually clicked on the link how effective our security and awareness programs these are very difficult things to measure by not doing this kind of testing because

we don't get real-world data but by stimulating and emulating we can find out exactly how many people clicked on that link exactly how many people installed that payload we can find out how well did our email and content and malware filtering solutions work was the red team able to get a payload through the email filter how many tries did it take did we detect it did we alert on it next comes establishing the foothold so the attacker is going to initially compromise a system they're going to deploy malware establish command and control and establish persistence into your environment and here where this kind of testing can help us is well exactly how well do all of these

so-called next-generation security technologies that we all pay dearly for actually work against a real-world targeted attacker I will tell you the first time that you pay big money to have a red team exercise done and none of your entire security technology stack actually fires off a single alert it's a very eye-opening experience for you and raises a lot of questions for your vendors next the attacker is going to escalate privileges local or remote vulnerability exploitation finding unsecured credentials again where this kind of can help you do we appropriately manage credentials in the organization have you given your employees a way to securely manage their credentials or store them if you haven't chances are there's text

files laying around with usernames and passwords and a red team is going to find those and be able to deal with that before the bad guys do and next begins sort of a cycle of internal recon lateral movement and maintaining presence as the attacker moves through your environment really where this kind of testing can help us here is essentially well how well do is as our capability work to detect an active compromised in our environment do we have the use cases and processes in place to detect that activity and report on it or alert on it and then lastly complete the mission so this is when the attacker is going to take the information from your environment and

send it out to the internet exfiltrate the data and unfortunately if the attack reaches this phase you're probably not going to be able to stop it but again where we can what we need here is a tried tested and true incident response plan where everybody at every level of the organization knows their roles and responsibilities like the back of their hand in the event of an incident

all right okay and lastly I just want to leave you with one thing if the worst does happen because incidents are going to continue there's going to be breaches there's going to be misconfigured api's there's going to be s3 buckets put on an Amazon with a leaking client data and when these kind of things happen we need to hold our peers at other organizations accountable but we also need to practice empathy collaboration and communication with them because I've noticed a pretty disturbing trend over the last few years and some may argue with me that it's always been there where we seem to blame the victim a lot of the time in when they have a breach and I can't really

think of another industry or another crime where it's so common to victim blame so we saw this for example with Equifax when Equifax have their breach some individuals discovered that their seaso had a formal education and music and then decided to essentially relentlessly attack the credibility of that individual and try to ruin her entire career because she was at the helm when a breach happened and clearly because she went to music school 30 years ago she doesn't know what she's doing and if you spent any time on InfoSec Twitter or any other of our wonderful communities like that you'll know exactly what I'm talking about you'll see this when the next breach happens but I guess my message to you is

really if you're just starting out in this industry if you're a student if you've been here a while I think it's incredibly important that we practice three three three values because I guarantee you the adversaries are doing this they have their forums and their use groups and their mailing lists and there's no pretense they continuously talk about their challenges their weaknesses and share information and intelligence on how to compromise organizations and they do that so much better than us because of this kind of toxic climate we've created its created a chilling effect I think in our industry a little bit and there's two problems with that number one it just makes us bad humans but two we're very

unlikely to share and talk openly about our challenges and our weaknesses where one another and you see this you go down to Vegas you go to the big conferences everybody wants to talk about the latest and greatest offensive exploit or this and that and you'll talk openly about that but try to ask somebody hey what are you not doing well in your organization and nobody's going to talk to you so I would encourage you please over the next two days at this conference at any other conferences that you go to have those conversations with your peers I guarantee you we're all facing the same similar set of challenges and if we do that then I

really do think that we have a pretty good chance of turning the tide and getting in front of this thing so with that I'd like to thank you very much for listening and I'm more than happy to answer any questions that you might have about the breach about our response or anything in the presentation [Applause] all right so what we'll do for questions we kind of realize that we need a mic to go running here so if you have questions we'll just ask you just weakly come to the stage we probably got time for about one or two so someone can come up here someone's gotta have some type of question yeah come on up and then just I

just ask you to the mic and I will pass the mic back to Matthew here Matthew thank you for the presentation for the use of third-party bug bounty programs what are your thoughts on the maturity scale of the organizations and before consuming that kind of services so that's a great question one of the things when we first decided that we were going to launch a bug bounty program we told this to to our board and the question was you know they looked at me like I was an alien like you want to invite hackers to attack us you know isn't that going to go horribly wrong and the short answer is it can but I

think the key there is if you use a managed platform such as hacker one or bugcrowd or anybody like that you can actually set yourself to be an invite-only program where you can slowly invite only reputable known people on the service with high signal ratio that have a lot of experience and you can control the flow of researchers participating in your program and ensure that you don't get overwhelmed because I agree if you launch this and you were an immature organization and then you opened up the floodgates and you can't keep up with remediating the vulnerabilities it's gonna get potentially very bad for you very quickly but they do give you that ability to kind of control that flow

which is a fantastic feature that's what we did we work to ramp up very slowly from you know at the at the beginning we had four researchers now we have 4,000 so okay okay thank you very much [Applause]