Identify | Protect | Detect | Respond: A Deep Dive Into the 2017 Equifax Breach

hello good afternoon my name is jim nimitz i'm going to discuss with you a famous breach from 2017 and to try to make sense of all the issues surrounding this breach I'm trying to place the all the different specific items the problems with the breach into the NIST cybersecurity framework functional areas to try to see where the the fails or the problems were just a very brief little bit about myself I am here on leave I work for the US Army certain Germany I'm here on leave at my own expense or this talk been doing in IT security for about 20 years I spent about two years in the IT field and then got into security and

I'm former law enforcement so in addition to the previous thing that says all my comments are myself I have to put this in my slide my PA uh guys told me I had to put this in my slide so this is not the government's or the Army US Army's to use these are my views so I gave this talk at a close conference a couple of months ago and I realized I didn't get to all the parts on what the attackers did I went over all that so it was like my fail I didn't really get to all the juicy details so I'm gonna put all the attackers actions up front so they gained root code execution or an

Internet facing web application they installed a lot's way too many web shells and we can kind of talk about a this theories on why so many web shells were installed but maybe it was all like a whole bunch of people doing the doing the thing as a group like a group intrusion they found plaintext passwords on multiple databases do you do lack of network segmentation they laterally moved throughout the network they extracted loads of PII from multiple databases and they sucked it all down in different chunks using double you get and they were on the system for about 75 days before running rampaging around doing all this stuff before they were detected so here is the vulnerability

they exploited the struts vulnerability and I just like to it's got a score of 10 network vector low complexity no privileges required pretty bad and here's the company we're talking about in case you didn't read the synopsis or didn't guests already it's the Equifax breach we're talking about here so for those of you who aren't from the US Equifax is one of the major credit reporting agencies they take they record all your credit transactions card transactions credit card mortgages all that jazz revolving credit accounts at stores and they record all this stuff and they they give you a score for your credit worthiness when you go for another loan a bank loan or a car loan

or whatever if you've got a bad score you're they're not gonna give you a loan the bank won't give you a loan because Equifax or Experian or Trans Union said you had a bad score or they might give you a loan with a higher interest rate so to make these decisions they have absolutely insane quantities of PII about everybody and it's just this goldmine of PII for an attacker to to go after it resulted in the theft of 145 million US consumers there were some other smaller amounted numbers from Canada and even the UK lots of lots of forced retirements and firings and this July fine of up to 700 million I although I don't think that they are

actually gonna pay that much here was the and this is all out of the Congressional reports on this here was the placement of the CSO and a lot of different guys online Krebs talked about this but uh like here is this is Oh the CSO and where is this person this position it's under the chief legal officer and apparently that it's not it's uncommon as I would have thought it would have been I think he said Krebs had like 20 percent of the respondents in a survey he did said that that's where their says it was but I think that is suboptimal I think it could work successfully if you had good communication between everybody everyone

was talking each other but in this case everyone was not talking each other when the CEO got a briefing on cybersecurity this is Oh didn't give the briefing a lawyer did and I have to wonder did the lawyer take the cybersecurity stuff seriously I mean did he did did the the CISO prepare something and did he read it offer to eat which he tried to tell say something like and I don't some of the stuff is like speculation because I don't exactly know but here the congressional testimony you'll kind of get your own ideas about how this is going on like if she really wanted to tell tell the CEO that something was really bad she wasn't allowed to like

tell him herself she had to go through the lawyer and is the law her boss the lawyer and his lawyer filtering that or not or maybe they'll maybe since he's not an IT guy maybe he's like I don't really think this is serious not gonna bug the CEO about this I mean you know you kind of don't really wonder you wonder how that was going on also not very good communication between thus is oh and the IT department which she was supposed to be giving oversight for so the system that was breach is this system that was created in the late 70s and I can't imagine that it was on the system that there was actually breached

but the the this AC is automated compute consumer interview system the system the version of the system that was breached was created in the 93 to 94 timeframe it was a bunch of solare boxes running Oracle and I can imagine anybody who's into like pentesting would think that since a very juicy target and I also think is an extremely juicy target because you've got all this middleware Java or call dispatch and corn scripts flowing back and forth to get this thing running and with on the front-end Apache struts so there's probably all kinds of juicy things to poke at in there and this was directly connected to the Internet and I could find no reference of an ever

being contested or audited and they made in the commercial testimony it kind of made it clear the upper level of management did not understand how this system works because it was so old and it was only still running because they had some of these super old dudes still original developers still working on it and that like absolutely freaked me out they think at least like single points of failure that these guys could have been like 70 plus years old I mean they could have retired at anytime or something could happen to him so this is potentially fit made of a tional quote here so there's you know chris dickerson probably a lot of you do he's quoting

potentially quoting a guy named vince lombardi which was a famous american football coach who was famous for his motivational quotes I thought this was a great quote here but I couldn't actually find it anywhere and I think maybe somebody made it up but it sounds like something from are you would have said but the point is is that if you're really good at the basics you're going to be successful and I think you'll find we go over these issues that Equifax was not good at the basics so the areas that I want to go through with the framework or the first for the identify like identify when you say identify what is on your network what is the scope of

your network the IP ranges what kind of hard words offer is on your network who owns this stuff if something needs to be taken offline or patched who is it that you're going to if you don't know what's on your network you're gonna have a harder time defending it for instance and we'll get to this but uh they didn't know they had struts on their network they forgot for this this had this event the the vulnerability was announced or discovered and March 217 and they had just patched their struts stuff in January for a different vulnerability and they forgot that they had struts so protect got to protect your stuff the whole defense-in-depth jazz patches all that type of stuff if

someone does make it past your protections you'd like you'd like ideally you'd like to detect attempts as well but certainly successful breaches or problems and once you've got once you see something that happened something you detected some kind of event that happened you would like to respond to it there's been places I had a customer and people in this room know exactly what I'm talking about where we had really a really strong detection game and we would see attackers these are these are Red Team guys not like real guys but we would see attackers coming in and doing stuff and we would offer them mitigation advice its own how to protect their network and they wouldn't do it so it's

very frustrating if you I mean if you got a really good detect thing you see all the stuff going on and then we try to get them to respond and they wouldn't respond we didn't have response control over their network we were only doing the detect piece so really all the stuff kind of like flows together I'm not gonna really get into the recover piece for this talk that recovered according to the nist the recover is not the same recover i'm thinking of when I think of recover I think of like go reimage that workstation if it got compromised they're not talking about that so again they identify what are your IP skips what hardware is on your network

with software is on your network who owns this stuff who was responsible for the care and feeding of it can your vulnerability scanner touch all these different subnets if you've got a big network is there something blocking variability Skinner that would help that would prevent you from from doing it letting it do its job and identify you know the stuff that's on your network that may or may not be patched ok lots of failures from Equifax on this they did not have a consolidated software list that was accurate the IT department and Security Department had lists but they were neither one was complete and they didn't have everything covered their coverage did not extend entirely into their

legacy network this a CIS system was at a data center and Alpharetta Georgia and they did not have full coverage into the status Center when they're scanning stuff with their scanning software mainly it was hired in response to the breach to to response to it and bright invest do an investigation write a report and they are seeing all these databases that were popped and they couldn't figure out who owned him and they all these databases are all kinds of PII and it was all somebody in Equifax but they couldn't figure out what department it was again they had just seen the struts vulnerability patched in 2017 and they forgot about it and when they finally discovered the

breach in July and if the stress had been breached they were surprised because they didn't think that they had it so protect this stuff you've seen it a million times but I think and a lot of people are like they're bored by it or whatever and it's not exciting it's not sexy but it will absolutely save your bacon if you have a good defense in depth over and over again I see like an attack we'll get through one piece but something else will stop it like it gets through the SMTP security gateway or and maybe the in point catches or it gets past everything and then it finally gets caught on the going out at

the proxy or the IPS will stop it or something so really big fan of defense a debt that's not just some buzzword it absolutely works the vulnerability management I think there's another slide on this we can talk more but uh the the patching was reactive at Equifax the administrators would not patch anything unless the security department told them to patch it so there was no like proactiveness Lake just the administrators were not like signing up for their the vulnerability mailing list from struts or like whatever product that they are responsible for other simple simple stuff you hear all the time like plaintext passwords they found those immediately shortly after they exploited struts they found them on the

system on the internet facing system that passwords to all these other databases they were exploited databases that the system didn't need to talk to so that's like a lease privilege and network segmentation issue here is this cool setup that they had which didn't work which I mean if it did work it would be pretty neat you know I mean um so internet border gateway router and then SSL visibility plants which I hadn't actually heard of I've heard of other terms before but not that exact term IPs and IDs and then this ASIS a CIS system here there are certificates for this visibility appliance had expired in depending on which report you read between six months and a year so

they had no visibility to see what was going on this traffic inside and out of the encrypted traffic inside and outside of this database well this PII on it no mention I mean I assume that there was a firewall in here somewhere but that's not mentioned and a wife was also not mentioned which probably wouldn't have worked if the certs were expired but would have been nice to have that in there so ineffective patching we went over that plaintext passwords no laughs the IPS was blind he couldn't prevent it couldn't do its prevention piece because it couldn't see inside the encrypted traffic no network segmentation the databases were also unencrypted there were just so many problems you see over

and over and over again it just wasn't just one thing I should have said this at the start like like I thought that a lot of you might have thought that the issue this was all caused because of a missing patch but it was way more than that even if that thing was patched it there were so many other issues there's chances if someone was actually determined they could have gotten in anyways so I have some my pen testers and former pen testers in the room but they don't have that they don't have to answer this question any they're going to so if you were gonna run a Rivera build assessment or a pen test on an

internet facing application what you think would be the right time to run it before it's put into production after it's put into production after has been compromised or after it has been decommissioned anybody but I will say that I don't know if it's like that over here but in America the common saying is if you're taking a multiple-choice question and you don't know the answer you should always pick C so oh my I think didn't work so it was C so they did a pen test on the application after they compromised it they found out it was vulnerable to SQL injection and like a bunch of other things and then they decided to take it offline so an

effective vulnerability scanning they scan on multiple stuff multiple different tools but they couldn't find they couldn't find it and I'm not sure if this was because their tools were configured properly or they had poor visibility all the same the attackers were able to find this and I don't know if yet for what direction were they scanning were they standing for these vulnerabilities internally or were they scanning them from an external incident the attackers found this found these vulnerable systems I mentioned the path patching policy the IT department didn't patch anything unless the security department told them to patch it the security department did tell them to patch struts if they had it but the email went out to like 300 people but it

didn't go out to the guy who was responsible for patching the struts instance they had an internal audit that discussed their there their patch management system and they were told they were they the the audit urged them to change to a piratical model but they did not do so so just like you have defense-in-depth I am a believer in detection and and logging in-depth for all of your witness devices all these items should hopefully be going into your seam and normalized so you can you can make sense of all this stuff and if you're not sure where if you've got a seam you've got these logs you're not sure where to start really advise you to

look at mitre attack had great luck with minor attack and they did hit detection piece so the attack that suckled the PI out occurred started in May but another attacker got in on the 10th of March now this could have been the same people that came back in May but we don't know someone came in popped the system and ran Who am I and just left so I don't know what they were doing you know could have been the same guys may not have been the same guys there was nothing monitoring the key integrity of files they were all the web shells were inserted they could have used I know they're like oh this is an old system

it's legacy we didn't want to touch it but they could have used tripwire I mean that's a 90s era product they could have written a bash script or a Python script to run an md5 hash on these files every night there's like simple super easy stuff they could have found to see if the tech that piece of the intrusion they had over 300 SS tls/ssl certificates expire in 2016 they were monitoring their perimeter stuff not just this system other system so they had a massive problem with that and so they could not detect these things even if they couldn't see it if they were looking at their their network logs they could have seen that they had bothered a look at

the network logs maybe some older would have asked the question why is all this outbound traffic going to China when all the PIO that data was getting sucked down so and finally and they had this long term super slow-moving project to update their SSL all their certs on their own their visibility appliance they installed it on the 29th and they looked at the logs he immediately saw outbound traffic and they blocked that one they blocked the network not just the IP the network that was coming from on the next day they saw another piece of another more traffic going to a German IP address it was also at least we had a Chinese connection to it then

they ran the pen test and found out it was vulnerable to SQL injection all this other stuff and then they decided interesting piece here I was so you know working for the army we always have like who is the authority to order this or who has the authority to order that like thing my possessed computer so they didn't really I didn't see that they asked anybody that they you know that the lower-level guys to disconnect this system they just did it I'm like well at least someone showed some leadership here and disconnected the system after was compromised um they hired mandiant to look into the breach and that based on the results over 105 45 million consumer data was

leaked so the conclusion of the breach is it really wasn't a priority for Equifax they had said that it was a priority security but it wasn't there were there were audits beforehand besides the internal patch audit that told them they had lots of problems they didn't obviously they didn't take it seriously enough all the potential issues that they could have in this one instance and very happy to see that a lot of the people in charge the response for this lost their jobs because that's what they should because mi in Pishin my my opinion that they were they were negligent big fine the competitors this was in the Senate report because there were two really

good reports from the house and one from the Senate they noted that the the competitors were also using struts but they admit they did not get hacked so there was a lot of fallout in this from this org chart this original or chart does anybody want to guess who didn't get fired or forced retirement legal who knows who had a cya more than anybody else the lawyers so the first people to go were David the CIO and thus is oh they got a forced retirement um the next guy was a CEO himself and the only guy that got fired was was this guy I'm not really sure he was more guilty than anybody else but he got the axe um not

on the slide they had there like a new problem we got a new C it we got an e CAA we're gonna bring in they're gonna bring this new guy in but they was involved in he was involved in the the recovery efforts and he got popped for insider trading so he pled guilty and had to go to jail for that so they couldn't have him be the CIO because he he tried to he dumped a bunch of his stock before it was publicly announced that they had a breach so a little bit of its discussion or questions where do you think this is au role should be somebody have an opinion on this the

optimal right foot right right out of the CEO that would that would be my preferred that would be Mac for her placement if you're under the CIO and the CIO you're beholding to the CIO and you're like grading him but you're grading the guy that rates that rates you so this kind of a conflict of interest there I was in a position at another job where I fell directly under the siz oh and he was trying to get me to falsify a compliance report to our headquarters and I wouldn't do it but uh I don't like my got my boss trying to get me to lie for him so I think really reporting the CEO is

the best solution I think depending on the situation maybe any one of these positions could actually work but I liked reporting directly the CEO the best because then if there's a problem and you think he needs to know he needs to know about it there's nothing stopping you from talking to him about it so there was a lot of a lot of kind of nasty things said about this is oh she had a degree in music and there was no record of her having any kind of professional security certifications so does anybody have an opinion on should the CISO have an IT degree or specifically cyber security degree or should they have should they have

certifications like CSP or sysm or any that jazz or does it really matter so pointing out just not beating up on her the CIO had a degree an undergrad degree in Russian and an MBA so he didn't have specifically I mean I don't know what kind of certs he had but does anybody have an opinion on what like what a SISO should have education and certification wise what like other roles in security I don't think it really matters as long as they know what they are actually doing and they understand the more important concepts or the skills that they need for example to master stuff like risk assessment and proper business dealing with security and all those what it is and stuff like

that as long as they do have those skills and become proof with some kind of track record or even through assessments or whatever it doesn't issue do matter what university degree or security certification they have it's just like an add-on in a sense for example yeah if you have C's okay sure nice it would it might increase your chances a little bit but if you don't have it you do didn't kill your chances either it's more about your skills I think I knowledge thank you that um that really sounds good I think all this stuff you know the IT degree that all the certs they can assist but you really need the right person in the

role that with a obviously if you're in a management position I think good managerial mindset a good leader is important but you really gotta have the security mindset and take stuff seriously so we kind of went over this and the last talk this was gone over but my question I will have one piece out of this what is the oldest something that you took action on that you found the logs and actually took action on he found something that they were told to find like over a year ago and actually took action on it no one looked at their old logs had to get someone told them that nothing no one's looking at their old logs I mean just because you how old

was it yeah so it helped you okay so um data was that just for an internal investigation or was that like for a law enforcement investigation or something well the second one what was it was was this in response to just an internal inquiry or investigation or like a lawn forcement investigation a front-runner in the State Journal said but it did help you yeah yes again anybody else don't really I'm not really thinking we're fine and we're generally about six months back is what I'm seeing every once in a while we'll get a request for something older but normally it doesn't go past six months for us oh I have supposed to hide this wrap up okay so um

a little bit it try to take away from this talk I hope you got that I hope you found the the Equifax stuff interesting the Congressional reports are really good you can use all their mistakes try to use all their mistakes to your advantage try not to make the same mistakes they did and use these points use these points to try to if you're trying to get something fix your organization and you're not getting any traction that you can do like use Equifax as a poster child as an example to try to get leverage to make something happen and if your leadership is not interested they're still interested you could remind them about everyone that was

fired or forced retirement from to make it sound make them know that they are not immune to just because they're a c-suite person not immune to being fired also have a gander at the NIST cybersecurity framework if you haven't done it already you can just look at that one piece and like what are these different areas mean to me there's all kinds of stuff if you dive into it it's actually not really hard to read I quite like it and again like a lot of other people I mean this the the mitre attack workshops right here too that's really good stuff if you haven't looked at it already I check out minor attack you can really

really really improve your detecting with it it's helped us a lot lots of references in the end here and any in in questions ten minutes oh I went way too fast any questions comments how that it yeah first of all thanks for a graph of the organizational overview which is quite interesting I know a lot of friends in different corporations which are Caesars under the IT department and this basically leads to IT security being a dumpster fire absolutely so in an interesting fact about the lawyers before being before the CEO or the board an interesting detail is that as soon as the board knows about major violations they are hardly liable with private equity so

there's in many corporations this concept that lower management is not allowed to talk to the board directly but have to go through briefings with lawyers and they kind of do some filtering on what you're allowed to say to the board otherwise they could be liable with private equity so from the side of the CEO this kind of makes sense to have two lawyers before the Caesar they known - my - they don't get too much information so that don't get liable private equity yeah fun fact okay I hear that it's interesting fact yeah it has to be like a major violation and if they're aware of it and don't act on it on something like that this can

really happen like this was a special cases but they they often have lawyers in between they're those conversations which briefed you and yeah also an fun fact about the congressional hearing I was hearing it I saw that the stream and one of those competitors basically was a reseller of FDA Equifax so short after the breach the stock went down a little bit but soon after all customers gone to the different competitors and in turn the the stock of Equifax even raced I think they even made money on this any could have yes I think they should have been hit a lot harder than they were then they were but but at least all these all those the

c-suite people got hit I mean at least because a lot of times you hear about these big disasters that happen in this and no one held responsible at all so at least something happened you know early retirement means that you get a shitload of money before you leave so I'm not sure if it's something good a bet happen to you if you're in that position I think that that early retirement was contingent upon how much the board liked the congressional testimony if you read the testimony it's very interesting and I did not have a favorable opinion of that scissor based on the stuff she was saying oh it's not really important to keep blogs I'd really you know it

doesn't really matter and not really matter that we didn't know what was on her Network it really wouldn't have helped us I'm like I don't know if the lawyers told him to say this she truly believed it herself but yeah anyways thank you oh thank you what else I'm very glad that you mentioned the NIST cybersecurity framework I know it's a bit new for now but have you seen more implementations of it in your work more people are trying to implement it because when you try to for example to look for examples they are very specific use cases like Siemens for example that implemented the entire framework and went through it so have you seen some

like in real life while working or consulting more people who applied the framework altogether and got some results or positive results from it I've only seen what we've done with it and there's a regulation that I can throw in here because I again I work for the US Army cert so I'm not going around consulting with people I mean I talked to people at conferences but we have we have some documentation you know the beauty manuals a lot of their regulations the stuff they highly reference fitness stuff the our our doctrine from ours of has similar to the the framework they call some of the areas differently and some months they have double things in

these sections but the current one we're using is from like 2012 and I actually prefer the NIST one because it's newer and it's it's actually clear we were I really look started looking into this like a year year and a half ago because the red team was like running circles around us and I was fed up with it and a lot of people or all other my colleagues were fed up with it and we got a bunch of people together and figured out how we could prevent the red team from owning us and that's when we relied heavily on protect detect and respond we went through all kinds of had our guys look and see how was the red key

breaking in and and tailored mitigations to each one sometimes people didn't want to cooperate but we managed to get a lot of different mitigations in that's some of the protect piece some of the detect we relied heavily on mitre attack and for a response we like to go down to the lowest level for putting isolation actions in place so we were so like the guy at 2:00 in the morning if he sees something going on in a workstation he has the authority to isolate it he doesn't have to ask permission from anybody to isolate you can always on isolate it I mean but we want to quarantine a box so that's that helped us I wasn't specifically looking in each

you know you can look at the NIST document I don't know if it's 70 100 pages and I encourage you to check it out if you if you can make it through all the fine print I really was looking at broad pieces like you you know the broad pieces I kind of went over and you know wish nope them thank you okay thank you so much [Music] [Applause]