PG - Everything is Not Awesome: How to Overcome Barriers to Proper Network Segmentation - Jason Beat

I think that's it so without further ado here is everything is not awesome with Jason bTW thank you so I got to talking with my co-workers about segmentation in the context of want to cry and then Petya and there's widespread consensus amongst us plus you know the Twitterverse that network segmentation is generally a good thing and that complex network segmentation or you know more robust network segmentation beyond like a DMZ and a guest network is a good thing but then why don't we see it implemented more often so in my consulting background I didn't see it very often and a lot of places were like yeah we know it's good but and so I want to talk about those

cases constructively so I don't want to be like hey these people are lazy or hey these people are dumb all these people meant well all these people tried and for some reason their projects failed or stalled and so we need to talk about that constructively and figure out ways to overcome these particular barriers so things I've been at some point in time besides the besides LV backup photographer um the I've been a systems admin for Windows and way back in the day net where firewall admin network engineer and then I got promoted to managing director so I ran a IT infrastructure group did budgeting and project management and all of the things associated with running a couple

different teams and then I moved to being a security consultant and so I got a lot of different looks a lot of different places a lot of different verticals and helping clients building and maturing their particular security programs and the second reason is because I made a lot of these bad excuses myself and so this is my chance to sort of come clean and to you know to talk to you all about some of the things that I've said where I said yeah you know we can't do this because of this reason so I'm just as guilty as anybody else so we're in play a little bit of a game get people moving a little bit but

if you've either said one of these things or been responded to with one of these particular things go ahead and raise your hand I want to raise my hand now because I've said all of these things but the first one is will be seen as disruptive if we block business related traffic we'll get a few steps in and they'll push back so anybody excellent okay if the attackers have gotten this far they already own us anyway so we're not going to excellent okay cool we're not staffed to conduct or maintain an internal segmentation of effort of this magnitude so we can't do this and our day jobs okay a little bit less but good excellent our firewall cannot

handle all of the internal traffic so if we push all this internal segmentation effort into this particular firewall it'll overrun it and our budget won't cover buying another anybody excellent okay cool so I'm not alone so there's a couple elephants in the room whenever we're talking about these particular things I have 20 minutes to go ahead and give this talk and so internal network segmentation is one facet of a much larger layered defense and death strategy and so there's obviously things that play in very very well into these particular strategies and so we're doing Network segmentation you want to look at application whitelisting you want to look at your endpoint protection data classification another another role

areas that play into this but because we've got a limited amount of time I want to try to keep it right on target but there will be a few tangents and so if we have time at the end we'll dive into more of those and we'll definitely talk about those so when we're talking about network segmentation sometimes it difficult to say what good segmentation looks like because it depends it depends what you do as a business and it depends kind of you know a lot of different factors associated with your budget your staffing and all of that but we definitely know what bad segmentation looks like and so we'll start out by defining the the negative if you will so

we know that crunchy exterior and chewy Center model is really really bad and in this particular diagram what I'm talking about this is the stuff on the bottom all of that is the internal network and all of it has implicit trust to everything else and so you've got sales laptops that can get to database servers and core databases and other proprietary information in the business and the other thing that we commonly see and this is fairly terrible but we see that implicit trust then you know water falling down to DMZ and possible web servers and so the internal network is assigned a higher security level if you will and the DMZ is assigned a lower one

and they're just allowing the implicit trust to roll all the way down to that so that's also terrible in terms of attack surfaces and you know other bad stuff that could occur so defining what good segmentation is like I said it's a little bit more difficult and it depends a little bit more but we can talk about in general what these sorts of things might look like and so DM Z's and guest networks are a great start and I don't want it for some reason malign those because they're definitely essential they serve a really really good purpose but don't stop there so what I'm talking about is is looking at particular business workflows and so diving in and saying if

you have say an accounting department or whatever else you're looking at just accounts payable you want to look at their particular workflows and say okay how do you go ahead and cut checks to a given vendor or to expect you know to fulfill expenses for your employees and so on and so forth and then define the internal boundaries around those things and so you want to make sure you understand exactly where to go ahead and set up that particular swimlane and then you want to fall back to least privileged models if they don't need it so no one else in the business should be able to access those particular servers if they don't need them that's what

we're talking about in terms of robust internal network segmentation again this is common sense stuff you know I see a lot of heads nodding and all that but if these are things are common sense why don't see them more often is sort of the thing that we're trying to attack here and so that first thing that we kind of you know the first two items in the list I'm going to kind of sum those up as segmentation is too difficult when but it's really will be seen as disruptive if we block this initial related traffic we'll get a few steps in and they'll push back and then the attackers have gotten this far they already own us

anyway so I owe this this terminology by the way to my old boss Mark Stanislav and networks at duo but he came up with the coin slot rail approach which is effectively if you're trying to sort coins and you're rolling things down a rail or whatever else you're gonna match the smallest coin first and go ahead and drop in next coin next one so on and so forth he referred to this in matching PCI saq types but it definitely applies in terms of this as well but generally you know if you're your strategy is block it and see who screams you're not going to get very far you're gonna get some sort of disruptive change that

occurs somebody's going to go ahead and escalate around to a c-level exact we need to find a way to do disruptive change and we need to take a slow and steady approach to it so when I'm talking about this one of the things that I would recommend is we know we need to secure IT those accounts are you know kind of the sweet spot for an attacker looking for a lot of lateral movement and we know we need to address those particular workflows you know really really soon call it in phase one of this particular project but they don't recommend them step one phase one maybe step three step four or whatever else so attack something that's a little

bit easier payroll and HR not that they're particularly simplistic they have their complexities as well but they have some built-in separation of duty in there and so what's nice is there's some discrete separations that might help you out in terms of a segmentation project and so when I'm talking about going ahead and doing non-disruptive change what I'm talking about is taking advantage of that first match the coin slot rail approach sort of things with common firewalls where you can go ahead and write in a very very specific ACL where you go ahead and say this one user group this accounts payable or you know smaller team can go ahead and access this particular server over this

particular group of ports or this particular port and you can even narrow it down even further if you know that there are certain time ranges that they access these particular things they don't work at night so on and so forth may not be true but get as specific as you possibly can and where you would normally put a deny beneath that the denial would be disruptive and so put a more general rule beneath that so that's the next layer of that coin slot model again sort of common sense but go ahead and roll that through but where we see people failing when I was doing firewall reviews I was looking at those things and then seeing that a lot of these

rules were stagnating so they weren't continuing to go ahead and groom these particular rules and find out where they could go ahead and then put that explicit deny to say okay no one else can go ahead and access these things and so one really really good strategy for that is to go ahead and set up regular recurring meetings to go ahead and review all of the workflows that you're trying to go ahead and narrow down and have your peers go ahead and enforce and say hey we need to figure out okay where the hits on these particular rules what things are going ahead and hitting these more general rules that we could write more specific ones on top of that go

ahead and keep on revisiting and keep on iterating in terms of that and so as I was doing this particular project we ran into a problem where we got a little confident a little bit too early and so we went ahead and started to do our explicit denies and narrowed things down and then we realized that accounting doesn't necessarily work on daily or weekly schedules there's monthly there's quarterly there's yearly stuff that that happens and and we got a little aggressive and so we you know we got called into the the principal's office to a certain extent cause some disruptive change it was no good and you know I my attitude wasn't great back then honestly and so I was like well you

know screw it if they're if they're telling us we need to go ahead and roll back all of this stuff then I guess we won't do it and I got some really really good mentoring at that time and it was there are two new things the easy way in the right way and if you do things the right way long enough they become easy and I'm worried that this fails at ethology check but at any rate I got sick of that yep hearing that but eventually I was one parroting that to my staff as well and it's absolutely true in terms of a network segmentation virtualization effort any of the other the big effort so you go ahead and do is

that you're going to need to go ahead and iterate but you're gonna hit some friction at some points and you need to just keep on pushing through that so it may be two steps forward one step back but keep on pushing through that and then if for some reason that's not getting easier I had to kind of reflect and look back and say part of the problem was kind of me and so when I was going out to the business and I was doing these interviews I was asking questions from the mindset of how do I get the information that I need and get the heck out of here so I could get back to doing my normal job rather than

aligning with them and saying here I understand that you need to do X Y & Z can you tell me a little bit more about why can you tell me when these processes kick in and to help align with them but also to kind of build that empathy a little bit I wasn't doing that initially and so this these projects were failing and I realized that they weren't failing because of lack of knowledge they were failing because of my attitude and so if there's any there's anything to be gained I guess just try to align yourself better try to ask the right questions there try to learn what the right questions are to ask kind of T

those up for people and try to empathize with what exactly they need in order to get their job done to keep the business running so the other one the attackers have gotten this far then it's over this is one where particularly end test they did the external and then it came in did internal and they were plugged in into an internal data center switch and the guy was telling me well hey you know there's a setting on this particular switch it allows me to go ahead and move laterally between ports and all of that and I kind of said you know that's great but you're you're physically present in my data center you know Gabe's kind of

up at that point you're kind of you know you've got physical access to all of the servers as well haha that response made it to the report that went to the CTO and again I was back in the principal's office again and it was like hey you can't just go ahead and dismiss this stuff it's not your necessarily your job to determine where and when to surrender and the CTO said it's not my job either that's the guy who signs the checks that's the CEOs job in the meantime we're supposed to work as hard as we can to go ahead and patch and and go ahead and remediating these things as we possibly can or at least discuss them

you don't get the to determine where when we decide to give up and so the tick away was treat every part of your network has absolute defensible ground don't just go ahead and pick and say ok if they get here then the gigs up and and we're done and so we started looking at printer segments we started looking at network management segments we started looking at backup segments and saying hey these provide a ton of movement and we're not doing any sort of detection or prevention inside these networks at all and so we started going ahead and securing those as well the nice thing was is that we knew those inside and out those were incredibly

predictable networks to go ahead and start segmenting and we just hadn't done it before because we figured ok if they're there already then you know the you know like I said the gigs up so we kept stacking those defenses both preventive and detective controls and so next one internal network segmentation is impossible to maintain so once we set it up we're not staffed to conduct or maintain an internal network segmentation effort of this magnitude we can't do this and our day jobs and and you know the short answer is it is your day job sorry but you know let's talk about this constructively to a certain extent it's rare that I saw a IT or is team that was

grossly over staffed or bored but what I did see was a lot of organizations that over purchased so they had too many or conflicting or overlapping security tools or they over licensed the tools that they had and the reason that they did add is that they didn't understand all of the workflows that were in place in their business right they were just going ahead and buying things and assuming that all of these particular workflows were equal and in reality when you you know you dug into it they didn't necessarily need all of the same tool sets and so that was kind of a take away as well it's you know a lot of these teams failed to

automate repetitive and time-consuming tasks and so they had some decision fatigue as well they burned time and effort on detection alerting schemes that had way too poor of a signal-to-noise ratio and so there was some frustration in and around that and they basically said you know I can't do all of this stuff and do the rest of this when in reality they needed to go ahead and reduce that particular friction this sort of akin to learning scripting or learning even like touch typing if you take the time to go ahead and muscle through it and learn it it pays dividends in the future you don't have to waste time doing that simple stuff and so that fact finding that

gathering of information and applying that network network segmentation effort basically leads you to placing more specific tools not just in the segmentation effort but just in general getting better in terms of your detection figuring out what does it matter and kind of pushing that the hey aside to find the needles in reducing the cost and that noise and the second thing is is as we went through and did this network segmentation effort that knowledge is really empowering and so you work more effectively but it's palpable excitement because you understand a lot in a much deeper level exactly how all of this stuff comes together internal network segmentation will run over run our current hardware so I've been phrasing a lot of these

sometimes as an excuse this one's more legit if you will not that any of these are particularly excuses but this one you know could definitely definitely bite you so our firewall cannot handle all the internal traffic on our network and our budget won't cover buying another so the first thing I guess is to you know don't try to go ahead and boil the ocean don't calculate out all of the traffic that you might possibly need to segment and then say okay well the firewall won't handle it so let's not even start that's kind of defeating the purpose to a certain extent and so start off with like I said a simpler workflow and figure out what kind of horsepower

you're going to go-ahead and need to get the rest of the job done and start scoping that out so to start figuring out exactly what models or you know what might make sense to go ahead and finish the job but don't just stop just because you can't go ahead and complete to a hundred percent the second thing is obviously you know use analysis to look for noise or garbage or abuse and and pv possibly either you know write policies or you know get to get somebody on your side to basically reduce some of that stuff so if you've got a lot of streaming audio and other stuff going over your network that's not necessarily business related

but it's clogging up you know firewall cycles and all of that it's possible you can go ahead and say hey we've got better ways to go ahead and get you audio and better ways to go ahead and get you fantasy football stats and so on and so forth vendors will often say you know let's reduce the logging capability or reduce the logging detail on the firewalls and I'm not a huge fan of that that cuts into your forensic audit trail but there is such a thing as it is information overload so if you're at a debug level and you don't necessarily need to be at that debug level make sure you're tuning that appropriately I saw

that a lot of firewall reviews is that you know all the way all the way up on the meter and they weren't using any of that particular information but obviously like if you have got functionality that you don't need like DNS and DHCP server functionality I saw that as well it's not going to get you a whole lot of games in terms of CPU cycles or memory on your firewall but it's something and so start looking at what your firewall needs to do necessarily and then figure out what else consider offloading some workstation ACLs I've had some good discussions in it around this the problem with that is you know some of the fancy next-generation firewalls has

some really nice features in terms of dynamic ACLs so a group then gets a ACL applied to it you can't necessarily do that at the switch level at least not unless you have a really really nice client that kind of routes that all the way through and so that might be the problem but if you have some simpler ACLs that are just go ahead and block you know that we don't need to go ahead and analyze this traffic let's just go ahead and drop it you can set up your your sim to go ahead and log that and alert if you're seeing some sort of lateral movement in your distro switches or below that might go ahead and help your

burden and then the last thing and again this I'll apologize for being Captain Obvious here but you need to push for that understanding that robust internal segmentation is not a nice to have anymore it's not the difference between an A and an A+ anymore the you know the attacks that we're seeing definitely require a much more detailed level of segmentation than we're seeing currently and so you need to make that case to whoever it is that we'd be going ahead and justifying the budget for building out this this effort we're at two minutes okay I have to speed through this just a little bit I apologize I had some learned helplessness so I had some

things where you know I obviously declared defeat long before we were actually defeated and so learn to recognize that to a certain extent learn to inoculate it by talking to your peers if you will and saying hey if you're seeing this coming on on my mouth call me on it call me on my BS so if I'm saying hey well you know if they're not going to give us funding for this I guess we're not going to do it instead you want to say okay well if they're not gonna give us funding how do we either talk to them about how do we formulate a plan to you know get around this particular barrier or how do we come up

with an open-source alternative how do we come up with a way to go ahead and do this it doesn't require nearly as much money or what else can we give up and if this is indeed more important but come up with solutions rather than just coming up with ways picking places where you'd go ahead and give up and then learning to love and leverage your compliance burden so I used to spend a lot of time basically saying hey you know the PCI are it's gonna be happening in May so I just blocked off my calendar for all of May and it's okay nothing's gonna get done from this time okay excellent five minutes almost done so you know I used

to be basically just dreaded because I you know I was always going to be behind at the end of May when in reality I should have been leveraging it I should have been using that compliance burden to justify a lot of the things that I absolutely needed because it was a really really good stick to go ahead and do that but unfortunately that means learning language of the business learned into compliance it's kind of you know kind of like radioactivity you don't want to get any on you but you know business risks task cycles you know role descriptions all of that stuff it's absolutely essential for communicating and getting budget justification for this sorts of stuff but it's how you

win and so find somebody that can do that or you have to take it on yourself if kind of to shoulder that burden as well and the second thing is and this is a story from mine but I got into the elevator and the CEO was in there and he's like hey how's that you know the PCI project going and what do you need if for some reason things go poorly and I struck out because I didn't have any sort of reporting I didn't have anything I could show him I didn't have something where I could say hey I wrote a report on that you want me to mail you on it or whatever else so luck favors

the prepared so get quotes figure out exactly the constructive solutions ways to attack things that you can't necessarily justify today but have that ready in case the situation presents itself to do it I referred to it as making ransomware lemonade awhile back but yeah and so what does the finish line look like and so we go from from this model where we got the chewy Center and go ahead and breaking all those things out and we no longer have that implicit trust model so all of these lines are explicit trust specific ACLs that were going ahead allowing and we're adding like I cheated and I added some some other technologies in here that I said we wouldn't talk

about but things like knack and all that fun stuff but that's kind of the goal that I'm talking about in the goal we're shooting for so you may never be done but here's where things get really good like I said your licensing you're alerting your budgeting start getting really more efficient you realize that not everybody might need the exact same tools or you figure out exactly what you need in order to get the job done for each of these particular workflows and I hope that you're going ahead and documenting these things so that for some reason somebody leaves they don't leave with all of that knowledge you can go ahead and transfer that knowledge over that was a big sticking point with

us as we had a ton of tribal knowledge that we lost every single time we lost an employee so you want to go ahead and do that the second reason is a lot of these workflows are only used maybe once or twice a year sometimes and so you won't remember you've got tons of other battles to fight you'll come back to this one you won't remember the details so definitely document it but your necklace your network in your ecl's will start mimicking your business processes and look a lot less like the vendor defaults and that'll help you a ton because the particular attacker is gonna have to spend a lot more time trying to feel out exactly what the heck you know

where are they on the particular network and where do they need to get to give them a little bit of a maze to run excellent two minutes local firewalls and host-based intrusion detection prevention agents so I get the excuse a lot of times well you know we haven't really done a whole lot of complexity in terms of host paste intrusion protection or host-based firewalls because we don't really know what they need well you will once you go ahead and do this in document and then a suggestion from Jerry Bell malicious link is you can go ahead and start looking at districts which port isolation so you basically say hey if you're on the same segment and you're

communicating don't go ahead and allow that implicit trust where one node can communicate to the other directly through the switch it's gonna have to go back up a layer or two that's definitely a really really strong tactic to go ahead and do if you want to limit that lateral movement and then like I said the attackers will need more time to run that maze set your detective controls such that you can go ahead and detect them and then possibly eject them from your network there was a question when I was discussing this as well of you know will you run into the top end limit of ACLs this is a terrible way to evaluate firewalls by the way I apologize so take

this with a you know giant grain of salt but yeah the theoretical limits on the number of ACLs for various platforms are relatively large you will hit throughput you will hit module memory limits you will hit tons of other things before you hit these things but people are asking exactly what the theoretical limits were so kind of look this up that's it I want to thank my my employer CTC for giving me a time to come here and talk to you I also wanted to thank Adam Brandt who mentored me in this and let's continue the conversation there's my twitter handle so if you want to DM me I apologize it over in a little bit of the

time for questions but there's a couple quick ones any questions we have time for like one question all right well you can always contact him on peer list one of our wonderful sponsors or on Twitter and we'll see you in a few minutes or the next talk thank you [Applause]