Defense In Depth: Designing Networks That Survive First Contact

all right so I'm going to go ahead and get started um this is a defense and death talk U mostly talking about building networks a little bit of a different take I'm a network engineer so I'm not necess you can't hear me okay I'm a network engineer so I'm a bit of a not really on the security side so much um so I'm going to be talking about building networks uh a little bit about me um like I said I'm not necessarily a security guy uh a lot of what I do uh I try to incorporate security into it but I'm not a security professional in and of itself uh my name is Jason um I'm the

senior network engineer for Lafayette College in Pennsylvania um been there about four a little over four years now um my first talk was actually uh last month at uh derbycon so um this was kind of of a on a whim thing Deron I went to in 2011 was uh quite interesting and kind of just decided I was going to talk um help me get a little bit more uh uh faith in myself if you will uh forces me to learn some stuff so um I'm here to learn like everybody else so let's start off with uh a little bit of a story so back in the day uh now I'm one of those pre-1986 people so uh we had

networks that weren't connected to anything cuz the internet didn't really I mean it existed but nobody was on it so you'd have just your your normal uh ethernet style networks uh back then it was token ring or deck net or you know whatever other Technologies they were using and it was all self-contained nobody needs to worry about anything here uh hackers weren't really able to get into the system you might have had a modem to dial up to something but typically it was all closed uh and then um something happened uh this thing came out called the internet and um so we connected to it because we decided it was a great way to get all this

connectivity uh to everybody else in the world um would revolutionize business and it has um but everybody else connected too so all of the attackers decided that they could now get into your network and they did um this is a bit of a problem so we responded and they responded and we responded and uh and the race was on so we started going in with uh firewalls and routers and DPI and wafts and traffic shaping and and it got more and more and more expensive so this has been going on for years and for networks that have existed for long periods of time a lot of this stuff is bolted on so nobody went in and

said if we started off with nothing how would we build this a lot of people have just said wait we need a a firewall here and just throwing it in front of it and the firewall is not doing enough so let's throw an IDs there and well that didn't work because now I have to watch it so let's use an IPS instead uh and then you know that's not working so now we have waffs and DLP and and all this other good stuff so what would happen if we started with a clean slate What could we do differently to build our networks um in such a way that maybe they were secure from the start

are there any design principles we can use um is there anything Earth shattering here that we can we can just look at our our you know how the networks are built before rebuild them now and then just have something secure from the start without having to get into all of the expensive toys um so that's what this talk is uh so we're going to start off with some basic principles um any network uh or any decent sized Network I mean we're talking a little bit more than your your home here um any decent sized Network should have some form of redundancy and resiliency um we want to make sure that this network that we're building stays up because if it

goes offline we've kind of defeated the purpose of having it um one thing we're going to talk about is Network segmentation when it's usable uh how deep we should we go with network segmentation uh the principle of lease privilege um and then our monitoring security um these are all related principles um in one way or another they're all going to get us to that end goal and for the most part all of this with just generic networking gear is already available um and then anything else beyond that that you want to put in here is usually available as open source um those fancy toys come in handy but you need to make sure that you need them

and you know what they're for and and how to put them into your network before you start throwing money everywhere so if we start off with redundancy and resiliency um redundancy is is basically duplication of uh the components the critical components that you're dealing with in a network or in uh a car or you know any any type of system um the idea is that if one of them breaks you have another one to take over uh resiliency on the other hand is building a system in such a way that it stays running even though something failed you still want that system to be able to continue running they go hand in hand one doesn't mean the other um

they're both necessary so if we start off with a system like this um it's pretty simple you've got a single connection to the internet you've got a single connection to your internal Network lots of point of failures um the least least expensive method of starting to get to where you're more redundant more resilient is just add another connection and you can do that on the outside and you can do it on the inside so now you have connections on the outside and inside um but you still have those single points of failure and there's a lot of them um I mean just from the the image you can see you have the router in the middle um that goes

away you've lost everything uh but there's more than that so these links that you have that go to the Internet or the links that even go to your internal Network are they connected to are they in the the so for your internet link so did they go to the same provider um are they in the same did they take the same path out of the building I mean if a back ho comes out front and hits that line are you done um these are all things that you need to think about um you know the glaring piece here is the router so of course you can just add another one routers are depending on what you're getting is expensive

um but depending on what your business is it may be worth it so you have to take into account all of your business processes and everything else so this gets us further down the road to a a redundant system um you still have other issues these routers are they in the same location um if your building burns down how are you going to survive I mean these you put a router in a different location how do you get between the two buildings um let's see uh routing uh routing through the network itself you've got different types of routing protocols um how do we deal with uh making sure that the traffic is going to the correct

location um how do we we uh how do we deal with the security of those protocols Etc um and this is only our core Network infrastructure here so you still have all of that internal Network stuff to deal with you can extend your redundancy but it gives you usually a limit so once you start pushing out towards the server end the desktop end you get to the point where it's going to get really expensive really fast I mean you can make a a desktop machine that's pretty resilient quite redundant but you're talking about multiple uh multiple hard drives multiple ethernet Jacks um I mean you've got you're going to have to run some sort of uh High availability protocol on

it um you know there's there's a lot that can be done but there is a a breaking point at which it's just too expensive to go on so um this would get us at least out to the desktop Port of part of it in a in a fairly redundant way um on the server side it's a little bit different I mean those are services that you want to keep up and running for long periods of time um so at that point you you do get into a little bit more expense and and servers are generally built to be more resilient um so one thing I failed to mention is that this is a uh intended to be a an

interactive talk so uh if you have thoughts on ideas as we go through here um start shat them out um so what other areas do we have that we can look at for redundancy resiliency Etc um any suggestions any any thoughts software side itself the software side itself okay so your application so if you're dealing with uh say a web application um you can do some sort of high availability uh maybe a proxy in front of it caching system that sort of that sort of uh setup um databases and whatnot they have high availability uh setups for that um some things that people don't think about um you know this past uh this past week was

a little rough with the uh the hurricane um so some of the stuff that that came up for us uh that we were looking at is is uh power power is a kind of important for all of these electronic devices that we have so how do you deal with uh redundant power um number of years ago I worked for an ISP uh who did pretty good with the the power they had their generators they had their UPS's all was great and wonderful went through a point in time uh pretty rough winter uh knocked out the power and the generators wouldn't start full gas generators had been tested run whole nine yards turns out the the fuel lines were frozen

so how do you deal with uh situations like that where you have you have everything in place everything works it's tested on a fairly regular basis but suddenly you have like a unique weather condition where the temperature drops and it's frozen right so these are all things that need to be thought about um the uh see the Fukushima accident do we know why the uh the backup generators weren't running there because because the generators were too low and the water and all that other good stuff was going on so everything was in place but there were there were deficiencies that prevented it from happening so uh looking at you know depending on what your business is and which direction you

need you know how expensive it is to be offline you may have to go those extra couple of miles to get that that extra redundancy in the system uh Network segmentation so um traditionally it was take all of your users throw them on a a network segment throw your servers over there on a network segment and then connect the internet middle and off you went um as we've we've learned since then especially with uh adding security into this you want to start segmenting your network into different areas now uh for our particular um Network we try to segment into um on the user side we're segmenting into uh user types um there's a lot of different ways to do this um a

lot of different technologies that that are used to do this uh we happen to use uh Knack and 802.1x for this um it used to be that you would actually go down and program the the ethernet Jack itself um those days are happily long gone um so one of the uh one of the reasons you do this is so that you don't have problems where your salespeople can get into your engineering Network or your your marketing people think they're they're great and they start getting into the tech support stuff um and you can do uh Security based on these these different roles so what we have is you plug your your system in you authenticate to the

network you're put into whatever role you're in and then you have a basic level of access control based on what network you're in and we do the same thing on the server side so we have uh different areas for um our web servers Bas or our our DMZ or you know security servers management Etc um everything is currently segmented on uh access and roll um but that's you know pretty simple change so when you're doing Network segmentation there's a number of questions that you can ask um to identify where uh systems need to to to live um you know where where does that where does that system belong um is it a web server should it belong in the web

cluster maybe you're dealing with some sort of web service that's internal only well if that's the case you have a cluster of web servers over here that are for outside access all your internet users and whatnot are there uh maybe you don't want that internal system there maybe you should have a different network segment for that um and that's you know also identifies what sort of access it needs so uh we have a number of servers that don't need internet access so those go into rfc1918 space which is just all your private addresses so without Nat they don't go anywhere um and then how do you decide where it goes well at the end of the

day if you don't have a location for that to go in you kind of have to decide whether it's worth the time or the effort to make a new network so if you have some sort of new service that doesn't fit into these roles that you already have you can build that new network which is is fine that you know you're given that ability but you don't want to take it too far I mean there's there's there's a limit to how many networks you really want to have to manage Um this can act as your your first line of defense um your network segmentation is I mean the intention is to have firewalls between all of your different

networks so if you craft it the right way those firewalls are going to take out the attack traffic for most of your common services so if your your desktop uh your desktops for instance um in our particular Network the firewall doesn't let anything get back to them so there is no outside traffic coming to those desktops whereas in our DMZ we may allow uh web traffic in and then that gets shuffled off to the proper location internally [Music] um which is where principle of lease privilege comes in so the idea behind principle of lease privilege is is pretty simple um you are allowed to get to the least you have the the privilege to get to only what you absolutely need

so I'm not going to let you just have full access to my database you're going to be able to get into the web server and that's it um this is normally not something that you see at the network level or at least not a lot of people talk about it at the network level this is usually a server based thing so uh I have a login to a server um I may be able to you know run a command uh as the user myself but I don't I can't become root and run that command so I don't have the power to do everything on the box or if there's something that needs to run U I may be

given access to run that but everything else that's on the system I'm not allowed to touch um you can do this at the network level um the the simple easy way to do it is just to use firewalls so if you segment your network properly build your firewall rules um you know we had the the talk previously about um the service service level firewalls or the service based firewalls um where instead of using all these core firewalls you can do localized firewalls on the systems themselves this is sort of complimentary at least the way that we're looking at it so for instance on our DMZ we allow web traffic in and that's it well our our network is set up

in such a way that those uh DMZ servers the caching servers that we have have to be able to get to the backend web servers so we just open up all of the all of the web traffic on 80 and 443 to those web servers we're not doing individual uh rules we just do a big simple rule that says yeah the web servers live here so just allow web traffic in and we do that for all of the different places that need to have you know General level access and then you can add in additional uh rules on top of that to find grain things if necessary so if you have very very sensitive uh

servers somewhere deep in the network you can put those those very fine grain rules in but otherwise you just use a general rule and then you depend on those server level firewalls IP tables or you know whatever you're running um to do the the fine grain stuff on the server itself um and is this really leaste privilege yes and no um like I said we're giving it a broad rule we're saying allow all the web traffic so it may be that I have a server on the back end that is a web server but doesn't use 80 and 443 well those ports are still going to be open in the firewall to it but we're we're

relying on pushing that firewall traffic out to the edge and letting the edge servers run it so that way the core stays nice and clean on a firewall level it's very easy to manage and then you have you know IP tables has become powerful enough to to handle that sort of thing um so this is the audience participation part again um so any other thoughts on on lease privilege networks Network wise um any other any other ideas on on what we can do

there okay so um right so you can do MAC address filtering um VLAN wise what we what we typically do VLAN wise is is that's where our our Network segmentation comes in um so we would have uh you can you can segment them out for say user networks down to um so that you have like a say a a an admin Network and then inside of that admin Network you do vlans per building so you still have a a smaller subnet that you're dealing with in all those buildings um it also gives us the ability to shut those down if we need to so if we have uh an outbreak of some sorts you know I

work at a college so if we have students that are suddenly infected with something I can go out and say shut down shut down that student Network in that building without having to affect all of the other buildings that are across campus so vlans is another way to to to handle that any other thoughts he was asking about SSH yeah what about separating by SSH based on what your I'm not sure I understand what you mean basically if someone tries to connect via SSH separating out what they're allowed to do based on what their SSH is ass okay so key based authentication on the on the

servers um monitoring is another another area of network building that some people uh not generally thought of as part of the building of the network itself um but it's still a good strong piece that you need in order to build a good Network um so I absolutely think this is part of of network building um without the monitoring piece it's I mean this is what's going to give you the insight into what's going on in your network um properly set up and and and rolled out um you're going to be able to get real-time information you will get historic information um it becomes really useful for troubleshooting when you start having problems um I said

historical data um and it's really good for security because those are the logs that we use to figure out well why did that server get hacked you know where did that attacker come from um so what can we use to monitor our Network so this is I get into some of the the tools and whatnot that are out there um one of my favorites is is rancid um is any everybody anybody here familiar with rancid so rancid is this this great tool that uh sits in the background and just goes to all of your routers switches and whatever else you've written to allow it to get into um by default it's usually uh Cisco stuff uh think get supports

Juniper uh we have it supporting some Aruba stuff um but in general it goes out and it starts pulling those configs down now it's great as a backup and that's that's useful um but what we find even more useful is that this sends us diffs every time there's a there's a change on the network so whether we made it or it was a dynamic change made by uh you know like the the 802.1x system or or you know somebody came and made a change or misconfigured something we get all the diffs on this and it happens on a constant basis so this is this is one of our are one of the tools that we find

absolutely invaluable um when you're dealing with uh uh vlans and Nat 2.1x Knack that sort of thing um this actually shows us um on occasion we can see where where vlans have flipped into networks and kind of look at it and go wait a minute that's a student building why is suddenly somebody there connecting to the faculty Network so this this helps us out a little bit um SNMP is another big one is what we use to monitor a lot of or all of the routers and and a lot of the other devices that we have um S&P has been around a long time it's really really easy to use uh you can get a lot of data

with it uh the other side of SNMP is when you have the different pieces of equipment telling you what to do telling you what's going on so traps come in um this is really handy for uh uh data centers when things start to get hot because somebody forgot to change the filters on the on the uh the air conditioning um so we've had a number of times that traps have come in and said you know hey things are a little hot in here you go running in at 8:00 at night and say wow air conditioner's off that's probably not a good thing uh ping is this thing up or not um we use this more with I mean it's great for

troubleshooting but this is this is the basic building block for any sort of monitoring system uh you know naio is is a great monitoring system and by default a host is that's how it tells if it's up can you ping it yes no when it goes down you can't ping it anymore uh so that's I mean that's one of the really simple easy tools that exists everywhere and CIS log because logging is good um CIS log is CIS log can be handful um you get kind of get careful uh depending on how much logging you're actually doing um and trying to go through those logs is can be a bear um there's a lot of really

good logging software out there um see uh stuff like grey log um Splunk um I mean you've got commercial you've got non-commercial open source stuff a lot of really good stuff out there um what do you guys use for monitoring things I mean is there anything Awesome by alien awesome OS i m okay invol right great IDs IPS thisp uh just point stuff yeah I've seen that mentioned a lot on the uh the the osac um it takes an OS yeah wire shark's a nice one um for doing real doing stuff with you sitting there um I don't know that I've ever seen wire shock automated I mean N Flow net flow yep which I thought I had in

here actually um yeah nefla is actually a really good one um you have the most of the uh ability to identify everything that's going on without having to actually capture all those packets so um N Flow is something that that we're rolling out and and really looking into solar winds solar winds as a uh well they have a whole Suite of of tools so everything from network monitoring is like a uh what is Orion yes Orion um so the security piece of this is is all of the stuff that we've built in thus far through the whole process of building the network um you've got your your' got Network segmentation you've taken all of those disparate units and

and put them into their own networks isolated their traffic um you've used firewalls to make sure that they can't get to uh those about locations that they're not supposed to so your web servers can get to your databases but your you know your desktop tech support people can't uh you've got monitoring Place uh we've built that INF fre from the start so we have a whole monitoring Suite that's that's able to to look at all of everything that's going on and Reporting back to us and we've got the logging to back it up so what else I mean you know I threw this together um this is a lot of what we do um and I'm I'm interest you know

the the previous talk that we that we had here about uh um the the system security was was really good I've got a lot of ideas on using chef and everything now so I'm looking for more ideas from from everybody so is anybody else doing other stuff out there that I have a question actually about your network since you're you're in a college which is unique um in in an office it I break my machine my boss gets flagged that's a bad thing colleges like to break other machines because it's either a joke or you know one down the line besides Vance what other things do you do to you know prevent let's say my machine or me from kind of just

running around the network infecting things or possibly breaking areas that shouldn't really be going into so um this is the first college network that I've uh worked in and they've explained to me numerous occasions that we are unique and that we actually monitor that sort of stuff um we do what we can to prevent that um so we have Central systems that are managing desktops um we want to get better at it but we have that in place now um the lab machines and whatnot are locked down um and they're they're Frozen um you I'm not on the desktop side so I'm not entirely certain how they do this but they're frozen in such a way that when you reboot the the

machine it it rebuilds it from an image so even if you've taken that machine out or installed like a whole bunch of software on it when it reboots it's all back to to uh you know whatever it was when it started um they're doing uh they're starting to do patch management push to the desktops um the it there's there's different areas where we're allowed to do things and and we want to do things and then there's then there's the students which we have no control over so we kind of isolate those over here put a big firewall over there and say don't touch anything so you you can't really go into a computer that you don't own like the

college has no dominion over those machines so we can't touch them um we do um our AO 2.1x Mac system um we don't allow like Rogue access points we you have to register everything that gets onto the network um so even down to like uh Xbox PS3 that sort of stuff like all of that has we have a system to go through and and check on all of that make sure that that's that's what it is you have a wait to prevent somebody from duplicating and just um yes and I mean not really um there are I listen on the traffic and I I see a mest that actually talks to the traffic I know they're registered all I

have to do is knock them off and go in get right the only place that I've really seen any of that is more on the wireless side I mean the um the switches are you know you have to you can sniff broadcast traffic so I mean you can you can grab something that's on your local subnet but of course if you're sniffing on your local subnet you're already in that Network so I just I sniff somebody else's Mac address that's somebody who's already on that Network um we're not allow allowing broadcasts through to other vlans so if if you're on the student Network you're not going to see traffic from the you know the The

Faculty Network um I mean there's there's ways to get their Mac addresses and I'm not saying that we're you know there's any any way to prevent that at this point um so we don't have a solution for that um we're trying to get better at it but um it's not an easy it's not an easy problem I don't know I I personally don't know of a solution for it I'd like to find one but um fortunately we haven't had a lot of problems with it um we've never really had to our knowledge had an issue where that's happened um we have people who end up uh you know you'll have a a machine that they're building that they

put out there and then um you know the person who's in charge of the machine is supposed to come over and log into it make sure that it's all set up and everything and somebody else gets to it first and logs in and now it's on the wrong Network so that happens all the time but um again those are those we usually figure that out pretty quick cuz they end up on the student Network and can't do anything thing with 821 on the wired side you're do2x um we're we're starting to do 802.1x on the wired side the plan is to roll it out on wireless as well um this is an area of active research for us if

you will I we're still learning some of the pieces of this but so you're looking at RFC 3580 tunnel attributes or layer to ACL um believe it's the former this is you know on the it's it's still very new so I don't know a whole lot about it um I know from what I'm told on the the 802.1 XL on the wireless um what they're they're able to do for us on the wireless side is exactly the same thing that we're doing on the wired so once once it identifies the user even though it's the same SSID it's it VL lands everybody apart so they're going to have a they're actually going to have a

student role there or a faculty Etc uh what about like application lelund for example if someone takes down service so we're actually working on that now we have a um we're we're almost we're almost fully virtualized at this point so um taking down a service and being able to spin another one up is is pretty quick right now um um we don't have anything that's automatically doing it but we actually have uh the on the system side the guys are looking at a way to to identify a service that has gone down and and bring the bring it up over here and then actually for because depending on how it's taken down I mean if it's a some

sort of a Dos attack um on that server you know you can bring it up over here the traffic's just going to move you have the same IP address so they're they're looking at how can we dynamically move IPS and and change things like that

right so um I don't have anything I mean oik depending on what it is oik can block some of this for us automat detect and block it based on the the rulle the uh the scripts that are running so if it detects enough traffic or uh whatever whatever pattern that we're using to detect that we can actually have that feedback into a rule or into a uh a program that will go out and block it on the routers or block it on the switches um it's a little bit more advanced than than we're dealing with now but it's called oik o it's a host-based intrusion detection tool all right or it's kind of a hybrid IDs IPS so it's it's not I

mean when you talk about an IPS or at least the the marketing speak for most ipss that I've heard is that it will stop it in transit and that traffic will never get to your server so in that respect it doesn't work that way because the traffic has to have gotten there and created logs or you know whatever it's looking at to identify it um but it's usually within depending on what the severity is it's usually within one or two uh packets that it actually is able to block it curi do you know any good open source IPS IPS o o oh open source ipss um snorts the one that usually pops up all right did you have a

anyone else so um in the beginning I talked about um all the the fancy toys that we have now the dlps the the uh um you know the waffs the DPI stuff um this is the end of the whole um building exercise so unless you have a need to put this sort of thing in immediately based on you know some sort of a uh requirement that you're under um this is the sort of thing that you look at at the very end um you know like I said a lot of this stuff is not necessary it's it's buzzword that are thrown out there they don't necessarily do what they're supposed to they're usually very very

expensive um so you have to look at these with a critical eye and and kind of identify is there a better way to do this um for us for uh a lot of these different tools we end up using open- Source um tools for this whether it's uh something like mod security for you know like a a web application firewall or oik for host host based intrusion detection um you know there's there's parallels on both sides um so you know I kind of caution people against you know getting all this stuff when I when I started um I started after they had done a complete forklift build rebuild in network um where I was and as part of that that

rebuild they got every fancy toy that was out there um and since then we've basically removed all of them because they don't do what they're supposed to or uh in a lot of cases it does what it's supposed to but it's you you you end up spending most of your time trying to manage that device um ids's are really really noisy and especially when you're dealing with a small team there's there's not enough time to deal with those so um these are these are things that are kind of neat to look at and you can get ideas from and and I would recommend looking at them and identifying what they do and seeing how they fit in your network and look and

see if there's some better way to do that um there's no Silver Bullet there's a lot of different ways to do things but if you build from uh the start to be secure and to segment things out and and put all of that in place it works out a lot better so that's pretty much all I have um this slide deck will be up on my site later today um as soon as I get myself some internet access um any questions comments you throw things at me Etc is do you have any suggestions for like building a Virtual Lab to practice stu um sort of so if you're going to build if you're going to most of what I deal

with is networking so on the networking side virtual Labs become sort of a problem when you're trying to figure out the networking piece of it there's a decent piece of software open source software called quag used to be called zebra you can actually do it's a basically build a router out of a Linux box um so if you built yourself a couple of those nodes and then you can actually use vlans depending on what what um what virtualization you using I'm sorry what was the software called quag it's qu a GGA QA right um it gives you a Cisco like interface um we've used we've used it for bgp um I I think it supports some

basic routing protocols rip it might do does it do OPF Yeah Yeah it's gotten a little bit more advanced since the last time I used it anything else one one last one primarily around your monitoring um I'm not familiar with the state of Delaware regarding uh dual party consent from deny which requires full party consent and monitoring of network traffic as AED ational institution uh do you all have any U as far as your choices for Technologies employed do you all have any concerns regarding privacy of students those lines we don't any hindrances we we don't have we don't worry about it generally um because we're not doing packet captures on a on a full-time

basis so I'll pull I'll pull cap packet captures when I need to but not not on a a normal dayto day that's we're we're expecting to be able to use netf flow for that um and from what I understand and the way that that net flow works because you're not actually seeing the the content of what's I mean I can see that you went to Playboy I can probably guess what you did but I don't have the exact uh packets to tell you what what what that transaction was that's covered you know we don't have to worry about it on that flow more than anything else than to mitigate something like that right right right and then we're we're also a

private school so um B basically when they come in it's you know we don't have to yeah it's it's kind of a consent thing like we don't look at your mail we can look at your mail but you know I really don't care father's right any other questions so thank you um around I'll be here tomorrow as well anybody wants to chat so thanks [Applause]

go