BSides RDU 2022

foreign

[Music]

[Music] I think so um we'll wait a second for some more people to come in all right cool so first off uh let me just make sure that we are live on the interwebs looks like we're good so uh welcome to uh b-sides RDU 2022 hacktoberfest um first uh I'm sure you've noticed uh your badge here um we uh we try to come up with a new design every year and actually like the rest of my core team to come up here as well um and I'm gonna actually let bass quickly talk about um the badge design because she definitely so let me just uh base 16 always sits at registration so she never attends Opening Ceremonies and

she said to me I always miss that stuff so if you're coming in and you're waiting on t-shirts she'll get back to you as soon as possible but she wanted to talk about the badge design it's been a joint effort between multiple people on our team and outside uh people as well so Bass right cool thank you so yeah uh I've designed the badge the last couple of years you probably remember a couple years ago we had the dumpster fire with the uh trash Panda like the raccoon and the animation and all that so I did all the pixel art for that and this year we're like oh let's do a theme hacktoberfest it's like

okay what kind of uh imagery and stuff can we do for that and I kind of give as I'm like lying in bed like right before I fall asleep and I had this fantastic idea and um I totally meant to do my hair up and braids to this is a full effect but like it was it was a little bit obvious um coming up with a badge design uh this year so I think it's another like pretty good design and represents conference pretty well um and hopefully a lot of people can see themselves in the kind of faceless figures uh hacker there so welcome awesome so yeah base always comes up with great ideas and we try to pull our community

too to see like hey what um what theme should be should our conference be this year um and we decided on hacktoberfest there was a lot of great ideas out there um but uh we ended up on hacktoberfest so welcome to hacktoberfest um this here is our link tree you'll see on the back of your badges um they all have a QR code this is where all our social media is so um there's our Twitter our LinkedIn our Facebook um YouTube um I think there's a couple other things on there oh we're in hacker tracker so if you have that app um you can actually link to us uh from the link tree um and then of course our website so all

of our schedule uh where things are who our sponsors are what our principles are they're all on our website and you can get to that website either at besides rdu.org or you can go to our link tree and it's linked in there one of the things about besides RDU and be sexist in general is that we're non-profit this conference here is free so we um we really rely on our sponsors this year we had four sponsors and without them really this wouldn't be possible so we especially want to thank Jupiter one velocity from Stern security uh Intel security and mend please make sure that if you're here in person that you stop at their booths talk you know

talk with them see what they have each sponsor an event or you know a little village that we have here we'll also have some fake money that fake money you will use for our silent auction that we do at the end of the night for different prizes and things that we have here so make sure you spend some time at The Villages as well as with our speakers here as well so um this year we have several different Villages and events uh we have the CTF by eversec um we love them they're here you know every year so super make sure you guys stop by and at least say hi to them they're going to be on the second floor

and whoops I accidentally touched something up here um and then we also have lockpick Village through uh ocl Oak City Lock sport they are also an awesome Community sponsor of us and uh really help uh you know bring besides together as well this year we're also having a career village with Holly Harrison um so please make sure to stop by that Village over there if you have a resume with you if you want to talk interviews if you're interested in getting into security um that's a good uh Village for you to stop and talk with the folks over there this year we also have a workshop the software spelunking Workshop which will be taking place on the third floor for

those that signed up it is sold out so um if you if you've already you know you may have already missed it but um special thanks to that team uh Ben uh demek and the infinity State uh people that will be here with him helping and then the last but not least we'll also have hacker Jeopardy uh and Patrick uh McNeil will be uh doing that uh as he has in years previously so this is going to be an awesome event one more thing too I don't I forgot to put it up here but don't forget we also have an after party at ponysaurus so after the event today we'll be moving over to ponysaurus where we'll have uh

free beer and pizza and um there are some allergic to Pizza there will be some options there as well foreign so one thing about b-sides is that we do have a code of conduct you can read all of our code of conduct here the first one is probably the most important if you're under the age of 13 you can just change that word out to jerk don't be a jerk but basically listen like this is a community event we're put on everybody here is um you know part of our community or wants to be a part of our community so we want to make sure that we Foster an environment where that is good for us to

communicate you know all the things related to security or things that are coming in security and just allowing new people to come in and I know if you've been at any of the conferences before or you know me personally you know that I'm a big proponent of trying to get more people into security especially newbies so make sure if you see somebody you haven't seen before around here welcome them to our community because we need more of everybody here so we have a lot of people to thank obviously we have a great line of speakers uh today um that will be here presenting great information um and we actually had a lot of people so submitted to talk to this and we had

to uh you know kind of dwindle it down to the numbers that we have now um but every talk that um will be up here today we all went through on our core team on our staff and uh really wanted to make sure that we had a diverse um a presentation for you guys today that breaks down you know different areas within security and from you know different uh walks of life so um thank you guys for being here uh for those that are on the internet that are also joining us virtually um we appreciate you guys you guys are our community so that's important this next group here is our core team and I just want to kind of quickly

introduce them because I know I have a short amount of time up here but these folks here and some that are up there and others that are not here at the conference today have spent hours and hours and hours and have most of us have been here for multiple years um just dedicating our time after work on Mondays for an hour sometimes two hours every Monday once a month sometimes every Monday depending on you know how close we are to the conference and having conversations outside of those meetings with people to try to get people interested in security and b-sides and helping us out and stuff like that so at down at the end we have

Chuck I'm sure some of you guys have seen them around the security Community he's been helping out with our speakers then we have Anthony next to him who's been helping out in a lot of different areas but helping us manage our like food trucks and different events that we have going on next to him we have my husband Phil so hey Phil he helps with our website and pretty much like um helps with everything when it comes to our logo making sure we have the right formats and the right uh resolution and the right colors and every time I'm like hey no that doesn't look right I need you to do this or do that or change the website or the

wordings bad or whatever he's there to to help me so he always comes in in the clutch and then James who's been around a while he helps with a lot of different things swag sponsors and all the prizes he's the guy that holds our money so we to some degree so we uh so we have to ask him to to buy the things we need um and then Tyler Tyler is really like coming the last uh uh bit here uh we needed somebody to help with uh volunteers because um our guys that were uh handling that previously um weren't going to be able to make it this weekend so Tyler stepped up and he's like I will help and uh he's really

like taking it on so I really appreciate that and then we have base 16 I'm sure a lot of you guys have seen her know she's been in the security Community for many years now helps with dc919 actually helps run it um kakalaki Khan all the different things so all of these guys are on my team I'm F noob I'm F Noob and um I just help coordinate things but essentially this is our team we have more that are not here we have Omar up here and Cyrus who are also in charge of audio and video so if things go wrong please don't complain to them though just just tell me I'd be all right I could take it yeah thank you

guys for coming uh there's many more that aren't listed here but um anyways thank you guys [Applause]

all right so I actually need to pull up my web the website real quick not my website our website all of our websites right um

is this microphone okay this will work all right so our first speaker is Jeffrey Lang he is the director of cyber defense operations at Virginia Tech um Jeff Lang has been with the Virginia Tech security office since August of 2012 and brings with him 16 years of I.T experience before joining the team Jeff was Computing technology manager for the Virginia Tech School of Architecture he has experience with network monitoring and forensics intrusion detection and configuring security appliances Jeff has a ba in Philosophy from UNC Greensboro and a masters of Information Technology from Virginia Tech he is a sand certified intrusion analysis Windows Security administrator Network forensic analysis python coder cyber threat intelligent intelligence analysts and certified Enterprise

Defender so Jeff is going to be doing a talk today on ransomware I just had to and I'm gonna let him take over now thank you guys good morning uh thank you for having me today I appreciate the opportunity to uh talk a little bit um we're going to talk today a little bit about an incident that happened at Virginia Tech last year we've had some departments uh get hit by the cassaya ransomware uh and uh we've been actually very open what happened uh the steps we walked through and the things that we kind of learned from it and so hopefully uh this will share some information with you and uh be a good talk

so ah thank you so again I'm I'm Jeff Lang I'm yeah had the introduction there uh the responsibilities that we have uh with our defensive operations are our security operations network monitoring incident response uh we've got computer forensics and we deploy all of our security tools maintain them uh for our monitoring operations uh my email is there if you have any questions feel free to shoot me an email I'm always happy to talk and share the information that we've got so just as kind of a background about Virginia Tech our main campus is in Blacksburg Virginia which is in Southwest Virginia and the beautiful mountains um we do have a large presence in Northern Virginia though and we're

actually building a uh an innovation campus there it's under construction now and that's in Alexandria Virginia uh when I took the last statistics we had about 30 000 students enrollment is actually up so that's probably closer to 42 000. uh and we're ranked number 48 in research uh institutions in the United States um you know nine colleges in our graduate school 110 170 programs or fairly large research one University uh but our I.T infrastructure is very distributed we have a few services that we provide centrally things like networking email our courseware various uh infrastructure mostly is all provided centrally our security office provides services centrally as well but most of the Hands-On I.T stuff is done at the

distributed level every department has their own set of I.T staff some of them are very competent very large staff some departments are much smaller and have a smaller set so we have a lot of challenges that we go through working with different levels of people different technical expertise and different availability with them and so those provide a lot of interesting challenges especially uh and this you know as you go through and you talk about best practices for security and you get a lot of uh gasps when people say things like never have RDP open to the internet never have SSH open well many of the research one institutions are open we have no network firewalls

except in limited places security is at the host level and that provides a lot of challenges as well we don't have that centralized management of things to try and control and do a lot of the security layer on kind of that outside edge our security office we have 10 people total um four people on our defense team including myself we've got our architecture red team our risk management team we're working on getting a 24x7 security operations center up so we have an associate director who's kind of working with that and then we also have added a software developer to try and help with the projects that we have going on uh so you know that's 10 people total

across the board doing all the security operations uh for the University itself so what ended up happening uh last year uh Friday about lunchtime uh right before the Fourth of July holiday weekend everybody was ready to have Monday off and ready to go and people started coming back from lunch and opened their computers and had these weird files on their desktop they clicked on them and they were encrypted and there were text files saying hey you've been hit by ransomware you need to go to this site to uh to check out how much it's going to cost to get your data back um we started getting some phone calls in at the security office from one of our

departments and they were like hey we manage this thing called caseya VSA it's a you know Security Administration software package it allows you to push patches install software remote manage and they were like you know we've got all these people who are connected to this device and they're all getting hit by this ransomware so it turned out there were some flaws in that application from cassaya and there were a number of people hit by this so we started scrambling to try and figure out what was going on uh and what was happening and it turns out uh that there was a group that you'd sodino kiwi and attacked all the machines we had on our

Network that were connected to the VSA server uh it turns out it wasn't just us it wasn't a targeted attack at us it was targeted at cassaya the VSA Appliance so this was a supply chain attack they didn't attack us directly but they attacked a vendor that we worked with and the primary targets were actually managed service providers so there were 60 msps that were impacted there were a handful of other areas like Virginia Tech that got hit that weren't really an MSP but it was over 1500 clients of those msps it was a huge attack surface and it but it only uh it only attacked their on-premise software so their software is a solution

in the cloud wasn't impacted but they did shut that down as well because just in case it was something they didn't want that to happen so kind of the details of what it was um it turns out there were seven flaws that were exploited in order to get this attack to happen um the cves are all Lister uh SQL injections cross-site scripting all sorts of you know pretty uh normal and and things to expect not to have in a management application like this um what they did is they uploaded an initial payload through a bypass and once they had that they used the SQL injection to execute that file and once they had that they had administrative

access on the entire VSA Appliance so once that was they had that footing uh they were able to push out software to every client that was managed by those VSA servers they downloaded an agent.crt file and they pushed it into the cassaya directories that uh were protected and were trusted so you know this is admin access this can install software and the operating system allows it because you know this is a trusted component uh so they actually were able to run a Powershell script that disabled Windows Defender and then downloaded um a some additional files uh they had the cert.exe which is a standard Windows executable they used it uh to decrypt an agent dot exe file so

now they had their own agent running on all the boxes that the VSA managed wrong way um so once they had that they actually used it to download vulnerable software and this was an old Microsoft anti-malware that had a side load vulnerability so they downloaded it uh it's Ms MP eng.exe that actually looks like a you know a valid file and for a long time it was a valid Windows File so if somebody happened to see that running you're not going to have any question about that you're like oh yeah that's that's the anti-malware and that's fine um they were able to sideload a dll file and then that dll file told the anti-malware to encrypt

files and to do it in such a way that you couldn't do things like Shadow recovery or rollback so basically in place you couldn't recover those at Virginia Tech it turned out we had three of these servers running only one of them was compromised it was the one that had its administrative access open to the world so 443 pretty common pretty normal but you know you question maybe why was it there and we have some questions about that as well now the deed was done and we needed to recover from it uh the department that Managed IT they also allowed six other departments to use it uh they had already invested in the infrastructure and were paying for

it and they said hey why don't you guys use this too uh so all of those departments were impacted uh 111 servers and 805 endpoints in those seven departments were encrypted all the files on the machine all the documents were encrypted also it followed any map drives any synchronization drives like uh Google drive or Microsoft OneDrive uh and we actually had multiple file servers that were encrypted so we had terabytes of data on file servers uh that were encrypted and from the endpoints themselves so what did we do um well first we shut down all of those VSA servers even the ones that did not get compromised um at the initial response we didn't know what that Vector was and so we were

like let's shut them all down we also shut down the network portals uh that were connected to them just in case a VM popped up uh accidentally we knew that no network traffic could get to it at Virginia Tech we have kind of our computer incident Response Team a cross-functional group of people we have some guiding documents I have a link for that later in here uh we activated that team and so we were able to bring a number of people together and start having conversations kind of across the university with the Departments and with additional resources that we could bring to try and discuss what our next steps were we notified all of our senior I.T

management which is part of that cert is to get out notifications so that the university knows what's going on uh we contacted the Virginia Tech police they are a full Police Department so they opened our initial police report which we then were able to report to the FBI and that was in Richmond Virginia that's our local Hub but they didn't have a lot of information because it was being actually run out of the Austin office so they forwarded some information to us as the as their regional office received it from uh from Texas and we kind of went from there uh we also opened a case with the ic3 uh submitted that so that we had kind of

our paperwork done and everything going on there uh we did open a ticket with caseya which was very important uh since this was a now a widespread event across their product they kind of took the lead on incident response we provided data to them that we had and other organizations provided data as well for them to test and to figure out everything that was going on we started having daily Zoom calls uh we had actually two on that Friday and then one Saturday and one Sunday I remember standing at the grill and then going into the house and uh sitting down in front of zoom and talking to Senior Management and the Departments to get

status updates uh from that and then to go back out and try and help finish off the Fourth of July party that we had going on um so once we kind of got our our grip on what was going on the department started to identify all those machines that were out there that they needed to basically go to and and figure out what needed to be done next with them we decided that basically we're going to restore from backup uh and then you know wipe every machine and replace it so they began that process of identifying things and giving a status with that um and once they did get that restored they started scanning for pii so any uh any confidential data

any Student Records anything like that we needed to know if anything was on those uh and then we also had to get in touch with the Department of Education or report to them so we had weekly meetings with that update them on our status throughout everything that was going on so the request for the ransom actually came through the our evil group they are pretty widespread ransomware is a service organization um they announced on their happy blog that yes we did this haha look at us we're great um the uh we received permission to negotiate and we got a number from them that was forty four thousand dollars per decryption key there was some confusion uh with them

trying to figure out exactly what that meant and ultimately it meant that every one of those machines had its own unique decryption key and it was going to be forty four thousand dollars to pay that um we did some negotiation for about 30 Keys we had some some thoughts that we might need to potentially pay some Ransom and so they I dropped that down to 700 000 for those 30 machines a few days later and you can see the announcement uh they decided that they would rather just offer out a universal decryption key for 70 million dollars and of course nobody took them up on that offer uh so as we were going through kind of the

review uh one of the Departments realized that they didn't have good backups failed and so they needed to think about paying the ransom for it and that's how we got to that uh number of 30. they said this would probably do us and we kind of added in some for some other departments in case we needed kind of that over overhead so we reached out to our cyber Risk insurance which at that point we realized we should have done that at the very beginning uh it's one of our lessons learned that we'll talk about in just a little bit but they would have been able to negotiate on our behalf they have people who that's kind of their job so we

potentially could have gotten you know a much better deal on that but fortunately we found out that we didn't actually have any systems that needed to be recovered from the ransomware using a decrypter uh so the the department that was running it they had their VSA server running on a hyper-v uh VM on a Windows server that was itself managed by the VSA so when the encryption started happening it started happening on those windows hosts that were hosting those VMS and they basically ate themselves so it shut down all the VMS shut down the servers and when they were able to actually get at those VMS they realized that they had encrypted the data was still on them and

it hadn't even downloaded the agents to those VMS so they were able to go through a process and pull that data and not have to do the ransom so fortunately we decided we were done with that we stopped all communication with them and kind of went from there uh we did have a tabletop uh in earlier this year and we had a lot of Senior Management with the university and our University president Dr Sands said uh hey any ransomware I have to approve no one else at this University can pay it uh so that was it was something good to kind of get that perspective and say oh yeah you know any decisions we might

have you have to take it to the top you can't have somewhere someone even the CIO of the University make that decision it had to go all the way to the top um as far as decryption keys go uh so uh our evil and their infrastructure disappeared on July 13th they fell off the face of the Earth no one really knows what happened well someone probably knows what happened but we don't know what happened um on the 22nd of July so this was 20 days later casaya announced that they had a universal decryption key uh so that was great we signed an NDA we actually had a our library had a digital live of scanned images we have a big

architecture program so it was a lot of architectural drawings that they had accumulated over the years it was a few terabytes worth of data they had basically decided not to recover that data they were just going to recreate it as people needed it and they were actually able to use the universal decrypter and get access back to those devices uh in September we've all found out that the FBI had had that Universal the crypto key for about a week and a half before they made provided it to cassaya so there was a lot of response time and a lot of effort that was going on that you know perhaps could have been avoided if we'd had that decrypter a little uh

sooner uh on September 7th uh the our evil group came back uh and in May of this year they started updating the uh ransomware software uh in its under current development now so they kind of went away for a little bit and then came back just as uh as annoying as before so things you know some things went really well uh as far as the response you know of course not having it would have been a better thing but it happened uh and so our cert activation those things went overall very well they allowed us to bring people together and to have communication across the Departments um one of the biggest things from the first call we had was Senior Management

uh it was never a blame game it was never fingers it was okay this has happened how do we recover how do we work together to make sure that we uh get our data back and we ensure the Integrity of the data that we have that made a huge difference because nobody then was blaming someone else so we were really sharing everything there was no impetus to hide things that we found or you know be disingenuous about something and so that made a huge uh benefit to us overall um the departmental response was very quick and effective you know within a few weeks they had gotten most of the systems recovered that they could uh and

they were moving uh forward to get things taken care of as they could we did run into some issues uh but that you know that will generally happen we maintained our daily updates for the first two weeks and then we had a weekly call and then we had you know once a month calls and then finally we did get it finished out so that was good everybody attended everyone was there from all sorts of different we had University legal represented our Risk Management Group we had all sorts of people involved um we also during the initial event we had a lot of response from our Central I.T although we it was in a department that this was

going on uh they supported all the work that needed to be done uh cutting those Network portals uh we had some centralized backups that they immediately changed the data retention periods so that we could roll back and we wouldn't overwrite dials on Central Storage uh they even provided physical Hardware so uh the initial group that managed the VSA server they had you know they needed a large box that was able to host uh those VMS so that they could see what kind of damage was done so our Enterprise systems had a box that would do that and so they lent it to them so that they could stage uh the way that they got things going

um and then we also have some staff that supports the division of I.T internally for desktop support but they also handle things for our VIP clients their services were offered and they helped reformat reinstall and re-image machines to try and get people back up and running we also had pretty good Network forensics we immediately started looking to try and find out so hey is this an encryption only event or is there also data exfiltration response is a very different uh different based on which one that is we determined that we did not believe that there was any data exfiltration going on and hasaya later confirmed that that this was an encrypt only attack and it was not a data

exfiltration so that worked pretty well uh but not everything was good uh and again uh that first thing you know why was that uh uh administrative interface uh exposed to the internet um there were some needs for some of the VSA that would be exposed to the internet um but perhaps they had a little too much so you know that was kind of key piece uh that's kind of got the ball rolling the other thing our business contacts and our cert document had gotten stale how many of you have ever put together a big document list a whole bunch of people's names only to find out later that they've all left the company and now you

don't know who to contact well that that happened with us it caused us to miss out on some important communication uh kind of the risk management and some of the other areas the university risk management also we didn't have any contact with the university Emergency Management they usually deal only with you know like physical emergencies a fire on campus an event that happens um you know they have a lot of resources though and they also you know we're bridging between you know things that happen physically and in the virtual world that are all linked together and so they didn't get contacted which caused some delays in them reporting the incident to the Commonwealth Emergency Management Group

um we also and kind of one of the big Keys is we didn't know where our high risk data was at we didn't know what had pii or student data um you know the Departments hadn't really kept tabs on a lot of stuff we had terabytes of file storage that had spreadsheets that went back 20 years and so we had to treat every device every machine almost a thousand machines as having high risk data on them so everything had to be scanned everything had to be done and that took months to do uh imagine trying to scan with a regex across three and a half terabytes of data it it's painful to say the least so

that that was a huge issue uh that we just didn't know what data we had and where it was we also uh had this little thing called move-in day uh come August and that 37 000 students showed up all at once and all the faculty and staff to support them all needed updates uh so uh for that start of the semester things got pushed as far as some of those scans went and it really delayed us getting the resolution finished um the Department of Education had recently made some changes about reporting deadlines we actually had 72 hours to report the incident from the moment we had any thought that something was going on um we barely made it in on Sunday

morning with the with that information to them because that group was one of them that had changed the leadership and we didn't have the contact information and so when we finally got it we were already about 48 hours into that period so there are a lot of things that just kind of just didn't happen the right way um we also so we asked them what machines are impacted by this and they're like I have no idea we're like well why not it's like well VSA Managed IT so the inventory of all the machines that were supported by VSA was VSA and now it was gone so they didn't have any reports they had nothing that they could go back to and

say hey these are our devices this is what we have and this is what we need to do so now again they had to go to Every computer that they owned in their departments and say was this a VSA managed machine or not did it get encrypted or did it not so that delayed things a lot I already kind of mentioned the size of some of the files even though the library got the decryption Universal decryption key they didn't have three and a half more terabytes of storage to do a decryption because in fact you need another three and a half terabytes of storage because you don't ever want to run that weird encrypt decryption key across your original file

because if it messes that file up that was your one chance at it so you want to make a copy of that file you want to decrypt that and if everything goes right that's great now you have the decrypted file so they actually had to do it in small batches and that just took time for them to run through that and get that process done our physical forensics we were able to support doing uh one or two forensics images at a time and now we had almost a thousand machines looking at doing forensics for so we weren't able to do that we basically just said bring us an example of every type of machine that

was uh compromised and encrypted so we had a copy of a server a copy of a desktop and like a copy of a laptop that was the best we could do fortunately we didn't need those forensics images but uh it would have been great to have them so what you know what recommendations kind of came out of this we had you know a lot of things worked well but a lot of things that kind of were gotchas for us um so immediately it's you know the minimum thing is if you've got an admin server of that you know that kind of admin privileges across your organization make sure that you really do have it protected and you don't leave

it open um for us it should have been restricted at least to on campus and anybody who needed to access it remotely uh that would you know use the VPN come through with that we also recommended now that we need to take all those sorts of systems and do security reviews for them make sure that they're doing the best practices and that they're protecting logging and monitoring uh additionally with those services so that we know that they're as protected as we can make them our business continuity I mean we we made we made it through but our business continuities in different departments they couldn't cover from some of that because they didn't have servers uh we

had they had to borrow servers from the central I.T they have the storage to decrypt things so all of that should be in your business continuity how do you get yourself back up and running and that wasn't there so that was an issue that we'll have to deal with moving forward as each department goes through and kind of builds those that incurs extra cost and extra thought in what you do with that um so one of those cross-functional groups was our Purchasing Office and they were there and they were looking at it and they were like so you know they pulled the paperwork and and we did a Security review of the of cassaya and

the VSA when it was first purchased but they'd had that thing for seven eight years now had done upgrades along the way and there was had never been another Security review done on it and so it made us start thinking that you know maybe at renewal time you need to at least get a certification back from the organizations that they're doing the things that they said they were you know they said they did it at the beginning but who knows what's changed and what's happened at that organization um we were also going through a major major uh incident I.T wide major incident process review at the time they were formalizing it in servicenow which is

our trouble ticket system uh and it basically deals with how you communicate uh to your your stakeholders the University at large when you have something like email going down or canvas the course management that's having issues and we needed to make a security side into that because that would have automated all of those initial Communications that we needed we wouldn't have missed people because we would have already formalized it had it and had review uh within there and we now have done that and hopefully anything moving forward needs to or would go through that process um we also want to start conducting some tabletop exercises fortunately everybody came together and worked really well but what we need to

do is make sure that when we bring all those people in they know their roles they know their responsibilities and what needs to happen and take place and so we can focus these table tops around things that we know are likely and some things that are unlikely that could happen and just to make sure that everybody knows what they're doing uh and you know what's going on with them so we finally got everything kind of taken care of all the machines that were encrypted were wiped completely and reinstalled there were no root kits there was nothing that caused us to have to throw away any hardware or hard drives so we were able to wipe and

rebuild all of them we rolled back all the data on the share drives to a clean State and then at where they could they did those decryptions we did have some interesting Adventures trying to roll back Google Drive and Microsoft OneDrive we had to have an engagement with Microsoft in order to do that because otherwise we would have had to have done it one OneDrive account at a time so Microsoft partnered with us helped us out with that but it still took a long time to do uh like I said we had to treat everything as high risk so we were scanning all those shared drives those terabytes of data for pii and for Student Records

that took until December and that was just it dragged out and dragged out and dragged out and finally in January of 2022 we provided the last data that was needed by the Department of Education and we basically closed out the incident at that point at the end of it there were over 129 000 uh Student Records that needed to be flagged in our system and that basically was saying that there was an event that happened with their account but there was no data exfiltrated out of it that's a requirement by the Department of Education and most of those were students who hadn't been with the university for 10 15 years because it was all old Excel

spreadsheets now everything like that is tracked you know in systems and data is encrypted in databases but back in the day all we used was Excel so we had digital cruft that just lasted and lasted and caused us to have a lot of these headaches and a lot of these issues going forward and now so you know say oh yeah the event's done but it isn't really done um our Board of uh Board of visitors yes our bov uh was very concerned of course uh at the end of this incident and decided to engage Deloitte to come and do a across the board university-wide review of security practices uh and in different areas not just the security

office but within the Departments and how they treat data and what they do and how they go through it they spent a couple months going through that putting together a lot of recommendations for us um and then kind of came up with forget at least six it related recommendations there were some management related recommendations but with the idea to try and strengthen security and make sure that we have all the communication that we need together it actually got us uh three uh positions at the security office that we're now hiring for um and then one position uh to work with endpoint management that uh is was one of the recommendations to improve uh what our endpoint management solution

was at the time it was left up to the Departments to roll whatever they thought was appropriate now we're kind of moving into something that's more centralized and can be managed in the division level at the division of I.T and used by everyone we're also in the middle of an internal audit of course the Auditors are taking a look at you know not just the processes that we have but they're looking at our all these results and all these recommendations and things from the cassaya incident and saying did you do what you were supposed to do how did you compare to what you say you're going to do and that's a good thing because it ensures that we're

following the procedures it ensures that we're looking at them of course and making sure that they're appropriate for what's going on one of the big things that kind of came was that you know we had that stale data we had a document that was supposed to be a living document uh that just sat and collected dust for four years without anybody looking at it so things like that are what we're focusing on uh trying to move get that moving forward putting in some new security standards on the high-risk data identifying and inventorying what that high risk data and those high-risk machines are within each department in each area and it's going to take some time to get there but

we definitely have a lot of steps there a lot of this were things that um we wanted to do things that had been uh kind of you know hit on an internal audits an external audits that we had before but it always comes down to money right how do you afford to do these things and because of the severity of this incident um the board said hey okay we're tired of money as an excuse we are going to fund these projects and we want results and we want them now and so we're in this process of transformation there is a link up here it's open to the public any information that's there you can read and see kind of the things that we

have going on um and that's really what I have uh for you today I don't know uh if there's if anybody has questions uh whether we have some time now uh to do that or afterwards in the hallway I'm more The Talk and go from there okay um I saw this hand first

uh so the question is was this uh preventable from the Virginia Tech perspective uh had that VSA server not had its web admin console open to the world it would not have been compromised um the group in question they actually they have machines around the world they're an Outreach group that does a lot in other countries all over and basically the VSA was open so that they could communicate with all of those machines no matter where it is that weren't on our campus that weren't in our Network basically unfortunately um the web admin interface didn't need to be open to the world uh so had that been done they could have left the ports open

that just needed for that communication uh and and in that case it would not have happened because they wouldn't have been able to access it

um

uh so the question is uh with our distributed I.T are they part of the decision-making process on how things go or are they basically given the information afterwards the answer to that is really both uh there are certain decisions that are made uh that are made from an infrastructure level as far as like networking those things that they're are kind of brought down and said this is what we're doing we do have meetings twice a year with the departmental groups we're getting ready to have one next week where we talk about what's coming and what's going on but as part of this transformation every one of those projects that we're doing has representation from our

departmental I.T and the different areas so that they can provide feedback and input into the process I believe one question in the back there

into the future

so the question is are there data retention and maybe data labeling uh things that we can do moving forward to try and mitigate having the information that lasted so long there is that okay um so we do have a data retention policy there are records management policies provided through the Commonwealth of Virginia and Virginia Tech policy and they just weren't followed in many cases so that's one of the kind of education things that has to happen is to make sure that we have departments doing what they need as far as data classification uh part of one of our Point protection uh transformation projects is also to do a DLP and labeling and so we're moving

forward with uh trying to get that in place so that we do have protections on that data so even if it does stay for longer than it should and that data got you know exposed somehow that we they wouldn't be able to actually access and read it so yeah those are those are projects that are underway and going okay thank you everyone um we can talk after

you have outside the book there thank you very much it was my pleasure to speak um enjoy the rest of the conference

okay well Jeff is uh getting disconnected here I'm going to invite the panel to come on up uh for our next session uh while they're getting up here and getting situated I just wanted to provide a couple of reminders for those of you that are doing the software splunking Workshop that will be starting now uh so that will be up on the third floor yeah third floor right so um you'll want to go to the elevator and hit the B2 button B2 equates to the third Fourier balcony too so um yes if you signed up for that please head on up there also just a reminder we do have a um career Village uh just

getting set up outside as well in the lobby uh if you are looking for help with your resume or if you maybe are an experienced hiring manager and might want to help Holly out if um you have some time stop by and see if she needs some help with that but definitely that's a big piece of our our community here is helping others with their career so we're happy to have that that Village for you here as well this morning so um let me see what else do we have going on just we're also with the vendors um as you stop by and see the vendors they have some money to hand out to you

so that money will be useful at the end of the day for a silent auction that we'll have for some gifts so another encouraging uh another uh way reason why you should stop by and talk to our vendors again our sponsors are why we're here today they helped pay for this entire event make it free for us so happy to have them here so with that I think we have our panels yeah come on up on the stage here panelists so um I'm not going to introduce everybody with their bios we're going to let them introduce themselves uh but we have Diane Morris here who's going to be leading the panel uh Omar Santos uh Josh

dimbling and Lisa Bradley we had one panelists unfortunately we couldn't make it here this morning art vanyan so um yeah unfortunately was not able to get out but fortunately we have the rest of the folks here and I'm going to turn it over and let you take it from here Diane are you are you good all right

no it's something you're good to go all right good morning everybody my name is Diane Morris I work at Cisco I am a content manager on Cisco's product security incident response team so we are the folks who are in charge of putting out security advisories my team within the p13 is in charge of editing and Publishing those advisories um before I let our panelists introduce themselves I want to introduce our acronyms of the day uh so our title is s-bomb plus X plus csap equals the future of vulnerability management so s-bomb is your software bill of materials so that is a list of ingredients that is in any piece of software all the different components that go into making

a product Vex is the vulnerability exploitability exchange and that was created by the national telecommunications and Information Administration as a framework for security advisories that let manufacturers both uh say when their product is affected and when it is not affected and csap is the common security advisory framework which is a framework for creating machine readable security advisories so I hope that helps okay now I'm going to ask the panelists to introduce themselves that was Omar why don't you start I'm a senior director at Dell on the product notification security team so I have um an oversee piece cert our open source component management our s-bomb Initiative for Dell security Champion security training program and our bug

value program I'm Josh dembling I'm senior director at Intel I run the product security incident Response Team the bug Bounty program uh security working artifacts team where we look at industry Trends and and new regulations and legislation and how that changes the industry um and uh work to educate both the community as well as Insight Intel on good product security practices excellent my name is Omar Santos I'm from The Cisco music team and working with Diane along with a few others that I see over there and I'm also the chair of the common security advisory framework which is one of the acronyms in there and I'm actually the founder of besides RDU a long time ago thank you

okay so let's start with just um s-bombs and how your companies are prioritizing the development let me start so um I'm very lucky in Zell that we are prioritizing it all the way up the executive chain so we have a lot of support um as soon as the executive order went out uh we put a team together and uh figured out um where we were good and where are we behind um as bomb was one of the ones that we needed to catch up on and we put together a phase one approach where all of our EO critical products uh which was fun in itself to figure out what those were um we went and drove to get us bombs so

we utilized black duck um to uh pull together our s-bombs and our inventory for open source but even with uh having a software composition analysis tool it still took us a significant amount of time and scripting work because we wanted to have what we considered a Dell customer facing us bomb so um you know we had I think maybe one third of our product portfolio that we now have us bombs that we can give to our customers but there's still a long journey to go in uh in in this matter And to clarify for folks um the executive order that went out regarding us bombs came out uh was it earlier this year or last year late last year late

last year um that basically said that companies that are going to be selling to the federal government have to have software bills secure of materials rather with their when they uh with their products or their products yeah and I think that's a good segue I want to take a moment and step back because some of you may not even know what software Bill material means think about like the interest of anything that you consume a product right whether it's a hardware-based product you know or software that is made up of third-party components in most cases and those third-party components in many cases open source so what the industry is trying to put together is a

for a long long time but now it's been been part of you know more legal documents like the executive order but we're trying to put our heads together is what is the most effective way to know what we're running to know what we're consuming and it's an ecosystem and it's not so much of a single vendor creating an asthma you're done if not that that vendor like in my case I also consume Technologies from other vendors so it's very nested so one of the so going back to your question one of the things that you know we're looking into is not only how to produce this artifact that my customers can consume but how can we also require our vendors

to have that level of consistency for us to consume that and then accelerate that so it will be instead of you know from from a from a software composition analysis tool which is after the fact how can I put this in my development practices even from the moment that I hit commit or I even decide to use specific third party or not so so that's in an is what or in a just what what basically an s-pom on this effort is all about you're right though like a lot of it is Tory like when you're you know you're developing to making sure that you're paying attention to what you're consuming um because oftentimes we get to the end

point and then it's hard to go back and figure out what you put in there because developers leave and your build environment change so um you know how do you get it within your typical developer ready platform activities and and and know that inventory from the beginning and and have those the the composition analysis tools start earlier on in the process because you know my good buddy Ellen Friedman um uh he will say Hey You Get Enough Bomb by just pushing the button it it's really just not that easy um I mean one of our product teams took six weeks just to create the s-bomb um because of the complexity of it and then we have vendors like like Intel

here where now we're putting the pressure on to having us bomb also yeah and our journey started similar to Lisa's journey and with the EO um let me even a bit before that I mean we all knew this was coming it all started to work on this within our respective companies but the challenges are um not just um how you put it together but to what depth and to what's using what format and making sure that when we do put it together we can communicate with each other share it with each other effectively as needed right and so it's it is definitely a shift left Journey but it's got to start more on the surface right taking what you've got and

putting it together yeah and the journey doesn't stop up after you have it I mean you have to figure out how are you going to give it to your customers what customers are you going to give it to how do you make sure that you protect that information so um you know our journey is still continuing on with the US bomb because uh for for me right now unless you have um some kind of contract with me or or most likely waving some kind of big money in front of me I'm I'm not willingly to give it yet because we're still maturing our processes with it um so we'll we'll continue to grow on that journey and eventually we will open

it up to all customers but until we really understand how it's going to be used by our customers what you know are we even delivering the the right format to them um we even had a customer that asked us for a certain naming Convention of that can you imagine if all your customers ask for a different file name of the s-bomb uh that you know that that's just a lot of work so how do we educate the customer at the same time of what we're going to deliver to them uh in a standard way so a lot of activity we've had to worry about now of course one of the things about uh s balance is

that was all these uh third-party software is tends to be where most of the vulnerabilities that we have to deal with why right so um let's talk a little bit about how s-bomb connects to security advisories and um csap I guess I can probably take that one so so csaf the common security advisory framework is only one of the components um that is in this ecosystem it doesn't need an s-bomb to function basically we've been doing this for a long time in the industry most vendors hopefully right they will have a pizza team like ours and then they will disclose up only the executable libraries that they have they do that in a security advisory some

people call their bulletin some people are noticed but it's the same thing it's a historically it's a human readable document that that we publish on somebody actually we have to read and say okay my program is affected by this I have to upgrade and then you know move on I used to have a joke in my company and this has been recorded this stream I'm actually streaming it up there but I I always say that if a human is really my security advisory is I should be fired right and I still say that because we have to do this at machine speed there's so many vulnerabilities out there so many vulnerabilities that actually do not affect you and it's a way of so we

have to put our heads together on how to do this in a machine readable format a long time ago in the galaxy is far far away we were also participant in a forum called icasic that is now actually part of first we whenever say we a whole bunch of vendors including Microsoft Intel and so on and we had multilateral mbas to exchange vulnerable information and we wanted to do this in a machine speed and we created this thing called the common vulnerability repairing framework so it's basically a machine readable advisory that's what csav basically replaced now instead of in in this form that was a little bit more closed it's part of a an organization called Oasis

So within csap um you have different profiles you have the normal security advisory that we all know and then whenever it comes a little bit more pertinent to the Aspen conversation is this thing called vex so Vex which stands for the vulnerability exploitability Exchange can be a little profile within an s-bomb document and there's two major standards that the industry is focusing on right now in s-bombs spdx and cycle on the X So within an s-bomb at the moment that you create it you can actually say you know this vulnerables don't affect me or affect me or under investigation the challenge with that is that two minutes after I publish that information becomes obsolete because

that vulnerability is under investigation that may be not affected or a new vulnerability comes in right so it makes that document and that status completely obsolete after you create the s-pom where csap comes into play with that Vex profile is that you have if you think about it an automatic security advisory being generated for you basically it's a response to whatever the status of that vulnerability is in time so you can query these you know essentially with apis or any other methods in a little bit more intelligent way and as you go through the vulnerability management process from obtaining the the report of that third-party software vulnerability all the way to the disposition of fixing it or I'm not affected you

actually can query you know any system and get that status and that's what affects it's all about so hopefully that makes a little bit of sense for folks that I haven't been exposed to that yeah I think Bax is interesting Maybe our um to figure out their impact as soon as possible because our customers have our response and the last thing I want is to have 15 to 50 to 100 who knows how many customers constantly asking every time there's a new open SSL vulnerability so you know but all of that work to get that vexed to get that you know that that working it is a lot you have to figure out how do you tie it in to your

ticketing system that you have for your product security ticketing system right now for vulnerabilities how do you make that impact statement uh be done by your teams earlier on and understand the importance of why they need to do it quickly how do you let that engineer team understand that pain that's going to potentially come from our customers if we don't get this information out to them and how do you make sure it's accurate and up-to-date and consistent I mean there's a lot of work that we're having to do and there's no it's an amazing journey that we're on um but you know uh the the the pressure that I feel is is that you know there's

a lot of people talking about how quickly it could be done but you know this is a lot of process coding people uh money uh to make to make it happen it's not just about you know the technology of implementing it it's about changing the culture of a company in the way that they think in the way that they operate the way they develop products and it can affect Innovation so you have to do this in a bit of a systematic way this while it's a shiplift journey again you have to start with what you have and integrate yourself into the business very carefully so that you don't disrupt the business revenue and the higher you

are up in the food chain the the technology ecosystem the more important it is that you are on this journey because everybody that you feed down uh is going to be relying on this because like Omar pointed out there everybody's gonna get flooded with all these advisories all these reports of security vulnerabilities and they're going to need to be able to make decisions faster and faster at the speed of a machine and that's why this is so important yes matter of fact you brought an amazing Point actually both of you and I'm trying to look for things that are controversial so you don't actually agree with a thousand percent with them like a friend of mine always says

everything in a PowerPoint orchestrates and works perfectly right in practice is a lot different it's a Monumental task especially whenever you have 1200 products in my case and you guys are pretty much in the same issues yeah and getting the culture in place getting the tools in place getting um the customers educated as you mentioned it is it is a Monumental task it's not going to be like these acronyms are just you know machine readable documents uh I'm not going to diminish the work that we have done in sisa you know because it's been tremendous but you know we can come up with the specification like that you know no time putting it to work

right is significant you bring up a really good point that it's uh you have to have real support from the the company that you're with like like we all have to be able to uh be involved in involving these kinds of Technologies these methodologies uh like Omar said I mean this has been years in the making the kind of interaction collaboration that we've had for more than a decade on on getting to this point has been critical for the success of our companies but but they've allowed us the opportunity to be a part of this they've embraced this and uh without that I think it's it's hard to be successful I do think though the

executive order is as much as it's right I mean it do and have a journey for that would have took a long time now we we had a hard fact document that the company had to follow or they'll lose business and uh you know when you speak uh those kind of terms to the right people things will happen um and I I think there you know as much as it's been a lot of work I I am grateful for the executive board I'm I'm not grateful for the amount of regulatory acts that are coming down on top of that um but I'm grateful for the first step with the executive order yeah it's it's a trigger the executive

order in the US is a trigger but also has triggered other governments too so it's not only about being compliant or being able to sell to the federal government but right now Anisa I'm in some form of concept with them they're looking into s-bombs as one of the components but also even predisclosure of vulnerabilities which is a conversation for another day we can really have another three panels on that yeah so it it is going to take a lot of effort from vendors from the consumers from the government from the industry to actually get you educated because it's a face approach what Lisa mentioned about software composition analysis tool that's the first step we're in the same

shoes from pretty much every the whole cup the whole industry whether they tell you otherwise or not you know it's pretty much in the same shoes right um you you're reacting into this looking into what is in your source code or in a binary right and then there's no magic bullet that you can just press a button and say I'm gonna put this into my cicd Pipeline and then tomorrow I'm gonna have S bounce backs and you know all the acronyms in the world that is not an assist right now hopefully we'll move to that in five ten years hopefully right hopefully I'll be alive by then but uh but it's gonna take us a significant

amount of time so the call for Action is that we all get educated and we all try to actually solve this together right right because it's an ecosystem yeah um talk a little bit about why the executive order came down what is the implication for National Security with response I was going to actually say I think this has got more to do with the trend that you're seeing with government agencies and even within the industry it's it's really finding a way to push people to disclose vulnerabilities faster to be more responsive to security vulnerabilities to have processes and standards that they follow consistently and how to handle security vulnerabilities and then bring people together so that we communicate it from

vendor to vendor and I think that that I mean that was a culmination of the trigger that that launched up but there's so many other decent legislation yes I think solarwinds was one of the major major triggers that the government will say because the solo wins but there also was you know the pipeline you know um and and everything in supply chain has just always been an issue but I mean the executive order has a lot more than just s-bomb and to to be honest if if your company uh feels that just having enough mom is the answer it's they're all there's just so much more to it having you know sdl practices having you know contracts with your with

your vendors having a good vulnerability response uh program they're the the that I think everybody's been focusing on us bomb because it's a trendy name and then backs came along is another trending name so it's a lot of fun to focus on that but but there's so much more that you need to have in order to be successful in the space and even when you're one of the best in it you still can easily fail um I mean every day there's new vulnerabilities I'm sure just as we're speaking there was you know hundreds probably came out uh you know and and that that part is is there's a lot of work ahead which is great because

we're in these fields I think we could do good job security but oh we've made that before we see the solution the the even though you mentioned Alan Freeman is is a good friend of all of ours um he led an effort in the ntia of s-bombs and even before him a lot of companies that's the reason that black box and the white source and a whole bunch of other companies assist they want to help individuals to see what they're consuming think about and this is an analogy that I think Alan or Josh Corman actually use um think about that you're actually consuming something you know you're eating and most of us we actually look at the list of ingredients

does this have too much sugar for me you know I'm probably diabetic and blah blah so the same thing that comes with software or Hardware um having knowing what you're running it is essential right if you don't know what you're running then how in the world you're going to be able to determine what vulnerabilities affect you or not about how these s-bombs and all these parts started was because of other legal compliance and Licensing like you know you're using open source so I'm going to disclose what open source packages I have and for years we have been doing that in PDF documents and then says you know you should open SSL I mean the next kernel but it

doesn't go into the originality of saying this is a version of Linux kernel or this is how I'm using it and for you to determine a vulnerability If You're vulnerable or not you have to know not only the version am I actually compiling this into the way that actually affects my product and a good friend of ours too from Oracle he actually did a study and most of their vulnerabilities that they actually face don't affect them they're not exploitable right and if you look at that you know through many other vendors and that's one of the reasons that we actually you know trying to put our against our heads together with Vex and the standards is to also allow for you

to know what are false positives and the real in the perfect world will be that somebody you know instead of actually running a scanner and then calling support and say are you really affected by this you know vulnerability like qualities or net tenable or whoever tells you that you have and you have to get that person to talk to one of my guys and one of my guys talk to engineering and then engineering look into their code if you can automate that in the future that's of course the perfect world scenario it is right but that's what that's what the the main purpose of this efforts are I have a different perfect world

is that the greatest but we all know that it's really expensive to do that which is why we take the time to do the analysis to indicate if we're affected or not because half the time when we bring in open source we write a lot of code around it we have wrappers and so it's very expensive and it don't even get me on the OS updates and how expensive they are um so in a perfect world though where I think we will eventually get is that every release which will hopefully be a consistent and only security update release that's why I always talk about my in my company um will you know a customer will always

know when they're getting that next update and they will always know that they have the latest and greatest or maybe it's just a continuous update that that's constantly going on um although that is scary when you simmer and I really really pay deploying this update or is it going to break something so there's a whole other Journey that goes on around it where I think we we need to we need to go my perfect role is accommodation too I think that it's important to recognize that what we're really doing is helping customers assess risk so by eliminating some of the the challenges that customers would have by ensuring we're updating to the latest version is

important I I think that's important for many reasons the stability of your product and keeping up with the um with the changes that are made in the components that you're integrating into your technology but it's also important to be able to give people a better understanding of how exposed they really are because as Omar pointed out today it's a very manual effort it's a it's an ad hoc discussion that happens between Engineers rather than something that people can look at and make a split second decision on and say yeah I understand what they what the vulnerabilities and how it affects that product I know on my risk although the severity of the vulnerability may be

high my risk is much lower in my implementation but today that's that's there's more it's more of a subjective decision using anecdotal information rather than um well and it also takes customers understanding and being okay with receiving that yes I mean a lot of our customers if they have a scan and it's indicating you have a vulnerability even though you have a whole documentation of why you're not affected some of them have rules so like I'm sorry we we have to chat our program you know our them down if we scan and see something so um you know I think there's a lot of Education that needs to happen to our customers and to those exact uh you know

that say you need to be on this like log4j was a really good example we had customers saying you need to be on that 2.17.1 I'm like no like eventually yes I want to get to that but you really need to be on that 2.16 and and and you know but that education and understanding we're very far from the still it um looking at your Ideal World sounds like customers maybe aren't ready there I mean how are your customers dealing with are they prepared to deal with F-bomb and Bex and csap are they ready to consume that and make sense of it so as my customers being able to feed you you partner with my customers on the

formats and the approach that we would take so that they're able to consume it I have to understand what they were looking for so there's a it's a bit of a Chicken and the Egg experience right think that it if you ask five people you're gonna get 20 answers in the customer side the reason that I said that is because depending on the amount of resources that you have whether you have response or not you know they have a different way of in you know ingesting security vulnerabilities that's a reason that I said you know the perfect world is not introducing the vulnerability from the beginning that's easy said than done right that's another effort but how you're

going to see these probably in the industries that some customers are not going to have the infrastructure in place to even know what what am I gonna do with this thousand you know lines in here am I gonna even ask the right questions you know is the vulnerable in the Linux kernel more um you know impactful than openssl right so you see a lot of even companies now as a matter of fact we have one outside uh with Stern security they are digesting s-bombs to provide a risk information to to an S to a to a customer the other thing is that um even if you look at s-bombs or backs this is a continuous thing right more

vulnerabilities will come in Mobile will always will be remediated or will be determined are false positive if you think about the way that historically we have put things in our networks or in the cloud that you have to go into a certification process or this product cannot have any vulnerabilities on it and you work with the customer for six months and then you know they point fingers to my team oh you just publish another vulnerability this product is now well we have the public we have to be you know trans transparent if you look at open source vulnerabilities as Lisa and Josh they're coming as we speak somebody's actually finding one right now so there are

constant state of not being compliant is that we have to educate the whole industry and the consumers because I will never I can guarantee you there will never be a single product in the whole world that will be vulnerability free ever is the matter of us actually being able to ingest this information so we can make decisions and to help protect it you know how to remediate it and you know how to prioritize this stuff there will never be whoever sells you that I have a vulnerability free product just turn around and just you know buy something else or a system for development products just you know I remember having this discussion at a conference once and

somebody asked me what's the single what's the single mode biggest risk to our products today and I said it's anybody that's sitting down to write code it's not it's not the technology it's how we design it how we write it how we build it and it's not to say to those people that do it are bad or flawed or uneducated it's fact of life we you know security has evolved over the decades how we develop products has evolved the technology that we had 30 years ago is outdated so the methodologies are also outdated and we've learned from that so we're continuing to grow and evolve and mature and that's what makes this a journey and not a destination

how are we doing on time we still have we're about 12 minutes um I just want to do one more question before we uh turn to the audience um so the response I hear so often when we get into a conversation about s-spons and Vex is aren't you giving the hackers a blueprint how do you guys respond to that I can think go ahead so um I was presenting with a few other folks in that that kind of around the same topic at the policy desk and it was kcle Academy Story you know a whole bunch of more intelligent people than me but we were all talking about the the expectation that some attacker threat actor can be a criminal can be a

national state uh organization that they will be running they probably have a better Aspen than you that's the number one thing right so the the way of us thinking that no I'm you know this is a big problem because I'm going to publish it out and somebody's gonna share it and then that's going to be taken advantage of yeah you can you can accelerate that but they can buy the black dots of the world as a matter of fact they are buying you know this type of things to do the software composition analysis and actually know what they're running right so it is time for us that we're trying to protect to know about that however

and what we have to do is not so much of can we put it in repositories and that's a technical nerdfights that I you know have to deal with right because I'm a nerd but where to put profile information in machine readable format so I can predict where you put you know this information right that's the things that we're trying to do in the standard but in the real world how you're gonna probably see this is that originally it's going to be on a high request type of transaction right and your customer that is going over and certifying your product and put it into a into some process to acquire it and they're probably going to ask that as a

checklist right and you're going to see that a lot because they're probably not even gonna know what they're asking for but they're actually just asking for this thing and you provide to them amen you know in the perfect world will be of course that you have it in a specific repository that the product itself actually has some type of manifest that it tells you that and then you can query there's another protocol that we didn't put in there it's called Mud and where the actual product itself can tell you where you put your security advisor is uh information on and so on so that's you know the futuristic type of it but we're seeing it at Cisco is more even a

transactional place yet and even for vex you don't need an s-com for vex That's my kind of joke uh it's a question somebody's going to ask you what is your status of cbe12345 in your products so that's what Vex starts for you know we we do have those mechanisms right now sometimes it's like phone calls sometimes it's an API call sometimes it's an advisory right but uh that's what you're gonna see you know kind of baby steps in out there sorry yeah no I I actually was thinking about it and I my concern is it's not that we're giving more information but will people understand what to do with information that are giving them um so that's where my head goes at with

it I I want to protect it uh as much as I can right now because there's so much more maturity with doing dependency management now that we have the US bombs how do we know about the vulnerabilities how do I make sure that all my product teams like if there's an open SSL vulnerability then I'm I'm instantly scanning those s-bombs know what products they're consuming it making sure they know it about that vulnerability and that we're ahead of the game and we're not having our customers saying hey you have this vulnerability in this because we have your s-bomb um so that's why for me I'm a little bit hesitant but it's more because I want to

make sure I have everything else in the background working really well so that you know my customers aren't telling me something I should already know um the other thing and I just want to mention it because I feel like we missed it um in it and I'm thinking about it is that one of the biggest things with us bomb like we're all having a little difficulty is that there's no common naming so you were talking about in the past and in my past company we had all the licensing but some would say Apache struts another person would say struts another person say Apache struts version one like so there's so much inconsistency in in the naming

convention and us in the industry haven't attached to one thing and I think that because of that we're all struggling and so if my ass bomb is your s-bomb uses a different name for shots I mean like what is our customer going to do like they have to code for all these different naming things so that I feel like is still a huge gap that we we haven't figured out and I don't feel like anyone's actually taken a stance to say hey this is what we should use we're all sort of you know we use black duck so I'm going to use a lot of black ducks but if the all the black docks of the

world don't talk to each other then they're all going to have different naming conventions yeah no and back to Omar's point though I don't I'm not worried about as much about the painting uh a clear path to the malicious actors of the world um I think their reverse engineering or technology and they understand it probably just as well as they need to be able to do what they do um at the same time I don't necessarily want to make it any easier than I have to so I want to provide enough information to our customers all down the chain so they can make the right risk decisions and they understand what they meaning how they need to fix their

environment but um that's less of a concern of learning with this model yeah all right let's see do we have Germany yeah okay you guys are here representing

all these things based on the open source projects by that one guy who maintains those lead yet what about those guys but there's some public or private Partnerships that's that's a great yeah that's a really great Point yeah I'm so let me repeat the question and you were talking talking about how are we helping that that Joe schmoe who's writing the open source code so open ssf the open source software security Foundation do they say that right yeah yeah um so uh there has been a big initiative and and a significant amount of big companies have poured money in into it to to and the focus there's a lot of focus on it um one of the focuses though is to try

to figure out how do we make the open source more secure in the first place how do we do that grading of of the open source so that you know um a product team when they're choosing they pick one that maybe is more secure um so you're exactly right though we need to figure out how do we if we're using these open source how do we pour the money back into it to make them more secure in the first place so that you know we don't have a log project issue happening on a Friday uh you know type of type of thing um so there's a big effort if you don't know about it you should look into it

but I know I know my company I'm assuming you're both of yours are contributing into it um to to be able to help the whole ecosystem of Open Source but I think even Beyond open ssf and being a part of that as a company if you're using open source you should be looking to invest in I mean Intel has got a big effort around that around not just investing in the industry uh groups that are supporting the idea but then if you're going to use that technology let's make sure that we're we're either contributing to it we're a maintainer of it we're somehow influencing it we're communicating with the maintainers in any way that they're willing to

communicate with us because some of the maintainers don't want to and that helps

and make sure you're investing in those that you're supporting those which are likely used by others so I have two examples so I agree a thousand percent what they said also agree a thousand percent and one with the last comment of anybody can actually make an open source project and somebody can consume it and even though as a vendors of a producer consumer we have to get better on determining what is good infrastructure of Open Source that I should be consuming and what should be potentially outside of you know so so going back to your question about the investment I'll give you an example after heartbeat forget about log projected long time ago right horrible

was kind of a wake-up call oh my God you know people are using open SSL and that's two-thirds of the internet uh we have to pay attention to security Cisco we looked at okay what are the things that are critical infrastructure uh is the kernel well that's people giving money to that is it X Y and Z and then you know we're spoiled because we have money and we give you know to these efforts right and we said what about time what about ntp and if ntp fails you know certificates fail and everything else so we started a little project with an organization called Asic and Talos and you know and University and we look at ntpd and

nobody had looked at vulnerabilities ever right not even aesthetic analysis because we're looking for vulnerabilities and it was like free you know CBS out there right so um so whenever it comes to that we probably have to collectively say okay what are the top you know consume critical infrastructure opens the open source components and that's what opens the self is actually trying to do and the lineage Foundation of many other folks second what can we invest and more importantly why we should not be consuming some things that probably are not supported anymore but we are using it in critical infrastructure products that we actually sell right so so that's one of the things that we have to also look in this

ecosystem and is not as easy as an s-pomorph X is actually a whole development shift yeah that's a whole journey so yeah any other questions out there win back

[Music]

yeah so the question is about cease has been publishing actively exploited vulnerabilities um and making uh the rest of the industry aware that they know that exploits are happening um they did it because they wanted um their own government agencies to address vulnerabilities at a certain speed so they actually said hey this is being actively exploited this is how quickly you should fix it um I I haven't tied it in with past bomb yet um although I will with the full dependency management but where I tied it in is more into my high profile process so um when uh vulnerability is being actively exploited we typically call it a high profile especially if it's rampant and utilizing a lot of

products so I've been utilizing that information to help hey is this something that Dell is using and are you know what a re-wide spread effective or not and we need to then faster do it but the is this is Dell actually using it is where usbomb comes into play if you have your s-bombs you could quickly know that you're utilizing that that open source or or a lot of them aren't even vendor issues um and then you can quickly notify those product teams so it certainly will play in the picture as we continue to mature our processes I think that's okay all right thank you very much everybody

[Music]

thank you so everybody we uh do have a 15 minute break uh we are going to try to get started right back at 10 45 so uh yeah please be prompt see you back then [Music]

oh actually while everybody is filing out um we did find out there are actually some open seats in the software splunking Workshop so if anybody's actually interested in doing that um there's space so feel free to stop by if that's something of Interest

it's my Direction but this one

[Music]

foreign [Music]

busy weekend ahead right yeah

oh yeah I've actually been on the news yeah

hi there how's it going but so I don't know

I'm gonna argue yeah [Laughter]

foreign

foreign [Music]

thank you foreign

terrible feedback

thank you it happens here

foreign

[Music]

high resolution

thank you

foreign

thank you

foreign

thank you

no just uh I just think it was sad because they're no longer

at your office

wow

foreign

um

foreign

that one's done

I don't know if this was turned off or not [Music] hey everybody as uh folks are filtering back in um just a quick reminder I said this right as we people are exiting maybe you didn't hear it uh but it does turn out we have some open seats in the software splunking Workshop up on the third floor so that is something if you're interested in checking it out feel free to stop by uh no prep is really needed um nice if you have a laptop but not required either I think there's some offline exercises that you can pick up there so it's just a heads up about that and since we are right here at 10 45 yes

keep the lights on there sorry about that I'm going to go ahead and start introducing Manny uh we still have folks that will be filtering back in here but uh we're definitely trying to keep us on schedule today and fortunately we are on schedule so far so introduce my good friend Manny landren here uh Manny's their principal information security consultant with Align Technology Group extensive experience building securing and monitoring high value and well-regulated applications and platforms on premises and in the cloud he's LED teams that implemented the Greenfield information security programs for IE IAT insurance as which is a special Insurance Group with about 2 billion in underwent and premium underwritten premiums premiums excuse me

and also for Citrix sharefile which is a Gartner magic water leading content collaboration solution hosted with AWS and Azure many is a graduate of Virginia Tech another Virginia Tech tie this morning and a veteran of the US Army he enjoys spending time with his family in playing his violin and also hiking in Umstead so with that I'm going to turn it over to Manny and uh let's get started thank you Chuck appreciate it uh so so thank you besides for the opportunity to present today and thanks for uh to all of you for attending so I'm Mandy Lander and I help organizations secure and um monitor their information technology platforms and applications on premises and Cloud today we'll be covering a talk

called understanding and securing containerized environments um as far as agenda is concerned that's right here so we'll provide you with an overview of container security with an emphasis on the fundamentals uh and differences between containerized environments and virtualization the pros and cons of containerization from a security perspective and the challenges associated with securing and monitoring containers throughout their life cycle from from container registry on through production we're also going to explore the concepts of container Escape review the role of Kernel name spaces and control groups and the role they play to enforce segmentation and control resource allocation and then we're going to gain an understanding of uh container runtime Behavior and the importance of implementing safeguards including lease

privilege access to promote container isolation so a container image you know differs somewhat from an actual container uh you build a container from a text-based a script of instructions usually a Docker file an image executes code in a container so think of a containers running an instance of an image a runtime environment is the environment in which the program or application is executed it loads the application and runs it with all the resources necessary for the program to run independently of the operating system and then simplistically it's software that's designed everywhere before it actually installs

so containers and virtual machines have similar resource isolation allocation benefits but function a little bit differently because a container virtualizes the operating system instead of the hardware containers are also much more portable and efficient than virtual machines because containers are typically much smaller in size and therefore take up less space containers are an abstraction really of the app layer that packages code and dependencies together so multiple containers can run on the same machine and share the operating system kernel with other containers and then each is isolated an isolated process rather in user space so virtual machines or VMS are an abstraction of the physical layer uh and you have one server running on on rather you have many servers rather

virtualized running on one server one one hot one server the hypervisor allows multiple VMS to run on a single machine and each VM includes a full copy of the operating system along with the application and any necessary binaries uh and libraries as well VMS are typically much larger uh than containers and traditional VMS can be really slow to boot there are new VMS like Amazon's fire firecracker and the Kata that are lightweight VMS that do start in in milliseconds rather than minutes but typically they're much uh they take a lot longer to boot um I'd like to highlight a couple of terms uh in the event we have some folks in the audience that are not familiar with them so user

space refers to the code in an operating system that lives outside the kernel the operating system kernel is a central module of an operating system uh that's really responsible for managing all the resources of memory processes devices the disk itself system calls and more and the fundamental difference between containers and virtual machines is that virtual machines run an entire copy of the operating system including the kernel while containers share the hosts kernel as a result container isolation is technically weaker than that provided by hypervisor in a in a virtual machine use case so Engineers create managed containers run applications relying on a combination of continuous integration continuous delivery or continuous deployment pipelines popular contamination Solutions such as

kubernetes Docker swarm Amazon elastic clouds container service rather ECS to name a few so Amazon elastic container service is fully managed it's a fully managed container orchestration service so Amazon elastic kubernetes service or eks for short is a managed service that makes it easy for you to run kubernetes on AWS without installing an operating system uh owning your own kubernetes control plane or work nodes worker nodes worker nodes basically is a machine that hosts uh one or more containers and then you can build and deploy container images using traditional virtual machines as well as serverless Computing engines such as AWS fargate so AWS fargate is a serverless compute engine for containers that works with both Amazon ECS and Amazon eks each

fargate task has its own isolation boundary and does not share the underlying kernel CPU resources memory resources or network interface with another disk so I credit Liz rice the author of securing containers for this image and list this slide represents a uh she she wrote that book it's published by O'Reilly if you haven't read it if you haven't um if you have a copy of that please get it it's well worth the read she's very succinct with uh with her writing um I think the whole the book and it's an entirety is about 160 pages and covers a lot of great detail this slide represents a high level overview of the common risk associated with containerized workloads so you know

software vulnerability is in the host operating system and third-party apps as well as all the misconfigurations in the host operating system and container they introduce exploitable vulnerabilities that may lead to privileged escalation and container Escape it's also important to consider the security and trustworthiness of the build environment and the container registry you want to ensure that the integrity and trustworthiness of the container images used to instantiate The Container applications in production and to avoid introducing untrusted and unauthorized images so managing Secrets also remains a challenge but orchestration platforms like kubernetes and Docker swarm along with secret vaults like hashicor Vault and AWS KMS they've introduced the capability to more securely manage Secrets throughout their life cycle as

well so crowded Julie Evans she's at work on Twitter she published she publishes zines rather on wizardscenes.com herzine how containers work goes for about 12 bucks and is entertaining uh entertaining rather introduction to how containers work um recall that applications uh do not run in user space and need to communicate with the kernel right the the Linux kernel using system calls to do things like read and write data to a file open or change the owner of a file execute another program Creator kill a process and remember that a Docker container or a container period is a process there are hundreds of possible system calls depending on the version of the Linux kernel that's running and

applications don't really need all the system calls at their disposal therefore evaluate the profile of the system of the system calls in the in the application um that the application makes rather throughout its life cycle to determine whether the appropriate system calls are present and they adhere to the principle of least privilege uh so next uh we'll use S trace or at least I'll show you how I used s trace a Linux debugging utility that monitors interactions between processes and Linux analytics kernel itself which includes system calls to observe what occurred when a container is terminated so a little setup is required to use S trace and again s Trace is primarily a debugging tool that's very capable this

isn't intended as a I'm not proposing that sjs be the primary tool that you use there are several third-party Solutions commercial off the shelf or open source that provides similar capability this is just a readily available tool rather in the Linux toolbox that you can use to evaluate what sort of system calls are made whether for educational or academic purposes or um for for to evaluate whether a container is is uh you know making proper system calls to the kernel so again a little setup is required to use s-trace you run pseudo drps to list all the containers running on the host note that the process ID isn't listed by default when you run Docker PS so to

obtain the process ID take note of the container ID and run the the docker inspect command you see on the screen to look up the process ID and you'll get copies of this slide I'm sure of these slides I'm sure so you don't have to take a picture right write it down next run sudo Trace with the p switch for process with the associated process ID uh which in this case is four six nine one in another terminal I terminated the running container and s-trace captured the raw output listing all the system calls made to the kernel uh it's cut off but in a second I'll show you the summary of those system calls um

let's see what else is important here um so again s Trace is just a it's really a debugging tool that you can use to at least evaluate what sort of system calls a container is making or any process for that uh whether it be PWD for your present working directory LS it doesn't matter if you're interested in doing that it's really easy to do just make sure that it's installed and then you can go ahead and start using S Trace at your at your liking so uh here we use um s Trace to view system calls and and put in a table format using the c-switch note that the process ID changed because I instantiated a new version of the

containerized application this application wasn't doing much it was a pixie vulnerable API uh I I as much as I tried to interact with it I couldn't get it to actually make a system call until I actually terminated the the the container itself um so we used uh s trays to provide you with an idea of what occurs under the covers but bear in mind that s Trace while capable you know does have its limitations however there are open source Solutions again like like Tracy Tracy is one that you can use in place of the S Trace that serve a similar purpose and then open source and Commercial solutions that rely on their capability to observe and

filter calls to Monitor and secure uh the host and containerized applications these are typically bundled with Cloud native Apple application protection platforms some that come into mind are Aqua systig Trend micros you know Cloud one that sort of thing so while I'm not prepared to give you a primer on Linux permissions today please note that Linux or at least in Linux everything's a file and file permissions do dictate who or what can access that file and what actions they can perform on those files understand that containerized applications run as a process and most run as root albeit with limited capabilities consider that there are rootless alternatives right that you can use to host containers um

that are emerging that promise to eliminate the need to run a container's route so be sure to review the containerized workload to determine if it's running as root and if it's an absolute requirement for it to run this route otherwise take steps to effectively limit permissions as necessary you can also run sandboxes on the host to filter those those system calls as well and then you can review those capabilities and we'll show you how to do that in a second to determine whether those capabilities are are necessary note that the set uid flag can be pretty dangerous because it might give an unauthorized user root access or at least access to run a program under

another user consider that the set uid sets the effective user ID of the calling process and a container is typically running as root so Linux divides the Privileges traditionally associated with Superuser into distinct units known as capabilities would be independently enabled and disabled the capabilities can be independently granted to processes including container processes you can use the get caps command along with the process ID of the container to identify the list of capabilities that are enabled you can reference the capabilities man page to learn more about each capability provides a definition of what the capability uh what sort of privileges that capability grants well this container doesn't have the capsys admin capability enabled I highlight it here because it should be

avoided as it enables many of the privileged capabilities associated with roof you'll see later when we talk about container skate briefly that the capsys admin capability is is typically what is leveraged through a vulnerability in a kernel to one escalate Privileges and then to escape the container itself you also want to avoid running containers and in privilege mode or root mode with the dash dash privileged flag although a container may run as root by default again several of the capabilities are not granted by default privilege mode means that if you are root in the container then you you essentially have full privileges of root on the host system itself uh recall that containers run as a

process right they are a process uh even though they have an application running even though they have dependencies in libraries they do run as a process there are three essential Linux kernel mechanisms to limit a processes access to host resources control groups namespaces and then changing the route so control groups are a feature of Linux that limits account and accounts for uh isolation so isolated it isolates resources including CPU memory disk i o and network and any really any process that any process can consume namespaces are a feature of Linux and it partitions kernel resources such that one of the set of processes doesn't see other resources that other processes have access to you can also view the name spaces

available on your host operating system by using the ls and S command there and I highlighted for example the process ID namespace there are several namespaces and one example is the PID namespace which provides processes with an independent set of process IDs including process IDs that Spawn from the from the parent process ID so as child process IDs another example full space this one's important because it provides a mechanism uh to map the root user inside a container to a limited privileged user on the host itself you can see the hosts or rather you cannot see the host entire route or rather the file system from a container typically because a root directory is changed during container creation

this effectively limits the set of files and directories that a container can view so um this particular vulnerability cve 20220185 was identified in January of 22. and um recall that capsys admin you know is largely a catch-all capability it can easily lead to additional capabilities or full root typically granting access to all capabilities so this container Escape vulnerability materialized because of an issue in the Linux kernel but container Escape can also result from issues with misconfiguration as well doesn't have to be a vulnerability in the kernel or in the operating system typical mitigation steps include you know obviously patching the kernel applying the necessary patch but compensating controls can include minimizing the use of capsys admin the

capsu's admin capability using a Sandbox like SEC comp to effectively filter system calls the default set comp profile is usually enough and a more recent Docker implementations actually Implement set comp by default foreign so here we start talking about each of the um security considerations for a containerized environment I listed them out so that when we do share the slides they they're they're readable they're they're digestible so you do want to scan images in the registry for vulnerability for vulnerabilities right so Registries especially public Registries can be untrusted so do you take great care in evaluating the images that you use the best approach in certain cases especially when you're considering posting sensitive workloads is to create

those images yourself apply available updates to third-party libraries including applications before you introduce it to the bill or at build rather so before you introduce it a production you don't want to do that at one time cryptographically sign the images to ensure its integrity and trustworthiness once you create an image it's important that you take steps to validate and preserve its integrity and also admission control emission control safeguards that are largely part of orchestration solutions like Docker and kubernetes uh Dr swarm and kubernetes rather they depend on that they can depend on that signature to evaluate trustworthy image before they actually introduce it to the build cycle in production so keep Registries private if it doesn't

need right and do you take great care when you um evaluate and select images that are in public repositories as they can contain malware they can contain a vulnerable third-party um libraries dependencies and of course you need to evaluate do code reviews to evaluate what the code is actually doing um secure the build process again to protect the image of the uh the Integrity image but only allow approved images to deploy in production so then you want you you want to secure the host just as you have been doing um uh for years right this is really the one constant um build a host from scratch if you if you need to right there are certain

workloads sensitive workloads that you should uh rely on a build from scratch um host or use a trusted Baseline the center for Internet Security benchmarks do a great job however they don't produce benchmarks for every single operating system version configure the host operating system according to CIS benchmarks but again if it's available right use a minimal operating system purpose built to host containers such as Alpine or core OS these operating systems typically remove unnecessary libraries dependencies functionality and you reduce the attack surface uh and of course they're much more lightweight as well um consider using a minimal operating system like sorry scan the host operating system for patch config and configuration related vulnerabilities uh obviously before you introduce it to

production and then of course in production uh consider refreshing the host operating system as opposed to patching it if you can uh work with your engineering team to figure out a way to automate and take advantage of you know immutability ephemerality immutability meaning that if you agree for example to enforce immutability in a production environment that means that anything in production doesn't really change you're not patching you're not adding any any third-party software you're not making any configuration changes the preference would be to introduce an entirely new operating system host right an entirely new container that has been patched in the in the application delivery process to ensure that it's easier to manage image and and to ensure that you don't

have a lot of you know basically uh configuration drift [Music] um apply available updates to the host operating system as as usual and then leverage a Sandbox or Linux security module to enhance isolation um these come in the form of SEC comp app armor SC Linux gvisor the G visor is really a hybrid between a virtual machine and a Linux security module or a Sandbox rather and list it you know there are there are ways now to host uh containers in very lightweight very fast um virtual machines again Kata comes to mind uh AWS firecracker comes to mind these are really lightweight virtual machines that operate just like uh typical virtual machines but they they

load much faster in milliseconds rather than minutes and they're much smaller so if you have a sensitive workload that you need to ensure the isolation for and look virtual machine isolation is for the most part tried and true it's proven and it does add another layer of defense for very sensitive workloads as opposed to relying only on the Kernel within the operating operating system itself you can argue that the kernel has been around for a while too however uh the kernel uh occasionally does reflect uh or produce some materialize some vulnerabilities as well secure the container you know run containers as non-root users if possible or a paired down version of root right remember the capabilities we talked

about not all containers need every single capability and they don't need every single namespace either right so avoid running containers in the privilege mode as we talked about because essentially that overrides all Security checks um you are true root in the container when you're running this privileged mode right so you you override any uh sandboxing any uh Linux security modules those kinds of checks you basically get a get a pass remove unnecessary Linux capabilities as I just discussed and then leverage a runtime protection solution to ensure that only authorized executables run these come in very in various forms uh it could be a traditional anti-malware solution or it can be a anti-malware solution that also incorporates some

degree of syscall filtering um perform continuous Cloud security posture management um leveraging you know open source or commercial off-the-shelf Solutions these continue these Cloud security posture Management Solutions not only evaluate um how well your uh environment typically your Cloud platform is configured but also the services within those um within that cloud platform as well uh run containers on hosts dedicated to running containers right don't mix and match workloads on a on a operating system host avoid mixing sensitive and non-sensitive workloads on the same host because there's always the possibility of container escape the isolation boundary with the kernel is not as strong as it is with a with a hypervisor and in a virtual traditional virtual machine use

case so and then also while not listed here do consider uh hosting workloads or or containers rather that communicate parties or external third parties outside of the network with those that are only communicating internally so very much like we segment our existing on-prem networks you should also segment your containerized environment then add executables to the container at image build time not runtime alluded to this earlier because you do want to you do want to ensure that the Integrity of your containerized application and then you also want to avoid uh drift you know configuration drift right you end up with an environment that's using different versions of a library you know different versions of a dependency

and ultimately it's really hard to secure an environment like that it's really frustrating to try to identify which containers are vulnerable in fact work with your engineering team if you're in this particular situation to embrace ephemerality even though ephemerality can be a bit frustrating as well ephemerality means that you know the instance isn't treated as an ongoing concern it it's temporary right and it's meant to be refreshed uh and if you can automate that um that that if you can automate that process at refresh time you're essentially introducing a container or a host operating system that is patched configured to the latest requirements and then you avoid that configuration drift as well and then run immutable containers to

avoid configuration drifts we talk about immutability not changing a production workload in production uh make that change in the continuous integration continues deliver your deployment pipeline apply the patches change any configuration that you have to update any containers and then redeploy them and do that on a periodic basis it uh it'll it's I I think that the work is well worth the effort to ensure that that environment is much easier to Monitor and ultimately much easier to secure so securing Communications between components my observation is that this is an area that isn't really typically enforced or or you know it's not emphasized so you have containers that are talking that can talk to any container in the

environment that can talk to any component in the environment usually it's unencrypted right there's no Mutual authentication right so enforced Mutual TLS typically either through an orchestration solution or through what they call a service mesh like istio one of the byproducts of a service mesh that does control communication at the application layer is enforcing you know Mutual TLS TLS being the protocol that is used to encrypt HTTP or https traffic uh consider using certificates to enable Mutual authentication as well so not only encrypting that traffic in but also ensuring that only machines that should be talking to each other are talking to each other and then use Network policies to control which components can communicate very

similar to to what you would do in a traditional Network again a service mesh can do that for you at the application layer but doesn't really handle the network layer so you do need some capability at the network layer including you know things like iptables to uh to enforce that those connectivity requirements and that segmentation so treat containers as ephemeral as I mentioned earlier refresh often and update the hosts and containers and then check for sensitive directories mounted on the host because any sensitive directory is mounted on the host especially if the container is running as root are visible to the container then you have uh protecting secrets in transit net rest and this is one of the most significant

challenges today so don't hard code secrets in an image at any stage of the image creation process because those an image is created by layering files and if you introduce and hard code a credential at any layer you're not really removing that layer right uh and so you can actually decompress the image and and actually see those credentials uh when it decompresses so please avoid doing that if you do find that you have to really rebuild the image from scratch don't Define secrets and environment variables either because if a if a container is running as rude it has access to those environment variables and then of course if the application crashes though typically those environment variables

are are displayed right and they're also captured in in log files as well encrypt secrets in purpose-built secret vaults like we talked about hashicorp Vault um uh and other vaults are purpose built to manage secrets and to uh encrypt them at rest and in transit and then leverage features and orchestration platforms like kubernetes and Docker swarm to manage Secrets as well if that capability exists and then consider encrypting us don't consider I mean do encrypt secrets in transit and don't pass Secrets over the network unencrypted and and really avoid sending secrets to the network interface that's just a bad practice it's better to uh the optimal approach rather is to pass Seekers by writing them to a a file

accessed through a temporary directory held in memory and the key word is temporary right does it live persistently in memory it basically instantiates in memory when it's required and then of course it's ephemeral and then avoid writing secrets to disk even in encrypted because at some it's it's unencrypted when you're writing it to disk um and so uh and also you have to figure out a way to decrypt it right and so that that introduces another challenge all right so that's all I have today um we are a line Technology Group we are local we are built and founded here all of us are from Raleigh we focus on um securing monitoring um uh you know technology platforms

whether they're in the cloud whether or not we're an AWS technology partner and again I appreciate your time thank you for coming questions

yeah questions we have them yes sir

uh I haven't I haven't reviewed them but I think because you're the NSA they're probably spot on yeah I haven't reviewed them and uh and so there is a fundamental difference between the actual container and the container orchestration solution um I will take a look though yeah

yeah I haven't reviewed them either anybody else yes sir

uh I think the question is around performing forensics both online and offline with containers is that fair um so a container is going to produce uh well first let's let's face it in a lot of environments containers are ephemeral right here today gone tomorrow um and so you may not have the container or even the host operating system uh to evaluate uh if that isn't if that is an important component then as I mentioned earlier you may want to um host that particular workload in a lightweight you know virtual machine and maybe treat it as more persistent otherwise you're going to be relying on the collection of all the system calls that the container made

um when it was when it was uh when it was exploited right one privileged escalation occurred uh and then of course one container Escape occurred on the host itself and so I would also evaluate the extent to which you're collecting the the information you need on the host operating system because the host operating system is the one that's uh the kernel is when it's uh evaluating those system calls as well does that make sense yes sir foreign

yeah um obviously it really depends on your needs right I mean most third-party application patch repositories or a pay for Play service I can't recall any of them off the top of my body even though I've used several of them um but you what you would do is evaluate the kinds of third-party applications that you're running in your environment and then determine the extent to which that third-party patch repository supports uh the the applications right and then obviously the host operating systems that you're running as well um one thing uh I want to I want to mention though is that you know it's a lot easier to patch in the application delivery process as than it is to patch

in production it just introduces a lot of overhead um and a lot of uncertainty too right you know having been embedded with engineering teams uh I've I've caused my share of problems when I patched you know even if I patched using the the proper patch management process you know going from test stage to prod inevitably something happens in prod so um going off on a little tangent here do consider uh you know patching in a non-prot environment leveraging a source that you know satisfies all of the requirements that you know you're using in your environment thank you okay oh it's like suggestion for Secrets management the secrets database that you maybe prefer or use or what works best uh it's not

there's there isn't one that I prefer um it obviously has it has a lot to do with you know the technology you're using right so a lot of orchestration solutions are introducing the capability to manage Secrets they may not be full featured though Hashi core Vault you know seems to be a very popular one if you're in AWS and secrets manager or KW KMS rather or Key Management Service is is going to help so my answer is it really depends on the platform and the solutions that you rely on uh to deploy manage your containerized environment or any technology for that matter that relies on Secrets right but ultimately you wanna avoid you know writing those

secrets to environment variables you want to avoid writing those secrets of disk um and uh you do want to leverage a purpose-built vault and then you want to encrypt those secrets in transit at rest um in between components that leverage those Secrets you're welcome yeah and I want to emphasize like none of this is easy right and I even underestimated um I even underestimated the the overhead associated with you know securing a containerized environment once you start to really evaluate the permutations of what could possibly happen in this kind of environment the operating system itself on the host whether it's in the container it gets really complicated so it's a matter of really taking a risk-based

approach making some trade-offs you know and then of course negotiating those trade-offs with the engineering team but then emphasizing with the engineering team that the more efficient approach is really to automate as much of that in the application delivery process as opposed to production I think they will be very amenable to doing that any other questions foreign last plug uh you know for for Liz Rice's book uh securing containers have been very influential um very informative uh really is the tip of the iceberg though um and so I do encourage you to read that book if you're interested in container security uh read some of the docker uh uh documentation the kubernetes documentation um it's still evolving even though it's

about you know 10 tennis years old um and then a lot of solutions are out there that are you know coming along that are purpose built to help you manage this environment uh to monitor it and ultimately secure it thank you [Applause] for you oh thank you so much

thanks again yeah thank you so uh while we have our next group of speakers uh making the way up to the stage here just a couple of logistical notes so lunch will be coming up after this session at noon uh there are already a couple of the food trucks out there I think we had three scheduled to be here um at least two of them are here I'm not sure about the third one but that was what I just heard a few minutes ago and also just a um a note if you are going to be going to the after party at Pony source and you have any sort of food allergies or restrictions um just if you have a chance let us know

you can either find Joe shotman kind of tall guy wearing a blue B-side shirt glasses let him know he's organizing that um also you could just tweet at us or something like that tweet it besides RDU to let us know but um yeah just uh wanted to get that out there while we're transitioning So Lisa and Julia if you want to come on upstage and we'll get you plugged in here let's see yeah

okay great so um yeah this one working too yeah that one's working too okay yeah oh well uh they're getting set up I'm just going to really quickly do an intro I'm not going to read through the full bio since we have two speakers here um but uh yeah we have both um I don't think make sure I have the right names here because I don't want to mess that up yeah so Lisa Bradley and Julia Hopkins uh both with Dell and part of the product security incident response team uh here is going to be about bringing order to the chaos of security vulnerabilities which is a topic that is near and dear to my heart so can't wait

to hear this one so um yeah I'll let you go

take this out [Music] let me know hello I like the pace a little okay you're good yeah yeah all right some fun stuff so um I spoke earlier a little bit in the panel but Lisa Bradley I'm with Dell um in my past um I worked at uh Nvidia and at IBM I've been in security for over 10 years I came from New York down here 23 years ago I went to NC State so we'll pack fans I got my PhD from there and um I uh call um my team Versace because I'd like to have a fun name around it uh vulnerability response customer security and community enablement so uh t-shirts underneath me s-bomb

um security Champion security training uh lots of other other fun things and I'm Julia Hawkins I'm a technical program manager um on the product security incident response team at Dell prior to that I was with Lenovo in their product Security office and then I also helped to establish their Chief Security office and I have a masters in cyber security from Utica College mm-hmm great so um I want to start off just in general a lot of people when they think cyber security instantly think the sea cert side so peace cert is maybe the red-headed stepchild of it um and actually uh it has um the name incident but we actually don't deal if there's like an incident that happens in the

company uh pcert doesn't handle it um our job is to best protect the brand and our customers by going through a typical uh process where we're trying to resolve vulnerabilities that are in the product line so P for the p sertas is product so very product focused here go to the next one so um any company that has a piece should have a policy a framework that they're following uh first uh piece services framework if you haven't heard of first um there's a csip framework a psert framework um a lot of uh are in doubt what we follow is uh that framework where we start off with the discovery how did we find out about the vulnerability was it

us through STL practices is it you know from our customer is it from a vendor uh researcher bug Bounty program whatever we're lucky in in April we started a public bug Bounty program for Dell I'm very happy about that um we go to the triage and Analysis so we found a vulnerability is it valid what's the impact what's a CVSs score so CVSs is the common vulnerability scoring system and um you know the higher the number the typically the worse worse the risk uh goes from a low medium high and and critical um then we go into remediation figuring out our plan remediating the vulnerability what is our comms plan um you know if we can't fix it on time

there's an exception procedure that we we fall and then lastly the disclosure uh dell security advisory we call it but security advisory sometimes security bulletin security notice how do we make sure that we tell our customers about the vulnerability the risk that is there with the security update that they're going to get so that they follow their own policies to figure out how quickly they should address uh a long time we through this we have a high profile procedure so uh log for J for example Spectrum ltown harp lead but even sometimes for us even sometimes we could have a a proprietary code vulnerability that we could call high profile because um some big researchers are going to

disclose it working with PR and all that governance and reporting a lot of metrics kpis some of the key kpis that we do are focused on how are you adhering to our standard how quickly are you addressing your vulnerabilities within our slos how are you doing with your open source component management are the customers finding the issues or are we finding the issues and with proprietary code are you know how many are finding from researchers versus us how well is our STL process and lastly stakeholder communication so out to the comms to the rest of the business so this is our formal part of of our talk here uh where we have it and then and then we'll we'll try to have a

little more more fun as we continue on here Okay so so I'm gonna put my uh my researcher costume on here uh so as a researcher what are my overarching goals here what is what is my motive um as a researcher it's getting payment after finding a valid vulnerability it's uh setting my reputation you know getting my claim to fame uh being able to go to conferences and speak about my Discovery uh it may be helping customers protecting them helping to keep them more secure or just people in general uh and then there's a consideration of future employment if I find a valid vulnerability I submitted to a company you know will they bring me on

so but what I don't know is what's really going on behind the scenes when I report that vulnerability to a company So Lisa maybe you can yeah the curtain back so so we will walk through this but right now my focus is on my customers and my Brands right how do I make sure that I'm protecting them and making sure I'm adhering to the industry standards to what my customers think my customers think and there's a lot of pressure that I have to make sure that I'm protecting the company and when I'm thinking about those goals my end goal is is please don't get the media involved I don't want to be that label that story I don't

want to have to do that I don't want to have to tell the president or the CSO all of the stuff that's going on because really I'm trying my best to handle this um but we do have some common goals I think in general we want to protect our customers and you know help the ecosystem like we we both want to do that

so walking into the first phase of discovery I'm a I'm a researcher I'm submitting the report what do I need to think about and what questions do I need to ask what's the product scope here is it is the vulnerability just in this one product is it in other products does it affect this one company or are other companies going to get involved that I can submit this report to oh crap I just got a vulnerability from a researcher here we go um is it valid let me think well first did they even give me enough information can I even understand what they're trying to tell me do they tell me what products affected that they is the product in scope should

I be paying them you know does it fall into my bug Bounty program am I going to acknowledge them and oh I have a have I worked with you before I hope I worked with you before because maybe we have a good rapport if not I have no clue what I'm going into I hope they don't zero day me I don't want to be zero day like let's make sure my team really has a raft together and is talking to you have we talked to you did we oh my god did I respond did I respond back to the vulnerability that you gave me not yet okay automatic response went out okay I I forgot I had the automatic response this

is great um but you but when do I get to talk to a real person Yeah well yeah so we're looking through um I'm figuring it out uh I don't know about this issue okay we're good um is there enough though that I'm going to triage it and the security team or am I going to go bring it to the product team am I ready to bring it to the product team am I ready to go and now scare the crap out of this whole product team that they now have a researcher of vulnerability I think I'm ready where are we going to next next we're going to severity what's the severity I didn't even

confirm the damn vulnerability yet like I got a little more work here um I just got everybody known I opened up the ticket I put all your information in there I think there's enough so I'm gonna now do my triage and Analysis and I'm gonna figure it out so and when the hell do I get paid yeah not yet not yet I didn't confirm it yet I'm still working on that I think you might have missed something like someone's submission on submission where where can I even submit this there's a bug Bounty program that's an option if the company has one set up uh I do security email I have that too awesome and if I want to write a paper

on this oh please don't write a paper don't write a paper but I definitely want to tweet about it oh wait don't don't tweet yet don't tweet don't give everybody away don't nope and how long do I have to wait until I can do all that yeah well let me confirm it still and if you make me wait too long I'm just gonna zero day you yeah yeah so so we're gonna go on to our our triage here I send it out to my product team they're analyzing it and we can't get from Step a to step C there's thinks of B like can you give me a little more information dang now they're asking me all these questions

did I give them enough information I don't feel like I did you didn't um um can you give me some more information maybe you could record yourself doing it and give me a little video and here's my proof of concept damn okay I have the proof of concept I could recreate this it's a vulnerability I don't know about it I now need to score it how severe is this does it only affect this product I found that it affects more products so with broader impact do I get a bigger reward now um no uh but wait a minute I gotta look is this my code is it my code is it my odms is my oems is it my vendors where

is this vulnerability who's whose problem is it is it really my problem can I pass you off to my vendor it would really be nice if they had to deal with it uh instead of me and then maybe they could pay you but they might not have a bug value program so then do I pay I don't know if I pay do I pay what did my rules say um maybe I pay I still probably should pay you found something I'd like to be paid yeah yeah I'm sure um so I determine it's my issue so I've got it I it's valid okay I'm gonna now pay you but I'm gonna pay you based on the

severity that I think it is so I think it's a a you know a medium it's a medium here it is we we got this we score things all the time I don't agree they're coming back and they're giving me a lower severity score yeah I think this should be higher I can maybe talk to you do I want to talk to you do I want to have a call do I want to just email I'm certainly not going to let my product team talk to you they will mess this all up um so I'm going to talk about how we know how to score what we think of this talk through the vector do you agree

with me yet I think we're getting close Okay Okay so let's go and talk about this we can have a call do you want to have a call let's have a call okay we'll figure this out so we have a call we're good we're good severity confirmation and agreement yeah I might have moved it up a little bit they were right about one thing so it moved up but it's still a medium but it's like a little higher medium higher medium still important or should we call it a high I don't know we're close we're around there um okay so right now I'm lucky it doesn't affect everybody if it affected all of my product lines I would be

calling this a high profile um also if it was coming from a vendor I'd be working with that vendor uh the odm and oh my God let me hope that that odm I actually have a contract of how quickly they should address their vulnerabilities because if not they could take a really long time um and uh you said you're gonna vlog oh yes I intend to blog okay and be good to cover it they were good um hey if it did though affect a lot of products um are you interested in working with cert or some coordination and it really would help you it would help you certainly help me but it would help you

if you were with them because if you try to coordinate working with all those other vendors it might be difficult yeah what does that entail I think you just tell them and then they they handle all the coordination it's sort of impressive all right yeah okay we're gonna just say right now that it just affects my product we're going to go on but if it we would have a whole another story of embargo days matching embargo days everything with all these other vendors uh hopefully we would utilize someone like cert uh if we wanted to um and then it would help our coordination

here we go you ready so now when when do I get to tell people about this yeah this is getting disclosed yeah um so I just released a security update or my security updates and with my features right now so I just had a big release um so we're just getting started on our journey here or oh crap did we did we not release yet now we're we're about to release we didn't release I got a few more weeks we're on a code freeze I'm in trouble I need now to tell you that it's going to take us a little bit longer you told me it was this date though yeah I'm gonna zero day you if you please

suck on that please don't zero Damien I'm really trying here but you know I have to get my QA resources aligned and wait did I even figure out how to fix this yet um okay it's gonna take a little longer it's in this component and this component and you know it's sort of complicated we have to grab a paper to write it and it's due at the end of the semester oh okay okay uh knock knock exact door um we have a researcher reported issue uh we have a timeline that we typically can't meet right now even though the team's trying their best to follow our slos our standard but um um yeah no uh we're gonna get zero date

if you don't fix it by this date um you need to fix it by the state because I really don't think you should accept the risk here of getting zero day they plan on blogging um okay I got a date you good with this date now what do you think I'm good with the date okay I'm ready to start coming up with my name and brand this thing no please don't fan name it no no no no no you remember like that bad luck sadlock thing like you don't want to be that like is this big enough it's big you of course think it's big maybe it's only one of my products maybe two that

are affected I don't know should we like what do you think I think we're good we're good we're on it I think we're good I think we got a remediation date or and we're in agreement yeah uh meanwhile I'm talking to PR I'm talking illegal I'm talking to the product team I'm making sure they're on top of things I have a reactive statement then I'm ready in case the media picks up on this but the media shouldn't pick up on this that should be good I should be good I'm good um I have my embargo date coordinating with other vendors if they were affected they don't screw it up um um hey are you gonna blog

oh yes oh can I see that okay I'll let you review it okay PR is all over this damn blog what am I gonna do they want to fix like 15 to 20 to 30 things on this blog I'm not going to screw this researcher out I'm not gonna do it I'm gonna pick my top five things and I'm gonna go to this researcher with um oh my God they have proof of concept code in this blog I can't proof of content code it's going to kill me my customers don't even have any time to fix this yet um knock knock researcher can you can you not put that proof of concept code in um in your blog can you can you just

wait like even if you give me like two weeks just wait a little bit can I still disclose the vulnerability without the proof of concept yeah certainly I'm gonna have to put a security advisory out anyways so I'm gonna put that out I'm gonna acknowledge I'm gonna acknowledge you yes I hate you right um so you're good uh yep I'll have the cve I have the description I have all this if you want to see it you could see the security advisor I would love to see it remember my name in it yeah we already agree with the score just remember that we agreed on this this is the agreed score that we have in there

um but if you could just I'm okay with you blogging just if you hold off on the approval council code can you do that for me we'll hold off on the proof of concept code yeah like a couple weeks two weeks yeah perfect great thanks good we're good I got this we're good you feel like we're good you got it I think we're good we're good we got this got it good okay it is disclosure day I get my claimed Fame today I got my cve I contacted miter they're activating it I published my security advisory I'm all ready to go I'm tweeting no no don't tweet yet Tweety resource just told me that they came

down with covet and I need like one more week one more week delete one more week yeah we're all good okay got the fix out media looks pretty good come actually I don't see any media that reactive statement I didn't even need that all that work I did I didn't need that I really need that what are they doing all these articles I Was preparing all my customer support not only my customers aren't even calling they probably aren't paying attention to it I don't have that API thing going on yet so you know they're human readable they're they should know about it I transferred my risk onto them they should pick it up but we're good we're

good we're good did you tweet I did oh crap okay I tweeted come up with my name I've got my conference abstract ready to go it's been accepted now I just need to create my swag that damn sweet that damn tweet got picked up by the media you're yeah you're doing I'm famous [Music] um I now have 50 customers that just called me today I realize that my security advisory is not clear enough I need to update it because they're confused of what they need to do did the media tell them that we fixed this if the media statement isn't we fixed this we got this we're in trouble your that Media blog post didn't

acknowledge we didn't even list it so now it looks like I'm affected I have no fix I got zero Dade but in fact I I really have this I do media reach out to those resources get them to update their posting please because we already fixed this we got this we fixed it we're good it sucks I had to go tell my CSO I had to go tell all these execs I had all these people emailing me tons like I don't even know these people they're emailing me we have it though we're good I maybe made the stock dip just a little like a little but how are you doing over there and now Defcon is coming next week and I am

going and I am presenting on my discovery okay um can you share that with me I can thanks that's awesome because I want to know what's coming um hey I'm I'm reviewing this and you have a new vulnerability in here that I didn't know about yeah okay media PR let's get ready for this we fixed this one part we didn't oh wait can you not show the proof of concept for that new thing that you were just showing me but it's the conference um this is the whole purpose of it okay not exact but SVP exec we have a zero day on our hands within a week but wait maybe I can reach out to Defcon

um maybe I could tell them like hey I don't have a fix for this yet maybe we could negotiate through the conference to yeah no okay it's not working okay man the conference is coming I'm ready PR is ready legal's ready I got that security note so I have security advisories security notices this is now a high profile right I'm now going to be zero day and I only have a week I need to have some statement I fixed one part of this we're working to analyze the rest of them uh the rest of the issues that you put in that presentation but we got it I'm good we're good I'm good we're good next time like I do have a bug Bounty

program so remember that because the next time if you went through that I would pay you for the other vulnerabilities you found I intend to get paid for more vulnerabilities yeah but it was very nice working with you it was wonderful working with you yeah yeah okay and then the learning the post-mortem what'd you learn I learned that communication coordination it's going to be to the researchers benefit I'm still working this issue we have a root cause analysis procedure that we put on on determining what scl controls might have been missed I'm doing a whole Lessons Learned did I have enough guidance why didn't I find this why didn't we find this first what do we need to improve on can I do

this better how do I get the media to understand what's going on can we pre-talk to Media next time when we know it's going to be hot I am working this I'm going to be working this for the next maybe month or two you might not know that I am still working it um I'm documenting I'm updating policies procedures all that they do not take overnight I have to get all my stakeholders to approve it I just change the control that means their maturity assessment is at that level pull it all up because they didn't do this one thing but if they did this one thing you wouldn't have found the issue we're going to find the issue next time

I hope I hope if not you know how to find me make sure you find me please find me I do hope you have more reward for me yeah yeah good okay hope you enjoyed our fun our fun skit

there any questions yes

foreign

we actually had a private bug Bounty program for uh I believe like a year and a half before we even did a public one um so we did some private programs and then we had our application one I think it's good to start in the private space as you get the teams used to it um to think about funding how are you going to fund it uh are you going to go to each of the teams or the business units are you going to fund it yourself it's a lot to go to the different business units but then if you take that funding you're responsible for keeping funding um executive make sure the executives are aware you

might have to prep media are you going to do a soft launch you're going to do a pro like a big launch um and there is a bug Bounty community of Interest which is um there's a lot of people in the industry but it's you don't act as that industry in it um and there's a lot of good advice uh that you could get from learning from other people they've posted a few blogs and some things already we could certainly talk offline but um a few of us here uh have a bug Bounty program Intel has a really fancy one um I'm still simple um as I I learn to grow with it because I didn't know what was going to be

coming um for me we have it on the application side like ardell.com and our product side so you know when you have the web apps you will get a lot of activity yep any other yes

yeah it's a the question is is um if it's a uh if the issue is an oem what do you do um and I have to tell you it's sort of not a one-size-fits-all it depends on the maturity um of the odm um is that odm even a naming Authority do they even ever assign cves do they have security advisories all of that um if they're more mature uh we certainly can figure out how to pass it off but there's other things that go in the background like are do we even publicly say that we use that odm for the code or not or do we just you know white label it as our as

our company um so it's not a one-size uh fits-all um in in the past we've had a mixed bag where we would just assign the cves themselves you know for for that odm but with the bug Bounty program it's brought into another complexity because it's like do we pay or do we push it off to the odm um in the end we want that researcher to be happy um so most likely we'll probably still still pay or figure out some way to pay or some kind of recognition um but it's certainly not a one size yep

a lot of researchers English is not their first language

it's difficult ones you want to take that one um Google translate helps no it uh it can be hard it can be difficult um but uh I mean if if you can get translators in or bring in the um the technical expertise if it's around the technical language leverage your resources where you can I think email is is a useful resource instead of getting on that phone we we tend to not get on the phone with researchers all that often um because you just never know what's on the other side um and and so that unknown can be really scary in in the call um so email gives it a pace because you can take your time to respond or

understand and then the the receiver if English is in their first language can take their time to re to respond yeah getting on getting on a call with a researcher could actually bring in more hostility especially if they're already in that state um so yeah email yeah it depends like there's some researchers that they're dealing with 50 you know reports all at the same time so they're busy they're not stressing they know you know big companies got this but then there's some researchers where they just think that they found that the the biggest vulnerability that there is and they're only focused on you and and you're in the background we're dealing with I mean we talk about open source we're dealing

with hundreds of vulnerabilities all the time so you know managing that and making sure that your team is staying up with the comms to the researcher even touching base you just say hey you know just touching base we still we still good like everything's still on track um it takes two seconds to do an email and it might save you from a zero day so I can't express how much to to communicate okay oh back there

yeah so the the question is is starting up a bug Bounty program did we see an increase of malicious activity um so through the bug Bounty program we utilize um one of the more top ones you know there's hacker one there's bug crowd um there's what is integrity Integrity there Intel is using um so um there's supposed to be some safeguards where they're watching like if um we if we have an incident where where a researcher went too far um we reach out to the program and the program we'll talk to them give them warning half the time it's a lot for Education they didn't realize what they were doing they were so excited into

what they're finding that they dropped a web shell because they wanted to keep finding more but you're not supposed to drop a web shell um in us um so you know I think a lot of that is educating the researchers to say hey you you can't go that far or or you know if you feel like you want to go further talk talk to us um so I think we saw a little bit of an uptick of it but I do think in general people were having the right intentions attackers it's their livelihood they don't want to get kicked out of one of those platforms um you know there are certain people that we did a like a soft launch with

our bug Bounty program we didn't go do a whole media Spiel or anything around it basically because it was our first time heading into it we didn't know what we were going to get um and I think that slowed us into the process so it's it's a mix I think having a bug Bounty program though I a few years ago you would have said bug Bounty and I said no no way but you know the whole industry is doing it now and these researchers are actually finding really really good things um and if we figure out how much we could what we get out of pen testing versus the bug Bounty program we're realizing per money per value we're

getting a lot more out of the bug Bounty program because we only pay if they find something where in pen testing you pay regardless if they find something or not so not to say that you should not do both you certainly should do both but it should be part of your sdl's is extra testing yeah oh there's a question up there

oh that's a hard that's a hard question the question is is how does uh dell compete with nation states who are um for for their finding bugs um I I don't think we can compete uh I think that um you know this is why having a good relationship with your researchers and growing that relationship makes sense so that the researchers actually want to come to you they want to do the right thing um but we can't stop those that don't want to do the right thing unfortunately we could just incentive like try to bring them to the to the side um but uh I mean we can't we just can't pour that money into it you have to

think with the bug Bounty program I love it and I like rewarding researchers but that same money is when we're not maybe taking away from other security practices so we have to have a good balance and we just can't pay the way that that some others can thank you okay thank you [Applause]

I have to say bug Bounty program single best investment I ever made in our security program love all that thank you so much and with that yes we're right at noon so we're at lunchtime food trucks are outside uh please go I'll go grab some food grab a friend have some great conversation and join us back here again at one o'clock for the next talk

it's a damn laptop right it was a great talk thank you I mean we've been learning ours for about 18 months now which finds our pin tests are like completely non-events yes nothing nothing contestants never find anything we've had definitely tons of comparisons were we like 60 Grand pen test like this findings funding and like we're like 10 000 on this and we found this this and this it it really is yeah I mean I think there are some security researchers in India that know our product Better Living our own Engineers I I definitely I definitely believe that yeah oh great talk I really appreciate that thanks

[Music]

[Music]

each other

um all right [Music] so there's

something yeah

it was trying

to fight

what stage

you start to go to this uh

woke up and then we have we have different stuff

you know because

our dreams right

I had one you have one right

foreign

[Music]

definition

foreign

foreign

foreign

yeah so like she's been screaming it was a good thing you were standing there because you can't see the screen at all [Music]

[Music]

foreign

covering time

management certainly do more aggressive from now on especially this point now so I can't remember things but that was easy to point out again and again and again you're going to repeat it

foreign

I like it thank you

thank you very much

thank you

foreign

foreign

thank you

a good job

all right

thank you

foreign

foreign

hmm

foreign

all right

foreign

[Music]

foreign

foreign

foreign

foreign

foreign

good morning

foreign

foreign

all right

foreign

foreign

foreign

okay

all right

foreign

foreign

foreign

thank you

foreign

[Music]

all right

okay

thank you

he checks

what's your topic

machine learning feature

yeah I mean I wish I had time to finish a resisting AI business it's a short book it's a good book it's amazing most they shouldn't be doing this security depression you should probably yeah read a lot of benefit out of our like threat modeling practice you know in our absent program it ends up being a lot more beneficial than who it is let's see if this works I'm kind of interested in uh what everyone else's experiences because I've definitely heard [Music] definitely heard some people have like all over the place uh

okay I was like it's just gonna be me

foreign is it execute the python script show area

okay

yes I brilliantly encrypted files remotely

in summary likes documents pictures uh under each current user because users are free to change their location the problem is that Microsoft included this existing digit study in a protected for that by default Microsoft should have that users adds them instead of including them

so how do we address this program the first is to add the folder you want to protect yourself however I do not recommend recommended this because many researchers have found holes in Los Angeles production and many more may be found in the future second always backup your important data naturally the backup destination should be a other than PCS media Nas could I and Etc I would be even better if it's possible to manage Our Generations please don't ask for each product is better I don't know the specific product name

finally as you can see this this time I could encrypt the user's data in a very easy and very ridiculous way it was so easy method that you probably thought you could you could do it yourself but please never create using this method okay my presentation is over thank you [Applause]

foreign

tomorrow it says repeatment what is the most important part of backing up for a copy of back doors and breaches

and as a note if you happen to be back doors and breaches enthusiasts for the silent auction at the end of the day we have an autographed set of them foreign Jason and Deb coming up next we've got Alexander Rubin he's going to be talking about some pretty cool microsql and pro score sequel hacking techniques one other quick bit of bookkeeping if you were waiting on a t-shirt we do have those available out front now for anyone who didn't buy them ahead of time you can go get them for 20 bucks and now we've got Alex Alexander thank you let me set it up [Music] foreign

foreign

no

all right sounds good this is my first time speaking at a theater so I don't know if it's normal that I cannot see anyone because of the lights I probably you've made that [Music] all right thank you all right cool so um my name is Alex I am um principal database engineer at Amazon web services and I will be talking today about databases and specific um security problem which is called confused Deputy problem and we'll talk about how this will apply to the databases a little bit about me I have been working with Maya skill my background is MySQL Consulting pretty much I started 2006 uh starting working for MySQL EB the company behind MySQL I was doing

uh Consulting there and my security interest started as a hobby about four years ago I started playing Capture the Flag games and developed a lot of interest into um a security into red teaming and I joined Amazon web services about two years ago and I switched to doing security I created what we called RTS red team and currently leading that um a team at Amazon web services so we're working with a relational database as a service which make customers make it easier for customers to start and manage their relational databases so what is confused Deputy problem so confused Deputy the deputy here is a computer program so it's a computer program that have um a number of privileges high privilege

program and another program take that program that high privilege program into doing something usually that's basically a privileged escalation and what I will do first I will show you how this can look at Linux and then we will talk about the databases and see how it can apply to mySQL so let's say that I'm a system administrator and I wanted to fix a security problem so I realized that this unprivileged user ec2 user on my ec2 instance has some files owned by root and I created this simple Chrome tab a crown job to fix this and basically every minute it will end putting it back to what it should be unprivileged user easy to use

right so what can go wrong here and to demonstrate that I recorded a quick video uh what may happen so here on one side I have a root access and I have this current job uh and um I am creating by mistake I am creating a user in the home directory and this user will be owned by root and then my current job will start in a minute and we'll fix that right because we don't want to have uh files in a home directory of unprivileged user owned by by root because in this case user will not be able to do anything with that right so in a second here we go so it fixed

right and then we have this unprivileged user and a privileged user can actually confuse the crown job making bad stuff so I can create here a Sim link to Etc password so I can Create a Sim Link in any directory which will point to any file even if I obviously if I'm not owner of that file right so then in a minute the current job will run and what will happen is that this shown command Will Follow The Sim link and change the password file to my unprivileged user so an attacker I bet after on that side on the unprivileged side will be able to trick the privilege program the current job into doing something to obtain an

additional privilege obtain a privilege escalation so now it is owned by ec2 user and I can as a easy to user I can edit Etc password I will create another root user or I can simply change my ID to be root so this is how this malicious user and privilege user got the privilege escalation right so this is the example of confused Deputy problem for the databases so let's go next so let's talk about the databases and in this case I will be presenting a sample architecture for MySQL in this architecture this is completely fictional scenario right so in this architecture we have some Health Care records service software as a service and they have their main application

which is highly privileged and highly protected and they also have a WordPress site they just wanted to tell about themselves right so there's nothing interesting here and then what happens our bad actor found a way through the SQL injection in some of the WordPress website modules for example and was able to get into the WordPress site but there's nothing interesting here so the WordPress database is pretty much the web pages so even if and this user MySQL user is redoing so there's nothing interesting there but what the attacker really wants to do is to get into the health records database where there's a Phi information health records and stuff like that so the question is how can an attacker

can potentially do that so to answer this question we need to look inside the database so as a privileged user I can see all the users in my database and what I can see here is that I have an admin user like a root user on Linux I have two dedicated users one user is for the WordPress database and the second user is for the healthcare service so my application has basically two applications here right so the first application uses the WordPress user uh and the second application use the health data service user so uh I also see this there is a monitor user so what is this monitor use so to answer this question we need to

look into the actual permissions so we can see that the a WordPress user only have privileges to the WordPress database and nothing else and the similar Health Data service user has privileges on the health health data service that is but the monitoring user which is used for performance monitoring usually has a Global Select but in addition to that it has the X you can execute the stored procedures and can execute can do a function hold so now the question is can we can a bad actor we can act confuse this monitoring solution that is using this monitor user to give a work as user privileged escalation to read the Healthcare System so if we are the Corp WordPress user we

are unable to read the mindscale.user MySQL that user is basically uh a table which stores both usernames and passwords so think about this as a both Etc password and Etc shadow if you have an access to that table you will have a username and you will also have a an access to Hash of the password but a WordPress user has no ability has no way it can it can read from that table at the same time the monitor user because it has a Global Select can read that and it can also read this authentication strain which is actually a hash of the password so you may be wondering what this monitor user is used for and what this

database performance monitoring system do let's do typically in the database world we use database monitoring systems to collect database metrics to collect slow queries and to actually generate the explained plans so explain plan is the way to see why this query is slow what this query is doing so the typically database administrator again review the database metrics and collect the slow queries and a typical database administrator job is to actually run the explain plan and see what this query is doing and understand how to optimize that and some of the monitoring systems performance monitoring systems they are doing that for you so they are displaying the queries the slow queries and they also execute the explain plan

and they execute the explain command on the query so it will be very handy for database administrator to use that because they they see it on a single page so I think you need to explain plan what either DBA or this performance monitoring system will do is to take the slow query and rerun this slow query with explain plan so we do explain and then we can see something in the MySQL world something like here right so it's using index it's using and scan this number of rows something but the question is will it re-execute the select right it will not re-execute the select because it's only need to take the statistics right right actually in some cases it will

and this is the worst part and this is the part where Maya scale actually didn't do a very good job on security um there was a blog post a couple years ago written by percona the company that's doing MySQL consultant Consulting and uh what it actually describes is that there are certain conditions where MySQL explain plan will need to execute the part of the query so in the database world we have this select query but we also have this notion of sub query and a sub query is a way to execute part of the query basically materialize create a temporary table as a result of this sub query so to generate the the good explain plan

my skill will need to do this so if there's a sub query it needs to calculate the number of rows in this sub query to generate the right explain so to do that it will materialize and re-execute this query so this is what happens here we have a sub query which is doing sleep and when we run the explain plane what we see that this is run more than an hour because it's sleeping it's waiting until this part the select slip part will be materialized so now we know what to do a bad actor knows what to do so how do we escalate our privilege how do we use this uh confused Deputy problem to escalate our privilege

this database in the worst WordPress that is this user can create tables functions student procedures anything so we'll start with creating this function which is called exploit in this function will be using this again that way specific thing which is called SQL security so SQL security um works is um I need to talk about this if this slide though so what SQL security is it basically allows to run from the with the privilege of a specific user so with this example uh it will run normally whoever runs this uh function will the the Privileges of that user will be applied here so what this function will do it will check what is the current user meaning

the user that is running yeah yeah if it's if it's any user except for the monitor user which is sleep because we'll need to generate a slow query however if this user is monitored we will select the authentication string from MySQL user right but the monitoring user has Global Select so monitoring user if monitoring user will rerun this query will be able to retrieve the password but the question is how do we pass the password back so again we have the ability to specify this definer and SQL security clouds in the database so the definer part works like a sewed bit on lens so if you run this functional privileged user run this function it will still be

run in the context of whatever user will be specified as a definer defining sort of owned file so now we can create another function which will save meaning insert into a table whatever we'll pass in this function will be run with the privilege of WordPress user so inside of that function we have the ability to actually do the right into the database that is controlled by attacker so now we can create this proof of concept here again we will use this SQL security invoker so it will run in the concept of in the context of user who invoked the function basically who did the select then inside of that we will select the password put it in a variable and then

use another function press the variable into that function and that function in turn will be running in the context of our user which will be able to write to the data so my monitor user actually doesn't have any ability to write but because if we're using this function which has the definer part we will be able to write so this is a trick we can now select obtain the password and save it into the table that is controlled by an attacker so I have created this demo and in this demo I have everything here everything set up my two functions right and then I'm running from that um Corp WordPress user because this is the

user that attack you got and then I have prepared this table to materialize my password into right and then the next thing what I will do is I will generate the slow query I will make it slow and will make the sub query here and I will simply do select WordPress exploit all right this is slow query it will run in what 30 seconds right 30 seconds now a DBA or even the monitoring system will pick this query up and we'll need to run the explain because it's slow query so the GBA let's say DBA runs this and GBE doesn't understand what happened right it's not a slow query now so now but then now the attacker

will select from the p here's our password so we have obtained the hash of the more privileged uh user so now the question is what we will do with this right so this is our monitoring user and our monitoring user has much more uh interesting uh stuff it's a it's a obviously privilegious collection but the question is how do I log in right it's a hash of the password so my hashcad instance with the uh eight GPU cores and we'll run the and try to uxd um actually brute force that path so let's see what will happen right so this is my password I removed the story and from the beginning because it's my skill specific and then put it into a

file and then run hashcare so let's see how long it will take wow it's done so the password is pass p-a-s-f that's the password of the monitor menu right and the reason why is people never pay attention into securing this the you know users for the monitoring system right so they think that this is this is unprivileged user nothing nothing more than that it's just for monitor right so in this example I was able to get the privilege escalation retrieve the password take the hash of the password and hack the password so now in attacker is able to use that to connect to the database and pretty much download the whole Health Service database including Phi

patient name whatever it is right so recap so what we did is this is a database specific we use the confused Deputy problem and apply it to the database and we use MySQL as an example so MySQL MySQL explain plan can actually execute the statement and it should not and another problem that I have demonstrated here is that monitoring user has Global Select and also executable execute privilege if the monitoring user don't have execute privilege then this whole attack will will not be possible so as a result we got monitoring user password hash it's simple password easy to crack and uh an attacker can connect as monitoring user now and because monitoring user has a Global Select it can download the

whole database and start selling it yeah whatever all right so this is the picture it's a fictional scenario uh an SQL injection to the low um privileged database a database that has nothing in it pretty much um going to the MySQL allowing an attacker to actually switch to the database use the confused Deputy problem to switch to the database and download the calculator um what about postgres I don't have a proof of concept for podcast yet uh that's much more complicated uh but still possible when I will have a proof of concept on the next conference I will do a demo for posters as well and that's all what I wanted to talk about today thank you very much

[Applause] uh two minutes four questions

yes foreign

so basically the questions there are two questions here right uh is it isn't there really a bug in in my skill we um it's it's debatable if it's bug or not we think that this is bug I personally think it's bug MySQL developers don't think that's a bug so so um technically it's uh it's a design choice but from the security perspective it's very bad uh other databases may also be susceptible to that there are certain conditions then I haven't talked about that but there are certain conditions where posts SQL actually execute the uh query even if you just do explain so in both MySQL and postgis SQL there's explain and there's explain analyze when you run explain analyze it

is actually executing the query the normal explain should not but this is what it is right so um both MySQL and positive SQL potentials for that so to fix that you need to be super careful of what user is running the select queries and who is running the explain plans and whatever whatever privileges you have on uh on that user that is re-executing the the custom career pretty much another question all right thank you very much [Applause]

thank you now I've got three more sets of back doors and breeches in hand if anyone wants to stand up and tell me something cool that they learned today they'll get a set and if you don't feel like talking publicly I'll tell you that we've also got some sets out by the t-shirts that we're giving away as well so if you want to set either tell me something cool or go out there and grab a set and part of why I'm doing this is as a reminder if this is your first time at besides RDU prior to what we do is a silent auction and if you go visit the vendors and we hope that you do because

they provide the money that lets us do this for free they will give you fake cash that you can then can use to buy things like the autographed set we've got a bunch of books by Cliff Stoll who was one of our most infamous keynote speakers he had us uh running around grabbing a actual overhead projector old school 1950s Style for his presentation so three sets up here or a couple sets outside and otherwise I think we're scheduled to be back in 10 minutes or so 15 minutes

what about that

okay

foreign

foreign

[Music]

[Music]

foreign

[Music]

[Music] thank you

[Music]

thank you

[Music]

foreign [Music]

foreign

[Music]

yes

nothing before you it's called the whole setup so yeah

so of course just um

right now

but it wasn't perfect

they're not um

you know

foreign

foreign

foreign

so

leadership

services

like Moscow

there's problems

foreign

[Music] foreign

thank you but only when it's close

almost out

thank you it's my favorite foreign

yeah I thought about that don't get it from the pictures maybe but I'll explain thank you though I'm assuming you know we had to look it up did you giggle when you when you got the result all right all right oh God no no you know what I never thought about that now that somebody pointed that out that is kind of okay yeah yeah

I'm good to go yeah you just tell me when to start I had to go to my backup plan but

I can start at 2 45 right on the back foreign

foreign

Hello everybody welcome back from the break um people like jokes I like jokes all right let's let's test how we've recovered from our carb loading at lunch knock knock a new kind of Port knocking that's who so one thing I had to switch to my backup technology here and so it's a little weird I have to slide if I'm a little off somebody just pointed out the slides aren't heavy we'll get through it if you were here last year you know how I started it's not with jokes my friends if you're new well here we go I start with questions to get us warmed up this is going to be my question to you and

I'm going to select a couple people at random so you cannot hide if you think I can't see you I can and I'll smell the fear like a good predator and I'll call you out here's my question I'll give you a second to think about it why set the rest of my presentation up what are Empires built upon think about that what are Empires built upon so what we're going to do today together is we're engaging a story I'm going to tell you a story I'm not going to tell you who I am my name is Jason by the way that's all I'm going to tell you because I don't know what else matters you can ask me any

question you want at any time and I'll answer if you're curious about something about me about the presentation about my thoughts on just about anything but I'm going to tell you a story when I was teaching at High Point University in High Point not far up the road I had a student come into my office one morning and if any of anybody ever remembers what it was like to be in Academia whether you're on the faculty side or the student side or like me you ended up doing both over your career some mornings are rougher than others some mornings are rougher than others that's code for I had a hangover so everybody remember the question what

are Empires built upon my friend right here as you took a drink you knew it was coming what do you think Empires are built upon first thing that pops in your mind Emperors that's a good answer my friend right here in the glasses in the black shirt beer oh man I'm thirsty fear and Emperors those go together right my man in the green ideals besides Raleigh I'm gonna tell you that Empires are built upon the bones of those beside you so I want us to look around to our left to our right these are our sisters and brothers and warriors who together as a community have the moral obligation the moral obligation to build the Empire

of cyber security together and secure the future are you with me

skulls for the skull throne come on it begged him so here's the nature of the story anybody who Raymond Chandler was all right you like olds okay Ryan he's sitting with a hangover in his office the girl comes in oh this bad thing happened Raymond Chandler was the author of all the old some of the best fiction ever written I say in the 20th century in America and this was a Raymond Chandler type of morning again I wasn't feeling at my best and I'm here to tell you the truth sometimes as much as I love lecturing as much as I loved presenting I didn't really want to talk to anybody especially on this Raymond Chandler

morning and I heard the door open up to the office suite I was the only faculty in the office that morning and I thought to myself here we go and I heard the footsteps coming towards my office and a little head poked in my door and he had this floppy hair I mean he looked like a muppet now I can say that because we actually built this technology together but he couldn't show up so I can make fun of him all day and he's not here to defend himself his name's Kyle one of the best students I've ever had in my life and he said to me is Port knocking detectable and I said do you even Network bro

now who in here has ever heard a port knocking be honest all right a non-shelly amount of people of the people that raise their hands have you ever used it either on your own systems on a small office Network in a giant Enterprise who's actually used it one in the back right there three that cut like in a third so of all the people that raised their hand that have ever heard of it probably less than a third of you have actually used it so the people that use it do you still use it a little bit okay right on he said to me I know the OSI model now if you've ever taught undergrads when

somebody says I know the OSI model my response is well then let's find out because now you have my attention not something a sophomore or undergrad usually knows so a little bit about Port knocking because the majority of you didn't raise your hand I'm going to assume maybe you've heard the word because I've said it like 10 times already but you out don't actually remember when it came out it came out in 2005 26 because it started with a really simple discussion on like a Usenet listserv type thread which was in a firewall rule what is the difference between drop versus reject or deny whatever the language is anybody know

that's right and that that reply often is like a combination of reset app right that is a big difference because if you reject denying you send that back you're transmitting which means by inference my friends what that you're there and listening if you just drop and just kick it in the trash there's no reply nobody knows you're there what a revolutionary concept and from there some clever people said hey wait a minute hey wait a minute what if I write a service a simple demon that interacts with a host firewall that listens for incoming ports drops them but then does stuff like who's there me because I know the knock sequence oh man that's really cool isn't it

and it does work it's a really simple concept because of that and one other thing this is so Paradigm shifting we're used to connecting to a system putting in a username and a password hitting enter and if it matches it lets us in right we do that like a thousand times a day I feel like I do this system says no if you know the secret knock you're authenticated by inference so once you knock correctly I'm letting you in that's a big paradigm shift and so Kyle and I had this long conversation about can you detect Port knocking and we published some research on it we went on to do bigger and better things he graduated and he's getting married

and I don't know what he's doing but he ain't here this weekend so now I can tell all kinds of crazy stories the core principles of Port knocking when it came out in actual software because it exists there's a dozen or so Forks of the main project there's all kinds of versions the core principles concealment not obfuscation we're concealing via that draw but we are actively listening for the knock sequence and the idea is that instead of exposing something like SSH as a common form I can conceal it because of that behavior and if you know the knock the service just opens up that port and now you can connect so think about that that's a linear chain of events right

knock and there's different forms of do I knock once do I knock twice do I knock three times four There's No Limit you can change it you can build your own I did that's what I'm going to talk about for the next 40 minutes and the authentication is baked into it really simple elegant Solution that's Empire Building here's a cool little graphic isn't that the best art you've ever seen on a PowerPoint slide yeah the client knocks once twice Thrice PKD that's just some abstract version of a port knocking demon says yeah you knocked correctly those are the right ports Because by the way Knox are ports so I just send a packet it could be send

flagged super simple to a destined Port so let's say six thousand seven thousand eight thousand why not let's just go in thousands in order and when that's correct the demon says hey that's correct just open up SSH and now anybody can connect assuming that the only person that's going to be trying in that millisecond is the person that knocked foreign is it detectable who thinks it's detectable yeah yeah yeah of course it is of course it is it's not encrypted and even if it were it still has a header that has to be visible right and somehow you're getting to those porch you can't totally hide that otherwise then I guess it is a fuse a

kid and I can't even see it like that back road back there I need a visor like a blackjack dealer but there's two caveats because now we've got to be honest hypothetically or in a lab environment have you actually ever tried to detect Port knocking has anybody actually ever tried it in a lab were you able to detect it okay me too check this out what tell me if you agree or disagree with this then because we're the only two in the room they've ever done this experiment right two caveats I'm gonna put on this tell me where I'm wrong one you have to be able to tap the traffic obviously right because I'm going to

capture it so how do you capture Port knocking on the internet has anybody ever tapped an internet link that's gonna admit to it all right okay so that's one caveat you agree with me right my friend thank you caveat two that that link cannot be super noisy because if it's super noisy full of all kinds of traffic how there's just too much volume now of course we could build some advanced technology to parse that all out fair enough if you're clean with wire shark or two speed dump you can build some filters but how wrong am I thank you my friend so with that in mind now we can step back and say okay look we got us cool

elegant solution for concealed remote access that although in practice and principle it ought to be detectable and it is there's two caveats that that probably like you're not tapping the internet and even if you could it's probably too damn noisy you ain't gonna see it okay but let's say you can or can't we'll put that aside are there any problems with it from a first principle so think about this think about how I describe this word what do you think what's wrong what other vulnerabilities in the system are there it'll alert a Defender to you it's like if you're trying it trying to brute force it yeah and so on that we can infer something

there's no client authentication the authentication is baked into the port knockings but I don't know who you are anybody that knows the knot can knock right I mean anybody ever hide behind the couch when somebody knocks on the door and you know they're selling vacuums like I don't want to talk to you right what else any other ideas yes yes great idea how about this what if I am able to capture the knock we just inferred that it's not client authenticated I can replay it right I'm not checking like sequence numbers and deep packet inspections I'm just looking for TCP ports think think okay come on in okay in fact there are a trunk full of issues

with Port knocking if you're analyzing it in like a first principle way I clearly some of us it hasn't stopped us from using it right but that's not bad but then the question becomes and this is part of the story of curiosity we know it's detectable at least in certain cases we know it's an elegant Simple Solution it works it works for sure it works but can we make it better can we do something or a set of things to at least negate some of these systemic issues and yes we can anybody ever try single packet authentication some people call it single packet authorization it works pretty good too isn't it and it's basically just like one UDP packet

stateless and you're good to go that's a really simplified explanation but I'm going to get into it later because we baked it into ours there are some deeper impact packet inspection schemes where they look at sequencing numbers in addition to the port Knox uh some people have tried some really slick stuff with cryptography using and encrypted basically ipsec vpns but if you got that why do you need this that's kind of screwy and now Coleman that's our solution how cool is that it's the badass assassin's name of all time I invented it so here's the design and I'm going to tell you up front I was originally going to run a demo of this but for my friends that have used Port

knocking or anything similar to that it's super boring to watch like it doesn't show you anything not like that cool SQL stuff that was slick that's a good demo or the windows hacking those are good demos watching Code yeah so I'm going to show you some code as I walk through this but we started from a design how do we Design This from the ground up so that it doesn't have those issues or at least not most of them and we get something that's robust first things first it has to be functional it has to work and from the get-go we designed this to be Enterprise ready not my home lab with like my wife thinks I have 20 computers I

actually don't I have like 22 I had two in the closet she didn't know about it that's all right it needs to be multi-user that's the other thing with Port knocking if I want you to knock to my server and I want you to knock to my server now I do I let you both have the same lock if I do how do I tell the difference do I give you two different knocks now I gotta run two demons two servers see that's why there's two in my closet you got the joke thank you thank you thank you here's the other thing too by the way I didn't mention this is important when you knock do you know what server you're

knocking to you assume and fair play right it's probably yours but you're still making an assumption so there's no bi-directional authentication either back to this we wanted this to be pseudo-anonymous because it's multi-user and so when I show you client registration I want to impress upon you that we have no way to know who somebody is we only know that we registered them or when you register your friends and family and the community cyber security you don't have to know your name's John I made that up I hope it's not I just know that you're in here and I have your keys that's all I have to know about you pseudo Anonymous has to be

resilient and it needs to scale and be centrally managed it's the 21st century I'm not building and deploying software even in my home environment for my own use that isn't centrally managed I'm not doing it anymore dudes not doing it so we have a design in principle now let's design the overarching algorithm of how it's going to work first and foremost we register clients so there's a facility to do that when we do that we do a set of sub things we generate two sets of keys a symmetric key and a public key set okay the next piece assuming clients are registered now the clients want to interact with the system this is where our version of spa comes

in okay we have what's called we call it Preamble and so it's a one packet blip when the software's in debug mode it will send an act just so you know it's working you can turn that off and it's just a little blip that contains a unique hash fingerprint the counter I'll explain the counter in a second and a little bit more information what that does is it triggers the system to say a registered client is getting ready to communicate with me I'm going to set up the knock system and then how does the NOC system actually works here's the secret sauce but it's not a secret anymore because I'm going to tell you about it okay

we use otps to generate the port sequences there's no static ports in this system they rotate every time a client sends this Preamble successfully and the server recognizes them now come on you've got to give me an Applause for that right that's some dark sorcery I think it was clever it's an elegant riff off of it because that's one of the problems with fortnite that's why you can replay it because it's the same ports every time so we thought to ourselves from first principles well how do you avoid that you use an OTP and you just change it who cares and then it's all IP tables after that this is the implementation there's three parts

Three core Parts there's Conan Coleman server and then common service really breaks down like this the heart of all this is in common underscore server that's the TCP level demon that's what's listening for the Preamble for the ports that are the knocks it's what interacts with iptables that's the core common is a command line interface to the server so when you want to register clients you can de-register clients you can reset counters you can revoke keys if I if I have a traditional Port knocking system and I let you connect how do I kick you out I mean I could change the ports but then it breaks it for everybody else too in this system everything's unique to each

client so I can revoke your access I can revoke your keys okay that seems pretty cool and then come and service goes with common anybody here familiar with flask python flask that's a python flask that here's why because we want to build a web admin panel for it because I told you it's Enterprise that's why as much as I love command line I'm getting old people my hands are slow and brittle so I like web so that's not implemented although it's there and it works so Komen Works through common service they can interact with co-min server to get a list of clients to list the keys register deregister revoke all that stuff and then the common server does the

heart of it that all breaks down in the code into two parts we call them The Supporting Cast there's three crypto handlers symmetric asymmetric and we need hashing and then there's eight utility handlers these are the utilities like registered clients um do what we call the remote access sequences fancy name for Knox they're just not knocks anymore now it's a 15 digit hash based OTP that gets broken down into three knots that's also configurable because I told you it's Enterprise all enterprise software is configurable right and so you could change it to one knock of five numbers you could scale all the way up to well I don't know see how much you can take and there's some

other utility handlers in there all right I promised a code by the way by the way if you like this code because I think it's super sexy I wrote it if you think it's trash python Kyle wrote it easy way to tell another reason why I decided to show the code is because I also comment my code and you can see in here we have some problems you'll see in the comments that I've left myself little notes that because this is beta we have some things to fix some of you this is who in here this is your first like conference ever anybody yeah in the back right all right I was I had a first day too we

all do and I had no idea I would ever be up here showing this I had no idea I could build software on the scale not on the first day but now I'm like on day I don't know what's 10 years times 365 a lot okay that's just a Time problem right so the other reason why I wanted to show the code with the comments because this is in our repo I'll give you the link to the GitHub at the end anybody can do this this is why we're all brothers and sisters in the same Community you're capable of this if you want to do it and you put the time in so anyway back to pseudo Anonymous

registration we generate Keys asymmetric symmetric we look for some client IDs this is really trick and clever how do you think we build a client ID because that is unique per client close good guess it's not the OTP it's the fingerprint of the key well it's a public key system so you need that for signing anyway like I'm not giving you anything if you're snooping like you could already derive that okay cool but it's Unique to that key which is by inference unique to the client last but not least the counter here's why we keep a counter because it is an OTP and so your eight OTP based on your ID which is a fingerprint that's unique to your keys

when it starts at zero dictates which OTP gets generated and then every time you start that process the counter goes up in the Preamble negotiation I'll show that in a second we have a mechanism to check for that because of course with otps you can get desynchronized so we have the ability to re-synchronize the counters see we thought of everything didn't we who wants to pay me a million dollars for this right now sell it no takers okay I haven't seen you already yet here's the preamble we actually thought of this last we built everything and it was kind of working and we're like man but something's not right dude like there's some there's some issues

that we didn't cover from that trunk of issues like what are we gonna do I'm like wow why don't we get a preamble right like when you knock pardon me when you knock on my door I say who is it and you're going to say it's Frank now if I know it Frank which I do and I hear your voice and you sound like Frank I assume you're Frank or maybe I got a people I can look out or both that's kind of the concept of the preamble so real simple it's just the client ID it's the rack sequence so a remote access code is an individual knock all of them put together in one string is

the sequence and we send them all in a burst okay and then the current counter the server checks that against its database of registration it checks the counter for that client and it generates an OTP to match it to see if it matches pretty mature OTP and then like I said in debug mode we send an act just so you know you can turn that off and it just gets dropped you don't know anything but the important part last but not least what this does is it sets in motion the interface to the IP tables by the way if you're curious this is what it looks like when we generate the sequence and so by default this is set

to 15 digits long do the math that's three remote access codes of five digits each why five digits well what's the maximum number of ports 65 535 and so each sequence can range from zero to the max each of the codes in the sequence you can configure that you can have it be five of those you could have it be one of them if you're feeling spicy that day I don't know maybe you want to try it I did it works it worked but that's important because of the space that's an almost I'm gonna go out on the record and say unbrute forcible space and then here's the part of it what I'm showing you here is the last

step is anybody familiar with IP chains and iptables okay I'll give you a quick breakdown it's actually really cool technology built into it you have a firewall and the firewall is a list of rules in the process sequentially right and then in IP tables this is Linux um you have different tables one for forwarding one for input one for output so you can filter on that table and that's the direction of the traffic right you can put chains on those tables and a chain is exactly what it says it's a linked list like a chain of links right and so the way this works simplistically explaining it is we set up a chain of subtables that represent the first

remote access code the second one the third one and then the fourth one which is the SSH access and those match the OTP this is why the client has to send the preamble that comes in registration gets checked if okay generate that off that sequence and now you're set and so the knot comes in the knock comes in that's the sequence server checks it and then generates the chain client now sends the remote access code sequence in step pass pass pass SSH time quick aside when we first did this we were thinking about what we tried a package to interface with iptables because it had a nice wrapper too and gave you nice method exposure this is

all object oriented um but man it made it really hard to read the code and figure out what was going on and because of the complexity and the space of possible most remote access codes we have it got tricky and so I made the executive decision to say screw it let's just do sub-process calls and write the iptable commands just like you would if you're working from the command line I don't think that's better code I think it's more readable code which makes it more maintainable that was to be transparent the decision now yeah and there you go by the way real quick with chains too if you're not familiar if you send the right first

code with the wrong second one you get kicked back to the first one that's pretty cool right if you get to this one and something goes wrong it'll kick you all the way back to the beginning all the rules are also set with expiration timers so this is only live for connection for the whole set no access code no access code no access code SSH for I think it's 30 10 10 seconds and that's