Matt Bosack - Kettle of Fish in a Barrel Cloud Automation for Subdomain Takeovers

[Music]

hey everyone happy b-sides i am matt bosack and i am here today to talk to you about kettle of fish in a barrel so this is a little bit about the journey that we've been on at comcast xfinity with our bug bounty program and one of the approaches that we've come up with about a problem that we have seen come up frequently around sub-domain vulnerabilities and takeovers

so i've worked at comcast since 2018 and i joined an in-flight team that was working to operationalize a bug bounty program to go public and to accept vulnerabilities and then begin the process of working with the research community to reward their findings and their work for that so i'm a senior product security engineer there still on the bug bounty team it's our pcert product security incident response team and before that i earned some engineering degrees at temple university um with focus on control systems i was there for quite a long time and i loved it there and then got into some other security spaces actually focused on machinery control systems and industrial control systems security

so that's how i kind of broke into the security space in general before going and joining comcast and i have been there on the bug bounty team and uh love that as well so excited to kind of bring some of the things that we've been working on um in that area or with them so i got through the boring stuff this is a little bit of an overview of where we're going to be going today i'll dive a little bit more into comcast and the bug bounty program we have there and what leads us into subdomain takeovers um this isn't a new problem and i'll talk a little bit about that but it's something that we've seen

pretty frequently and so i'll give a little bit of a background on that some of the first researchers to find it and then a little bit of the the technicalities of what they are once we cover that we'll be able to talk about what our solution approach was how we went about implementing it and then some of the findings and the future work that came out of it so it's it's really that approach uh that is kind of the exciting part about it um combined with a little bit of the the bug bounty program and uh thinking in perspectives about this that hopefully you might be able to think about this in a different way also

so comcast from an outside perspective i think is usually thought of as an internet and tv company which is fair um i i think more accurately uh and they'll probably say it's a technology and a media company so it is internet it is tv um there's nbc and sky there's business enterprise aspects to it also um but when you actually get inside and uh this is this was also kind of surprising and cool for me to see also uh it's a full-up engineering and innovation shop so you know everything that a lot of the technologies that you cover in engineering networking programming uh front-end development back-end development cloud security hardware networking if i didn't say that so it

covers a huge gambit of technologies and there's a lot of really great engineers working to kind of bring these things forward for the company um and so as you might imagine across all of those areas and with as big as the company is i think to this community what that probably translates to is a really big attack surface and that's also true and so one of the goals of having the bug bounty program is that you know as securities kind of exploded in the last 10 years and bug bounty has kind of exploded in the last five years or handful of years or so um as well people are looking at this stuff they're not going to stop looking

at this stuff and you know we provide a front door for them to be able to find that if they do come across something in their work and in their research um they know where to go to be able to get it into the proper hands to come to a resolution a lot of the people that are working in security now do it for the the betterment of the community and society so it kind of it offers that um opportunity to do that so we also work with a company called bug crowd and they're they're our vendor for doing this and they offer a great intermediary service for us where it's not just an enterprise versus an individual or

individual versus enterprise or working together or uh they said they said or anything like that um they are a really good advocate intermediary um for everyone involved and you can kind of submit through there they'll also do some triage assistance for us um and we get to work with their engineers and then also interface with the research community through all of the bug bounty programs um there's obviously other uh bounty service providers out there hacker one synack and uh they also offer kind of a augmentation to a security force especially when you have a really large company or you're working in an area that really has nothing to do with security and you don't have

security expertise in-house that you can go out and kind of work with these companies and then have the crowd help you out so it's a really cool space to work in we do have two public facing programs right now that you can find through bug crowd um you just go and search for us either through comcast or xfinity both of them will pop up um we have the the comcast vulnerability disclosure program which really kind of catches everything under on the comcast xfinity umbrella and then there's also a specific subset of the bounty through or with our xfinity home home automation products so um both of those are paying cash bounties right now so you

can go and do work and if it ends up being validated um you know you you can earn some extra money for some red bulls or something um so with this program being launched in 2018 we have had an opportunity to kind of see some trends come across a lot of the things that we do see are a wasp top 10 type things but we've also come to realize that some domain takeovers has been a pretty frequent submission that we've seen throughout our time so a little bit of background on that i'll cover what i think i see as like the genesis of this of where this kind of originally hit mainstream um the two big ones that we deal with a

lot are c name takeovers and dangling name servers um and uh talk about some of the caveats with those and then i kind of talked a little bit about the frequency uh leading into this but we'll cover these things as we move forward so uh these two researchers on the left uh is a hacker one submission from franz rosen and i actually think that he was working for detectify at the time when he wrote this blog and there's a really good talk from him in 2015 that covers a lot of the nuts and bolts of subdomain takeovers he was one of the first to come across this problem of when um your your dns records kind of

go out of maintenance or go stale in some way someone else can claim it uh and set up a malicious service behind that so you can see on this ticket he actually ended up earning 1 680 bucks for this one which isn't bad especially when you see the the relatively i think low effort that goes into um that goes into finding it low effort might not be the the best way to describe it but i think it's it's once you get things operational it's a low technical effort um when when you see what's going on so here's one of the first uh so by no means is this any kind of new problem that we're presenting on

here but this is just another approach of a way that you can go and kind of look at this problem so the one on the right is patrick hudak and he actually has a really fantastic blog that if you're into this i would go check out um kind of does descriptions and playbooks for every type of sub they may take over and when he found this one on starbucks in 2018 i think this is when it kind of hit prominence and uh the bounty community at least really started to go after this and try to sink some teeth into this uh and it really paid out i think in the long run so interesting perspective on just kind

of bounty hunters in general you can there's many different ways to do it and at least two types of bounty hunters that we see um are you can kind of look for everything do all types of scans go after every type of vulnerability every type of business and just kind of do a heap spray and find vulnerabilities and file reports that way or you can also do farming uh so i feel like patrick is or ox patrick is one of the bounty hunters that's really good at farming he set up some really nice tooling around his process um to the point where i think he's even having uh doing api calls to write his reports for him and submit them into

bounty programs and so it's it's really just a cash operation at that point which is pretty cool so um that's a little bit of of the background of where it came from the cname takeovers or the canonical names in the dns system are really uh aliases or symbolic names so the example that's on here is there's a pretty slick service called surge.sh which um does static deployment for front-end websites and when you go and you register at search it'll give you kind of this random domain name this one here is better techarmy.surge.sh so that's fine and good unless it really has nothing to do with the products that you're trying to promote or work on uh and so you can go

and you can buy a domain or you can register a domain you have a domain like we do at comcast and then you can set up the cname record to point to that search domain that it was given to you so now when i navigate to that more appropriate alias it'll still pull the a record and send me to um where that service is being hosted so in the middle there when you first go to a search that sh domain name there's nothing there so it's project not found and then if you dive into the source a little bit there you'll see that for all of the one for basically every service that you can go

and register for if something's not there it'll give you kind of this 404 type of message and each one looks very different so you can go and actually identify a fingerprint or signature of what this looks like and and that'll become useful later so what ends up happening is i register a name i use the cname to point to my service and then i deploy a site behind it um and then it works and it's very cool the problem lies in when i'm finished uh or you know i pack up and i move elsewhere or i'm just done with this domain um i will delete my search account but i'll never actually clean up my dns records so if i

discovered this i can go and navigate back to betterattackarmy.search.sh realize that nothing is there and then i can go and register this account on search and then basically set up anything i want as the website behind this so the problem is that at that point you can abuse the trust in this case of the comcast.com domain name and set up anything you want there so that's the cname type the other type which uh i think is the more nefarious type here are dangling name servers uh or ns records so um right under that title there you'll kind of see if you do a dig of this domain it'll come back as a serve fail so what this means is the dns system

can't fulfill your request because it actually doesn't have any information about the thing that you're looking for in this case on the bottom right part of the screen if you go and you dig the comcast dns servers this is saying that all of the name server records for this particular domain have been delegated to aws um and then when the kind of public domain servers go to aws and ask for information about this domain it doesn't have anything uh and so what this means is that um these particular name servers that it's asking are no longer the actual name servers that maintain the records for this an interesting thing about route 53 is that you can go there

and you can register any domain name for anything in the world the difference being um that's fine because as long as you maintain the records for them the name servers will never point to uh your registration of that it'll only go to where the system of authority says to check with name servers so the the discrepancy that happens here is that in the top right of the screen there's this is the record set from vinyl dns so at comcast we recently migrated all of our dns records into vinyl which is open source and very cool and very useful and has been kind of a great service for us and then in that vinyl system i'll let

the vinyl dns know that these are the name servers that i want to use uh and then so uh i'll register my domain in route 53 bring that information back to vinyl register it and then it's good to go the problem that occurs here is just like with the cname if i go and i delete this or if it gets deployed or redeployed in some way and these name servers get overwritten um and i don't update the records in vinyl now i have that discrepancy that someone else can go and register it so uh some of the things that you can do with that why it can be more nefarious is at that point you have

full control over that sub domain you can create new subdomains under that and kind of spawn out into a whole other different set of problems um you can set up mx records and mail servers and then send emails from the domain that you have kind of taken control of this is one instance of the cbe uh for this type of problem and cvss version three if you're into that type of thing is a medium um there still kind of requires some human interaction to go and get people to navigate to these things um and you know if it's a really wonky sub-domain then it might be obvious that something is going on but nonetheless uh you know there are non-wonky

sub-domains that you can take over um and and do a lot of harm if you're not maintaining your records properly so after we got all of these submissions coming in uh we said all right well maybe this is something that we can be a little bit more proactive about and so we started to make um the the researcher automated framework for information knowledge and insight or rafiki and it was really kind of based off this idea of um being the researcher kind of seeing what they were doing and then picking up on their trends uh and then going out and searching for things on our own um kind of trying to morph it into a

little bit of a framework to to get other trends built into it but the primary focus now is for sub domain takeovers uh and as we talked about a little bit it's really been influenced by the large and dynamic ever changing perimeter that comcast has with all of these dns records um one of the cool things uh that's really promoted at comcast is kind of innovation and rapid innovation and so uh you know we allow a lot of teams to maintain their own records it would be kind of haphazard to funnel all that through one place anyway um so there's a little bit of anarchy to it and and this has brought light on to

that and we're working to address it both from um our team the security perspective and uh from the development side of things so with rafiki originally what we did was uh we glued a whole bunch of tools together so there's there's a great list of things on the right hand side that we basically wrote a bash script around to go out we fed it domains on comcast and on xfinity and went through sublister dns recon and a mass and just aggregated um tens of thousands of records of all these domains that we wanted to look at uh and then at that point we fed it through subject which um was really fingerprinting in the same way that we

talked about earlier to go out and look for cname types of takeovers there's another cool repo from ed overflow called can i take over xyz that maintains a really good list of the types of services that um are examples of this where you register for something you're given a you're given a domain name for it and then you can set up a cname record and then it becomes kind of this race condition if you delete your account someone else can register it and you have to maintain your records so this was the first iteration um it was in bash there's a very brief period of time that we put it in a meta split module but uh we actually

don't really use menosploid on our team so it didn't make sense and it was still very manual it was still me running this on my computer daily so um we wanted to just kind of offload it into the cloud and amazon provided a great place for us to go and do that with this kind of architecture that we built it around so um an engineer on our team went and built a mass into a container and then we just got the authoritative records from vinyl and aggregated all of those together and then pushed them up into an s3 bucket so it sits there and gets batch processed by a lambda um so it's put into a csv and sits in

the s3 and then is run over from a lambda and then just brought into a dynamo and sits there until it sets off a trigger or on some of these run on time delay scans and then each one of these will serve a different function uh looking for these subdomain takeovers so we have a lambda that does um that reads from the database goes and does c name scanning we have a separate lambda that does um that does scanning for the uh dangling and s records and then there's also a module that we have in a lambda uh that will go and then register and delete and re-register uh your name that you're looking at until it

actually pops and grabs one of these amazon name servers so there's a finite amount maybe 2500 so you know eventually if one of those records is available you can go and register it and then at that point you own that domain as well uh if if any of these come up as a positive um and there's some sort of match then we have it set up to um email our team and at that point it's kind of it is back to manual at that point of going and verifying that this issue actually exists uh so for the cname scanning lambda um and if you haven't checked out lambdas they're they're super cool uh serverless way to um do

computing uh off of you know in the cloud but off of your computer um supports python ruby go uh node and then you can actually um set up your own engines uh if you want to use some other language i think java's in there too but so we're using python and so uh it pulls everything down from the dynamo we'll we're using the python requests library and we'll check the cname and then we'll actually do a get on the cname and that's where kind of those fingerprints come in we'll use at overflows database and kind of just have that file that's in the bottom right of that unique signature in there and we'll parse over

on the get request and if there's any kind of match in here then we know that that service is no longer being used and there's a potential for a takeover happening here so at that point we'll send out the notification and kind of just loop over it i think right now um the mass scan for all the different types of domains that we're pulling down is actually in the tens of thousands i'm going to save like 65 000 records um so it's going and then scanning all of those um for these fingerprints if it can pull something down for the lambdas that we have for on the dangling ns again goes and pulls things from the

dynamo um and then we'll actually just kind of do the the dig against it um using python dns and then check for that serve fail if there is a surf fail it indicates a potential vulnerability and we can send it to route 53 which will then do that register delete re-register until it actually gets a match um there's another way that's on here that we're working on we obviously have access into all of our aws accounts so we can get kind of those sources of truth of what's in route 53 and what's in our vinyl and do a comparison against that one of the gaps that's there is um you know with delegating those records

into aws uh it points to that one subdomain but then teams can spin up other numerous subdomains under that which we might not necessarily have the source of truth for so this works really well for for something in that regards um at the end of the day rafiki the researcher automated framework for information knowledge and insight uh has been operational for on and off for a year and we have 54 actionable findings and since we've been tracking this uh in our bug bounty program we've had 86 actionable findings i think since late 2018 so two years um you can kind of go back and look at some of the other bounty payments extrapolate what that

may or may not equal out to it it'll be close um uh but you know we're catching up we're still looking uh we're adding a lot of domains back in uh and that's one of the other cool things about the bounty program is that um people are so creative and they always come up with awesome ways of approaching things or thinking about things in new ways that you see it come in uh and i get yelled at that we didn't find it but then it's a cool opportunity to kind of tweak and adjust um running all this in dynamo uh and in in the lambdas and the s3s is pretty cheap um if you found one uh high impact

domain it would pay for your costs and um a bunch of cases of red bull or whatever so all in all it's it's been successful we're looking to expand on it uh it has it's been uh kind of a joy to work on it also uh and it also represents kind of an evolution in our security posture in-house in that you know we weren't really aware of this problem until we realized the frequency that it was getting submitted and then started doing our own scanning and kind of had that oh yeah all right this is a problem um so it changed the way we do things on our side uh and then now because we're improving

our security and because we're also improving the way that we're hunting things um it also makes uh the security researchers and and the bounty hunters kind of change and morph in what they're doing too uh so not to um be like over philosophical about it but it is kind of symbiotic and a back and forth i guess it's more of a chess match but a very respectful chess match with your friends that that's that's really great so those are the results um where we're going is uh you know there's always new domains that we're realizing that we use on apis or other backend systems so kind of expand our hunt for those um continuing to update and modify

the fingerprints that we have for cnames um pull more sources of information so we're using vinyl and we're looking at the root 53 stuff but you know this isn't a problem that's specific to amazon so uh you know where else can we get data from to inform our process in doing this um and i we have support to do this um it's not in a place to do it yet i don't want to be that guy that says we're going to publish the code and not do it so this is a huge goal of mine hopefully we can share this with the community in the very near future so that's all i got thank you everyone

i i hope that um this was somewhat entertaining and useful uh thanks to the peeps at comcast for supporting me to do um both the work and the talk uh thank you to b-sides and thanks to the community

you