PG - Lotus Notes Password Hash Redux - Willaim Ghote

it's live hi Willard Dawson AKA billy goat um glad to be here it's my first presentation at a conference so uh a little bit of nervous a little bit of that but not too much okay so the presentation today is about Lotus Notes password password hashes and those that are today exposed to the internet with basically no protections and my activities around that who am I I'm an information security architect at New York Life Insurance Company I've been in that position for year and a half before that I worked as a penetration tester for uh sunu guard availability Services a variety of jobs before that um so I've been basically living in Atlanta for the

past 20 years some OD years so uh well that's about me um so what this is about Lotus Notes we're going to talk a little bit about what notes is what what it what it actually does for you it's more than just email but uh it's has a rich history uh we're going to talk about some of that history and talk about the background uh behind some of the vulnerabilities that have been announced and discovered over the years and some previous presentations going to go over some tool tools that are available for testing Lotus Notes and um there are quite a few tools and going to give some samples of what I was able to find uh

during my testing and some observations about what I discovered after doing some password cracking and other testing Lotus Domin it's so easy to say Lotus Notes all the time I keep finding myself doing that Lotus Notes is actually the client side application that goes against Lotus Domino servers so um there are there are some some special protections that are enabled for client uh uh access to notes servers and I'm really not going to dig into that too much but so there are some mitigating factors to to the passwords being exposed so if if for example Lotus I notes which is we get into an moment isn't available then you may not be able to leverage the password user

authentication components that you cracked to actually access email but email is not the only target all right so messaging and caling Cal calendaring the the the most common use of notes there are also um quick web page development uh um software available with with lotus Domino um and that is kind of a if you wanted to draw a parallel to the Microsoft world it's it's probably most like SharePoint so there there are easy easy web page development tools that that are part of this and it's called quicker so there's also um inotes which is a web- based um capability uh and and some other other things we'll see some some do the rundown on another slide in a moment it

has support for multiple operating systems runs on Windows runs on various flavors of Unix and uh supports server clustering which comes into play later so in we mention I mentioned quicker there's also a product uh called same time which is a meeting coordinate a meeting um planning and coordination tool the same time it has some interesting aspects to it if your penetration testing um that sometimes you find meetings advertised locally open openly with with no password and you can sometimes invite yourselves to meetings and participate in their meetings um without having been invited of course uh officially so and of course then you can also see who's attending the meetings because generally that's also not obscure and you can use that

for social engineering attacks so that's pretty nice to know uh I notes is the the web mail component for motus Domino and pretty much just like ow for Microsoft in that um Lotus Notes Traveler is a product that allows you to get your Lotus Notes email on your iPhone and it has uh basically a basic authentication component which of course you could boot force and uh generally speaking uh I I don't see a lot of companies deploying certificate based authentication for for traveler although for for the actual thick client for notes you sometimes find them certificate bed tication and that is we we'll talk about that some more when we get to mitigations who who actually uses load

of stom well that that's kind of varied over the years but as recently as 2006 and 2010 time frame probably somewhere around 40% corporate world is using Lotus Notes in terms of the actual number of mailboxes that are deployed so a lot of large Enterprises and big companies extremely large companies are are using them uh and and this this was these these bullets were lifted from the documents that I mentioned here so not my own research there's also you can find a list of companies so you want to get down to specifics about companies who are actually using it in fact the company I work for uses it previous company I work for uses it and I have to

say I'm not a happy user I don't like load of snow so here I am to bash it right yeah me neither to be honest and and we're we're actively looking to migrate away from it but the challenge you get into as a large corporation is that you you develop applications and databases on it so it's it takes some planning and coordination to migrate away from and it's it's a non-trivial task in a lot of companies because really are Maj major migration headaches around it so uh next we'll talk about some some background history and previous work a couple of previous uh research papers uh that are worth looking at from 2001 Defcon presentation was really good material

and also hacking Domino which was uh a presentation or actually some papers that came out of some company in Russia has some very good background as a matter of fact on how to extend access into Lotus Domino servers to gain admin access to the servers to leverage that access using um cat uh um incat for example or or other tools and and of course pivot and let extend your your presence into the company's networks uh doing fast pin tests I manage to do some things of that nature so I don't want to go into those kind of specifics because this really isn't about pin testing per se but just to know once you get access

to Lotus do uh Domino password hashes and crack those if you gain access to admin hashes now you're talking about gold as far as penetration goes so there is history of weakness and and we'll talk about that a a couple of cves that are worth noting in fact I'm also giving this talk at at Sky talks as my my title there was 2005 once their hatches so as as long ago as 2005 we knew about the specific vulnerabilities that I'm talking about here and yet corporations are not configuring their servers to defend against because they either don't know about it maybe there maybe their staff has changed and and the folks who did know about it

aren't there anymore but however you explain it it it it the guards asleep at the gate oh and the link at the bottom is is a good site to look at because you uh you you get it it brings into into view additional cves that may be of interest so it's more than just looking at the specific cve so I found that to be a useful link some tools um not sure how I'm doing on time so I don't want to rush to it I'm good all right so uh a few links there some really good uh articles especially I found on search Domino about securing Lotus and and I should mention if you want the presentation I'll be happy to

share yeah yeah yeah so uh yeah it's not trying to hide it or or stay in hiding too too to off that but uh so there there are uh I found these to be useful tutorials about heing Lotus Notes and and also discussing some of the the password still a threat on Lotus domains not the first presentation that's been made about the the problem at hand and of course if you want those links it's much easier to get the presentation but these are the tools that I found for testing Lotus Domino um the one that I would say is probably to me the most helpful uh Domino hash Breer does more than just test or passest word hashes but it also

tests for other weaknesses so that's a really useful tool and um the um so the Raptor Domino hash Link at the top is probably the best one for actually extracting hashes from servers the problem that I have with that tool is that it basically pulls one field from the the user directory and the hashes and what I found I found it to be useful to get all of the fields and to scrape the web pages so that I had all that content at my disposal and so uh rather than rely on that tool I I I did some scripting of my own that I'll show you momentarily the the thing about the um their tool though it's it

leverages additional weaknesses besides just UNP password um user directories that aren't passwort protected so it it has some capabilities that the scripts I wrote don't have and and it th that tool can be very useful when when simple pass password evasion is not an option so how do you find these Domino servers they're they're scattered all over and mainly what I'm what I'm talking about today is web access to Domino servers so there's three three ways that I I went about that there's actually some known Google queries that talked about it in in Google dorts so we'll show you a couple examples there erip is the every routable IP project and that's that has some some interesting uh results when

you query that and of course I guess you probably know about showan there's nothing new about that I will say um in a moment Google dorks uh a link to where you can find probably the first Google dorks that were published but you really don't find many hits from that so as as you start looking at this and searching with with Google um I use tour a lot and Google doesn't like tour very much so yeah it makes it makes life interesting when you're trying to get around that but uh anyways still managed to do so and I I found especially we scroll down the page a bit when you iterate for US states US territories and Global top

level domains that Google really doesn't block you that much especially on tour so just the the nature of a query that you make U they they they home in on certain types of queries so breaking it up in that fashion they they really just left me alone for the most part so just one quick it's kind of not very visible but one one quick result quering for Alabama uh in this case I I found it better to break down the US domain into the individual states just I wanted to limit the results that I got and at this time I was really just doing things manually just to see what I I would find I started this off in

September of last year just playing around and and actually you know I I really didn't mention I forgot to mention it why why I started this little project um that that would have been good to know right um well a I hate using notes and really uh during my pin testing career I I I was a very Avid user of John and John the Ripper and and password crack so two years ago the first crack me if you can contest the core logic or organized I participated in that and and had a lot of fun doing that um this system right here is is my cracker Eng so unlike the competitors there who who actually won the

competition with GPU enabled uh computers and and spiy software like hashcat um I didn't have I didn't have any of that so I I thought it was very respectable that I didn't come in I did not come in dead last you know for me that was that was my goal don't come in last and and and actually managed to crack around um I I forget the actual number but a very decent number around 35 or 50,000 I forget somewhere in there a drop in the bucket compared to 500 you million that that they managed to spr uh really that's that's cool and everything but the bad news is hashcat and the GPU enable tools do

nothing for Lotus hashes and I think it's an awareness issue I mean certainly they they they developed the software to go after the hash formats that people know about like md5 or Shaw one because those get all the press release you know you get LinkedIn with Shaw one big news right Lotus Notes nobody cares right hopefully we'll make them care so some additional Google queries that that got some different results and and especially I like this quicker one so I'm actually looking for with some experience okay some some experience in looking at these sites I I start to see postn name conventions that companies are using and they're quite open about what kind of server they have

embedded in the domain name it's really really quite helpful to us pin testers thank you very much so all right some samples it's kind of hard to see not not too bad though um so here a search we did a I did a Google uh look up and I I found I tried to uh Reda the actual identifying information so I actually haven't disclosed these these vulnerabilities to the companies that are affected and yet plan to do so as time permits but when you talk about the the results and the number of sites that I'm finding it's it's going to be a large task to let them all know to be honest it's it's uh it's more than than

I can do a couple hours a day and and actually say I've accomplished it somebody needs that as a full-time job I nominate IBM reach out to your customers and help them secure their environments so I clicked the link and this is what I found um and I keep looking because I want to make sure I actually rejected the the host name and all that I think I did so this is a quicker site um it actually has some publicly publicly accessible content with obviously no authentication but in this case no sites but if you add names.nsf now you're looking at the user directory the the published user web directory for this server and it has identifying

information which I've attempted to cover up now there's a special uh uh view for for uh that you can leverage in Lotus Domino uh well let's let's take another look uh drill down actually into the menu options but the important thing here to note is that you can look at the all server documents list then it will tell you every server that makes up the cluster in this case it's a single server but often times you find more detail about every server in their Lotus domain we we found this one's actually a Windows Server so I guess if we were going to Pivot that would be a nice place to PIV it because then we could

use something like metasport jump all over it um anyway with the users view dollar users and parentheses around that you get a little bit more information and and I don't show you all the all the columns that are available here just the name and the list name but as you scroll further to the right there are many more other data points and there's one called HTTP password the H HTTP password column has a hashed p uh content is the hash password for the user and that's that's what we're after that is exactly what we're after um sometimes you will find that companies have uh obscured that information but if they haven't bothered to to protect the directory generally

they haven't I found one or two sites that that actually obscured their their um there there are we'll talk about it in the mitigation section but one or two places that that had the directory open but actually allowed that that hid the hash but for the most part if the directory is open they've also not protected against the hash and there are other um other other ways to approach the problem too to to limit access so this is this is another site um just one more example this this actually is is what you see when you go to the website itself yes they want you to authenticate but the director is open so if you simply check for names.nsf like

the the Google dorks query and saw the password prompt and assumed that you couldn't get access to it well you you passed up an opportunity so this this uh is kind of the same layout we saw before um but an example of well that's user data sorry more user data uh interesting to see things like a Blackberry administrator in in the full name listing so we we get some appreciation about the other Technologies they' deployed and probably if we crack that password we might have useful information on other systems the um and and here here's the example of a clustered environment so you can see more server detail which you use so from all my Google searches I I

found 93 unique sites excluding all the Clusters uh so taking out the cluster servers and just unique unique uh sets of passort cashes I was able to out but because I'm doing it man manually because the API went away because Bing doesn't like I don't like their didn't like their API as much even though it was freely available available time on there's something else we can do so let let's look at erip uh every rable IP project it's a it's a project that basically attempts to to do Google one one better instead of following uh web uh web spidering it it simply attempts to enumerate through IP addresses but doesn't seem like it's been updated

involved because every time I check it I get the same results for this particular query but like showan you can query you can query U specific search items and get different results back and at the time I tested n none of these sites were were exposing their director so I didn't the need to block their information um but you know you can see basically the information they were able to pull which version version info and domain name it's very very helpful so I wrote a script to enumerate it and it's I'm I'm lazy so I I started off my career as software designer doing C and C++ a lot of heavy database lifing but these days I I mainly throw things

together in bash with off and maybe a little Pearl if I have to I thought maybe maybe I will turn this into a metasport module yeah that's too much work I didn't get around to it yet I think it would be cooler to add it to the framework so here's here's the thing running basically just a bunch of w gets and then I uh pars through the output and I've obscured the IPS because those actually are roll potential and actually those happen to be so simply by scarfing through scraping scraping the results from erip and then pulling the IP information out I've created a list of ips that I still need to process um and I'll show you

what I do with that later so those are just lists of of of lotus Domino serves at that point in time I still really didn't know whether the server is exposing its hashes or not just we all listed of things to go test so you'll get tons and tons of server because how many well lots of servers to go take a look at um did I say how many yeah four 459 servers that erip told me about for the lus Domino servers so just from that one query so I did a handful of other queries and unfortunately I didn't really track the results to know which one gave me I just threw it all into a big pile but one

thing I did test for example I IBM gets you lots of Apache servers and really the Apache servers are generally speaking not Domino servers that's kind of a wasted effort and but I went through the effort anyway to test each and every one of them to see if they happen to have Domino hiding hugely time consum so here's my little batch for uh bash for to uh actually run through and and enumerate the tests against each one of them so do a w w get against each IP on ports 80 443 and 880 uh and drop that into a test file uh and then I'm going to actually just use G the output HTML to see if it contains

the word HTTP password so if it does they're vulnerable that's simple and that gave me 314 vulnerable loest Domino servers from ER out of those thousands which I thought was pretty good result showan I think probably no one anyone not know showan right showan is is cool you can search things like Lotus Domino as a string in this case pipen and a total result of 77,500

I started using the API for that and pulling results down and um uh I'm awful with names I should have written it down with the gentleman who actually um put it together I I reached out to him for some advice and some support and he found out I was doing it for a research project and he just gave me all the results he said here here they are so totally nice guy um and I'm super ashamed I can't remember his name early onset really um um truthfully so 71,000 results now these are just raw Domino servers that at one point in time were exposed to the internet and of course they may or may not be up

now so what do I do with that well I've shown you basically how how I scraped those web pages now I want to gather those those hashes together and this is where the Raptor Domino hash program does its magic but I wanted something a little bit different and then I'm going to practice hashes so this is uh me scraping the individual um servers it's very simple it's actually a couple of pages so it's longer than this three pages so I didn't actually provide the text for for this so if anybody is interested in using it reach out to me and I'll be happy to give you the the actual scripts you don't have to type them in but um pretty

straightforward just just uh enumerate through all the pages now there there can be some trickiness around that because we looked um I guess sorry if I have to bounce back so I really won't do it but depending on the version of domino the the presented directory has different error buttons and some of those if you click them you can click directly to the end page and then an additional page and it tells you actually that the numeric URL parameter that takes you directly to the end generally speaking these these pages are are broken up into blocks of 29 by default uh when you scroll if you just keep clicking the next Arrow as you're manually viewing the web page you get

them in blocks of of blocks of 29 line items on uh within the directory so I can preset how I enumerate through that by just root forcing my way page of the time if I know when to stop um so that that's handy but in older versions of lotus Domino you really don't have that information at your disposal so I actually parsed out JavaScript and and calculate from that where um where it's taking me for one page to the next and that's handy because sometimes the pages don't follow numeric boundaries they have funny little period dividers so they're not really just numeric parameters they're alers and then there are the occasional pages that I couldn't do any of that

with so I have a manual script where I simply root Force until and watch it uh by eye until I think it's done and then kill it which I haven't included here and when all else fails go back to Raptor Domino hash because it really works just about every time um so a few pages of that all right so once I've actually gathered all these HTML pages I put together another script to to extract out from the fields that I'm interested in and the reason I want to build I'm building I'm building an output file that I'm going to feed to John John the Ripper so I want I want there to be two columns

uh separated by a a which is the default separator and John the Ripper and the left column can be anything but what I found when you're using single mode in John the Ripper is that if you build out your input data with as much Poss information possible you actually wind up with a lot better results when you're cracking so that's another reason for not using romino hashes because when you actually feed that to John the Ripper you you don't wind up cracking as many quite so quickly course you can let it run in incremental mode and and brute force it but that's going to be very time consumer so that's that's my motivation for this little script um and

I also match on on the because for for some strange reason username um the the email address version of the username can appear at various locations within the uh output file so it's kind of hard to predict which column is going to contain the the uh email address and so I I did basically did some characters matching on that uh I I won't actually show an example of the hash tap uh output that I did because that is very revealing it's massively revealing um so um we won't go there and hash hash formats there are two basic basically two well-known hash formats within Lotus Domino one is the unsalted which goes back many many years but you still find

it uh pervasive in in the in use and salted um is the so-called more secure password hash format type and John the Ripper calls these Lotus 5 and Domino set um and I used uh some some guidance from Rick Redmond's um paper on the topic about different ways of of of doing um pattern pattern matching and it's very useful but uh these these examples here um don't really take any of that into account and you get quite a lot of results using just this simple uh type of thing uh as you might imagine running on my I5 laptop it's a slow process it's not very not very Speedy word word lists are definitely the way to go rocku matches often very

often pure hates word list this is a very good one got milk and the inside Pro dictionaries are very good and there's a pretty good listing at skill security as well and I I use these often no is it very very helpful um so what what did I do I so far and it's it's it's a snapshot time because things are still moving um 11 over approaching 12 million gathered hashes unique uh well lines lines of input I should say and the lines of input uh for me means multi multiple lines that uh because of the way I'm assembling that hash input file uh in terms of just the account data here's where I'm actually

saying this the unique email addresses that I've identified um and unique hashes this takes into account also that uh loot Lotus that Lotus 5 format is unsalted so the same password has the same hash every time um so with the Salted uh hashes of course because it is salted the same password May generate a different salted hash and you get you it's hard to predict exactly how many unique passwords will come out of it once it's cracked um but so far you can see basically what I managed to Performance issues especially with uh do the the more secure format on on this machine when I load my input file takes probably 72 or 80 hours just to start cracking

because it takes the way John works is it loads the entire thing under memory first and with with the uh Domino the the the Domino SE form that's very timeing even even when I'm running from my SSD so it's it's not running from an SSD is is definitely the right way to go because it does speed lad time but very very problematic the um the the the old format the Lotus 5 format John is smart enough to spread that across multiple or or hyperthreaded CPU instances so there's actually I'm consuming 400% of my CPU on the laptop with that format but the the Domino SEC form it only it's it's very serial in nature so it can

only do one of those it's not hypersquad it's not parallel at all so there's I don't know if there's good opportunity to parallelize it for GPU but I'm I'm hopeful that there is at some point in the future we may have better choice so just a quick rundown of some of the findings unique accounts very large organizations which I won't mention by name because that's kind of too revealing but uh a large automobile manufacturer a few Banks um computer manufacturer in China that you probably know very very decent results I also found and I didn't really make a slide to address it but because these these quicker pages and other tools are being used for open sites

there's also uh a frequent use of lotus Domino to support self-registration websites so a lot of times what you find are email addresses that are things that people have entered they're not an employee of the company but sites being managed for a community at large and there I especially there was at least one organization in Florida that I found that had um an enormous number of EX users and and another organization in Texas that was aimed at teachers that has an enormous number of passwords and then I found a Caribbean Bank that has a large number of users and oddly enough they're all mostly in the financial industry in the US so imagine what you could do with

that some mitigations you've got uh different settings that are specific to to blocking access to information and um um these these first two basically cover that so you can obscure the fields you can uh limit Anonymous access to the directory so the challenge with that is that if you don't hide the password fields and you do allow it or you don't and if you if you allow the password fields to remain but you disable Anonymous access all you've done is hidden the fields from inside users so if if you're an authenticated user you could still get access to those hashes if if that's the case um you can actually extend my script if you if you um if you want to

use something like burp and capture your authentication headers and stick those into the script you can actually use that to capture hashes as an authenticated user not that I but um I it could be done and you might find some very interesting details um and and of course if if if that's if and only if the password fields are still uh exposed and hashes even to authenticated the the last uh recommendation I found on Military sites in the US that the common access card as they call it or two Factor authentication is fre is frequently in use uh this is a pretty decent article because the actually helped them do that and they they speak to what they did to

make that happen so they they address some special things that had to be done on same time servers that don't support authenticated uh certificate based authentication and as we approach the end of the presentation uh is it forgotten yeah kind of sort of is it dead not really it's still out there and uh they' really sharing that with us waiting for us to test so my my last slide I'm begging someone to step up and take action I please help your customers secure your n their networks and owner operators go to these documents there I didn't present a link for it but there are some very good uh red book articles on securing Lotus Domino IBM generally

speaking has some very good documentation I think the the weak the weaknesses here are that um it organizations are not following through on those documents and building uh proactively bu architecture um to to take advantage of those to harden their servers and and need to step it up so that's

y well some of the some of the approaches that you can use to to move away from actually user username password authentication I think I haven't evaluated that one specifically but in my own company we're we're making the same moves so we we obscure ours using um some other protective Technologies and and I didn't I started to but I didn't provide snapshots here but some companies are protecting um their inter their infrastructure all together by blocking it from the internet and and using setting up another user authentication layer using things like UAG or or other reverse secure reverse proxies to Li access I think those are all very good moves oh sure

proba blowing up

videoa and I have cards that's my corporate I didn't person so that's

itk

e e