BsidesLV 2024 - PasswordsCon - Tuesday

he

[Music] h

[Music]

[Music] oh a [Music] [Applause] [Music] [Applause] [Music] I'm just to something I I'm just trying to give you [Music] something I'm just trying to give you something I do you I'm just trying to give you [Music] something he [Music] [Applause] [Music] [Music]

[Music] [Music] I'm just try to I you I'm just TR to give you something [Music] I'm just trying to give you something [Music] I I'm just trying to give something [Music] w

[Music]

[Music]

[Music] [Music]

[Music] he

[Music]

[Music] [Applause]

[Music]

[Music] [Music]

[Applause]

[Music]

[Music]

[Music]

[Music] oh [Music] oh [Music]

[Music]

[Music] [Music]

[Music]

[Music]

[Music]

[Music]

[Music]

[Music] [Music] [Music]

[Music]

[Music]

[Music] [Music] he

[Applause] [Music] he [Applause] [Music] [Applause] [Music] he [Music]

he

[Music]

[Music]

[Music]

e TR [Music] hey hey [Applause] [Music]

hey hey hey hey hey hey [Applause] [Music] [Music]

he [Music]

[Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Music]

[Music] [Applause] [Music] oh [Music]

[Music]

oh

[Music] h [Music]

[Music] [Applause] [Music] [Applause] [Music] yeah w [Music] [Applause] [Music] I'm just I'm just trying to give you [Music] something I'm just trying to give you something do I'm just TR to give you something [Music] m [Music] a

[Music]

[Music] [Music] I'm just TR to give something I'm just trying to give you [Music] something I'm just trying to give you something I do I'm just trying to give you something [Music] o he [Music]

[Music]

[Music] [Music]

[Music]

[Music]

[Music] [Applause]

oh [Music]

[Music] [Music]

he [Music]

[Music]

[Music] e [Music]

good yeah let's see just GNA turn this down and then okay cool hey everybody uh welcome to bsides Las Vegas 2024 this is the ground truth track focused on data science and other mathy and data things in security thanks for coming uh we want to thank all our sponsors for making this possible and we're going to get started right now uh with Douglas mcke come on up Douglas thank [Applause] you all right thank you I'm excited to be speaking at my first uh bsid Las Vegas it's great to get a chance to to talk to you all today I'm going to talk about something called proprietary protocols it's something I think we leverage not enough in the industry both

from an offensive and defensive perspective so I'm going to go through uh both of those things first off I like to say who's the C talking in front of you I've been in the industry 15 years almost which is scary to say out loud mostly on the offensive side of the house doing hacking I'm also the author instructor for sand security 568 which is a product security pen testing class uh and during the day I'm the executive director for uh threat at sonic wall so spend a lot of time in threat intelligence and networking stuff which is what's important for today's talk you can look me up on almost any social media platform at Fullmetal packets

happy to discuss anything cyber security related if you are interested so set the stage a little bit uh I'm a little guilty right being a Sans instructor I always end up defining everything before uh before I get started so we need to start by talking about what do I mean when I say proprietary protocols so an open protocol are things that you're very well familiar with from a networking perspective these are things like HTTP FTP these are protocols that have what's called an RFC so these protocols you if you want to know how they work you can go somewhere you can look up that RFC it'll put you to sleep but that's okay it'll still tell you exactly how the

protocol function on the other side we have proprietary protocols and these are the opposite these are protocols that have been created by vendors typically for a specific purpose now I did some Google online looking at like what do people think about proprietary protocols and I found this graphic posted online and I I I love this because it illustrates the point that the internet always knows everything that it's talking about right so here it's the vendor is marketing that proprietary protocols what's so good about them is they are safer because they are unknown now i' I've only been in the industry a short time but security through obscurity I don't think is the that typically pays out from a secur

perspective so this is not accurate right it's actually we want to take the time to understand these protocols because they're beneficial whether or not you're on the red side of the house or the blue side of the house so I've put together what I call an eight-step process for dissecting and understanding proprietary protocol I cover this in security 568 about a day and a half which I do not have time obviously for here today so we're going to talk about four of the eight steps and I'm doing this because I believe if you want to walk out of here today and apply this immediately these four steps will get you started on the road to immediately

apply these principles and start helping you uh within within the workplace so to Jump Right In I'm going about the first concept here referred to simplifying scope so what do I mean by simplifying scope to put it simply it's the concept of reducing data down to a reasonable size so taking a large quantity in this case of packets some we have thousands hundreds of thousands of packets and how do we get less data that's the goal of simplifying scope because I don't but I do not have a time to look at a th000 packets 10,000 pack 100,000 pack do it now the key to this is we can't just

randomly e

but starting with the first one protocols itself or higher level protocols almost every proprietary protocol rides on top of another protocol and it's not uncommon to see multiple higher or I actually I'm saying that in Reverse lower level protocols multiple lower level protocols being used within a proprietary protocol so you might have TCP and UDP being leveraged by a proprietary protocol simply spitting them out into two different types or grps of packets starts to get you two different representations of those protocols another way to do this is by conversations or flows probably familiar with in wire shark you can uh right click and do you show me the TCP flow or UDP flow at times what we see is that a

flow of traffic is one format or one type of protocol again what we're trying to do is to group these so that we have likeness between our groups so we can analyze one or a couple of packets in that group and have an understanding of that larger group another way to do this is by port numbers often times especially destination port numbers will ingest or parse a different type or format of packet think about this from an operating system perspective what is a thread right port numbers are often tied to threads they're they're independent sections of code well that code is different often times between thread or parsing different different typ types of packets we can also use pay of length

this often surprises a lot of people but generally speaking and again this is not all the time right not 100% of the time but generally speaking packets of the same length and have the same format or the same function and so using length as a tool can be very useful and lastly let's not forget about basic computer science tools for things like Integrity if the md5 the same between two packets they're the same right the same packet and we can leverage that to our advantage so using these five methods the best way to communicate this is to do this in real time so I'm going to walk through leveraging this concept of simplifying scope using a real propri

protocol here in this presentation so here on the screen is a proprietary protocol wir shark clearly knows not not what to do with it this is obviously just a screenshot of a much larger pcap we want to start reducing this scope of this pcap you the larger context in this pcap there's about 516 packets captured in about 188 seconds or about 3 minutes that's a lot of packets right no not really that's kind of a trick question that's actually like nothing three minutes of a network capture can be like 2,000 packets right so I say that to illustrate that we're going to do this for just 500 packets you're likely going to be working with a

lot more but the point the princi will hold so one of the things we can leverage they talked about ground truth being potentially a data sciency track is this concept of Eda Eda is exploratory data analysis we can actually use code to make it so we have to what uh my co-author for 568 ishill likes to say is not we don't have to read The Matrix accuses me of reading the matrix by just looking at packets we can leverage code so here is a panda's or here's I'm sorry here's python code which is leveraging scapy packets into a panda data frame by packets into a panda data frame we can start doing statistical analysis on these packets

and we can use the grouping methan mechanisms that I talked the previous slide in code a lot faster we don't have to actually be able to recognize this visually here in four lines of code I argue it's really two lines of code because two of the lines of code are print statements ingest a pcap and display the different protoc calls known protocols in that packet so here we've got UDP and TCP now if we do this for the example pcap that I provided you or provided on the screen for the grouping principle here are four of the different grouping techniques I talk about the first said you can Group by a lower level protocol unfortunately here that

becomes useless right the entire pcap is UDP so we're going to have to try again so we're going to look at another method I said destination ports could be something that is useful for grouping if we you leverage our Panda's data to graph out destination ports we instantly get four distinct groups of packets okay now we're on to something there's a chance that these are all of similar format or different formats if you will what if we'd use length if we use length we also get four distinct different groups which begs the question is there an overlap between length and destination ports we always when we're simp scope the stronger case we can make is the more data points we can use and

it just so happens that if we tell python to group these together we actually get perfect marrying of length and destination port and this is very simple do leveraging Ed and leveraging pandas you don't have to always visually see this you can totally do this what I would call the and I've done it that way many times but this it will speed up your process so now we have this concept of okay if we take the length and the destination port number this could help reduce our data set but we've looked at one pcap is that statistically relevant no right one pcap of data of one capture that's great but this could just be a random Trend that we found and

that's very important in dissecting proprietary protocols you need multiple captures multiple instance of the same behavior and that's also very key so let's go ahead and look at this again for a second take of the same behavior of the protocol we're looking at and what happen is we actually get a very similar representation there's a few minor differen here but the important thing to point out is that our L by destination port number pairing still holds true and we can this a couple more times but the concept is likely a good place to start to reduce our scope down to four packet types so here we had 516 in 188 seconds and in just a few minutes

weed four that require analys is not 516 and I can't stress enough that all of the eight step process I have put together for reversing proprietary protocols this sets you up for either success or failure this is probably the most important step so now moving on to some of the other steps the other thing that I will note uh here quickly is the first step be first you have to do simplify scope the rest of this is kind of in an order that I was useful to me and to my workflow and can definitely be rearrange to your workflow once you've done simplified scope you can rearrange these to whatever you feel is the most

useful so I'm going to talk about a concept called embedded networking as the next scope it's it's actually uh very common in proprietary protocols to see embedded networking with payload of the protocol that matches the networking D data in the other lower layers so what do I mean by that things like IP addresses Mac addresses port numbers are duplicated in the application layer payload layer of the packet the number one question I get when I talk about this information is why to be honest I don't have a definitive answer to that question never written a proprietary protocol my hypothesis which I think is is based in a decent amount of data is there's a chance there's a large chance

that the higher level application that's receiving this information does have access to the networking layer of information if they want to make a decision based on let's say an IP address of where something coming from or a port number they embed that in that in the payload so they can access that information I think is typically why we see this but this is seen all over the place so here we wire shark in the IP layer that the source address is 126 1.1.1 I love this cheat if you highlight it in wi shark it highlights it in the payload and hexadecimal for it makes it very easy to visually notice that there's a value of the exact same pattern in the

payload of the packet and this is the concept of embedded networking if we take this and we look at all four of the packets the representation that we took from the larger groups we'll actually notice that three out of the four have a IP address embedded in the payload of the packet so now we're well on our way to understanding how protocol Works our goal here ultimately is to go from knowing nothing to knowing something it's sometimes we're able to find out what every bite means sometimes it's not possible or not easy to do but we get one step closer to understanding that protocol the next uh concept or step if you will something I call enumerating

patterns and this is the idea of leveraging patterns across packets across peaps and within the context of a capture and I'll show you by that here in a moment and to to determine what bytes in a payload represent so let's first zoom out and look at the higher level pcap again uh just from a wire shark perspective now here we're by length and port number because that concluded was one group of packet now we're looking at the group that we've decided and we're saying what are some patterns within this group and it doesn't have to only be within the payload I see that as a common mistake immediately you open up the hex de View

and I want to look at the payload well how about looking at the overall conversation that sticks out as a pattern if you will is this repetitiveness of four packets in a row from the client four packets in a row from the server okay that's interesting and we can also see a Time perspective these packets are sent almost the exact same moment in time well because uh we have this data in a panda data frame it's very easy to check other group grouping methods that I talked about in the beginning with just one line of code we can actually check the md5 hash of these packets are they the same or are we dealing with different packets and what

find out is that four packets is actually the same pet repeated so what we care about or what are the differences between the repeated packet and the packet that is changed understanding how cryptography works this could be one B this could be the entire packet that's different so we want start tackling this question what is different and what can we learn about the fact that there's a difference here so here if we open up the three three of the packet to see the difference across the different payloads we simply no nothing fancy or trickery here we start looking at one bite at a time after where we have known data we know the IP address we know where it is in all three

of these packets see that it's the same that's expected because it's coming from the same place what is the next bite Well turns out we start looking bite by bite next three bytes are the same where we get a difference is the fourth bite interesting so three packets in a row the first thing that is different is there's four B three bites that is different now in networking it's it's possible to have one bite that indicates something that's why scapy has a bite field for example however it's more common to see two or bites or more increments so you dissecting packets is a little bit of an art you have to make some assumptions and then either prove

or disprove those assumptions most of the time I'm going to not assume that one bite is different on its own I'm going to assume there's a group of bytes involved and here one bite changing I'm going to look at these as a four bite group and see if I can determine what these look like now if you can spot it using the hexadecimal here there is a difference of 10 between each one of these apologize for mixing decimal math and hexadecimal but I think it illustrates the point a little bit more on on the slide so here we see it incrementing by 10 so what could that mean that could mean a lot of things

could in theory be a coincidence even though I don't believe in most coincidences what is causing this to change if we zoom out again and we look at the the the entire peap or filtered pcap look at those four packets we can see that there's something else changing and it's the value of time so almost exactly to time there is a 10 second difference between each one of theet does that mean that this value represents time not it could but it ALS not be I I just randomly started at the next four bytes and found a 10 difference so how do we validate this hypothesis we can take those four bytes and we can convert them using an Epoch

inverter why Epoch because that's likely thing you're going to see in uh in HEX AES a packet and we get a time stamp of Tuesday January 23rd 2018 does that mean it's a time stamp could just randomly translate into a time we want to validate this by looking the frame of the packet the frame of the packet also shows January 23rd 2018 the time itself is slightly different but the the probability of the date being exactly the same across those two values is pretty low it's a pretty strong indicator that this is like a Time Value in the packet so that's just one way that you can leverage one thing you can determine by looking at patterns across

payloads and AC different uh across a pcap I'm going to jump down for a second to step seven and I'm going to talk about appliation reverse engineering this is something that is very useful to you for dissecting proprietary protocols if it is an option and I stress the If part because a lot of times we're dealing with proprietary protocols in iot ic infrastructure you can't always have access to the application to do the reverse engineer but if you do have the access the if an application is sending and receiving a protocol it understands it if it understands it you have the source code and you have to just be able to read that source code so this is a

very strong tool in your tool belt if it's applicable if we go back to our pcap uh this a little bit about this protocol this is a medical protocol uh it's called the r what protocol and here what we were looking at was the client the this case is a medical device not impossible to get the parsing code for sure we could do some Hardware hacking techniques we could pull off firmware uh we could see if it was downloadable but definitely the more difficult approach the server in this instance is a Windows machine much more friendly to work at if we want to do some application reverse so for a moment if we say that we didn't

figure out what that time value was and we wanted to investigate if the server side had some clues of what that was we could dump the server uh in this case dll into something like Ida Pro and right off the bat in Ida Pro this is right in the very beginning of the code uh we get some Clues to what be going what might be going on here now Lucky in this case a lot of debugging symbols were left in function names were left in the majority of this was left here for for me uh which is extremely helpful right and deter I recognize that might not always be the case dur during your event but

it's something to look out for here we have a function called broadcast rwood I did not rename that that was provided to me by the developers thank you very much we know from a little bit of osint R what's the name of our protocol broadcast it's going to the broadcast address this makes sense I'm looking in the right place right we go down and we see that there's another function called get R what period it's being saved into a variable I did rename the variable so the variable name is I named that for just for readability on on the slide but I name the function and we can see that that variable is being used in a debug

statement I love logging in debug statements they provide a lot of useful especially for something like this and it says the r what period is so many seconds if we dig into that function just a little bit further we can see that there is a value that's being returned either 10 15 or 20 uh depending on what a configuration also named this variable as well so this one was not left in but I renamed it for for readability so this is just an example again of tools in your tool belt to help determine what is going on The Wire what's going on on the wire or the ground truth if if you will so I'm going to take a break of go

from going into the different processes of reverse engineering those protocols that gives you kind of a a quick ramp up as I said I do this for a day and a half in in security 58 uh but I'm going to turn to talk a little bit about how you can leverage this from an offensive and a defensive perspective so starting with the offensive side I'm biased I did mostly offensive security so I I generally there I want to go over three principles really quick that you can leverage protocols for assuming you understand just at least a little bit about them and so I'm going to talk about exfiltration emulation excuse me and falsification so starting with

exfiltration if you remember there was four packet types we looked at we Group by length and port number this is the longest packet on the screen of the ones uh that we selected a concept that I didn't have a lot of I time to discuss today is the concept of padding proprietary generally leverage a lot of different padding and they can do this one common way is to do it in the form of null bites not always it can be any value but often times it's either a null byes or F or what I see most commonly used here I've highlighted on the screen all the null bites within the larger packet you can see there's

quite a large quantity of that in fact additionally if we zoom into the top part we can actually see there's R of 32 38 bytes of consistent null bytes if you think command and control if you think exfiltration of and you know that these are going to a broadcast address it's very simple to embed what you from an attacker's perspective hidden Within These padding bites to leverage on the network and frankly in the reality of it is most Defenders have no idea what theol is it's not being parsed by ids's or other security tools unless they're doing this you likely don't even have to get the format right you can probably strip out the rest of the payload and

and put the whole six with whatever you want maybe have a few values in the beginning correct so that way look like the other stuff but it's very easy to exfiltrate data with proprietary protocols next thing I'm going to talk talk about is emulation often times from an offensive perspective it's useful to make it look like a device is there that's not there or that you took down and you want it to keep looking like it was doing what it was supposed to do I'm going to use the rat the medical example to to demonstr this in a a quick little video here don't get caught up on the medical nature of this so this is the

server uh I'm plugging in a Raspberry Pi obviously not a medical device right this is just a Raspberry Pi and is going to interpret the data that the Raspberry Pi is sending as valid Medical Data it's actually going to report it as a and so as long as this Raspberry Pi is plugged in it's going to keep getting that data from the proprietary protocol now don't get hung up on again the medical example here you could use this in a large of different use cases where you could uh pretend that something's working at a level that it's not and very closely aligned to that is the idea of maybe I don't want to emulate the entire device

but I want to falsify the information that the device is sending so again leveraging this medical example uh here we have the end medical device this is a patient Monitor and it's showing that beats per minute is what's being translated and we're going to bring that up in the central monitoring system here or the server side of the device you're going to see that data communicated across now an attacker that understands how this protocol is supposed to work can manipulate that protocol as it goes over the wire already demonstrated Python's very good at and using scapy uh at bringing in these packets anding them we can in a simple script here we can send we can intake packets we can send them

back out and we can have them with a completely different value in this case jumping to 180 now get hung up on the medical nature of this example obviously there's implementations From perspective but think about this what about an alarm system whether you wanted an alarm to go off or not go off right think about any type of IC System whether you want to make sure that a a valve was open or a valve was closed all of this applies to that as well so that's from the offensive perspective let's talk take a little bit about the defensive perspective of proprietary protocols even though I come from a background that's largely offensive I actually think from a proprietary protocol

perspective this is way more important we have to understand the data on our networks in order to be able to defend it properly we have to have that Baseline of information and reversing these protocols helps provide us that Baseline so I'm going to talk about three different uh Concepts again from a defensive perspective I'm want to talk about threat modeling mitigations and protection and because it's 2024 and I'm speaking at a conference I have to leverage AI or they literally cart me out of the room so I am going to make sure get some of it in this presentation there is a model out there called stride GPT I don't know heard this before but

it's it's used for threat modeling and it can produce mitigations it's actually a neat tool like most AI powered functions it is it perfect absolutely do I think it should replace your threat modeling and mitigation strategy absolutely not but it is a great place to get started to give you some ideas on how to move forward from a defensive perspective one of the things that uh I we talk a lot about threat modeling in uh security 568 and one of the things we talk about is a threat modeling Manifesto which has four questions for threat modeling two of them are what are we work on and what can go wrong and we document them doing something called a

DFD or data flow diagram so what you see here on the screen is a data flow diagram of a different protocol this is not the medical one that I was talking about before that is that has been produced by reverse engineer this protocol and understanding how a system works as a result what could go wrong with it this is created in draw.io what's really cool we can upload this image to stri GPT and stri GPT is able to then understand a bit about how the system works it actually populates this description field that you see on the automatically based on that diagram I fill out a few other questions on the right hand side about the application

it's an iot application it has basic Authentication you hit this generate threat model button atot and you're off with a a generated threat model to start your work with it does things on top of generating because the threat model it's going to generate I'm sorry is stride based hence the stride G stride is not always the most useful methodology to use one of the things that I like to Leverage is something called attack trees and this will actually generate an attack tree for you and provide it to you in a graph I find this extremely useful when thinking about things like mitigation and what can go wrong within your net if we look at the nodes all the

way at the bottom we can see these are the outcomes or the potential malicious activity risk that can occur by using this protocol of course this is based on what the I the AI thinks gener the in the the input that you gave it but I think is more actually impactful the act the outcomes at the end are the fact that you see what I like to call these choke points within the attack tree you can see that there's this one thing leading to multiple potential risky outcomes so if I'm able to mitigate this one thing I get the biggest bang for my buck is protecting against these these type of attacks here obviously in this particular instance

the fact that it's an unencrypted TCP protocol is providing the way or Paving the way for four of the outcomes provided the other thing you can do with a factory is you can look for things that your organization may be specifically concerned about uh a lot of times in the government World location leakage is an important thing and here the threat model has identified that a location leakage can occur well I can work backwards now so I don't really care about the other potential outcomes but I care about one I can work backwards up the tree and say what do I have to do to mitigate the risk ofation linkage if you're paying attention you

can also see on this slide a little bit of the imperfections or not ESS of something like stri GP stri GPT excuse me in fact having a know that UDP Network traffic risks is not very helpful right but again this is a a model it's representation uh as George box used to say all models are wrong some are useful what we're striving for here is something that is useful to help you mitigate risk in your organization not for something that is 100% correct taking this one step further uh stride gbt will also provide you a mitigation suggestions based on the attes and threat model it provides sometimes these are not super useful they're super overarching or industry

best practice but again they can jog the thought process on what are things I can try to implement I like this first one here that it produces of course it's going to align these with stride again that's why you poofing here and tampering and the screen is cut short uh so it that may or may not fit your method but the first one remember we talked about emulation on the offensive side this first one's actually actually very well geared to emulation it a scenario and it gives you a potential suggested mitigation the first scenario is an attacker uses a for IP address to impersonate a legitimate client it sounds like emulation that's what we talked about in the offensive section so

what is something we can Implement to help prevent that here it suggests implementing IP Whit listing good suggestion that would definitely help with that emulation problem that we're seeing on from an offensive side so lastly uh I want to talk about uh detection uh which something that can be uh extremely important from a defensive perspective I'm going to do so uh in utilizing Zeke I think Zeke is probably one of the most well-known open- Source IDs tools out there so like to use stuff that people are likely familiar with one of the things that I was short in having time to talk about is the this concept of documentation by scapy it's probably the second important principle

in dissecting Pro proprietary protocols and is that you have to document your findings and nobody likes documentation I highly suggest that findings are documented using Code so as you learn about your protocol if you write what's called a custom scapy layer that is reusable then for both defensive and offensive purposes offensive things like fuzzing for example defensive we can use it for detection or logging or baselining so here would be an output from the uh the second protocol I showed you of es scapy layer for that UDP packet and this is just writing a scapy layer for uh what we've learned doing the the reversing now to the best my knowledge there probably people in the

room that are way more familiar with Zeke than myself so uh happy to talk if I'm incorrect with this but I don't believe Zeke understands scapy directly you've got to do something else in order to get it there well one of the things that it does understand is a language called spicy and it's actually very simple to take a scapy protocol and to convert it to a Skypey uh I'm sorry a spiky syntax I kind of like the way that phras is uh phrase works out and so here what I've done is I've taken the scapy protocol and I've implemented in uh spicy uh syntax now this is something that you can compile that Zeke

understands how to use so we can compile this using the spiky C compiler that comes with Zeke and we can now create a Zeke event this is very very basic I'm just simply showing that you can print out information about the proprietary protocol uh to the screen show you what that might look like you can run the custom Proto or the proprietary protocol through uh through Zeke directly and you can see that it's parsing out the embedded networking here we have something called a type field and a session ID that's in the protocol but there's that location data this protocol is actually sending you coordinates every time a UDP packet goes out on the location the UDP packet came

from here it's repeated five times in this payload and you can see that it's now understood by i z what can you do with this you can now detect anomalies right you can now print this to log files that you're ingesting in your other security tool sets to write rules around you can't do that if you don't take the time to understand the proprietary uh protocol so I think there's a lot of value in this so that's what I've got for you today I thank you for taking the time to to listen to my talk I hope you enjoyed it if we have time left I don't know exactly where we are on the schedule uh happy to happy to

take any questions live or afterwards thank [Applause] you yes sir I think they have a mic they want you to use on hello hello great talk than um do you know of or can there exist a scapy to spicy to wire shark disector pipeline I don't know of a pipeline that exists I I do think or some variation there yeah I'm not aware of one that currently exists I think it would be a great thing if you want to start open sour project that would be fantastic but I I'm unaware of I'm not aware of one any other questions yes sir uh when you're working with these uh proprietary protocols and you find issues that are um abusable from an

offensive security standpoint do you work with the vendors on those like what's your process there stuff absolutely I've found fact I've got several cves to my name test to that specific point is issues with that yeah follow responsible disclosure is is typically the way I go about as if a vulnerability discover Discovery process uh there it does get very complex depending on the industry in which the proprietary protocol is being used uh in in uh one that I discovered that was in the um building automation industry for example that was a relatively easy fix that fix was in in software on the firmware that was digesting the protocol and they were able to push an update to

that uh and so that was a very good process and another instance where we found flaws in the protocol that was in the medical industry right that gets a lot more complicated right they uh they had said that they'd have to go through their accreditations all over again to make that change right and then you have and that's the constant problem you have in ic iot critical infrastructure is it's not as easy as patch it right but yes to answer your question directly when I've discovered flaws in the protocol before work directly with the vendor to responsibly disclose them and then do some type of coordinated release uh have you done anything with identifying hashes or signatures in

packet streams I'm sorry say again hashes or signatures where hashes or signatures in the packet stream yeah ABS absolutely uh there's a lot of times a characteristic I see a lot of is different CR R c's for example which are not I know they're not a hash but the same same concept that that you're talking about uh those are obviously get a little bit more complex to identify one thing that uh there often times if you're looking at cryptographic tokens of some type A lot of times they're at the end of a packet and my my advice for when you run into that is you don't want to try to look for those first right you

want to weed out like the other stuff that I'm showing uh embedded networking clear text communication uh running tool external tools like binwalk and things like that on payloads to help you identify what you're going to end up with is you're going to get all that stuff taken care of and now there's going to be a group of five bytes over here that you have no idea what it is you're going to find two bytes over here that you have no idea what it is right you can start looking at those then with a cryptographic mindset uh to say is this potentially a hash is this potentially some type of CRC uh a lot of times you want to use

the I think it's step six in the process is behavior modification uh and the concept with behavior modification is you want to set up the protocol to work in a normal environment intercept that traffic make a change and then see how that change affects the client in the server if you're messing with a hash or a CRC you will immediately see something not work the way it's supposed to and that's a very good way to identify that again I talk about that more in depth uh in in the longer program next question does that answer your question sir oh yeah that's great thank you just a thought I had when you were talking about this especially like for

the you know for these medical device things um you know the first thought in my mind from a defensive standpoint it' be really helpful if they provided you if they provided like the U if they were able to provide um these templates for for these proprietary messages and my argument would be you know you go you have to go of course they would have to consider this during development but you could convince them that this is a good idea because then this will make it easier for the the people trying to defend the hospital networks against Rogue and you know against attack um has has there been any thought or has this been like discussed

within at least with within industry I I think it's it's a very wellknown uh problem SLS solution space I I think uh and if you look now especially in the medical industry you're starting to see some of them pull away from proprietary po or what they're doing is they're doing a better job implementing encryption on top of them okay so it's it's the product is still there but it's sent through an encrypted tunnel the the the challenge that you're faced especially with iotm or medical critical infrastructure is a lot of this is rooted in 20 30 years ago right and so it's not new design it's design that's been here for a long time when security

was not the Forefront or we didn't have these concept of the attacks that we have now or sophistication and changing that is is a multi- deade problem no I I I understand completely what you talk about like as an example like people who find vulnerabilities in uh in the baseband firmware and modems you know they they're they're always wondering well why why can't they just fix this and it's like well because if they make any changes then they typically the the the the modem providers typically have to go through recertification with the carriers and the medical industry is very similar you make a change well you got yeah it's not just that but then you

have backwards comp accountability issues as well you know when I was working on some of the medical devices in order to fix the flaw that was discovered they would have had to replace all of the devices in the hospital yes well if you go tell a hospital that they need to go replace all of their infusion pumps that's like you know a 10 million $25 million Endeavor and that's not necessarily yeah then you have to start looking at okay is there better ways anyway so yes okay I think I've been told we're at time here so thank you for your time and happy to take any question questions [Applause] outside do what I can yeah I'm used to things running behind I

try to catch up some time for you I did [Music] now

[Music] h

[Music]

for

[Music]

he

[Music] h

oh [Music]

[Music] w [Music] [Applause] [Music] [Applause] [Music] [Applause] [Music] I'm just something okay I you I'm just trying to give you something [Music] I'm just try to something I I'm just trying to give you something [Music] w

[Music]

[Music] [Music] I'm just TR to get this have to F you I'm just dring in [Music] something I'm just dring in [Music] something I'm just trying to give you something [Music] m [Music]

[Music]

[Music]

[Music] uh people in the back if you can hear me via the microphone please nod there's nodding thank you very much uh I'll be brief when it comes to me uh this is the slide that I hate the most but um I grew up online my teenage years was spent on a computer on the internet so when I was going to University I decided Well I want don't want to study what I already know so I'm going to study something else and I studied something else and people keep ask me about this uh what's your education my education has nothing to do with it I studied uh psychology that's my bachelor degree educational psychology for those

who are really into details and I have a master in what's called philosophy of technology I decided to study what I don't know uh I did however take the uh introduction course to programming one and two and that got me a job that's not what I do I don't program my field of specialty what's called Abus ability when I look at your system I'm interested to know how what what can go wrong with this system how can I mess up people's life with this system and because I am actually a nice person I want to figure this out not to hurt people but because I want to change it so we don't hurt people with the

technology that we make if I'm going to give you an advice when it comes to uh having a hobby uh if you're going to fight pick a fight you can win I was stupid I've uh been picking a fight with a bank don't go after Banks that's a very bad idea it's a bad idea because banks have money they have money they have lawyers they have legislations they have laws they have all kinds of resources you don't so if you want to go and start trouble with them be prepared for them to push back um yeah it's a very uphill battle so far uh I'm going to explain to you why it's an uphill battle but I'm also going to tell

you what's the problem and I also going to show you uh what is going on and why maybe the ball is rolling so um in every system that we make there will be design flaws some are small some are big some are proper security issues not every issue is serious so when I spend my free time looking into other people's system I will call it bug hunting without the Bounty there's nobody Norway pays you money for anything it's it's 100% uh as a um volunteer basis yeah but I do this because I care about humans and I care about humans lives and I absolutely adore people in general there are individuals that I really loathe but

people humans I like them I like you so I spend my time thinking about what's the the worst that can happen how can this system hurt you it's a bit of a sad State of Mind to be in so I don't recommend it for everybody I spent a lot of time thinking about domestic violence abuse child abuse grief death sickness poverty and all the other awful things that can hit someone my main question whenever I look at inter thing is what would the worst person in the world do with this Tech and I hope you take a note a mental note at least of this because this is my one guiding question that I think

summarize what is Abus ability and uh when I talk to Tech students I will tell them think about this and if your boss is not impressed but what the worst person in the world you can show them a picture of me and say uh but what can she find in our system we should fix that before she finds it so you can do that I give you the permission to use my picture as a threat that's okay yeah um I did a project uh while back I did this very cute little project uh together with p uh and another awesome human called Yun or John that he keeps introducing himself as when Yun in norian um we did a project where we

trying to figure out what kind of old stuff is lying around and still operative and one of the old stuff that is still R lying around when it comes to banking is paper forms I know I know uh by the way how many of you are from the US could you put your hand up living or from yeah perhaps easier how many of you are not living in the US awesome uh but the US banking system is slightly further on the timeline than a lot of the other countries uh so you may still have paper forms that are being paid attention to but in Norway we have paper forms that are still valid you can fill them out

and send them into the bank but nobody's really paying attention to this so what happened to summarize it very fast I got access to his account and his money uh and that was fun thank you for the money pad that paid my coffee um but in addition to getting access to his money I discovered I got access to a lot of other things as well so I started pting around and seeing what can I do when I have access to someone's account uh I think somebody L dropped their

glasses anyways and I found a lot of small issues small issues are everywhere but I also found some big issues my problem as an external volunteer yeah uh let's not say hacker uh in this context but yeah as a external person I have absolutely no leverage when it comes to the banking in Norway I don't get to set the order of the day I don't get to change their priorities at all so all the big issues that I found I made a note of it I took a screenshot of it and I check up on the issues did they fix it uh once in a while uh but there's very little I can do because to change anything I need to

get somebody's attention and it's difficult because Banks don't care about you they don't pay attention to anything that doesn't cost them

money I can of course go to the media and I have done that in the past when I thought it was ethically sound and a good idea going to the media is not a straightforward thing to do because journalists may not understand what you're talking about and then they won't make anything out of it that happens all the time they were like and like will uh the bank stop working no they won't then why should we write about it you know in addition they absolutely love love the whole story hacker found security issues but when I talk about my type of issues Abus ability type of issues uh they will call the bank and they say hey do you have a

statement because this hacker person or this security person says that you have security issues and they will say that's behind login it's not a problem they're overdoing it it's it's a feature and then there's no story so I just had to hang on to my knowledge to whenever the time is right and the time was right a few months ago because one of the directors of personal banking in Norway got made an interview in the biggest news channel that we have and she talked about his her her campaign about making women invest their money not in Saving accounts but in uh stocks and font which is a good that's a good project but she was blaming women for

not having a good overall insight into the uh economy their personal economy and she was being I think pretty unfair so I did this really old school thing called um a reader post where you write and you send it into a newspaper and I choose to send it into the biggest financial newspaper in Norway and they said yes thank you and they made a huge page out of it this is paper newspaper but they did also come online uh and it was great because they started to listen and the biggest news agency who did the interview today before called me and said do you want to be on TV and radio in a debate with that

director and I said of course I want to which is terrifying because she has a team of people that will back her up and give her PR advice and I had to compl compl my work day and then maybe brush my hair before I got on TV that was horrible um but I did and the debate went well for me I think um and things started happening uh it triggered a lot of other um reader posts and social media and other financial institutions start to have opinions and people started calling me and things well the ball is starting to roll and it's great maybe maybe we can have better banking for private individuals and of course I'm doing this

talk and I'm doing one at Crypton privacy Village uh together with p and I'm also going to this we'll call it a political convention it's a weird thing a bunch of politicians and lobbyists and journalists are meeting up perhaps it should be a Meetup it's called Aran and somebody invited the new director of banking in that bank to have a debate with me again great they're starting well they're I wouldn't say listen but at least they're not ignoring me anymore and fighting me is better than ignoring me when they're fighting me they're starting to pay attention and also doing a national security conference in Norway they invited us to talk and it's great it's we're getting

somewhere but let me talk about the issues because securing a bank banks have a lot of security systems like whole heaps of measures in place there's a saying I guess uh I translate it into English and I'm pretty sure you have it here as well uh as secure as in the bank is it does it make sense yeah there's a nodding thank you uh we have that in nor Norwegian as well banks are supposed to be super secure and by the way I love this picture because because as a lot of you know you don't have to open all those locks you only need to open the one but before you try spend time opening the one check if the door is

already open anyways um the security measures that the bank has is not your security measures a lot of the time this is an overlap between your interest and their interest because you don't want to get robbed and they don't want to going to be the bank that got robbed but this talk is when it's no longer an overlap of interests about security measures that are not there but you

need oh yeah you should go ahead and take pictures I don't mind um because you can get robbed even though the bank doesn't get robbed

basic banking if you're a single person or living in the 1970s would be like this you have a key to an account I call it Vault for this occasion um but most of us live in household with other adults not everybody but a lot of us um if you live in a household with other adults um they may be your romantic partner your children or your parents or relatives in some kind and in Norway at least um we most families are two income families it may be different where you're from I know uh you Americans have much more single income families than we have um but it doesn't really matter um because you are sharing your finances

even if you say that you have a split economy because you will probably only have like one bottle of ketchup you will have one utility bill one internet bill and so on you don't have two it doesn't make sense to have two power bills to your house so you are sharing even if you don't intend to share so when you've been living with the same person for a while you build trust with them and you start realizing well we are sharing so instead of having this hustle every week or month or whatever let's share an account it makes more sense oh by the way um equality I know people disagree about equality um and I don't think it matters what you

think of equality because I think you can agree with me that um shared access to Shared assets is fair and also it should be possible to secure your assets if you want to when I say asset if you're the one who's making money that will mean money but if you're the one who's living as an adult in a household with other adults and not making money you will probably be contributing in other ways so your asset is the money being made to the household that that is money that matters to you you should make take measures measures so that you secure that you're not homeless or starving right and if you want to do that access

control to a bank is not enough this is sharing a bank account we call it power of Eternity sorry attorney um I keep calling it eternity but it's not for eternity it's attorney um it's a fun little trick sharing a bank account like this because in essence there are two locks but you only need to open one of them to get access to the

money what you get access to you intend to share your money probably for paying bills food bills or uh utility bills but what also happens is that the other person or the other persons you're sharing with get access to your history and that's fun because you may be having a new relationship do they need to know what you've been up to the latest 10 years maybe not um and of course the money and depending on your local loss um by giving them access to one of your accounts that is a declaration of that you have have shared finances even though you don't intend to have it what you don't get get access by uh to uh by with the power of attorney is

the bills they're still coming to that one person in that one person's name and of course that is very unpractical and in the US you got something called a joint account instead joint accounts are not available in every country I have not seen them in Germany and Denmark and Norway and a few others I only see them here um I would love feedback on that if you have joint account as a concept in your country let me know and also I will like the name of the bank joint accounts are nice because when you're setting up a joint account you're setting up a new account so there's no history they don't need to know what you've been spending your

money on um and you're making sort of like a third legal person when you're paying your bills it's no longer your or the other person but whatever that account is registered as that is paying the bills and they can uh receive bills as well it's neat it makes sense but again depending on your local laws you're declaring that you have joint finances and this matters when life gets harder anyways um joint accounts and a power of attorney will let you uh opens up for you to be robbed and if that happens you can complain all you want to the bank or your mother or whoever but they would just tell you well you gave them access to your money

you gave them legal access to your money of course you get get going to get robbed it's your problem you trusted the wrong person and I think that's really unfair this is a feature by the way trusting the wrong person is a feature in the banks because even if you trust the right person uh life happens you may have change in priorities that may not always mean a divorce but a lot of the time people fall in love and fall out of love or feel different about priorities in life grief when you lose someone you your mental state are impacted by it accidents happen um that can leave the other person that was the right person now

longer choosing the right things or the things that are in your both interest and I will be very direct when it comes to illnesses because it's very common that people have either diabetes cancer dementia anxiety addictions depression and a whole other things that will affect their mental capacity to making good financial decisions so even if you chose the right person the love of your life this can unfortunately still happen to you and they can also be um scammed so we need more measures in place in banking that take care of your interest not just the bank's interest

to add insult to injury banks have this they have account types that have all the measures that you want that have double signing or triple signing even if you want that have transaction limits daily limits monthly limits weekly limits and so on they have alarms they have notifications so you can have uh either on email or on your phone or wherever even in your banking app uh and they have logging logs of who logged on and did what but you don't get access to it as a private person you only get access to it if you are a business you don't have to be a big business you can be a really small business you can earn less than

your household and you still get access to this if you're a business and I think that is unfair and I want this for private personal banking yeah if you have a bank that do any of this for personal accounts I would love to know because in order to well win or at least make the fight harder for the bank where I'm at I would like a good role model or um a little pear pressure to put them under so that they uh give me this because I know technically speaking this is quite possible it's just a matter of priorities so yeah um that was it I I was super fast this time do you have had

any questions yes I I can make mention about some of the issues that can and that would be um thank you one of the things that I find that's in our country that a a banking issue and no offense to any parents here but when your parent um when you want to initiate a banking I remember one of my friends she was 17 uh she had to have her parents sign for it um now that that person I think she's in her 50s uh till this day she can't get her mom removed so of course her mom is asking her you know uh what did you spend that $100 on and you know it's kind of annoying to her not me but her

and um that's something that you know she's banked with this company or this banking system for over 40 years but she she can't get her mom removed yeah so I think that's that's a issue and vice versa if you have a person who a child and you open an account with them for 17 years and they still have that banking account and that person is um is not very staed in their finances overdraft you know all matters of issues then that parent is still sign signed to that account yeah um I'm guessing you're in the US yes yes uh you have a credit system here that is um different um than other countries uh first of all uh when you grow up or if

your life changed a lot start over when it comes to banking in the sense that get another bank because that's how you sanitize access um trying to tell the bank that they should do something different they don't listen I'm sorry they don't listen uh start over uh be a good consumer threaten them tell go to another bank and say you know what um give me a good offer you may not always be in the position where it's um you you are the best uh customer but they don't know that yet so try be bold um

yes the last comment question was actually kind of tied to um my thoughts around the credit coring this the the credit scoring system here so in that case if she starts fresh right her credit tanks because it's based off age of accounts too here for different FICO levels yes because FICO sucks I hate FICO yeah if you want to talk to me about FICO I plan to have a rage campaign against it but it's another story but as far as like credit scoring and say I'm sorry Norway right yeah so how does that how does like account age and stuff like that impact accounts in like Norway and I don't I don't know what the credit system looks like there

but I assume it's better very um but I can talk a little bit about it because um in Norway we have a well first of all we have a um credit system where you can um you have a credit score there but it's only available for businesses usually and they have to have a solid good reason for checking your credit we also have a registry for all your debt we didn't used to have that but now we have that uh it is so that you cannot take up a lot of credit around everywhere and be deeply in debt um because it's unfair to put you in that position uh and it's also very unfair to put your loved ones and people

you're sharing your economy with in that position um so we have that um both in Norway and in every other country I've talked to or people um as a romantic partner I cannot do a credit check of my uh love interest um I'm not sure if you should be able to but it's an interesting thought to follow through uh or think about um because when you're starting a relationship you're entering it with trust and that trust may not be um um what we call it uh valid or a reasonable warranted thank you um so yeah on the other hand it's very important for personal freedom to be able to take up credit without your current romantic partner or parents know

about it because you may need those money to get away domestic violence uh Financial violence is a huge part of it um so it's not an easy thing to change things but I will support you if you want to Riot for a better solution yeah uh and also when you're changing Bank you can do do it slowly like ghosting them you don't have to do it like on day one just ghost them slowly hi um so I guess for your last slide what I guess what I'm trying to take away from it is you know a lot of fex now have that for personal accounts even um where I work does the same um I

I guess what are you trying to like gain out of that uh because for your your last example a lot of that would show that someone's trying to set money aside if they're trying to escape a situation like you have all the like where do you want that line so um the line is a whole debate uh because it depends on your local laws and it depends on um other context uh like your income and how that is in so on um but one important part of this is that if both need for everyday money like for cash like pocket monies and so on you can have your own account that's not a big deal but for savings um it's kind

of absurd that my spouse can take all our savings and run away with them or drink them up or buy that Bloody motorcycle um without me knowing and approving so if both have to sign for things uh don't do that for all your money that will be insanely impractical but do it for the huge amounts and yes then you end up in a debate where you probably will have to divide assets called a divorce or a splitting of assets um if you cannot agree but if you cannot agree you're not agreeing but at least one person is not robbing the other one and leaving them without any money um

yeah oh and by the way this happens so many times hi Cecil great presentation thank you yours too very nice um I just happened to leave in Norway and we had the chance to meet before I was just wondering during your presentation it came to my mind if you have any insight from uh related to Abus ability applied to the bank ID system have you would you have any thought to share and if of course you could just spare spend like 30 seconds explaining to people what B is which in my opinion is pretty robust yeah yeah so bank ID is the um well I I was the ciso for bank ID for three years that was my previous job

bank bank ID in Norway is uh the system that all Norwegians are using to identify not only to Banks but all to every single government service you can think of and insurance companies and so more bank ID has 95% market share in Norway at least so it's the solution being used uh and and it's a digital solution ID solution uh that comes in in uh two levels high and should I say medium security um and you use that and as an example when I sort of we we did our little attack that we are also going to talk about at the Crypton privacy Village at Defcon on Friday um I was you know um I wanted Sicilia to obtain legal

access to my account but we didn't want to do anything illegal uh so I asked would you like access to my account and my money and she happily accepted that we fill out a paper form of one single uh one single sheet of paper with my information and her information and two witnesses that were absolutely clueless to what they were signing for we sent the paper form to the bank and suddenly she got access and uh additional bonus after the access had been set up I didn't receive any message from Bank from my bank that they had given her access and she didn't receive any information either from them like a text message email or anything

that she had access to my account and after I gave her account I suddenly realized [ __ ] I have 10 years of payment history on that account as well and she can see everything all the money I have ever received and all the money I have spent for anything that was like okay well so you're a good trusted friend uh but Facebook would love that info yeah so she's going to auction that off later on today I guess my personal information yeah but yeah that's the bank ID system uh and from there it's uh it's interesting to see that you still need Alternatives like paper forms for people that are not digital people that cannot

use bank ID for many different reasons like you having a legal guardian being sick uh we still have you know one challenge to us is as an example we are taking in quite a few uh UK Ian refugees and they are not immediately allowed to have bank ID because we don't have any credit history we don't know who they are it's difficult to verify their identity so we don't give them immediate access to using bank ID as an example and then we have paper forms yeah um there's there's a lot of good reason to have paper forms for now anyway uh the EU is doing a very interesting project that if you have a passport uh and you

have a established ID uh you also get an digital ID bank ID has been doing that for 95% of the population in Norway for a while but the last five is still a significant amount of people um so there's there there needs to be something done for them and for for now they're living on their parents' accounts or other relatives or romantic partners and so on and so on and it's a mess it's not a good mess um and I accept C that payer forms has a function but nobody's paying attention to them so that's a problem um but most of all uh there is no way that I can give anyone a little bit of access

to my money it's either nothing at all or everything and that is too binary um yeah do you know do you know raise your hand do you know anyone any business or any Corporation any government organization that actually verifies written signature so a piece of paper oh that's fun um one one two three okay yeah we haven't seen anyone in Norway for several years that was one of the fun parts of that project that was I was signing this paper and they're going to well if they was checking my if they were checking my signatur against what when I was five um it didn't make sense at all because we have digital banking so much digital banking um so yeah that

was and when I was going to fill out the paper form I I looked around my entire apartment I didn't have a single pen I haven't used a pen for for many years I'm All Digital so I had to go out and buy a pen to fill out a paper form just trying to do social engineering against the bank it's we don't do that anymore yeah uh the future uh two things uh you can find me on socials and everything uh LinkedIn is probably the easiest way to get in contact with me if you want to I would love to drink coffee and have conversations with you um but also I'm hoping the banks will start giving me at

least a way of making two signatures happen because it's possible um and uh also I'm hoping with modern banking that maybe my next bank is not a Norwegian one but a German one who possibly may have maybe maybe these uh control measures in place I don't know but I think um I think banks are in deep water uh it's a global market now yeah yeah thank you [Applause] Cecilia and it's uh lunch time we'll be back here at Pas riscon at 2 o'cl with Troy and Kathy uh with detecting credential abuse [Music] [Music]

[Music] is

[Music]

[Music] [Applause]

oh [Music]

[Music]

a [Music]

[Music]

[Music] he a [Music] the [Music]

oh e [Music]

[Music]

[Music] [Music] [Music]

[Music]

[Music] n

[Music] [Music] [Music]

[Music]

[Music]

[Music] [Music]

[Applause] [Music] he hey hey hey he [Music] [Applause] [Music] he [Music]

he

[Music]

[Music]

[Music] track [Music] hey hey hey [Applause] [Music]

hey hey hey hey hey hey [Applause] [Music] [Music]

[Music]

[Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music]

he [Music]

[Music] [Applause] [Music] he [Music]

[Music]

he

[Music] h

[Music]

[Music] [Applause] no w w w [Music] [Applause] [Music]

I'm just trying to give you [Music] something I'm just tring I do I'm just trying to give you something [Music] w

[Music]

[Music] [Music] I'm just TR to I'm just TR to [Music] something I'm just tring something to I to BR you I'm just trying to give you something [Music] w

[Music]

[Music]

[Music] n

[Music]

[Music] he

[Music]

[Music] [Applause]

[Music]

[Music]

[Music]

oh

[Music]

[Music]

a [Music] e [Music] the

[Music]

a [Music] [Music]

[Music] [Applause] [Music]

[Music]

[Music]

[Music]

[Music] [Music]

[Music] a [Music]

[Music]

[Music]

[Music]

[Applause] he

[Music] [Applause] [Music] [Applause] [Music] [Applause] [Music]

heah [Music]

he

[Music]

[Music]

[Music] track [Music] hey hey hey hey [Applause] [Music]

hey hey hey hey hey [Music] I [Music]

[Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Music] [Music]

[Music]

[Music] [Applause] [Music] he

[Music] what [Music]

he

[Music] oh a [Music]

w [Music] [Applause] [Music] [Applause] [Music] [Applause] [Music] I'm just trying to something this I to you I'm just TR to give you [Music] something I'm just TR to give you something I'm just TR to give you something [Music] oh [Music] [Applause]

[Music]

[Music] [Music] I'm just try to get this something I do I'm just TR to [Music] something I'm just trying to give you something I do I'm just trying to give you something [Music]

n [Music] w

[Music]

[Music]

[Music] [Music]

[Music]

[Music]

[Music] [Applause]

oh [Music]

[Music]

[Applause]

[Music]

[Music]

[Music] e [Music]

is [Music] a [Music] [Music] [Applause] [Music]

[Music]

[Music]

n [Music] [Music] [Music] [Applause] [Music]

[Music]

[Music]

[Music]

[Applause] [Music] hey [Applause] [Music] [Applause] [Music]

he [Music] a [Music]

[Music] a [Music]

[Music]

[Music] TR [Music] hey hey [Applause] [Music]

hey hey hey hey hey hey [Applause] [Music]

he [Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Music] [Music] he [Music] [Applause] [Music] he oh [Music]

[Music] now

oh

[Music] h [Music]

[Music] [Applause] [Music] [Applause] [Music] [Applause] [Music] I'm just okay I I'm just TR to give you [Music] something I'm just try to give you something I do you I'm just trying to give you something [Music] m [Music] a [Music] [Applause] [Music] [Music]

[Music] [Music] I'm just in I'm just in [Music] something I'm [Music] just I'm just trying to give you something [Music] he [Music] m [Music]

[Music]

a

[Music]

[Music] [Music] a

[Music]

he

[Music]

[Music] [Applause]

[Music]

[Music] [Music] oh yeah [Applause]

[Music]

[Music]

[Music]

[Music] a I

[Music] h

[Music]

[Music]

[Music] [Music] [Music] a [Music] [Applause] [Music]

[Music]

[Music] oh

[Music] a [Music] [Music] [Applause] [Music]

[Music]

[Music]

oh [Music]

[Applause] [Music] hey he [Music] [Applause] [Music] a

[Music]

he

[Music]

[Music]

[Music]

[Music] TR [Music] hey hey hey [Applause] [Music]

hey hey hey he he [Applause] [Music] he e [Music]

[Music]

[Music] [Applause] [Music] a [Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Music] [Music]

[Music]

[Music] [Applause] [Music] he

[Music]

[Music]

oh

[Music] h

[Music]

[Music] w oh [Applause] [Music] [Applause] [Music] I'm just something I'm just dring [Music] something I'm just dring something to BR you I'm just want to give you something [Music] [Applause]

[Music]

[Music] [Music] I'm just trying to I'm just TR to give you something [Music] I'm just trying to give you something okay I do for you I'm just trying to give you something [Music] m [Music]

[Music]

[Music] [Music]

[Music]

[Music]

[Music] oh [Applause]

oh [Music]

[Music] [Music]

[Applause]

[Music] oh [Music] e [Music]

[Music] [Music]

[Music]

[Music] a [Music]

[Music]

[Music] [Music] [Music] [Applause] [Music]

[Music]

[Music]

[Music] the [Music]

[Applause] [Music] heyy hey hey hey

[Music] [Applause] [Music] [Music] n [Music]

he

[Music]

[Music]

[Music] TR [Music] hey hey hey [Applause] [Music]

hey hey hey hey hey hey [Applause] [Music] [Music]

[Music] a [Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Music] [Music]

[Music] [Applause] [Music]

[Music]

[Music]

he

[Music] h oh [Music] oh [Music] [Applause] [Music] [Applause] [Music] [Applause] [Music] I'm just trying get some this okay I to you I'm just TR to give you [Music] something I'm just TR to give you something I'm just TR to give you something [Music] w he

[Music]

[Music] [Music] I'm just TR to give something I I'm just TR to give [Music] something I'm [Music] just I'm just trying to give you something [Music] w [Music] w

[Music]

[Music] [Music]

[Music]

2 o' we have Troy and Kathy from uh Google from all the way from Australia really happy to have you here and uh detecting itial abuse looking forward to this one please welcome both Troy and [Applause]

Kathy if I turn it on yes that's it okay is that yeah I think that's good right yeah okay cool um hi there folks thanks again for your time today as well um and thanks for the opportunity to present to you all and to the volunteers making it happen um before we introduce ourselves I'd like to pose a question to the audience here how many of us in this room full of security enthusiasts has been the victim of a count compromise fishing stuff like that I was really nervous when I thought I was going to ask this question cuz I had no idea how many people would put their hand up actually it looks to be

roughly speaking about like a third maybe 40% of the room the rest does know yet maybe no maybe so right and I think the interesting here the interesting part of this is like we're a hardened Target right and if you can imagine if we asked this question in a in a room for of folks who are maybe not as enthusiastic about security as we are um that number is likely going to be significantly High um so we're here to kind of talk about a pretty common attack Vector nowadays um that we still see kind of happening in pretty large scale in the wild um it's worth noting that the detections we'll be talking about here are um possible and

commercially available tools open source and there's nothing specific to the company here um but in the interests of like furthering both the red and the blue sides of the per coin um we're going to look for basically looking through the needle looking for the needle in the kind of like log Haystack just to introduce myself quickly I'm Troy I'm originally from South Africa I've spent most of my life in the UK and Australia i' secure rarely leave the context upon which they were generated this contest could be a browser for cookie for example a device or a TPM a trusted platform module for those who are not familiar with the term um they there are exceptions of course

for example credentials such as passwords or barrer tokens uh they often times intended to be transportable but many credentials um rarely leave the original context they were generated from and therefore can leverage that for our detection credentials are also rarely accessed outside of a specific set of executables for example cookie jobs are only accessed by browsers most of the times but if you see a random binary c. EXE accessing your cookie jar then there's something wrong another example of this could be we see many executables signed by the same certificate accessing uh C credential and all of a sudden an unsign credential is accessing it this will stand out aside from password pass ke Cas MFA many

credentials also rarely get access by end users typically access to credentials are automated or per formed by a program although human access is not impossible it can be interesting in the context of other detections as well okay so before we start talking about how to use these properties to implement credential abuse detections I would also like to address a very important prerequisite we need to talk about logs we cannot talk about detections without emphasizing this point logs are extremely important a crucial precursor to everything we we're about to talk about going forward we we'll be assuming we have good log availability and access we have the ability to ingest filter and detect maliciousness within a reasonable

time frame or latency we're not going to talk about how this can be done as this is not a detection pipeline talk but we can chat about this afterwards if folks in the room are interested okay so just to reiterate the points the main takeaway here is your first step shouldn't be try to implement the detections it should be making sure your logs do what you need them to do as your detections are only as good as the data is built off you know following the garbage in garbage out principle you'd want your pipeline to log the right things reliably log the right things retain these data for a reasonable time frame make these logs available for

ingestion and tell you when these things aren't working as intended obviously this requires resource compute memory and disk space to enable but they are very important to build reliable detections okay I'll pass to Troy to talk about credential abuse detections now thanks but all this being said let's talk about um how we can use what we know about credentials and and how attackers can use them to to try and detect some Badness so credential theft often manifests itself as some kind of identity impersonation compromise of a user account something along these lines and there's many many vectors through which credentials can be stolen this is commonly the result of things like info stealing malware um of which there are

many many variants every day in an even greater number of infections that we see around the world but it also could be things like cross side scripting um dumping and exfiltration of credential databases all these kinds of things that effectively result in credentials ending up in places we don't want them to be um we won't be talking about all possible variations of detecting credential abuse there's a lot of it as you can imagine uh we'll be talking about um a couple of examples including the impossible travel problem which a lot of people might be familiar with um credential binding violations and also discussing credential forgery as well um which is similar in terms of impact um on the end

user accounts here but very different in nature when it comes to detection engineering um whoop that was too fast so to talk about impossible travel so for those of you who haven't kind of experienced as much before let's say we have a user that's normally based in bogatar in Colombia that's their main working location this is where we see them appear from in log data and where we see their devices talk to our infrastructure where they originate from hypothetically after a period of time we then see authenticated user traffic originating from Seattle the question becomes is it possible for for a human to feasibly travel between these two locations within the time frame that we've

seen this essentially boils down to a question of how fast would they have to travel to get from A to B and is their velocity above what we would consider possible in that time frame for example the velocity of a commercial passenger aircraft is is probably fine within reason but if we see people traveling beyond the speed of sound unless you're in a very specific set of jobs you're probably not going to be doing that regularly but remember that all of these observations and speed calculations come from log data as Kathy mentioned a second ago we need ways of determining where the user is appearing from first of all this is kind of this concept of

geolocation IP address is probably the most common means of figuring this out generally speaking but if we see the user in two places when looking at IP indicators there could be many other legitimate reasons why this happens things like VPN usage for example is the most common one and remoting it to devices somewhere else or geolocation attribution sometimes changing on a given IP address block because this is just how the internet works so we need to move Beyond just did we see a low Fidelity in indicator present and we can't really solely rely on IP addresses and we really want to try and figure out where the real user actually is based on the other indicators that we see so

password is unfortunately also a weak indicator things like you know fishable credentials that are designed to be somewhat portable for the user if we see a user authentication event and all we all they provide is a password we can't really say for certain that it's the real user actually at the keyboard and this is particularly D due to the port portability of passwords and and their fishability soft back backed MFA creds things like SMS security codes for example are slightly better and particularly if we see them in combination with other factors passwords and security codes this is like the concept of two Factor authentication nowadays basically but again SMS things and like soft back security codes are designed to be

portable again and reasonably fishable and some of these are vulnerable to outof band attacks like sorting the effectiveness of soft um MFA tokens really depends on the specific token type and these can be quite varied which makes it quite hard but we might be able to determine for example that a company-owned device is present at a given location if a hardware backed credential issued to a device is used as a second factor or if we issue users with Hardware MFA tokens UB keys for example or if our infrastructure requires the use of mutual TLS with a certificate that's stored in the TPM this at least suggests some kind of legitimate strongly tested factor is present but it still doesn't

mean the real user is there stallen devices cred sale um which we'll talk about shortly all of these are situations where we might not see the real user at the keyboard here and we kind of need Biometrics to be sure almost which I'll talk about in a sec but finally observing the same session based credentials used from two devices or two IP addresses at the same time is typically a strong indicator for account compromise session based grant things like cookies for example typically are not designed to be transferable between devices so why would you see them moving that's the real question you have to ask so when we see an event take place where indicators like these are present

we need to figure out like you know which indicators do we see of geolocation and of user device presence how confident are we in their Fidelity their portability their transferability what activity do we see before and after the event and does any of this seem strange um and do we have other log sources from which we can enrich this data did the user badge in recently to an office for example if so was that office location near one of the locations we see them in log data and and combining a bunch of these indicators can give us a pretty good idea where the user really is uh and therefore May what may have happened to

trigger a detection like this but this is really dependent on the indicators that we see and what we know about the user and this changes our confidence level in the detection itself and also how we investigate it but again the effectiveness of this detection this detection pattern The Impossible travel problem depends on another variety of factors for example we want to be able to correlate as many log sources as possible the this we really want like strong log diversity you can think of this as where we have as many relevant indicators as present are present across as many log types as we can because the more logs that we have the more diverse data points we can

geolocate and calculate the distance between this gives us generally speaking a higher Fidelity signal otherwise we heavily limit our visibility to the specific log sources that we choose and the attacker just may hide in one of the ones we're not looking in similarly but kind of almost inversely we want High log granularity we want as many event types to be logged for any given log type and this means that the more events that we have for a given log the more geolocation indicators that we can compute and we can compute the signaling more frequently and with higher um higher Fidelity but there is a trade-off here the biggest problem is the more stuff that you log the more stuff you ingest

this basically means the more thing you have to account for in your detection logic and the more expensive things become you have to store a hell of a lot more information so implementing a good detection here gets increasingly complex um both in time and resources but even Beyond beyond all of these indicators it's important to consider the context of the events that we see firstly we need to consider the devices we know the user might use for example um in traffic rooting terms mobile devices over mobile networks um or Knack gateways at Wi-Fi on Wi-Fi networks at home General Mobile roaming um Network rooting um all of these things affect user attribution and geolocation accuracy users also Express different

behaviors from mobile devices for example in comparison to desktops or laptops and some devices just might fundamentally not support the security features we need for some of this stuff some things may not have a TPM for example we may not be able to Hardware back credentials it's also typically rare to see multiple concurrent active sessions on multiple devices most of the time people will use one device at a time interactively they may flip between two in very quick succession if you have two laptops next to you for example and but they'll typically have one main device and this won't be for a very long time so that's to say that observing two concurrent fully interactive sessions

from different devices should almost definitely arise suspicion and particularly if this happens over a long period of time and it can be hard to determine interactive here but it's possible within reason um but the fundamental question here really still comes down to how can we be sure that it is the actual user at the keyboard at the location where we see them or more specifically how can we be sure that it's the actual user expressing the credentials that we see user behavior and attack of behavior for those of you in red teaming roles we'll know are very very similar um depending on the role of of the user and we can tune our detection output to any volume we like

almost and we need to consider the volume that we can deal with either manually or through automation those in blue team roles might be very familiar with noisy detections being tuned based on capacity rather than accuracy and the CH this is also a challenge here in the impossible travel problem um unless we have the right data the right enrichments the right automation um to tune this stuff appropriately and for those of you in red team roles um this is where the attacker really wants to blend into the noise they want to be able blend into that volumetric tuning limitation of this detection pattern and as blue team as we also need to consider this angle

when we investigate these events but we also need to be considerate of the user we don't want to inconvenience the user with unnecessary remedia of AC actions or we risk them circumventing controls that are too cumbersome but those strongly back credentials those mtls certificates UB keys and things like that we mentioned before can help us answer some pretty important questions for example as we talked about briefly did we see any company issued devices to present at a given Source IP address if so we need to consider the context of what happened a company device being seen at a given IP address might suggest that this is the user's home IP address location for example or their typical working

location or at least that someone in that location has access to a company issued device not that may not be the real user this could be an attacker could be a family member could be almost anyone if not we need to consider the credentials that we see how these are issued and how they're stored on the device but this can also help us answer a question of do we see evidence of human Presence at a given Source IP address observing human presence from IP addresses that are geographically far apart is the effectively the essence of this detection pattern of the impossible travel problem but again we need to attest user identity to be 100% sure

within reason of of who's actually there because human presence can be attested more easily than attesting the real user identity that might be present at the location usage of second Factor tokens that aren't biometric login activities using fishable or transferable credentials interactive netflow or process logs on the device these are all signs of human behavior they may be automation as well to be fair and you can pretty normally figure that out quite easily but it's not actually the real user you can't be sure from logs which although is a more difficult question to answer we now come to our final question of can we determine which of the active sessions is the real use of the person we issued the credential

in the device to if we have biometric Hardware back second Factor authentication here this can be pretty conclusive in determining what's going on um specifically that we can know within reasonable bounds which session is likely the real user this also has caveats when it comes to enrollment we can also interdict user processes with additional biometric checks things like selfie verification for example but ml models are getting really good at rendering video and you can imagine the application of that there but effectively determining real user presence typically come comes down to the best possible combinations of trade-offs and limitations in the impossible travel detection problem and it's worth noting that for non-biometric Hardware second factors we

we can't make the same attestation here we could only state that a human was present at a given location but we'll talk more about this when we talk about credential sharing and sale but to summarize the points on the impossible travel detection problem real quickly um if you find yourself implementing or investigating the output of one such detection like this try and combine as many indicators as you can and wait them as appropriate some logs are going to be more relevant than others we want to filter out known background activity and or noise where you can um because we don't want those things to trigger impossible travel events and as a result you don't need to

ingest everything but we want to try and avoid shooting a detection like this for triage capacity because it's much better to implement better detection logic and noise reduction and enrichment so we can actually outscale the attacker rather than just burdening ourselves with toil we should also consider how much trust we place in credentials that we can see being presented if we see high trust credentials and we got confident in their security posture we might be able to imply a given session as the real user within reason we also need to consider geolocation granularity and maximum velocity you might have to ask this question of like how far does someone have to travel for this detection to

fire given how accurately we can measure geolocation and how fast do we think is too fast some cases maybe fundamentally too fast like bual to Seattle example but some may be just unreasonable like unreasonable sorry like for someone who would fly from Manhattan to newer because it's probably not going to happen in real life sometimes it might be better just to ask the user what happened if you aren't sure um often if we aren't sure it's just easier to ask to understand the weirdness but there's caveats with this which we'll talk about shortly as well but it's also generally worth trusting gut instinct from what we know of the user their role within the company their likely work times and

locations is this likely to be them you can kind of normally figure this out although blending into the noise as is a thing um attacker Behavior does generally stand out again with with a massive set of assumptions there so Instinct and ooms Razer can help make these gut decisions a lot of the time it's also worth keeping in mind the limitations of impossible travel detections um unless session based credentials are available available as indicators here the detection itself is going to be heavily dependent on geolocation changes to a users account and we talked about how some of those are better than others and also some of the challenges there as well and finally in practice as coverage of detection

increases uh so does the noise level it's important to complement the detection with additional controls for example credential binding which I'll hand back over to Kathy to discuss all right so as Troy just mentioned credential binding is a great control to complement credential abuse detections so this links to a property mentioned previously some credentials we look to secure should rarely leave the context upon which they were generated and if we can bind a credential to a context for example device we can in theory detect its theft when it leaves that context and get presented back to us so let's take a closer look at how credential binding works so we have a identity provider IDP and a client where a user

wants to authenticate via client and get the credentials they can sub subsequently use for example a cookie we have some binding conditions configured that we want to enforce we need to compute The Binding criteria when we receive their authentication data generate the credentials we want to return and then apply our binding criteria and log the credential generation event and then return these Bond credentials to the client and while a user try to use the bond credentials they would present these credentials back to the IDP when um and then the IDP then compute The Binding criteria again check that our computation matches those present in the check that our computation matches um the bind the the B the binded

criteria that's present in a credential or presented by the authentication client and then make a decision of whether the credential binding control has been met and in this case um because it's a legitimate client let's assume it has and log the credential usage event with the binding control validation result and then return the data or content to the user as requested but in the case where these credentials were stolen or used from a different malicious device for example we still do the same thing compute The Binding criteria again check the criteria make a decision and then log the event but in this case as you can imagine the control will not validate and we we will be logging a violation of

The Binding control and this is the event that we look for in our logs when we write our detections and then we would like some sort of actions to be taken to remediate the situation we might expire the suspicious session lock the user out log false re authentication there are false positive use cases here too for example um like a user might accidentally back up their device and or reimage their device and get issued a different certificate that doesn't match the existing binding but it is their credential they're trying to use use uh it's important to consider these edge cases um When developing detections and similarly to our impossible travel data points there are a variety of binding mechanisms to

consider although the usefulness of these controls may vary depending on the context and their use cases so starting from the top we can bind a client site credential to a machine certificate for example that is backed by a TPM this is a strong binding as it is non-trivial for the attacker to retri retrieve and lift data from a TPM remotely TS Channel binding is an alternative mitigation which could also be highly effective Channel Bond credentials are bound to the underline TRS session for example cookies may be Channel Bond or all tokens Can Be Channel bond with ch Channel binding leaked or stolen credentials cannot be used in the attacker session the attacker must compromise the

victim's browser session directly both the top two op ions um they protest against credentials taken out of the original context and used elsewhere however they don't protest against abuse from the compromised device or session directly and the compromised credential may be ex changed for new Unbound credentials that can be used elsewhere this is not foolproof but it's highly effective to spot a class of attack and next we have IP addresses and net blocks attacker in possession of a credential needs to be in the same IP address range or originate from the same IP address from where the credential was generated to make use of the credential we can detect you know offs SEC mistakes basically um very easily using this

approach but there are a lot of options for the attacker to bypass this um binding like using VPN or large retail ISP ranges specific IP address might be a bit harder but not impossible depending on the circumstance as well and finally we'll have time it can be considered as as a binding control in the form of an expiration window Unbound credentials can be expired to mitigate the impact of any potential compromise having said that time in general is a weak binding control this is where we see a lot of credentials targeted by credential St today and automation can be easily used to bypass time windowing as well so we just talked about controls that try to prevent credentials from

leaving the original context to leverage these controls for detection it is important to understand why a binding control was engaged in the first place if a binding control files this implies the credential has been moved moved elsewhere we can mitigate this by force the user to log out but it we should still know why did this happen and investigate it um it could be a Mal infected laptop for example or it could be just user procuring a credential for attacker we never know until we look it is important to remember that the attacker can easily try again uh so just because we mitigated this one time we need to make sure we keep the cost high for the attacker per iteration

as well we've been talking in depth about credential theft so far it's time to talk about a similar threat credential sharing and sale this technically is not a standalone detection pattern but nonetheless an augmentation of what we've discussed so far the difference is complicity and intent in the case of credential share sharing a sell the user likely completed this means we need to consider intent in our investigation someone might be sharing the credential so they can get access to a personal dog save on their CP account uh because their family member they're traveling and their family member is still at home they could also be sharing their credential because they have sold them post deuse context enrichement is very

important here ideally we should find out what actually happened after the credential were used in an unexpect unexpected Manor um and also biometric Hardware backed MFA tokens are very useful here if we see evidence of a hardware backed biometric token used this is a strong indicator that the user is either present or complicit not if a user is complicit we also need to be careful during our investigation and response first we need to consider when and how to reach out to the user this can be a very sensitive discussion don't don't underestimate your time investment um be especially conscious of your tone and don't assume malale intent just because someone gave away their credential doesn't mean they

personally hate you for example they could just be asking a family member to get a personal dog they could be a in a very hot position and this seems to be the way out manufactur many factors can be easily Mis assumed so be sure hypothesis and prove don't assume next we have one notably different detection pattern this one is forgery um as Troy mentioned previously we have the storm 0558 case from the last year as an example at a high level we can think of the attack in a similar way to that particular case normally an IDP will query a key material stored in a in a storage device to sign a credential this is where highly

sensitive key material is typically stored but then an attacker might use a foothold to compromise or leak the key material somehow and then ex exfiltrate the key material to their C2 infrastructure and the key material can be used to create and sign a credential out of Band of the canonical IDP and then they can provide the credential to authenticate themselves which the IDP would recognize as cryptographically valid and Grant access and this is the whole flow to detect this type of attack let's take a look at some logs let's say we have time extending down we have two streams of events from the standard credential workflows we have a creation stream and a usage stream both streams should have

a unique ID for each generated credential such that we can map every usage event back to exactly one creation event that happened in the past with this premise holds true we can then check for the violation of this condition and this is basically what the use um um the usage of a forged credential look like a credential usage event with without a cross bonding creation event we don't have the creation event because the attacker minted the credential out of band and it wasn't created by the canonical IDP which wris the log having said that there are some underlying assumptions and limitations for this detection that we haven't mentioned so far so the first thing is

our patent here assumes that the forge credentials are generated out of the band all the banned by the attacker and then used against the canonical IDP so the system that issues the credential does not retain the state of any generated credits so The credibility of the presented identity is solely established based on the valid validity of the cry cryptographic signature and then secondly the credentials here cannot be forged online IE the IDP itself is not vulnerable to any exploits that affect credential generation so the Integrity of the generation process is strong and we can trust it in terms of the limitations of this detection this detection pattern requires logging to be as close to 100%

accurate and complete as possible for example if we don't have a credential creation event we need to be sure um like it's not a log loss somewhere an alternative to this is State statefulness um if the IDP itself the server side maintains States for the issue credentials we can then just check the state machine for given credential to ensure its authenticity and alert on any invalid credentials being presented to the system and I'll hand back to Troy about what to do about these detections thank you all right so you can imagine if some of these detections fire we obviously want to do something about it right um so you might be wondering tangentially to this why not just detect all things

like info stealing malware um well the problem is there's just so many ways to steal credentials and we often have quite not that much control over how and where they're stored trying to detect all the theft options here can be really expensive timec consuming and also highly variant meaning a lot of this can be hard to write but there's a very small number of places where the credentials that we're talking about are used validated and checked and these are all like core identity providers and we also have much typically at least we have much more control over those core idps um as well as where and how they log things so we can Implement detections basically at the choke point

and save time and effort there with greater accuracy rather than trying to solve the M execution case for example that being said this isn't to say that we shouldn't do anything obviously except right right rules looking at idps right like detection and depth is still a completely valid concept here in the same way that we want multiple detections to fire for Badness we don't want this one cigra detection rule to fire in in the worst case either okay let's say we've detected some credential abuse and we know it's legit like now what do we do and we've probably already answered questions such as where was the credential generated was it generated out of band which user

was it account account was it tied to how was it stolen how we just do we suspect it was stolen how was the user potentially complicit things like this but we still have a lot of questions to answer like we need to determine the impact like how widespread was the activity that we saw what did the credential do Post compromise and then we need to figure out what to do next we have to first of all figure out is the credential still even valid and and if so when do we kick out the attacker and then what do we do next like do we need to like urgently fix something even as like just a stop Gap fix or do we like

urgently need to deploy additional controls and what do we do later like do we have to fundamentally fix something do we have to fundamentally deploy more controls really fundamentally rework our application of identity and this is this last one's probably overkill for the most part but many an active directory of forest is re being rebuilt for this very reason but at the end of the day if it's the user credentials that have been compromised we always have to be cognizant of the human in the picture we mentioned before the need not to assume situations when in investigating suspected credential abuse um we don't want to assume the intent the circumstances because log data does not

contain the entire context of existence users might be complicit for reasons beyond their control so we really can't always assume Mal inent here we should be sure of what's going on and then act upon it as Kathy mentioned before we should we're probably going to have some hypotheses which we should be able to reasonably prove but then we should take action only after we've done that obviously here a strong security culture blameless culture all of these things are really worthwhile Investments because we need to acknowledge that all of us including the users that we're protecting are on the same side and we need to work together to make this problem harder for the attacker because really you don't want

to be the security team that's feared this makes collaboration significantly more difficult and fundamentally makes it harder to build trust with the very things you're trying to to protect including your user base so then we're going to talk about a few controls quickly we talked about credential binding as an example here um but statefulness Kathy mentioned is is a really good example if you can track the state of a credential we can validate it State when presented back to us and invalid invalidate sessions where we SE an invalid State transition and this requires the attacker to be a aware of a state machine and of the credentials given state in a point of time within

that state machine which is very hard to do unless you're internal we can also start automating expiration or forc reauthentication processes these are good in theory um but you know if the initial authentication challenge is weak or fishable then the attacker might already have fished all of the components that they need to be able to pass the checks we also especially need to consider usability when it comes to controlling credential abuse if we challenge too often or if the challenge effort is too high then users are going to start working against us and usability here is really a function of the challenge overhead we're implementing the frequency at which we present it and the actual effectiveness

of the control and we need to get this function right to avoid overburdening the user human expiration and Remediation these are kind of like the um actions that may be taken by someone who investigate one of these incidents sometimes you might need to know more before control kicks in but the challenge here is to know when to act um with this can really vary by the user that's been targeted the processes in place but we don't want to alert the attacker or inconvenience the user and this is a hard balance to strike but we also don't want the attacker to progress too far either so there's a very delicate balance in making these decisions but this doesn't mean however

uh sorry this does mean however that we can be considerate of what is appropriate given the user's needs just want to talk about obscurity before we close out um it's obviously a meme to like solely rely on on obscurity um but in attacker doesn't know what they're up against so a single obsc failure can often lead to detection if they don't know all the impossible travel data end points a data point Sorry or if they don't know the binding mechanism or if they don't consider user working patterns or something and they guess and they fail that's when you can detect the weirdness it's obviously a not to not solely rely on obscurity right but like

don't discount it in reality cuz if we Implement a control with high complexity which is as transparent as possible to the user this forces the attacker to make a leap of faith and we want the attacker to guess and we want that guess to be really hard but to conclude really quickly um we talked about some C tenant of credentials and the importance of logging and remember that your detections are only as good as your logging um we hope this has kind of helped further the understanding of some of the detection patterns we've talked about today and some of the complexities involved in implementing this logic um as well as Within reason how we can

overcome some of these challenges for those of you on the red side um common attacker techniques that you might think about may not always be as effective as you think like VPN to a local network range for example but nonetheless we still see CR Angels being abused in the wild but we hope that the attacker needs to tread lightly that's the whole premise here the detections and controls we discussed can introduce uncertainty into their operations which gives us a great a chance of finding the Badness and finally if one such control fires your detection team should likely be interested if a control exists for a reason so even if the risk is mitigated how did you find yourselves in the

position where a control had to fire that should be the question your detection team should be trying to answer and finally finally finally um if you're interested in one such project to make credential abuse harder um and which gives us actually really good opportunities to detect this sort of stuff we suggest looking up um device bound session credentials and this is a control um underway it's being designed and implemented the at the moment by a bunch of folks looking you make this particularly hard um but we don't want to spoil it in this talk um so feel free to look it up afterwards and in the meantime thank you for your [Applause] time questions good

sorry somebody was standing up behind me so I thought it was him um thank you that was that's great I have two questions I'm kind of just curious about your thoughts on one was to do the kinds of things you're talking about for a single event is you know something a human can run down but when you've got a large Enterprise with hundreds of these th well I mean thousands of things happening that you might want to look into obviously you have to have an enormous amount of automation I'm curious about your thoughts about that and the other thing I'm curious about is most organizations have scores of different types of you know authentication systems um and

hardening them all and monitoring them all is difficult you might be tempted lots of us do to start to harmonize down to a single try but now everything is in one place so if that gets pop you're really screwed I'm kind of curious about your thoughts about that also good question do you want to take the automation one yeah sure uh so your first question is about how do we automate all these detections at scale basically um oops too short uh um yeah this is this really can be a different talk um but like to summarize what can be done to automate things at scale like for detection pipeline we have like three main stages we need to

normalize the log write detections and add enrichment if possible and finally automate the investigation as much as possible um normalizing the logs is very very important because that's the stage where we can make sure we have the data available we have really good valid data um and is structured in a way that we can use use it the detection part is also very important obviously um it's usually depends on the log volume you might need to leverage some sort of data processing pipeline that supports a big data kind of scale processing um and in the end of the day it needs to be flexible enough to support different detection patterns it just be like a

filtering of certain thing you're looking for it might need to support like batch look up streaming streaming kind of ingestion like comparing different Windows of of things like we can talk about the detection patterns later but like to to to enable some of these detections you usually need to ingest more than like a few data sources and the way they come in they get delivered and how you can process them like varies very uh quite a bit um and finally I guess we have um we have enrichment um this is where we can like autoc clo things at scale um because like you said lots of these things will happen like thousand thousands of users

will make all sorts of weird like what what we think is weird anomalies in a in a log and now we would end up like not being able to investigate all of them we would come up with criterias of like how can we determine this is false positive like really like confidently and cify all of these and put that into the detection as well as enrichment and then Auto close as much of this as possible so the only thing that gets surface to the queue is mostly like High Fidelity um uh detections anyways I will time or respond we can have one one or two more questions just answer the second part of that one really quickly actually um

sorry to just inct real quick um I think like the the trade-off between like consolidation and siloing of identity providers is like there's also another angle of like business requ because some stuff you know you can't have in certain regions or whatever it might be um I think first of all if you have like sharded you have to prioritize them like typically I would say like look at the things that are known in The Wider world first right if things can procure credentials to The Wider world that is you know well understood as an attack Vector like I'd probably look at those first um internally is also equally as important but like there's a higher bar

to meet so maybe get around to those eventually but not right away um on the kind of like siloing versus like distribution problem I think um yeah siloing makes it easier but then you have to look at really all it does is shift the complexity from many systems into the specific subcomponents of one and then you have more subcomponents in one than you did in the others um it's just a a different kind of complexity you often have to have more knowledge across more things in the sharded sense of the model and have a lot more in-depth knowledge about the the ladder which I don't have an answer for you basically it's a tradeoff at the end of

the day thanks though yep hi hi uh given limited resources where would uh what's the best way to start what's the what's the lowest hanging fruit here do you think give go I can give it go well for me like maybe Troy has something different in mind but like if you have unlimited amount of resources make sure you get access to as many data points as possible that's usually quite hard way harder than people would imagine I think yeah managing stakeholder relationship and a lot of a lot of the times the data source comes from like external Partners as well which makes it even worse just turn it all off you know just don't let anyone log into anything and no just

kidding um honestly I think in limited resources right it's like it's prioritize what you're detect I mean it's a similar theme unfortunately so it's a bit of a cop out of an answer but um prioritizing what you need right it's like threat intelligence is great here like if you know what you're trying to detect then just prioritize your resources towards those things if you can um that may not always be possible for various Nuance reasons but um typically speaking when you're looking at like you really have to make a decision on you're going to detect large amounts of low sophistication high volume or high sorry yeah low sop High sophistication low volume in in the

inverse if you're looking at like a lot of low sophistication attack vectors then you're probably going to look for a small number of log sources because most of your data is probably going to land in like your core IDP logs inversely if you're looking for like high sophistication then you're probably looking at more log sources but probably not looking as many events per log source so it's kind of like the point I mentioned before about um like log diversity and log specificity you have to probably have a trade-off between those two things but that's a good question thank you okay that was the last one thank you Kathy and Troy another round of applause thank you

folks and we'll start again in 10 minutes with Kenton uh do me talk zero don't time credential rotation thank you

[Music]

[Music] [Applause]

oh [Music]

[Music] [Music]

[Applause]

[Music]

[Music]

[Music] h

n [Music]

[Music]

[Music] [Music] [Music] a [Music]

[Music]

[Music]

[Music] [Music] a [Music]

[Music]

w [Music]

[Music] a [Music] [Music]

[Applause] [Music] he [Applause] [Music]

we go sound check this sound good all right very good so my name's Kenton I go by Kent I work for visat you probably saw us in the news a couple years ago we got hit real hard in Ukraine um there was a talk at Defcon last year from our siso but I'm I'm talking about something else unrelated today this is zero downtime credential rotation designs and lessons learn so I'm Kent I graduated Virginia Tech in 2021 so I'm fairly new to the industry go easy on me um I I had a paper about uh container resource uh fuzzing that I thought was pretty cool but I I'm a security automation engineer at biat and I like to say I get paid to

break things before somebody else does so one of my first big projects was on credential rotation and credential management for one of our our newer systems so let me just run you through a few scenarios that you might want to think about um you recently fired a disgruntled employee who had access to your secrets manager and you suspect they may have copied some things before they left maybe a developer laptop with a bunch of plain text credentials on it was infected with malware and you think the dis may have gotten owned your sock has detected one of your service accounts being used from an unusual IP in a country you don't operate in impossible travel you know Google was

just talking about that or maybe you find out that a third party with access to your infrastructure was breached you don't know what credentials they had access to because why would you you trusted them right so in all of these scenarios uh once you say oh crap right your next move should be hey let's rotate some credentials right so maybe we go ask our devops teams hey I need to change all the credentials for your backend system let's see what they say you might get a response like I can't change that without a system reboot so I'll schedule that for the next maintenance window downtime possible that's why this team is doing it in a

maintenance window maybe you'll hear uh I can try to change that but we might break everything is the business okay with that those magic words the business right that means downtime is likely this this app isn't going to handle that and then a third response you might get is why do we need to do that you're going to take my slas for this month leave me alone downtime is guaranteed this app is held together by twigs and chewing gum right so here's the unifying sentiment behind all of this cred changes are complex doing complex things introduces risk of downtime which means loss of money and the best way to reduce risk is to not do things I guess so we should

never rotate anything ever once we created it set no there's an obvious answer in here is we should be practicing right and why don't we just practice so here's here's my big idea uh credentials are going to leak it's a question of when not if didn't we just have the biggest password dump like in history recently it's only getting bigger right since credentials will leak you should be able to change them quickly and painlessly and since you need to be able to change them at any time you may as well practice frequently how frequently is up to you monthly weekly daily hourly whatever you want but the point is you should be able to do it and the act of practicing will

teach you a lot about your system and show you how to improve resiliency it will teach you things that you didn't even know could go wrong because you're destroying some fundamental assumptions anyone who's familiar with the chaos engineering principle this fumps firmly into that bucket right best thing you can do is hire a monkey to just poke things and see what falls apart okay so I'm going to run through some terminology now just about credential management again I'm not talking about humans here this is all like backend systems service credentials service users certificates right the stuff that holds your system together you probably have a secret manager and if you don't you should a secret manager

is a place to store Secrets here's four very common ones hash Corp vault is pretty good there's a company called thycotic that's now known as deline that has a product called squirrel you might have used that AWS has one built into AWS if you have big presence there even GitHub has one now for you know your CI pipelines right these are way they offer encrypted storage they offer you some logs to tell you who's been accessing your secrets what where and hopefully some controls to prevent them from getting out where they're not supposed to be granular access what are you keeping in your secrets manager there's two main types of credentials a symmetric credential is a credential

that must be known by two parties to validate typically that's a client talking to a server this is an example of a password an API key in some cases symmetric key material although disclaimer I'm not going to talk about that because symmetric key material is kind of its own world of you know management and complexity and how you transfer for that across the network that that's kind of a special case but you might be keeping that in your secrets manager with symmetric credentials it's just like a string right there's no typically no lifetime built in so a client if they have one doesn't know how long it lasts right as far as they're concerned it could last

forever it could change five minutes from now uh the other type of credential is what we call a derived credential which are typically shortlived shortlived in quotes I mean you could make something last a year if you want right um but the these are derived from a symmetric CR so this is like RFC 7519 Json web tokens oaf Bearer tokens x509 certificates right um clients are aware of the credential lifetime typically you don't store these in a secret manager but you're going to store a symmetric thing in your secret manager that is used to derive one of these right you could pass these around a secret manager probably a bad idea because they have times right

they expire the important thing about these is that a client that has one of these knows how long it lasts if it cares right you can crack open a JWT and check the expiration Time same with an all off token or read your certificate right it's it's in plain text uh that is not the right slide I skipped the end so we're going to go back up to where we're supposed to be uh here all right uh the last thing that you need if you're managing credentials is you need a secret Rotator of some kind uh this is whatever you use to rotate your secrets most likely it's a collection of scripts or API libraries

or something right that changes the credentials and puts the new one in the secret manager when it's done some secret managers have apis that let you do this like some of the Vault backends will let you call an API and rotate the credential for you and push the result into L app or something right um managed cloud services sometimes just do this for you transparently and it all just works like a AWS ACM certificates that you can put on a load balancer they rotate them for you and you never notice right um the design of a rotator varies depending on what kind of Secrets you have and isn't covered here I think there's a talk tomorrow where we might

talk about some stuff like this but uh the important thing is that you have one and it works right okay so now that you have all of this stuff here are three algorithms for zero downtime credential rotation if you take nothing else away these are very simple things that you can put into your apps to make this stuff work uh before we actually talk about the algorithms I want to talk about kind of the big picture how these work so you either have for credentials a push model or a pull model right so in a push model when you deploy your app your deploy pipeline whatever it is is going to read secrets from your secret manager and

hold them in memory right it's these secrets are going to get injected into your environment somehow they either go into environment variables configuration files or some you know uh ephemeral storage in kubernetes or something technically that's memory right um they go somewhere where your app can read them right and they're pretty much static right so they're injected into the environment and they don't change from there on after the app reads them once on Startup and if you want to change them you got to reboot the whole thing contrast this with a poll model where the app environment is provision with a mechanism for interacting with the secret manager directly so your your app either has a library that it can use

to talk to the secret manager on Startup or maybe it's like a proxy server in your kubernetes deployment that has a mechanism for reaching back to your Vault or something um the point here is that your your app rereads the secrets as necessary during operation instead of giving it the credentials to start you tell it where it can read the credentials and it knows how to read them whenever it needs to so again big picture in a push model if you want to change creds you got to redeploy the thing in a poll model your app can recover from cred changes automatically although there are caveats there's a lot of corner cases you can get

into okay so here's strategy number one most people are probably familiar with some form of this this is what we call a blue green deployment so over on the left is the secret manager where we have two credentials which I'm calling blue one and green one these are for all purposes identical they're separate sets of credentials but they're valid for the same thing they act the same way uh we're going to rotate them in different times but that's the only thing that's different on the right I have some replic because of an application it's not really important what these are but we have this blue one credential is used in three separate places for this app

that's running behind a load balancer right so when I decide that I want to rotate blue which is in use right I start by pushing out green and I can do this bit by bit right so since they're both valid I push out green to the first replica and make sure nothing breaks that means that green works right and then I haven't made some fatal assumption I'm going to break my whole app if it didn't work I've still got two replicas right I take my one replica offline figure out what I did did wrong and and nothing bad happens right so we've already kind of built in a roll back scenario uh I push out green again

I don't have to do this immediately I could wait but why wait right I'm going to push it out I push it out a third time blue one is now gone right nobody is using that because all three of my replicas are updated so now uh I can change one to Blue two right nobody's using it Pros it's easy to do it's all segregated you can do it in pieces right you have built-in roll back mechanism the one thing that's kind of a con here is that it's hard to Define when you're done if you don't know how many replicas you have or let's say you know one of my replicas was unreachable because the

network was down or I was having a problem somewhere I can't rotate blue if I only got two out of three unless I'm going to kill that replica and everything's okay right it puts you into a funny state where can I finish now can I not finish now right that credential is being abused like right now I need to get rid of it but I'm G to take down part of my my system right so it can also be slow right if if you can't do them all at once you might be there for a while so that's strategy number one strategy number two is uh fail and retry so I have a slightly different picture here on the left I

have a secret manager where I still have blue one I've got my three Apple replicas that are all using blue one and I have some backend service that also has blue one so this is like an ldap database or something right doesn't really matter um so in this scenario I change blue one to Blue two and I do it in both places right my back end is updated and my secret manager is updated those are synchronized now the thing that's at a sync right now is the apps so the apps one of them's going to try to use blue one and fail right I've gone to Blue two blue one is no longer valid so what's the app going to do okay well

it failed it's going to go back to the secret manager retrieve blueo now try to use blueto find out it works the pro here is that uh I didn't need to do anything to my apps right they all knew how to read the secrets they know how to interpret a failure case they know that the credential rolled over right they just have to go get it and try again uh they can all also do this asynchronously if they don't talk to this back end very often I don't need to force synchronize all three replicas they'll recover when they get around to it right and they have to recover in order to keep doing service um now astute viewer May note

that I said zero downtime and technically there's a failed network connection in here do we really count that as downtime well I mean the network could go down for unrelated reasons or it could be buffering or you could have a a packet drop or something right so I don't really consider this downtime it's a adverse Network condition almost in that I needed to refresh my credentials my my argument underlying this is that this should be a normal operation that you can recover from just the same as if your network blipped or you had a route switch right or you lost your connectivity for a moment you just come back up right so in that case it's not

really downtime um uh another caveat here is that I mean your application code has to be more complex because you have to tell it how to read from the secret manager it can't just read a file and be done and if you're using commercial off the shelf code talk about that later but support varies and it's not always pretty so anyway this is strategy two strategy 3 uh probably should have mentioned this so strategy 1 and two are really only for symmetric credentials that's mostly where you would use those this strategy 3 is hands down the only thing you should be ever doing for a derived credential where you know how long it lasts so same setup I've got my secret

manager with blue one I've got my app replicas with blue one my back end this time doesn't have blue one in it because we're using a derived credential so each one of my apps is going to derive blue one prime I know the naming convention sucks but you get the point across so I present blue one prime to the back end it's accepted and I can do this for as long as I want blue one prime has an expiration time in it because it's derived right so we're going to wait 75% of the lifetime that's probably about fair so for 12 hours that'd be 9 hours for an hour that'd be 45 minutes whatever so we set a timer and we wake

up right uh with 25% left at this point I've already switched over to Blue two right and i' I've done this asynchronously so my secret manager is updated uh the app when it's time to refresh is going to go to the secret manager first before it does anything else and go rep this cred so now it's going to pick up blue to I can now derive blue two prime while I still have blue one prime and blue one prime is still valid I've still got 25% of its life left so I can try well blue one still works right I haven't done anything to that I can now try blue two Prime and find out that also works great

I just reset my lifetime with a totally new credential now I can throw away blue one done right I've replaced it and I can throw away the derived credential there's really no reason to not do this right I mean there should never be a problem if you had some problem where your secret manager had a problem and you couldn't generate blueto Prime you've got whatever 25% of your lifetime is to figure it out and if you don't figure it out in that time you have other problems to worry about because something's really out of out of whack so all right there's three pretty basic algorithms at least we think so here's the the rough stuff uh we took at ViaSat

one of our next Generation ground systems and we had a requirement that we needed to do this frequently we wanted to rotate credentials very frequently what basically whenever we wanted maximum of 90 days you know whatever you want to set the the time limit on and implementing all of this stuff into a variety of apps with various support for it taught us a lot so here's the fun part where you can see all the ways that we shot ourselves in the foot all right problem number one I I have these structured as problem and then the lesson that we learned problem number one is what we called ubiquitous credentials how many of your service credentials are shared between totally

different things for example consider you have like a read Service account client one uses it to read from some database and because it was convenient at the time when we wrote it and it was a Friday and I didn't feel like making another service account we also stuck it on client 2 that uses it to query information from a totally separate API alternatively uh let's say I slapped an identity certificate on a box and I have two web servers running there that are doing totally different things it was just convenient to put them on the same box for reasons right they're using the same identity C if both clients or both apps or processes whatever use the same

credential they become inexorably linked right they are stuck together forever and you might not think about that when you did it but it's going to probably cause a problem later when you rotate the credential they rotate and if they have downtime because something goes terribly wrong that's going to happen together so your blast radius is increased and if one client gets compromised both systems are at risk right if I lose service account a from client a and it's also used by client B my attacker might figure that out and now they've got two for the price of one right this flies in the face of a lot of the um you know least privileged principle that we espouse a lot but it's

convenient to do it sometimes right I think we've all done stuff like this so lesson number one granularity is important uh service credentials should always be split along logical boundaries whatever that is for you so applications probably a good idea deployment environment you should have separate for Dev test pre-prod prod however many environments you have your backend system if they're different you should have separate credentials your rback level if somebody only needs read and all you have right now is read WR or crud make a new one right it's it's not worth sharing um anytime you need to uh make a new credential for a system you should ask what is the current blast radius of the credential I already have

and how would sharing it increase risk if it gets compromised right again this is very basic it's the principle of lease privilege but as developers sometimes we get lazy we cut Corners right and you'll find this out once you start rotating credentials and stuff breaks in totally random places that you had no idea about and you know you get a call from somebody why did my app break in pre-prod when I rotated a prod credential for a totally different system right here's a a picture that I like of a gate that has six locks on it uh but you only need one key to open it yeah I lifted this off Reddit this is like a ranch thing that people do when

they have lots of people with different keys with different access to stuff but they all need access to one place all you need is one key to open this but if somebody loses their key you only have to replace one lock you don't have to replace all five or six rather okay problem number two this one's a little bit more fun do you know what your client software does if it doesn't have valid credentials well I can tell you what ours did uh scenario one rry with no DeLay So the client got stuck in a CPU or network bound retry Loop bashing against the API it was trying to talk to with the bad credential it didn't even

check the case that the credential might be wrong wasn't even considered this overwhelmed the target API or actually we we overwhelmed the secret manager at one point and a self-inflicted Dos attack right we we dossed ourselves uh problem number two so we we fixed the Doss problem maybe but um so we implemented some back off you can end up with this thing called a Thundering Herd so if you have 30 replicas and all of them need to do a refresh cycle at once right um maybe they all wait for one second and then they try again and then they wait for two seconds and they try again and then three seconds right you're just creating a little wave

effect right where you just pound instead of a fully uh you know uh Max efficiency Doss attack you're just dossing very slowly over time right this is a well documented Thundering Herd problem so uh yeah Jitter is important when you do time calculations you always need to add a little Randomness to fudge the time so that you don't do this um here's another uh this one's kind of fun this one's a little bit more Insidious so this was a what we called leaky threads so we had some designs that um especially in like a go or or a language that uses like a user space threading model you just spin off a thread for your connection right and uh assume that

it will raise an error of some kind when you have a credential problem right if you don't actually terminate that thread once it raises the error you're just going to leak threads over time so whenever we would do a credential roll over we'd have like 20 threads running on connections right we would leave all 20 of those still running and then bring in another 20 with the new credential right and if we rolled it again now we would go to 60 or whatever that was right so over time these threads would pile up and they just filled up the network right just boxes got you know uh we were wondering where is all this traffic coming from I'm only running

three apps it's because one app had like 500 threads in it right that weren't doing anything but just spamming credential errors over and over again so if this sounds like you might have this problem and you've never tried it you might want to find out because uh this is bad right um so lesson number two is that wa's web application firewalls are always worth it you should never assume that your internal clients will behave themselves um if the service that you're talking to is critical it needs a laugh cuz if an internal client on a fiberlink can knock you over like if you can generate a gigabyte worth of traffic like or gigabit per second like that

that's insane you're created a liability for yourself because if I'm an attacker and I get inside your network I don't even need to poke anything right if I can just generate an internal dos attack you're dead uh might be showing my age but this is a a clip from SpongeBob that I like where uh if I were to die right now in the some sort of fire explosion due to the carelessness of a friend well that would just be okay don't don't say that right build a WAFF okay here's here's another fun one problem three lockout events we've talked a lot about like human credentials today and typically if you have a human credential and you just

Spam passwords against an API you're going to get locked out right the parameters May differ about like how long you get locked out or for what time right or you know you could do a refresh to change it but um if you have service accounts that look a lot like human users do you know if their password policy is different than your human users because for a while ours wasn't um if that if that password policy is the same all it takes is one client with a bad password to lock the account out and then nobody can use it and if it's shared in multiple plac among like replicas of an app you've just taken all

of your replicas offline and all you needed was one bad client you in fact you don't even need the password right you just need the username if your API supports lockout by just having a username then anybody can do that if they just have it the username becomes secret right so that that's kind of insane um a human can do this right if I can grab one of your usernames oh if you have a public API that's you know exposed to the public internet the service accounts can off to I don't even need to be inside your network if I can learn the name of one of your service accounts I can just Spam your API and

lock it out whatever I feel like it right ask us how we found that out um I mean this is an easy problem to fix right you just change your password policy for service accounts but the same general problem app applies to wafs and client IPS that are behind a natat so let's say that you have a bunch of apps running on a node and one of them is just spamming an API because it's broken or it's an old deployment or something that you forgot about in a test environment if that creates enough traffic to trigger a wff IP block to a service that you're using for legitimate clients that whole node is dead right

because the W just blocked the node IP uh this is particularly Insidious on kubernetes when you have apps that move around between nodes so if you have one that's just moving around from node to node and just spamming and locking out the whole node ask us how we found that one okay uh so the lesson to learn here is that One Bad Apple can spoil your bushel literally uh service account lockout policies can easily be replaced by monitoring and high complexity requirements um yeah if you treat your service accounts like you treat humans you're probably doing something wrong you probably shouldn't be using service accounts in general but if you're stuck with them you you're doing something

wrong the the not blocking problem is a lot harder to fix because I mean in a lot of places we don't have anything better than a client IP so kind of ask yourself yourself how can you figure out like what's doing it right um if you get an alert from your W that an IP internal to your network has gotten blocked how quickly can you figure out what it is on that node that's causing that problem because it's not as easy as just going to PS right and looking at a process listing right we've tried that if you're on kubernetes you know you got to find a pod you got to reverse your way through

aat to the Pod and then figure out the Pod IP and then start looking at pod logs it like it takes forever so get good at doing this if this sounds like it might be relevant to you all right problem number four we talked about secret managers do you know what happens if your secret manager has bad data in it either because your rotation pipeline broke down or a user decided to put in haha funny I'm quitting right into place of your service password or something right um or I mean here's a a worst case scenario let's say that your rotator is running it's updated a password it's waiting to write it into the secret manager and the network goes

down at the critical moment right where we have the new password but we can't give it to any body because we literally can't talk to anybody um I mean this is an Avenue for attack right A lot of the time we think about secret managers being compromised or like read out or dumped but I don't need to dump it if I can just like wipe it or fill it with garbage right eventually I'll crash your system either way um how reliable is your rotator program right um if you're waiting for reverse alert from like a customer or like a developer team to say hey something isn't working then you're going to have downtime right so uh the lesson to be learned here is

that for everything credential related whenever you're going to rotate anything you need to have layers of logs and alerts right so reverse alerts here are like the final layer of protection for credential related issue if you waited that long you've already failed right um You should be getting forward alerts or events from your rotator if it knows that it failed it should let you know maybe it should even xmatters you because that could be really bad um your application if your application is not logging hey I tried to use this credential and it didn't work help before it retries right that's a problem this should never happen silently um you also probably want some big picture

alerts to kind of monitor hey what is this credential doing in my system right we often think about this for people to see whether they're authenticating from and what they're doing but you should be doing the same thing for your backend system too um after you rotate a password I mean you're probably going to see failures right of some kind as the apps kind of figure out hey this thing rotated right I need to refresh the new one do you know how you're expecting to see right how long do you see them right I mean what's your stagger window of if it's more than 12 hours afterward everything should have recovered so anything I see beyond that is probably

weird right or or something that's not recovering uh how can you detect if a system is not recovering right can can you detect that without even looking at the app logs just from seeing hey this credential just keeps failing from this IP long after it should have can I raise the alarm to the devops team to go take a look or something right because my you know the security log indicates there's a problem here uh this is a picture of like modern tank armor right you know you have your titanium alloy and then a bunch of ceramic and then more titanium alloy that like you should think about your logs for credential related stuff like

this if you only have one layer you're the first bullet that hits you is like you're dead Okay uh problem five is anybody these days is probably using a lot of commercial off the- shelf software and a lot of these for various reasons don't include look hooks for what we' call touchless credential refresh right a lot of C software is designed around you deploy it and the cred never changes some of them especially more modern stuff I've seen kind of thinks especially for like certificates is like hey we put a watcher on the disc to see if your certificate has changed we'll reread it in we'll do that every five minutes and so just drop it off in the

disc force and we'll be fine right um something like Apache where like you have to do a soft reload to get it to reload the CT apache's fine it's a wonderful product but it didn't in they didn't think to include this right you have to reboot it um these kind of assumptions are built into a lot of cot stuff and it can make it challenging to use when you're really trying to be robust about credential automation because you have to start bolting on additional steps it's like okay after I rotate this credential and I push it into the secret manager I have to log onto this node and do this thing or I have to write an API to let me remotely

reboot all of my proxies right because they don't understand that the credential has been changed or something and that in general I think that's just a a relic of kind of assumptions we made long time ago that credentials don't change very often often right or there's no need to keep up with this kind of thing but um dealing with this and CS is instant technical debt right if you have to be writing this like bolt-on code to hey I've updated my password please reload it right like it's it's just silly right at best it's flimsy at worst it's technical debt um even some stuff that does uh let you change password they don't let you tune anything that

would be good to tune like how long do you want to back off for what back off strategy do you want to use what Jitter calculations do you want to use right to prevent these Thundering Herd problems s it's just not exposed so lesson five this is a plea that we made to all of our developers and a plea that I'm kind of making to all of you is if you write software for security purposes that interacts with credentials let this stuff be tunable right assume I'm going to want to change every credential at runtime unless there's some critical reason why I can't and I must reboot if I have the option to change it for like

networking or whatever just let me do that right make it easy give me a callback mechanism in your client library that I can use to refresh add a file watch functionality you can ReRe from the disc give me options to control back off Jitter and reai do not hide credential related logs in debug they belong in warning error or critical the number of times that we've had to turn on a debug log to see that a credential was failing is ridiculous because debug logs I can't run in production they produce like terabytes of data right these don't belong there and never swallow errors there's some applications that will like fail like hundreds of times before they even log an error or

let you know right like you must like let me know right if if this is me me trying to use your client application you've done something wrong right you're failing me okay um so I have some final thoughts here uh I'm a little bit ahead of schedule so we'll have time for some Q&A I think um credentials are only useful if you can change them rotation algorithms are pretty basic to implement at least I hope I made them seem basic but they're really hard to get right because there's a lot of corner cases and conditions that you need to worry about you suddenly you need to worry about what if the network is down how

long am I retrying right under what conditions do I retry right uh practice makes perfect and it requires a collaboration between developers security and operations hey if you look at the highlighted letters there's a buzzword in there Dev SEC Ops right doing this can turn a devops team into a devs Ops Team right you make them care about their credentials and if they don't care Break Stuff right break their test deployments and make them care um and and finally right once you gain confidence and you can start doing these rotations with zero downtime and everything just works don't stop right A lot of the time we look at requirements that say you know oh I need to rotate

these 90 days or once a year why if I can do it once a week why not do it once a week right or you know if if your developers only want to let you do it in a maintenance window fight that right I want to be able to do it during business hours so you don't have to call me if something goes wrong right I'm already at my desk all right time for questions [Applause]

so so hey thanks for your time and then um you know back I you know I used to work in a database monitoring uh role a lot a long time ago and one of the questions that kept coming uh up again and again is hey hibm guardium Flags this event as a credential failure event but I'm looking at my application my application doesn't think uh about my my application doesn't flag this event as a credential fail or a password uh you know error so how do you reconcile uh you know your common definition of what is a bad credential event cuz I know it's up to the um you know individual monitoring tools to then flag that

behavior as such but then how do you come up with a common standard of classifying a bad credential event so this is actually a really good question when because we built detections for this stuff right to try to rec uh to understand when a client is recovering and when it's not the best I can tell you is it took a lot of practice cuz the numbers change depending on how many replicas of an app happen to be deployed when we do the rotation right if we're doing like a weekly rotation and the developer team from last week doubled the amount of apps that they're running maybe that's 4X the number of credentials right that are that are

floating around like two per thread or something like that um I I don't I wish I had a magic answer that was like here's a heuristic you can use but literally we did it a bunch saw what the numbers looked like saw what was normal right and then figured out okay here's the Line in the Sand let's build a detection around that so we said anything that's significantly higher than this basically we have we have two types of alerts we have a threshold alert that if we see way too many failures in a very short time period that means that a client is broken and is in Doss mode and we need to go like

get that now and then we've got a separate failure that we check for what we call persistent mix configurations which is like something didn't update but it's not causing enough trouble to like disrupt the network so we have a different alert that looks back over like the last two days and was like did anything fail at least once an hour for like six hour buckets right like we we built a thing to kind of do queries like that and it it turns up like broken deployments and like pre-prod and stuff as different from like a failure event and if the Threshold alarm triggers and then triggers again uh like we know that as a cue to like go look into it I I

hope that answered your question yeah great talk man don't even know where to start really enjoyed it uh Curious ious to see if you actually had to deal with incentivization uh to get I guess the dep SE Ops reality to really happen in terms of did money if we cut the BS have to become part of the conversation for the devop side of the house to be properly incentivized to take the security requests in tow I guess so I'm I'm actually pretty pleased like the culture of ISAT has been I think pretty great about this but we did have to quote unquote grease some Palms in the form of like I wrote a lot of

this stuff myself right like we wrote a ticket had a hard time getting it prioritized especially for like inner Source stuff for like libraries that people work on sparingly it was like we just had to step in and write it and then just once we had it written the teams were pretty open to implementing it right and and helping us tune the algorithms and help us find out where we did stuff wrong and it kind of rolled from there but we've we've gotten a lot better about it but we did need a burst of developers to kind of get it going hi here uh great talk by the way this way yes uh sorry so my question was

maybe partially answered by your earlier comment around detections but how do we prevent an internal crowd strike event with the a bad change going out a bad credentials being done as part of the rotation how how do you see that happening practice practice makes perfect so like I I mentioned these Doss events and stuff right we we also had scenarios where we just desync the secret manager like I can think of one where we had a network condition where um we tried to rotate the password and it got lost coming back so we tried to rotate it again like we retried and then the first one came back so we wrote the first one in but the request had already

gone out to change it again so the second one just vanished Into The Ether right um so there's almost no way to stop that right unless you have a roll back mechanism like you know hash Corp Vault or something let you like roll back a version so once you figure out what's going on you you should have a button to go backwards if you've already updated your L app store you're stuck you just need to redeploy so you kind of fall back to basic Dev off spr principles of I need to be able to redeploy really fast I should be able to do reboots really fast if I'm not going to do a full redeploy your detection should

catch this stuff and the app teams you know if you're not warning them they should be coming to you right we shouldn't be waiting for a customer to tell us something is broken the app team in the middle the SE devops team if you will should also catch that and come let us know hey critical problem what did you do recently we actually built uh dashboards because to a certain extent a lot of the automation like our Rotator like my team runs that on behalf of other teams and we don't always watch it necessarily because it just works but we built dashboards for all the teams that just post the data here's all the credential stuff that we've done for you

recently and we kind of train them if they see a problem they suspect is credential related first thing they go to is that dashboard and if they see that something happened recently they call us immediately and say hey help right so how often how often do you uh verify that the app asking for something from the secrets manager knows the old secret words I want a new secret I Pro to you that I'm a legitimate app by giving you the old secret do you guys find that necessary no so in general the connection to the secret manager is a totally separate credential from the credential that is being consumed by the app right so we do monitor that like

aside from the secrets that are in uh like in our case we use a lot of Vault right so aside from the secrets that we have in Vault that are being consumed we also watch Vault for weird events of apps that are not authenticating to Vault properly or constantly failing to log in Vault or trying to read paths that don't exist right or are reading a path over and over again when they probably don't have a reason to we have stuff the flag all of that and you know again poke the devops team and say hey what gives like are you doing this on purpose do we have a broken deployment right you know thankfully almost all of

this has been constrained to like non-pr environments for us by the time stuff got to prod it's actually been pretty well behaved which been very good fingers crossed uh so hi yes um I'm a devops engineer uh regularly hanging out on the security side of town um so most of the things that I do with my team is kind of devops as a service MH uh so how would I win over the hearts and minds of the developers to be able to for lack of a better phrasing uh take care of their sh um and just do what they need to do with all of your uh that last thing where you you got that plea the call out for the

developers how do you win over those hearts and Minds so I'd say the the kind of the formula that's worked for us is um there's this idea of like the security dragon that like guards the gate and says I won't let you to prod unless you do X Y and Z you have to use that very occasionally when there's like a very blatant like crypto violation or like critical vulnerability but for all the kind of squishy stuff in between right you got to meet with the developers on their level and say how can I help do you need me to write the code for you do you need me to sketch the algorithm like uh can I help you

write the pipeline to help test it can I supervise the test right I'll stay on call with you right like when we try this for the first time right work with the constraints that they've got but also I think push them to be better it's it's almost more of like a quest for excellence than it is just like meeting the bar the the bar would be you know maintenance window I can change my creds and I'm done right the Excellence is let's build confidence let's do this at any time and understand how to respond right and that last bit that that's what really scares people is hey my credential changed out from under me right I don't know what to do who do I

call right how do I recover right and we've written like pages and pages of docs about either how to get a hold of us how to handle it yourself how to interpret the alerts that we're giving you right and over time things things have gotten better because the not the tribal knowledge but the the organizational kind of uh key right builds up about how to handle this stuff and that we should be expecting it right and then the Excellence kind of it's a Snowball Effect I feel in one of your slides you said Implement a file watch MH uh file watch doesn't really work well in a containerized environment uh especially like for some implementations like

spring um so have you you usually have to pull for like the file updates and see if the file has changed have you gone around it or what's your experience with this um no polling um in the containerized environment specifically I've seen a couple of approaches to this so like we we have a um uh like a sidecar model I I could do like probably different stuff on CS if you want to talk about that that but a very basic way to do this if you're not interested in like a full-blown framework like C manager is implements a side car that uses a um uh like a shared memory like Mount to pass files between

and then just have both sides kind of pull it every once in a while um again like how you configure those parameters they up to you but I I I understand where you're coming from about certain Frameworks make this harder right and that's where you have to kind of bolt on like custom stuff and it's just you just got to try to keep the technical debt down as low as you can while maybe you know if you have a refactor coming down the line right that makes it easier you know just make sure that that's at least on the list of requirements when you're thinking about you know your next redesign a refactor or what so uh this is not a question but a

follow up to your question um if you store uh certificate as a secret in communities uh uh the you can uh trigger on that event to reload the application um as Le is what we we did you should talk to each other yeah certificate Management in kubernetes is a whole ball of wax just yeah make sure you have your uh authenticator configured correctly for your domains um a last question for Kenton before we uh go for break no more questions one comment okay we'll do one comment if that's okay you can correct me if I'm wrong um everyone has a hard time getting the business involved in security if you can get your board on uh into the mode of security

then you can ask them to make sure everyone below the CEO has it on their performance review that they do their work securely and then they will come to the security department and ask for help that's my suggestion Microsoft just did that right like seite yeah seite pay is like now tied to security performance because they have too many critical cyber attacks right like everyone should be doing that quitting oh yeah I I have to be honest as well during your talk here you know several times it's a thing about time and you know short Liv credentials and stuff I just saw some research coming out I think it was from academic research from I'm not sure

maybe it was Germany but they had been looking into GPS and systems using GPS uh because GPS doesn't only send you your position it also sends time and they have found out that well in most systems using GPS and GPS time uh there is a built-in protection for you cannot turn time back but there's no protection against turning time forward so they had figured out that are quite a few uh airplanes and the systems on them they are using GPS both for position and time and they also rely on certificates I see so they said that if you can just say outside McCaron airport and do GPS spoofing you don't have to care about location just spoof time

and put time like one year into the future there won't be a single plane leaving that airport for quite some time because you have to update firmware by connecting uh all the systems physically so uh have a nice flight back home and thank you

C so it's going to be uh 1 hour and 15 minutes break and then [Applause] [Music] [Applause] [Music]

he [Music] [Music]

[Music] track [Music] TR

[Music] hey hey hey hey [Applause] [Music]

hey hey hey hey hey [Applause] [Music]

[Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Music] [Music]

[Music]

[Music] [Applause] [Music] he

[Music] w a

[Music] h w [Music] a [Music] [Applause] [Music] [Applause] [Music] [Applause] [Music] a I'm just tring give you something okay I I'm just TR to give you [Music] something I'm just TR to something I I'm just TR to give you something [Music] he [Music] w

[Music]

[Music] [Music] I'm just I'm just dring [Music] something I'm just dring [Music] something I'm just trying to give you something [Music] oh [Music] w

[Music]

[Music]

[Music] [Music]

[Music]

[Music]

[Music] [Applause]

oh [Music]

[Music] [Music]

[Applause]

[Music]

[Music]

[Music]

the [Music] oh [Music]

[Music]

[Music] n [Music]

[Music] oh [Music]

[Music]

[Music]

[Music]

[Music] [Music] a [Music] [Applause] [Music]

[Music]

m [Music]

[Music]

[Applause] [Music] hey [Applause] [Music] [Applause] [Music] he [Music] he

a [Music]

[Music]

[Music]

[Music] track [Music] hey hey hey [Applause] [Music]

hey hey hey [Applause] [Music] he [Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Music] a

[Music] [Applause] [Music] oh [Music]

[Music]

oh

[Music] h

[Music]

[Music] [Applause] [Music] [Applause] w a [Music] [Applause] [Music] I'm just I'm just try to give you [Music] something I'm just try to give you something I do I'm just tring to give you something he [Music] w

[Music] [Music] I'm just I'm just [Music] something I'm just TR something I do I'm just tring to give you something [Music] o [Music]

a

[Music]

[Music] a [Music]

[Music]

[Music]

[Music] be [Applause]

oh [Music]

[Music] [Music] oh

[Applause]

[Music]

oh

[Music]

[Music]

[Music] he [Music] oh [Music]

[Music]

[Music] [Music]

[Music] n [Music] [Applause] [Music]

[Music]

[Music] I [Music]

[Music]

[Music] [Music] [Music] [Applause] [Music] a [Music]

[Music]

[Music] oh [Music]

[Applause] [Music] hey hey hey hey 2 [Music] [Applause] [Music] he he

[Music]

[Music]

[Music]

[Music] track [Music] hey hey hey hey [Applause] [Music] hey hey hey hey hey [Applause] [Music]

[Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Music] [Music]

[Music] he [Music]

[Music] [Applause] [Music]

[Music]

[Music]

oh

[Music]

h h [Music]

[Music] oh [Music] [Applause] [Music] [Applause] [Music] [Applause] [Music] I'm just TR to you I'm just trying to give you something [Music] I'm just TR to give something I I'm just TR to give you something [Music] n [Music] [Applause]

[Music]

[Music] [Music] I'm just trying to me to I'm just just tring to give you [Music] something I'm just TR to give you something I do BR I'm just trying to give you something [Music] m [Music]

[Music]

[Music] [Music]

[Music] he

[Music]

[Music] [Applause]

oh [Music]

[Music] [Music]

[Applause]

he

[Music]

the [Music] that [Music] n [Music] a

[Music]

[Music] [Music] [Music]

[Music]

[Music]

[Music] [Music] [Music] n [Music]

[Music]

[Music] St

[Music] [Music]

he [Music]

[Applause] [Music] he hey hey [Applause] [Music] [Applause] [Music] he [Music]

he

[Music]

[Music]

[Music]

[Music] St [Music] I [Music] hey hey hey [Applause] [Music]

hey hey hey hey hey [Applause] [Music] [Music]

[Music]

[Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music] he [Music] [Music]

[Music] [Applause] [Music] he [Music]

[Music]

oh

[Music] h

[Music]

[Music] w [Music] [Applause] [Music] [Applause] [Music] all

[Music] I'm just to I'm just TR to give you [Music] something I'm just TR to give you something I do I'm just to give you something [Music]

oh [Music] [Applause]

[Music]

[Music] [Music] I'm just trying to I'm just TR to give you [Music] something I'm just trying to give you something do I'm just trying to give you something [Music] get

Le

ass

now give him a round of yeah

yeah can everybody hear me y all right so if anybody has any questions ask ask me now don't save them to the end of the talk I I I it just drives me crazy because people forget questions and stuff so somebody said to have an about me page I have an about me page I'm not going to read it you guys can read it um I've done a lot of stuff mostly software some cyber security everybody got that so tools there's two big tools there's John the Ripper and there's hashcat they both have really good features they both are super awesome I use them both a lot in my experience I did much better with hashcat and GPU

accelerated Pastor cracking with John Ripper I having a little bit more trouble making that work um me switch glasses so with with John the Ripper there there seems to be a lot more updates there generally of a less formal nature um but they're constantly under development I got very good support from the community that when I ever asked a question um the custom rules is going to be important I'm going to get to that later and so it does it doesn't really paralyze so well unless you do the magical things the command line to make it work when you have only a few passwords left with hashcat it it's really an awesome tool it is highly optimized to work with

gpus it tends to work very well um the the rule syntax is not as sophisticated as the John the Ripper rule syntax and I'll get to that in a minute and and there are reasons there are philosophical design decisions and neither one is good or bad but I like tur rules so if you go somewhere and you see like here's a bunch of rules you have to find out if the rules are for John the rip or for hashcat because one won't work for the other one so word lists everybody's got them everybody says that they're the best most of them are junk most of them are really really bad the one that I really like the most is

Rocky but I've got probably 50 different rule list I'm sorry word lists um I needed to make tools to parse these things some of them had non- asky characters some of them had incredibly long lines some of them had completely random junk and so you've got a file that's a couple gigabytes in size and you've got to somehow make sense of it which is not very easy to do so I had to write some tools of my own to do that um but if you want if you want a really good one the rocky 2021 is a super awesome list extremely high quality there might be better ones out there but that that's the one that I that was my go-to list so

custom word lists so again since some of these are really long and a p so a password in GES is limitate characters so if you got a line that's thousand characters long it's not a password and it's probably junk um I wrote most of the stuff in Python short we take a big file and just get get truncate the first few characters um there's sort which before gnu sort could do offline sorting you were limited in how big things you sort and when you when you want to combine different word lists you really have to sort them too merge them together and make sense of things sometimes they don't sort them sometimes they have the

most common words at the beginning which makes really good sense if you're in a rush but not when you want to manipulate multiple word lists um so I have multi merge which will merge any number of files um simple sample just grabs a line every so often just to make give you an idea what's in a file so when you've got a billion lines you're not going to read them all because you just can't um I I wrote other programs I have line length and I have count I also have a thing which does substring so if you have a thousand character line you can grab like the first eight characters then you can shift over and grab the

next eight characters and you can take you know Sub sub samples of everything and that didn't work out to0 for me but it's a tool that I wrote and I use that well so some of the standard tools gnu sort super awesome now that it can work with more than physical memory it will just make temporary files and it'll just keep on chugging along unique will take repeated things and squish them down to you only have have one of them Comm will give you things in one file that aren't in another file which is very handy and the one true editor emac so with emac you can edit multi- gigabyte files you can make sense of

them you can display things I know some people don't like emacs you're all wrong for in the context of password in the context of editing word list and password files I I have yet to find a text editor that does a better job so hashing speed so NT landman is the current Windows hashing algorithm it is very quick and and quick is good because it doesn't use a lot of CPU resources and so you can do lots of passwords attempts very quickly on the other hand from a security standpoint fast is super terrible so md5 is actually slower than N Land man there's landman which is the predecessor of n landman I think that was used with

Windows xpn earlier I could be wrong DS cryp is the one that I use so these are these are all relative speed so you can see it's like a lot slower so for modern algorithms you've got sha one uh you've got script uh the wap2 and you have bcrypt bcrypt is probably what's used by most Linux implementations now it is very very secure it is very very slow and it has I think 128 bits of Salt by default which makes it very very secure so salt in 1979 the Unix guys put 12 bits of salt in their password does everybody know what salt is anybody not know what salt is all right very good so let's say you

have your password and your password is password one 123 and let's say he's got the same password his password is password 1 two 3 so if you just what's that I'm I'm sorry I'm sorry so if you're just using some encryption function his password and my password encrypt it would be the same so that's bad because if you break my password you have his One automatically so what salt does they they sprinkle this into the password it it's a little bit more complicated that they they sprinkle it in that's why they call it salt and in the case of Linux in 1979 they had 12 bits of random entropy so the chance of his password and my password having the

same encrypted form is 1 and two to the 12th which you know that's 4,000 so that makes things a lot better so you can have 4,000 people with the same password and you might get a few collisions in there but it makes things much more secure so again this happened 1979 in 1980 Unix added a lot more salt to 48 bits BCPS got 128 bits argon's another cool thing so D script uses that so the bottom line online NT landman today from Microsoft does not use salt so this is not a new idea don't ask me why they don't use salt so the thing is if if my password's password 123 and his password 123 on on a NT landman thing we're going

to have the same encrypted thing so you say who cares well people pre-compute lots and lots of passwords and you can either have pre-computed things you can have something more complicated called rainbow tables and when you don't have salt it's economically viable to pre-compute all this stuff so let's say you have a dictionary of a billion passwords and each password is eight characters long so that's 8 gigabytes so that's not a lot so if you want to have the encrypted form let's say it's 8 gigabytes of data but see now if you have linux's I'm sorry unix's 12 bits of salt that's 4,000 times bigger so instead of 8 gigabytes you have a lot more somebody help me out

many terabytes of data so eight 8 gigabits you're going to you're going to put on a flash drive it's real easy but when the modern algorithms with 128 bits of entropy that's a [ __ ] ton and there's no way you're going to be able to pre-compute that and store it's just completely non-feasible but with Windows and land man you can do that until people do do that and so that makes so if you look at the encryption speed in land man is really fast and there's no salting so maybe this made sense in 1960 but it's not 1960 anymore somebody should probably tell Microsoft that I I don't I don't know why they're not doing

it again this this is 1979 that's what 44 years ago 45 years ago so this this is not a best practice this is not a standard practice this has been a standard practice for several decades don't don't ask me why they don't do it so my DS dump I had 1576 passwords so the interesting about DS Crypt is you're limited to eight character passwords that's a stupid thing but in the context of the 70s and ' 80s it made a fair amount of sense um so I decided I wanted to break every Pastor it took me five years which is a lot longer than it would take me to do it again today um so using John the Ripper and

using my and using no options at all this is this is how fast you can get passwords so realistically I got 1,000 passwords in 2200 22,000 seconds so that's like 2third of them so if you want to break into a computer and you've got two-thirds of the password you're probably going to get some administrative privilege access by then probably long before then so only somebody with no life is going to break all the passwords because it's a stupid thing to do but I decided it was time to do that it's a hobby it's a hobby it's a hobby it it's a hobby but but but the fact is is that John the Ripper is is very very good and very

fast and it has very very good default dictionaries and this is fast I mean look at this 537 passwords in 126 seconds that's 2 minutes so that that's the third the passwords so it's incredibly good at finding passwords so I was using this version and these are several different dictionaries that I use there's the small rocku dictionary it's 139 megabytes there's the weak past dictionary the rocku 2021 which is pretty big 98 gigabytes and you can get quite a lot of password so with with rocku I got almost 1 th000 passwords that that's without any rules or anything I just let it go um so there's this option called print I'm not going to go into what it is it's

probability infinite chain elements it's a very good tool to use to break passwords it's beyond the scope of a password 101 talk but if you're actually breaking passwords look into it it's supported by both hash Cad and John the Ripper very very good thing there there's other options but this is probably your first go-to option when you're breaking passwords so my Hardware I have a 64 core AMD epic processor which I bought used I started this with a Nvidia 1060 I went to a 1070 then I went to a 3060 TI and what I wanted was something that would break a lot of P hashes per dollar and all those made cents per dollar at the time nobody's going to go

buy a 3060 today you're going to buy a 40 70 or whatever you're going to be buying um but these These are the tools that I used and I I also wanted to maximize the performance per price I also want to maximize the performance per watt so the the really fancy graphics cards the 4090 I don't know how many watts he uses 500 watts or something you know I can cool that but it's going to heat up the room and it's going to make a lot of noise and I didn't want to do that so I picked something that would be a little bit slower but not not melt things so I went over the word list that the

quality was all over the place um so there's basically two ways of breaking password either you have word list or use boot Force attacks word lists are thousands of times more efficient if if your password is in a word list or you can come up with some permutation using some magical rule it's like add a number or dollar sign or you know change a three to an E or something like that you use use word lists so here's how long it took takes to do boot Force stuff all eight character lowercase takes about 3 minutes on a GPU um 8 character upper and lower case 13 hours so that's a good amount of time now we add the lowers numbers and

special characters 5 and a half days now we have all the printable stuff 80 days so that that's for one password so you don't want to be doing this when you have you know 100 passwords that you haven't found but when things really get bad you've gone through all your dictionaries and all your rules and all your per ations it's time for The Brute Force passer attack no questions everybody still asleep all right so rainbow tables rainbow tables are a time space tradeoff and it doesn't really work with salts but it works really really well for landman and land man and md5 so with with the rainbow tables you can usually get 99 to 99.99% chance of getting a

password if it's in the so so a rainbow table it's going to say like all lowercase and number things up to nine characters long and and it's going to be a big file but when you have the rainbow table and you run the software it's going to be a matter of seconds to find your password so this is not going to work with any modern Unix which is going to have you know 120 bits of salt but this is is going to work great for all your windows systems it so so the guys at Defcon have the data duplication Village you give them a hard drive that's either six or8 terabytes and they load you up with

rainbow tables so go go buy yourself some hard drives load them up with rainbow tables and and start your hatching or your password hash cracking um it's an extraordinarily effective thing to do but again with salting you know with with the old eses CPT you'd have to have 4,000 times as big a file so instead of a terab hard drive you would need you know a 4,000 tab hard drive which probably doesn't exist anywhere yes are they called rainbow I don't know I does anybody know no but I believe that there was the early papers that talked about this technique they kind of explain I remember just like glancing through it and they explain why

they called it a rainb yeah there's an economic pay Philip Orin who created sort of invented this stuff uh so I can't remember the paper now but there is an academic paper explaining the concept so maybe you will find the answer in there but but it is it is a well accepted name and any I'm just I I don't know why it's called that I'm I'm sure some smart person had a good reason for it like sure but but it it's just it's ridiculously fast um I had an old landman machine I think it was a Windows XP machine and it was like 30 seconds and it found all the passwords it was just ridiculously fast I mean it just

you know years versus seconds it it's not even funny um so here's some password statistics this is with my data dump of the 1500 passwords so we have the lengths most of them you know 49% rate characters there were a bunch of shorter ones then we have the character strings we have all lowercase 60% it's a good amount and and we have a whole bunch of mix of stuff um we have different string classes I just thought this was a useful thing I don't see a lot of this being published with with the the password statistics so I I thought this was this was cool stuff um so so obviously if you're going to start breaking password you're going to

boot Force you're going to start with all lowercase because you got 60% of people doing that and it's all a game of Statistics so now we get to the weird passwords so when people were using this stuff they were using they had big computers they had like VX 8 650s and stuff and people would tell net into this is days before SSH so if you type the password it would have to go through telnet unmolested so I had a tab character so there's there's standard character classes of uppercase and lower case and numbers and specials and all that stuff none of them have tabs so you have to make a custom work custom character set

and have a tab in it so I had a control character here oh so there's my custom character set so control characters I had a control R and there's another password thing I'm going to talk about later on that had a control W so the control character characters were really quite vexing because I did a Brute Force attack and I didn't find the password I said well I've exhausted I've gone through everything how could it not be there the answer is some clever fellow put in a control R so if you want to make a password that's super hard to break put it into control character obviously you have to make sure it's going to be

accepted by whatever password tool you're using but I have never seen any standard rule lists or any standard character list that has control characters in them has anybody ever seen that you have a question no yes so how prevalent is something like this it's not very prevalent so again I had one controlr one Tab and another very small password thing I had a control W but I decided I'm going to get all the passwords and you know you you do an exhaustive search say I've done everything you say what's wrong has has my computer broken and and the answer is no you're you're not looking the right place and the right place says some clever fellow put

a control AR there so I'm going to I'm going to get to the the rules about this I think it's on the next slide all right so I asked the John the Ripper mailing list I said how can I do a substitution I'm sorry question yes oh just to add something there in Unicode 15 say that again in Unicode 15 there 150,000 characters that you can use as passwords including anything fair enough so DS Crypt Works basically with seven bit asky so this is very old school but in today's world you have to worry about people doing all kinds of things with all kinds of different dictionaries which makes the words World much more complicated and

difficult to deal with including Emoji cons including what including emojis Emoji cons emojis oh yeah and and people keep making new emojis so that that's the difference between GS scrypt and and modern stuff I actually saw a paper of somebody that was looking into an iot camera currently like last week and the camera was using GS Crypt for its encryption and this is something you buy today so this is not completely dead some embedded people are still using this so this is this is not only of historical interest anyway so with John the Ripper I asked how can I insert a control character in in the first position the second position all the way up to the eighth position and this is

the magic rule I can't explain what it does but it actually works and the second thing is how can I do a substitution of this so so the problem is if you do this with John the Ripper these are the two lines and and the first thing in the square brackets is the comment if you do this with hashcat you need 32 * 8 lines to get the same Rule now you can obviously generate those lines but it's going to be a lot more verbose than with John the Ripper but but hashcat can absolutely do this so this is what I use to find those control characters I used the insertion and the substitution along with the

rocky 2021 dictionary and you know in like 10 or 15 minutes it went through the whole dictionary all these permutations because there's 32 time8 permutations in there and that's how I found my password because these people that use these weird words one of the passord was song birds the S was capitalized and the r was a control R the other password I'll get to you later was chess whz with the w was the control R so they just substituted a prin of Basi character for its equivalent control character so if I would have had to a boot Force attack of all this stuff I'd probably still be doing this but these These are the rules I asked the John the

Ripper guys they gave me an answer it worked I found passwords so I was very very happy with that so now we get to the 1980 BSD un password dump somewhere I think is on archive.org somebody found a bunch of BSD files including the password and these were these are live passwords so GS scrypt was deemed to be so secure that they made the password file World readable they didn't have Shadow password so anybody could read the password file so the way I found this password file I just read it I didn't do anything illegal immoral it was World readable I had World access I read it so the BSD password dump I don't know how it got out there

on the internet but it got on the internet and people cracked most of the passwords so was Bill Joy's password was chess whz so the control W that that's the I'm sorry yeah control W that was his password and I'm sure he has better passwords by now Ken Thompson's password was this weird thing it's a chess moo with the P's and the exclamation marks so yeah yeah and and Steven Bourne's password was his last name I find that a lot I find I find that a lot I find that a lot so hashcat strips away all the information with the pastor fall and just has the hashes John the Ripper has some context like the person's name and the first

thing it tries is the guy stupid enough to use their you know first name or last name or something this this happens it's a real thing I'm I'm just telling you what I found so here's a list I'm sorry well so here's the statistics on the BSD passwords and I think there was something like I don't know what like 20 passs and something in that rough neighborhood but it's interesting statistics here's the actual passwords these are all of them um they were all found except the chess whiz I actually broke that myself using the John the Ripper rule with the I I tried some Brute Force attacks which didn't work other people have done that too and then

I tried the one throwing in the control characters and because chess whiz was in some Di AR it did a substitution of a control W boom popped out so I found all the BSD passwords from this particular password dump um so how do you defend against this I mean if you can't defend against it you're screwed so two- Factor authentication probably the best strongest way um so I have a Ubbi key I have a smart card you know if you have a cell phone there's fingerprints there's face ID there's a the cell phone guys want their phones to be secure whether it's Android or Apple stuff I know there's ways of getting around all that stuff but

they're they're reasonably secure it does act as second factor of authentication um you can use cryptographically strong password that's what I do so I generate passwords this this is the output of my password generator by default they're 20 characters long they have 129 bits of entropy obviously I don't memorize them I have a password manager but if if you have some tool that can break my 129 bits of cryptographically random passwords go for it um but but in in modern terms you need to have something that's cryptographically strong and random it's got to be long and you're probably not going to be able to memorize it may maybe you guys can memorize you know 10

or 20 of these things or a hundred of these I I sure can't that's why I have a good password manager there's a bunch of ones out there I personally use keypass I don't trust anything that's stored in the cloud if it's stored in the cloud it's not your computer don't do that so with gpus you can actually undervolt them and underclock them and you can save roughly a third of the power it's a very good thing to do there's a tool called MSI afterburner it works very well uh it was developed by some guy in Russia so it hasn't been update in a while because we got some political termo with Russian folks but but it's a

good tool they're there probably tools on Linux that do the same thing but you know if you're if you're saving a third of your power bill that that's a that's a significant amount the other thing is your GPU will run cooler which means it's going to last longer so it's a bit fussy because you have to iterate eventually thing's going to crash and you have to crank the voltage up a a little bit more but with within half an hour or less time than that you can get it running at a stable point or work and use a lot less power and this is a reference from the uh John the Ripper guys on some recent

stuff that they did it's just one reference there's a billion references out there for for password cracking and here's the dictionaries that I used and how big they are there's a lot of them um some of them are good so this this is not the size of the file I downloaded off the internet this is the size of the file I downloaded and then processed and I was using personally so the the file names I didn't change around except I had the do dick at the end um there there are more password files out there they're probably better ones there's certainly a lot worse ones out there and that's it

questions hello is that working yeah so I have a question sure um I've always thought that I could take my lousy password and make an md5 hash and then use the hash as my password that way I can always recover it that's an inter idea okay um obviously if somebody knows what you're doing that that's not going to work too well so the thing is there with common md5 hashes you can literally type cut and paste them into Google and I'll tell you the plain text of your md5 hash but if somebody doesn't know what you're doing that sounds like a cool thing to do it it sounds kind of brittle I mean I wouldn't use it for you know nuclear

missile Secrets or something but for your personal files sounds great more questions please yes so just to be clear uh you yourself use this password generation uh tool and then you store your passwords in uh I forget the name of the Tool uh yeah you use a password manager and that's that's your strategy for that that is my personal strategy because look at these passwords maybe you can memorize one of them maybe you're a good guy if you want to have a different password for every place you log in which is a standard practice you know you're going to have 100 or 200 different passwords I can't memorize 100 or 200 of these and so that that's why I

have cryptographically strong password and I decided I'm going to go with 20 characters so the thing is when you have something on on the internet let's say it's Facebook so you don't know what algorithm Facebook is using maybe they're using something really strong maybe they're idiots and storing stuff in plain text so if you use the same password in Facebook as you're using for Gmail you know and one of them is using some bad algorithm or in plain text and people do store stuff in plain text you know if you're Facebook account gets compromised I'm just using them as an example I don't think they're storing stuff in plain text then you're really screwed but if

every account has a unique password only that one account gets screwed if they have a data breach or bad things happen or they have an Insider that's you know doing bad stuff so you're you're limiting your exposure by having a unique password for every different thing you're logging into okay we had Jeremy Gosney here two years ago doing a a lightning talk uh about Pastor stuff he's the guy who have been helping me out organized uh Pastor con for many years uh and in his lightning talk two years ago he talks about you know well reasons for why you know password strength and your password doesn't really matter that much because there are a million and one problems in the

world regarding passwords and where you used them how you used them and so on and he said that well you don't know how different services are storing your password they might just as well be storing them in plain text and you don't know if their administrators will get