Joshua Jones - The Compromise of the Baskervilles - Wholistic Testing in the "Automagic" Era

all right good morning bides thank you guys all for coming out appreciate it 11:00 a.m. two talks done there is a whole lot of good material that I've been sitting through this morning and really really enjoyed it thank you guys for coming to my talk um looking forward to talking to everybody this morning hope to keep it really uh short and sweet and give you guys some meaningful material so uh let's get started so uh who am I right this is a lot of people do at talks and I find them to be very important so I'm going to go through that first uh on our agenda I also want to talk about what my goals are for this

talk with you uh sometimes speakers have this fervor for the material and they launch right into their talk and then uh I'm going okay I'm I'm not following the goal so I want to give you guys some goals um I want to talk about a historical context of why this talk was just came to my mind and why I wanted to talk about it I also want to talk about the ontological context behind this talk we in this room are living in an AI age we are humans we have ontology we have existence we have perception and thought and meaning so that is important to this talk as well um I'm obviously going to be talking about AI if you're probably

came to this talk that may be one of the things that Drew you there I really hope to not disappoint there um and then finally I'm going to talk about holistic testing um I know that they British and English pronun or spelling one with a W and one with not uh but I am going to talk about what that means and what that means for you by the end of this talk and finally I should have time for some questions so let's go ahead and get started um who am I and why do people have these who am I talks right at the beginning why do we have these introductions well I fundamentally believe that every human has a worldview

whether we like it or not we have a way that we perceive the world we have a way that we act we have a way that we talk we have a way that we are and as a result of that that frames what I do every day why I get up uh and the level of effort and what I do for my work what drives me to function right so I want to specifically talk about that you'll notice that throughout my presentation I am going to have a lot of Sherlock Holmes quotes for those of you familiar with Arthur Conan Doyle series um my talk is titled that I'll talk about that a little bit later and the goal of this

is really just to honor some of I believe the Timeless principles throughout Sherlock Holmes series and what Doyle's writing means and why it can still talk to us today okay so uh really quick just going to fly through these personal who am I right I'm a servant of Christ I'm a Christian ideologically that frames my worldview that is the central core of who I am and I'm unashamed of that I'm a husband and a father I have a wonderful wife of almost 16 years I have four amazing children that I love and get to spend time with and get to raise to be men and women um I love technology we're all here because that's what we love right this

is the field we're in I hope you love technology you're here um I am a homesteader I'm one of those weirdos that you've seen on Twitter X that are like nuts about the new tech that's coming out today and then I go home and I have chickens and turkeys and hopefully goats by the end of this year so professional life I didn't start out on it I've talked to a lot of people who didn't start out on it uh I I started out as Japanese language major um but that kind of jump ship ran into break fix cmin uh cut my teeth doing that for a few years went into Cloud infrastructure moved into building and

deployments uh and then I'm jumping into cyers SEC testing and now I'm in the GRC World um and I promise for anybody who's like oh no GRC Red Flags uh I'm out I I promise it won't be boring just because I'm in GRC right now uh I think GRC is legit and I think it's important all right so what is the Genesis of this talk why am I here and why did I want to talk about this talk well the fundamental basis behind why I'm here and what I want to talk about is the AI craze if you've gone to a bsides or a black hat or any other information security conference in the last year

there has been an AI talk probably for the vast majority of them there have been many AI talks it is taking over everybody's Minds we see big Tech your your apples your microsofts your Googles that is what their whole focus is on their entire Twitter X feed is just dumping what they're doing in AI it's obviously important Tech so I want to talk about that but I also want to talk about what security gaps would exist in that world and some of the things that we may or may not be thinking of as a result of that I then want to talk about Advanced detection and this Advan detection today is not going to come from a machine and

and I hope that uh my talk some of the Sherlock Holmes quotes that you guys will see down there at the bottom throughout this is going to help bring you all into the goal of what I'm hoping to talk about holistic testing which is going to involve us uh I'm going to talk about a business cultural analysis why it's important for your business your culture and who you are as a worker in this industry finally I need to talk about that we need an answer um I I see people developing all kinds of crazy great tools in and with AI and my greatest concern is that we're flying down the road it's almost a minute by minute hour by hour update on

the social media feed and we have I I haven't seen a lot of answers coming back to take the step back so the goal for this talk is to take the step back and to do that I need to go over three pillars of study okay um excuse me so those three pillars of study are going to frame the way that we look at Ai and ultimately a holistic testing approach so history um these are just some screenshots I randomly grabbed off the internet different points in history different times fictional non-fictional uh works and and times in history pieces of Art and what I want to communicate with this is that history is an absolutely invaluable tool every one of

you uses history every single day in your job we use history because we look at data data is points in time they are stamps of sometimes fact sometimes trajectories sometimes elements that we have to coagulate together to create correlate together to create functions and ways that we move forward but have we actually stepped outside of the data in our org outside of our org or even the data of the last 10 20 30 40 even 50 years when some of the first world computers were being developed and looked at historical backgrounds and why those are important um you know one of the great thinkers that I love to look at in ancient history Solomon the wise

he had a comment he said there is nothing new Under the Sun and what he didn't mean there was that it's not like oh Solomon you missed it you you didn't see AI coming you didn't see computers what he meant is the role and function of humans has always been to develop our society Andor destroy our society and that has not changed so as a result it's not a oh there's nothing new Under The Sun It's also boring what that means is we can look back and see what has come before us and that can inform the way that we act now so take a look at history think about it and how that is important pillar one pillar two

security right this is the world that we live in right castles walls this is what we do every single day every single one of your orgs is essentially a modern-day castle with walls trying to guard something precious Bank safes vaults this is what we do we protect important things AI now we're taking tools and we're using these tools to inform the way that we protect important things but what kind of things do we protect why do we protect them people can protect things of monetary value people we can protect all different kinds of things we can even uh desire to protect our reputation with someone if you go back to the like an Old Testament story of

Cain and Abel um the idea was Cain wasn't accepted by God Abel was and Cain was like I'm sick of this guy I'm getting him out of the way and so uh there's my one meme for this talk I just wanted to throw in one meme um uh all right so now the third pillar is Humanity this is who we are okay nothing I'm going to say here today is probably going to be new but I do want to talk about Concepts in a way that I hope we haven't thought about them and can inform the way that we do think about them in the future Humanity creates Humanity builds Humanity defends right Humanity also destroys

Humanity tears down Humanity attacks now probably one of the only questions that I'm going to throw out there for thought is which one of these lists sounds better than the other most of you would probably say the list on the left I would say it depends on context right I can create chaos in a city I can destroy barriers between cultures I can tear down political walls in an organization I can build uh I can build uh and and defend selfishly right I can do these things and so we can't look at the way that we perceive the world and just simply say objects only have one function right uh that's a way that we

maybe look at like a server for example if you're trying to F fulfill a PCI or a Cy toop 18 but that may not be the way that we look at the way that humans interact there we have tools we have uh agency and those are things that we should be thinking about finally you are what you eat and this is my first challenge that I'll put out for this talk what do you do on a daily basis to create destroy tear down attack bu build defend what are the ways that you use the agency the ontology that you have as a human to interact with the world and are you developing yourself in this

industry I'm very encouraged to see everybody here because when I see people here I see these are people that are interested in developing who they are they're coming out here and they're listening to a talk uh they're listening to all of these talks today because they want to get better at what they do so we are what we eat right are we going to we are going to become the kind of people people the things that we're interested in denot so that's going to frame part of this talk through too okay so enter AI this is not the fourth pillar and that's really important to to point out AI is not the fourth pillar the spoiler

alert I've already said it a couple times AI at its fundamental core level is a tool right it is a hammer it is something that can get stuff done it is incredibly powerful and I think through a majority of AI talks that I've listened to people have said some crazy sci-fi things that are some now coming now coming true people have said some things that haven't come true and people have said this is what's going to happen and just like a friend of mine told me last night yeah that's probably not going to come true right so we're in a world where we don't know what AI is capable of but at the at the world we

exist in today it is still very much a tool here's some examples of my most recent social media feed about different things that are being done with AI right Google deep mind uh looking into what Google is doing with AI imagery audio those kind of things excuse me excuse me Microsoft getting into GPT 40 looking at what that can do with your uh system looking at the way that that can interact with you as you play a video game like Minecraft right uh you can go to search AI pentesting right and you're going to find all different kinds of websites across the internet that are going to say this is our AI machine learning tool

here's how we're going to use it to pentest your organization there's a lot more that I want to say about that later on in the talk right and just another example um here's how we can use all this data to get inside your organization and learn more about it and then we can pentest against you and all you have to do is press a button and watch the magic right finally just an example just one example that hit my feed three or 4 days ago uh excuse me and um just looking at hey AI python code exploits they exist right so we look at AI in many ways as fundamentally a different entity from anything that

we've looked at but I still want to for the purposes of this talk and the high level that I'm sitting at right now 15,000 ft I want to look at it as it is still in the nature of a tool now why do I say that right well what is AI right now now I just grabb this off the internet there are arguments plenty out there there's the eight laws of AI or the the 10 types of AI or the seven types I I found the seven to be the most right now we're still all figuring this out but I think it's important to note that we're somewhere in this AGI world right now we're we've we've got this

artificial general intelligence that is capable of doing incredible things right but it still needs users input we're not really at the super intelligence level uh although I've seen some stuff just in the last days maybe even hours that is showing we're not too far away from getting there right and then you got these four five six and seven are different purposes of AI most of you probably are already fully aware of this right now and and know what this is but I just wanted to point out that right now we are still at the point in time with AI where it requires a user input and thoughtfulness of a user to ultimately be of value for the purposes

that I want it to be of value right it's not like the jeton where Rosie just floats up to me in the morning and goes hey what do you want a coffee you know like maybe um I was watching a demo of uh a robot just obeying a woman's commands and uh putting oranges into a basket and then going and put them on a table like we're not far away right but we are still needing it to give user input okay so Ai and we framed our pillars of History security and Humanity what's next right um I just like I had to have an obligatory meme I have to have an obligatory 1985 Goonies quote um so it

is our time down here right it is important for us to know that we are at this point right now we are Security Professionals we are in our fields and we have capabilities to do incredible things with this tool you can go anybody can go as you well know on open AI site you can hit up co-pilot on your desktop you can uh download LM studio and start running your own 53 or olama 3 uh instances sandboxed and what are we doing with this tool what how are we utilizing this tool what do we want to accomplish what do our orgs want to accomplish that is an important topic that we need to be thinking about right um and

excuse me I've been using offline llms for some months right now um I I don't give any like product or anything recommendations uh if you are new to the AI world and you want to do a sandboxed llm uh probably the easiest in is LM studio uh it is pretty easy to use and you can download just about anything that's modern out right now uh that is open source and and can be dumped in there people are updating instances all the time and it's very fun to play with and that's just a great place to start but it is changing every vertical so what are we doing with it right now okay so that's the the

5K uh foot view we're we're zooming in and sort of Landing the plane towards the ground of what we want to do with this tool well um I want to talk about this is not the end of my slides but this is the end as in Telos um the Greek word Telos means end it also means goal it is the purpose to which things are driven and so my question for all of us is what is our purpose as a human what is our end what is our goal none of that has changed we still all are here with organizations for a purpose we still all want to accomplish something we still all are hired to do a job what is our purpose

what is the end goal to which my organization is working what are the assets that they are protecting what is the Telos a system right now that can say pentest your organization can do a whole lot with uh authentication and maybe do some Kerber roasting and lateral movement privilege escalation but does it know what my company's tellos is probably not and why is that important okay so uh the the end of this tool is that it is a tool and and we should absolutely begin to use it um I can guarantee you this tool will be something that is in organizations if it's not already it's going to be in the next decade and that's probably the only

prediction I will make because I want to stay to very safe predictions I see this being used we're using it in my organization I've talked to people at this conference most people are using it right now in their orgs so what does that mean are we using it right finally or not finally but next um what what are the differences right this is in in a sense of tools it's no different than a lot of things that we see do you want to build or buy right are you looking off the shelf those slides that I those pictures that I just showed a minute ago they're repl out there there are people looking to say

take advantage of this use something like miter attack or any number of Frameworks and say I can build an AI algorithm a machine learning tool that can attack your organization give you a report Off to the Races right is that something you want in your org or or do you want to build in house and what is that going to take also what is your org again what's the tell us what's the goal of your org and why do you want that AI in that specific point why do you want to give it permissions in that specific area why is that important right so finally and this is probably the most science fiction that I will get in this

talk AI ultimately does have a worldview why does it have a worldview because we program it to have a worldview if anybody might want to disagree or have a a different thought about that I'd love to talk after the talk but all I have to do is throw in some different questions to GPT for example and I get a whole bunch of answers back to show that AI has been programmed to have a worldview it has a way of viewing the world this is fiction this is fact this is a story this is history those kinds of things right and and this is probably the most science fiction question that'll ask this entire talk does its worldview

align with your company Mission what is your company Mission only you know that right we haven't trained the algorithm maybe some of you are and that would be incredible are you training the algorithm to know your org inside and out the way that you do so that it can defend your org or attack your org in the way that you want it to those are important considerations okay so let's get use our pillar of history and get some historical examples probably the most common example that we ever hear of in the information security world is the Trojan Horse right this has been around since viruses have been around this has been around since attacks have been around I

probably heard about Trojans in the late 90s when I was it still in uh public school right um this is a thing that has been around now why was historically the troan horse so successful well it was seen as a very important item in the uh this Greco uh Grecian um Athenian War right we we we are the the the Greeks had this great idea we're going to pack some people in this thing we're going to cart ourselves off they're going to go and grab it and Pull It in and now we've just gained privileged access into their Castle right and so this worked and that's because the attack or sorry the Defenders thought that it was a trophy

they thought that this was something I'm going to grab and I'm going to take hold of and I'm going to call this mine and the attackers thought that's exactly what they're going to do so I'm going to I'm going to take advantage of that I'm going to exploit it what's another example well here's one that ultimately failed for those of you who are familiar with the French magino line um after World War I and with the rise of Nazi Germany in the 30s France said you know what we're going to spend hundreds of millions of dollars and we're going to build a wall and we're going to make sure that Germany can't cross that wall

now I'm I don't think I have a laser pointer here but if you look towards the top of the map there near where Belgium is in the Arden Forest they didn't build a wall why well they couldn't get over it in World War I their tanks didn't work they got slogged down so they're probably not coming that way and where exactly did Hitler move his troops right through the ardan forest right so time affects history and the way that we respond to attacks it didn't work out for the French because their View and scope of protecting their organization their infrastructure their Community was we know what works and we're going to do what works but are we accounting for

what's going to happen what could be innovated right this is another reason why I think it's really important for all of us to be well up on what's going on in the AI infrastructure and and AI developments next um the Enigma machine a German developed tool for World War II right uh almost uh a a a pretty at its time incredible cryptographic device and something that encoded many messages for the Germans but ultimately there was a weakness and uh I don't think I have that quote but uh from Sherlock Holmes but he has a really good quote about uh discovering and detecting that may be on a future slide so with all of these tools we have two that failed and one

that succeeded we know that historic things like a trojan horse still work because it looks like a legitimate asset it looks like something that works right walls certain walls if not properly defended and properly looked after will eventually not work cryptograms and cryptographic hashing Etc will not work over time we those things defenses must age attacks can innovate right and attacks can even still work over history so we need to be able to look at that now how does AI figure into all of this well AI could do what a lot of this stuff does I mean it could probably build a magino line that would work right somebody could probably say bu build me an infrastructure magino line

and uh make it to where the Germans couldn't get through it right or make it work for today right it could build a potentially cryptographically stronger uh algorithm than just about anything that's out there today there are numerous capabilities that this tool can do um right and same thing with like the Trojan Horse when I think of cryptography my favorite thing to think about is has anybody ever tried to work on the be papers with AI because if you haven't uh I'm definitely working on it and I think it'd be really cool because there's $43 million of gold in Virginia somewhere and uh those papers cryptographically tell where it is presumably um so something to think

about okay so let's look at examples of some ways that AI is at work in organizations today and some security concerns right a lot of the concerns that you'll probably see on Twitter X today are going to be privacy concerns right um now almost every article I'm going to show probably has an update within the last day or so because that's just how fast things are changing but these are some time stamps I took from about a week and a half ago right um examples of using user data right that is something everybody who uses Reddit we told Reddit we're cool with here's the Ula checkbox done Off to the Races slack same story building an AI

algorithm more privacy concerns potential security concerns for slack too um finally and one that really interests me is Microsoft's recent announcement of co-pilot AI for Windows hey I've got a power of Recall now I can go back and see every action that you've done historically on your Windows machine okay I'm I'm not scared I'm just somewhat concerned there's a don't trust but verify going on right now right we need to be thinking about how these tools are being leveraged on us in the world that we live in in the communication tools that we use and does that mean something for our organization so we have defenses right there are holes and patches historically this has always been the same way that

it is and it will still be today we are still going to defend our organizations AI may be a new tool tool used against us or a tool that we're using but it is still patching walls right it is still Shoring up defenses okay so walls need Defenders who know those weaknesses that's you all in this room okay walls need watchful Defenders right we need to be folks who never phone it in on our job every single day we get up this is something that we're going to just jump right into okay what is it doing what's the latest what how do I look at everything my org knows how do I look at the tellos of my org who am I how do I

factor into this right how can I Be watchful with AI how can I defend my organization right and then an understanding that AI is still changing the how of most organizations and will be over the next one year five years two days who knows um but it's not changing the why right we still every single day get up and go to defend our organizations because we care about our organization success not only for our jobs but many of you are putting lights on you're putting gas in people's tanks you're helping health care you're helping efficiencies with information technology and computers it's it's nothing changed we still care about defending our organizations because ultimately we care about jobs

and human flourishing I don't want to watch this area this culture fade well to do that I have to be a stalwart def vender right so AI is now a tool that I'm going to leverage against that but AI doesn't need to be seen as this is it everything changes I think a whole lot's going to change right but the why is still going to be the same we're still going to be people defending Towers okay finally um at the pavement it is not a silver bullet okay um it is it must be specifically vetted we have to look carefully at what these tools are doing I cannot stand when I talk to organizations and they're like we got

this new tool and I'm like okay did you do some due diligence before you got that tool well I mean it was awesome the sales Tech got in there and he showed us what it can do I'm like yeah does it call back to their org does it feed their or your data do you even know how it tests concerns there there are some concerns for me I think AI is going to be amazing let's look at it carefully and and um also I think we should have in our orgs whether it's small or big even if it's a one pager look carefully at what it does and then have a document that says this is who

our org is this is what our org does this is who I am and this is why I say AI only goes this far and I will use tools or develop tools that Accord with my company's Mission and tellos right that's important and we need to take those into consideration so why the name for this talk why did I choose the compromise of the Baskerville well uh I love charlock Holmes I love author Conan Doyle um and the The Hound of the Baskerville is one of my favorite talks that books that he does um so for those of you who don't know this is a 100 plus year old book so I'm sorry if I'm going

to spoil this for you I won't spoil all the details I won't tell Holmes's famous ending speech where he says everything that he does um but you have a story of a man who moves to the Moors he gets a great inheritance there's this uh supposedly supernaturally evil Hound there and he's killing people and this guy's really scared calls in Holmes and Watson Holmes doesn't go this time he sends Watson he says only do these things Watson Watson is now acting in a way like a Defender AI he needs to go learn about the organization and find out what's going on the Hound is the attacker AI right and and just bear with me here um what you find out later is

that there is someone behind the Hound the Hound has been bred developed starved and chemicals used on it to make it seem larger than life and Supernatural right Watson thinks he's only acting alone thinks Holmes is too busy for this he's writing letters back to him Holmes shows up on the scene at just the right time and says you were integral in solving this right because Holmes knew how to get to the end he knew how to beat the supernatural right because he knew how to use the people around around him and the tools in his belt to get to the answer that's why I love Sherlock Holmes because every single story is going to

be the same every single story is going to tell us here's how to detect here's how to inspect here's how to look at things and this is probably my favorite quote I out of the whole talk right it's of the highest importance in the art of detection to be able to recognize of the number of facts which are incidental and which are vital otherwise your energy attention must be dissipated rather than concentrated ated I promise you the best tools without the right inputs will still do that in today's world you all have to be there you have to know who you are know your tellos know your company's Mission and you have to be able to direct that tool to use and

interact with your company in a right way or I promise you'll have a magin o line you will so holistic testing the whole person for the whole organization for the whole protection and whole automation that is what holistic testing is it's you it's me it's us we're working together we're funding the funneling the inputs we're creating the organization breaking it down it all comes down to know your field Know Your Castle know your attackers know how to protect that's everything that I said today it's nothing new it's probably a reminder but really hope it hits home we are still critically important in the AI AG thank [Applause] you any

questions let people think a little bit

there's great power in all the tools that are now wanting to leverage Ai and get on that bandwagon what do you think for the security industry in particular is the best application for AI right now for us while we're still learning so much about what it can do for us yeah that is an excellent question so I will say that something is going to be critically important over the next I think but within 5 years is going to be information security defense information or sorry information testing defense and information testing attack I think those are going to be critical tools I think people are go to market really fast with them right now and I think it's just get

it out there as fast as possible I think there are sandbox versions LM Studio has some people are already developing them you can put them in a a sandboxed area and develop them and deploy them safely um I'm just deeply concerned about taking somebody's off the shelf right now so I think if I would tell somebody to start somewhere I would say uh start searching for those attack and defense tools and take a look at them also something I didn't mention in the talk and thank you so much for bringing up that question a lot of tools you're using right now are going to have ai dumped into them within the next year if they don't already your heuristic

defense probably has some kind of AI powered stuff and it would be incumbent upon all of us to say what kind of data is going back how is it funneling how is this tool now being leveraged for my organization and against my organization so those are already there so we should definitely be playing with those great thank you yeah great presentation thank you um yeah we have ai you know on the offense AFI on the defense I was just at RSA a couple weeks ago every vendor used the word AI this AI that it's all enhanced integrated it's it's over the top do you have a particular llm that you like you know Ama or mral open AI you like Gemini

any particular ones you you prefer and then the other thing what do you think about agents and the evolution of agents that are going to be coming yeah so um I'll answer your first question I almost sit in hear your second one um agents agents AI agents oh so yeah agents um so I I do tell people a lot like for for newcomers for for people who are brand new um LM studio is probably the easiest way to get started uh it's just a you know like uh steam or something for AI you can just go Marketplace download what you want and then get started based on what you think I I use Reddit a lot to talk to

different folks in the industry in security and uh look at what they're looking at and and see some of my favorite llms that I've been playing with over the past few weeks and months uh I did I've used AMA 3 and I've used 53 um just to see how they compare with each other I like to put them head-to-head I like to put them in contest with each other so uh I'm guessing that some if they're not already out by now some people are trying to get some open source GPT 40 copycats out there and to see what they can do um and as soon as those come out I want to try those as

well all right let's thank our speaker one more time [Applause]