LLM-Assisted Risk Management for Small Teams & Budgets

Well, we are at 1:30 sharp. We only have so much time, so let's get right into this. So, hello everybody. Uh, my name is Connor Turpin and this is the LLM assisted risk management for small teams and small budgets uh presentation. So, we're going to dig right into it. So, about me, I've worked IT and private sector, the defense industrial base, and in federal roles. I've been using Linux consistently since about 2012 or so. That was I think the Iuntu 1204 days. This was my first main dro. But my first experience with Linux was back in 2002 with an old Red Hat CD. I am a current member of the USF civilian service and outside of tech I

enjoy gardening, battletech, radio and wireless communication. So the fact we got a meshtastic monitor for those uh fancy badges was really fun. And this is actually my first public tech conference talk. Emphasis on the uh public. I've presented stuff like this before, but this is the first time in front of a bunch of random people in a conference. So maybe be gentle. Hopefully this works. Hopefully it's, you know, not terrible. We'll see how it goes. So this is an actual image of me explaining risk management framework policy implementation. Uh my co-workers at one point while I was working on a whiteboard of a particular architectural problem within the cyber security policy setup of the place I currently work,

they printed that out and then stuck it to the wall next to me and I didn't notice for another 20 minutes. So that's kind of the vibe of what we're going to be experiencing in this presentation. If you've ever seen It's Always Sunny, you already know what's happening. This is a legally required disclaimer, by the way. I am a current employee of the federal government. I am attending a presenting of my own choice today without specific sponsorship from USAF. So, my thoughts, opinions, and presentation are my own. They do not officially represent the views, actions, or plans of the US Air Force or the government at large. Yes, I had to put this in here, and trust me, it was way

easier uh than the alternative, I guess. Have you ever seen Oh, it's went dark. Yeah, they decided they didn't want me to keep talking about that, I suppose. There you go. They found me. But if you've ever seen uh if you've ever seen a few good men, right? Tom Cruz, the marine lawyer. Yeah, I narrowly avoided having to talk with those people to get this approved. So, we're here. So, the introduction, why might LLM help risk management? There's three main points I want to make here. One, to reduce drag on risk management documentation. What is going on with that thing? I am. It is actually just freaking out and not keeping a connection. The whole projector is

losing out there. Anyway, we can just go on without that while it uh comes back up. We are going to work on reducing drag on risk management document uh documentation generation and retrieval. We're going to lower the skill saving for information sharing and generative AI. And we're going to increase the velocity for undermann and underresourced teams. How many of you here feel like your team is under manned or underresourced to manage everything all the time? Got a couple sheepish hands. Okay, a good about half of you or so. Some of you really raising it high. I feel that pain. Um there was a point a couple of years ago where I was working for a federal contractor and the game

plan was to switch from the CMMC 1.0 uh compliance framework essentially for federal and defense contractors to the 2.0 model. And that was an endless sea of Excel uh road maps and they call them crosswalk documents. How you map all these different policies and controls onto each other through completely disparit documents that took a long time to get any headway. So if you're feeling undermand for something that convoluted or maybe PCIDSS or something else related to the finance world, this could be useful for you. Who is this talk for? To be a little more pointed, this is for beginner and intermediate CIS admins, cyber analysts, and technically minded risk management and enterprise risk professionals. Everyone else is also

welcome if that doesn't fit you, but things will get kind of technical in this talk. So, if that's if this isn't something that necessarily fits your fancy, I'm sure someone on your team might like it. Managers are also welcome because your technical folks are the ones driving these results. They need your sponsorship. They need your program and project management efforts in order to achieve ROI in a timely manner. This costs money. This costs time. If you already have very little of both, management working with you and not against you is pivotal. So, some background here. These are just some recent breaches. We've talked about a bunch over the course of the last two days at Bsides. And so, I

want to go into the details of a few of these. If you've not heard, Granite School District here in Utah had a complete loss of confidentiality for all students that had ever had information stored within their current uh information system. Just all of them, your student number, your PII, birthday, full name, all that jazz. Perhaps up to socials, I can't remember the um the dissertation that was put out whether I mentioned those or not, but very large breach. Change Healthcare and many others over the course of the last several years have compromised in total about half of the American populace's PII and PHI. So that's all of your health treatments as well as once again socials, name, rank,

serial number, address, all of that jazz. It has been breached multiple times. And in 2011, this one right here hits a little more home. This is the OPM 2011 hack. If you guys can uh reach that reach back that far to the ancient before times precoid pre- lots of things. They experienced one of the most severe data breaches of a government information system to date. It took 2 years nearly for the House Oversight Committee to put together a report and the issues ran really deep. And that screen blanking out on me is going to drive me nuts. Here's another uh example here. 74% of account takeover attacks start with fishing. This research comes from Checkpoint

Research as of Q2 2024. This stuff changes rapidly, so bear in mind the shifting threat landscape is just a fact of life, but it's pretty recent. I want to highlight the dichotomy here between Microsoft and everyone else. The most targeted companies for fishing scams. Argo they it's kind of ambiguous the way that they write it but this can be implied to mean the company's targeted both for fishing attacks as in individual employees and anecdotally that's also where a lot of fishing attacks center from uh if you're actually targeting a specific organization it's your Microsoft emails for your account your licensing or hey one drive's going down please click this to use the alternate site stuff like

that I actually managed to catch one of my CTO's years ago ago with a fake email like that. We were doing a spear fishing test and I managed to get him to click on the very fake health insurance update email. One of my proudest moments. But that's huge. That means for as important as Microsoft is for the average business process and process map that your organization's value streams run through, you've got to make sure this is safe. That's where a lot of risk management centers on is how you do things with Microsoft tooling. These are also really important. 10% of millions is still a lot. but it's not proportionally the same. So, here's a fun little picture. I've

got a few of these. So, this is a script kitty right here. And this is the API key. It's a little orange in the corner that you have committed to GitHub openly. This is a dumb little dog with a sword. They're not really going to be that harmful except you left the door wide open. This is one of the main points of risk management and why we need to make sure we understand our processes and our workloads and our potential failure points ahead of time. Risk management isn't avoiding risk at so much as mitigating it or accepting it based on severity and splash radius. So don't think that it won't happen because it'll happen to you. How many people

have actually experienced some sort of loss of confidentiality or integrity of your information? You've had an official breach of some kind. A few. Okay. How many of you have gotten hired onto an organization in the aftermath of one of those breaches? You're helping doing cleanup. Okay. A few more. A couple separate people. This happens all the time. Statistically, it will come. You can't guarantee when that attack is going to happen. Only that at some point it will. And I have a bunch of references in here, by the way, that you're not going to see in the PowerPoint because we have only 40 minutes left already. So, I will be posting this in the Discord as well

as the GitHub repo that I'm storing all of this on. You'll be able to see both of those. I have just under 40 different citations, a bunch of different federal and um private sector framework documents, which was a big selling point of the talk. A bunch of those that you can test on along with instructions. So, just bear in mind we're going to be moving pretty quick with these. Speaking of framework, guide and documents, you're going to see stuff mostly coming from NIST, SISA, Google, Microsoft, and AWS. These processes, these methodologies can be applied pretty much agnostically to any other documentary body that you're trying to build and pattern your own infrastructure designs off of. But these are some of the heavy

hitters. So, in case you've not heard of who these are, this is the Cyber Security and Infrastructure Security Agency. SISA are the people who do uh MITER for example. If you've ever heard of MITER attack, the framework for red teaming, that's SISA. And what is NIST? This is the National Institute of Standards and Technology. These are the people who invented DES in the 1970s. You guys are probably familiar with triple desk. How many of you heard of that before? Three. Okay, good. Most of you. So, three deaths, which was only very recently fully deprecated, uh, piggybacked off of something that was written half a century ago. And they also actually helped invent and formalize role-based access control.

They did the first framework in 1992. So if you're thinking about how active directory is structured even novel to a certain point if you have used novel way back about the turn of the century for uh systems administration and account management uh Azure AWS Google to a certain point role-based access control is based off of things that were written about 30 and some years ago by this point by NIST they've created special publications or SPS for the current hot topics in IT and cyber security you'll see SP800 the 800 series is a big group of documents. There's uh many of them. Just go to the site and search. You'll see the links in the citations. But zero

trust, for example, how many of you heard of zero trust before? Okay. How many of you feel like you know what it is beyond a buzzword? Let's be honest. Anyone? Okay, a little less. Zero trust is a really convoluted topic. I've read 800207 cover to cover and it can get really deep into the weeds when you start talking about implementation. Same with AI risk management. Did you know that NIST actually has put together a risk management framework specifically for the adoption, testing, and trusting of AI? Anyone? It's a brand new thing. It came out of the oven in 2024. And I actually had an opportunity to speak with one of the people who was on the

board who wrote it at one point. I got to pick their brain for a while. It's a very interesting tool set. I know that it's already being used in certain pieces of the government. I'll leave it there on where that might be, but it's actively being used. Check it out. If you don't know how to get yourself to trust AI within your workspace and you know it's happening anyway, that's an excellent tool set. Secure software development 800218. What it says on the tin, secure software development framework is a way that you can adjust your development patterns and workflows to avoid introduction of vulnerability in ways that are fairly easy. It's a big paradigm shift for some people. It's

hardly a move for others. It really depends on your organization's culture. Then supply chain risk management or scram down below. Um there have actually been talks today on supply chain risk management and we're going to go into those a little bit. This provides a good yard stick for the baseline of most major types of IT topics like we mentioned before including information system assessment, secure storage lifestyle. The point I want to make is that a lot of vendor docs, guides, patterns, and best practices draw on things that NIST or the National Laboratories have already made, especially PNNL or the Pacific Northwest National Lab. You guys have all used CDs at some point. I'm sure Pacific Northwest invented CDs like

just period. They they invented them. Kodak got a hold of them. And then through the 70s and 80s, you start seeing that adoption curve move up. But it came out of one of the national labs. So chances are the things that you consider gospel in your software development patterns, your cyber security frameworks, your business methods revolving around information and computer security, they came from these organizations if you're in the US. Don't underestimate the power of leaning on what has come before you. Steal their homework. That's what it's there for. This is all already fedally funded. We just all kind of pay for it along with everything else that we pay taxes for. It exists for free. Go use it. There is

no membership requirement, no nothing to use these are uh documents. So first example is going to be NIST 800161. This covers scram as a point of authority here. Executive order 14144. You can go ahead and look it up. Even the feds are moving on supply chain risk management and open source. So to put it lightly, you no longer have an excuse. Even the feds that are notorious for moving slow and being terrible at adopting to things quickly, that's a presidential executive order saying use open source and don't suck at it. So in case someone's saying, well, you know, we'll do it when the government does, guess what? That's what you can show your siso right there. Several key

examples in the last few years of why scram is getting really important. You have endless exper um examples of malicious packages or loss of package ownership through API hijacking and other methods within the Python package index or within npm. I'm not going to go deeper into those. I am not a red teamer. I am not a cyber guy by default. I actually do it infrastructure, but they exist. And then Jan, which was mentioned in an earlier talk today, that social engineering effort nearly hacked the entire internet. They pinpointed the XZ library that uh a lot of baseline Linux distros like DBN and Red Hat use for compression of data. Had they gotten that to get compromised and use that as

a side channel for other uh pieces of their attack tool chain, basically the entire internet would have eventually succumbed to it and it was caught mostly by luck. So it happens. You've got a framework there 8161 that can uh help you understand how to do supply chain risk management. Next one 853 rev four and five. This is the risk management framework. This is how the federal government does risk management. There are 20 control families 18 in rev 4 and 20 in rev 5. These are bundles of configurations and metrics and design requirements. And you have anywhere from 5 to 51 individual controls per family depending on your revision and enhancements which are more specific. Uh

think of those as the weird edge cases. But you have well over a thousand it's actually like 1,600 counting enhancements in Rev 4 individual actions in both revisions. This is the thing that you do that's tied to a given control. It's a lot. Why I'm doing this talk is because I have direct experience working within the RMF framework from tip to stern. I have written policy. I have implemented policy. It's hard. It's convoluted. But there's so much good information in there and so many things I don't have to worry about anymore in the infrastructure designs I use because I have templated it off of these designs. That's why I'm doing this talk because AI can make understanding this

way easier. Moving on. For example, you've got seven steps already. Each of these having their own bodies of documentation and process. There's a lot of them, especially authorized here. Anyone who has worked in the federal government before? Anyone raise your hand? Okay. Anyone worked in getting a new system authorized by chance? Okay, we got one more. So, atto is a pain, is it not? It's a bit of an adventure. Were you fed side or were you contractor side when you worked on that, might I ask? Contractor. Okay. Well, still either way, uh, contractors are a huge point of getting that authorization over the line because of the sheer body of work it takes. So, we have at least one other

person in the room familiar with how much work this is. So piggybacking onto that is the NIST AI RMF. You can learn how to do the following for any AI tool that you're going to use. You can govern, map out, measure risks, manage and prioritize your risks, and explain them in a clean and concise way. The link is here in the slide deck. Like I said, you'll see this on the GitHub repo. You can peruse all of this yourself, including whatever notes I've put on the slides themselves for the presenter. We're going to dig into the MITER AI maturity model next. And why I'm going through these so fast as a sidebar is we have about 30 minutes

now to walk through the actual usage of parsing these giant documents with AI. So I'm trying to get you a sense of the landscape here. There's a lot. That's the whole point of why I'm showing these. This is one is what I want you to really focus on. The MITER AI maturity model. This is a gap analysis tool for AI implementation. It's based on CMMI. If you've ever heard of that tool set before, that's for uh process and uh compliance management. You have once again six pillars and 20 total activities cut up into five tiers of progress. Ergo, you have granularity that you can use to explain exactly how well your AI adoption is going within

any context within your organization. It can help you be brutally honest. It doesn't help you lie about it. It can explain your AI adoption and maturity. And it will also help with suggesting supporting documentation. Chances are that none of you in this room have started from zero with your documentation at your current job. How many of you have at least something? You have at least some documentation technically in your workspace. Okay, about a quarter of you. Maybe it's a little rougher out there than I thought. Either way, this can give you some good suggestions on what those supporting documents can be that will help you explain AI. This is excellent for uh cyber security insurance for example. How many of you

have participated in board reviews for cyber security insurance within your or anyone? We got one back there. Woo! High five. Yeah, just the other day. Exactly. I remember it like it was yesterday because I was stuck in a suit coat for four hours explaining line by line why we did things the right way. So in case that kind of pain is familiar to you or someone else on your team, this model can help you correlate that with your AI work because it ain't going away. By raise of hand, how many of you think that AI is actually going to just fizzle out in the next 5 10 years? Be real. Anyone? God, I love this crowd already.

We're going to have a Q&A session here in just a little bit as well. And I'm going to have a couple breakout points while I demonstrate. Feel free to just yell out questions at that point. We can discuss what we need. I have endless backup slides. I could take twice as long and we still would have more stuff to go through. So just yell them out. Just an example. This is the actual uh pillar diagram from the AI maturity model. Most of your work is going to be around here from the initial and engaged sections. This is wow the AI seems kind of cool. Wow, chat GPT is awesome, but I'm just uploading government or

proprietary business information into chat GPT. The oh crap moment is usually between level two and three. Ergo, you are using it, but you've not yet defined how or why. Your rules of engagement for how you leverage LLMs or machine learning models within your environment's data infrastructure has not been defined. You aren't sure where or who or what is getting stuck inside Jet TPT's chats. That's a scary place to be. You have a whole set of guides and a whole bunch of different activities to cut up and explain your efforts to go from using it to defining a rule set. These are just optimizations. I don't want to underscore the importance of getting to the finish line, but most of

your big heavy experience is going to be right here. This stuff kind of take it or leave it depending on how important AI and LLM are to your organization because the value of death is cruel. This is coming from MIT Sloan Management Review and BCG Gamma and Henderson Institute. Despite increased investment in activity, only 10% of organizations are achieving significant financial benefits with AI. I'm going to pause for a second and let that sink in. You know how much AI is being used in the world already? You know how many countless billions of dollars are being poured into building up data centers and putting even more thinking robots onto silicon. 10% of the people that leverage all of

that investment and all of that subsidization by major corporations are actually net positive. I'm probably willing to bet that within this room we have maybe one or two people that would fall into that category at their work. Most of us probably are not. It's not necessarily an insult on anyone, but it is a fact of reality. Using this to avoid burning cash here is the next step or at least in your case, your organization's roll out of AI will fizzle out. It's just too much money to maintain. It's too hard to keep it secure. It's too hard to prove to insurance or to the government or regulatory bodies that we do it right and then it just falls by the wayside.

Goodbye fast emails and parsing through the menial crap that HR sends you. You go back to the stone age. So, this is an optional one. How many have you even heard of 800171? Anyone? A couple people. This is optional. Abandon all hope ye who enter here. 8171 underpins how every single nonfederal information system that does stuff for the government works and your requirements cyber security-wise. But if you are really interested on how PII and PHI are handled within your organization or you need tips 171 and 171A the assessment tool will help 800 just keep this on your radar. It is 18 years out of date. Do not use the current version but a current request for a new version and commentary

on the development of it went out in 2024. So assuming current context, without going too far into the weeds on that, assuming current context doesn't screw up those timelines and teams too much, you might see an updated manual aimed at infosc managers and corporate folk sometime this year or next. Just keep it on your radar. I'd love at some point in my previous part of my career to have grabbed a book, said, "This is the industry standard from the federal government. Do it." I would have loved to be able to do that. So a new version would be really nice. We're going to get into a new topic here. This is setting up a local LLM

service. By raise of hands, how many of y'all have ever done that before in any aspect at all? We got Okay, about about a sixth or fifth of people in the room maybe. Okay, so we're going to set up open web UI and Olama through Docker containers. How many of you used Docker before? Most of the room. Good. Congratulations. You're already halfway there. We're we're basically already halfway there. So Docker is ubiquitous and fairly easy to use. It might be free or have licensing costs and it depends on the org, but so long as you have an OCI compliant containerization engine like K3s, Kates, Docker engine, whatever, it'll work. Whatever you use within your organization or on your local dev box,

just use that. Why open Web UI? You have account management, admin controls, and a bunch of other cool toys in there without additional dependencies being brought in. This is all in one box. One Docker image, one Docker file. You can do it all within one spot. You're not spawning multiple other containers within the cluster to actually do your IAM, your account management, for example. And you can run this all locally in Docker or you can run it on whatever Kubernetes setup you have in the test rig inside your environment if you're lucky enough to have one of those. So, breakout. We're going to walk through LOM setup and instructions on GitHub. I'm going to flip my page over

real quick. We're going to exit that slideshow right here. So, if you've got if it would actually go full screen, that'd be nice. So, you'll see me as Rilo on GitHub. I think it's actually Elvish for squirrel, which is funny, and that's why I have a little uh it's not showing the whole screen. It's being kind of funky that way. But you'll see this is besides SLC 2025 resources. This will be linked and put in the Discord as well once we're done. Setting this up really is as simple as a couple of different Docker pull commands. First, we are going to Docker pull the actual Open Web UI image itself. And then we are going to run

that image. All we're doing at this point is mapping port 3000 on our external box to port 8080 on the internal. We are going to open up the backend uh data section there for the volume. And there we go. Basic container is set up. It will not have volume storage mounted. you will not have persistent storage, but you can add that on pretty trivially through another uh command and through reviewing the open web UI documentation. This test does not have persistent setup because it's meant to be a dev toy. It's not meant to run in prod. How many of you run Linux for your actual workbox? Anyone? We've got two and a one and a

half. Two and a half. Okay. How many of you use Windows for your actual workbox? The majority of the room. Okay, Mac OS for the rest of you. I'm assuming that's everyone else. Okay, there we go. Not everyone's raising their hand, so they're either not paying attention or some of you are running OpenBSD at work. And congratulations, you scare me. So, if you want to set up on Linux, uh this is an excerpt from the actual post on Olana right now. If you want to just use the CPU on Linux, it's really that simple. Otherwise, if you have an Nvidia GPU, you're going to pull it down and specifically instruct it to use all of your GPUs. This is also a

handy hack for laptop GPU setups where you have an integrated and a dedicated GPU within the same because I'm sure we've all worked with the uh GPU switcher stuff and Optimus within Linux before and it's kind of painful. So, just tell it to use all of it and then you can run a model. It literally is that simple. pull down the image, set it up on port 11434. That's the important part right there. And then you're running the model. If you want, you can also include this inside your Docker file to actually execute. I pulled down Llama 3.2 because it is the simplest and easiest model with a decent amount of parameters that you can run locally on

any device, including this M1 MacBook Air basic entry model. If I can run it on this little thing that cost me about half a grand at this point, I guarantee you any gaming or work laptop you are using can set up this uh local model pretty much seamlessly. I've tested it on worse. To note for Mac people though, Olama has to be installed directly if you're using Docker Desktop. Docker Desktop on Mac does not support GPU acceleration. I can't verify if K3s or Kate spins do that. I just haven't tested it. But with Mac being weird about how its GPU integration works with additional processes, it's possible that might not be a thing. Your mileage may

vary. Next, we're going to connect Open Web UI and Olama together. This will happen largely automatically. Open WebUI is smart to how Lama handles on your system. That's why we're keeping the default ports. So, you are now going to be able to connect to port 3000 on your machine and then you will be able to tell Open Web UI running on that port 11434. That's Olama. and it'll run. And you can go into the admin settings below and mess with that more. Going to control C over there. Boom. We already in here. I'm going to mess with the uh settings real quick actually and see if I can fix that screen because that is really starting

to be a little problematic there. Extended display invitation

advanced resolutions. That's the best one that it's going to have. Okay. All right. So, we're just going to focus on the middle of the screen. Then, I'm going to have to shrink this down a little

bit. Might turn down the gain on your microphone. Sure thing. I can do that. I'm hearing the thumps and whatnot. Let me see if I can just go. Oh yeah, you just do it here. Make sure this is where it's at. Cool. There we go. It's a lot better now. Thank you for that. [Music] [Music] Yeah,

there we go. [Music] Local host of 3000. Maybe it'll just turn it off. I know that's true, too. Connection is not secure. It does not come with an actual validert. Yeah, they are recording it. So, we'll just have to deal with the thumb piece, I suppose. I'll try and not smack my uh keyboard too hard. So, this is open web UI right here. I've got Llama 3.2 running, you can tell here. And the way I was able to set that up was by going up into and now it's decided to take the screen away again. Okay, that is an interesting uh problem that we keep having there. Not really sure how to fix that or why it keeps

doing that. Everyone else has been having no issues. So, we're gonna go ahead and move on from that point. We've only got about 20 minutes left and I'm going to show off a couple other things real quick and then we'll get into the meat of it. So, 3.2 latest. Go ahead and open up into settings. Here we got connections, manage direct connections to OpenAI compatible endpoints. We don't even need to worry about that. We can just go into admin settings here. And then there is an OAM setting download. Is really hard to see this on both sides. Okay. 3.2 latest. Able to edit that there. You can actually edit the model itself here as well. And this will

automatically. So, let me uh run off a quick slide real quick and then we are going to directly test and generate some stuff inside uh this little container that we've made. Play this from the current slide. So, document generation and retrieval. Why would you do this? So, you can discover things about your current risk management stature using plain words. You can take whatever documentation you've currently got, any guides for CIS admins or cyber, any procedure documents or work aids that already exist within your organization and feed those up into what we call a rag database. How many of you are familiar with the term rag in context of AI? A few. All right. So, we're going to differentiate between raw

file uploads or context injection and retrieval augmented generation. basically inferencing. That's what you're doing when you're chatting with an LLM. Directly adding data into the context window. Just copy and paste it in or drag and drop it directly into the window depending on what website you're using for it. It's easy. It just works automatically. And that means there's no additional training or fine-tuning needed. It can read that information and answer you on it immediately. It's also reduced latency. There's no additional data storage. You just upload it from your local box and it goes into the models processing world. This strongly depends on how you architect your LLM tools roll out or whatever tools you use

though. Next is retrieval augmented generation. Rag is scalable. You can pull in data from other sources and formats and feed that into your vector database. Essentially what happens is you grip you gri that's not a word. You give the model service a file. So, for example, open web UI, and I'll show you how this works on that uh site in just a moment. Upload it. It will cut it up uh sushi style into a bunch of small pieces called chunks, and then feed those into a database. And then what happens is you actually have a small part of the large language models text stack, read over that vector database, and check similarity against things. Think of it

as regex on steroids. That's essentially what's happening here. It's testing against the similarity of what you're asking for and what it thinks you want based on what it's already know inside the uh inside I can't talk today. Good heavens. You're watching a seg live. So it will automatically check those parts of the database. Yeah, I can't say database anymore. I'm going to start eating my own tub. But domain specific highly complex data landscapes are probably going to be better off with rag. If you have tons and tons of documents might be a better option. You can also move the compute load elsewhere. You're not fine-tuning and doing all that funky data science AI engineer wizardry stuff. You are taking

the baseline model that's decently smart and then feeding information in directly through rag. So we're going to demonstrate that right now on the site. Let's quit out of this right quick. We're going to go into this workspace section right here inside knowledge. So I've got a couple of different piles of documentation here. All you would do in open web UI is click this create a knowledge base. What are you working on and trying to receive? Wingus. Dingus. Create knowledge. Boom. You now have a vector database. It literally is that simple. So, we're going to take a look at one that I've already created that has some decent stuff in there. This is the NIST and MITER

documents. So, you can click here on the little plus sign to actually add that information in. Oh, it's trying to load Rev Five. Rage against the dying of the light machine. Don't try and load that one all by yourself. That is 12 megabytes of pure text. Don't do it. Look at this one. It's a little simpler this one. 800 to8. So, click here, add content, and then it'll pop up. It's popping up on my other screen. So, don't worry about that. You know the you know how to upload a file. This is actually what it's seeing and what's been uploaded. And that is now a part of your AI's brain. So, we're going to make a

new chat here to demonstrate this. You can see on the side I've tested this a bunch. I can demonstrate those later if we have time. Tell me about the fundamentals of implementing Azure or implementing an AI tool within Azure based on the context I've given you. That's a little bit of a buzzword for the large language model. It's saying based on all the stuff that you already have now referenced. Example, hashtag Microsoft Azure AI document body. Now there's a pile of documents that it can sift through and rag against. It will actually retrieve pieces of that information and then use that in generating its response. The best part of this being you're stealing directly from the horse's mouth

here. This is Microsoft's own documentation on the matter. You're not worrying about random websites or things like that. You're getting it from the point of truth, the authoritative source. Click it, generate it. Like I mentioned, this is Llama 3.2. So, this can run on anything that has about 4 GB of VRAM, preferably eight. It'll go a little faster. But for example, my uh MacBook Air has 8 gigs of the unified Apple style RAM, so it runs perfectly, generates quickly. So if you have no idea what any of that stuff is, based on the provided context, you're looking on information on implementing an AI tool within Azure. It mentions an application platform for AI workloads on Azure and a

link to the Microsoft Learn page. To implement it, you would start by selecting an appropriate service or solution such as an Azure machine learning, cognitive services, or bot service. These services provide pre-built features and tools. You can also use Azure's cloud-based infrastructure to host your own if you're familiar with Azure OpenAI and the fact that Microsoft and OpenAI are really tight right now. You are already familiar with the end state for this kind of design. We're going to take a look over here for example and take a look at an Azure AI rollout guide that I had to create with the context I've given you. Create a step-by-step guide. Write it in markdown format. Write it

for a mixed team. There will be people across the entire spectrum of the racy matrix if you're familiar with that term before. people that are responsible, accountable, consulted, or informed. So, people that are actually involved in doing the thing and people who just have to sign off on it or check emails. Identify courses of action for each part of the racy matrix. Don't avoid jargon and tech words. And finally, mix them, but don't call it out specifically. And it has now generated a whole basic markdown document body here, including nice little chapter headings. Simply instructing the large language model to generate its return in markdown will result in stuff like this. That's the fun part about stochcastic models.

You'll get a little bit different every time, but you'll be able to opinionate it based on the format that you want. So you now have an introduction. You have a design phase here. And if you're wondering about how true to life this is with the documentation I'm giving you, that's one of the key points of AI is that you are using it and building it for your own understanding. You are not going to get a carbon copy of everything that comes out of that document list. You're going to get conclusions. You're going to get summarizations. You're going to get individual bits and pieces from all of those different documents. That's what retrieval augmented generation does. How

many of you are familiar with the terms top K, top P? Um, anyone at all? Okay. So, I'm going to skip over that. The point is you can tweak the heat, the temperature, the willingness to get creative of the LLM should you so choose, should you want to learn how that works. Even at default settings, which is what I used here, it generated a very sane baseline document that you can then start filling out details for yourself. Just control F through it. Change a certain thing. Create a highle architectural design. Cool. Crl F architectural design. Remove that. create a highlevel software development plan. You're now getting a baseline of documents that you can use for business

cases, for setting up and onboarding new people. Say you hire on a new guy and he seems to be an expert, but man, you don't really have anything that explains what you do and you've kind of been going by the seat of your pants for six months or more. Use something like this. Give them a basic layout of what you're working on. You can explain responsible and accountable uh courses of action. Developing a costbenefit analysis, creating a comprehensive project schedule, defining and documenting all technical dependencies and integration points. That right there is a fun piece that actually is taken directly from the documentation on Azure that I just showed you. It'll summarize and build all of this stuff together. So, I know

what all these documents mean already, but I can tell it's shining through here because it is directly referencing and making inferences off of that whole setup of information. We only have a couple of minutes left. We have about 10 left here. So, I'm going to start generating a couple of more and showing you how some of this stuff works. We're going to open up to questions at this point, though, because I have a whole bunch of other options on what I could demonstrate. Questions so far? Anyone? Not even one. Okay. Pretty much. Yep. I am running uh bigger models actually on my Asus gaming laptop. It has 8 gigs of VRAM, 24 gigs of normal RAM. If I run something in CPU

mode, I can hit a 17 billion parameter model instead of three and it'll run pretty fast. So what I want to emphasize there that's a very good point you are not stuck with paying for open AAI or for claude or whatever other tools that you see those are very very powerful models and they have web integration and deep research all sorts of tools that are hard to emulate at the local level but if you just need to generate documents you can do that for free from the comfort of your couch and that is a much easier cell a much easier entry point to adopting AI within your organization than it is saying, "I need $200 a month

for ChatGpt 40 Pro per user." That's a little harder to have them sign off on. You can use the existing hardware you have within your teams right now to do this. We're going to go ahead and generate another one. Then this is a bit of a warning here for really, really big complex documents. I'm talking thousands of pages and hundreds of thousands of lines. You need to be spec uh specific when it comes to what you want to generate. You do not want to give it a giant pile of files and give it a oneliner question. Summarize this content I've given you. Give me each of the control family names in order. The order is alphabetical in the

document. Uh something tells me that CA a ps is not how the alphabet goes. There are limitations, especially with local language models. You've got to be the smart person in the room still that's willing to verify how these things work. You can't just blindly trust it. But when you play to its strengths, it can work really well. Like uh something here for putting together a new piece of software. I'm working on a project that involves building a new piece of software. The software needs to follow NIST SP80028's rule sets. I've included that file for your review. Walk me through how a team would road mapap that new project in view of this framework and it'll give you actually a

set of phases that are based on what is inside 218's rule sets including directly referencing them. This is all fairly highlevel stuff. What I want to emphasize here is if you feel out of depth with either what you're hearing here today, if this is already too far off the beaten path for you, or if this is something that you are not sure could get traction within your team, start small. I'm not here to necessarily be a motivational speaker. I kind of suck at being motivational in that way. I'm much more a get down and get to it kind of guy. But I want to emphasize that this is within your reach and it is both

technically and physically something that you can work on. If you don't know and you are just so confused with how you would start this next project, you don't even know where to begin to start. Get a requirements document from the project manager or from the sea level executive that's sponsoring this little plan they have. Get them to just summarize that. Pull down one of these documents like NIST 800 218 and then mix them together. start creating the next step and you will learn on the fly. You will learn by the seat of your pants as it happens. One of the greatest force multipliers and this is my closing uh commentary here. We got about five

minutes for some remaining uh topics. But if you take nothing else away from this talk, I would love to know out there in the world that there are more people who are using this kind of tool set to speed up their learning and to refine that core gameplay loop that we all have at work of do a thing, realize we don't really know how to do the thing, Google it, try it again, realize we still don't know. That iterative loop of true learning within a professional work environment doesn't have to be quite as painful. You can use this to help uh summarize, synthesize, and move forward those projects while you play catch-up. It is not a replacement for

human intellect. It is not a replacement for you being crazy enough to say yes to a project despite some really weird constraints. But it will help you generate a plan that other people can spend a day or two chewing on while you figure out what the heck PCIDSS even means. If you feel lost in something, this is both a diversion tactic as well as something that'll help round you out as a person in the workplace. So, at this point, I'm going to go over some basic topics of why those uh returns are so variable and so strange. This is an optional topic here, the data versus AI question. How many of you come from more of a data or finance

or sales background? Anyone? We got one. Okay. How many of you are software developers by trade? Okay. Good handful of you. How many cyber? Okay. How many in like uh management or enabler roles? Anyone? Okay. One, two, three, four, a couple. Okay. So, if you're not familiar, the axim is really simple. Garbage in, garbage out. If you don't have clean data, then you won't get clean returns from your language models. The language model does not know what a true and proper compliance framework is supposed to look like or what your assessor or insurance representative is going to want. If you haven't updated, say, your tech list for your laptops in a year, the model's not

going to suddenly fix that for you. But it can explain if the mo data is clean where there might be issues with your tech list. That machine's really old. That machine has actually been reassigned. You can use that especially if you have online enabled AI tools that connect to other systems. But garbage in, garbage out. There's a ton of different angles and attributes that you can work on and uh study out to figure this stuff out. That's a whole different conversation and usually should happen at the enterprise level with dedicated specialists, but it's there. So, here's some starting points as a final as a parting note before I do the uh conclusion and show off my contact

info. This is what you should ask yourself. Where is my data stored? Where should I start? And what effort to reward ratio makes the most sense to me? If you want to start working on large language model usage within your teams, if you want to start working on implementing tools like this to assist your processes, make things more automated, just ask yourself, where should I start? What's the best bang for my buck? The simplest, easiest thing. I hardly have to lift a finger and I'm already getting better results. Maybe that is hooking up your email stream to Copilot and Outlook. I personally wouldn't do that. I don't think that's necessarily the best option, but maybe

for you it is. figure that out what that is and then start working on it. You can even get suggestions from the lang uh language model itself on how you should start and they probably won't be horribly wrong. But you have to understand the implications of your data and your business before you go into that. Quick highlight here. This was Oh, that displays horribly on this screen. Yikes. Uh if you've never heard of Dockling before, essentially it will convert PDF docx files like that into markdown. And it's not really that lossy. It will also include images and just base 64 encode them. So if you have a crapload of PDFs full of information, then this will actually OCR scan through

all of them with a very reasonable rate of accuracy, turn it into a markdown file, and that's something way more easily parsed by a local language model in open web UI or whatever other service you use. Uh that's something that Bryce Coons yesterday brought up. I didn't know it existed. Very nice tool. Go ahead and check it out. So that being said, this talk is basically over. Congratulations, we made it. Hopefully your reaction is this victory and not this. Thank you. The GitHub repo with resources is right there if you want to get a picture. And then my contact info also is below if you have questions or want to tag me on LinkedIn for whatever

reason. Then go ahead. If you have any input or feedback from this, I will be here for a few minutes. I would really appreciate any inputs, good or bad. This is the first time I've ever done this in a public setting with a really varied skill base. Have a nice day. Thanks for coming.

Sure. I was wondering So it's local, right? So yes, you set it up in a container. Yes. And you could throw the container on the cloud. Yes. Fore!