VRM 201: Effectively Assessing Vendor AI Risk - Chris Honda

at and things hey everyone uh thanks for having me we're I I I'm I was telling everyone that was up here I don't really willing believe that you all willingly came here to listen to brm it is okay to put your heads down take a nap post lunch sleepy is kind of kicking right now so at any rate for those of you that really do want to hear stuff again um please don't get your expectations up too much the only gets so exciting but today we're going to be talking spefic specifically about assessing AI risk in a vendor setting so before we start uh again apologies you're going to hear some really bad takes of mine um you're

you're free to leave whenever it won't hurt my feelings none of this is meant to be advice right we're sharing thoughts ideas experiences that's it uh if you say I advise you to do something I will fight it to the nail so let's just avoid that I'm also not an AI expert um I think anyone that claims to be an AI expert is either really really really smart or lying and I'd like to be neither because I know I'm not smart and I don't like lying so non AI expert this is just kind of stuff that we found in the last year with AI being the buzzword and everyone kind of being in a position

where they have to learn about it so I'm also going to do a fun thing um I don't hear or see super well but I also think that most of the people in here are have something really great to contribute I've seen it in a lot of different conference talks throughout the last few years and I like to get those shared now so if I need to skip through my presentation I would much rather hear what you have to hear what you've seen what you've done I think that's going to be a lot better than the sweet memes that I have added in here so um I'm really not that interesting I've done a bunch of different stuff

this is kind of the first home in which I've been able to do something that I think I don't entirely suck at I wanted to be a band teacher I'm most definitely not a teacher I think that was a great financial decision at the very least um I am the reigning cahoot Champion my company doesn't mean much not a lot of people but I think it's fun so and I work at wistic which is a a we're we're a I was going to say we're a small company like every other company recently we've changed sizes recently um the the the chuckles have it yes um but we're we're a company of a bunch of Scrappy folks doing a bunch of different

stuff keyword we wear a bunch of different hats out in Utah south of Salt Lake City um but we're out on a mission to pretty much make vendor management suck less put it bluntly um and I think that between sending questionnaires manually through email over the course of six months cutting that down to our general you can have it done same day maybe next day uh is has been a net good so that is my sales pitch for wistic I don't I don't believe in selling during so what we're not going to be talking about in other words setting realistic expectations right you're not going to walk away from here being AI Pros because I'm not an AI Pro or an expert

or really good at it all um but we are definitely going to be talking about it I'm also not going to be able to teach you in teach talk to you in 20 25 minutes about how to completely der risk AI use with your vendors that is that that's a book uh and and right how fast technology moves that's not something that we can reasonably do but we can touch on the topic and like I said we can share the ideas the experiences and try and make it a little bit better and we just do that as often as we can I'm also not going to teach you how to boss your vendors around to make them do

security your way right um this happens right I think we've been on both sides in some capacity and I I just got to say it doesn't work so just want just wanted to bring that up in case you thought that I was going to give you some sweet tips and tricks to uh get a squeaky clean assessment on your next vendor review I'm not going to be able to tell you how to do that that's that that's it's just not feasible so what we are going to talk about though is what AI vendor risk is um right I think that definitions are important so we're going to cover that we're going to cover what AI vendor RK is and is not from a high

level we're going to look at some things that will be helpful to avoid make the process a little bit painful as you either review vendors that use AI more actively right it's not going away or as you look to integrate AI into your product how to maybe position yourself a little bit better so that when people start coming to us as you you can react with a little bit more foresight um as those come because it comes it comes with its own Suite of challenges which is I'd say it's fun but like I said I'm not a liar so it's it's not fun but it's going to happen anyways and we're also going to look at some

resources to get you started as well I told you I'm not going to sell I don't think a mention of my company doing which does vrm software counts as selling so just again heads up and like I said some terrible memes but I think you have to what right okay I'm going to take a quick second before we really dive in we we work with a lot of really serious stuff right we one of the last presentations was how do we make our metric suck less and the seriousness that comes with that and I think that's really important especially when we come to conferences like this but I think that if we're always in a mode of shoot

like there's always this next thing that we got to do there's this next thing there's this big scary out there we just kind of burn ourselves out we get really jaded and I think it's nice sometimes just sit in a cool room smile maybe chuckle a little bit and it primes us for the next thing so again just setting those proper expectations if you're looking for this to be purely educational aha I got you the doors are on both sides but I would love to have you so uh for and this is the primer again for those of you that do want to leave at any given point the main points we're going to be talking about are AI

doesn't have to be scary it is scary and I think a lot of that just comes from us not being super comfortable with a yet which is okay CU it's pretty new right chat gpt's really only been GA for well like 18 months if that right that's that's not a long time but I think the more time that we spend understanding technology what it does how we're going to use it the scaries go away we can do our jobs a little bit better from that point there is no one right way to handle AI vrm there's too many acronyms guys but there are some not great ways and that's kind of what we're going to

be hitting on right we want to at least not make mistakes and we know we can avoid and then as we get into the unknown we can say hey this works even better um also I'm a big believer in GRC right I I I shouldn't say right you guys know me this is my first time meeting you all and I I I love being here this is a great place um I'm a big fan of GRC and GRC more is a skill as opposed to a function that's separate from a bunch of other stuff so GRC comes into order for a reason and it helps to avoid the c a bunch of cool stuff hope that it doesn't

get hacked because security right um we do these things for a reason we have to think about what we're doing why we're going to do how it's going to help the mission and then how we do it in a way that's not going to blow up in our face essentially um and we do it so we avoid this kind of stuff right A lot of people are scared of this I regularly have nightmares of this maybe not specifically Terminator but I don't love the idea of machines thinking for themselves I'm coming over to the other side of it and realizing and part of why I'm giving this pitch now is because AI is just software and

Hardware right it's technology and as we come to understand it we realize that yes this is a great I I guess you could say arguably a great set of movies but that's what it is right this is something in the world that we have to deal with and again before people start leaving um I I think the take-home for this is as you develop a way to handle ARS in your company and for the vendors that you choose to bring into your environment think about the wife this is a fancy fun um nostalgic way of thinking about okay before we do this should we all right so Jurassic Park rule I like to throw this out a lot

as much as I can so before you do something I I would recommend I guess this is where I can give some good advice is before you do something think about why think about is this really necessary and I I I I think that this will help solve a lot of the problems that you run into as we as a collective industry as a collective set of professionals start figuring out okay this just doesn't work when we're have le having legal discussions around how we're going to protect ourselves from our vendors using Ai and we can just get away from some of these worst practices so let's get into after that long diet tribe um what is AI

risk right essentially I think it'd be boiled down to that the risk of using AI right and that's how we use it right do we allow our employees to use chat gbt are we going to integrate um what is that anthropics claw into our product right that's something I know a lot of people were doing especially last year all right RSA black hat that was the talk of pretty much every conference and that's kind of it right I don't think there's really need to over complicate it and I think having that proper Framing and making sure that we're not going to but but what if expanding that right let's put a good decent workable scope around it but really is that

really helpful I I I'd like to think it is but if it's not right I think if you were to Google anything about AI risk you're going to see a lot of articles that bring up these sweet pretty infographs with lots of colors lots of numbers and figures things that we love and are just like the absolute fruit of of of of of board discussions right like these are things that we want to distill and show our management and I'm going to say perhaps a little controversially that these are really not helpful at all um I could be wrong I'm not saying these are bad or that that there is no value them but from us as a practitioner standpoint

I don't see a ton of value on this because it doesn't tell me okay how H how is the introduction of AI really going to hurt my organization or or present threats right right if you're looking at the top one we could see oh well it feels dos like that dos was around before AI right A lot of these things have been around for a long time I mean I hope so right either that or I'm just speaking nonsense so I I I'd argue that there is no good way yet not saying that there isn't a way to do it but as of right now there's not really good way to sum up okay what does what risks do AI present

to us overall right at least not discretely from the rest of Technology right when we think about how we manage risk a lot of it is it's not different from any other software right and it's understanding the technology understanding how this impacts our organization that's really going to help us understand what needs to be done and then the next steps we take to mitigate said risk and right I mentioned it before as we understand what AI really does how it works right and especially when you get into the concept of training right that's that's the big scaries that I've been running into and having to explain and give good Assurance on is we don't train our AI

wistic has been integrating AI into our product we don't train AI on your data we have multiple level of agreements that says you may not train using any of the data that we sent you right and I think once we're a once we were able to demonstrate that we had a lot less scaries from our customers not saying that it it inherently closed better deals from that perspective right this isn't a sales conference but right talking from One Security professional to another that was a big point of assurance right and that that's something new right when we think about regular software regular sass right we don't really talk about data ownership because we've gotten to the point to

where it's just kind of generally accepted that you own your data so we've just kind of come full circle we have to come to that same general bucket of terms where we understand okay either we own our data or we don't for example so uh and right and what I'm a reiteration of what I meant by full circle right like you get a lot of you'll probably remember what the internet did to the world right I don't remember when we started using the term app I we don't usually right we say oh you go to this website right but it's a web app and I remember there being a lot of scrutiny well I grew up in a small farm toown in

California I mean California not Silicon Valley but I mean like my high school was half a farm um and the internet was a scary thing so it it was a very big transition from being able to understand okay I can go and check my school schedule for example right or I can see I can go get my teachers email from that so I can send email through Google that's pretty sweet and then right think about the cloud right people still refer to the cloud as some mystical magical thing where if you have your app hosted in the in the cloud it's secure right we're good we're hosted in the cloud how many of you have heard that please I I

want some interaction thank you sweet um everyone else please this is nap time this is prime nap time and how ridic and and how ridiculous is that though right not not the nap thing the cloud thing right the I'm telling you guys it's it's just that time for what was that that's what cat Meats for for the nap thank you very much yeah but and that's where I'm trying to reiterate like the fact that there's still the perception of we're in the cloud weet right is is is is I think kind of where we're at with AI right now just because we're using AI doesn't mean the things are going to be correct that the Integrity is still intact or that

insert security or reliability or risk problem here right we're just using another facet of technology and if we rely on it as it's given to us we're going to run into some problems it's worth our time as professionals to understand what's being done how it's happening so that we can give recommendations as needed right I mean same thing you guys remember when smartphones came out first how Wild that was I remember distinctly sitting around and seeing the first iPhone commercial and thinking why would anyone need to Compu in your pocket and when was the last time you left house without your phone right I I think we're going to get there one day but I think that's going

to come at after we put in the effort to really understand and grasp how this is going to work so um moving on from AI risk what is vendor risk I'm going to spend less time here because I can feel it all of you in your souls projecting and rating radiating up front how fun it is to deal with questionnaires both asking for them and having to fill them out um I gave a talk a couple years ago at a local event up in Utah Valley called St con about V vrm 101 this is VM 2011 so this is this is kind of a fun continuation right we're we're essentially looking okay when I introduce external influences how is

this going to impact me and when we smash them together we essentially want to see how AI being used by our vendors is going to impact us and again that's kind of it right we can always expand the scope but let's start at a workable place so and the reason that I keep coming back to this really repetitive really grading concept of hey let's put this scope on here is I've spoken with management it is an ongoing discussion within some teams and some other stakeholders that well we need to really understand what is what are the future implications like if we keep thinking what if what if what if what if what if without understanding the basics and

saying look we know at least for sure with this we're going to we're never going to reach a common point of understanding and if that's something you're run into this is something that I recommend doing all right if we're really concerned about how our vendors are using AI what do we mean by that do we mean do we care whether or not that our that our email provider lets their employees use chat gbt probably not but we want to see that they know how to properly govern themselves regarding that risk so what AI risk is not and this is I don't know why I did Point by point but this is probably the changes that I

was doing it to o00 earlier this morning you can you can tell where that comes from a lot of the questions that I've had come in the form of questionnaires and customer calls and contracts and this is I think just a byproduct of like I've been saying AI being such a new technology integrated in in in in business is hey how do you train the models like we don't develop our own model so we don't train it kind of deal um but then also access to data right that from a fair point of view you do want to know how okay how is the AI doing the the AI how is it doing what it's supposed to be

doing right how does it know what to do with my data and that's where you get into the product level discussion but in terms of okay how are we preventing Sky nut from being built that's great you're gonna have to go ask anthropic or open AI right that's that's something that from a product level unless you're building your own model which I think would be really cool and I know does happen sometimes this is where again small print this is where I think we need to put a good set of boundaries on it it's not that it's not important I will never say that questions like this aren't important but putting the scope on okay in the context

of our relationship from me buying your service or you buy mine right if you're integrating AI into yours you need to be able to say okay what is your concern and and and in the most polite professional way say how is this relevant to our discussion right if if you let things spiral they will spiral and that happens with all new and emerging technology right the whole example with the cloud how often was that happening before or you're in the cloud okay that means either you're super secure which is wrong or you're super duper insecure which is not super wrong but right and and things have changed right there's been a lot of changes for example AWS S3 you

guys remember just a few years ago everything came public by default and you had to lock it down that's not the case anymore and it's going to be the same thing with AI eventually as well where it's very clear who has ownership how that's impacted and so on but until then we have to get good at having the discussion saying I'm happy to get this information for you also how relevant is that to you understanding how this is going to impact your actual risk right are you looking sorry was hand oh no no you're good sorry sorry I I I told you eyes ears they don't work that that good but I do I do want thoughts and opinions

so if you're waving your hand want to say something I don't see you or say something just shout hey you and and and we'll go with it so again to reiterate like there's there's a lot of different ways to go about it but if you're only focusing on the minutia the super technical minutia it's not that that's bad but you might want to ask yourself how valuable is this in actually mitigating the risk that we have with this particular vendor right so um it's it's like it's software right I was saying before and I know that's overly reductive but right how many of you familiar with little Bobby tables KX CD right they have a new one and guess what

there's like two minor changes right the same script I guess arguably better art right but it's all about oh you you named your kid something squl injection and then you should have sanitized your inputs right and and I thought I was going to get more Chuckles honestly but but but but I mean what hit me hardest about this was look this is this is it this really is it there's not been a lot of change right did you change do You' sanitize and validate your inputs it was like that before we had the internet then the internet was big thing and then it was still are you sanitizing your inputs now we're into AI generative AI all these different models

and we have the same problem right it's it it's I and I think that's where a lot of the scaries go is hey look like how we do it is a little different but if you're thinking conceptually from a risk mitigation standpoint is the technology doing only what I think it should be doing whether it be you as a developer or you as a consumer are you confident that it's only doing what it should be doing or it's close to only right because nothing's ever 100% risk-free but having that confidence and understanding that Paradigm of look like it's technically new technology but it's all based on stuff that we've been using for a long time it takes away the

scaries right we just want to peel back those layers and see it for what it is um I couldn't come up with a funny cuter name of saying this so these are some no no that I I've kind of experienced in handling AI vrm is essentially the pitfalls right I should have said pitfalls but these are no no um unofficial no NOS remember I'm not giving advice uh I'm not a lawyer thank goodness um there's a lot of discussion about denying the use of vendors purely because they allow AI somewhere in their environment or their product not saying that you can't because there's some legitimate purposes in which that might be reasonable there's also some that will

only use a vendor because they use AI which I think equally comes with some issues to maybe reflect on right a it's it's it's like saying hey are you cloud-based right regardless of what you're environment is like that that doesn't affect a ton it doesn't affect nothing right because if you have an on-prem solution versus a hosted solution right you have to do the you have to do the upkeep but I I don't think those reasons in and of itself are a reason to either use or not use something right there's also treating all AI usage is the same right so when we first started integrating AI into our product we jumped in with the uh open AI API and a

lot of people freaking out was like they just got breached like nope that was chat GPT which is a different platform I understand the concern all right uh and then also this is just kind of a pet peeve of mine because one of our products that we use is kind of like trust Center type stuff um where I say here is hundreds of hours of work that I put into completing these questionnaires compiling our policies all our s to all our certifications and stuff here you go that's nice we want you to fill our a custom questionnaire someone's experience that I heard someone laugh about that um and and and and what I mean by that is I

know it's hard to validate but I think it's a good place to start right when you look at anthropic when you look at open Ai and all the other folks that are doing that and you're looking at using them take a look at the security stuff they prepare that for a reason and that's specifically the reason that they create that and give it to you and it's the same thing with your vendors right hopefully rather your your vendors that have ai and their products have taken the time to create documentation say this is how we use it this is what we want you to feel comfortable doing if there's holes absolutely ask say oh well I don't see

any mention about ownership or about deletion or about training ask about that absolutely I think it would also be a disservice to both you and your vendor to not ask first ask and then read if you're given something so even worse vrm yes yeses maybe some recommendations from some experience from just a little bit of experience now right don't I I'd re essentially recommend not getting too hung up on oh well it's the AI it's scary right you want to understand the whole ecosystem because there's a lot of service providers a lot of software that have ai as an option that's something that um at my company we built in AI powered features as an option and you

have the full realm of functionality and being able to do what you want to do both from a buyer as a buyer and a seller from a vrm perspective without a drop of AI without one bite going to or from our um the model right so should you assess the capabilities absolutely how they're securing it for sure but there's a whole underlying set of Technology that's going on there and it would be shortsighted to fix it only on the AI right um and and and right we're looking at risk right we're Security Professionals we are Security Professionals but like I was saying at the at the beginning I'm a big GRC fan as a skill set as opposed to

a as an isolated function we are also risk professionals right we are good we as Security Professionals are good at understanding how technology works and how to secure it because it mitigates risk so understanding how that fits into the big picture is going to drive a lot of good conversation and then help make this more efficient as you go on oh and also if you got really important stuff going on get it in the contract right if if if it's not in the contract if it's not in the signed contract it doesn't exist so if there's an option to say I don't want my data to be used in training whatever the underlying model is get that in writing

and then you're good to go um at the very least right if something shady is done you have recourse right again we're risk professionals we're security but we're also risk professionals that's how we handle this type particular type of risk forgot to breathe for a minute there y'all um some great learning resources again I told you I'm not going to sell too hard um but I do believe in what we're doing at wistic um we have great relationships with other organizations that are author to make this process more efficient less painful um and and just overall looking to make things better the tpr is how I know the third party risk Association does fantastic stuff a lot of other ones too

right so when you think about the CSA the cloud security Alliance the vendor security Alliance they have resources too that's awesome nist right who doesn't know and love nist they came out with a AI risk management framework open source oh by the way I'm going to have a QR code to these slides later please feel free to link it because any of these underlying underlined fun things fun resources are linked so you don't have to Google it I saved you two seconds you're welcome um there's also a couple of ISO standards out there 23503 for um implementation guidance I think is the technical term right it's kind of the iso 2702 to the uh ISO 4201 which is an

actual certification that could come out it's still new but I'm also a Believer in First movement Advantage so if you want to go look at how that's going to impact you even if you don't certify this is going to be a good way to kind of get your chops wet and learn how that's going to drive adoption of AI in in the ecosystem later what was the name of the C again uh ISO 23 2353 and 4201 yeah so the the latter the 4201 is the one that you actually certify to okay yep um also I'm I'm I'm not a paid rep or anything but Walter hadock on LinkedIn if you not seen him has a bunch

of great stuff not just AI related but more recently his company stack ofare has done fantastic stuff um I'm not saying that he is the authority but I I I like his stuff I learn a lot from him pretty regularly so I'd recommend him uh go follow him ask him questions he's got a bunch of stuff on there uh the QR code and that underlying link is a to a talk an article that he published recently I think is a great primer all right um n RMF has been out a little while but AI RMF so I I think I mistyped that but at any rate great stuff great resources and it comes out regularly so uh I double

linked a lot of these that's my bad so ignore the left half uh there's also some great new legislation some of you love that I'm not particularly a big fan but like any other law buing citizen it doesn't matter whether I like it or not if we have to be compliant we're going to be compliant so some example legislation is coming out of the EU right like it tends to do um gdpr everyone's favorite thing driving a bunch of privacy legislation here is likely also going to be a precursor here for the EU AI act um it has been adopted and is slowly going to be ruled out I think over the next three and a half

years and they have given a tool to help people get ready go them uh called the cap AI assessment which essentially helps you determine whether or not you are ethically efficiently and securely developing and implementing AI systems I'm not a sales man I just realized that sounded very markety and that's that's not me I'm sorry about that all right in summary cuz I got like 20 seconds is AI is just software doesn't have to be scary as long as you're willing to put in a little bit of time to try it out practice learn about it and how it works and the scaries go away um therefore because AI is technically just software right the same Basics apply good old

Bobby tables Bobby drop all the tables it it it's it again goes a long way right if you understand how it works you can apply the same Basics and really right I'm not saying you have to use AI or that you have to avoid it all right there's going to be an in between so decide what you need to do then go fix some of the problems that come with that um I think I was just in a jurass part Jeff Goblin mood so you're welcome everybody and uh like I promis slide deck over here here's my LinkedIn if you want it and I'm sure you're going to figure this out I believe in you

you're all amazing as pleasure standing and talking here with you and I I I I'm sure you'll figure it out like Ron Swanson said thank you very

much any questions is there I think for a lot of especially small companies co-pilot similar systems are already starting to become like super useful but I worry that we don't have Poli books on what you could be putting into chat GPT or or co-pilot or helping you know write papers or other things like that so right now we are not inting at the corporate level um but people are still using it off the side for things so are there draft corporate policies out there for governance and for like hey don't put this into it but you can put that I think there is um I wish I could tell you where I I honestly I think W

like I was saying before I'm sorry I'll go back to that in a sec folks I think Walter hayok might have a couple of stock templates but what he would probably say if you talk with him and what I'll also say is the chances are very good that um there's never going to be a stock policy as good as one that you draft yourself because right only you know what you need what you what what your environment can't tolerate um but in essence right you have the purpose for using a type of technology in this case something like co-pilot we have something called I think because co-pilot is technically Genera of AI we have a generative AI

acceptable use policy um if you message me on LinkedIn I can actually get you a sample of that I just try to connect you so perfect yeah I would love to because you trying to get a handle on some of the back in like just basic policies for AI there's nothing out there to look at yeah like Mrs over here said chat GPT can write you on I'm sure chat GP now whether you want it to or not exactly ultimately right um like any effective policy is a making sure it says You must or must not and then making sure that those it applies to are very well aware of what boundaries are and what happens if they don't and then

right we're getting deeper into the governance bit but then enforcement and all that so um but ultimately I think if you can put pen to paper and say this is exactly the line with which we're not okay with you doing whatever anymore that's a great place to start work backwards and say these are the things you can do yeah so but happy to have a longer conversation about that for sure yeah use like the same acceptable use policy the same confidenti establish it is about sharing sharing information you you definitely can right I I think that you run into some cavat when you think about use case right but I think that's where it again it's very

individual right like if we don't share any data with any third policies and that includes Cloud providers right so if US host being hosted in the cloud we can't share that that becomes a problem because we're not on Prem but right there's there's definitely ways around that it just sometimes comes with re architect and policy so yeah yes yeah your slide deck doesn't have access it it does it allow you to request it you can request it but it's but it's not automatic for the read access hang on for the view access gotcha

I'm hang on I think I'm I think I'm on hot spot um in the mean I'll look at that right now um did you yeah um if third part vendors have a partnership with um with a company such as open AI to keep the data private they probably change their prices have you there um because that to pay more for the partnership have you encountered that sorry I think I'm might a Miss part can we say again so in order to keep your data within your company you have you typically have to have a partnership with the company that's providing the Genera AI enterpr verion of it in which Cas you might have Tok so the the the heart of the of the

question of the comment is essentially have is there a difference in in pricing because of a supplemental agreement that comes with saying hey don't use my data um I I think that kind of depends on the business it's probably going to be the more probable outcome um I know it kind of was with us because we sell our AI feature separately uh but we but we did enter but that's not always the case right um there is something with specifically open Ai and I know every other provider has it called an Enterprise agreement that essentially says because we are a business we insist that you not and that usually becomes with okay that means you have to pay at

least this much a month instead of the usage cost right where it's just pay as you go so it's possible but again I think that's kind of like a case by case where wherein right like so for for example slacker Zoom they've kind of integrated AI features at a more General level that might not be the case because you would well rather that would probably more definitely be the case because as a non-paying user right for users you can use those for free um they process your data and for training which generally means that's how you yeah yeah but great thought great question it's I'm sorry I wish I could get something better than it depends

but so yeah seen it both ways um access requests there we go um for everyone that's saying hey that's requesting access to the slides I'm going to slowly work through that and changing it so you can just publicly view it sorry thank you very much any other questions before I step off and let something more interesting you're most welcome no I think he answered I was how many people in here are managing their third parties risks third party risks just I do okay internal internal risk who who does that y um thank you all so much I so like I said I'm from out of town I haven't been in San Antonio for 15 years

and this is just one of the most fun conferences been at and it's because of you guys thank you