Securing Azure OpenAI Apps in the Enterprise

excellent last session of the day uh I'm standing literally standing between you and the uh you and the Afterparty so let's let's make this an interactive one please shoot out those uh slide of questions throughout the session and I'm I'm more than happy to uh just focus on those give more time on those U as as we go forward but first some slides uh so we'll get those over with uh as well so very briefly uh hi I'm Carl um I'm consultant I I talk Cloud I talk security and nowadays has everyone a bit of AI on on top of that uh as as well so as part of my day job I I do consult uh

mostly Enterprises uh but I I I do have a background as a I had a real job I was an engineer uh now it's mostly mostly hand waving uh and PowerPoint is in instead of pow shell most of the time but still a lot of fun so um this this is really um this this whole talk is based on around a specific experience specific case so a year ago or so if someone remembers there was this thing called generative AI uh CH GPD had just come out a couple of months before and Microsoft rushed also out labeled the gener gener generally available label on top of their own uh version of the product uh that as well but basically I

was working with this with this massive uh banking or organization usually very uh slow moving usually very kind of let's do everything right this especially this kind of not invented here uh we they would rather build stuff stuff from scratch for years uh to make things absolutely right not get uh you know flagged in audits that sort of stuff but now with with generative AI uh everything changed so now they suddenly we're like hey we are doing this can you uh we we are now doing this uh generative AI thing we are using Microsoft Azure Microsoft told us that it's now generally available it's ready it's done Microsoft has this great security story on top of this uh can you

just you know a few Sprints I'm sure it's going to be uh you know just a couple of weeks you're going going to get this right and we can get to using this ji stuff and you know save the industry what whatever and as a consultant or improviser or whatever you know sure how I mean how hard could this really be i' I've done multiple other Azure projects before I've done I've heard about this this AI stuff before uh I know math you know what could go wrong right well it's a whole whole talk um so let's uh let's get uh get into it so this was my plan then so I'm kind of uh

going through the same workflow or same flow of logic uh that I was going through with with the client and it's it's a bit of Lessons Learned but also the kind of um benefit of uh benefit of hindsight as we as we go through uh go through this session and as a reminder please do ask those questions uh in the slid out throughout the session then we we have a good backlog to start going through them uh at the end of a few minutes there so basically very briefly even though they they knew that they wanted to do this in this Enterprise part on the Azure side they did want to understand what's what what are these

kind of different different hosting models available uh how to how to get it working on their a environment kind of sandbox kind of disconnected environment and how to put it uh together with their existing you know Enterprise Landing zones continuously audited environment etc etc so that was uh that was our plan and that's what we are going to follow them okay so uh very briefly I'm kind of summarizing uh at that time there was this thing called uh jet gpt3 plus I don't think was was announced uh back then and there was a blog post announcing J J gbd Enterprise and that was pretty much it so um so this is a little bit uh kind of updated version

this is this is where we are right now there are these different versions of of the chat gbd uh which is I'm sure someone uh who is either from open AI from mic Microsoft uh if they're in the room they can comment the exact legal approved language but it is uh similar in features but not Cod sharing whatever the official language there is but very similar in in terms of Fe features and capabilities um but not exact not not exactly the same thing but close enough for our purposes so basically we are looking at this chat GPT versus open Azure open AI what's the difference what can we get uh out of the box from this software as a service

version uh in the chat gbd world and what can we get this on the platform as a service World on Azure open AI uh nowadays there's also also this uh approach U from some of the vendors also from Microsoft there's this uh if you if you if you're all familiar with the uh Cloud shared responsib responsibility Matrix there is this similar shared responsibility Matrix now available for AI Services as well so from that perspective you can also kind of think of this as software as a service on the left and more and more control available for us more closer to platform as a service on the right and then what's kind of missing missing from from here

is is basically the part that then then would add somewhere over here the kind of bring your own model host your own infro type of stuff then this will be the kind of closest thing to infrastructure as a service there but very briefly there is this uh free version from chat GPT there is a claim that Enterprise gives you some more features it does claim some features you can go to the trust Center sign up for an NDA take a look at their uh reports and you'll understand that there's some some more features there's actually an SSO uh integration coming up there's you know domain allow listing that you can do for hooking up your teams team

members so that you you can actually have shared workspaces and then there are some weird features uh such as uh such as this uh encryption at rest I'm very curious you know knowing that this is built on exactly the same technology this is built on Azure kubernetes service and Cosmos DB uh after this session uh I don't need to break any ndas if someone can tell me how can how can they say that encryption address is not available for the freeer please let me know because I I I couldn't couldn't couldn't get it working working mentally on my side but you know there's some things that are a little bit more kind of slide slideware uh and some that are

kind of technical differences there so basically if you if you use chat GPT uh first Party Services it's kind of closer to software service it really doesn't give you that much uh security features or from an Enterprise perspective controls uh in there and then there is this ad open AI Service uh there's kind of tldr apparently I have animations so let's open those up so so basically there are those free and plus versions uh there's some other versions coming up uh nowadays as well both are really based on the same thing both are apparently hosted only in the US if you're if you're coming from somewhere else or working on different different locations like this bank was working had

to have an environment for Europe had to have an environment for multiple Asian environments as well you know know they they didn't have the opportunity to just go for this kind of globally hosted or hosted in one place only environment and otherwise my kind of own personal tldr is that you get a little bit of more kind of paid privacy or at least the promise for paid privacy if you if you choose for that but if you need to choose you know bring your own uh keys for encryption if you need to have some network controls Etc pretty much your option quite easily is uh is the Azure open AI service which obviously it's it's a it's in the name of talk so

hopefully no one's surprised uh that we we chose chose that uh for for what we were doing there so let's uh let's start looking at it so as I said I I come from from that Azure background I was like yeah it's a new Azure service it's supposedly even in general availabilities which means that it should have passed through all of those quality Gates it should have all of those bells and whistles that any Azure service that is claimed uh that that is labeled generally available you know production ready um so the general approach for for for this uh that I had uh and that you should be able to replicate the Sim similar similar

approach overall is this kind of concept of Microsoft cloud security Benchmark so this is really just this beautiful origami of different different type of um outside requirements so from an auditor perspective or from a uh Caesar perspective if you want need to be compliant with you know dozens of different outside uh security requirements uh there's usually you will just do all of this work in Excel and then try to prove to an auditor that yes the fact that you click click here or the fact that you have encryption on really really takes this and this security boxes in whatever CIS control framework etc etc now microsof has done that part for us so that that should be

the part that you should get started with so there's documentation there's policies there's technical controls available for all of the services built by the same product product teams so again uh at least in theory any product who goes through this it's a product group's responsibility on Microsoft on their side to actually produce this uh Cloud security Benchmark controls and documentation in there fantastic for the first 10 months there wasn't anything so just like the client uh also also in this case our uh our Cloud partner had also skipped a little bit ahead and skipped all of their internal processes internal kind of best practices on this side and just focused purely on just getting it out of the door and hoping

that eventually the features will catch on but now uh since September October can't recall now we have this control B security uh Benchmark available there's 35 controls that are responsibility of the client or or user of the cloud and seven of them are actually provided right now now uh for uh for aure open AI service so not exactly best uh best kind of feature parody Or best kind of coverage as such but at least we have some controls available that we can point to saying that this is a checklist uh if if I if one of my internal controls as an Enterprise is to basically just do a check mark on everything that the vendor promises or

vendor tells me to do if I go through this checklist everything is good maybe at least there's something to follow there um but basically as as we look at these a little bit more closer you will see that there's a little bit of replication in there basically this uh two two ones on the bottom the network security ones are just uh very similar similar ones over there uh resource logs uh if you if you know Azure this has nothing to do with the service itself it's always on by default in there uh key key Vol integration this is actually pretty good I have no idea why this is split into two control but that's fine

and and actually as as we are seeing there's even even a third one here with uh with the uh customer managed Keys uh Keys option in here and then finally there's something called Data leakage and loss prevention which is specific to this type of service uh this Azure uh open AI or Azure cognitive Service uh group of services only so we'll take a look at that and as well as these other controls here as well basically very simple it's just the kind of auditor and Enterprise friend L way of saying that uh you you should figure out your Access Control you should figure out your logs and maybe have some network controls in place here as well make makes

sense so so for uh for networks uh very very kind of simplified view view into this is basically you have uh you have inbound and outbound Network into this platform as a service component it eventually this Azure open AI service service anyway this thing in here this is just a resource that you provision in your Azure subscription if you're from like AWS world this is like AWS account uh or Google Google Cloud project you know it's it's a thing that sits in your that sits in your subscription that's that's great and usually it has a location uh of which data service data centers it's being deployed into uh and then you add some configuration in there

in this case basically uh for inbound inbound we can actually config something called a resource firewall despite its name it's not a firewall uh but let's let's run with this nevertheless it's basically a short uh Access Control list uh this is configurable in in any other way as well this is familiar to you with any if you focused on any other Azure data service before it's the same thing in the UI it's it just says excuse me um yes uh in the UI it just says networking you go into a tab called uh firewalls and virtual networks and then you select this uh instead of all networks you go and select selected networks or even disabled in there uh

two things you need to know there's uh both the virtual networks as well as uh IP address range these are uh actually cumulative so you can have both virtual networks listed in here and then you can add any type of IP address that doesn't have to be included in that list of virtual networks uh in you in there so uh especially if you are trying to configure all of this at scale you do want to later on audit audit this part and make sure that this set of IP address ranges doesn't for example include Consultants home office IP addresses uh in in here good so that's very familiar to us it puts uh kind of inbound inbound

network controls in there but then there is also an option now to even though we can't really configure the whole thing it's not a firewall we can't really go configure in inbound outbound uh even in the level of rules and priorities and logging as we could for like an infrastructure as a service component uh we do have something for outbound uh network controls as well and this is the thing that was called Data exfiltration prevention or even DLP feature uh in there has nothing to do with Microsoft like the pview or this this kind of M has other tools that are related to DLP it has nothing to do with any other type of d LP this is just a firewall quote

unquote now instead of being a virtual uh virtual Network or or IP address based uh firewall in here this is basically a uh an allow list of fully qualified domain names that uh we can configure uh that our instance of the aure open AI service can talk to outside so for example if you are downloading those dependencies from hugging phase and that sort of stuff that we've been hearing also in this room before that's where you could start building that uh uh Access Control list there's no UI there's no SDK uh us just have to know uh that this is actually available to you and then you can uh query then you can just send send in

send in the changes uh using the rest API by yourself but eventually this is going to be there it's restricted outbound network access say you set it on and then you provide it uh with a with a table of up to up to a thousand fully qualified domain names and and that's it then you hope that it's on then you maybe test it uh you don't get any sort of audit logs about failed or uh not failed uh attempts on this but it's something that we can turn on and yeah I have this beautiful burning Cloud platform uh sign when I think something is not exactly right but I didn't just want to add it to every

slide so I'm highlighting some of them with uh with that icon on the corner there all right encryption address very similar to any other type of azure data service by default you do get encryption encryption uh again this is the default encryption that somehow uh the the open AI folks who are building chat GPT free and and the paid the user based um free and premium fre and plus whatever the non-enterprise versions of chat GPT somehow are able to turn uh disable this feature uh that even though as user buying things from Microsoft you can't or maybe they just give it to you for free um so that's called Microsoft manage keys mmk and then you can bring

your own keys with customer manage Keys you point it to a Microsoft first party keyb no you can't bring your own like hhic cor Vault or anything like that uh here by default uh if you want to use that as a kind of centralized way then you need to First have your own VA and then synchronize from there to a natural key VA etc etc or if you want to control those keys uh Keys yourself uh then the physical Keys as well then Azure key does support that but again may may or may not uh work work for you there very very you kind of from Azure perspective this is business as usual stuff if someone wants to ask what

does this actually encrypt that's even a better question because uh I truly cannot answer that based on any sort of Public public information the only kind of wag recollection there is that maybe hopefully the whole cluster is encrypted but no idea if it's actually the the the the runtime if there is some sort of a separate storage area that is being encrypted we don't know but encryption is on so I guess Auditors are happy this is from an Enterprise perspective again same thing for audit logging uh it's it's uses a standardized Azure feature you f you go to the resource you turn on the log export you select from these three different items audit logs request and

response and Trace logs so the one in the middle here the request and response oops sorry for that spoilers the one in the middle here request and response this is the one that uh when this is disabled that then you have your then your application provides this paid privacy of not not training uh not training your uh using your client's uh data if this is turned on then this is this is exactly the feature that the the free and plus versions of the chat GPT for example are are using and then you of course like on any other Azure service that uses the same logic then you select where are you storing the exporting those logs into

you can export those out of azure of course in aure natively you can export them to storage account or loganic workspace in there logs look like logs uh there's interesting stuff uh that you can then go ahead and dig a little bit deeper there Access Control just like asure storage accounts there is this account uh level uh Super Root admin key that if you get access to this key you can do anything uh with that uh that's called local authentication nowadays uh you should not use that but you should prefer to use uh Azure or inro ID authentication with your users uh and your um and your workloads uh you can disable this again no API no UI for that but disabled local

o said that to true that sounds great the only little thing if you actually are still developing your environment uh then you your developers might actually use the web UI they might go to use this Azure AI Studio that actually uses this local authentication so if you uh if you're using that that will break if you turn this uh turn this feature on but again this kind of traditional Enterprise once it's kind of waterfall level once this is done we ship it to ction then we turn all of these options on that fits uh this workflow workflow very well all right so very briefly in in Azure of course uh anything that we want all of those features that we just saw

everything that may may or may not have a UI all of those security controls it's great that someone goes in and looks uh at one point of time that if these are turned on or not but we also need to have this continuous auditability we need to be able to export and verify that those settings are still what we think they are and maybe even combine that with uh with our infrastructure as code deployments maybe do some what if deployments Etc the native tool for that in Azure is the Azure policy framework uh there are no built-in policies or policy initiatives currently for uh for this uh specifically Azure open AI service but fear not this is actually

under the microsoft. cognitive Services resource provider so anything that has been built for microsoft. cognitive services will work it will still tag these things for you but that means that if you have any other resources is already deployed uh and you are only looking at as your open AI uh then you might get some false positives so for example if you're previously deployed whatever Azure search is called nowadays cognitive search AI search whatever it is um then that will also get flagged with the same exact policies um you need to do some manual exclusions out of that uh so that that was that was a concern uh for example that we had in there so

if you if you are building your own anything that you are doing in there just use the microsoft. cognitive Services policies uh not all of the aliases are there just yet but you should be able to do some reg regex regex magic over there uh by just filtering out everything that is not under Microsoft do Azure open the open AI here are some examples of the those policies that I think are useful to to repeat from these others so again for all of these controls uh you can just reuse the existing cognitive cognitive service policy that is built in it's first party provided by Microsoft so from a Enterprise perspective you know Microsoft is responsible for updating

them etc etc so it's just a list of them they are nice nice nice policies so because there are policies you get kind of continuous information you get nice crafts and you can export this sort of stuff their policies if you are building these on your own here's a couple of examples and I have a link to a public U Azure communities GitHub repo at the end uh here's one example where we have uh again we are just using let me actually I don't know if you need to zoom in I need I need to manually zoom in um so we have in here the microsoft. cognitive Services uh public uh public network access that we are looking at and then

we are specifically as as you can see here we are just filtering manually we are oops well it's underneath here we're manually filtering this uh this type of service called open AI instead of search in here very very small small chains here this is actually more specific to open AI we are going into the details of the deployment we are selecting that only certain types of models can be used or certain versions of the models can be used this is uh in this case we are looking at this a little bit more details uh details in in here good and that brings me just uh very briefly uh into quote unquote reference architecture um there's this

is a little bit different different from the Microsoft option uh but basically this is pretty much putting all of these things together what we had of course this is usually consumed by some sort of a this is an AP application basically that you building you should expose that through some sort of API Gateway uh to your uis uh or your you know web app web apps basically on Azure that are then hosting some sort of UI or web chat Logic for your uh for your users and then if you are if this is being consumed by by really uh people people everywhere then you can deploy that using as your front door you can put

your v in there etc etc in this picture I just don't you know I just didn't didn't didn't fit uh fit the uh API Gateway but maybe it could fit for example for example somewhere in here um and then usually you would also have this some sort of retrieve retrieval augmented generation so you will store your indexes uh for example in AZ cognitive search and then uh and then you could maybe store your vectors in AZ Cosmos DB there as well and that's it we have five minutes set for questions there's a bunch of of links that you can mentally click these links and and that these slides are available in the same place that the

rest of the slides are uh still TVD I guess uh I am trying to put some thoughts on this together uh so do follow me on them them internets for some more stuff uh if you are coming over uh during well here if you are going to the AI Village I'm sure there's lots of interesting stuff and over the summer summer also at the summer camp as well so thanks a lot and let's jump over to questions all right that's kls um as per usual we'll take questions via slido uh if you have any uh you can submit them instructions on how to do so are besides asf.com qn uh and that's all you'll need to do

um we have one question on Deck right now uh how do you go about finding all of these undocumented features and configuration options available in Azure like the fqdn allow list um that's fantastic my favorite I I don't have Network hooked up in there but there is uh well basically you can just query uh query their regular regular um Azure Resource Management API yourself um if you're using you know Postman whatever you're doing I kind of have a regular regular kind of uh Explorer going on every week seeing okay if there's something something new going on in there if you just want to browse around you can just go to resources . azure.com uh resources. portal. azure.com instead

of the regular management portal you'll do the same you'll see the same thing but just with a nice little visual layout uh that's that that's what gives you the uh gives you that uh raw rest API view without the UI stuff that is uh that is on the management portal in there and that's how you'll you'll find it uh usually then I'll just edit edit a field in there and try try send it see if it sticks sometimes it doesn't sometimes it does beautiful uh next question what are the biggest questions and concerns large Enterprises still have deploying chat GPT via Azure any big fears or has security progressed far enough uh I hope the tldr here is that

it's not far enough at all um I think there is always something to choke about having job security etc etc but but no there is there's lots of concerns especially these kind of black boxes of yes there is encryption address but what does it really actually encrypt how can we be sure that the the fact that there is this network control in place that it actually does what it says on the tin for example if you compare to AZ your kubernetes service you actually get visibility not just to the resource itself but you get visibility into this manage Resource Group that creates all the cluster the all of the rules if you if you scale scale your AKs up if you

add more noes you'll see in that Resource Group that there's a new virtual machine added there's there are new Nicks Nicks added there's new Networks security groups added all of that is visible to you and you can audit you can even go go and you know log into those nodes if you want there's no such visibility transparency here in here just yet um maybe we'll get there maybe we'll we won't Frost uh next question you think other vendors outside of Microsoft have more security features or is landscape so sad in general it is the tldr is that this actually is Cutting Edge uh honestly I I I do have like I have like a Backup

backup uh slide of somewhere else but uh basically the the technical features are are almost non-existent for from all of the vendors right now Microsoft has a little bit more because it Taps into the existing existing uh Frameworks in here uh from an Enterprises perspective uh AWS has has a good set of tools when if when you're hosting your own model you can tap into that that logic quite a bit um there's always the question of do we want to go for those own models but that's some for another discussion on the Google side uh right now the kind of uh the trust Ender material hasn't been super amicable to to the Enterprises that I've been working with uh but I

think everyone's putting all of their resources to building both the engineering and the kind of the doc cap capabilities right now so I'm sure this is very different content next next year we got one minute left and one more question is there a cicd pipeline you recommend for managing services like this something similar to a cicd pipeline um yes I mean it's a it's an Azure it's a cloud resource so of course you should deploy everything using uh using uh you know infrastructur as code um I had UI here because it's just a little bit nicer to show UI um for for the stuff that you had those rest API calls uh right now those are not

available Azure uh terraform uh provider or the uh or the bicep or Azure resource managers just yet uh I mean it's RSA week uh next week so maybe something happens then maybe something happens by Microsoft build in two weeks but uh but basically anything that has that can be deployed using the infrastructure as code tools in there natively obviously you should put it hook it up into cicd those are those are literally the seven things that you can you can check for uh in in your code whether it is natively using the tools or adding like web Hooks and man manually check those same things that we saw from those rest quaries right beautiful uh that's all

the time we have for questions today uh happy hour is ongoing now a party will happen right after all of that's happening on the fourth floor top floor uh and I'd like to thank our speaker Carl O uh for the wonderful talk and we have a goodie bag for you from our sponsor soet security everyone please have a great time and I'm going to have to ask you to leave cuz we need this room cleared by 6:00 all right let's go