AI Won't Save You—But It Might Have Helped: A Retrospective on DevSecOps and Cloud IR

[Music] Well, I talk really fast and I sometimes mumble my words, but if you need to find me uh after the meeting or after the session, I'm happy to meet with you and we can you can ask me all the questions you would love that I potentially could answer. So, all right. So, upward and onward. Again, my name is Josh Henkins. A little bit about me. My favorite part of the talk. Um, of course, I got the disclaimer. This is me. Nobody else. If you want to sue me, don't sue me. I'll make let me give me a five minute head start so I can get an LLC up and you can sue that. Um, been working a very long

time obviously as Justin said. Um all the cyber defense u roles I've done engineer architect analyst incident handler director security we did for the analytics arm for Kroger um built that program up from scratch all the cyber defense and some compliance fun with that as well um keep busy with asserts and right now I do the fun part of advisoring or on the advisory boards for certain places. So let's move on. Here's the agenda for the talk. Four bullet points. Kick off with the AI panacea. I think that has a different a different definition for everybody, but I'll give you mine. And we'll go through use case, the unplanned. Spoiler alert, that's the instant response. And then the use case

for a more planned uh cyber security project. And then I'll give you AI's through the look of AI's lens through that. And I'll conclude with a summary slide. And hopefully there's time to get through the reference materials. If not, I'll show it on the screen and you folks can get it later in YouTube or whatever. So there'll be some questions too, I guess, at the end. There's time. So anyway, uh the AI pianace of promise. We're going to get t um gains in efficiency, productivity, and re regain time with all this fancy technology. What are we going to do with this regain time? Of course, we're going to put it to work. We're not going to do anything

fun with it. You can do what we want. I don't care. But this regained time at work becomes repurposed time, right? I don't I was a director for nine years. So I never said, "Man, I got enough hours in the day to get everything done. I got every employee I need with all the skills, you know? I think we all have a laundry list of things that if I had one more person or if I had more time, I could get something done, right?" So, we're going to repurpose the time hopefully the a AI that will um grant us and we can use that to take creative approaches. Um, a lot of my talks are

more about that layer 8, how to get buy in and stuff, but there's some technical uh pieces in here, too. We'll see if AI can help out with that. I guess non-technical pieces. Spoiler alert, it can. So, I think so. All right. The first use case, some um IR in the cloud are fun, right? So, like I said, a some of these details are left out on purpose. I can't say them in here, but if you want more detail, just find me later. Maybe I'll tell you, maybe you won't. Depends on the question. I don't know. So, uh, let's go over the IR process. Our good friends from SANS, our six steps. Um, for this talk, I'll be

focusing on identification, containment, and eradication. Yeah. So, first, let's go over the threat overview. So, I was watching a baseball game. I was looking at Twitter. At the time, it was Twitter. I refuse to call it by the letter because I think that's weird. But someone was tweeting out about this um ChaosDB vulnerability um by Whiz. Did anybody know about that or hear about that one? You guys have to combat that one at all? Maybe ring a bell here in a second. So, you know, first thing like I hear about this, well, it deals with Azure. I'm like, well, I know we run Azure wherever I was working at the time. Um, I know we use

Cosmos DB. I was like, please don't let the third one be right, but we do use Jumper Notebooks. And this is an artist redemption of me. What happened was I think it was a Thursday night. I was watching my uh baseball team and they were losing unfortunately. So that took about an hour looking at the tweet thinking about it thinking do we have this cloud architecture are we using these uh databases and are we using the Jumper notebooks at this point it's in the cloud doesn't matter if it's prod or dev right test it's it's still an attack target. So all right here's an overview of the vulnerability. So, the good makers of Whiz found out um

our friends at Microsoft made a change because of the shared security model that did something to optimize with Jupiter notebook. When they did that, it allowed for some uh unfestive things noted here. So, you got the remote account takeover and then cloud credential theft, which is always a lot of fun. Um and then exploitation became trivial. They went in through the Juniper notebook and then once they got that key, they're able to do lateral movement. Um, things like that. You can look it up, but I feel like the Wiz did a really good job explaining it. I guess they should since they found it. Um, I like this diagram. And of course, what's the impact? Well,

it's high risk to the business. Could lead to ransomware and maybe data loss, I guess, is a subset of ransomware. it. Um there's also some things that folks don't talk about like even they don't do anything this they could just gather metadata artifacts about your network about your cloud maybe sell it to IAB access broker maybe bring maybe enrich their doseier or dossier on you attack you later. Well, even when um folks don't get access or take data, they're going to or do something nefarious, they're going to get those adjectives about you and keep building that um script against you, right? So, little fun fact. I'm sure you folks know that. It's nice to repeat it. Um so, another

hour passed. So, bummer. So during that p time I contacted the on call for the um the people that own the cloud whatever that meant. Um but parathetically before that I got somebody on my team to start sifting through the logs to see if there was anything that was germanine to Azure Juniper Jupiter sorry an old network guy um I should say net screen a little joke um but u and then the other um notebook DB whatever we'll figure it out so they're looking through the sim looking for those artifacts right so went and called the uncalled person for the cloud went perfect I got the right person at the first. No, I didn't. So, um, one

thing we did really good did really well. Um, make sure the line uh, we had owners tied to every resource in the cloud. So, at least there was some human I could call and that human would tell me that's not me. Go call the other person, which I was what I expect. Um, so I eventually got to the manager on call for data engineering and had to talk to the database guys. Um, this is a very long time. You'll see it take about just under four hours to get to the end of the rainbow to talk, make a decision. Um, and we found out that the person that we needed to regenerate the key, that's

what you have to do because the key was stolen. um the best person to do that was not available to the next day. So we have a decision to make. Do we do it now with the second best person or do we wait? And so I'm on the other side channel talking to my team, spoiler alert. Um do you have anything definitive saying that we need to do this now? Yes or no? There wasn't anything definitive, but there wasn't anything that said not. So that's all I can say. So without some definitive proof, we decide to wait till the next day. Uh we did some other things to increase our cyber defense mechanisms with extra

alerting and things of that nature. So for more alerting side monitoring, right? So move into a containment eradication. So we schedule a meeting for the next day, get everybody on the phone. your typical party line solve a problem. I'm sure we've all been there. And but now the risk is low. And I feel like I could stand behind my decision in terms of, hey, why did you wait at 8 hours? Why'd you do this? Like I I feel like I had good answers for all of them, which I did. Um I was informing upper management what was going on. So I knew they were going to ask me, "What do you recommend?" That's why I'm here, right?

I was like I wasn't too happy about waiting but it was lower risk the business. So So after the key was regenerated and all that fun stuff I we still treated like an incident. We did all the postmortm things themes but could never get any definitive proof the thing was any um nefar nefariously done or anything like that. So six hour passes. All right let's see how AI's turn. So, since we're in Microsoft land, I thought it'd be good to work with AI or co-pilot. That was how by the way, if you don't remember that from 2001. All right. So, let's try to get taking baby steps here because I'm assuming that that the co-pilot doesn't have any idea

about what I'm talking about. So, I want to make sure it knows before I start asking it questions. Um, don't worry. This is the first time you'll see me type out recorded because I suck at typing. Rest I just cut and paste. So, so it's describing the vulnerability obviously at the end you'll see um what links references it has. So, I mean to me it all looks legit like it got the right thing on the first try. Now it knows about it in that session. So [Music] I also used this co-pilot version. Um this was on my own co-pilot dime like about a year ago. So I'm sure it has some changes. I did six months personal

testing which um co-pilot and the other six months I've been on chat TV. Sometimes I call it chatty G when it's we're best friends but remember we're like doing really well when it's not it goes back to its surname I don't know what so let's what's the IR equitation details I kind of need to understand how the attack vector works like to get to a point though if I've got Azure environment and I have an enterprise um co-pilot license why can't I just hop in the co-pilot and have it interrogate my cloud and find it quicker Right. Um I'm sure that's coming. I think that could be done now. Just depends which level of access you have,

right? But at the moment, I'm assuming that I don't. Right. Um like, well, wow, this is not good. All the same stuff we just talked about. Fun like that. So, oh wow, that was so nice to answer it. Okay, that's actually really important. So, let's move the next guy. And let's see what is the impact in response. Yeah, it takes me a while to to find the thing. I like looking at two screens at the same time. Make sure I paste it in.

So let's ask the impact. I mean that's really at the end of the day is there an impact to the business um to the operations to anything that's going to be catastrophic right for keeping things in motion. So gives me an impact like huh unauthorized access vulnerability by spread this isn't good gives me the mitigation steps and still I'm looking at the references it's dealings which is important right you don't want to have any hallucinations while you're basing your entire approach on how to potentially solve this incident and maybe it got maybe I mistyped the word and it thinks it's something else that'll be unfestive so so I knew Uh, we had a decent amount of

knowledge about Azure, but I knew there was something called KQL. Don't ask me to say that fast. Five times or five times fast, but I knew this could go through it. It's like SQLish. Um, I was like, well, this is a way that we can run a query to find all the instances, right? So, we know what we're up against. Maybe it's just one. Maybe it's a thousand. I don't know. Obviously, one's enough, right? But at the end of the day, they're going to be like, I'm going to get asked, how many um vulnerable instances did we have of XYZ? Did you get them all? Are you sure you got them all? Can you double check?

Well, questions I would ask the person that the tables returned, right? So, usually there's a typical bit more forceful question in there. So, um so it gives me that which is good. I'm like, uh let's try to find this for everything. This is very basic. Looking for a query that looks for those uh three instances of those characteristics of Jupiter notebook, chaos DB and um if they're together, right? So hopefully you folks can see that. Okay, [Music] it tells me what it's going to do. I can copy it out. It looks okayish. I don't know, but it looks better than what I could come up with in how many seconds they did that. You know, I would have to

look up what KQL is. How do I write it? Even if I wrote it or someone on my team wrote it, I'm going to have to get somebody who's more of a DBAish person to take a sniff at it, right? So, uh, let's say I can't get a hold of anybody or I'm too impatient. That usually the deal. Yeah, I want to test this before I use it. Oh, that seems like a novel idea. So, because first I want to be asked, did you test it before you went through and eradicated all our Cosmos DB? Cosmos Cosmos DB. I'm like, no, I didn't. Right. I just got a pay raise. I'm leaving. Right. Um, I was supposed to be

hitting the go button instead of pontificating. So,

>> what's that? >> How long >> the real incident? Yeah. >> Um about four years later. So, you know, we were like we're like everybody else in our cloud journey. We had a different platform and then one day somebody came in like, "Well, we're not using anymore. We're going to use Azure." Everyone's like, "Okay, great." Um and how I found out that we had Cosmod and Juniper notebooks running. I just happened to be walking past one of the architects going to the bathroom and I heard Juper notebook. That's how I that's how I knew. Not the most savvy way to keep an inventory of your cloud assets, but or instances or whatever. But I would also

argue back that time there wasn't a lot of CSPM and CA this and C this and cloud this and cloud that, right? So, not that it's an excuse. It's not. I mean, it is, but they're not going to buy it, right? So, um let's see. Did I just run this? What am I doing now? I went through this once. Oh, okay. Um, I'm going to test this. Yeah. Oh, how do I test it if I don't have the instance? Oh, yeah. Oh, I didn't even know this thing exists. Existed. The custo export. Oh, that sounds good. Um, ADX. I didn't know that existed either. I can test this before I get make sure nothing at

least is going to blow up, right? That it's going to get me faster to running the query to find out what my exposure is, right? So I'm like, well, okay, that worked out pretty well. I mean, I could have Googled that, but at least it takes my code or its code and like this is how you do it. This is where you run it, right? So, so what were the results with that? Um, so understanding the vulnerability and the attack path. Okay. Um, I gained about 45 minutes. It was about under two hours of work, but we got it done in less than an hour. So, I would say about 45 minutes. These times are estimates. You have to

remember that was like four to six years ago. But I know exactly what happened. I know what time the baseball game was, what inning, and I know what time I went to bed. I know what time I got up. I know what time I tried to get my Diet Coke and the place was closed. And then, which was true, and because we couldn't get on the call to eat. So, I like my Diet Coke in the morning. Understanding the scope and impact, all that fun stuff took about 60 minutes. So if you look at those two two items together, um regained about I would say three to four hours, say four hours solid of time

back, right? I could do a lot of stuff by myself. I didn't have to call as many people. I could do some testing. Um so and implementing the fix. Yeah, I did co-pilot. No, I didn't. I would never do that. At least not now. But uh maybe when things are better and things are tested and we love our Azure instance with co-pilot co-mingled maybe we can use aenic AI to do that and we've tested it. I would still do a ticket. All right. So for the next one this is the unplanned or planned one. So this is launching um DevSack ops for an enterprise at a place I worked was very developer heavy and friendly. Um, I have

a whole talk on that, but I'm not doing I did it Ottawa Bides 2023. Anybody was there? No worries. I have a bunch of tips from that too. I put them in here, but I can go deeper in that if you want later. So, this was my um mandate. Make operations for IT and everybody and security and developers happy work together in synchrony. and I had uh goals for the three main parties and uh I had some engagement strategies results and we'll like go back in time and look at it through the lens of AI. So, so I think we can all agree that tech is most time easy part. It's getting the buy in, talking to people, getting them

to do what they want to do or what you want to do, what's in it for them and all that fun stuff. So now this isn't a tool. I mean I did use a tool as part of this but this was a lot of politicking u information sharing empowerment um took a lot of my political capital to get this one done so but I had some clear goals for the developers and security and IT ops you can see all that but I I noted the secure coding is code quality um that was a mantra I went back to all the time so I didn't want them to see like I'm just doing this for security you got to pull request. Maybe

your code um doesn't do parameterization correctly or it's vulnerable to SQL injection or whatever. Well, yeah, that's a security issue, but your code quality needs to be fixed, right? And the place I worked, the analytics company, there was always a focus on automation regardless of what your level, what your job, what you had to deliver. You could just get one remove one step every every time you do it. That's what they did. We kept a list ledger too of like what we could automate. A lot of times you can't do it because you don't have the time, the money, right? Well, the technology isn't there, but keeping a list, I know it sounds kind of trit because you never

know. I used to get some things back in the day called favorability, which was finance counting term for like we got extra money, who wants it? So, I would take it because I always had my stuff ready to go. They know I'm a cheap person, keep it PG, and make sure we spend the money correctly, right? So that's the adoption strategy I took. Um the last two points the education and matrix teams I can talk about that after here but I'm going to f for this talk I'm going to focus on that. So how do you get the buy in from the top right? If I get that I'm going to make things slightly easier. um

I work with them to get this process as part of uh enterprise coding and engineering's goals is to adopt dev secc ops. So once I got adopted from the or pushed down from the top it became their goals which gave them the the time to actually work on it. So this time let's use chat gbt and see what it can do. I'm gonna I'm sure it knows what dev sec ops is, but I'm going to make sure I think it knows right what I based on. I don't want it to uh be like uh it gives me some recipe for something. I don't know. So, [Music] yeah, I should probably make these a little bit shorter. There we go.

Um yeah so core principles automation collaboration monitoring shared responsibility key practices they mentioned SAS and DASC SCA I'll talk about SAS here shortly what the benefits are and like oh this all sounds really good sounds similar to what I'm doing it sounds pretty legit so let's move on let's see if I can read this one So when transitioning to an enterprise company to this new ways of working, what are the common resistance items I may encounter specifically with cyber security advocating for this new way of working for developers and software engineering? So I'm basically asking how does he help me with adoption? Right? So see what it says. It'd be funny if it said talk to Josh,

but it didn't. So it's like I'm Josh. I'm talking to you. Great question. Well, thank you. Um, culture resistance. Oh, shocker. Or resistance. Double shocker. Tech resistance. Yeah, I guess if people like their tools, whatever. Leadership and governance barriers. Yeah. Practical strategies. That looks good. Um, shift from policing to partnering. Huh, I did that. Develop centric approach. Yeah, I had that. Uh, executive alignment. Yep. Did that training enablement? I spent my own money on that. Well, I got money from finance to help with training because I didn't want them to be like, I don't know how to fix this stuff. Well, here's the training, right? And incremental adoption. Yeah, we did that. We did a lot of feedback loops, too.

This was like over a two-year period, too. So, like, that sounds pretty that sounds pretty good. So, let's see. Is it going? Oh yeah, I think it is maybe. Oh, there it is.

Yeah, I was trying to move the mouse over to the other screen so I could hit the go button. So, oh, uh, connect to the interest for them. A change will help them involve them in the process. Yeah, I did I made sure I did that with IT ops. They had, um, the people that ran it's called the platform team. They're ones that did the integration with the uh pipelines and things like that, Jira and all that. Um, make sure they had some say in the game because they were going to run the tool, but we're going to take artifacts out of it. It was a SAS or whatever. But they made sure that was hooked into the

pipeline correctly. So, I have to give them a little, not little, I gave them definitely time to help think out and help pick it because they're going to be the ones um supporting it. So, they better be on board with it. doesn't mean they had the final decision, but I need their input. So, yeah, start with the small wins and this key principle at the bottom. I think we all know that that's not a big epiphany, but it's good to hear it. It validates things, right? So, so went back to my Oops, that's the best slide. All right. I went back to my uh college days and thought about the philosophy class I took. I was like,

well, I think some of the great thinkers on the plank can help me forge my um argument or persuasion that um that I have a solid case going to talk to the VP or whatever. Um I don't have a problem talking to people obviously. Um I don't I don't I'll talk to anybody about anything at work. I'll talk to the CEO if I think I have to. Some some of you who are in leadership roles may find that a bit daunting, but um I use these prompts to help come up with ways to think about what they would say and have the counterargument built in. I kind of call it the pong approach. Um you have

the ball going thing going back and forth. So submit your idea, the prompt, and you play the role of the person you're trying to convince and have the two machines argue against each other, the two different screens, right? So then you can take that data um the outcomes from both put in a different session have it do like a heruristic heruristic combination so your points are across they're still being heard but at the end of the day you don't really care as long as you get what you want move your project forward without being a jerk that's the most important thing so I may be old I didn't take philosophy with those guys I didn't teach close to

All right. So this is the output from the first year. So RGA means revenue generating um application. So companies exist basically two reasons make money or provide a service or both. Right? So I started with the the two applications that made the most amount of money for the or I wanted their stuff connected into the SAS tool in the pipeline. I wanted their um experience with it. I was going to help build credibility that it was working. It wasn't breaking. And so when I went to other dev streams down the road, I was like, I don't have time for that. Well, the one that makes us 30% of our money did it. Go talk to

them. They have a good experience, you know, things like that. So, so we got to our goals um with July and November. Um, also had some pretty talented college kids I had access to and at the time we were pretty big on dashboarding. I'm sure everybody is still, but the the new CIO or whatever was having this enterprise dashboard. So each uh team or group had their own and they would all feed back into the roll up into the big enterprise one. So that's why we had a dashboard fronting the SAS tool. The SAS tool was good. Um I have to admit um it was a low ent entry point for us and it did a lot of

good for us. So but it was also beyond the metrics of saying which devstream was getting better or whatever. We also had um metrics in terms of you know the or the number of um defects going down or which devstream has the least amount which one has the most. You know is it easy for the developers to get to the dashboard and click and get to the SAS tool? Yeah, they just had to click it and it took them right to the error message they needed to do. They didn't know how to fix it. The person that committed the code, that person's name was in there. They could go talk to that person, right? I couldn't get that

person to talk to the person, but I figured at that point I've done everything for them. So, we also did we also did some surveys. 15 minutes, I got 20 more slides. We also did sentiment surveys. We're really big on that where I worked like you have the right tools to do your job. So, I sent my own out. Hey, what do how do you think DevSec Ops is going? good, bad, ugly. Um, I also had DevSec Ops office hours. People could drop in, ask questions, um, things like that. So, definitely wasn't from a lack of trying and getting it done. So, um, I we're going to look into what AI could have done in terms of the dashboard for SAS.

So, this was the original dashboard. I know it's live. It's blacked out. Sorry. Uh, it's Graphana. You can say that. So basically if you see anything that's green it was 90% or higher that means that devstream uh uh met the threshold and was in certain parameters the people are 25 or bad and yellow was like in the 80s um you could click on that on that button that would take you or the you could click on the the radial there it would take you to that specific dev screen and these are the top items that need to be fixed. So it would just take you right to the tool and it tell you exactly what you need to do. But what I

liked about it, it told exactly where people were on their journey, right? Um so I told I met with the developer leads all the time and uh VPs and stuff, not all the time, but when I needed to. I had some reoccurring meetings with um some of the people that were more strategic getting the tactical things done. The point of this is I kept on telling them this dashboard's coming. This dashboard's coming. three months, two months, one month, about three weeks. They're like, "Oh, this I go, you know, this one is gonna be seen by the new guy, right? You know that." Like, "Oh, yeah." So, I got on a call with them. I said, "Click that." I said, and

I took him there. I like, "These are the things you need to fix." Oh, this is pretty easy. It tells me what I need to do. I'm like, "Yes, thank you." Then I went home and screamed in my pillow. Now, it wasn't quite like that. People are busy. I get it. Just had to click two twice that would find their answer. That's fine. It's okay. Josh, right? Let's see what um um AI can do for this. So, that'll make sense once once I go through the um other slide. So, let's act like that is a button that can be clicked and it will give us what AI thinks. Well, next slide will tell you what that's going to be

about. So, so yeah, let's uh we got a lot of data coming out of SAS tools. I'm not going to say names, but sometimes SAS tools can generate a lot of noise. Sometimes it's valuable, sometimes it's not. How do you know? It's a lot to go through, right? Um so let's like well maybe um chatbt trained, right? What know it's looking for? See what it can do. Let's look for some low uh low hanging fruit. Let's also have it, you know, give it some um developer ready fixes. I call them more proposals. I don't want to call it dicks, but this was a sample prompt I came up with in terms of if I had if the chat

GBT had access to all the data that came out of the SAS tool and I wanted to mine it and then look for things to accelerate um making things more uh frictionless. I think that's a term uh for the developers be more self-s serve um and stuff like that. So, this is a pie in the sky prompt. I I I like it. It say everything I would want it to do. I like the fact that too it'll also put in their backlog. I'm sure we all heard like we'll do it in our backlog. It's on our backlog. Backlog never gets looked at. Anyway, so did the panic come true? Well, for the first one, for both I help with

efficiency and less friction for sure. I got about four to five hours back from the IR event. You know, again, this is a retrospective going back in time. I'll say my Delorean, right? Looking at see what's going on. Uh DSO means DevSec Ops. Um did it help with the project launch and maturing? I you know what it validated the approach at least on the right path. Um it would validate my ideas of how to gain uh upper management developers in org and hopefully less frustration for me. And then pro tip, um, when you're working with people you need to convince, you use that pong method of constructing a prompt. I think we all know about social engineering to

help. I'm using that term kind of tongue and cheek, but use that sprinkling in wisdom from your favorite um debaters. Then you want to bolster your um outcome of your ask, right? So, let's all face it. Uh most of the time people don't always see us as the happiest people to work with or they want to come work with us. So anytime you can do things in terms of making it easier for them and removing that moniker of the the big bad boogeyman security person. So it's helpful. Um the AI save us I don't know but saves some time saves less frustration. So that's good. So here are the key reference materials. Of course, replay the SANS list. Um, I'm

sure folks know this, but every cloud service provider has a well architected blueprints you can use to talk about your setup, your security, your compliance framework. Those are good to mention. Um, go through this. Uh, in case you didn't know, all OASP has some really good resources out there in terms of your AI journey. And I like the checklist. I'm a big fan of checklists. um and your resources and NIST. I'm sure we're all familiar with that. I like this one a lot. This will take you to some spreadsheet they have in a Google's doc that talks about the right language and it defines it. Let's get through that. Um you get this nice road map. Um the risk

management framework for AI that's really good. I do a lot of assessments for my clients whether I'm using NIST 853 or 800-37 all those things kind of run together but like how are we measuring it and NIST is always good because it's universal right so all right that is the end I appreciate your time thanks [Applause]