BSidesSF 2026 - State of (Absolute) AppSec (Panel)

Um, so my name is Nicolina. I'm your host for the day. We have two liaison, Fee and Sunny. We also have people on AV that are working hard for you. So at the end, we're going to acknowledge them as well. Um, let me introduce our speakers. Let's see. Um, at the podium who's going to be moderating and reading off all the questions is Ken Johnson. He's a CTO and co-founder of Dryrun Security. Give him a little wave. >> Hey. >> Hey. Nice. Nice. All right. Then we have Ken Johnson. Oh, sorry. Seth Law, right? His counterpart. Apparently, they do a podcast together. Apparently, yeah. So, we've got he's a principal consultant at Redpoint Security. Very nice. Round of

applause. All right. Kevin McDermott. There he is. All right. He's head of security at Superhuman. Kind of like that name. Pretty cool, right? And then we have >> You can do it. >> Okay. Ash >> Aa. >> Aa Singhal. and she is our only female which I love and she is the director at I love that uh she's the director of security at one >> Netflix yeah one is not enough >> no it's not enough but I love that she's here so thank you for attending and she also used to help volunteer and run bside so that's pretty amazing yeah all right we have a final last but not least hold on let me get my notes

Okay, we have Clint Gibler who is now head of security research at SERB. Yeah, very good. All right, welcome everybody. We're excited to get started. Uh we are going to be discussing the state of Absolute Absse. All right, I'm going to have you take it away, my friend. >> All right, and we're live with an in-person edition of Absolute ABSSE. I'm Ken Johnson joined by my co-host, Seth Law. Seth, say hi. >> Hey everyone, welcome back to another session, I guess, right? Not another episode this time around. We are really, really excited to have our panelists here this today. Um, it's been a minute since we did one of these, right? I think it was last year here. Um, or

whatever, it doesn't matter, right? Um, we are going to be talking about AI and APSSEAC because apparently the whole industry has changed and there is an earthquake that's happening underneath all of our feets. >> All of our feet. >> Yeah, >> whatever it is. All right. >> Feet seas. But >> Clint, Kevin, Asta, thank you so much for being here. We've got uh a few questions that we have curated. Um, and we will be taking questions from the audience. Um, so some of those will be coming in. If you have a question, make sure to submit it so we can add it to our list. I'm not sure exactly what we'll get to. Um, if you've ever

listened to the podcast, you know, it always goes off the rails because we just don't Yeah, that's just the way it is. >> And please place your uh questions on Slido. >> Thank you. >> That'll be the best way. I like I really can't see anything besides a big bright light and I feel like I'm >> Yeah, literally. >> Is there a number? >> Is there a number for the slidoh? There's a QR code in your program. I'll find out what page right now. Okay. >> Cool. All right. All right. So, you want to start with that first question, Ken? >> Yes. So, and we'll have AA kick this off, but uh so the first question is,

what's one thing the industry is underestimating when it comes to AI and application security right now? Yeah, I think uh one thing that maybe we are underestimating as an industry is just like the fundamental shift in how software will be built and written and then like the ways in which we have to change our assumptions around application security processes, points of context, uh the controls that we're going to need. Um and I think defensive security teams just need to think a lot more deeply about how they're going to do application security in an AI native world. That's totally fair, Kevin. >> I think it's going to be the impact that non-deterministic software has. Most of

our systems and AppSac are built around very well-ontrolled, well understood workflows where user input goes through a certain level of processing. You get an expected output out the other side. We're pretty good at that as an industry. paradigm doesn't exist as much in a lot of our products anymore. A lot of our products have AI in them and we don't really know or control what they're going to do at any given time. I don't know that the industry has started adapting to that yet. Everyone understands AI is non-deterministic, but I don't think we've really changed how we operate a whole lot as a result. I think we need to start. >> Yeah. No, I have a lot of opinions on

that, so I'm I'm just not weighing in. Uh, so not to eat into your time, but yeah, totally. That makes a lot of sense. I'm I'm also curious. I'll probably add at the end something about economics to see if if you all have any thoughts there, but Clint, I'm curious. Do you have anything to weigh in on this? >> Yeah, I think there's a couple of things that have already been broadly discussed, but I think they're still underestimated. Uh, one in terms of the amount of code that's going to be written and how quickly. I think that's going to like I think people expect it's going to speed up in a lot but I think

it's still underestimated. Um and then I think the uh rate or the speed and ease of exploitation of new vulnerabilities. Um so there's been a a bunch of blogs and talks about um uh both finding vulnerabilities in the first place as well as taking existing uh CVEs and like auto exploiting them uh based on just publicly available information. Um, and then I think we're also underestimating our ability to build as builders um, and build uh, secure by default libraries and paved road and things like that. So I think we can be more ambitious as apps professionals than we were in the past. >> We can totally build a lot faster. You're absolutely right. Like that is

one thing it's it's really giving us is empowering uh, the ability to just yeah ship things we wanted uh, that uh, you normally would have to go through a vendor for. The economics I think are interesting too. I don't know are you all seeing any disruption economically in the workforce with budgets with anything like that related to AI at all yet? >> Who who wants to go? >> So I I mean I can go as a as a consultant, right? Like we're definitely seeing a shift in the number of contracts that are coming in, right? And what they're related to. Um we're having to justify our apps, I guess, right? like I don't know how else to say it of

how we are actually performing testing and code review um because everyone has access to slash security code review right like so you know um Ken and I started teaching you know secure code review manual secure code review I mean Clint was in that course what eight years ago whatever however long ago that was but a lot of that we can automate nowadays right and like that is a single prompt that we can give to agents and have that actually run things and we've got dry run and other places and so it feels like economically um I don't know where things go, right? Like and that's that's where my underestimation is there's this there's this idea that apps

is going to stay the same and there's still a lot of people that are trying to build their careers around that and I I don't think that's the case. I think we're underestimating what that ground shift is going to do to application security in general. >> Yeah. And you know, I think like also on your front since you're in the consulting space like engagements, you know, obviously engagements should should go a lot quicker now as well, right? Um yeah, just with with that capability. >> So I'm going to move on to the next question which is around Sorry, pulling this up. Uh how do you Yeah. Okay. So I'm going to go backwards. Clint, we're

going to start with you first. So how do you reconcile modern ABSSEAC practices with legacy expectations for SAS DAS like regular, you know, traditional tooling? uh and auto audit requirements. Um is the current ABSSEACEC tooling approach working? Are teams uh lean it sounded like you were talking about teams kind of leaning into building building things more? Uh but yeah um so the real question I guess let's just summarize this very succinctly. Uh what's changing in the tooling space for apps and um are people leaning it sounds like they are but are they leaning more into building versus uh buying and and things of that nature? >> Uh sure. Yeah. I think um a couple of

things. One, oh maybe the mic is on. Maybe it's not. I was going to say I think the mic is not on. Um let's see. I think um like I like to go back to first principles in some ways in terms of um like what ultimately are we trying to do? We're trying to meaningfully uh reduce risk for the organization. And I think like what is the right tool for the job? So there are some things that you're like this is a type of issue I want to find consistently and reliably. So um uh what is the best mechanism to do that right like probably LLMs are not going to replace uh code formatterers or

llinters or things like that cuz it just costwise doesn't make sense. Um, so if there's like specific patterns, you know, are bad, I think using sort of traditional approaches like static analysis makes sense for that, like compilers still work and still do how they do. Um uh but yeah, I think instead of uh focusing on like finding lots of issues like how can we um solve classes of issues with secure defaults um uh yeah not everyone has a Netflix team but uh in terms of so many good SWES uh but now security professionals can build um and things that were just previously intractable uh are more tractable now in terms of I want visibility into what

every developer team is building in terms of um like what is um high risk um so in uh looking at various um design documents uh Google Docs like where people are writing about what they're going to build like oh cool hey this seems high-risk versus uh not um or similar with code changes you can have like a very handwavy view into like oh this seems like it's introducing code that maybe a a human security person might want to look at um even if it's not necessarily a vulnerability it's like um just exposed risk um oh this is an unauthenticated route or it's uh changing the off flow or things like that that are not like a strict pattern

you would look for but are more like a vibe type thing. Um yeah, so I think sorry I feel like I'm giving a a long ramble but I think in short there's um some classes of things such as broad visibility into what's happening in your ecosystem that is tractable in a way that uh had to be humanpowered in the past. >> Yeah. And like just to touch on that real quick, something that you said was really interesting is the design aspect of like here's what you here's what you've written as your specification, what you're going to build, and then did you actually like I I I think that's maybe what you're saying is like did you

adhere to that kind of thing? It's not like a strict check, but it's sort of like a Yeah, you you you have some risk just from the architectural review here that we're we're uncovering just from Yeah. >> Yeah. Maybe uh I'll pass on the focus in two seconds, but just um yeah, I like what you're saying just to make it more concrete. It's like, okay, the um developer uh or PM is like writing uh here's a thing that we want to build and then an LLM looks at it and it's like, oh, it looks like you're uh touching this sensitive database and you're uh having users log in. So here's an authentication library we recommend you

use and also make sure to use this other mitigating control. Um and it like auto puts that into the requirements for it. And then you can also to your point um like once that code has been built you can also do like a post-process maybe uh with like a a cloud code or similar coding agent and be like okay cool you said you were going to do the thing did you actually do the thing um based on actually looking at the code um versus uh just assuming they built it or uh having a human go review it in the future but sort of like mapping uh both security requirements into the design document as well as seeing is it

actually enforced in code as well. Um, I think you can do pretty well today even just with existing models. >> Yeah. Yeah, definitely. And and that really does push things like security reviews to to, you know, shift left, but just really to be quicker and to actually keep up. So, that's really interesting. Yeah. I'm curious, uh, Kevin, any thoughts on how tooling can change, how processes change, and if we're Yeah. building, buying, etc. >> Yeah, I'll, uh, I'll tackle the audit piece a little bit, uh, since it's a different direction than Clint went. And so disclaimer so that my lawyers and company don't get pissed off at me. I am not an auditor. So talk to auditors.

But we at Superhuman, we overhauled our GRC controls over the course of the last year, our internal controls to make them align more to what we are actually doing. And we did talk to auditors. We brought our got some consulting hours from them. We brought them in. We ran through all of our new control processes with them to make sure that when we hit audit, we will be fine. But uh we did that uh to ensure that bringing AI into our compliance process wasn't problematic and spent a whole bunch of time doing all the compliance things, making sure we had everything documented properly, that we had the appropriate audit trails. But ultimately what matters is

that you're adhering to the intent and making sure that the the outcomes are the same and like Clint said the outcomes that we are striving for reducing risk in our applications that hasn't changed. The way in which we go about it has but uh we can reconcile them. we can move our processes forward and still adhere to our compliance obligations without necessarily having to sacrifice that as long as you dot your eyes and cross your tees because they're still auditors. >> Interesting. Interesting. So I guess a followup I I have there is you know like for when it come especially when it comes to compliance um there you know and we all deal with like security

vendor questionnaires and things of that nature where it's like you know if you have data here on like your policies why can't those things kind of be automated. There's a lot of ways you can automate that. So I'm curious there are there any like concrete like examples you can throw out of ways that you're sort of getting through things easier just automating. Yeah. >> Yeah. So the the place where we have had the most success in appsac at least is automating our dast triage. We do almost no human triage of dast issues anymore. It is all through a claude code skill uh that kicks off well an agent uh cla agent I guess uh with a skill to do all

of our dast triage for us. Um yeah my appsac team doesn't really look at dast output anymore. It's auto triaged and then tickets appear. >> I'm hearing that's the way it's going. So, it's it's a pretty interesting time to be alive. Austin, what about you all? >> Yeah, first of all, my team is amazing. So, thank you, Clint. Like, I will take um um no, I think, you know, we as an industry talked about DevSec Ops for like a whole decade. And I think we're at a point where there is going to be sort of like a new agentic SDLC and like we have to think about what are all the security controls that fit into

different parts of that from like design to implementation to post uh build validation and things like that and I think we're going to have the opportunity to do that in like very very scalable ways with uh agentic technologies ourselves. So hopefully we live in a world where we don't even have to cut tickets and we're doing a lot of that stuff um uh kind of in line and I think that's kind of where we're going as an industry. >> Yeah. So my question on that specifically is around this this model of development that we've had for the last 30 40 years right does this make sense anymore right this goes back to this question is is appsseack in its

current form just dead um because we've bu we've built tools I'll let you answer here shortly but we we have built tools that are based around a waterfall methodology and yes we've shoved them into agile and other things but you know you talk about DAST and where we do the the the testing we talk about you know threat modeling and where that's at and I mean even the terminology we use shift left is oh you know we're thinking about a waterfall SDLC and this isn't the way things that are happening the amount of code that's being generated it does that make sense anymore so >> I think it is like the most exciting time in application security ever

because this is like being there when access was invented right like we're discovering new classes of vulnerabilities that didn't exist before that don't have solutions and we're inventing solutions as the tech is evolving. So apps is not dead. It's actually like you you can kind of go back to a lot of those core competencies and apply them to new problem spaces. >> Okay. >> And then the way we do it will look different. >> Yeah. Yeah. I mean it's it has to change, right? There's been posts out there about like new um new I I don't even know like a AI DLC, right? like whatever like life cycle that's it's a little bit different than the SDLC the

traditional SDLC but we've always built it we've always bolted security on after the fact right and I I feel like I'm with you that there is this this opportunity for us to get in front of that or get embedded in it in a way that we've never been before so sorry Kevin did you have >> ABSAC is dead >> is dead sweet >> long live APSC long ABSAC there we go >> um no I I agree. I think the the reasons why we have this practice are just as important, if not more important now. How we do that work is going to change. I agree with that 100%. And I think it will I think there's already a pretty

big divide in industry between team security teams that operate as engineering functions and security teams that operate more as a compliance function. I think that gap is going to get bigger. As teams who are able to adapt to AI are able to move faster, they're going to be significantly more able to reduce risk than the teams who are still in a traditional apps model, running tools, consuming output, generating tickets. I think um if you want to keep up with the speed that engineering development is going to move at, you need to be able to adapt. >> Yep. Couldn't have said it better myself. I mean, you're you're absolutely right. Like a few points here. For one,

I mean, I don't know how if Seth and I have talked about this publicly, but there was a period of time like maybe four years ago or something where, you know, it's like we can only go on the podcast and talk about breaches so many times. I can only talk about variations of excess so many times, right? And frankly, it was getting a little like tedious and a little like, you know, >> stale. >> Yeah. Stale rep repetition and all that. Uh, but with this new sort of emergence of AI writing code and how it's changing the SDLC into an ADLC or whatever we're going to call it, it's fun again. It is fun again, I have to say. Um, and then

sorry, there was something else. Well, we'll move on. What's the next question? There was something else I wanted to rant about. >> Speaking of questions, real quick. >> That's okay. Um, so for our audience, Slido, if you want to ask your if you want to ask your own questions, you can log into Slido Sli.com and then you want to put in the code besides SF2026. So that's B SiSf2026 and we're in the 12. >> Thank you. Thank you. Yeah, actually there's Hold on. I thought Sorry, kick back in. I'm old. Um, yeah. So, no, I was just going to uh u basically what you're saying about this this divide in ABSSE, I we've been talking about it,

Seth and I, quite a bit and there is like this um to stay relevant and to stay in this, I think you have to really really drive home on deep domain expertise. I think that that's like if you're just doing what you said, which is running scanners, consuming output, throwing it over to the wall, and that's your ABSSEAC program, you're going to struggle in about I mean, if you know, it depends on where you work, but it could be a struggle now or it could be a struggle in two years, but get yourself up to speed because like, yeah, this this is happening and and we've had friends that privately, you know, they're like kind of, well, we our our

developers are not going to do that. Like, they're, you know, arteasonally handcrafted coders and all that stuff. And then wmp w six months go by and they're like oh no actually you know VPG or CTO or whoever said everybody's got to go AI so or uh use AI to build so um it's going to affect everybody and I just yeah I totally agree with that. So yeah >> sorry what were you going to say? >> No I wanted to give Clint an opportunity to to talk to Absse is dead long live app ABSE. >> Yeah. Um, yeah, I have thoughts on that, but also I uh can I maybe just like drop some spicy takes and and see if people

have mic. >> Yeah. Yeah, I'll I'll take it off and drop it. Um uh yeah, I just um adjacent predictions. Um I think in the next 6 to 12 months, I predict Bug Bounty will stop being a good entry platform for security people because either the top bounty hunters or security product companies will just scoop up all lowanging bugs. So junior people will be able to find zero non-duplicate bugs in like either today to like 6 to 12 months. So very soon uh I predict maybe secure coding training for developers may go away. I think we can meaningfully change the OAS top 10. Um so if you look uh so it replace uh there's like new

ones every couple of years and if you look back historically it's like well yeah uh so those are mostly the same ones. Maybe we gave them different names, but like injection and stuff like that has still been in the top two or three for like 20 years. Um, but I think we can actually meaningfully change it uh now. So we won't have another version that's like what would you say you do here? >> Yeah, exactly. >> Well, and that that does lead into one of those other questions that we wanted to ask specifically about the OS top 10. Everyone's seen the new iteration, correct? Right. Like we are at >> Has anyone not seen the new iteration?

you know, 2025, uh, there's a new list of the OS top 10. It is more categorical than it has been in the past. There are reasons for that. Um, you can talk to Tanya or Brian or whoever about it. Um, it is pretty interesting. But the question here is what apps risks change and are there new categories we expect to emerge? So, to that point, is there anything specific that you would call out that needs to be in there that we're not seeing yet? Yeah, I mean it's tough to say because there's also the whole separate like uh AI OASP top 10. Um occasion uh so I feel like a lot of those things um are

already captured there. >> Okay. Um but I think that the just standard SQL injection cross-ite scripting and things like that I think we can tractably solve that in the next few years hopefully by coding models get better uh automatically rewriting code gets better we have more and better secure by default libraries and we just mass rewrite things to make it more secure um in a way that is tough for legacy code but I think is tractable now. Oh, I mean this goes back to the whole idea of there is the one true framework that is secure, right? Um and like so I while in principle I agree, I also feel like there is there is so much

legacy code that's out there that doesn't get touched and what we're seeing from models isn't this panacea. There's there's way too many sharp edges. Um and that the code that's being generated is so it may solve some variance of XSS and SQL injection but it is also introducing new variants and it's introducing new holes I guess new vulnerabilities that we don't necessarily have a handle on yet. Um so like >> you mean new vulnerability classes or the same vulnerability classes manifesting in a new way? >> The same vulnerability classes manifesting in the same in in a different way. I mean you're aren't you seeing that at dry run all the time like I mean we train the models I mean if

part of this is a training problem so it it should get better but we're seeing SQL injection pop up. I'm seeing a resurgence of SQL injection that I you know simple forms that I haven't seen in the last 10 years because of AI generated code. Yeah, I actually had like skills and an anthropic superpower installed and I and like when we upgraded to and I've talked about this on the podcast, but when we went from 4.5 to 4.6 with Opus, it just completely ignored it. And all of a sudden, I'm getting PRs where it's like literally SQL injection, basic SQL injection and these PRs written by AI assisted editor or coders and I'm like, how are we back

to this? Like I don't I don't understand. Yeah, >> I hear you that like it is kind of like some of that basic stuff is emerging right now and generally I do think that there are some trends around generated code and the security of that. I just think that both from the standpoint of like our ability to like make those things better quickly at the model level as well as this like the industry motivation to solve those problems um as kind of like core competencies that they're going to offer to enterprises. I think those factors will make it uh kind of go away pretty quickly >> improve. >> So, but with that and going back to

Clint's point about um training, right? So, secure code training goes away for developers because the models solve a lot of that and they will like they will fix that. We're not actually hand artisally crafting code anymore. How do you how do you envision someone new getting into application security? So I I'm not sure who wants to answer that first. I mean you've all have teams. I know I'm coming at it from a third party consulting perspective. This is on my mind. We have juniors and yeah like I have a huge question in my mind. How do we teach them about SQL injection? Right? Like we have the traditional patterns but nowadays they have cloud code. They have other tools

available to them. Where do I actually point them so they can get the expertise they need so they will be the principles in 20 years? Right? >> You can try to start. >> Go for it. Yeah, >> that is a big question. Um, I think the the actual path into application security isn't really changing a whole lot because I think most people who are in application security started their career as developers. And I think that's not the only way into application security, but I do think it is the main way into application security. That is the path I took. Maybe I am biased, but I think that's where you start. And how you create code today is very different and

it it looks very different. You're using different tools but fundamentally you learn application security and you get exposure to it by building applications. I don't think that will change. It is can you adapt to what engineering looks like today. Can you effectively utilize tools that have a very different learning curve and just behave very differently from what we are used to? But if you can make it as a software engineer, I think you're still going to be be able to make it as an application security engineer. Um, but you'll need to learn those modern applica or uh yeah, modern appdev fundamentals which look more like managing AI agents than they do actually handwriting code at

this point. Okay, Austa. >> Yeah, I I think it's a good question like for us as an industry generally to think about talent growth um and being intentional about it. Um I think it's going to be a combination of not like I agree with the point about like working as an engineer in like those modern systems but also we have to think about ways to teach like sort of the architecture and systems thinking and like all of those types of concepts cuz I do think that those are the things that are not changing. It's like the way you understand prompt injection is still kind of like the thing about okay untrusted data is code and like what is

the hard thing about it and like how are you going to secure that. So I think we'll have to figure out as an industry like how are we teaching folks that type of sort of systems thinking and architecture. >> Okay. >> Yeah, I would definitely agree with the systems and architecture and big picture thinking. a lot of the uh low-level execution with a good plan uh coding agents can execute very well. But um there's also like a a taste part of it where it's like well like actually the put this functionality here this should actually be a re reusable function versus reimplementing it twice here and here. Um so I think that uh is useful.

Um, and maybe just quick uh about the I think foundation models will get better at writing uh secure code. But what I meant by eliminating the vulnerability classes is also um like having uh skills deployed to every engineer that's like here's secure coding guidelines that are specific to our company for our tech stack. Uh after you generate the code, maybe there's a hook that like autoscans that code with some carefully crafted prompts. So it'll still make mistakes, but then there's like a postprocessing step with separate context that's like not biased by what it just wrote. that's then going to um uh flag those issues. Um so it's like a series of things in an ecosystem, not just like oh it never

makes a mistake. Okay. >> Similar to how we code today where we have human code review and there's multiple steps and there's tools and other things. >> Yeah. So it's it's the DLC whatever right like the new water file. I have got a good question from the audience I wanted to >> we actually have a lot lot of good questions but one of these ones I wanted to get to was um if you were building an appsec program from scratch right now what would you be thinking about first >> I'll start with AA you spot >> yeah put you on the spot give you no time to think I'm sorry >> uh no no I think uh the thing to think

about there would really be hiring security engineers that like to build in kind of like our um infrastructure, but also folks that are really really curious and eager to learn because you know the like I I think Clint you had this quote in on his keynote about the you know the only limit to learning right now is your own creativity and agency and I think that's so true that like I think folks that are curious have the security fundamentals and really want to learn and grow are really going to succeed. That's Yeah. I mean, if you have a core really really good core group and you've got the ability to build, it makes sense that

you I like that you're focusing on people. I just I really like that. Yeah. Um Kevin, >> I'll take I agree. I'll take a slightly different approach. Uh what does the business actually need? Because not all businesses need the exact same apps and the exact same apps team. You may be in a heavily regulated industry. you should start by being able to ensure that you can meet the table stakes requirements of that industry. Um, that's kind of a boring answer, but the the business needs are exceptionally important when you are building from scratch because you you arrive in a state that you're probably not starting day one of the company. So, there's a lot of tech debt, there's a lot of just

stuff going on and it's really easy to get derailed into things that are not important. So focusing on that business need and eliminating that risk first. Um that hasn't changed. That would have been the answer I think 10 years ago. I think it's still the answer today. >> Yeah. And to be fair, we don't exist if the business doesn't exist. So it does make sense to align well with that. Yeah. Clint, uh what's your take on this? >> Yeah. Uh I would agree with Austa. I think having some uh like cracked AI builders would be a great place to start. Um I would also say that um people talk about how things are changing but one thing that I believe or

at least that I've observed is that I think a lot of things that used to be important and valuable are still important but they're even more important now. So really just like engineering fundamentals in terms of um do you have good review processes? Can you um like patch quickly? Can you get things in production quickly? Can you uh effectively roll back if things break? um uh like uh do you have good visibility into your environment? Um do you know who's building what and when and like what they're working on? Um uh sort of Asta and others from the Netflix team had a number of talks over the years about how the security team should become best buds with platform

engineering teams and I think that's uh even more uh true today where if you can hook into the SDLC and then again maybe put these security gates or uh secure coding skills or different things um that those are very high leverage points. Um so yeah with an appsac team I'd like okay cool how can we make the engineering team even more effective uh but also insert some sort of secure by defaults along the way. Um so again I think aa and Patrick gave a talk at apps at Cali in like 2018 about this uh and I think it's still true >> eight years later still ringing true >> but yeah Austin do you um I don't want

to >> no no I think I think generally that principle uh is probably going to be true for like long-term success. I think one of the things that is sort of unique in the AI world right now is like the experimentation and like the new thing comes in and everybody wants to try it immediately. And I think there is sort of like that short-term risk management that we have to think about while we're building the long-term path. I actually think that that's probably where um we'll have to focus like a little bit more on like that short-term uh risk as well. >> Yeah. Open claw and prod. What could go wrong? >> Were there any questions off this list

that you wanted to highlight? >> There's a lot that I would like to dig dig into here. We're gonna take these today. We can just go another hour, right? That's keep going. >> Yeah. >> Five minutes. Five minutes is what we're being told. >> Um >> we can take it Yeah. after. >> Yeah. Yeah. We we will be available. We'll make ourselves available after and we can we can answer some of these questions. Um there's other avenues that you can use to Yeah. to talk to us basically. Um, yeah. Do you wanna We've got five minutes left. >> Go, go, go. >> Seven. >> Okay, sweet. Um, we we did want to kind of put a bow on

things. So, yeah, like now I'm looking over all these questions and >> just take that away. >> Seven minutes later. >> Yes. >> Seven minutes later and I'm still like scrolling. Um let's let's just close out with what's one thing the industry is doing today around AI and ABSSEAC that won't age well? What are we doing wrong? >> Uh don't retrofit your current processes to AI data world. >> Okay, very similar but bolting on AI security without actually building appropriate systems for it. Uh there's a lot of AI augmentation going on right now as we figure out how to use it, what it's good for, that kind of thing. Um we need to get past that pretty quick and build

actual systems that work as opposed to having the same paradigm but just AI it. >> Okay. >> Uh I think failing to embrace evals and benchmarking. Um, so, uh, I spend a terminal amount of time reading things on the internet and one thing that I've noticed is everybody has like, oh, here's my security code review skill and there's like 10,000 of them and it's like, which one is better? I don't know, guess, I guess, just like run them. Um, so I think security teams that take a engineering and sort of data sciency mindset in terms of like let's build a security thing, but actually measure if it's good in practice and be like thorough about it. I can tell you as

someone who obviously is in this space, yes, like you absolutely have to have a lot of rigor around evaluation. Things go off the rails quite a bit and you're always doing basically drift detection, context infusion. Yeah, you're always fixing things basically. There's a million ways you can fix things, but you're right, having evaluations is probably the number one thing to start with if you're going to build around AI for sure. >> Can I add one more? >> Oh, please do. Yeah, please do. No, we're done. I think like I will channel Anna from her keynote earlier. I do think that it's like we as an industry are focusing too much on the bad stuff, but like

there is so much opportunity >> and I think we have to focus on that. >> A lot of opportunity. Does anybody want to make any predictions for the year? >> Any predictions at all? >> Yeah. What What are we talking about next year when we do this? >> I'll give you my prediction. My prediction is on this note of what you just said, Clint, I've said this publicly. I think we're going to see a lot of blog posts of oo this worked great at first and then we had to do a lot of work and to your point about evals I think that'll be a lesson learned a hard lesson learned if people don't start with that kind of rigor

>> if we look at the past year MCP was adopted cloud code was released cloud agent SDK was released we moved from agents to skills I don't think we'll be talking about any of the same things a year from now. I think it's going to be completely different. >> That's a fair one. And it feels like we're at the winds of the frontier models for what we're going to be talking about. >> Yeah. Next year. And that that could be next week, not just not next year, right? Uh yeah, I feel like everyone in their brother has a blog post about here's some vans we found with uh AI and um I think partly the models are good,

partly open source code is just not good. Um or rather the defect density is higher than you would imagine. >> Um so I think both of those are simultaneously true. So I would expect to see a lot more bugs being found. Yeah, someone did make a comment about in the questions about like uh why are we something like why are we pretending for 30 years we haven't been putting slop out there fully. So yeah. >> So I mean with that do you feel like we are building sorry I'm going on but you know are we building more CVE we're building I mean the amount of open source code has exploded right there's >> we'll be able to do remediation at scale

this year. >> Uhhuh. >> Yeah. >> Optimism. >> Optimism. Optimism. Aa is here for the optimism. Right. >> I think we'll we'll start it this year but maybe in two to three it'll be good. >> Yeah >> that's fair. >> I I think it's going to be a rough road. Right. Um I I I don't think we have a good idea of what the plan is to actually secure any of this or to even comprehend the scale that's coming. Um and so that's what's going to drive us this next year is basically can we stay on top of it and can we come up with a plan that makes sense and actually apply some rigor to that and data engineering

principles in order to I mean keep our jobs number one but also actually secure these environments. Wasn't that amazing? >> Yeah, I think we're being told to get I do want to say real quick, thank you to our panelists for agreeing to do this. It took I mean there was actually a lot of coordination for this and it's you know thank you. We really we really appreciate it. So it means a lot. >> Thank you. >> Yeah. Thanks for having >> 150%. Give them a round of applause. Our eloquent and amazing panelists. Thank you so much.