Vendor Hacking: How to Make Your Tools Suck Less

all right well we're gonna roll with the slides anyway hopefully they work uh welcome to vendor hacking how to get your tools to suck less i'm mandy piazza this is james nixon nice to meet you guys uh we're gonna talk a little bit about who we are and why we are here in the first place so uh i'm the chief evangelist for fee llc i'm also yes one of the organizers here for b-sides nova i'm a former combat mp i got into threat intelligence a few years ago i am the first employee with my company which is really exciting because james is the first employee of his company my wife meredith as you saw earlier on

the keynote is also employee of our company so she likes to remind me just because i'm the first employee doesn't make me the number one employee um so we james and i talked about this a couple years ago or last year at besides nova the the idea of right making our tools work better um i am on the user side i usually get very angry when i start talking to developers and they're like oh do you want to do this kind of a join or this kind of a joint i'm like i want a big stupid green button that says go right where's the easy button for a user um and james is that guy so james go

ahead and give your background yeah so uh where i'm at today is the first time i've taken on the responsibility of leading customer success previously i spent a lot of time being one of those software developers that would ask those well how do you want the join to work behind the big stupid button uh and transitioned into doing a lot of data science enterprise architecture so first to join the co-founders over here and it's now my job to make sure that when people like andy say i need this this is awesome this sucks how are we going to get it into our roadmap or how to train them to you know use the product the way it was

designed to to meet solve the problem meet their needs yeah so uh this is not a vendor pitch uh you see my my corporate logo at the bottom i am representing the the greatest company in northern virginia called fia if you're looking for a job and we do a lot of government contracting uh usually u.s security clearances are required uh do you hear us up more than happy to to talk about career opportunities uh james is representing analyst one uh besides nova is a no vendor pitch thing so i will say if you're looking at the threat intelligence platform go check out their platform it's one of the best he can't say that here but i

can um it is one of my favorites but uh yeah it's it's a really cool opportunity right to to talk about i think really something important in anywhere in infosec but really here in northern virginia too because we've got our own problems with acquisition being government or government contractors we really want to talk about like you know vendors are always like you should check out our solution i'm like your solutions more problems than my problem was man so let's figure out how to tear down some of those walls um so we know we as users we suck right we're gonna say suck a lot we're trying to try to be fun and if you are offended you're probably in

the wrong talk um you know we're really good at as users we're really good at naming problems if you come and say hey what are your requirements to fix this thing i'm going to tell you everything that sucks about my current job right um go get a cup of coffee or a glass of whiskey and be like this sucks this sucks this sucks this sucks this sucks what we're really really bad at is reminding or telling in the requirements what we really like too so we go buy a new solution they buy the things that fix all the things that suck and they forget or don't know to keep all the things that we do like

um and so we want to talk about that that's my role here from a user perspective is how can we be better users and get better feedback to the acquisition teams if we have those or to the vendor directly right um how to kind of be a better steward of our money and our technology so we can actually get those that work for us um we also have this problem too where we buy a tool right and like you go out and you buy a prius and then you want to put a lift kit on it like that doesn't help with the prius right we want to break the tools that we just bought one of the reasons i actually really

respect um analyst one when i first met them didn't know anything about them they came and pitched to agency that i was working at um while they were pitching we were asking if they could do something and uh scott i see in the chat um scott one of the one of the founders of analyst one was like no we can't do that and they're like well what if we you know pay you more money and he's like no that would break our tool and i just really appreciated that from a vendor's perspective of like you know they didn't chase the dollar bill and it's like that is fundamentally different than what you're buying right like

and he was straight up like if that's not what you need go buy a different tool i'm not going to try to strap everything onto the top of this thing and weigh down my my car uh because you have additional requirements that don't really make sense um and i think that that is the type of communication we're talking about today hey i want you know i wanted an ids they can look at network traffic and also pull forensics logs that's probably not something you're going to do maybe with an edr but it's probably going to do both things poorly right instead get customized tools for the things you need is kind of the conversation right and we usually

suck too with our acquisition cycles on the user side we talk about requirements like 18 months later we start testing tools our capabilities our requirements have completely changed because the management may have changed um or we you know got a sweet excel spreadsheet working already by then um and so we complete completely throw a new list of requirements at the vendor and the vendor's like what the hell this is not what we talked about 18 months ago and vendors can really suck um right seriously are they every even sometimes you talk to them they're like what is your use case for this tool right i don't use ids like you don't know what an ids does

you're selling me an ids you should know 90 of the use cases of what an ids does right um so but vendors you know they make most of the tools that we need um and so we need to be able to work with them to get the functionality that we need and it shouldn't be a fight right we shouldn't be arguing with our vendors um or uh what i commonly see is most solutions are most companies they buy a tool they have like a weekly meeting or monthly meeting with their vendor the people will complain for an entire month and they'll get on the call and just stare at each other hey you got anything anything we can

work on james is like i would really like work to do can i make it better and i'm just like not talking to him i'm not i'm not telling him what's wrong right so ultimately it comes down to you know how do we as an analyst get our words nicely over to james right so he can go fix our problems and go talk to the developer team or the architects or engineers or whatever it is um any any comments on the pain points james the uh there there is a reality that vendors do need to understand that does lead to some of those those awkward conversations i've had them recently with a couple customers where

it's you know you got to go deeper on their use cases and needs and do that assessment of okay you're you should be able to do that with our product or like andy said you should you should not this or this integration's missing the that reality of that you got you as the users are living the day-to-day of it you know we get to work with the product sometimes even less than our users because they might be in it for their full eight hours a day or their or whatever their shift is so understanding making sure that i try to always come at that perspective which does need that what andy stock can do we have to be able to

have this conversation on what's succeeding like like you got out earlier we need to know what's working so that we can build on it make it even faster or better quality and time matters a lot and then also talk about what's what's blockers and that that's going to go into when we talk later about prioritization to understand where yes okay you can make a big list of what hurts and what's what's painful but we got to put that in order to make sure that the highest value of improvement gets into your hands in a timely way so that the overall performance of the job the tools meant to do gets there fast right yeah and i think it's important

too right um a little admin note about our slides we have a lot of bullets with with stuff that we're not necessarily going to talk to we're going to try to tell some stories throughout the talk so slides may not align to what we're not per se but these are the real key takeaways um but we do a little story time uh in a couple of sections so if you see me jumping over a slide you know feel free to stop make sure i'm not like in the middle of talking when you take a screenshot but um just so you know like that just want to put that admin out we we have some good

talking points in here that we may not actually talk about right um so what we're gonna talk about we're gonna talk about the pre-procurement challenges right how do we identify actually what we need what we don't need um assessing all of these flash pitches right you're walking around the the conference and they're like scanning your badge and like we could shoot down malware from 10 000 feet above the sky and you're like that's weird that doesn't make sense or you know you get the vendor pitch and they're screaming we do it on the wire 15 times in a 10 minute demo um waiting through all that and trying to figure out like what do i actually need does

this vendor do what it says it does right if their website says that they use a certain model right if they say oh we use attack cool that was two years ago attack is still cool they should still be using it but it's probably not a sales pitch anymore um everyone should be using attack at this point right so going through what do you mean by attack oh you just add it as a tag like everybody else does cool or can i actually see an attack graph right and actually understand what your requirements are if i want to build a thread actor playbook right in a threat intelligence platform can i say you know show me this playbook

and show me all the attack techniques that were mapped to that reporting or do i just get a search to search a tag and see on that those are the same requirement two completely different views can come out of that requirement right um and we're going to talk about post procurement navigation communicating our needs for future bugs fixes or new requirements um identifying any rail red lines right fail fast don't get into a relationship that's going to last you three to five years in a tool that nobody uses if it's a year after you you sign paperwork and the tool is not installed and it's not your fault and it's the vendor's fault or it's your fault you may not be

ready for that tool as an environment right if it's 18 months after you purchased it and your users aren't using it daily or monthly or whatever the intended schedule was you may have a problem and you may need to be talking to your users and you're probably pissing away money i get to say that here right yeah pissing away money um if your users aren't using it daily right you buy a fancy car and park it in your parking lot or parking in your car driveway you don't take it out other than covid reasons you know you're kind of wasting money on a car right so don't spend a bunch of money every year on a product that your people refuse to

use learn how to fail fast make sure your contracts are set up for annual reviews or there's there's clear cut guidance and how you can get out of those contracts um think smart and start getting your procurement cycle for the next tool before the other one breaks right so if you do throw a vendor off site because they don't meet your needs don't go a year without without another tool that really sucks for your users too so um being a little bit better in your post procurement cycles to figure out when do i need to go buy something different or have i outgrown this tool even those kinds of things and most importantly let's talk about managing a

relationship so that you can in the future be on a talk with a vendor and not talking crap with them you can consider them your friend and go have five guys with them afterwards right the the attack example you use is a great one because it's talking about that spectrum of concerns that when you go into those procurement conversations like the travel and solutions architect is going to talk to it's like yeah of course we we hit this line item you know it's somewhere in the process we're going to talk through here you've got to make sure that people have a chance to look at your people have a chance to look at things and give an assessment

where this i can use it or i can't use it or i can use it partially and that's that managing relations post acquisition that can help people move in so that you don't hit that many months later people aren't using it and get that feedback to me and to any all your vendors of if it was like this it would be more efficient or more useful and then finding a way to see if there's some kind of partnership between yourself and the vendor to to make it better for your users today while the product evaluates is that in our wheelhouse to do is that an integration to do how do we you know solve the problem with that

honest conversation yeah before i move on full credit to joe sloick for this amazing meme i spent like two hours the other day going through his twitter page just finding the perfect meme from him because he's always putting out great infosec ones but there that's a great one um so procurement requirements and and conversations right so communicating the needs right from a user perspective um we need to identify what's broken we need to identify what's a new capability um oftentimes depending on where you work um you know your management or your director team or even your if even if your managers and directors are squared away but your acquisition team doesn't do the work they may not understand the capability

that you're asking for right and so you say do you do miter attack and they go out and they buy a tool that does miter attack and you're like that's completely the wrong tool but we've mapped to the requirement right so actually spending time like showing them how you do your job um is really really fruitful acquisition team asking them if you can give them a demo of what you do before you have the tool um is really really important if you're an acquisition person on here uh congrats i don't know how you ended up at a hacker conference but really excited for you to be here go sit down with your with your analysts

right if you're a manager or a director go sit down with your analysts uh when i was at uh i'll say when i was at u.s cert in kik at the time uh priest is a days uh director felker was the director of it okay he came in and he did like his nine of his teams and stuff and then after that he was like i want to sit down and work with analysts and understand how they do their job and so they came over and he's like i don't need a briefing i already know what your branch does he's like andy show me how to be an analyst here and i opened up my email showed him how

i brought stuff in from a stakeholder submitting me thread intel i opened i remember 21 tabs on my on uh firefox i said now i'm gonna bring these indicators through firefox at all these tabs and do some open source enrichment and he looked at me and he's like i haven't gotten you a tool for that i'm like well you've been here for like 90 days and he's like no that's a failure at my level i need to get that fixed right and so he understood um at his level his decision making and purchasing and stuff and what that impact was at the analyst level because he actually sat down and saw the work was the first time i've ever seen

an executive do that it's absolutely amazing i've seen one other sis out in my career so far do that as well if you're a manager or director or an executive and you haven't seen how the sausage is made in your factory i highly recommend you go spend some time because within a year he'd gone out in the funding so i could stop opening 21 tabs i could open one go into a platform and actually do this enrichment automatically but that's what's important right is understanding really what the work is and what those requirements are and really being able to then categorize them as this is a gap this is a new capability you may see a vendor demo and just be

like i never even thought we needed to do that but we need to do that right so identifying if it's a new capability an enhancement right does it make your job faster does it enable you and integrate uh you know getting a soar platform being able to integrate your different tools right so you can get an alert on a host you can go look for network events too that kind of thing um most importantly users we have to get involved we cannot ignore the acquisition in invites not show up to the table and then get handed the tool and complain that the tool isn't what we wanted if you're not going joining the table and you're invited to the table that's

your fault for getting the wrong tool it's something yeah that so that that time and people thing matters a lot we've gone through a few evaluations as a vendor we make an investment with really competent people really competent analysts and like three days five days before the evaluation they get pulled for for some reason it's their their contractors and they're at their hours or the there's a critical mission they need to engage with and then like we got to scramble and teach new people about not just the product but also the scope of the evaluation what the rules of the evaluation are what they're allowed to do and it it hit the people that get

engaged in like some in my opinion the people getting engaged kind of defining the need and shaping the need have to be allocated at an organizational level that time to stay invested in this decision all the way through the end and help do the evaluation and uh the the um the challenge that you face there is it can't be bonus time it can't be like the above and beyond their average daily work making that finding the right people who can do that investment do it in fish efficiently and help you make that organizational decision is is worthwhile because andy story is dead on you you you as an executive buyer as a team leader or as an end user all have a different

perspective of what's going on and if the only way the user's perspective is rolled up through some document where everyone puts together their well categorized needs and it just rolls up to some eventual evaluation by people disconnected from what those words mean when it was authored you're gonna run a higher probability of making a decision to buy something that looks like it actually meets the end user's need but but doesn't actually because they got disconnected from the workflow and the last thing i'll say here on this slide is the we've been engaged in conversations where people are frequently telling us like exactly how the product should should work it should work in this way or that way and

that's actually a harder conversation to have on most situations what we need to understand is the the pain that you're experiencing because the product does or doesn't do something and we need to oh or if it's a hey we want to go next level here's something we want to do next we need to understand this the solution the goal that would come into play like having a conversation with a customer this week on products they're producing on behalf of their organization outside of our tool to broader people kind of crossing directorates i had to take time my team had to take time to understand the scope of that product before we could give them answers on how

their intelligence product before we could give them answers on how our vendor produced product could provide the data in an efficient way to go and get stuff done that if they had just told us like on a simple line item we want we want data like this i guarantee you the first couple iterations of us helping them take their step forward would have been wrong that investment in conversation with the end users makes the whole process of evaluation before acquisition and after acquisition infinitely more efficient and more accurate absolutely and as i said we have a lot more material in here than we're probably actually going to talk through today i plan on doing a write-up for my

medium blog and i'll get uh james and put it on as well so anything we don't cover today we will absolutely get you a plan uh it will get planned in the next month or two to get a blog out um and get more details in how we approach these things um because there are some actual real world takeaway examples in here um but we we find that teaching through stories and talking is a lot better than uh just going yes and then do this and then do this so um i think we kind of covered this a little bit but uh already but you know justifying the time blocking out the time so i'll give

you an example it's important here um when i was at a previous organization we went to to buy a tool that acquisition team did a great job they did a down select they picked um you know three vendors to come in and actually pitch us they pitched us for about two hours two three hours and i say us there was about 10 of us analysts from different teams in the room there were some engineers that had to deploy the tool in the room and we all got to see the pitch we all got to talk to the vendors directly and then um we uh they deployed the tool into our demo environment or into a

training lab and we got to play with each one of the tool for about a week each um each tool got evaluated we had spreadsheets i'm going to show you a generic example it's not from that thing um of how we evaluated every tool we went through our requirements and was like can it upload a document like yes it can right that's a very binary question but all of the tools should be able to do that so we actually did a gradient like did it meet expectations did it exceed expectations did it do a really awesome job kind of a thing let's get that that company more points because they did something intuitively um and so we car our management carved out

our time for us to be there basically for a month going into this conference room and nerding out together and white board and talking about our problems with the tools i mean we were able to get the tool that we needed um so time is very important outcomes right can this can the tool meet the subjective or not um is it repeatable or does it suck if you're constantly going back to the vendor or the vendor documents to ask how to do something the tool's not intuitive enough you know it should be replacing what you do already one of the number one things that drives me crazy when i talk to analysts and they have a new tool in

front of them they're like we need training on this tool vendors if you're if you constantly have to provide training for your tool your tool sucks tools should be intuitive you hand me a hammer i know what to do with that open a can of beans obviously no i know what to do with a hammer right if you hand me a platform and i'm constantly going i need training on this for like the basic stuff your tool sucks it's not in go back through ui school right what is the old meme uh user interface is like a joke if it needs to be explained it's a bad one um yeah you shouldn't have to provide

training every month to the users just to do basic stuff if they want advanced training that's cool but if you're constantly providing training got a helicopter going over sorry uh your tool ui probably sucks right um quality right does it make your job easier as you're about integrate with what you want it right splunky s all that good stuff or elk or any of your other seams not not vendor specific that was a inside joke um and then other you know considerations um grouping these things you know or these core requirements um or be cool but they're not really required ultimately hopefully not just cost is justifying what you purchase but the end user experience

with the tool i don't care if you save half a million dollars on a tool if your users don't use it you didn't save half a million you're wasting whatever you're paying for it right so sometimes it's better to go in a little cost if your users are actually using it the um the the the key thing on on this idea of prioritization and justifying the needs across how time gets saved you get good outcomes good quality that it thinking about it in that way helps then kind of create your list and helps encourage your users where it's okay just ask them the questions what do you need to save you time what do you need for a better outcome

what do you need for a better quality you're probably going to get a better list and andy's point about priority where we're going now and gradient where it's not just a yes no answer it's a it's some kind of scaled score you're going to get higher and higher likelihood of making a good selection in the pre-procurement phase the more flexibility you have to let the evaluators really rank things and not just bear it down to yes no if you if you keep things that just a yes no do i meet the requirement and the requirements are prioritized you're going to get into the continued cycle of lowest price technically acceptable winning the day and i think we all know

how that goes sometimes um it's it it i'm trying to read the questions too and we're gonna try to work the questions into the talk um it takes a lot right to get this stuff right um the the question is about you know what about dealing with third-party risk or you know getting it through all these other teams you know that's ultimately a management issue um if you don't know your own organization right go have a cup of coffee with those guys in a postcode world or gals send them virtual coffee if you can but get to know what the actual process is i've seen it in bought a tool right and they like had

their their technical requirements list like you know minimum servers blah blah blah went to deploy it and then the itops team was like we don't support that os at all and when we were delayed months for deploying it because they didn't even talk to the folks that had to deploy the dam tool and so they had to go through a whole change control process to get a new os introduced into our environment and stuff that's all pre-procurement management stuff and that's what your managers are doing i get it you're managing analysts or engineers or whatever but part of your job is called management is to manage those relationships inside of your organization so that not just your user requirements

but the tools requirements can actually be deployed i love you guys but that is a management failure if you can't get a tool deployed after you purchased it and that's a manager for the folks that were buying the tool that's a manager for the itops team those are those folks need to be talking um and if they're not i don't know what else to do for you right um uh so here's a super generic i threw this together uh this was not the official one that when i referenced doing this uh previously with a client but this is kind of a gradient scale of how you can assess those things right actually get a spreadsheet and list out

all of your requirements and not just like one word requirements actually write some scenarios around them and then go through and rate these things in the tools if you can get hands-on in the tools rate those tools and actually go through it accounts or one license or whatever to be able to test things like go out and do those things and actually spend the time as the user of the engineer um get the hands on and actually rate each one of these things um yeah you know rating it for timeliness is really really important i know everybody wants to go to automation i'm just going to jump right over that one if you can automate obviously that's why you're

buying a tool is to do things faster um right is the human in the loop human out of the loop human is the loop that kind of a thing but really the timeliness of the things too uh is it making my life faster is it making the process i do faster or is it taking more time it's not very intuitive going down on the quality side impacts time if it's not very intuitive right i've used a platform where i had to do the same process process twice into two different parts of the ui just to do the thing that i wanted it to do one time i know i'm not explaining it very well but i'm trying

not to call out a vendor for their shitty upload process but i had to go through one side do a thing go through another side where the other vendor i just did it as one seamless process and it was very intuitive right so those kind of things give it a quality assessment um you know can a hammer breakout tile sure but it sucks maybe if you did a vendor assessment you would have gotten a jackhammer instead right assess your tools it's not just about a yes or no question give it these gradients at the end you can rack and stack them score them and if you have more than one person scoring it you should at least like three or

four people you can run the averages and now you can talk about is the cost worth the the highest score maybe we need to go with the second place score because it's going to save money or we're actually going to get it deployed because it's has the os to support it but at least now you've got more than just a yes or no james go ahead and then variety of users there on that averaging point matters a lot because if you're in a and which depends on the tool right if you have a tool that does one thing for one niche user group and you have a segmented budget and line item that's going to hit to

make that user group better you're really focused but when you get a product that's going to span organizations which fits with the kind of use case andy's talking about the evaluation he's speaking to it's going to span teams and you've got now like multiple people in play representing every kind of person in the long run that's going to be there and giving them this kind of flexibility to score is going to then get you at that better solution that that fits the better need of all the teams and then you know each team should be able to go in this and be able to help weight not just the outcome of the scores but also the

priority at dandy's point if if budget is constrained or budget is significantly important um in the grand scheme of user facing functions weighting those up in the scoring process is actually going to help you come to that kind of hard decision at the end of can we should we make the decision to allocate more funding for one solution or are we going to be lacking a better way to say it good enough with with this with another the ideal solution is the ideal place is the best solution is also the lowest cost but that's not always what you're going to find yeah um is it mergloss maybe might butcher that point too right i don't have here on

but you should give the weight for the has to do should have more weight right those are your core requirements should have more weight in your scoring system than the nice to haves right i would also like it to not just shoot down malware from the sky but maybe now we're in the ocean right those are nice to have um those should obviously be way differently if you got you know all threes across the board for the nice to haves but it doesn't do a the core required functions you probably shouldn't buy that tool either right maybe though you've identified a different set of requirements for a different tool uh you know talking from my my own

experience again not naming vendors a lot of folks are going down the route of their threat intelligence platform also being their short platform those are two completely different functions and capabilities again that's like having a system that can do host analysis and network analysis it's going to do both of them all right but it's not going to be good at either of them probably not in 2021 maybe with some ai quantum physics in the future but uh you know right now the reality is maybe i needed a sore platform more than i needed a threat intelligence platform and using a lookup table in my uh scene was good enough for my tip for now or if typic

maybe i shouldn't break that core function by working too much on the soar capability right um you got to consider those things those are related those are absolutely related but they're not necessarily should be the same tool i don't my experience i haven't seen any of them do both of them great i see them both do it mediocre maybe mediocre is good enough you're saving money i don't know um but consider that you may go through all these requirements and man these nice to haves were spoken in on this platform i really really want this but we didn't meet any of our requirements okay maybe you need to purchase two things and you got to go back to your

board and go get more money or you may make the decision to do a hard left and buy a different tool for a different capability but don't fool yourself into thinking that you're going to get you know two things for one kind of a thing hopefully that makes sense uh here's a generic example for trying to replace excel just kidding nobody would ever do that it's the greatest tool ever made um but we wanted to be as vendor non-specific as possible and figured microsoft could take the ding to their stock and nobody will notice after that solarwinds incident but whatever um so we figured we'd go after excel here right how would you rate you know a tool to

replace excel you know everyone's like i don't know why you're using microsoft products man the google products are great like no they're not right my slides all looked a little weird converting from powerpoint over to google slides but whatever um but you can assess that right should we we're a small company should i be paying for microsoft licenses or google licenses for a corporation good enough right um and start doing those evaluation those cost benefits savings are important sure but also usability right if i'm going to have some janky spreadsheet presented for my financials or my you know sales pitch as a vendor and i go to a client and i'm like oh yeah i'm using like the free software

that was installed on linux let me just show you this vim document that i wrote up this contract in they're probably gonna be like these guys aren't professional enough for us to yeah leave her office man i can't figure out how to use that still um i'm a bad hacker apparently but if you go pitch right in something and the document looks like crap maybe it wasn't worth saving the money i don't know um but here how to do the scores you got the combined scores over here um i do have the criticalities right and so you can get in some excel formulas and i'll write those into the blog but you can do like

account if criticality is this um you know sum up the the scores over here and combine scores so you can see how it meets your must haves your should haves and like to haves get separate scores in your total dashboard at the end and you can say this one has all my musts or most of my musks but a lot of shoulds or i would like to have so maybe it's worth reevaluating my requirements right um hopefully that makes sense i'm not going to beat that up too much but james got any thoughts on that yeah so the on screen the there's a word in chat called weight that would match with what merlot's put forward as criticality

you see here just for keeping things simple and then there's like those there's gradient with variety of scores walking across the top again he's talking to a key thing for anybody working procurement or anybody setting this up moving into that pm level is uh in my opinion you don't want to be in that place where just simply the highest score is guaranteed to be the right choice because there might be other factors or you might need to revisit scoring when you see things coming out because maybe we really do um need to reassess do we need to meet all the musts or do we need to re-look at things and we put something in a

it must do these things does it actually need to like being able to make sure you have some flexibility when you get to that point post evaluation to make sure that you agree with the scoring it is worthwhile or doing scoring test runs like maybe do some scoring test runs on your current product before you like if you're going to do it through a re-evaluation do some scores on current products to make sure your process works ahead of time so that at the end the ability to make a good decision is is clearer yeah absolutely of course i gotta apologize to excel with one of my world's favorite favorite speeds i don't know actually who

provided this but if you follow me on twitter you've seen me publish this uh meme pretty much anytime anybody mentions excel because it literally is the best solution or at least the second best solution to every problem yeah i went the wrong way nope i went i'm just gonna mess everything up today okay there we go go you've done your assessment now you do your fancy math and you select the right one with the highest score right now we already talked about that um it's not just the one with the highest score you know cost is involved can you deploy it is also involved right big thing though is cost shouldn't just be the only

factor that's really what we want to drive home is usability really should be a factor requirements met really should be a factor right i can save a lot of money by using chinese product but intellectual property theft is also expensive so cost isn't the only factor sorry china i'm not kidding um pre-procurement to-do list right obviously we talked about requirements you should probably write those down um right make a list list out all your needs and wants don't just think about the things that suck think about the things that you do really really like in your tools absolutely write down the things that you like that is the number one mistake i see users make we

forget to say the things that we need to have and that we love to have and we want to see stay there um which is why it's so important to define these things more by achieving a job function than it is about a line item like you support this integration or you support this feature or you have uh this kind of graph or this kind of display if you can if you can write your requirements that will eventually be evaluated against the way people do work and the responsibilities they need to fill speaking again to the needs and not like i want blue versus green level detail you're going to get to an evaluation point to give people that flexibility to

assess can i do my job with this tool because in the end we talk about tools we talk about automation i love where our product does automation i love where it works with others that do automation that's awesome but in the end it's people making decisions that define what those automation should be or people making decisions in products to go and do things to scale out those decisions over time so the closer you can get your requirements to representing the evaluating against a person's job the better write-up you're going to get from the vendors at the outset to prove that they do or don't do or don't meet that job need and then the better evaluation when you get um

when you get to that the the testing the new solutions at the bottom you're going to be able to do yeah absolutely um free accounts like i said earlier a lot of a lot of companies and vendors will give you a free account if they have an online portal um take advantage of those right if you're a manager and you're coordinating those that user access give your users some left or right limit sorry it's a military range term tell your your users what they're allowed to do and not allowed to do maybe in the free version it's cloud and it's multi-tenant maybe don't upload personal or internal documents but also give them expectations that you

want them to spend an hour a day in the platform or give you feedback on the platform i don't know how many times i've been given free accounts they're like hey we gave you a free account then i just get it as an email and i'll literally i'll get it from the vendor go over to the manager and go hey i just got is this phishing what is this and they're like oh yeah we paid for those accounts or we're getting free accounts go test it and i'm like what do you want me to test it for right give some expectations of what your users are doing um have a conversation with them and and

then hold them accountable at the end of the week or a couple day trial period whatever that is have a conversation about how the tool did um and consider your requirements list and maybe do your scoring there from the free version understanding the free version might be a little bit different than the capabilities and what you could do with the in-house version because you can't put your intellectual property or your log data out into a cloud multi-tenant cloud right that might be a little bit scary depending on the type of platform but set expectations for your users clearly communicate what you want them to do and not want them to do in the free version

and have a post conversation about it too i think we already kind of talked about this approach anyway with the you know down select play with the last three um do not limit the vendor and your organization's connections i absolutely know you know you don't want them pitching to everybody i get that um but one of the things i always see is there's always this bottleneck where they're like if you have any questions for the vendor you got to send them all to this one guy and this other team that nobody likes um or just there's always that single point of failure right uh for for vendor relationships especially if you have contractors on site you don't want the contractors

talking directly to the vendors about your requirements because they don't represent your organization well they're usually your users so set up that that time where your users don't care the color of their badge have the opportunity to talk to the vendors and not just the sales person if you're talking to a vendor and they're only showing up their sales person go talk to a different vendor they should be bringing engineers users analysts people who know how to use the platform on the front end right and have that conversation if every a question they're like i gotta take it back to my team ask for a new rep give them the second chance to send you

a better rep otherwise go to a new vendor that's a very bad sign from my perspective as a user oh people are just going to call me in the middle of this thing sorry um probably about my car warranty or somebody suing me for all these bad things i'm saying about vendors one of the two um good james so the and from the from the vendor side like that communication is helpful because it it helps break down the barriers because if the only translation between what you need and what the vendor will do or be evaluated against is a document the probability of mistranslation and misunderstanding is high um the other factors here is when your

priorities are constantly in flux the ability to do a healthy evaluation in a decision is is hard you should have the right where if they should be able to change but there really needs to be some operational reason behind it that's causing that not different peoples or different opinions or i hate to say it just because you did a different manager who has a different set of opinions that shouldn't necessarily change your organizational perspective on what's important or the weight the criticality um and then regarding inefficiencies in the relationship and in the products themselves like over communication is is my advice i had actually a conversation this week where i had a very adamant end user on

how things today are were creating a struggle for them and it was it was hard to hear just admit it but i now have more data that i can act on to give them a means to succeed today and make sure that our dev team is understands the actual impact to their daily operations which makes the dev team come back and be like oh well let's not do exactly what they say let's do this that's going to actually meet their need and make everything more efficient because they're talking to me which is where this this customer this user is talking to me about their need how is their job affected you know the worst case scenario for the

vendors what you got there at the bottom is where the road map's always changing and we have no effective means of measure against it pre or post procurement or we'd never hear the problem you know because if we don't know we can't give advice or provide solution uh all credit to my buddy chris crawford i hope you're dialing in buddy the i call him the professor that's not actually his handle but i just made it up for him uh he said this i asked him to take a look at the slides and ask for feedback and one of the things he said is without requirements is it the wrong tool or the right one the world so i was like dude i got to

make a meme out of that so all credit to chris that was a good one he let us use it but it truly truly is the problem right like requirements list if they exist at all they're usually like a handful of bullets and they usually suck and that's how you end up with the wrong tool uh there was a question earlier like what do you do when you get get handed the wrong tool that's where it's a conversation with your managers about failing fast right getting training maybe it is the right tool and you just don't have the right training um but getting it getting that conversation with them as early as possible so that

at that one year renewal or two-year renewal that they're already pre-procurement cycle again and they're looking at another tool and you're part of the conversation in the future if it happens multiple times just change organizations because your management sucks no i'm just kidding um that's true though and go work if you go yeah yeah thanks come work for me um post procurement management right so you selected a tool you bought the right one right analyst one no you bought the right tool um excel and you want to integrate it in your environment and you know you're starting to get trained up on it you're starting to find maybe problems with it or it's not working the way you

thought or you had a good idea ferry you're driving in stuck in traffic like i do and you're like oh cool you know what that thing we should really do it should use lasers to shoot down malware um how do we handle those post-procurement management really uh those relationships right so colorless green or blue or red um i mean red would be evil right according to star wars uh obviously uh blue green or orange because those are the fiat colors um this this is where we started to get to merlot's question earlier about what do you do if you're given a product well internally as a as a receiver as a user of a product

the first step is understanding the people involved both on your side with you as a user you've got to log in and do a thing with the thing and the people in your organization and then working with potentially your managers or upper-level decisions or acquisition to understand who is on the vendor side which is where andy's going to go now to kind of outline this but this is where the first step is understanding people their relationships you can start figuring out how to get help or make better decisions yeah notice the key players is more than just the technical lead or the government leader management lead or whoever it is and the sales rep or customer sale success rate

um there are a lot of key players involved in this conversation um if they're if your organization is j with the vendor right whether it's a contractor issue right or uh they just you know don't like their users you know holier than that type of managers whatever um those are definitely some serious communication problems there should be more than one rep from both sides of the relationship involved if you're constantly meeting with a customer success rep and they're never bringing vendor engineers to the meeting or architects of the meeting um there may be a problem if you're doing your training and it's the customer success rep doing the training and then he's the only person in the room or he or she is the

only person in the room there may be a problem um there's maybe some red sign red flag signs right that you have a problem with your vendor or your own organization like i said um there should be multiple folks involved and get them involved as early as possible you have you know pre-post procurement you bought the thing but you haven't deployed it you want to get your engineering team on the user end engage with the cli with the vendors engineering leads right or the installation team they shouldn't be talking to a sales person anymore they should be talking to the person who can tell them what commands to run to get the damn thing installed right um

and hopefully you can make those connections um so you want engineers or developer types if you have a sore team get your get your sore lead and their their super smart store engineer involved early as possible so you can get those integrations rolling um you know integrations don't have to take a year after the tools deploy to get them integrated unless you're going for like ips mode and start blocking stuff yeah you may turn those along later and evaluate the tool a little bit more but otherwise you should be able to you know start getting that thing rolling um some reds red flags again uh you start deploying this thing and it takes over a year to deploy it in your

environment that's either a problem with your architecture or you know you got to fly around globally to install these things like sensors and certain physical aspects or you have some political problems internally you got to figure out and it's not the vendor's fault the number one problem i have i've seen it so many so many times where we put on the vendor that is never ever the vendor's fault we don't communicate with them properly we didn't give them their requirements properly we've got the wrong pocs talking to them or we don't as users touch it um and we're like oh that's a stupid tool i'm not gonna use it anyway and we just put all this shame on

the vendor like oh they it's their fault their fault they haven't done anything and we've done nothing to build the relationship and put our right people at the table um always do your best to put the right people to the table if you're not the smee but you're the one we go into the meeting make sure this me is invited and have them sit at the table with you right or at the outer conference room chairs if you're like in dod space where they got all the generals at the table but all the really smees are out in the outer ring bring those people to the outer ring right let them talk and we put users here aren't key players

yeah they're they're they get their own slide because in the end if the people using the thing the product aren't logging in and using it at the frequency like any reference earlier that's the core fundamental problem they have to be involved with the opinions on is it working today what needs to change can we update the sops empowered to update the sops one of the things we've actually seen operationally is users want to use it but then the key players on the previous slide haven't done the due diligence to actually give them permission they're like hey my job would be so much better over here but the procedure that i'm contractually obligated to or that i must follow says

i have to go do it in this way so give empowering them in all ways to press forward on your investment in tools is is critical so that their efficiency is there and that their ability to use the thing you've invested in the think positive right to not always be about the suck be about the success if you've bought a tool that works those other people on the other slides that are internal to the customer are the ones that have to get all those barriers out of the way so that users can use the thing that's been invested in yeah so like i mentioned some of the red flags already fail fast idea fail fast

basically is you identify a problem get it fixed or get it replaced like go get a new vendor rep go get a new vendor completely um think about what that life cycle means for you if you've got to start procurement over again and it takes you 10 months you'll probably want to start as early as possible do not go down with the sinking ship you're wasting time and money for your organization um like i said i've made it's made some of these references already tools and deploy in the first year um user's not adopting the tool it's in prod for 90 days um or plus 90 days and users aren't really using it they're still talking

about it being in beta really if we're buying solutions users should automatically know how to put this into their sops right if you if i'm like i really get tired of notepad i wish there was a way to sort this data and you hand me excel i'm gonna be like hell yeah and five minutes later i'm gonna be sorting data in excel right like caveman finding fire if the tool's not giving you that aha moment you may have the wrong tool if your users don't want to use it because it's clunky or doesn't do what they thought it was going to do you may have the wrong tool and you got to learn to fail fast for that if majority of your

requirements aren't met you know 12 to 18 months after deployment um that's very concerning procurement right right and it's not the vendor's fault necessarily um if they promise things especially in writing or if they promise things then they didn't deliver it yeah that might be the vendor's fault but it really could you know take some ownership you could have screwed up pre-procurement if you're not fully deployed or mostly 90 fully deployed 12 to 18 months after you bought the damn thing um the other thing if uh you're working with the vendor and like you know more than 30 of your requirements are adding additional costs where every time you talk to your vendor they're like oh

yeah we could do that for an upcharge or you know we can we'll get you a quote on what that's going to cost um that's really really scary um and you know the other thing i've seen too is uh vendors roll out you know their annual event they roll out a whole new capability everybody's like yes and they're like and if you upgrade now for 9.99 you get this capability and they don't roll out that capability of their existing customers like they're a freaking cell phone company like that's dirty business in my opinion a company should be adding enhancements throughout the product where you're they're creating a whole new capability a new platform sure maybe we should talk

about you know having a different product line um that would make sense but if they're adding enhancements to the existing product line and they want to upcharge you to migrate to it um maybe you bought the wrong fender in the first place uh this is a true bullet at the bottom if you're you or your children start using the name of a tool as a swear word my daughter was like 18 months old and i won't say the name of the tool but it was built in house and a client and i would say it under my breath like the f word and she was like 18 months old walking around the house saying it

like it was a bad word that's a really bad sign that you got the wrong tool you got an interview no i think as as the vendor here i'm going to say a little a little a little silent i will on on the slide there was a question there in chat on isn't that a selling module to roll out one thing and then add modules it it is and i would i would contend that that's a a way that many companies validly have a successful business model the challenge though is when you talk about meeting requirements with this whole thing is about evaluation if you get to the point where after acquisition you're like hey you

said you meet these requirements and we bought you how do we go about doing that and the answer from the vendor is oh yeah you got to buy this other module that's that's a red flag because in my opinion that cost should have been baked in to meet all the requirements in your acquisition decision um so that you would know all the modules or all the pieces if the company pieces things out uh if if the company doesn't piece things out and they sell one product that kind of does everything and then the licensing is clearer the probability of then having those awkward conversations down the line is is is lessened and that's why i think

andy's putting that on on that's andy that's why i think you put that specific point about upsell as a red flag to you is because from your view you would have thought that that cost and that module would have been there at the acquisition cycle otherwise you wouldn't be told later on you need to up you need to buy more to do the dog do the job yeah um so let's talk about manage relationship um we are going to start to run into the q a section here in a minute but we i think we've been addressing most of the questions so if you have questions start throwing them in the chat we'll try to address

them if we don't address them here i'll try to address them in the write-up that james and i are going to do on medium later on killer grizz klr grz on medium follow me on twitter as well i i really do plan on addressing all the questions if we can't do it today so um you know expectation settings related to vendors um you know i i think james this is really your slide um it's a little bit of therapy for you i think at this point go ahead yeah so the i'm gonna talk as briefly as i can on this which everybody who knows me knows it's not my strong suit as a vendor i'm gonna know what my product can do

and i'm gonna know what it should do in your environment and i'm gonna have strong opinions on how your people should use my product but in the end i'm not able to consult for you to identify your problems i'm not able to get in there and define for you your own processes i can tell you options on how to use the product well and how to not use a product well and then you as a as a consumer as a customer have to make that that choice the the deadline conversation kind of that that next one is you know unless you're explicitly funding a vendor like you're a dev team which is not a model our company follows

which is why it's important for me to point it out here unless you're paying them specifically to do things on a specific timeline the ability to hold them too i want features a b and c in a month or three months is going to be difficult having them respond to your priorities and maybe giving them giving you a straight answer of your priority number one is priority x on our list and we're going to keep it there until it gets fixed and then you they you as a cust consumer can kind of see things roll out over time uh and you know your things moving up to eventually get hit like that's a that's a good

that's a good relationship in my opinion but but that not all vendors are going to give you that option to give them cash to build the thing you want and you may not want that option either one of the things we've seen across the user base and we've actually one of our most healthiest customers brought us in it was hundreds of users and they sat us down with their team leads and they said we don't trust our processes tell us how to use your product well to solve these problems and we were able to show them how the product solved those problems and they adjusted processes sops everything to empower their users to use the product way it

was designed to work and in that workflow we found limitations et cetera that we were able to work into our roadmap and make things better for them over time and that that was a much healthier conversation for us and them because they were willing to admit that we are bought a product that meets our needs but we might need to change the way we work you know in in andy's example getting he said he had 21 tabs open pre-acquisition and he has one tab open afterwards you actually have to break people's habits to stop doing the things that they used to do if the product in play is going to make them better at doing their job

and so that's where understanding the design through documentation through training events helps that transition period um and andy makes a solid point earlier a tool that constantly needs training is is not that's a that's a red flag or a negative the other red flag i would put in that conversation is the inability training unwillingness to have you a vendor do a two hour or a couple segment recorded sessions for you one time because you hire new people people come in sometimes someone has a starter at ground zero you have to be able to empower them from the beginning and resources or training events are in my opinion the right way to do that i'm certainly comfortable with

sure we'll run a few sessions you can record them and replay them as much as you want as a solution to get people you know get customers what they need to move forward uh yeah so you know understanding the the size of them what kind of financing they have or they venture capital it back they're probably just trying to make money no um right understanding the type of business that they are um and how that impacts you know your your service and support contracts how it impacts whether or not you're getting free training um how is the company grown right where they um just created you know because some tech guy had a solution and you know want to drive a ferrari or

was it a group of analysts who got tired of doing this thing manually over and over again and so they created themselves a tool and they made it available as a solution right that you can purchase or you know get from open source and the same same problem with open source if you you know there's no vendor support it's an open source tool right or there's only community support there's a lot of costs involved in there you might have gotten a free tool but there's a lot of cost in figuring it out yourself and spending a lot of time on documentation online or reddit or wherever the hell you can get information so just because it's free doesn't make

it free right so understanding how how that works yeah i'm not going to beat this up too much um i think it speaks for itself and we're running out of time i'm pretty sure the live video will cut on your guys's end and we may not know so uh i apologize in advance if it does that that's the platform's trick to help things be efficient this slide was put together that once you understand the nature of your your vendor in play uh the you can understand how to use it to your benefit which is like where andy put this together like how do i hack the vendors to get where you want to go you know a privately held company is

going to tend to be smaller than a venture backed andy venture-backed not the of what you have on screen company is going to make them a little more likelihood of being partners getting strategic you might be more likely to be like yeah you know what we can put this thing on a road map we can use you as a use case or we can kind of do some collaboration here but then you know if you're going to have a large backed company they in theory should have more resources to handle things or be potentially a more mature solution at the outset uh some in the less privately held and and larger i kind of blended

the small versus large small um has some some risks to be kind of they get they get stuck or efficient large could be successful or maybe they're inefficient a little bloated you've got to make that assessment with how you see your vendor interact you know if you have a large vendor that's taking months to get to basic requests or solutions or just improve documentation versus a small vendor who's like i get email comms pretty quickly from them and they get updates to their guides or workarounds to me in addition to i see the product change six months down the line and when i lost video that was the end of it

uh i know we keep going for people in the room or they just off off they're yeah they can't hear us or see us anymore um we almost got oh i think the important stuff uh i was freaking out and talking really fast the first 20 minutes because i thought we were a 30 minute slot not an hour was like we've got way too much material and then i while you were talking one of the times i handed it to you i checked what it like changed tabs and i was like oh it's a 60 minute talk yep i think i think with getting to there and then the things we're missing at the end is

know yourself wrong vendor and then the time to hack your vendors you know that getting the slides out there and doing the follow-up post i think is going to be the the the right way to follow up with the crew i think we got through the important stuff and uh people had i think we answered all the questions as we went either in chat or um either in chat or uh oh dave says we're still broadcasting dave you still hear us oh well we still have like 30 attendees in room andy maybe we should finish this out wait why did it drop for some people i'm very confused i am not the vendor of this platform so i don't know

but um all right sorry guys so it looks like it dropped for some people um but yeah we we have more stuff uh we'll cover it here and for those that dropped we'll we'll share it out later uh yeah so know yourself um what type of risk are you willing to accept um you know a lot leveraging the adaptability to you know change operations right to suit the product right we're gonna talk about addressing our sops being flexible yeah not being stuck in the mud um uh go ahead james you can cover some more of this stuff so understanding yourself is important so i've i've gone through a big career change of my the majority of my career was

spent at a very large government integrator contracting firm i've been switched to come over and join the co-founders as first employee here and switching to a small privately held vendor company um personally i love the change i love the company i love the product that we're doing obviously because i'm doing it right but that gave me this this perspective on this gamut you know the small company i can talk to everybody and we're pretty risk accepting in in terms of like going after big things but not like uh in some areas and in other areas like on the way we manage our our growth potential we're we're intelligent to stay stable working at the big the big entity prior

like the ability to get things to move was was very difficult um they were very risk-averse and there was a lot of kind of structures in there so i had to learn how to use that structure to my benefit to kind of communicate and get buy-in for more seniors to be like oh this is a important thing to go forward you know the event that andy talked about with felker at uscert is a great example of an intelligent leader understanding top to bottom what's going on and then making strategic decisions if you're in that very structured organization you've got to find a way in that organization to use your seniors as a user to understand your problem and become

your direct seniors to be your advocate up the chain to understand the problem communicate in their language like if you go back to the earlier slides and you say to a senior in my current way of doing i spend 40 hours a week doing a with this product it would take me five and then i spent 35 hours doing this other thing you want to do that like that catches people attention and it's going to have your needs be justified and argued on your benefit versus on the other side like if you're a very adaptable flat low hierarchy organization the ability to move and pivot and make good decisions quick is going to come into play

but the the challenge there i would advise is you as you as a user have some diligence on your on your decisions you know i i've given bad recommendations before and i was very grateful to work with people that told me no let's look into this a little more and it stopped me i was much younger at the time without the beard or the gray so i'll chalk that up to youth but the that diligence to prevent just jumping at the latest thing is going to save you a lot of time and energy um on vendor selection and in general like even going into that process change stuff we talked about yeah absolutely you know i talk about uh

example you know large versus small um i remember a couple years ago we were going through a procurement life cycle and this small company no one had ever heard of came to pitch us one day and they were talking and they're like oh yeah and uh for the folks that saw the demo a couple weeks ago we didn't have this capability but you mentioned it so we got it rolled out let us know if this is how you wanted to see it and uh they started to continue the demo and i remember i think it was me or one of the other guys like wait when did you do that demo with them and they're like oh like two or three

weeks ago and then this small company right they were young and agile they rolled out a new capability as a platform for a requirement for a customer they hadn't even earned you know had a contract with yet um and that really impressed us as a team like we've had so many vendors like oh yeah it's on a roadmap on a road map and here was a company who was very agile able to respond and give us an absolute requirement i mean it was a show stopper not to have it for us um deployed in their platform before we'd even uh considered purchasing it um so there is that perk right but you know james's experience right smaller company

may not have the depth depth in their bench to go do bigger crazier projects um too so there is some some perks are using a larger vendor who's got a deeper team it's there's just pros and cons you got to consider so and this goes into if you listen to andy's language you use the term showstopper which aligns to what we talked about at waiting and criticality as a must you know being in that situation ourself when we heard of a customer we really want be like we can't move forward with you for a b c and d we've had some evaluations come at us and be like listen we really love everything you do but

this one thing that you don't do is why we can't proceed it's sad and it's it's rough um the first situation was one where we were told explicitly up front and early and given a window to come back and kind of show them we're going to adapt and be good to you and that led that was very positive uh because that customer ended up acquiring us the other was they went through the eval and in their down select process they just kind of made that call of you're not you're not checking the box uh for this one thing we which goes back to that waiting right we hit everything else in the conversation with them we hit all their

other musts and we did it better than their other things but this one must was honestly we don't do it and we can but we need to have some understanding on if you're going to proceed with us if we do and that lack of flexibility to give a we want to go with you but we need this by this date we probably would have proceeded with with that and committed to it as a as a good relationship so having that flexibility with your vendors when you get to that scoring and eval so that you can tell them flat out you we can't proceed with forward with you without these things empowers them to be adaptable or not and

if they're not that's an easy decision if you hit that big red no button uh yeah so wrong you know some of the red flags you got the wrong vendor we kind of talked about you know they're rolling out new capabilities every year for an upcharge um you know they come out with a big open keynote speech and they're like we built this thing 3 000 and like but i have the 2000 like it's going to cost the upgrade right um and they keep doing that you know year after year that might be a red flag maybe you know that's just their sales model and that's cool but think about whether or not you're still getting the support

if they've got the time to build a whole new platform every year a whole new capability every year and they're not rolling out to their existing customers they're probably not building on the requirements you're giving them year after year so consider that right um every conversation adds to additional cost or if you ask about something and they're like oh there's a workaround that's fine right you know david mentioned here in the chat about you know having good success working with overcome some show stoppers that's cool for like one or two things but if everything you do and especially what you consider like a simple task or a basic function of the platform and they're like oh yeah this is how you would work

around that if they keep using that language that's a red flag you've got the wrong vendor right it should again a ui like a joke shouldn't have to be explained um this is my number one complaint about vendors you come in to do a demo and you ask me what my use cases are before you showed me your platform if you don't know what a sock does or a threat intelligence team does and you're pitching get the hell out of the office like you don't deserve to be here you built a tool to solve my problems as a stock analyst or a threat analyst show me your pitch first leave some time at the end for specific use cases

and say is there any use case we didn't cover because if you don't know what a sock does or a cti team does you probably didn't build a solution to solve my problem in the first place and you shouldn't be at the table that's that's a really big red flag and i want to shoot a flare at you for that one but james go ahead you know the work around things that my opinion are healthy are things where like oh hey the the platform as a threat until platform it's all about data right so i'm going to speak to that in our use cases the the you customers talk to us like hey our your platform has all the data

inside of it but i kind of need it out in this i kind of need it in this format in this way because i have these mandatory requirements that i haven't been able to change the sops yet it's like okay let's give you the means to get the data out that's you know not necessarily a built-in feature but let's give you the means to get the data out in the format you need to answer your question you know a key use case has happened where um one of our customers has very strong relationships with a wide variety of international partners and they bring that knowledge they get from them in to make good intelligence sharing

security decisions doing what a tip should do but then they've actually migrated all their previous methods of reporting to leveraging data outputs we've helped them use so they're using the platform day-to-day to do their job it's been adopted and used well but then it's like every year they have this one-time activity it's like hey we gotta go do that report thing again it's like okay yeah use here's the thing that meets your specific need and handles your specific requirement and it's so niche and unique that software development in the product probably would take more time and energy on everyone's side and the probability of missing their need is is higher versus a let's just be your

partner with the in the relationship with like the vendor rep on screen to get you what you need in a timely fashion and make sure it's durable because they've now done it for a few years in a row successfully and um this goes back to the miter attack thing originally there there was uh there was other data points they cared about and because they figured out oh wait we have attack here as well you do it's like okay we're actually going to expand our product what what they're producing outside of platform for reporting and benefits and they're able to do better assessments of the relationships they have so the that's the kind of workarounds i

see as helpful long run just to own it our goal is that those workarounds are always in some capacity temporary except when they're really in that niche use case because things should move in to be product features eventually work around should be should be temporary and then they should roll into a product solution down the line the the the risk the red flag is when vendors can't make you succeed today or can't give you an idea of their understanding about how important that they get how important something is in that post-acquisition reality to then get you where you need to be to succeed yeah and i would say the biggest red flag i will say is not on the slide if you

got the wrong vendor or the wrong solution is if it's not a solution if it didn't solve any of problems or if it added more problems than it solved it is not a solution furthermore i just want to use furthermore uh time to hack your vendors right develop a plan to assess and manage your requirements we talked about putting your spreadsheet together um right put out your requirements actually put some narrative around the requirements need to be able to upload well i need to upload it right need to be able to upload auto types of indicators for my case i'm a thread intel guy or whatever your data points are needed to be actually usable and not

just a blob of tech right is it searchable um actually writing out those things right uh and ultimately testing them getting engaged uh multiple teams right if you're buying a threat intelligence platform and only the cti teams at the table and the stock's not there instant response isn't there or the risk team isn't there you've got problems it's cti team right probably buying the tip and their requirements should be most of the must but those other teams have some requirements too and they should be at the table one team or one or two people at the table you are doing acquisition wrong and most importantly if you're buying a tool purely from a demo uh vendor led demo and you didn't get

hands-on and your people didn't get hands-on you're doing procurement wrong completely don't buy a car without tech no wait i always buy cars without test driving don't buy a tool without test driving i'll tell you that much and communicate with your vendors they're ultimately not evil um you know if you're if you're not in the government space they can actually buy you lunch so at least you get a free lunch out of the conversation um you know they've got some of the softest swag shirts and polos in the in the world uh more than just the shitty little pens you get at the conference is but no talk to them they are people they want

their solution to work right um you know we had it earlier james said like the biggest thing in the world biggest problem in the world is if uh if they're not being talked to or you know they're creating more problems one coffee have an honest conversation i will tell you this if you are a vendor and you want to know if your tool sucks give me a free account i've done this for every one of the threat intelligence major threat intelligence platform vendors in the work and uh the market i've had free accounts with them for years uh before i've given their sales people and their and their engineers feedback i've done it over dinner i've done it over email

i'm willing to do that with pretty much any anything that's kind of in my space if it's not in my space that would just be unfair for me to give you that kind of feedback but i'm going to give you honest and feedback that you're probably not going to get from most customers because they don't want to violate whatever rules or things i'm totally cool with it don't get your her feelings hurt make your tool better in a year or two from now if i don't see some of those things addressed i'm gonna start losing respect for you five years from now and i've known some of these companies for five years and they haven't

rolled out any of the things that i told them that they suck i no longer recommend those companies when people ask me for a solution because they haven't done anything not to say that my opinion is the most important but i give you unfiltered advice right when uh analyst one reached out to me a couple of months ago and said hey we're getting ready to do a new ui you want to say a demo and i was like if you broke my tool i'm going to kick your ass right like that's the kind of thing you're going to get from me and i'm more than like i love working with vendors because me smarter and i understand how tools

work better but i know that those are going out to the community and people are buying those things and deploying those things and i want you know my bank to be more secure so if i can tell a vendor to do something better and that makes my bank more secure my electric grid more secure or my supermarket more secure that's a good thing for all of us right and so give that feedback to vendors yep yeah we are we are humans to emphasize that again we are not cylons if you know the reference and the the thing on that communication is is crucial because you're going to experience in communicating with your vendor like andy's saying that the

communication happens early happens often overshare and then if if there's not honesty in return if there's not able to give feedback and if there's not change then that's starting to realize some of those those flags that andy said not me and priority the only time communication goes wrong is when you you aren't able to keep yourself as an organization prioritized to the vendor which is which does mean as a customer you've got to sometimes put one user group needs above another so if you if you as a as a as a customer have a small team you're buying for the ability to keep things easy and deconflict is going to be less but in some of our customers

with with hundreds of users and dozens of teams the ability it's it's not on me to tell you what's most important i i will i will prioritize our what we will do internally to our dev team i will invest our time according to what you say is most important but if in the end you tell me that this team's needs are most important but a year from now it was team b's teams that actually are affecting the acquisition or the re-up decision that's going to put us all in a negative place because what if we had had teams bees needs in the higher and the west what if we had had their feedback higher and the less what if we had

shown them the workarounds or the training beforehand then we would be in a different spot the more important needs would be met and you wouldn't have this conflict between team a loves us and team b hates us you would have the more important team moving forward and then we'd be on a team a's needs that priority and deconflicting it is is big um and sometimes you know with the advent of new data standards not that i'm naming names right now but you probably know what i'm referring to in the threat intel space you know that's pushing down a lot of user-facing needs because we've the community's evolving we have to adjust to get the data standards done because and

afterwards we'll get back to the specific feedback requests and that's an honest conversation we will have with you so that you know where we're going and and why and where your needs are in that list so ultimately i think we answered a lot of questions on the on the road as we were going otherwise people lost they're not hearing us anymore uh ultimately i want to thank you know james for and his team for uh being an awesome vendor to me over the years but also being uh here to do this uh talk hopefully your tools after this will suck less your procurement process will suck less hope hopefully this talk didn't suck itself so uh

thank you guys for attending go watch other talks you're going to be late now i feel bad for going over um but at least you know there's not speakers waiting on this room so i feel a little less bad but uh hope you guys enjoy the rest of b-sides these slides will come out come uh i'll probably just put them out on linkedin slideshare soon and then we'll actually get a write up supporting that in the next couple of weeks if my life can slow down a little bit thanks james thanks everybody else thanks andy thanks all