BSidesNcl 2021 Framework your PoC Michael Johnson

okay good evening everyone uh mike johnson i'm a devops security engineer and um part of my job functions in the sort of security tooling space and obviously validating tools before we implement them um and even just comparing against our current tool sets one of the issues presented with this was as the technology advances at such an exponential rate it's very difficult to justify a change for maybe two or three features so there's a i developed a framework that would aid in a proof of concept for security tools and the rationale behind this is there's been in the past 18 months a few cyber security companies that have hit the unicorn status in the they've been

valued at over a billion dollars and this just means that there's going to be far more funding in the future and where there's money there's obviously innovation the issue posed to a security team however is if you become stagnant you could be missing out on a huge number of either benefits or you know reducing your attack surface or risk one of the issues as well is um if you look at tools that from an infrastructure perspective such as ansible tower or something like that the worst case scenario engineers that have to work around some of the nuances of the tooling when you're looking at security tools they can be the sort of difference between you knowing that there's someone

in your network and not knowing not necessarily these sort of toolings it could be something that looks at company policies like iam

a lot of proof of concepts that you run with vendors will be probably a couple of weeks to give you that feel for it so the scenario i present is you've logged in to your email you've received a linking message and it tends to be a called call from some security vetter that is offering you the next generation of machine learning artificial intelligence backed security sounds fantastic now you've gone through the motions with them you've been speaking to them and they've given you the demo now for something like a hidden xdr platform they'll likely show you a live demonstration of the tax it'll show you a very controlled environment where you know here we can

monitor this we're looking for lateral movement from containers and things like that and you can take their word for it and you can obviously see it as believing however there's always a side of caution especially if you're working with a small startup or a very sort of small company in terms of there might be limitations to technology that aren't immediately obvious so you're looking to fill this gap and uh you go through the motions and you're now doing your two-week proof of concept well how do you get the best value out of the two weeks before you either have to commit to a probably 12-month contract that could cost quite a lot of money or you have to make the justification of

why there's better solutions or there's deficits in comparison to your current tooling so the goal behind designing a framework would be you'd sit down and you'd scope out all of your requirements it could be that you've got a gap a b and c you then have to take a look at the sort of bigger picture of well is there any solution that's doing it for less money i mean it depends on where you get to if it's budget or if it's just the technology if you're in a position where money's not really an issue in your security budget then that's fantastic you can start looking at some of the top flight companies and you can start comparing against them

but it all comes down to this documented process where when you're making a justification to potentially the decision makers or management to try and release the budget you have this paper trail that will justify each way that it's been carried out and conducted so the initial scope would be maybe you've got these five key features that you want it could be you've got a compliance framework that requires you to monitor data exfiltration tools now a lot of the time um this wouldn't do like something like cool which is as many people know completely benign most of the time um but could be used to obviously do some uh malicious things and there are some frameworks that mandate you

wanted for this now it could be a case of you like some dirty bash script that will every time the uh command run it will send off a slack message or it'll email you and yeah that would be great however if you're looking at a tool that's obviously going to protect you and offer you the solution that you need it would be nice to have that is your sort of that's one of our appointments and then you can compare it against a second tool and at the end you've got these comprehensive and detailed comparisons between maybe two or three solutions so the best example i can give is for something like a host intrusion detection system or an xdr platform now

this is probably the more it's more the fun one to framework and have like a lot of fun with and so you look at the salute like the problem and you define your requirements so obviously you're going to want to be alerted if there's lateral movement through your network you're going to be wanting to know if a malicious binary is executed and things like this and once you've got this core set of requirements um you would develop a sandbox environment a lot of the times a proof of concept will be run on if it's an agent-based one it'll be right okay well we'll drop it on a select few boxes in our staging environment or our dev

environment and that's beneficial when you want to get the feel for what the deployment's going to be like depending on how you're structured it could be that you've worked with the stakeholders who run the service such as the infrastructure team so you know that the deployment's going to go quite smoothly you know that there's no nuances with the deployment in there it doesn't like another agent that's running on your estate so this is where the you create your own testing sandbox it could be a very controlled set of five servers it might not even be necessarily in your infrastructure it could be on any cloud-based provider going and the idea is you will have a scenario

i used for the extended detection response would be a ssh host uh you've probably got a db server storage box um maybe a vulnerable web application and things like that and you create this it's almost like a capture the flag scenario in the after we've seen our demonstration and they've shown us all these amazing features well you don't get the full feel for how does it alert me what can i do to escalate if it detects uh three o'clock in the morning um data my database has been accessed and you know these um tables have been copied and things like that i need to you need to get that feel for how the information is going to be

conveyed to you in a way that as soon as you've deployed it and you've gone through the process you get this alert and you're like oh well what do i do from here it could be that the alerts only come via email which is no good when you're using something like slack or microsoft teams and it could be that there's no real um scaling system so it just alerts everything or it alerts too easily so a lot of security tools depending on the nature of it something like a runtime detection agent will monitor it for just about anything and they create a lot of noise so you have to get that feel for what the noise is

going to be like if you've done the two weeks with a concept in your development environment or your staging environment i would hope that it's very very quiet um it would be very worrying if you've deployed it to a subset of systems and you all of a sudden start getting alerts everywhere because you know there's someone inside your network it would be never nice to find out so this is where the scenario comes and you have a little bit of fun so you give the uh like a scenario to maybe a junior member of the team it could be that you've got a graduate program and you've got some new starters and it gets that process taken out of the decision

makers hands and it shares it between you could come in with a biased so if you've brought this if you're trying to champion this tool the this process kind of alleviates any bias that you can drive forward and sometimes that can cause um some narrow sightedness in terms of a solution it might be that you've been using this vendor for years and you know you've got a really good reputation with a sales rep and you want to champion that at your new company or it could be that you just really like the product you've seen it it looks really interesting and you're sort of you're all into it but again if you've got a team of people

that are all comparing against this framework it means that you're evaluating two or three products and you can get the real apples of apples comparison against maybe your top three products and the idea behind it would be you create this sandbox and i mean with the benefit of infrastructure as code and things like that you can terraform your infrastructure deployment you can ansible your sort of a vulnerable network setup and things like your mobile application sort of and once that's all being deployed it's the next time you do it it's as simple as dropping a few lines of code and everything will be rebuilt so it makes it highly reproducible the red versus blue team scenario is

also a really nice one in this sort of space where if you've got a team playing as the red team who are sort of attacking this scenario that you've set up you as the blue team can then monitor and see how is it alerting me

and that's the premise of it it's reproducible i think some places you can get into a rut with a tool and after a year the way technology is moving after a year it could potentially be the wrong solution now so if you've got this reproducible documentation it means in six months to eight months when you start to look at the renewal process and things like that you can pull up your framework and you can test the newest solution um completely like for like to the solution that you're currently using and it means there's no unfair bias to oh well this is just newer it could be that the solution you found 18 months ago a year ago is the

best solution or it could be that you know what there's actually massive deficits that we've noticed which you can then add to your framework the ability to add to it in a year's time after you've sort of bedded it in because there's always nuances with tooling especially if you're working with some smaller companies or some startups it could be that the roadmap's not necessarily where they promise to be 12 months down the line it could be that a company's come up with something completely groundbreaking that would benefit you hugely it's just this reproducibility of testing that it makes everything a lot easier the sort of you can look at it is um an iam analyzer there's a lot of sas

based platforms that will link into your aws account for instance they'll look at three months of cloud trailer logs and they will give you least privileged access configurations based on you know which users are interacting with which areas and things like that well that's fantastic if you take this one tool and go okay well this must be gospel for the least privileged access configuration that they've generated then it's not a it's not completely done your diligence based on [Music] based on the fact that it's just taking one opinion if you look at some cloud-based providers will have their own role-based analyzer that will generate the best least privilege but have a look at two or three solutions use the same

test it could be if you're looking at something like a cspm create a deliberately vulnerable uh configuration something that you know should be flared up and then compare it against two or three tooling sets and you can start to get a feel for it the idea is to be quite creative and have fun with it it's it's um don't limit yourself it could be that you're looking at a um as i say a hidden section you create this capture the flag program it could be there every year when you come to renew it and you've got obviously a graduate program things like that that could be okay well we're going to try all these new products this year

let's give the junior members of the team or the new starters this fun interactive way to get a feel for it and it becomes a sort of once you've done the initial grunt work so to speak of generating the policy it becomes a lot easier i think the creativity creativity here is the sort of key to the idea if you look at the problem and you want to you can more or less framework for any type of security tool um an antivirus solution it could be that you're using um a big named antivirus solution and the new kid on the block has come out well you look at a lot of things budget's obviously one of them

but then once you've done this framework and you've maybe compared three or four of them apples to apples with a framework you can go back and you can use that as leverage the idea to negotiate or haggle almost based on you're speaking to your vendor okay you've done your proof of concept well how was it oh well we've got these deficits you know your competitor is doing this slightly better we feel that this would be a better fit for us um you can get into that point where you can maybe fill up some of your budget and the fact that you've highlighted the issues they've got by the end of the day they're still going to want your

business it might not necessarily be the biggest discount going but there's still that leverage and you can give them the paper trail so you can easily compare against two different sets and essentially you're making this sort of argument for you've done your due diligence and once you've done that diligence it gives you that leverage the entire premise of the framework is to [Music] add this sort of it's almost like um we've got infrastructure as code to create everything reproducible easily it's reproducible testing for infrastructure tooling it could be cloud-based solutions it could be anything it could be that you're looking at a vulnerability scanner and you've got three or four on-premise vulnerability scanners and you're comparing each of them which

one's finding the latest zero days who's got the most up-to-date news uh where are they looking for this information and things like that and again once you've frameworked it out and you've got it all documented you can notice that there's certain ones that have a massive deficit or there's some that while some of the technology that they're using is really good they're also the user interface is terrible and it takes someone three or four hours to try and sort of do anything the biggest highlight for the framework i've noticed was with the customization of certain rules um when you're looking at the rules that your sort of security team can generate and create themselves it depends on obviously the tool and the

solution but personally i think the more control you have over the um like the rules and the alerts it makes it a lot easier you can tune out a lot of the noise that you'd otherwise end up hearing and you can then create the sort of it may be useful for you in some cases you might open a tool and it has issues where you can't necessarily tweak the rules or turn them off or change the alerts you can have workarounds and things like that and but you've not got that same level of fine grained control that you would otherwise have and i think this is the this is where i've run a number of proof of concepts

and you've got two weeks and you've dropped it on your staging environment and that's great you get a feel for everything like that you have the strongest strong enough opinion to make a justification for okay well this is what's going to keep us safe for the next 12 months and i think the the overview of things of how does it interact with everything but you've got this paper trail so when you go to management and you're asking for obviously the budget increase or maybe um it could be that you're trying to move away from something that you've been partnered with for years you have to make these justifications and if you can show the exact paper

trail and apples for apples there's not a lot of argument in the [Music] uh in the discussion any questions

thank you does anyone have any questions oh you're all very well behaved it's strange that's the b-side you know well thank you oh chris has got a question spicy ah there you go that i'm like oh do you have any examples anywhere of how you put it on paper to map out some of these differences so that you can measure them out so this is the one of the sort of fluid sides of it where there's no right or wrong answer it could be you've got a very sort of you write the scenario at the top and you've got tick boxes it could be the each section you tested against you break it down it would depend on each

organization as to how you want to do it you could literally do a um i don't know a graph and have well okay this has a b c features each one of these has the feature and it's just this sort of how whatever works for the team or whatever works for you so there's no right or wrong answer and that's where they sort of get creative with it and just have this documentation that you can reproduce each time and it's just easy to sort of go back to him like a year's time well why are we spending all this money on this tool well okay here's the diligence we've done we spent two weeks we've tested this fully we can guarantee

at the time this was the best solution for us because there's situations where obviously budgets and things come into scrutiny and if you can give someone this piece of paper that just goes well here's all the tests we've done and here's my justification for why this was the best at the time it makes it a lot easier to navigate that sort of either renewal process or moving away cool thank you very much everyone give a warm round of applause to mate [Applause]