Technical Deep-Dive: SentinelOne

foreign [Music]

[Music] fantastic well uh good afternoon everybody uh hope everyone's having a good uh a good virtual besides calgary today this is norm chan i am the pre-sales se for sentinel one in western canada i am based out of uh out of vancouver and my goal today is to walk through about 100 powerpoint slides i am kidding i am kidding i have uh almost no slides uh today to share i've got a bit of a freeform uh agenda and uh one thing that besides all the b-sides events are in my opinion really good at is uh is trying to minimize uh the fluff in the content that sponsors are supposed to present we try to make it educational

uh higher on the education factor lower on the on the sales and marketing so the the topic today that i'm that i want to delve into is actually on this tab over here on my browser which is the the miter attack framework so we're going to talk about mitre attack uh i want to talk about something uh known as the miter attack evaluations as well and if anybody who is tuned into my session today is looking into edr looking at either acquiring an edr solution or upgrading your existing uh epp then i i think this session is going to be of interest to you because i'm going to talk about the market position or just

what the market looks like today for edr and epp and then i'm gonna delve into mitre and uh sort of give you some some tools to sort of better enable you to make decisions uh you know whether you're about to make an edr purchase or you you have an edr solution today and you're just looking at what else is available in the market so uh as i i mentioned this at the beginning of the of the session but i based on my setup right now with my single screen i don't have visibility into the chat and q a so if you do want to ask a question or chat i'll get to them uh near the end of the

near the end of the session so let's uh let's go ahead i actually have close to zero slides i'm gonna start with uh with this tab over here and uh this for for those of you who have not seen this before this is uh gartner's magic quadrant for epp or endpoint protection platforms and for a lot of people who are in the market for epp edr in fact gartner more or less rolls edr or endpoint detection and response into something that just should be fundamental part should be a fundamental part of a endpoint protection platform they don't really distinguish between the two anymore so what you can see and for for those of you this is

probably who've seen this before it's probably pretty obvious in the gartner magic quadrant report people always look to the top right corner okay because these are the the companies that are perceived market leaders or identified as gartner as market leaders so you can see who is in the leaders quadrant today but what you also likely know is that and sentinel one is right up there with many of my esteemed competitors what you probably uh know as well is the gartner magic quadrant it is not a technical evaluation of products offered by the uh by the manufacturers it's a two-dimensional chart and on the horizontal axis you have their completeness of vision so more or less um

does the company and their offerings line up with where gartner believes the market should be going to or what the market should be offering and then on the vertical axis you have the ability to execute does gartner believe that the company will be able to execute on said vision marketing how many calls is garten receiving about the said vendor etc so this is not a technical evaluation of the technologies where and there's many different ways that you can technically evaluate endpoint protection or edr but today what i'm going to cover is uh is the ones that are done by mitre so i'm going to pivot over to here and uh this is the so by the way so miter corporation uh in

in case you do not know whiter corporation a u.s based nonprofit they're charged with doing a lot of advanced research into a number of uh important areas uh of technology uh in the realm of cyber security probably one of the more useful most useful things that they have developed is the attack matrix or the attack matrix for enterprise or just attack in general attack is a an acronym and it stands for attacker tactics and techniques and common knowledge one way that i like to think of mitre attack is it attempts to enumerate in its entirety all the different ways and means that attackers and hackers and threat adversaries are using to to hack into the enterprise so what i have up here

right now is the attack matrix for enterprise and what you can see is across the top of the columns is you have so-called tactics and i think of tactics as higher level objectives that a threat actor is trying to achieve as they are conducting their campaign against you now beneath each of the tactics you have the ways and means or the techniques that a threat actor may may use to achieve this particular tactic so if the if the threat actor is trying to achieve initial access when you look down the column there's many different techniques that they can use to achieve initial access so phishing is one such a very common technique that thread actors will use and in fact

you can see when i hover over it every single technique is denoted by a specific identifier in this case t 1566 is the technique number um for fishing there's actually a number of sub techniques as well so if we drill down into the fishing technique you can see that there's three sub techniques spear fishing with an attachment spear fishing with a link or spear phishing by service or as a service so this is a uh this is a dynamic uh matrix as there are new uh techniques that are discovered this is this is constantly updated so this is fundamentally different than something such as uh what i have over on this tab here this is

uh say virus bulletin uh this i have numbers from june uh 2020 i also have numbers from 2021 as well but i just bring this up just as an example of sort of a legacy or traditional way that uh people would try to compare endpoint protection solutions is they would look at something like a vb-100 or a virus um you know just to measure how good is a product at detecting malware this is fundamentally different from what mitre is tracking here this is how good is you know for example is a product at detecting specific uh ttps or tactics techniques and procedures that are uh that are used by the attacker so let me pivot a little bit

so one thing that mitre has been doing for a number of years three years in a row as a matter of fact is they have been conducting their miter attack evaluations so each year what they do is they pick a thread actor so in 2018 they picked the apt-3 thread actor in 2019 they picked the apt-29 thread actor and in 2020 they picked the carbonac you got a two for one they picked two thread actors carbonac and fin seven so they run the tests typically in the fall uh and then they present the results in the spring of the following year so the carbonac and fin 7 tests for 2021 that were published in 2021 these were actually conducted in

the in the fall so right about now but uh but 2020. as a matter of fact right now for in the fall of 2021 they are conducting their wizard spider and sand worm evaluations and we expect to see those results in the spring of next year in 2022 so what they do so you can see here and this is all free by the way uh part of my hope is that if you are looking back into the market for epp is that you're going to feel a little bit more comfortable by the end of my session to actually look at the data for yourself and analyze this data because one thing for sure is every year

that mitre has been running this each time that they produce and publish the results as as a result of their evaluations every epp every endpoint protection platform company their marketing departments they tend to go into a little bit of overdrive and everybody comes out to say how well they did at uh the mitre attack evaluations uh some people are some people's marketing departments are bold enough to say things like they won or that they were the absolute best at mitre what i tell people is uh is to to ignore everybody's marketing heck you know you don't even listen to sentinel ones marketing come straight to the actual miter uh evaluations site and look at the data for yourself the thing

is that miter doesn't do evaluations on the data so it can be a little bit difficult to figure out what's what so um so here for example for this year's for the most recent results here in terms of the threat actors they tell you look carbonac is a threat group and they mainly go after banks or they or they talk about the fact that fin 7 they are financially motivated and they primarily target u.s retail restaurant hospitality and then and when you scroll down to this section here you can see more or less it is the who's who of um endpoint protection right so basically all of these vendors participated in some way shape or form in most recent

testing okay uh and by the way for those who are who are uh who have tuned in i want you to you know take a look at this list and what i will do is i'll i'll go back to the to the chat uh before i pull up results and maybe what we can do just for fun is uh if there's anybody that you would like to see the results for okay so it's audience participation time right you can take a look at at the list and then in a moment i'll head back over to the chat and and if there's anyone that you're interested in seeing the results for i will pull them up i may be

inclined to pull up my results as well because this session is being sponsored by sentinel one or i may not if if you know i'm happy to always pull up my results but again uh any other vendor i'm happy to pull those results up so i'll come back to that in a second you might be wondering so what happens during the uh the miter evaluation uh testing so what they do again so they they pick these these thread actors and they run a series of steps and sub steps that emulate and use the same tactics techniques and procedures that these thread actors are known to to use in the wild so for example again it's fully

transparent it's fully free if you wanted to see what the exact steps that were run over here there is their technique comparison tool and it shows you over here on the left hand margin the so-called operational flow it's all 174 sub steps that that they were run during their testing right so starting from step 1a1 all the way down to step 20 b 5. so for example if you wanted to know what was the first step of the attack that they the very first step of the attack you can see here that explorer spawns windward.exe when a user clicks this rtf file okay so the the idea is that when this step is run that the product that's being tested

it should notice right because that's fundamentally what good edr does is it is it tracks uh behavior it tracks telemetry uh on an endpoint regardless if that um uh if that behavior was uh was benign malicious or or suspicious so when this step gets run then it should at very least generate telemetry okay now so in this case what i've done is i pulled up for example uh sentinel one so when this step was run we generated telemetry uh and then if the product was able to figure out that the execution of this step resembles a particular tactic or technique those would be higher level detections they're analytic detections they have higher value than merely pointing out the fact that it

occurred if you have a lot of telemetry with not a lot of corresponding tactics or techniques then you're you're really seeing a lot of noise okay so what else do i want to say about this maybe what i'll do is i'm going to pop back just real quick over to the the chat and i want to see if anyone's nominated any particular vendors whose results that we want to bring up so let me just pop back over there real quick back over here and i'm going to pop over to the chat okay people are asking for uh fortinet people are asking for uh symantec checkpoint okay so let's uh let's go back let's uh let's

checkpoint uh fortinet symantec okay let me go back happy to do those standby okay so what we can do is we can pull up checkpoints so i heard somebody ask about checkpoint okay so i'll pull up their results in the tab let's pull up fortinet on one more tab i think someone said symantec okay so i'll pull up semantics over here and heck just because nobody asked for it i'm going to pull up mine so we'll pull up sentinel one so i've got four tabs and this can get a little bit dicey okay because i've got uh four vendors to to work through but that's that's okay i've got time so here are the results for sentinel one

here's results for symantec here's also fortinet and here's the results for uh for checkpoint so this is there's a number of of uh key data points here so let me what i like to do is because i'm a visual individual is i like to scroll down to the results graph okay so i'm on checkpoints results okay and remember how i said that look if when you run a sub step you should at least be able to score telemetry okay and if you're able to figure out if the product is able to figure out that it's a tactic or technique or so-called general detection then you score you know these green colors which is higher value if you miss

then it's considered a nun or a miss because well there was no detection so what i like to do is i like to uh i like to go through this filtering exercise because you can dynamically filter things out so i want to ignore my telemetry detections my general detections my tactics uh and my uh my techniques and that leaves me with all of the no detections or all of the misses so you can see here with uh who am i on checkpoint yes i'm on checkpoint checkpoint harmony uh client uh this is where checkpoint was uh was weak or this is where checkpoint was was blind right so you know three times over here yeah probably two times over here and

probably one over here we could you know this is a delayed malware execution uh step and we could drill down into that into each of these to figure out exactly where checkpoint was uh was unable to detect and this in and of itself is a useful exercise uh if you are a checkpoint customer because uh it's one of two things you can go back in here and you know get checkpoint to uh to help fix this or maybe you're gonna have some other compensating control if you knew that uh it was weak in this particular area so again we can do this for the other vendors and again if you're uh signed in right now sitting

at home try to follow along you know head over to the mitre attack evaluations and you can click along with me or i'm sure this session is being recorded you can do it after the fact let's take a look at fortinet's results so again i like to scroll down over to here and just filter out their telemetry their general their tactics uh and their techniques and so you know if i was just uh oh by the way what there's a i don't know if this shows up very well on the video but there's also a very darker blue uh section here um and uh what this section represents is this is the first year that mitre did

multi-platform testing so in the previous two years for both apt-3 and apt-29 they did pretty much exclusively windows os attack testing this year multi-platform they tested both linux and they tested windows and if the vendor didn't have or chose not to bring their linux client then you you had a a not not applicable uh in the in the column over here so i could probably this tells me that uh for whatever reason fortinet you know did not have or did not bring their linux agent so you know i could just filter that out so here is where fortinet missed and if i was to eyeball it back and forth between checkpoint and fortinet i would say it

pretty clear to me that fortinet probably missed a little bit more than other than checkpoint and let's do it again let's do it again with uh with symantec okay filter out their telemetry and their general and their tactics and techniques and uh so better okay in terms of missed detections we're only looking at missed detections that's only one element of the miter testing compared with fortinet right here's fortinet here is symantec here is a checkpoint right looks like checkpoint's doing pretty darn good okay um so here's the results for sentinel one i'll just scroll down to my results and i will eliminate my detections in both telemetry general tactics and techniques and this is where sentinel one missed

in this year's testing so here this is not a misprint ladies and gentlemen uh but uh sentinel one did not miss this year uh it's never happened in three years of testing no other vendor has uh has nailed the detections uh perfectly we're the only vendor to do this so we did pretty darn good in terms of uh of not missing a step during the carbonac and fin7 scenarios so that's one way to look at the data there's a number of other ways that you can look at the the miter attack data so for example uh one thing they don't tabulate for you let me show you is what they call uh uh yeah yeah what they call um

okay so if you click on here okay uh you scroll down what you see

in this case the last uh console okay but from this view what i'm able to do is i can just do a quick control f and i'm looking for the the string configuration change okay so a configuration change is basically uh it is a change that the vendor has made to their product in some way shape or form during testing if the product was unable to score you know telemetry technique or or whatever so the vendor is allowed on the spot to make some kind of a change sometimes the change is relatively it's relatively benign like maybe it's a uh it's a ui change right like i've seen instances where uh they they recompile code on the spot uh to exp to

extend uh the number of characters in a field maybe because you know they caught a this huge uh obfuscated powershell script but it you know it just didn't show because they didn't have enough characters in the field that's pretty benign the most egregious changes or sorry another relatively benign configuration change is sometimes a data source change so a lot of vendors have multiple consoles where you have to maybe look at the endpoint console versus the uh the sandboxing console or maybe they had to look at a sim or maybe they had to look at an ips or a firewall console then it'll it'll spell out it's a data source they had to look someplace else

to get the answer the most egregious would be a logic change where the vendor had to make some specific change maybe they had to add a hash to a cloud maybe they had to um just add some additional logic in order to score the detection so here what i'm seeing for checkpoint is 32 configuration changes and i've just scrolled through it detection logic detection logic detection logic uh looking at different uh data sources detection logic right so uh basically in this testing for scenario one the carbon testing they had to make 32 changes to the product in order to have scored and if i do the same on scenario 2 which is the fin 7 test if i just do a quick

control f and so 25 uh configuration changes on the fin 7 testing so 32 25 we're looking at you know 57 somewhat changes so basically the the product uh didn't work out of box when you when you see when you see that so let's let's do the same thing for uh let's do the same thing for fortinet let's uh click on here okay and we have one partner two seven we'll just do a quick control f six configuration changes for carbonac and if we take a quick look on uh did i do that right is it another six did they only do oops go back control find yeah six so 12 12 in total 12 versus what was it 50 57 changes for

uh uh for checkpoint and how did how did no that sentinel one who else did we have we got uh symantec how did symantec do in the realm of change [Music] then changes here and on and then zero changes so yeah that's not bad so 12 configuration changes for fortinet 50 some odd for checkpoint and then 10 for for symantec i'm not going to bore you with it if anyone is following along back at their desk if you did this if you repeated this same exercise with sentinel one we actually had zero configuration changes um this year right so i'm sure some of you are going to be thinking okay this is a complete fluke norm you know it's it's a

flash in the pan for sentinel one but mitre has been doing this for three years in a row and uh year after year our detection results are better we didn't score 100 last year the way that we did this year but the our detection uh results are are phenomenal i'm gonna go over two more uh data points uh on here one is uh under this so i'll stay on symantec for a moment um is this uh uh this button here this is the protections test and if you click on it what you see is 10 tests where they were testing the actual protective capability of the products because you can imagine when they run the detection capability

they normally have to turn the products down just to do detection because most products they're going to stop the attack you would hope they would stop the attack so they normally turn products down to only do detection this year was one of the first years that they did protection and they ran 10 tests and actually looks like 15 but it's actually tests 1 through 5 and then tests one through five again but prefixed with uh with a number one so if you see a lock that's a good thing if you see an unlock then that means the product did not protect against said test so if you just look really quick uh here i am on symantec you take a look at

each of their uh their tests they i think they scored like a 10 out of 10 in terms of their uh protection and if we do the same thing for fortinet just real quick we just eyeball it right so here i am on fortinet's protection testing and and again we're expecting to see a lock okay and i just scroll through it really quick and just click on each of the uh the various tests i think what you'll see is again i think fortinet did really well right they they were able to do uh like 10 out of 10 on the protect and if i go over to checkpoint check point did not publish um the results so i i

just mentioned before that this is the first year that mitre was testing for protection and as a vendor you were actually allowed to not publish your results if you chose uh not to oh in case you're wondering uh on for sentinel one under protection i think we scored a nine uh out of ten nine out of ten uh let's just do it real quick here over here little protections and if i just do a quick same exercise we had an unlock on five so we missed one on on test number five okay okey dokey there are many things um that uh we can go into on the micro attack testing i i i like looking

at the detections because if you if the product can't detect what an attacker or threat adversary is trying to do then why why you know what is what is your reason for being if you can't detect so i i always like to look at the misses or the the no detections i like to look at how many configuration changes a product uh had to undergo uh to get the results that it had i like to look at the protections uh naturally actually there's one last thing that is uh sometimes fun to look at which is the uh the configuration right so specifically what did what version did the vendor have to bring and you will find that here

so under vendor configuration all right for a given vendor um show me the configuration for uh this particular test and you can see symantec uh this is their edr 4.5 uh their step prevention uh sorry threat protection and something called their threat defense for uh for active directory and then they have a bit of a description of what uh what they brought same thing for fortinet if you uh hop over to fortinet and click on this you can see it's the version the versioning for their 40 edr and then what did checkpoint bring checkpoint brought uh their management console their agent uh and uh their harmony advanced uh product okay okay what i'm gonna do

is pop over to back to the uh to the q a oh we got that lovely effect there i'm just going to stop my sharing so that uh i don't give you guys a headache okay and uh maybe what i'll do is uh i will uh look for any questions in the chat any questions in the q a and i'm taking a quick look i do not have any questions in the chat or the q a fantastic um so ladies and gents uh i hope you uh hope you learned something new or at least one thing hope you learned at least one uh one one new thing from this from today's session i hope that you

feel a little a little bit more uh a little bit more confident if you had to go and look at the miter data and sort of look at some of the data points that would be of interest to you there are other ways to slice and dice that data i was uh engaged with a customer who was overrun with uh their current vendor it's just noisy right so they they wanted to know by looking at the miter evaluation data how can i figure out which product is less noisy there's ways of looking at at that by simply counting up the number of telemetry data points or the number of techniques that a product is spitting

out given one step and sometimes it's nuts there's one of the vendors i won't call them out right now uh but they had like five technique uh detections with just one step like you ran one step in the testing and the console just lights up and says it could be this technique could be that technique maybe it's this technique it's like five techniques right so that's noisy that's probably not uh you know a vendor that you would likely want to go with if you're trying to reduce noise i digress let's uh i'm going to wrap the session a little bit early thank you very much for those of you who have tuned in again my name

is norm chan i am the pre-sales sc for sentinel one if there's anything uh that you want to chat with me about about mitre uh about edr and xdr in general or if you want to talk to me about sentinel one specifically happy to do that i'm monitoring uh the booth or feel free to reach out to me on linkedin thank you very much ladies and gents hope to have a great afternoon and have a a great b-sides thank you