Network Security Monitoring Product Evaluation

[Music] I will start off to be typical disclaimers opinions expressed here commentary those are our own not attributed to our employers vendors customers or besides that's all see we are more than willing to discuss sending pretty much everything we do we are not willing to discuss who our customers are and who we do it for beyond high level we do urge caution and testing some of those stuff that we're talking about here we're not going to give you like a how-to guide to have to blow away Network but we do touch on some topics that if you're left to your own devices on your production network all of your administrators and bosses will be quite

upset so we've already covered who we are here's our twitter handle : ah well my background is primary software development I've been thrust into the cyber security world several years ago so I kind of try to fuse the teeth question is why are we here so we monitor were tasked with monitoring and providing intrusion detection and analysis for a very large set of networks they're geographically distributed those networks are constantly under threat from all of your average run-of-the-mill average hackers and nation-state actors so we have to constantly evolve the tactics that are our sake analysts are using to combat that and kind of give us a visibility of that space I'm sure you're all aware

that there are constant announcements of new products on the market they're all the latest greatest thing or everything you like to be yeah they're basically Swiss Army knives used to we used to operate in a space where there were a handful of really good security products and you just had all three of those and you're good and now you've got competition from a lot of different directors there's a lot of overlap you don't want to play a lot of redundant capabilities because it's increases your administrative overhead so we spend a lot of time testing all of those products to kind of see how they fit into our problem space which ones do what they do or what they claim to do

how well they do it and how well they'll do it at our space which tends to be a little different but a lot of more commercial spaces so and primarily we're here because we want to be able to share some of the approaches we've taken watch you guys be able to learn from some of our missteps landmines I also want to learn from you guys find out what sort of stuff you guys have been doing to kind of advance your cyber security missions so some baseline assumptions we assume that you're all here because you're interested in building test labs and figuring out how to break tools how to make it work better that you're interested in you

know catching bad guys and you have some at least at least some base understanding of networks carrying water here principles good reference for that would be Balak spoke towel network security monitoring is very good set of principles outlined in there he's got a couple other books that we'll get to but ultimately you guys care about network security so so do we our livelihoods and our hobbies our expectations for you we'll set those now you can understand what we're going to do what we're not going to be here so we're not going to tell you what you need to do what you test you need to perform to evaluate products those are going to be very

specific to your environment your use cases and to your specific problem sets that you're trying to address we'll provide you some guidance on how to identify those and flush those out for your specific organizational needs both beside the excuse me will describe some guidelines for how to set up a test environment some of the tools you're going to need some of the dependencies some of the skills that you're going to need to develop to execute these tests and is repeatable and effective that we do expect that you will practice it this

cumbersome the topics here is the number of reasons why you are going to be evaluating security products my favorites at the bottom there's always performance issues with pretty much every tool that's out there and most tools do not provide you any sort of insight into how a new analysis process or a signature or rule or detection methodology will impact the performance of that platform so you're kind of on your own dime there to figure that out and it kind of control your destiny later you're taking your all your latest threat feeds from 300 different vendors and pushing them to your senses you need to know how your sensors are going to maintain operations and keep your sock

filled with useful data sometimes you have to a new capability a lot of businesses approach cybersecurity from the compliance perspectives of the base gear meeting a Sox compliance rule or PCI compliance rule other government mandates some of them have no real basis in security they're just somebody decided is important for their business so you need to define capabilities to meet those requirements the latest trend as I mentioned in the keynote earlier endpoint security the go latest greatest set a cool way but my favorite one is well your boss goes this every good security conference or any sort of conference comes back with a good bunch of great ideas and you're on the hook to figure out how to make it work so fun

little Hobie we use here to kind of justify playing we go we've built the lab to test a lot of stuff but ultimately built with detect to play around with a lot of different toys and a lot of fun stuff so research is what I'm doing what I don't know what I'm doing all right so before you start building a lab you got to have some sort idea what you're trying to accomplish and with my background in software development I've come from a very well-defined process for testing and evaluating capabilities it usually cross over to Quality Assurance ultimately a test plan map to some sort of requirements but the challenge is when you're looking at new products that you

need to know you need it until the vendor walking it or you know you have to kind of develop those requirements and see if they're legitimate to your needs the best way to do that is to roll them into a test plan understand what sort of objective you you want to test or achieve with those new platforms new tools and ultimately you want to make sure that you perform all of your test activity but a very concise repeatable fashion so that you have can have some level of confidence in your test results because historically if my experiences with a lot of the guys we have in our cybersecurity engineering team they're always chasing the latest cool thing

that pop up on their screens or they start down a path that you're running executing a bunch of tests run into some obstacle and then pivot to something else that shows up or they run down that rabbit hole next thing you know they're trying all kinds of new configuration changes and your entire test set your results have just completely gone on vacation there's no basis for interpreting and at the end of process so the sharper you can define those test cases or more boundaries you can put around the execution plant of those test cases will help you in the long term interpreting those result sets you also do not want to complete it will blow

yourself it's trying to test everything all at once you can have multiple iterations of a test plan focus on specific functional areas so that you're not trying to test things that may be stepping on each other and the more the more points you have testing and test plan the more work you're going to have to do to analyze all that data in the back end so while you may start off with the idea of testing X Y and V if you test a through Z before you can get any determination on the XYZ you have to test all that other side which may ultimately usually is not going to inform your decision as a finding so the main point here is to

look at your organizational needs figure out what it is you're trying to accomplish with a particular product to find some metrics that would let you collect information that help inform your decision about viability for that pod there's metrics google metric all you want there's all kinds of stuff different metrics mean different things on the compiles packet loss is a big one that shows up in a lot of security analysis products throughput packets per second and with monitored segments you really need to figure out which ones are most applicable to you and focus on those because once you figure out what those those important metrics are you then have to figure out how to inspect for them how to monitor them how to

instrument your tests to collect information about those metrics or what those metrics really mean because just a metric on a page is just a number you have to understand what that number means yeah yeah be able to communicate that meaning to your peers your management staff and let them understand why people oh yes gokhan requirements my favorite topic you have this conversation on a weekly basis because the tendency is to run down the rabbit hole we pull the threat figure out what where it takes you but ultimately that is kind of productive to XPD a text it's perfectly acceptable make notes that come back and it looking investigate those issues or those topics but do it

outside of a particular test we treat our testing process very much like an agile software development model we schedule Sprint's better identify what topics we're going to test during a particular period of time usually two to three weeks well let test plan that fits that cycle if we have to test more than 50 and what we can execute in two to three weeks that's more than one test behind also it and then we can prioritize the results of the different test plans and merge those results analyze them at the back end of all this yeah Eltham Utley you're going to watch your high-priority stuff first though you're going to want to you know use a

bail fast approach so the finance cope is number one you also need to understand what type of environment from a network perspective kind of operational mode these systems are going to be under how many staff are going to be maintaining them because those all need to factor into your test criteria I've yet to see a commercial or open source product that behaves exactly how they advertise maybe that's just because we tend to implement it a little bit deeper than the and their target audience but ultimately there are thresholds which those devices stop performing correctly so failure is eminent it's going to happen you need to define what your minimum acceptable level threshold is or and here's

performance metrics ideally it's going to be beyond what your current operating capabilities the other part of that is you don't want your your security team to be run around looking at a bunch of different tools constantly want them to be able to focus their investigations you know the market likes to use the term single pane of glass every vendor sells you a single pane of glass but when we buy 30 single panes of glass you have 30 days so the integration piece is key you need to identify what the interoperability requirements are within their environment you standard way that is handled with blogging syslog other things there are commercial products about that later API integration so

restful api so yet anybody still good so probably don't wanna talk to them epub dated orchestrations key is it when we say orchestration what we're talking about in automation so there's there's a lot of tools out there right now how doing the orchestration so you got puppet chefs all danceable those are all around the automating system deployment system configuration there's also orchestration tools around Incident Response which is about retrieving information from tools those are all very important topics but when you are defining those focus on which side of that year on for us right now our immediate needs are more along the lines and system administration stuff integrating making sure that when we push rules up a systems they actually

get loaded the systems actually ingest them and report errors and the operating status we need to patch you don't want to take three buckets passing to be normal everyday operating process orchestration is cute a lot of that yes this is this is an air our team went around around a little bit because we were trying to develop approach that lets us repeatedly test products these that set up the test so that we can easily get new equipment in and out without having a completely real rebuilding of the structure because you know testing had always been an ad hoc thing as well as a oh yeah we got to do this it's very responsive so we pull a

few guys aside set up a couple of systems that generate you know 20 gigs of data flow different people are doing it every time as inferences so we did is we developed a testing methodology which basically encapsulates how we go about executing tests it's not the test plan this is about it's kind of like a lifestyle really because it's about starting small scaling up to in that unit test for software developers anybody expect that background but testing concise things and then scaling those and then incorporating more complexity James will talk more detail about that but it's really about you know developing a process to execute tests a process to collect test results and data and analyze those test results

and report on those because ultimately we need to have a report comes up again that influences a either a buying decision or building decision and then we need to have a standard way of so it's the only thing that methodology is probably we spent three or four months after spending a few years testing products developing that methodology and now that we have that methodology below that we just focus on testing all right so finding the weaknesses and strengths of products this is what we like doing this is why we get out of bed you like crushing the dreams of vendors we like sending sales and marketing guys crying because they all come in they call come

in claiming to do everything they want next-gen firewall who here knows everything a next-gen firewall supposed to do what makes it next-gen is it the same team that manages your current firewall is now going to be managing your firewall and your IPS devices that usually doesn't work the marketing team says it's one person doesn't it standardize on your firewall but the route is once you start pushing IPs rules on your firewall the first thing that is done with performance so you've got to figure out what types of functions work well on those products which ones vote separate the fact from the fiction we found several devices that they both very high performance throughput metrics and turns out that

those throughput metrics are based on pure packet rate there's no analysis way out we've been talking if we each are pitfalls I didn't tell you some specific cases that were really funded but by what we found a lot of areas where certain sensors work really well with certain types of detection and analysis rules while the vast majority of the ones we use image didn't work at all you spent a year and a half fighting with vendors over product codes that's that's kind of what shield this whole process we would Stan she ated develop this test logic test methodology so that we can pick those battles with the vendors that sell us stuff and we had a strong basis for

making our claims without logic yeah so performance testing estimate it's a fun topic really because there is the raw performance testing which you see those metrics all over the place there on Gartner Forrester you know this device will be 40 and a half million packets per second sustained it up tell you is you know that all those packets are the same size into your small size they don't tell you what types of validations are in place for those packet counts they don't tell you what helps the blocking being wants to choose you're just doing patching packets so what we tend to focus on is well what we're referring to is rated with phone records we want to know how a particular device

performs while handling 20 gigs of throughput with you know IP based signature rules voted on it well rules other types of attentional because those are the those are the normal operating conditions for our system so we can plea by that protein athlete in packets per second box put it on our network connection monitor a three gate connection you expect it to do really well most cases they don't toolbox so one of the biggest challenge we had when standing up our test lab was kind of wrapping our heads around all the different types of tools that needed to build this lab and along with all of those tools and a lot of skills that had to be developed the outside that people

left who won it took us a little while we came up with a pretty good set we started testing and realized hey we didn't really understand how these tools look so you stop and start to get so we're going to cover a couple areas where that we feel are the most instrumental parts of having a test lab for these articulations but before we do I do a lot of stress make sure you're proficient with your tools you can gain some proficiency while you're building out your test cases other when it actually comes time to executing your tests you need to make sure you understand what your tool is actually doing and how to make it do it I need to

do because your lack of proficiency in a tool can actually skew your test results so you will not end up with the shiny needs great product that you wanted the beginning so take the time invest in yourselves in the team to develop those skills that is you're going to have to build those with those testcase execute those tests case tune those test cases troubleshoot those tests case is there's a lot of failures in this such barriers of every level we need to be able to figure out which part of that process is causing the problems and then the last bullet here is before you build any basis of credibility on your test results you need to make sure that the tests are

doing what you think they're doing but working the way you want the board does otherwise you'll build a whole decision thousands all right so here's the high-level list of tools set so we start off with test orchestration this could be a number of different things there's also a traffic generation piece your data acquisition and analytics and a whole myriad of a good layer didn't have existed so the next test orchestration so this this could be as simple as scripting bash scripts Pro scripts shell scripts PowerShell scripts fear of that like Python is very popular there are also a lot of commercial tools out there the main point here that we're talking about on this test orchestration

is you need to be able to coordinate all of the layers of your test process at the same time so if you write a script to execute your test plan or a single test case you're going to want to generate some log data that says hey this test XYZ has was initiated at this specific time and here's the label and then all of your instrumentation points along your test lab will be also feeding data back so this will make your correlation a lot easier at the end if there's an error you know get that log and then you can actually see the act across your test level in that area curves and then when the test is done

this is the key is another key part you go long to stop but you can also clean up your next because you don't want you know debris from your previous test case distorting your next test case it's also very helpful when you have more than one person to in a testing to use this approach because then you can keep up these test requests you can execute a lot of stuff in parallel long as you have enough artifacts to correlate at the end in pretty good shape there are commercial orchestration tools out there most of them centered on Quality Assurance tools I won't get into specific brands all up here the microphone being recorded but we can

talk about it later traffic generation this is one area we struggled with a lot the vast majority of our network that we monitored our a sub one gigabit but we have a lot of customers that are they 1040 under 800 gigabit page you cannot just assume that your traffic generation method all usable one gig and ten gig will yield similar results at 100 gig 800 gig and beyond you can also not assume that any of your test cases will scale linearly as well so if you get a monitor and a kind of gigabit connection you need to be able to generate other gigabits of data and that is not a small team so those tools need

to be reliable you need to know exactly what is the what the need to be able to control the flow of data you need to execute in the same manner every time you execute it ultimately you don't want your generation issues to skew your test results at the end of this right it does choir require specialized knowledge how to maximize your traffic generation especially if you go with commercial products you need to learn metal used a even more so on the open source stuff because there's not a lot of documentation out there that's available so the last two bullets here are what I consider vital on the tariff regeneration stuff it's one thing to generate traffic data or generate

traffic for analysis that is a random key cap or a key cap with a specific threat the problem is if that approach is a lot of the newer products will see that traffic coming across the wire and say and I like the timestamp on that hey this is a real conversation I'm going to ignore it so you're not going to get a positive result when you think you should this is a problem we haven't several commercial product security products because it would recognize that hey this is a simulated conversation this isn't a real thing it's not worth my time they're showing a graphic move on the other side of that is very rarely do your of these sensors are going to

operate in a world where you only have bad cups whenever they operate in enterprise networks small business networks internet connection so you need to be able to generate a lot of that noise which is anything from male traffic and Pandora's big streaming video streaming dated YouTube we need to be able to generate all that kind of content and then be able to inject specific threat live threats into that space so that the device knows it's live and actually analyze it here and then you can actually see how it responds to real data sample data get you half way there but final legs are going to have to use real live malware to do this analysis which is why we're

talking about safe testing don't do this onion production network if you can that'd be great give us a call of yeah so it's a couple different open-source traffic generators we've used TTC PvP episode of pulse up standard for everybody this is the one we ran into a lot of problems with because consistency issues it wasn't real data what wasn't providing real you have client-server conversations escapee it's another one ostinato astana was a really cool tool it takes a little while figure how to use it it actually it's a client-server setup so you actually got a master server and a bunch of traffic drum rack virtual machines running on your network it is they generate this

type of traffic and it will actually create traffic cones between those client server again commercial traffic generators here we use one you love it and I'm not going to endorse them this forum but after work like in all kinds of information right data acquisition this is this one's a big one to handle you're going to be generating a lot of data from a lot of different points you need to have a reliable ingestion pipeline and you need to have some pretty significant performance on the backend indexing idea so you can actually search and analyze the results typically you have set of log orders you put out everywhere syslog the main one everybody uses log stash is pretty

popular these days ago everybody knows the leading four most important tools there I thought that later to look so not everything that you're going to want to inspect or the data from is going to be able to forward log data and it also may not give you the level of granularity you want on that problem so being able to integrate with Web Services restful Web Services soap web services lets you pull or push data you know from those places lets you tie in in mecca performance metrics at many steps of the process and also lets you have a positive control over that communication box so that that can also tie in to your orchestration piece so

that you know every 30 seconds i'm doing a memory snapshot on the machine trigger that vien api it all back for correlation the elk stack is a great open source login analysis platform it handles your indexing is so loath indexing elasticsearch solar as a pretty interface implemented on a log stash its rival see the back of my jacket it's a very powerful tool requires a good bit of knowledge to have it performing one at a high level you need to know how to tune your indexers you need to know how to tune your search queries you need to be able to generate the analysis reports correlate the data and you need it's also hold on job demands that's act so

we need to have you know expertise to ensure the quality of that ingestion by that any analysis that comes out of it

so this is just a high level equivalent list the various things that you're going to need in your lab you're going to need servers these are going to each standalone servers running Europe whatever collectively or hypervisors runs the ends a lot of different varieties basically you're going to want to replicate whatever you're operational in there if you've got VMware as your primary OS back that's what you're going to walk in a Tesla VMware also you know lets you give you some flexibility of virtual appliances there are a lot of virtualization platforms look almost every security product out there ships VMware lDA's that you can use for tests to get into other products like ATM and

the other virtualization platforms you have to go through a conversion process because appliances early attempt to learn you're going to need a set of network switches most as mostly to ensure you know your management network access those as test components your log delivery and eight your data acquisition network suck all that data off the sensors and the device is under test and your traffic generation platforms we use a 10 gig network for that not everybody needs to though you just were testing high bandwidth also if you run grow one 10 20 gigs of data abroad such as everything turned on then you're going to want a 40 gig connection on one taps stands whatever you're using in your in

your production network if you're using taps to monitor your network segment use taps in your lab using span and spans these are both to use but because this needs to be basically be a scale model of your production environment whatever your monitoring whatever your securing you want scale representation the packet broker piece is essential if you're testing more than one product can credit you're doing one test at a time or one did I set it kind of not big yet it does give you a little bit more visibility into the traffic as it goes into the sensor and comes out of the sensors lets you slice and dice some of that traffic but what we primarily use

it for is replicating test data to a set of devices that are under test concurrently so what they may be the same product two instances three instances the same product so that you have a basis of comparison using deviations across that same stack or they could be competitors ultimately this is a force multiplier for us so if we are evaluating twelve devices we can iterate through three iterations of a test versus well one thing how dry it is up here and if last angles cables and SPS single-mode SSB won't people in Ossipee those are the environments you're operating in have those a lot of people wonder okay well it doesn't really matter what type of cable it is what

type of media it is but ultimately the devices that is plug into are different so just make sure you have a variety to supportive again you do

once again key considerations isolate this test lab from the production systems your lives will be much easier the best thing about it is you don't have to deal with a bunch of chrome pieces add nodes with anything it's no changes you can pivot as you need to you're not tied to the same configuration management process that your production network is tied to you can run the live malware on there it could blow away the whole thing and rebuild it if you want you need to be able to support the same protocols media types and network speeds that are you have in your production line we get read over and over and over software developer and Inter sorry but

yeah you do need to understand the constraints of that line you do need to understand that your longing infrastructure may be only set up to handle two terabytes of data yesterday 500 gigabytes CA you need it doesn't do any good to test beyond that because you'll spend all your time waiting there's an example where we ran a test model growth sensor took five minutes execute the test took 15 hours to ingest namadeva that was a problem a learning problem I hand it over to James who can talk about yeah so good afternoon James [Music] so you know it's big audio ok we did perfect so you know I still you thank Chris today that is um you know you know

our approach was you know starting with small that's you know and sort of incrementing up you building as we go so you know when we first started you're having that hoc approach and you know we're just executing accessible device you know the single you packet generation and then just sort of assembled and took that and said okay well you know we need to mature this as a as a process you know we built out our test I'll habit now as he was saying you know you know we're using 10 gig but you know this approach could scalp you the size of your organization and your needs right so you know it doesn't have to be

you know yeah it would have to be hundred thousand dollars and can fit to whatever your purchase

so here we have you know sort of a high level diagram sort of displaying you know how data flows through the system right so you know we have the test worker orchestration piece you know sort of you know kicking everything off you know you know and we have the 2 hypervisors with the VMS passing traffic through from one to the other and in the middle we've implemented attack we're sending data to the broker and the buckler is then you know sending data to the devices under tests then the data from you know the packet generation the broker the devices under tests all that feeds up to your lock correlation so and then you know it's log at the log

correlation you know we're having you know getting indexed and then on top of that analysis and the reporting and you sort of building those artifacts yeah

the instrumenting of the packet broker of sort of the our approach here is you know every every step of the way the packet is going needs to be instrumented that way you sort of you know if something goes sideways you see where one sideways now with this this diagram here all we're trying to illustrate is um if you're testing an inline solution right this is this would be sort of how you would do that right so all the devices under tests they're using the packet broker as the inline traffic right so as the traffic passes through the device under test it can drop or however can do that you know it was in do reset or what have you

all right data collection so you know one of the points we want to talk about is you know the time super super nation so um all the devices need to be you know with you know time scenes whether using NTP or what-have-you that way there's no discrepancies because if you get discrepancies timewise that that can really throw a for loop and will take hours and hours to figure out okay why why did packet a not get to where it was supposed to be well actually know it was you know blogs were looking at are not actually the correct logs so that's a critical piece there the B so as Chris was saying you know you can generate significant amount

of data and so you need to scale properly you know and that can be challenging could you know with products you may not know what that is you know so you need to talk to the vendor or you know peers as well can be a useful resource for trying to figure out how to scale your login infrastructure you know and then you know so we've made a couple mistakes in this process which is sort of informed our decision-making and that's so like with the logging infrastructure right so you know it can lead to misleading results if you are expecting the you know all your event data that's been being generated is coming in right after the test and think

of at least a couple scenarios where is like okay you know we thought that the device was performing under our limits but it was actually you know all our entha structure around it was causing the issues so it just considerations

analytics and reporting so you want to validate your test assumptions right so this sort of goes with the fell fast approach right so if you know I say hey this ideas expected to do XY and Z well you know design tests around X Y Z and try to break that assumption and if it you know if you validate okay it works local you know what I mean but you know if it fells you know that's a that's a data point that you can then use engage with conversations vendor or whoever right to see you know see what that narrative look like interpreting the data right so now data is just numbers right so if you don't do anything to you

know build these artifacts sort of build a narrative random you know it's really meaningless so you have to have to build that narrative and say you know hey we tested X and you know it it ran really poorly in these scenarios but the reason that was is because X Y or Z that we under tuned or we over tuned or what-have-you so just capturing the narrative is important for now sort of building that a knowledge base so use peers to validate conclusions right so um you know if if during your testing you're making configuration changes and you say you know we made this change and we think this is gonna make it perform better because of these reasons but I'll

leave that with the teet say hey I think this and they'll say does that make sense to us let's let's do that and then you know they'll you just build a consent and you know sort of smoke check your conclusions so visualization of the data that's coming out that's right you know it's helpful you know graphs are pretty every likes graphs don't go overboard that's pretty much I'm not going to be more to than that and I'm just done quick crazy with it and then finally you know pressing the big red button hitting go sending packets executing on your test bond you know find the limit of the system break it break it again and you know blue that's a data like I

said earlier now when you see where the limits are the system seems sort of define the normal operating injuries right and then additionally you can define sort of the out of normal vectors as well well what happens if all the traffic's for our pens is what happens if all the traffic is now segmented or what happy so and then as you do that being skelet back in your normal operating procedures and see how it performs lat so details matter right so as you're doing this you want to be generating artifacts about what you're doing you know and you know the framework that you've developed should help you with that logging infrastructure etc etc but you know just

document stuff I excuted this test at this time these are lizard results and or you know it fell because of these reasons then on top of that you know monitor the tests and the components and sure test components are doing as expected and then if you do have errors investigate those errors try to do some root cause analysis you know sometimes that's always not successful you know sometimes you know the vendor has the black box and they're not going to tell you how the box ticks but you know sometimes that will provide you know further points with which you can yeah use this data and then you know just iater ate you know keep testing and

building that sort of repository of knowledge you know you basically are familiarizing yourself as a product and these tests just give you a quantifiable measured weight in which to do that and as you iater ate you know tweak to make you know because some of your assumptions might be faulty or ate like we expected it would do this and that wasn't the case so let's redesign the test to meet the valid conditions which this will operate so on this and this process takes time right so you know you're going to have to go back to the you know the scoping requirements yeah so you need to sort of set expectations but okay this is is

what it's going to take to you know test and build repository of knowledge to make a correct product selection you know we could take the easy way and you know just say vendor a but you know we're trying to quantify what that looks like and quote from Adam Savage here because he's pretty cool so the only difference between screwing around and science is writing it and I think oh oh yeah I think that's pretty much is unless what we're done for time in a few minutes okay well the next one starts at two three

anyone has any questions right now what I guess would be the time to sort of let it fly yep go ahead so that's an excellent question he asks the tools and the methodologies that we built is that gonna be publicly available I think Chris as mentioned we won't really talk about like commercial tools just because we don't want to influence anyone but what we mean I think we're planning on releasing the slides and maybe the framework so yeah we had a few instances where people were threatened some lawsuits about endorsement was done so we can talk to you individually it kind of give you that information more than happy to do so you can also we've got our contact

information I think on the next slide where we really seen the slide deck and along with our speaker notes which have more details as well but you can also reach out to us and we're going to have that dialogue or give you more specifics about what exactly we're using we just don't want to provide any sort of free endorsement and a recorded poem yeah this is our we both have Twitter accounts we all have emails you can reach out to us anytime practice afterwards we can give you the whole rundown we talked about this set up to a lot of different people on a regular basis so we like talking we'd like to know what you guys are using as well

anything else sit all right thanks for coming we came up from Las Vegas I don't think we got that out to you guys we're from Las Vegas so it came up here just hanging out with you guys [Applause]