BG - Lessons From the Front Lines: New York City Cyber Command - Colin Ahern

this is Collin Aaron and lessons from the frontlines great thank you firstly thanks everybody for spending the last talk with us it's really really appreciate it I'm gonna try very hard to make it a little shorter than advertised so we can all get to the important stuff like a pool party firstly standard disclaimer I do work for a government so these are my views not the views of the government I represent these aren't official policies these aren't official positions these are my thoughts and my thoughts alone so please take them with that in mind here's the agenda for today we will have time for questions at the end as she mentioned and you know firstly Who am I and what is New York

City cyber command and what makes that organization special then I'm going to talk about some potentially useful frameworks that we use to help us frame this really really really gigantic problem some of our thoughts on security governance and Incident Response and why those fields are really closely related and what we're doing with data science and research and why you know a lot of problems are actually data problems in disguise and how we as the organization that is charged with defending the government of the city of New York's networks thinks about future architecture of choices that ensure reliability and security because ultimately while we're aiming to do is provide reliable services to residents visitors and businesses that's the name

of this game you know so joining us on this journey really really appreciate the folks at besides for letting us talk about it firstly how many people live in a city any major metro area at all and how many people live in New York City that's for you great so I think it goes without saying that firstly New York City is the best city and secondly that cities are becoming increasingly interconnected things that we're doing as municipal governments and businesses were unimaginable five or even three years ago in some cases so throughout this talk I'd like you guys to think about if you've never considered public service consider serving in the public consider a job at

New York City Cyber Command or if you live in another city you know your municipality you know there is no secret team that is gonna solve these problems you know the keynote that Josh gave he really couldn't said it any better you know the people in this room the people at this con we are gonna solve these problems together or they're not gonna get solved so really you know as we talk about this we want to share with you the way we think about this problem why we're doing interesting things and why we think that you can contribute either you know signing up for one of the one of the jobs we have posted or continue

to be involved in your community in a positive and constructive way but as for me I live in Brooklyn with my wife I like long walks on the beach supplied crypto public service and ten string flows I have degrees from Tulane and NYU and I spent a number of years in military intelligence in the US Army I deployed twice to Afghanistan and I dip it and I commanded a company at US Army Cyber Command after I left government service I worked the security engineering and cyber threat research at a major financial institution and I think like everybody the main takeaway from this slide is that my github does need updating so if you see it bear that

in mind so history of cyber security and history of computing in the largest major metro area in north america new york city actually has a long and storied history of using computing to deliver city services in fact there's a member of our staff that has been on the mainframe team for forty-five years so when I say that New York City has a long history of computing excellence I mean a long history of computing excellence and in fact not the same actual mainframes but those same mainframe applications and their descendants continue to provide services there was a great talk earlier about mainframes on the internet you know so there's a very real interesting technical challenges associated with mainframes and I can

geek out about mainframes for a long time but I won't but I will say is that that longevity and that history is is an advantage well we see that as an asset we also build on a long history of security at the city's Department of Information Technology and telecommunications which was formed in 1994 and the department of info tech and telecom is known as do it do ITT is led by the city's chief information officer and is the city's technology core it's also the city's internet service provider they have hundreds of miles of dark fiber cable connecting city agencies hundreds of points of presence and it's a high-speed dark fiber network you know outbound internet connection

and you know one thing that do it the Department of IT does that you I hope have heard of his New York City open data that was in 2012 a local law made available huge amounts of publicly generated data for anyone anywhere and that really started the municipal open data movement in a real and important way and some of the things that I'm gonna talk about with research and and other things we see that as our contribution as our role as you know the city's cybersecurity organization but really part and parcel is supporting open data and that includes going awesome talks at besides and that data by the way is hosted by city-owned systems you know if you ask for New York

City open data you are getting data from a server probably not a mainframe but definitely a server owned by the city of New York and you know that's inspired some things as countless student projects doing citibike data you know really the and behind the scenes have open data is the huge effort to make that data ready as all good data engineers know 90% of the boring part is getting the data ready and 10% of the fun part is running the model so you know this is just part of part of the city's history and it's legacy so New York City cyber command as I said descended from do it formed by executive order of Mayor Bill DeBlasio on 11 July

2017 and it's cyber defense and Incident Response for New York City government and New York City government just at a high level is a hundred and forty three separate organizations everything from 10 person policy organizations doing cutting-edge policy research and immigrant affairs and women's health and other things to 50,000 New York City police officers 30,000 New York City Fire Department and all told it's about 300,000 employees and if the 300,000 employees of the city government were their own country they would be as populous as Iceland and if the city budget which is about 85 billion dollars where our country itself it would have approximate the GDP of Lithuania so when we talk about large complex

organizations we are talking complexity at a scale that is really more akin to a nation state than at a local government and the network itself as I mentioned is also quite large there are 3/16 Class B public IP addresses that's almost 200,000 and you know that connects over 400,000 devices from traffic lights to mainframes to server serving micro serviced applications on the internet today so you know very a wide breadth of technology excellence so the real meat of this executive order and it's riveting riveting stuff I encourage you all to read it is that we do you know we do 5 primary things one we ensure compliance with information security policies and standards 2 we

mitigate cyber threats 3 we mandate deployment of technology and administrative controls on city-owned systems for we review cyber spending for efficiency and effectiveness 5 we collaborate with governments and outside entities like giving talks at these sides and additionally we provide guidance on cyber defense and information risk to the mayor and city agencies and there have been some great talks about policies and one of the things I want to spend a couple minutes doing is why this is special for a.gov of such enormous scale and I kind of sketched out the enormous scale of the city government and budget Authority is really important and what I think I want you to take away from the budget

authority pieces for a gov meant to give a cyber defense organization budgetary approval is akin to if the Director of National Intelligence could tell other intelligence community partners what they could and could not spend their money on and you know that's a major criticism of the way the intelligence community reorganized after 9/11 and was a recommendation of the President's Commission on enhancing national cybersecurity which was released by President Obama in December December before he left office so you know we the city of New York took those recommendations very very seriously and integrated them into this framework and too centralized ability to deploy tech how can you defend a whole city if you can't see the whole city and you know

given the large complexity and Federation of this environment you know kind of contrast that with the federal government whereby the federal system at the moment sits in OMB but there's a DHS relationship and the end kick and you know there's a lot going on right so this move to centralization and unification is also right out of the committee get out of the Commission on enhancing national cybersecurity and a cloud first strategy so we're are have a number of initiatives cloud first we're working with a major cloud provider on a data science and Incident Response environment of which I will illuminate a little more later and really this is about being agile being flexible and being responsible being responsible with

the the resources that the taxpayers have been generous enough to give us to solve this problem and us being agile enough and agile enough for any organization not just a government to apply cutting-edge tools tactics techniques and procedures in an environment of high reliability and high security and so how is New York City Cyber Command organized at a high level we have small unified teams with a clear mission we have three primary lines of effort one is a threat management organization that's where the Incident Response happens there's a Security Operations Center which operates 24 seven it has forensic analysts there's assert sake analysts threat intelligence you know threat hunters taxers you know there dwells all manner of people

working hard every day to combat cyber threats on behalf of the city - we have a security architecture and engineering team that's where our data science team and our data lead data scientists work and those that data science team directly informs the threat responders in that tower and the executives in the governance tower there's nothing worse than having a data science team or the insights they generate don't end up eating anything so our commitment to our data science team is you produce insight we make decisions and that's something that for if you're looking to build a set data science team in your own organization we think that's a necessary predicate to that being successful we

also have security engineers including identity engineers firewall engineers and security architects lastly and and maybe most importantly at some some level is governance we have a policy organization application security folks you know governance specialists grant writers auditors you know all the things that really turn theory into practice and that's incredibly important and one thing we want to highlight on this slide that we think is unique is we manage talent via an alumni model so New York City's Cyber Command isn't the place you have to enter when you're 22 and leave when you're 60 although Frank from the mainframe team is doing great and wonderful things every day not everybody can be frank and we recognize and

understand that and so you know our commitment to our staff is that if you're here for two years or 20 years or 45 years you can contribute that entire time and that takes a you know that takes dedication from the leadership team that doesn't just that doesn't just happen and so that's an important part of of us and how we think we can interact with the outside community and if you are considering a career in public service why an organization that thinks like this and talks like this might be more appealing than an organization where everything is highly dependent on tribal knowledge things are not very process-oriented you know these are things that we are

actively trying to evolve in our own organization because we are very committed to working with outside partners you know be they people who work for our organization or people who work with us so kind of with that background out of the way want to spend a few minutes on operational approach and operational design and though this is a you know is a hacking conference the difference between a great idea and a successful project is operational design and the operational design that we use is called Hayden's principle named after General Mike Hayden who is the director of the CIA and the director of the NSA as well and the purpose of this operational design is to minimize

the impact of magical thinking and hack tip tomara and her phenomenal talk early this afternoon for those of you that caught it and the first thing to avoid magical thinking is operational relevance to the bad guys who care about me care about this that is the first question we ask ourselves and we undergo major transformative efforts like New York City Cyber Command like major technology roadmap journeys and threat modeling there's been a lot of great talks here about threat modeling there's a lot of great ways to do threat modeling but having a codified practice surrounding threat modeling making your assumptions explicit obviously all models are bad some models are useful and having a useful model that people can measure

themselves against makes your threat models more reproducible more data-driven more quantifiable and when we talk about data as a problem you know these need to be inputs to some model someplace ultimately and when we talk about operational relevance it's not just the good guys that are doing behavioral things and doing machine learning things you know the velocity and variety and sophistication of attacks has really increased and so operational relevance really needs to be taken at machine speed technically feasible can we actually do this today and if we can't what are the things we need to have tomorrow in terms of resources people time and money and what you know that Delta and how do we how do

we bridge that gap and there was a great talk earlier by you know policy and procurement how relevant that is we work very very hard as I said to safeguard the resources of our taxpayers and that includes thinking a lot about the relationship between technical feasibility and our ability to procure things so you know if for those of you in government are considering it that is a very real concern and political sustainability and the reason we put this one at the bottom is because for practitioners of some persuasion this is both the most boring and the most important part of you know operational decision-making and political sustainability I don't mean politics with a capital P like elections

November I mean politics with a small P like fitting the culture of your organization and meshing that with your operational design and seeing executives as key stakeholders in your operational design executives aren't people for you to manage executives are team members who have a valuable input to how you do operations and seeing them because this is the thing they do if it's not clear to you what your executive team does what they do is have an extremely keen understanding of what is politically sustainable and you know if you don't include them in designing your strategies it's unlikely that you'll get their buy-in or that this will succeed and moving from a hero model to a team

of teams approach and last and certainly not least legal the legal landscape and cyber evolves every day and you know lawyers are force multipliers that like if you know what is legal today might not be legal tomorrow you know legal risk is the thing to be managed by professionals so I thought I read it on Wikipedia and not a sufficient answer I checked with the general counsel submission later so governance want to spend a couple minutes on governance and you know New York City greatest city in the world that's looking south Freedom Tower in the background like I said we have agencies that reign from 7 to 50,000 people and managing and unifying those efforts is incredibly

challenging but the upshot is that if you work on governance at an organization like this even for a little bit you can change the way people talk about this in the city government for years to come the words people use the way they think about problems the tools they think they have at their disposal and as a technical professional that's incredibly powerful you can change what people think is probable to occur and when we think about problems looking out over this great Vista connected devices in smart cities how do we think about governance and its relationship to technical feasibility when your watch can report a crime to the police and what is the difference between data integrity and

disinformation and how do we address that technically and from a policy perspective firstly this quote this is what governance is if I could give you what is governance governance is in theory there is no difference between theory and practice in practice there is this quote is often misattributed to Einstein but actually was said by this guy Yogi Berra Yogi Berra New York City gotta have a Yankee and governance is making the theory and the practice as close together as possible an incident response one of the things we think is unique is guilty until proven innocent and that's supposed to be the back of a monitor and that's a layer 2 network contained for those of you that don't

get my amazing clipart skills and how do we contain a threat while balancing accessibility and reliable access of those services or residents if you think you can figure that out as the attack is unfolding you are probably wrong if you don't have a governance structure that addresses how these decisions are made that has executive support that is legal it is unlikely that this is going to work what you're gonna end up doing is being frustrated getting pwned and being sad so why we put this after all the governance stuff as though this is the cool part this is a pointy end of the spear and like I said you can see our organization we spend a lot of effort on

but we spend a lot of effort on the governance part because they enable these people to do their work they enable them to make decisions within a framework that is mutually intelligible and you know collectively exhaustive insofar as the inputs that are needed for the decision-maker of an appropriate level to make it's one thing to say I'm going to network contain some endpoint that just got fished it's enough to say that I'm going to turn off the server serving nyc.gov so if you don't have a framework for deciding how both of those decisions are made it's important and who thinks that all of Incident Response can be automated maybe so Incident Response is making decisions and to make

decisions we need data and the question we ask ourselves is can a robot do this and you know we talk about cybersecurity as a data problem decisions need insight insight in these information and information needs data so things really when we get to Incident Response and I talked about their relationship to the data science team you know an extract normalized transform data ingestion pipeline isn't just like a cool thing it's just incredibly relevant for the incident responders and you know how do we decide what we are automating and why I'd like to spend just a couple of minutes on a framework we use to help us allocate our scarce resources of automation obviously we have an

incredibly diverse organization not everybody can do automation and maybe a lot of people in this audience can you know swag a Perl script write a bash you know write a bash script or you know repeatedly do some repetitive task but in the large organizations that's ultimately a finite resource and it's not about making one person's life easier it's about advancing this mission and so how do we think about that this is a framework and for those of you many of you have probably seen it it's called ofin although I'm sure I mispronounced that it was developed by Harvard researchers in the late 90s to a to optimize decision making and how we use it is that you know most of our

problems are called multi-modal they're made up of more than one of these four things so different parts of our problems should be either automated entirely or orchestrated in order to enable our human responders to maximally apply their effort and talent and this is obviously related to the doodle-loop observe orient decide act which is made famous by the Marine Corps and you can see a lot of the similar words and I'll go through it and not everything can be automated yet although I do think it's useful to look at this this is a framework and obviously it has its shortcomings to look at this within the context of what your data science team can provide for you and what their

relationship is in each one of these areas because obviously different parts of different components can be automated in different ways but very briefly the first one is best practice I can get my pointer to work yeah over there in the bottom right and this is the area of known knowns you know it is a stable situation where cause and effect can be observed from the outside week you know and the way we respond to this is we establish the facts by sensing them categorize them and then apply the best practice should I accept an email are they in spam house no best practice should I allow connection to this IP are they known is it a no malicious I feed

no okay don't this is also you know standard operating procedures things like never draw to an inside straight I always bet on the pass line of a crafts table find the proper rule and apply it helpdesk tasks scripts you know when you do a task more than twice you write a script for it that's this evolving this you know we have the area of good practice here this is also called the complicated domain and this is known unknowns and sense analyze respond to what we mean by that is there's a range of right answers but they depend on some context outside of the facts that are immediately visible so we assess the facts we analyze them to establish them

their context to me and then I apply a good operating practice and an example of this is intelligence analysis the legal profession software Engineering in certain contexts and artificial intelligence does well here you know narrow AI we ki you know big blue obviously or deep blue doing things in chess and alphago you know over the last year you know we see too you know artificial intelligence and data attacking problems and automation doing things that were not previously thought possible but it's reducing those problems to this domain in one view and thirdly emerge in practice or complet that complex domain is unknown unknowns and cause and effect can only be deduced in retrospect when you're in the middle

of unfolding an incident or you're analyzing a new piece of malware eventually you will determine cause and effect but as this process is unfolding it is not apparent to you and this is where we need to allow experimentation and in this constructor experimentation is called probe sense respond but this is where we as leaders and organizations need to allow people to spend time experimenting and screwing around why do we want to allow them to do this why should we allocate scarce resources on experimentation because if we can move emergent practice to good practice but eventually the best practice well then I definitely can automate it and so that's a way to communicate to executives and

say I have these guys just sitting here doing nothing no sir they're not doing nothing they are experimenting in order to increase the efficiency of this entire effort and finally there's novel practice this is also called the chaotic domain and this is where cause and effect are unclear even in retrospect so you find things like a battlefield a disaster the middle of a complex incident where you're not even sure if you can reproduce the issue or where it started and this is where you need to act in order to establish order in this chaos and then move out of this domain so acting sensing and responding means if the whole world is to being turned

upside down the most rational thing to do is start acting to immediately apply order to this situation and again this is just a framework but we use this in our operational design for Incident Response because it allows us to contextualize what we're doing with Python you know at some level all automation is some Python script via some API but if we can take it up one level we can really evolve the way we think about it as an organization and data science and research moving you know moving along but we talked about threat modeling New York City cyber community's collaborating with the University of Maryland we have a paper in draft doing enterprise level threat modeling and

theory that's our commitment to open data we're happy to partner it's actually Roch Stevens who has come to be ties for many years if you know Rock he's working with us in this effort and hopefully next year rock and I can come back and talk about the paper and some more deep in some more detail but threat modeling is also not a thing that happens in a vacuum or should be done in secret so it's it's important that as a community we share the right things when we talk about information sharing and data and research and open data in this you know obviously we're committed to sharing ephemeral indicators of attack and compromise like everybody else but

why we wanted to go in this research journey worth of the University of Maryland is that we want to share things that are hard for the attacker to reproduce a DNS or an IP is easy for name your attacker to get and throw away but their modus operandi that we can deduce through advanced threat modeling is hard for them to to discard so that's part of the reason we're on that journey and working with a major cloud provider on a data environment and obviously that's that's a thing that is important to all the things we discussed so future architecture visibility without quote actionability and I'm not sure that actionability is a word we have some

debate about whether that's a real word or not but somewhere that we use we can make up words we're New York City cyber command having something I see but can't do anything about it's not useful to me so when we talk about future architectures it's not that we want visibility we want to see things and then stop those things if it is appropriate do so so no orphan alerts nothing that is on span that could be in line and again when we talk about all the things leading up to this why is it that some organizations have problems putting their email appliance into a block instead of you know transparent PCC it's because I will saw at some level a lack

of communication about why that's important and not integrating your decision-maker into this process from the beginning and you know the success that we've had is you know do you want me to tell you something bad has happened or do you want me to tell you I stopped something bad and obviously the latter and the maintenance of logs for retrospective analysis obviously breaches get harder and harder to detect so when we think about architecture choices and visibility we think about maintaining logs and telemetry and you know net flows and other things like that for retrospective analysis not only for our you know our big data but also again we think about automation and orchestration so we can experiment so we

can learn new things and perimeter focus cyber security so in the bottom right that is the pinnacle of military engineering of the middle of the 20th century that is an actual fort that was on the Maginot Line and the Maginot Line was a series of fortifications built by the French government the French military after World War one they took the absolute greatest minds in France and they said how do we defend against an invasion that we we know is coming we know it and they said we'll build a bunch of forts but I mean obviously what did the Germans do they went through Belgium and the fort's were irrelevant so when we think about the future of

architecture we think of zero stress networks and we are actively building zero trust networks for our responders because we have the same concerns that these other organizations that have given talks over the last few besides we want reliability and we want security and not only that we can't be dependent on these models that have been proven to be inadequate over many many years so it Network segmentation firewalls are incredibly important and we seek to do them very well but we also seek to evolve our architecture and be thought leaders within municipal but within a.gov by adopting zero trust network architectures and you know our view on my view on zero trust networking is that it is really approaching this

from an information dominance perspective there's a saying from where I used to work that the moment you know more about their network than they do it's not their network anymore it's your network so on some level this is an information game this is a game about data so what zero trust networking makes possible or even sometimes probable is that you might at any one moment know more about your network than the adversary because only you know the sum total of your devices and your identities you know you only know the union of those two things whereas if you have a traditional network architecture if at any one network segment an adversary an adversary can achieve information

dominance over you how do you know that what they're doing is bad you don't and that's why you know the capture the flag you know then we can see this attack and defend scenario play out again and again and again and at some level a way to think of it as an information dominance game and when we communicate to our executives when we say why is this crazy thing important we say well this is an approach that it's not a perfect approach no approach is perfect but it is approach that approaches you know probabilistically a model that might work and if you're not sure what kind of problem you have it's probably a data problem and my final slide public

service is really really important like I said there's no secret team that is going to figure this out there's no magic answer the secret is there is no secret this is hard work and you know we have no glory here only the honor of doing a hard job well and then when I see Govs last jobs we are hiring cert team data Wranglers second Janiero at hunters policy wonks you know I encourage you to think of public service not as a boring thing that people you don't really like do but as a thing that you could do to advance your career broaden your horizons and do some cool stuff so take some time to not only think about this

no organization that I work for and that I'm passionate about but whatever however you can contribute to to solving this problem that to me is what b-sides is about and why you know the first time we talk about this New York City I recommend is that besides that like we chose to do that because we believe in that kind of community involvement so that is you know again thank you guys so much I'm gonna do the best I can to answer questions in the nine or so minutes that remain and again you know my thoughts and my thoughts alone and with me is my partner in crime Andrew so if you guys have some questions happy to

try to feel them thanks again anybody I

was wondering about earlier in the slides I think he said something like you basically enforce policy and I was wondering what kind of relationship you have to the creation of the policy and who creates it yeah great question so New York City Cyber Command creates the policy and we enforce we ensure we I would say we ensure the policy and we ensured via administrative and technical control that makes sense yeah so it's not some other person who's never met an incident responder policy yeah so that's good question so to reference back your automation versus orchestration have you guys utilized any of the neurological networking to advance some of your threat modeling if you will or neurological and

neurological yeah so yeah we're using a variety of models what I would say is our data scientists he seeks it he and his team seek to use the model that is the most efficient not necessarily coolest model although recursive and convolutional neural networks are a part of you know the data science that we do not only because they're sweet and they're fun to say but also because for some things they are more efficient and you know they do generate a higher ROC curve yeah so short answer yes long answer not a priori because they're deep neural networks but more because they're the most appropriate model for that problem and one of the things that he's

really excited to explore is what is the relationship between those models and you know obviously generative adversarial networks that have been talked about before you know using these models together and providing the the system you know the model generation itself reasons to think of things that you couldn't have thought of because you're not a computer so yeah so just trying to fit the model to the problem not the other way around I had a cushion on your data science practice I mean looks like you're really building your data science capabilities it but I didn't see any when you showed the slide on open positions looks like you are you're not hiring for data science it's more for data wrangling and

yeah so we are building out the team we are we will be posting more positions to digress very briefly in the boring city stuff the city does its hiring process these four times a year so if you don't see them now you will see them later and if you connect with me after we're always looking to you know have resumes and have relationships with people that want jobs that when they open up we can make sure that we connect so thank you question obviously a very large and very diverse organization you'd mentioned the importance of visibility having the information you need so you can act what are your biggest challenges in achieving the desired visibility that your team

requires and can you share a little bit of how you've achieved some of those goals and what's been the most operationally effective ways that you've met your visibility goals great question so the the way we think about it is we have really one problem with two parts we have a visibility and actionability problem so the way we've done it is addressing it at the endpoint and the network edge one of the advantages that New York City has unlike other 300,000 person organizations of you know that I have 85 billion dollars everybody's within the five boroughs so our network edge is actually relative you know relatively speaking the large is finite so you know we can get

visibility there and actionability there obviously I said the city has an isp you know so you can think of it that only serves city entities and additionally we can mandate a technical control like an endpoint agent so the ways that we address that is is those two ways working in concert with the city's network and mandating an endpoint agent so good question anybody else great well thanks everybody much appreciated [Applause]