Fundamental Understanding of Baseline Analysis for ICS

Please join me in welcoming Miss Julie Joiner and Mr. Jeffrey Mezer. [Applause] Good morning. I'm Julie Joiner and presenting with me today is Sorry, you're not um on the big computer. Okay. Laptop. Okay, never mind. Go. Technical difficulties. It's all good. Before I get started with our presentation, um, I'd like to introduce you to some of the rest of my cyber security team who are here with us today. Um, Steve Blake, Sean Nixon, of course, I introduced Jeffrey. Um, another one of our members is not here today, Larry Sprouse. And we of course have our fabulous GRU intern Harry Zayn who's also um helping us out. We're very lucky to have Harry as our intern. Um

how many of you guys have anything to do with securing supporting um dealing with industrial control systems at all? Not very many. So what made the rest of you want to come hear this talk? was young and full because I probably will at some point. Excellent. Excellent. So, um I hope that Jeffrey and I today can give you um some good starting points. Um we're still starting as well. As you'll see, we're a very young program. Um so, we'll we'll hopefully give you some some things that some ideas that you can take back for your industrial control systems yourself. Okay. So, this is uh just a brief overview of what we're going to cover today. I'm going to talk a little bit to

you about Savannah River Remediation, our company, who we are, what we do, where we're located, and the importance of the industrial control system environment um that supports our company's mission as a spinner river site. Um we'll talk about the challenges that we all face um when incorporating cyber security into those existing ICS processes. Um some successful security configuration baseline development that we've done for our ICS and how we're now using those baselines as part of our continuous monitoring program to supplement and enhance our vulnerability scanning program. Um in addition to um identifying issues and misconfigurations for remediation, we'll talk a little bit about some success stories and program recognition that we've had. Um we'll go

into just a little bit of some future goals and plans that we have for continuous improvement. And I'll give you guys a chance for questions at the end and hopefully we'll my team members will come up with some good trivia questions that we can do for the giveaways. So that's exciting prizes. Okay. SRR is the right here. SRR is the liquid waste contractor for the Department of Energy Environmental Management um directorate at the Savannah River site. We are located in um Aken, South Carolina, not Savannah, Georgia. It's just right across the river. Um and we um our our corporate president or corporate um head company is AECOM. Our other partner partner companies are Bectal CH2M

Babcock and Wood Cox and Arriva. Um there are currently about 2,200 people who work for SRR at the Savannah Riverside right now. Um, our mission as a company is to dispose of liquid waste from 50 years of nuclear weapons program materials that's currently being held in very large underground storage tanks. Yes, it is radioactive. No, we don't glow in the dark. Our company performs removal and remediation of the waste from the storage chains, transmuting it into other environmentally safe storage forms such as borosilica glass and a type of cement called salt stone. Um there are three main processing areas each with its own collection of ICS components and support systems. Our cyber security program is

relatively young. We started or and were formed in 2011. Um, initially the program only focused on business and financial systems for our company and the ICS systems themselves were um, managed under another entities program on a very limited basis and they were not incorporated into the SRR cyber security program until 2013. So just recently. So all of this um integration into our program was going on while I was hiring people, developing my team, building our program, coming up with ideas for how we wanted the program to look and what we wanted to to do. Um and identifying and obtaining the tool sets that we want to use for or had in mind to use for

helping to secure our systems. Um, so to provide the greatest positive impact in the shortest period of time, we chose to focus on elements of the top 20 critical security controls. The two that we're going to cover in this presentation are number three, secure configuration for hardware and software, and number four, continuous assessment and remediation. And we'll talk about that in the context of our general industrial control system framework which we work to develop and supply um security configuration baselines.

This is an example of one type of industrial control system called a distributed control system or DCS. The example on the slide was taken from NIS guide to industrial control system security. The bottom level is the field level. At this level, there are controllers which are used to control and monitor various field devices. Some examples of field devices are sensors and motors. Uh also at this level are human machine interfaces or HMIs which are used by operators to monitor and interact with the process if necessary. The next level up is the supervisory level. At this level, there's a control server which hosts the supervisory control software that's used to uh communicate with the various field level

controllers. Also, at this level, there may be a data historian which is a database used to store historical process information. The top level is the corporate network. Um at this this is your more traditional IT network. And at this level uh you have workstations, uh printers and application servers. These are some constraints that you might run into when uh trying to implement cyber security controls on an industrial control system. uh one constraint or challenge is that availability is a primary focus of industrial control systems. When you're implementing security controls, you have to make sure that you don't do anything that will cause the the industrial control software to fail, which could leave the process in an

unsafe state. Also, sometimes you'll be integrating cyber security into long established processes. and some of these processes when they were set up uh cyber security was not a main concern. Um also in some cases uh the vendor software used to run the IC is tightly coupled with the operating system. Um so the vendor may dictate what patches and security configurations can be applied without negatively impacting the software.

All right. Some additional challenges that we faced um at our site DOE um the Department of Energy requires the implementation of NIST security controls. We specifically work to NIST 853 and now the new NIST 882 for industrial security controls. Um these guidance documents change very rapidly and sometimes very drastically. In addition um for example NIST 882 was first published in 2011. Rev one was published in 2013 and Rev 2 came out in May of this year. From rev one to rev two of that document, there were 200 security control changes that affect just the industrial control system environment. NIST 853 changed Rev 3 to Rev 4 added 51 security controls equating to 295 changes. And there is a now a total of

861 separate security controls that we have to address for all of our systems in addition to the industrial control systems. So due to DOE contractual restrictions and related funding limitations as well as the need to test changes for potential process impacts and vendor constraints to incorporate the changes into their application software and systems. There are some really long lead times that we have to deal with for implementing any changes in the industrial control system environment. We also ran into some some communications barriers stemming from lexicon differences. Engineering language is very different from it language and terminology. Um there are a lot of acronyms that um mean one thing to one group of people and something

else to another group of people. So we had to understand some terminology differences. Also Ed touched on one of my favorites in his topic. um airgapped, but it's airgapped. Well, no, it really isn't. Um, segmented, isolated, and what that means to the engineers in the field versus what it truly means an IT context is very different in a lot of cases. So, there were some communications barriers that we had to um work around and get over with dealing trying to implement cyber security in this new environment. Um so my team and I both had to learn a new environment and how we could securely implement um those changes within the industrial control system environment without negatively impacting

the processes. Remember Jeffrey said that availability is key that a gets turned upside down that triangle gets turned upside down in your process control environment. Um so we've had to deal with our own erroneous assumptions also um with um for example we have some very very sharp people who um are subject matter experts for the distributed control system application and its workings. Also, we assumed erroneously that those individuals also held the same level of expertise when it came to dealing with the operating system and that was not always the case. So, we had to do some talk around that as well. um we had to overcome or work to overcome and we're still working on this

um probably always will be the perception of an of cyber security being an impediment or a barrier. We've often been called the necessary evil. Um, we've had some success by comparing cyber security programs and the implementation of that those things to known established processes and showing the positive influence that secure systems can have on safety and quality assurance, quality control for other sections of the process. It's been very difficult getting the system owners and managers to understand that need and then also to embrace the need. So Jeffrey is now going to share with you some methods that we utilize to focus on integrating those two critical security controls in order to meet our cyber security requirements and also

improve the systems overall health without negatively impacting the processing that they were required to do. This is a general process uh for one way you can go about creating a baseline for a security configuration baseline for an industrial control system. Uh we're going to be focusing on creating an operating system baseline but the al the same process could be used for creating other baselines like application baselines. Uh the first two steps are more prerequisites to the process. In the first step, uh the vendor develops the software that runs on the OS that you're going to be creating the operating system security configuration baseline for. Uh when the vendor does their testing, we'll do it with the

operating system in a certain configuration and we'll support that the software works in this operating system configuration. But if you deviate from this the configuration that they use then the software could potentially fail and negatively impact processing. So you have to be careful uh before changing settings and to consult with the vendor beforehand. Uh in the next step you'll receive the the product from the vendor and you'll install and configure it to their recommended configuration. This configuration is probably not as secure as it could be. So to increase the security one thing you can do is select an IND industry standard security configuration benchmark. Some popular benchmarks are created by the defense information systems agency

and also the center for internet security. These settings in the benchmark will basically be your goal settings. The benchmarks will contain a list of settings where they're set in the operating system and the values that they recommended being set to. So then you'll determine the pre-benchmark settings. This is where you determine how the settings are actually set right now in the default configuration. Uh so this is where it can get a little bit tricky. Sometimes in these benchmarks there are over 200 settings and let's say you have 15 different computers to analyze. So this can be a lot of settings to look up manually. Uh so you'll likely use some type of automated method like uh getting a

default template or getting a a template built to the industry benchmark and uh put it into your scanner. You put it in your scanner and uh scan the systems to see how they match up against the industry benchmark. The problem is that the scanner usually returns low-level values like registry values on a Windows system, but it won't give you the translation of the low values into higher level values like group policy values. For example, the benchmark may say to set a particular setting to enable and they'll say that the registry item associated with that setting may be set to one. Uh but then if your scanner returns back that the registry on your system it shows a zero you'll have to

figure out how that lowle setting maps to the higher level setting to see what it really means. Um and um then there are some patterns that we've noticed when mapping low-level um values to higher level values like registry settings to group policy settings. I can't say that these patterns will always hold true. Uh but these are some of the patterns that we've seen when the options and group policy have for a setting only have disabled and enabled as options with no other options. Um we found that disabled is stored in the registry as a zero and enabled is stored as a one. In GPOS when you see an option of not defined uh we found this to mean that the GPO

will not affect the registry. In GPOS's when you see an option of not configured this means we found this to mean that the associated registry item will not exist in registry assuming that no other group policies are are being applied and are configuring that setting. A good tool to use on Windows systems um to see how the settings are actually set on the local computer is the local security policy editor sec. It will show you how the values are set in the highle values like the GPO level options instead of the lowle registry values. The problem with secpool.msc is it'll often only it will only show you a a subset of the settings that are usually

in the benchmark. Uh, one way to get those other values is to set up a test environment and try setting different options in the GPO, pushing the GPO to the test computer and seeing how each GPO option affects the registry. This way you can kind of reverse engineer how the registry options affect registry.

The next step is to compare the benchmark to the system settings. This is where you identify which settings are different on your system than what the industry standard recommends. Then you have to determine which of these settings are acceptable to change from settings in the default vendor configuration to what the industry standard recommends. One example of what might be an acceptable change is increasing the level of auditing in your security logging. Let's say that the vendor uh left a default of no auditing for min settings, but the benchmark recommends putting a higher level of auditing for your security logging. Uh this might cause the security logs to fill up faster and it uh may add a little bit of

extra overhead, but overall it probably will not affect the process. But having the extra logging will greatly help when you're trying to identify and investigate uh potential intrusions. One example of what might be an unacceptable change is changing the session lock from disabled to enabled on an operator workstation. The vendor may have set this session lock setting to disabled because the operator needs to be able to interact with the computer at all times. For example, you could imagine um if the computer's locked, the operator's there and alarms start going off in the plant and then the operator gets flustered. They try to type in the password to unlock lock it. They end up typing it in

a few times wrong and they lock themselves out. And meanwhile, this critical event is going on in the plant.

The next step is to analyze the risks of the deviations from the benchmark. Uh in this you'll be seeing how the you'll be analyzing the security risks for the settings that you can't change to meet the industry benchmark and you'll be developing mitigations to help lower that risk. Then you'll implement and enforce the settings using in in some type of consistent process. um in a repeatable fashion. Often this will be automated for example um using scripts or group policies depending on your operating system. So now that you have a a baseline created and also a scanning template created, um you're now able to incorporate baseline compliance scanning into your continuous scan assessments. It's a good

practice to do both vulner both vulnerability and compliance scanning and it would be a good separation of duties practice to have one group like the system administrators applying the patches and setting the settings to the baseline and another group like cyber security validating that the patches and security settings have been set to the baseline. One approach for determining how frequently to do your scanning is based on the level of risk of the system. Uh a this would be like a risk based approach. A level with a high a system with a higher level of risk would um be scanned more frequently and a system with a lower level of risk would be scanned less frequently.

Okay. Our um cyber security program has twice um received some recognition for some positive program elements that we have. Um first being um our web application vulnerability scanning and penetration testing program and our enduser awareness and training program and more recently we were identified for having an exemplary ICS cyber security program. Um we were invited to um the Department of Energy headquarters in Washington DC to um share our successes and how we got there um not only with our Department of Energy headquarters folks but also with um some cyber security program managers from other DOE facilities. So it using that collaboration effort to um help them get some quick gains on implementing security programs for their

industrial control systems environment. Um we utilized shameless self-promotion to get the word out. Um, anyone who's been in cyber security for any length of time understands the importance of of publicizing your successes and no one can do that for you any better than you can. Um, we wrote our own articles and then partnered with our local public affairs group to um have them published in online newsletters on site. We sent targeted braggad emails to our management and then they in turn sent them to corporate management and to our local DOE folks and then head DOE headquarters. Um we're lucky enough that we have one of our local DOE folks here um our cyber security oversight Robert

Ganaway. Um can you would you like to expand on any of of this Robert? Would you mind kind of put him on the spot? Sure. So, I'm Robert G. I'm the chief information security officer at Savannah River. We challenge SR Juls [Music]

because we knew that it could be done to work with the vendors uh to to put baselines in place to to gradually protect the systems better. than than it was already and and they've stepped up to the challenge and as Julie noted that we're actually working with NAP National Institute standards because those of you who have worked with this uh sometimes MI can they can sit in the lab and tell you what's what's best for you but they don't really know the ground truth. Um so we're going to induce uh a lot of the industrial control systems at the Savannah River site and the things that we've done as a benchmark for the national institute

standards working with BOE headquarters because DOE uh from what I understand has the majority of industrial control systems uh throughout the country. So obviously uh protecting those type systems in the process of that are very important. So uh listen what with the with what Julie and her team is going to tell you today uh because again it does work uh and you just have to bridge the gap between uh cyber and the engineers right it's a gap that has to be bridge uh before it works. Thank you Robert. Appreciate that. So why is it important to do this shameless self-promotion? Where does your funding come from? Who signs that funding agreement? Who signs your procurement, your purchase

requisitions for your new equipment? That's your management. So if they don't know what you're doing and how great a job you're doing at it, then what's your likelihood of getting additional funding for additional things that you want to do in the future? So that's something else to think about. Um the other thing that's important for us to think about as I mentioned earlier one of the biggest challenges that we faced the biggest and we still face is incorporating a cyber security program into longexisting processes that have been there for a long long time and the culture that goes along with those existing processes and being able to show those the managers of the engineering organ

organizations and the field organizations the benefit the value that cyber security is providing to them in securing their systems and um the what's in it for me they it's really hard to to get that through to them but if you can show them your successes a lot of times that helps um and helps pave the way and helps them understand the importance of implementing the cyber security program to improve their understanding. So, but we're not resting on our laurels. All right? We um we're you always have to strive for continuous improvement. So, here are some of the ideas that that we're working on. Um we want to improve our configuration management processes. We want to make sure that cyber security

is incorporated earlier in those configuration management processes. When they get the idea to purchase something or go out and make something or install something, we want them to say, "Oh, wait. I've got to get cyber security involved to make sure that I get it in there as secure as I possibly can and not leave us until the end of the process." We want to be earlier in the process, but we also want to be incorporated throughout the entire configuration management process, all phases, all steps. We need to improve the accuracy and quality of our configuration management data as well as the ease and the efficiency for collecting that data. If you don't have it, you don't know what you've got, you

don't know what you've got to manage, you don't know how to secure it. Um we've imple implemented a centralized software library for approved software versions and also for our secure configurations themselves known up good updates for products developing and maintaining and supporting procedures and other documentations. Engineers are always going to want to know well where's the procedure that tells me I have to do that. So if you don't have the supporting documentation, the supporting procedures that tell them how and what they're supposed to do to implement cyber security, then it's difficult for them to do that. You also need to make sure that you keep those documents updated and accurately reflect current state of your systems and

processes. We're improving standardization. um trying to across the board. Um one way we're in naming conventions, systems, accounts, scan results, even our filing systems are we're trying to be standardized across the board. This improves your network and system transparency, especially when it comes to performing scans. So you know what system you scan, when you scanned it, who did the scan, what the results were from that particular scan on that particular date. Okay. When and we were improving our process for implementing the secure configuration itself. So when this is done in tandem with having that configuration stored in the software library, it ensures the same configuration is being used and applied the same way across all of the systems

in your environment. That gives you a verifiable and repeatable process. And that's really important to the security of your systems, to people who change in and out, and um to auditors who want to come look at your system. So, some future goals um that we're setting for our program. We want to improve our processes for scanning. We want to make it easier for our folks. Um you saw my team. This is my team, five of us, six of us counting Harry. and he's actually being pressed to do a lot of a lot of work in the field. Um so we have a lot of systems that we have to go out and and scan on a regular basis.

Um we need to make it easier, faster for us to do that. Um we're conducting some research and some testing. Um we were um fortunate enough to get funding from our management from for a separate standalone cyber security lab that totally mimics from top to bottom uh an industrial our industrial control system environment and we do a lot of testing a lot of research in that environment. Um we're looking at some new tools like security onion. Um some you know other things that we're looking at what can we implement what can we do better um not only for ourselves to make our job easier but to make it better for the for our users on the other end. What kind of

metrics could we provide to them that would be beneficial and useful to them to their management and you know to get that funding and that approval. Um, so we're doing this all by keeping a breast of new vulnerabilities and potential issues that are specific to your industrial control system environment. As Ed mentioned in his talk, the ICS environment is your new frontier. That's your new attack frontier. That's where all the hackers, that's where we're getting the sexy vote now. You know, we get to be the target. Um, and that's that's really coming true. So, we have to make sure that we get there before they do and secure our systems. So, just real quickly to wrap up and

then we'll go to our our questions. Um, these are the main points that Jeffrey and I covered today. Um, we talked about Savannah River remediation and who we are and what we do in in um managing and remediating the 50 years plus of nuclear weapons waste, liquid waste and the industrial control system environment that we have that supports our mission at the site. Um, the constraints and the challenges that we talked about um the CIA triangle being the AIC triangle. Um, it's availability processes are king. you know, you got to make it work before we elect to secure it. Um, so we um also some of the other challenges that we um we talked about

was um implementing cyber security in long-term existing processes and culture, the tightly vendor controlled implementation challenges trying to address rapidly changing requirements, differences in lexicon and other communications barriers that we had to deal with. and the steep learning curve from for a very small cyber security team. We achieved some success by focusing on critical security controls and the two that we talked about here were the secure configuration for hardware and software and continuous assessment and remediation and the use of shameless self-promotion to influence our user community and our management for future endeavors. So that was all we had. Do we have any questions before we go through the giveaway or you guys just want to do the

the giveaway? Yes, I have a question back. See if this works. You mentioned one of the challenges with integration and getting them to cooperate and support modern security configurations. How responsive have they been to that? Are they quick? Does it take years to get anything in place? You know, what has that been like for you? Um, it has been very slow. Um, for one thing, um, I think one of the biggest challenges that we faced early on was even getting them to admit that they might have a problem. Um, and then they're always because they have to do so much regression testing because of all the different versions of their software they're supporting um, throughout multiple companies and

entities where they have it done. They're very slow to uh provide updates and patches and our folks will not do make any changes to the systems unless it comes directly from the vendor. So, it's been it's been a challenge. They're they're usually month and a half at least behind patch Tuesday. Um, but it's um it's a little it's it's a challenge, but we we finally got um we have a very good local vendor rep. Um, and we've been able to bring him in our um local area and show him our our lab system. And I mean, he's got, "Oh, my cyber security guy would love to see this. Can he come look?" And it's absolutely fun

to bring whatever he wants to bring. So we pound on it and have a good time. So the month and a half is better than I expected to hear. Yeah. In the back. Yes, ma'am. Uh could you please uh tell me your thoughts on SI version five? What kind of impact that's going to have on you in the industry in general? Positive, negative, that kind of thing. Okay, there's your first um trivia question. I don't have a clue. I'll Can you help that, Robert? Thank you. So, so SIP SIP version 5 is usually a utility power industry standard, right? So, in the BOE realm, we we don't use SIP in DOE and environmental manage management realm, right? That's going to

be a power uh type standard there. So that it will have no effect on us. Currently we're actually already on using this and and this framework for for selecting our controls. Thank you. Anybody? Okay. We are running short on time so I I hate to interrupt the questions but we do need to do these giveaways and then we'll take a quick break come back for our next talk.