By Patrick Miller

Two industry veterans who cultivated NERC CIP over the past 20 years discuss how it all started, and what’s next for electric power industry security regulations. Patrick C. Miller, one of the first NERC CIP auditors in the country, and Carter Manucy, a utility IT/OT Security Director, talk about the regulation that changed the electric sector cybersecurity landscape forever.

PATRICK  
I'm Patrick Miller with Ampere Industrial Security. I'm the CEO. I'm here in beautiful Orlando, Florida with my good friend Carter Manucy. Tell us about yourself, Carter.

CARTER
Hi, Patrick. I’m Carter Manucy, Florida Municipal Power Agency. I’m the IT/OT Cybersecurity Director here at Florida Municipal Power Agency (FMPA) in Orlando. Great to see you here in person in my own backyard.

PATRICK 
I know, nice! I was in town for an event. So, we're here to talk about NERC CIP. And of course, this is the transformational regulation that really kind of set the bar for cybersecurity and physical security in the electric sector. That standard, it’s monumental. It’s actually been around for 20 years. So we're going to celebrate —“celebrate” — 20 years, right? With this 20 years of NERC CIP, I think back to where I was 20 years ago. I was at a Pacific Northwest utility, and they were like, “Hey, we think there's some regulation coming down the pike, kind of go take a look at this stuff.” And this was the FERC SMD which started all this. It came out of an old Clinton-era decision directive from the president. That was where we kind of started to understand that our world was really about to change. So where were you 20 years ago when all this started?

CARTER
Twenty years ago, I was still FMPA. Twenty-six some-odd years now. So, it's a bit of an adventure, but I was actually manager at the time. And so I was gleefully unaware that this thing was coming down the pipeline. A lot of my focus was on corporate security and networks and all these other things, the firewalls and virtualization at the time, even so, that's been around for quite a bit now.

PATRICK

Still around.

CARTER

It is still around. I didn't see a lot of the news hit my desk until we started talking about fines. And what was going to happen to the regulations if we didn't comply. And not for the CIP space, but even the O&P (Operations & Planning) space, they didn’t have the concept of that fining component to a lot of the regulation. And the knock on my door was when folks were really concerned about that, and also needing folks to get involved with what's going to happen and understanding. Because a lot of the people that were initially let on to this stuff were not the ones that would have had to implement it.

PATRICK  
Right. And before NERC CIP, everything was voluntary, and basically collaborative, there wasn't any real teeth or force to do it. And I remember when that shifted, it was really difficult for me to get my utility to embrace the fact that this was now federal law. It wasn't just this kind of international standards body that said, you should do these things. It was you must, you're going to get penalized.

CARTER 
Yes, and declaring your own stuff as critical as part of that.

PATRICK 
Yeah. Choose your own adventure!

CARTER
That was that was certainly a problem.

PATRICK
Yeah, definitely. So my CIP life started at basically just trying to help the company figure out what this new FERC motion was going to look like. And then I got attached to the Critical Infrastructure Protection Advisory Group (CIPAG) and NIPC, the National Infrastructure Protection Center with FBI. And then we got into the early drafting stages, and did the implementation with the utility for a while, which was really interesting. Taking a company that's 100 years old, with different systems, trying to figure out how we're going to get this thing through the new - what would be the CIP mesh sieve - to actually becoming the first CIP auditor in the country, and then trying to figure out how we're going to audit this new thing that no one has ever done before. And then now, of course, well, we're on a lot of the same working groups and committees together, which is why I wanted you to have this conversation. Now I see almost every NERC CIP thing that's out there, you are somehow attached to this. I know you've got a day job. But all of these NERC CIP things, tell me how you got into this and how you are now woven into the fabric of how this is unfolding.

CARTER 
That's a great question, because sometimes I ask myself that same thing. But it all started with the transition into NERC CIP, and getting involved with FRCC and some of the committees there, and then listening to the conversations that were going around. And I'm like, “Why? Why are you guys trying to do things like this?” It just didn't make sense to me coming from a networking and security background and everything else. It'd be an awful lot easier to secure things in this way. And then the conversation we got to the room would go quiet, and people would think, and we continue on. “We need you to get more involved in these other efforts where we're having a conversation. Could you could you do drafting teams?” And yes, drafting teams are a whole different commitment. And actually, I considered it for version 4, but it was such a such a time commitment above and beyond everything else. But the other things that happened was, “Now we need you to get involved with CIPC.” So at NERC CIPC, we had voting committees and we had different groups in there, and, “We want you to do the cybersecurity side for NERC CIPC and getting involved in drafting the CSSWG the company control centers guidance.”

PATRICK
Control Systems Security Working Group (CSSWG)?

CARTER  
Yeah, Control Systems Security Working Group efforts on writing white papers on how to do firewalls. Some basic things sometimes. But how do you do that in a compliance space? And then, as CIP evolved even more, and those conversations happened more, we had trade organizations looking and asking for help, and asking “Carter come on in. Can you help educate?” And I was always looking at this, like, “Why, why are you asking me? You know? You guys are reading the same paper, right?” And it seemed like we all read the same paper, but somehow I came to a different conclusion as to what the end result would be. I don't know if it was a crystal ball, but everything ended up gravitating that direction. So that started a snowball effect, and more and more, and then we end up with other shared facilities, they seem to be a huge issue down in Florida, for whatever reason, a lot of our members have that same problem. We noodled on that concept within the Florida region for a while, so I brought that forward and said sort of pounding the table like, “Guys, this is a national issue, right? We need to figure this out! Write a white paper, why don't you?” So I almost voluntold myself a lot of this stuff.

PATRICK
So it's like no good deed shall go unpunished?

CARTER

That's right. That’s right. The next thing you know, it's 20 years later, and here we are talking about it.

PATRICK
The industry owes you a huge debt, because you've done a lot of really good, sane, balanced, level-headed approaches to help steer the standard in the direction it's gone for the better. So, thank you. When you started to implement this, did you think this was going to succeed? What did you think this was? How did you think this was going to go?

CARTER
That was really interesting. Because, you know, the first involvement that I had was folks saying, “Hey, we need more people that know IT involved in a lot of these standards and compliance.” And I had no idea what I was getting myself into at the time, but ended up with a lot of things within FRCC, our region down here in Florida at the time. “Hey, can you get more involved in these meetings? Can you find out what this NERC CIP thing is and help out our other member utilities that we have that do have compliance obligations? We don't think we're going to have anything yet. But that might change.” That did eventually happen further on down the line with NERC CIP for us, but the direct impact was to a lot of our members and the larger utilities. Those were the first cut for a lot of this stuff. That was my involvement, was just wondering what that world was going to look like. Did I know what this is really going to turn into? Oh, heck no. But I do distinctly recall having a lot of conversations about cost and implementations for those utilities that were deeply in it. And the unknowns. This is coming, or it's not. I’ve got to go to my board for more money, or I don't. And that back and forth was, I know, put in a lot of gray hairs for a lot of folks.

PATRICK
Yeah, I saw a lot of the same. I knew that we were going to be regulated. And I was just curious to see if it was going to stay the CIP standards, or if this was, was this really the thing that we were going to end up living with for the rest of our utility lives? If you remember the early days, we were talking about “Do we need a generation standard, and a transmission standard, and a control center standard, because the differences in those types of systems and assets?” We ended up with a kind of a one-size-fits-some and we've shaped it and molded it over the years, I guess, to kind of embrace all of those things. But I remember when this first started, I was really wondering, “Okay, did we set off on the right path? Did we do we actually get this going in the right way?”

CARTER
I think we've changed a lot in how we do things. Did we get it right? Probably not. There's a lot of things we could do a lot differently. Actually focus on security versus compliance. But in hindsight, this has made improvements in the industry. So, did we make the right decisions? Perhaps.

PATRICK
Right. It gets a lot of, I will say, mixed press, I guess. The CIP standards do. I am one of the biggest - I mean - I helped write the darn thing. I was the first CIP auditor. I give it a lot of noise, but it's also worth celebrating the successes that it had. It actually did move the needle, literally. What are some of the things you think are good successes the standards actually gave us?

CARTER
I think for sure it raised awareness. Yeah, hands down. We've got people actually implementing security - and not so security - things within our industry that never would have moved otherwise. So that is definitely one of the better things. We probably got to use a different framework to do a better job of doing things a little differently. But, in the end, that's what moves the needle: everybody has to move forward. We're seeing that time and time and time again. As time evolves, and things change, that's proven itself out, I think. We're starting to see that even now with the low impact stuff, the needle is continuing to shift and moving folks along.

PATRICK
Yeah, was talking with Earl about this, Earl Shockley. He was mentioning that his math shows that at least 50% of the standards are constantly in draft and being changed and modified to adapt. Which is interesting for a utility that's used to things not changing very frequently. So it puts a lot of, I guess, strain on keeping up with the target.

CARTER
Yeah, it's a strain of that, and a strain of not having anything definitive that you can look to. You would think that after 20 years, we would have had some kind of a steady state here going on.

PATRICK

Yeah.

CARTER

Which we don't. We're still struggling with subjects like virtualization and remote access, and some of these very fundamental things we still haven't figured out. We're still fighting, quite honestly, with a lot of the ‘how to do patching properly.’ Is this the right thing or not? Those are, unfortunately, the battles that we're having and the energy that we're spending on compliance is on things that may or may not be the best way of securing the facility,

PATRICK
Right, or really maintaining reliability. Which is what the goal is at the end of the day. Yes, security is there, but it's really there to support the reliability mission. Okay, so where do you think this is all going in the future? So, maybe five years, 10 years out - is it still going to be the CIP standards? Is it going to be something else? Will the standards change so much that they look different than they are now? What's your crystal ball, pure guess for where do you think this is going to go? Or, you've been in this long enough. What's your educated opinion on this might go?

CARTER
I think we're going to shift a bit to more of the NIST standards.

PATRICK

Yeah, I agree.

CARTER

For the whole industry to do the same thing. Power shouldn't be any different than chemical, oil, gas, water, pipelines. Why? Why do we think we're so different? We use different electrons to run Windows in a power plant than we do in the water facility? I don't know. But that's where it's going to go. Now, I think technology-wise, that's helping us get there. We've got better tools at our disposal. We've got better vendors able to help us meet a lot of these unknowns. Visibility becoming key. I think that's where we're going to end up, is that hopefully, we have ability to show compliance through monitoring so we don't have to do as much on a day-to-day to prove that some state was true 10 years ago, five years ago, last month, whatever. I think that's what we need to get away from to actually make it so we can focus on the reliability component.

PATRICK
Being more of a control objectives, controls, control tests, and then continuous monitoring approach?

CARTER

Yes.

PATRICK

Yeah, I agree. I think it's going to go that way. And we've even seen some messaging within the National Security Memorandum where they talked about a common set of metrics for those four or five industries: electric, gas, chemical, water and wastewater. They were leaning toward NIST 800-82, which, of course, is the overlay on to 800-53. I can't argue with that approach. I can't say that's a bad idea.

CARTER
I don't think it's a bad idea to have that standard of standards. If we can all focus and on all march along the same lines, and we don't have to remember 15 different revisions of five different things. Which version are we on? Which ones actually - oh, wait, no - that one was voted down. Then we have this one to replace that one. All the all the issues that were going on with that. Because, in reality, this is a bigger problem than just one industry.

PATRICK
It is. And I think it would help normalize even the discussion, because we're now using the same words and the same frame of reference. It would help the vendors, it would help the assessors. I just think overall it might make life a little easier. It's going to be difficult for our industry to think past the CIP standards to something else. I mean, are we going to have a window - your guess of course, your opinion - are we going to have a window where we have to do both the CIP standards and something like NIST, or?

CARTER
Well, I think that if we're smart about it, what would end up happening is we would have one transcend the other. But I've lived through the Version 4 off-and-on, and all the other combinations of these things happening and it never is clean. It's never pretty. So I'm going to go on the record by saying it will be a train wreck, but we will get there.

PATRICK
Absolutely agree. Yeah, I think it's going to be an interesting train wreck of a transition to something else if and when we actually do. Awesome. Thanks so much for your time. Really appreciate the insight. And hey, let's do 20 more years of whatever this new thing looks like.

CARTER

I'll be here.

PATRICK

Thanks.

Featured Posts