What to do about FERC's new INSM Order 887
By Patrick Miller
FERC has issued Order 887, directing NERC to create new Critical Infrastructure Protection (CIP) cybersecurity standards for Internal Network Monitoring Systems (INSM). Hear from a real electric utility asset owner, Carter Manucy of FMPA, on what this means for the industry and what you should do next.
Also see our blog on the INSM order…
The following text is a transcript from the Ampere Industrial Security Critical Assets Podcast, Episode 1 - What to do about FERC’s new INSM Order 887.
PATRICK MILLER:
Hi, everyone. Today with us I've got Carter Manucy. He's with Florida Municipal Power Agency (FMPA). I'll let him introduce himself in just a minute, but today we're going to talk about the INSM Order from FERC that recently came out. Carter, tell us a little about yourself and then I'm just going to kind of pepper you with questions about this Order.
CARTER MANUCY:
Wonderful. Carter Manucy, Florida Municipal Power Agency IT/OT Cybersecurity Director there, oh it’s been a little while, I've been with FMPA for almost 28 years. So I've been doing this NERC CIP thing since pretty much right around version 3. Started my introduction to this wonderful world, and ever since then, I’ve been involved from many different gamuts, Critical Infrastructure Protection Committee (CIPC) involvement, regional development/involvement, development of standards, the whole nine. Of course, everybody loves to comment, so I've been part of those too, but also corralling folks, and trying to make sure that we’re all rowing in the same direction when it comes to a public power perspective on the standards.
PATRICK:
Yes, I've seen your work and I want to thank you for all the stuff you've done both for the CIP standards in general and of course, for the municipal power authorities as well.
CARTER:
Thanks, sir.
PATRICK:
Well, many thanks. Because it's been a long haul. We've been doing this NERC CIP thing for a while and I've seen your name in so many awesome places that have been beneficial to the standards, making them better with security in mind and with compliance in mind. So, I just wanted to give you a shout out for all the awesome stuff that you've done for the industry.
CARTER:
I appreciate that. It's always better to do something that's practical instead of checking boxes.
PATRICK:
I agree. Speaking of checking boxes, we have this INSM order from FERC that just came out. We've both spent a lot of time you know, looking at FERC Orders, responding to FERC Orders, seeing how FERC Orders turn out… So, first, let's just start with what is INSM? What does this mean to your average utility out there trying to figure out what did FERC just tell NERC to do? So INSM in a nutshell.
CARTER:
Thanks. So, the INSM stands for - we love our acronyms, right? This is yet another one, Internal Network Security Monitoring. So, if you think about network traffic in general, stuff that's happening on your systems, this is what we consider “East-West” traffic. “North-South” traffic is typically what you'd have with a firewall, or an ESP, the Electronic Security Perimeters that you might be accustomed to. But these devices are not firewalls. They actually take traffic from network switches or other ways to collect it on your network and watch traffic that’s happening between devices perhaps behind the firewall, maybe even not. I don't want to get too far into the weeds with a lot of these things because there's taps and spans and different ways of collecting the data. Just think of it as a read-only way of viewing traffic on your system that they can go and analyze. Now, of course, everything I say it's a little caveat, because everybody does things a little differently and I'm trying to be very agnostic to any particular vendor technology. Some may do it different ways. They can also be used for non-security data analytics, such as sending device configuration changes, or seeing those, look at firmware updates that are going across the network, determine what devices are communicating with other ones, maybe give you asset discovery, network visualization. But all these things understand industrial protocols when in an OT network. That's what the focus of this INSM is for, from FERC. Because obviously, we're looking at the CIP standards, we're looking at the electric industry. So, usually the tools that are beneficial to us understand those things.
PATRICK:
So, it's looking at East-West traffic. What does it cover that's not already covered by the CIP standards? They've got controls at the perimeter already. They've got controls on the individual systems inside the network. And, arguably, they've even got some East-West controls with the antivirus components. Especially if you trunk those paths through a firewall – and many do because you can't get any virus agents on everything in the network. So, what does this give us that we don't already have in the standards?
CARTER:
Right, these things are not firewalls. They're designed to alert and not necessarily block traffic. So they give you some visibility when you may have a compromised device. There's concern around a compromised ESP. These would give you that ability to see something behind your standard firewalls. If you have it designed right, it looks at your entire network traffic – everything end to end. So, it covers things that can't have antivirus on them, things that can't have agents on them such as PLCs, controllers, SCADA controllers, all these simple devices. And they may be communicating in a way that they think is correct, because they've been told to do something. So that's the big difference you're looking for.
PATRICK:
Okay, from an anomaly detection perspective, sending a command that would otherwise look legitimate, but for some other reason based on conditions would not be a good thing, like a stop command to a PLC during the wrong situation.
CARTER:
Right, exactly. So those types of anomalous behaviors are what these things are looking for.
PATRICK:
Okay, so things that are kind of outside the normal realm of detection. Because we got a lot of other malware detection at the firewall, we've got it at the devices themselves. So, this is looking for things that are beyond that. Like maybe a configuration change, or a use of accounts that shouldn't be used.
CARTER:
You certainly have those things in there. But also, you have to keep in mind that sometimes, your adversary servers, if they are in your network, may be just very slowly gathering data. So, they may be probing things without you even realizing it or knowing it. Or they may be using an encrypted channel to come back to their mothership, which you may be allowing, because it is something that you believe is legitimate. Or maybe hiding with a vendor or something like that.
PATRICK:
So, okay, that makes sense. All right. I think we got a good picture for what INSM is, what it does, and what benefit we get above the CIP standards. Why do you think FERC thinks we need this right now? I mean, what's the catalyst for pushing for this right now?
CARTER:
Well, there's the old adage (that's old now to us in security industry), you know… SolarWinds, SolarWinds, SolarWinds. I think that's the overarching thing. And you know, if you think about it from a congressional push, “hey, are we doing things?” The NOPR in general says, “Hey, we've got gaps in our CIP requirements, we have the ESPs for high and medium impacts, but we don't have any monitoring inside the actual network.” Like you said, we do have it on specific devices, antivirus, those types of things. But those are generally in your high impact control centers, where there are more PC-like-things. We don't necessarily have that out in the field for a lot of our OT assets, per se. They may not exist in a control systems environment, because those are much more controlled Windows PCs talking to PLCs or similar. We also have the NOPR pointing to the Standard Practice Guide that was issued in 2021. That practice guide was in response to the 100-day Sprint that we had. There was a real push for more network monitoring to happen at that point in time. So, entities were real concerned. How was an auditor going to handle what this thing is? So that practice guide helped explain some of the expectations and guardrails. That was mentioned as one of the benefits. In that Practice Guide, was the use of these Internal Network Security Monitors and I think FERC was listening to that as well.
PATRICK:
Okay, I almost forgot about that Practice Guide. Thank you for reminding me. I've also seen the same language in the NOPR that came out about a year ago – almost to the day – from FERC, kind of giving us a heads up this was coming. But also, about a year before that, it was in the National Security Memorandum from Biden.
CARTER:
That's right. We've seen this historically through a number of different avenues. We've seen it with the second 1600 Data Requests that we did looking for more information on supply chain risks and everything else. So, a lot of this all originates from risk to the supply chain and an ever-increasing threat landscape. And are we doing enough? You know, I guess, it’s a step forward a little bit. But that's really where I think a lot of this is coming from.
PATRICK:
Okay. I know FERC thinks we need it. And it sounds like, that the Biden administration thinks we need it. I expect something like this to also be in the upcoming National Security Strategy that's coming out, as well. Given that it's kind of been everywhere, I'm not surprised that we're seeing it from FERC. A lot happens at that layer based on knee-jerk reactions to things like SolarWinds. We got CIP-013 from a knee-jerk reaction to Metcalf. We're probably going to get more physical security as a result of the distribution physical attacks that have been happening. We're already seeing those things. The machinery is moving in that direction already. Is this just another knee-jerk reaction? Or is this something we really need? Is this truly a gap in our visibility or in our in our coverage? Let me ask a different way… Is the security juice worth the squeeze here?
CARTER:
I love that. So, I'm going to give you the response I hear from a lot of my legal colleagues, “it depends.” What does it depend on? How good is your security culture? What is your track record? Do you have any way of monitoring traffic now that doesn't cross the security boundary? How tight are your security controls? How much visibility do you have on your networks? How can you help secure those communications links that you have inside your own ESP? I'd really say there's no one size fits all answer here. But I think it's the intent to set a new bar. Because of the CIP standards, the electricity sector hasn't seen the number of attacks that other industries have been facing. And I think a lot of us can probably agree that the number and quality of attacks on the internet networks and in our digital lives is on an upward trajectory. So, if we're not improving, that means we're falling behind. Right?
PATRICK:
Yeah. I've had some conversations, and another way that they described thinking about this was, we've done a pretty good job of securing the electric sector. Such a good job that we're now having to go this far to get additional visibility. So, I guess it's a good security problem to have, sort of? It does mean a lot of work.
CARTER:
It does mean a lot of work. And like you said, SolarWinds, right? How would we have possibly detected that? And I would argue till I’m dead that I don't think anybody would have caught that with even the largest budgets out there. Probably couldn't have caught that without having some very special talent looking at that. We're not that type of an industry. We don't have the crème de la crème. We're not, you know, security agents at Amazon, trying to protect all of these wonderful things. We're utilities, and we do this as best we can with the resources we've got.
PATRICK:
Right. Like municipalities, your world in particular has a very difficult problem with they are there. To be honest, they're cash strapped in a lot of ways. And frankly, the electric utility, the water utility is often kind of a cash cow for the rest of the municipality. The investor-owned utilities, granted, they may have more money, but they're have a regulated rate of return. They can only make so much profit in the process. And a lot of where their budgets go, it's not as visible as yours with the Sunshine laws, but it's still pretty open and disclosed in terms of what they're spending to the state regulators and various other oversight components. Utilities just don't have an enormous amount of disposable income to spend on these things. So, I totally agree. We're not really funded to fight off nation state level adversaries.
CARTER:
That's right. Yeah, that's right.
PATRICK:
Okay. Given the situation that utilities are in, we just described what it looks like for them in terms of their current capabilities (at least generally), what do you think they should be doing next to get ready for this order? What's the immediate stuff they should be doing, if anything immediate? Then maybe six months to a year out, what should they be doing? And then of course, that two-to-three-year window when this is going to be effective and possibly auditable? What does auditability look like? Run through that spectrum of time with me. What if you were going to do this at your utility, and let's pretend you had the budget and the people, what would you be doing?
CARTER:
Ha ha, right. If you don't have something like one of these INSMs now, and you have a high or medium impact cyber systems, I'd start looking at your network. Both from a logical perspective, as well as a technology perspective. Meaning, figure out where you would conceivably collect traffic from, and what devices you have that would be capable of collecting. And then, what could be sending traffic into this device to monitor. By the way, don't forget that you must make sure the devices can handle it too. Sometimes, we have antiquated switching devices or other network things and they will just fall over if you put too much traffic on them. Keep that in mind. Talk to your OEMs and vendors to see what else they have that might work for you, your control system, your environment, for example. But there are some really long lead times right now on equipment. Sometimes up to a year, or maybe even more with some things. And if there's a sudden rush to market, I don’t think our supply chain is going to react to that very well. We have to put all these things in place. If we think about budget cycles and everything else (especially with municipals), we’re trying to figure out if we need to do an RFP and then I've got to buy the equipment and then procure it, and then I've got to do this and do that, you might be looking at two-to-three-year runway here. These are the types of things to look at up front. I'd also suggest talking to some of these vendors out there that currently have these types of platforms and get a feel for what they do, and how their technology might work or impact your requirements so that you can start supporting it and get a better idea of where you might need to be in the future with these things. Personally, I like network taps over other technologies.
PATRICK:
Yeah, I do too.
CARTER:
But you should really make sure they can support your network and you understand the impacts that they may or may not have. Fundamentally, how are you going to tackle this problem? Which way are you going to go? I'm a firm believer that if you put all this network infrastructure in place now, you can be agnostic to the device that is actually doing the analysis for you later. You could plug in vendor A today and if that doesn't work out so well for you, you can just go plug in vendor B. You don't have to change the infrastructure out. You've already got that ready to go. So these are the types of things you can start doing now that won't necessarily affect what true vendor analysis platform you're using in the end. But yeah, really looking at how you might be able to architect your network too, would be the next the next phase. Do you zone it? Do you segment it? Do you try to create more effective choke points? Of course, then we have to be careful with that as well, to make sure that we don't adversely affect our liability by throwing too much IT security on things. Everybody loves it when we when we go down those roads. But yes, start having those conversations now to better prepare for them. Start talking with your leadership at your utility, make sure that they're aware that these things can get expensive. And by the way, there are grants that can help with these too. So, talk with your trade organizations, find out what opportunities you have. Your state may have something that could help. Start having those conversations now because, again, these have long runways. It's not something you can just go rush out and check a box and open the package tomorrow and throw it in.
PATRICK:
Especially if you're going for grants. That’s a long lead time as well.
CARTER:
Yes, and if you're going to install something in, say, a generation or transmission facility? Those can take outages sometimes to get them to work. So get on those schedules. By the time you put all of these Gantt charts together, that's a long Gantt chart to stare down. But certainly talk to other utilities, find out what they're doing. And I throw in a plug for going to places like S4. There's a lot of really smart people down there you can discuss these topics with.
PATRICK:
And the SANS ICS Summit coming up as well. Both of those would be good places to have that conversation.
CARTER:
When you start asking about what compliance looks like, that's crystal ball time for me.
PATRICK:
What do you think? How will the practice guide influence or impact the way the auditors will approach this? Because in theory, that is endorsed guidance that should be followed. I think it's impossible that it would have zero influence. But what are your thoughts?
CARTER:
So, the practice guide talks through different technologies because there are different ways to enact these types of devices. Some may have clients that are actually on the devices, and they probe and then they send data back. Others are through network taps, so they're more passive. But you have to be careful with what that device is, and where it sits. They talk about that in the practice guide. You could easily put this device and quickly have a to classify it as an Electronic Access Control or Monitoring System, EACMS. And then you’ve got another problem on your hands. So be careful about that. Think about, when you implement these things, whether or not it becomes a BES Cyber Asset. If it has any BES, Cyber System Information on it that you need to be careful to protect. Where is that data going? Who's looking at it? Can it be anonymized? If you're sharing it? All those types of things come into play, when you talk about compliance. But I think we also have some hints with current CIP requirements, CIP-007, maybe check out R4.2, R4.3 R4.4. You may have some hints in there as to what you might be doing with the same type of device. Evidence collection wise, I could certainly see, having these things as part of that Incident Response Plan and putting that as part of it. So when you when you execute those, you keep this as evidence too. But, showing network configurations, maps, topologies, documents - where they are - that's certainly going to be part of a compliance requirement.
PATRICK:
Okay, and what do you think about something like a level two request on the ERT, looking for maybe a sample of some of this log data? Because in the order they state, you're basically supposed to be doing packet captures.
CARTER:
Yes. And some of these devices, they do real time network captures. Some of the ones out there, you can say, show me all the data on the network for the last two days or whatever. And they’ve got a finite amount of space, so we're not going to be capturing two years worth of data on those things. But you know, 30-60-90 days, maybe? But you should certainly be able to pull that data out. So, think about that when you install it. But again, it'll probably be to the capability of the device. So, if vendor B doesn't do network captures, you can't be required to produce them, although that doesn't stop them from asking for them.
PATRICK:
Even the requirements for collecting logs, if you have to have it for like 90 days and it doesn't do that, they've allowed that to be requested at audit. So you can get that window. Because there's this weird situation where the audit freeze date can be a little bit before your audit date. And you may run past your 90-day clock on the buffer. I'm hoping they factor that in when they actually do the drafting team. But kind of all bets are off until we get some drafts to look at. And they have a 15-month window to get this done. That's pretty fast.
CARTER:
A 15-month window, and then don't forget the 12-month lead time on whether or not it gets pushed to lows. Right?
PATRICK:
Yeah. Speaking of that, what are your thoughts for medium without ERC and for low? Tell me what you think and I'll tell you my thoughts (or my bet).
CARTER:
Ha ha! So, let's start with the basic question. And I'm going to say the answer is probably “some” when it gets pushed to low impact. If we follow the path of how things have gone for low impact over the past few years, you can see a constant push for doing more than we're currently doing. The section 1600 data requests that came out a few years ago, led the staff report on supply chain risks, that can't be ignored. Then we had the whole low impact criteria review paper that came out on supply chain risks for low impact as that follow up directed by the New York Board. I sense the connectivity here between that report and FERC’s NOPR. We recognize that not all lows are created equally. You can have a small substation with a couple of 100 KV lines wrapped up in the same categories of 1499 megawatt power plant, for example. I think now people are starting to become aware that taking a small substation off the grid might not have the same impact as a generator that is swinging a steam turbine around a little bit. They definitely have different impacts on the grid, and we need to make sure that we categorize them in different risk buckets because of that. I think you're going to start seeing some categories show up with ways to say these are in, these are out. Remote access, remote monitoring at a generation facility, may be one of those tick marks. Control Centers, low impact Control Centers, that may be one of those tick marks. I think you're going to start seeing some of the components that are already in the CIP standards being utilized in different ways we haven't before, so we don't have to redo all of CIP-002, but we can still move forward and categorize certain things at a higher risk than the stuff that really doesn't matter to the Bulk Electric System as much.
PATRICK:
Okay. And I see that. That's really good insight, because as you were talking about it, I was thinking about that study that was just done for the transmission Control Centers and their recommendation is basically they're looking for some additional inclusion and exclusion criteria to factor in these circumstances that were bundled in earlier on in the standards that probably need to be broken out.
CARTER:
Yeah, that's exactly.
PATRICK:
I agree. My bet is… every time FERC does something like this, it is almost one of those situations where FERC is saying, “thank you for your report, we're still going to do whatever we want to do anyway.” I suspect that there will be a definite push into the lows and the mediums without ERC in some way, shape or form. I don't know if it's going to be wholesale. I do believe there'll be some granularity, like you mentioned, but it's going to happen to some of it – in no uncertain terms.
CARTER:
Yeah. If I were a betting man, I'd certainly put my money on generation above a certain limit – I don't know what that is. Call it 500 megawatts. I don't know if that's right or not. But there will be some threshold that they will probably pick on. And then, if you have External Routable Connectivity, if you have remote monitoring or remote access, I think those are all checkboxes that are at that much closer to saying, “yep, you got to have it too.”
PATRICK:
Now, is this going to have another one of those unintended consequences, where if they say, if you have any sort of ERC then everything applies, and therefore people start ripping out ERC, and we go back to dial up modems and silliness like that?
CARTER:
Maybe a few years ago, but I think now, there's too much value to the business for a lot of that data. And it's really needed to do a better job of performing in the grid, and we see all the issues that we've had currently with generation not being available when we really need it. So, I think the pressures from the market will help offset that a bit. I know that those conversations will for sure happen – whether that's something the utility chooses to do or not. But I certainly would hope that they don't rip stuff out that is actually beneficial to the business or to the actual reliability of the grid in general, just to spite CIP.
PATRICK:
I've seen some that will. They'll spend more money to get their toes right up to the line of compliance than actually spending the money on sustainable compliance. I have seen some do that. And I've their argument in some cases has been because it staves off future problems. I think, maybe, yes, but not forever. Eventually, they're going to find a way to rope you in one way or another. So, I would say, go ahead and do the ERC. Get all the business benefits. Get those robust network connectivity components in there. Have the visibility into your environment, and use it. I think they're missing a really big operational benefit to getting this data in a lot of ways.
CARTER:
The other thing that may actually happen is, not only the grants, but perhaps the vendors actually start showing real value to operators, real value to the asset owners to say, “you know what, it's not just about the security, but it's also about… hey, let me show you exactly what assets you’ve got… hey, let me show you all of the firmware levels that you've got across all of your devices.” So the next time you see this alert come through, you can very quickly determine whether or not you've got an impact in your facility without having to send anybody anywhere to ask any questions. Having a lot of that data collected and available to you, showing communication paths between devices… I think all of that stuff would be of a huge benefit. And if they don't cost a ton of money to implement, that'll certainly help push people towards that that goal. But on the other hand, I’ve had the real experience. I've had the firsthand experience of installing this stuff in generation facilities – and it is not easy. You have to plan this stuff out. You have to work with the vendors. You have to plan it on outage cycles, or even upgrade cycles, if you know the switching infrastructure isn't right. And if you don't do that, then you spend even more money on infrastructure just to bring the data back to a single point, which doesn't make any sense either. So, there's a lot of problems here. I don't I don't see this as being extraordinarily helpful in small substations. It may entice smaller substations to isolate and have single networks at each sub instead of a large ring where everything's on a single subnet. So I think there could be some of those types of ancillary gains, just by people not wanting to perhaps put an internal network system or security monitor in and say, you know what? Maybe there's a fallback position. Maybe there's something I can do to increase my security without having to have that massive collection of data going through. And maybe that's something that comes out of these requirements too, to say if you don't do this, then here are some alternatives and this is what you must do. And that's a great thing.
PATRICK:
Yeah. I agree. I agree. And I'm also thinking that there is probably a good lever to go to your executives and even to your grumpy plant managers that never want an outage ever, and say, “look, the Feds are making us do this, and it's also got a really good operational benefit, so we should go ahead and do this” versus “we should fight this in every way possible, and actually spend more effort and time and blood, sweat and tears to fight against it.”
CARTER:
Yeah, I see a lot more of the OEMs going towards this anyway with their infrastructure. It's something a lot of places are getting now if you do an upgrade, but doing upgrades cost a lot of money. It does take an outage and outages are money. So, downtime, all that stuff – nobody wants any of that. But maybe these other benefits will actually help make that a little easier to swallow than only doing this because NERC said so.
PATRICK:
Right. It's arguable, and not guaranteed in every case, but in many cases (or even in most cases) that this could actually improve the uptime. So then, in the future, you have fewer outages.
CARTER:
Yes, with some of these technologies you can actually see misconfigurations happening on your network. You can see… why is that controller getting all this extra data that's being either blocked or ignored from these other devices? Oh, man, we forgot last time we did the upgrade, we never removed that PLC from 10 years ago or whatever. So, there may be some of that going on, that could actually help you out too and make things less complicated. But again, that information has got to be brought forward into a pane of glass that an operator will actually sit down, look at, and pay attention to, understand, and find value in.
PATRICK:
Yeah. Which gets into a completely different discussion about how do we get security telemetry on their operations screens, but maybe a discussion for another time.
CARTER:
No, that's a tough it's a tough one. It’s not via MODBUS, I’ll tell you that!
PATRICK:
All right. Well, I really appreciate the conversation today and fantastic ideas, analysis. Always, always, always a pleasure. Thank you so much.
CARTER:
Thanks, Patrick. Good talking to you.
Show links:
Carter Manucy LinkedIn Profile - https://www.linkedin.com/in/cmanucy/
FERC Order 887 - https://www.ferc.gov/media/e-1-rm22-3-000
National Security Memorandum on Improving Cybersecurity for Critical Infrastructure Control Systems - https://www.amperesec.com/blog/industry-brief-national-security-memorandum-on-improving-cybersecurity-for-critical-infrastructure-control-systems
NERC INSM Practice Guide - https://www.nerc.com/pa/comp/guidance/CMEPPracticeGuidesDL/CMEP%20Practice%20Guide%20-%20Network%20Monitoring%20Sensors.pdf
