Forgot your password?
typodupeerror
China Security IT News Technology

Utilities Racing To Secure Electric Grid 113

Posted by Soulskill
from the shouldn't-they-have-done-this-5-years-ago dept.
FreeMichael61 writes "In the latest episode of Spy vs. Spy, China rejects accusations it's hacking U.S. companies to steal IP or bring down the grid. But there's no doubt the grid can be hacked, CIO Journal's Steve Rosenbush and Rachael King report. Industrial control networks are supposed to be protected from the Internet by an air gap that, it turns out, is largely theoretical. Internal security is often lax, laptops and other devices are frequently moved between corporate networks and control networks, and some SCADA systems are still directly connected to the internet. What security standards actually exist are out of date and don't cover enough, and corporations often use questionable supply chains because they are cheaper."
This discussion has been archived. No new comments can be posted.

Utilities Racing To Secure Electric Grid

Comments Filter:
  • by Jah-Wren Ryel (80510) on Tuesday February 19, 2013 @08:00PM (#42950585)

    Theoretical Air Gap!

  • by gTsiros (205624) on Tuesday February 19, 2013 @08:07PM (#42950647)

    ... why are mission critical devices connected to the internet

    sure we know that the weakest link is the meatware, not the hardware, but still...

    • by Anonymous Coward

      TFA answered this. It stated:

      "The production systems are supposed to be kept off line... so they aren’t vulnerable to viruses distributed via the Internet." followed by, "...these air gaps can be hopped if the two systems use common computer peripherals such as... USB sticks..."

      He is saying not connecting these to the internet is not enough.

    • by Beardo the Bearded (321478) on Tuesday February 19, 2013 @08:38PM (#42950947)

      They aren't supposed to be online, no. What you have though is the desire to do remote monitoring. One of the SCADA systems I used had an email module so you could get an email when things got all fucked up. That's a super awesome feature to have on a mission critical device.

      "Hey, Beardo, it's Loader 1. Probably nothing to worry about, but sensors picked up a fluctuation in the output. Last time this happened the system crashed hard. Yeah, I know you're in a movie. Come check on meeee."

      Now if this was up to me, and I know it's not, I'd build that module with an optoelectronic relay so it can send messages but be physically incapable of receiving them. Of course that does limit the usefulness, I can't send back messages, but I could call the place and let the night crew know there's a problem (if they aren't already aware) and how to mediate it.

      • by dave562 (969951) on Tuesday February 19, 2013 @10:07PM (#42951733) Journal

        If the SCADA system is architected properly, remote monitoring is done via a Historian server that does not have the ability to affect the control systems.

        I helped setup a Honeywell system to run a power plant in central California. My job was to architect the network piece of it. The hardware itself was completely mirrored in a typical master / slave relationship so that if the master failed, the slave was completely synchronized and could pick up the load.

        There was a hardware firewall in between the production network and the Historian. The connection between the two was one way so that the it could report historical data for reporting purposes.

        The corporate network connected to the historian via an IPSEC/AES-256 VPN connection. The switch fabric was redundant and the firewall used dual-homed, active/passive connections to mitigate against the potential of a switch failure.

        • by drinkypoo (153816) <martin.espinoza@gmail.com> on Tuesday February 19, 2013 @10:24PM (#42951851) Homepage Journal

          There was a hardware firewall in between the production network and the Historian. The connection between the two was one way so that the it could report historical data for reporting purposes.

          What I'm seeing you say is what I'd like to hear, but unfortunately what I'm reading is you were depending on a piece of software on a firewall to be vulnerability-free in order to provide your one-way communications.

          • by dave562 (969951)

            I am relying on the firewall. If someone is hacking a Cisco ASA that is set to default deny, I have bigger problems.

            • by jadv (1437949)
              I agree with the parent poster. At some point you have to stop needing to have everything under your control and start relying on third-party suppliers. Or did you build your own car instead of buying one because you were afraid that the brakes might have a manufacturing flaw?
          • by adolf (21054)

            Without knowing anything at all about SCADA except that it is a thing (or group of things) that exists:

            Real, solid 1-way data connections are entirely possible. As a basic and slow example, RS-232 with only TXD and ground connected will only allow data to go in one direction.

            • by drinkypoo (153816)

              Real, solid 1-way data connections are entirely possible. As a basic and slow example, RS-232 with only TXD and ground connected will only allow data to go in one direction.

              Yes, this is precisely the line along which I am thinking. This is an interface that, barring strange quirks in the serial driver, you're not going to be hacked through. When someone says "it's got a one-way network connection" I say "bullshit". If they say it's got a one-way serial connection or something like that, I can buy that. Even then it's not impossible, merely astronomically improbable that someone can successfully pull off an attack through that channel. And if all you have is the TX connected to

              • by cusco (717999)
                That's fine, until the device that you need to monitor is 27 miles away. Even RS-485 is only good for 4000 feet. Frequently the operator will have to decide that since Input 01 and 03 are live but 02, 04 and 05 are not they need to trip Relay 1F, and the whole mess is located at a power dam that's an hour's drive away and has been snowbound for the last three months (real world example, BTW). One-way comms aren't enough for SCADA, and certainly not serial comms.
                • That's fine, until the device that you need to monitor is 27 miles away. Even RS-485 is only good for 4000 feet.

                  There are optical data-diodes (yes that is basically the industry term for a hardware unidirectional network connection).

                  the whole mess is located at a power dam that's an hour's drive away and has been snowbound for the last three months (real world example, BTW).

                  Most plants are going to manned. Which makes a manually-enabled 2-way network an option. Only pjhysically power it on for maintenance with a VPN and you've narrowed the vulnerability window down to practically nothing.

              • by dave562 (969951)

                Show me someone hacking through a default deny policy on a Cisco ASA and I will start to worry. Until then, this is going into the "mountains out of molehills" category.

                The other side of the connection (the remote monitoring side, not the SCADA side) is a VPN tunnel with a single IP endpoint and a policy that only allows it to connect to the historian server. So first you'd have to hack the VPN concentrator and convince it to allow you to route traffic to a host that is not defined by the policy, and then

      • Even easier. Pull out a RS232 cable to run between a internet connected computer and the SCADA system.

        You can still send detailed alerts and possibly even remotely monitor the site. Just keep it read only and use a teeny tiny bit of quality code and its safe.

        Its using stuff like IP which is the stupid part.

        • by cusco (717999)
          Fine, and how do you open the spillway on a dam that's inaccessible because of flooded roads if you only have one-way comms? SCADA systems aren't just for monitoring, operators need to react to that data as well.
          • TX isn't necessarily bad with a serial line either for situations that require it.
            You just need a decent chunk of good code to authenticate everything and validate it.

            Plugging it directly in to the internet isn't the right solution pretty much ever.
            Mind you thats only slightly worse than a serial line with dodgy code controlling it.

            • by cusco (717999)
              Of course you don't want to plug it into the Internet, but one-way comms aren't anything like an adequate answer either. Serial connections are only good for short distances, even RS-485 only reaches out to 4000 feet. When the relays that you need to be controlling might be in the next county or the next state that's just not going to cut it. I don't think you understand how these systems are used.

              The SCADA system that I have been most exposed to covered an entire very large county's utility district.
              • Erm you can connect the serial to the internet or private internal network or VPN.

                The point of the serial connection is to limit the attack area.
                No ports, no other services running, nothing excessive which isn't needed to communicate the raw data.
                That was my point.

      • There is always the physical layer.
      • Re: (Score:2, Informative)

        by nazsco (695026)

        Cut the crap.

        Thereare millions of mission critical things that are online for good reasons.

        Just do it right.

        Assuming,you don't have to do it right,because there's a air gap or anything else the sales guy would say when explaining why you don't have to hire an expensive network security guy will just get you in trouble.

        It's like trusting a car salesman that this car is cheaper because it uses full synthetic oil so you never have to change it again.

        • by BVis (267028)

          Just do it right.

          There are millions of IT folks that would love to do that, but, inexplicably, they are not the people who make the decisions as to what technology to use. Those decisions are frequently made by MBAs/C-level morons who haven't seen a line of code in twenty years, or by the bean counters in Accounting who can barely open Excel. The IT folks just have to clean up after the tremendously bad decision-making that is a result.

          It's like trusting a car salesman that this car is cheaper because it

      • by AmiMoJo (196126) *

        Would be better if it just said "loader on fire".

      • Or they could just use, oh I don't know... let's say some other sort of dedicated communication device. We could call it a "telephone."

        • I've done those too. Hooked up an output signal to an autodialler to call myself.

          "Hey Beardo, that loader is all fucked up again. Pick up bananas on the way home."

          The only downside is you can't get data on the fly, just a generic error.

    • Happens all the time (Score:5, Informative)

      by Anonymous Coward on Tuesday February 19, 2013 @08:45PM (#42951033)

      Do you think that the energy industry is any easier on IT folks than anybody else?

      Big dollar consultants instead of trained employees, given full unescorted access because the manager doesn't want to have to sit in the datacenter and escort them to the restrooms and such.

      My SCADA datacenter still allows a cleaning crew in unescorted.

      And electricians, and HVAC contractors and so on.

      I found out they were PAINTING my datacenter the day that my storage started freaking out with heat alarms. Went running downstairs to find the facilities team had left a painting crew in the datacenter to cover all of my cabinets (and vented tiles) with tarps.

      So these devices might not start connected to the internet, but a USB key here, a rogue cellular wi-fi bridge there, and some wild stuff can happen.

      I've heard of other shops that had their SCADA people upset that they couldn't work from home, so they set up "secret" networks that only they knew about so they could still get in. Secret to their co-workers/management, but easy to find for the people who do that for a living.

      Going anon for good reason.

      • by dreamchaser (49529) on Tuesday February 19, 2013 @09:23PM (#42951375) Homepage Journal

        One of my clients is a large electric utility. Their security, both physical and for IT systems, is top notch. None of their SCADA systems are online, they do routine and regular audits of all security, and even 'trusted' people like myself have to jump through hoops to get into the Data Center, and are always escorted.

        They have really cool doors to get in too. They are like decontamination booths. You step into a vertical tube and wait to be cleared then the tube rotates and opens the other side.

        On the other hand, I've done work for other utilities where yes, the cleaning crew goes in through what amounts to an open door, without an escort.

        • by bcong (1125705)
          Those doors are called man traps, and they do exactly what it sounds like they do if you are entering an area you are not supposed to.
      • not just work from home but more remote plants switching / sub stations. Also the control centers need to be able to control all of that or do want to have some at each mid size to big substation 24/7 tied to a phone and control bank? As well a ready to go on call linemen who will drive out to the smaller ones to filp the switches?

        power lines have reclosers some kind of wireless links on them.

    • by Puff_Of_Hot_Air (995689) on Tuesday February 19, 2013 @08:47PM (#42951061)

      ... why are mission critical devices connected to the internet

      sure we know that the weakest link is the meatware, not the hardware, but still...

      They aren't, at least, not directly. They are however generally connected at various points to the "business" network which is connected to the Internet (people gotta email). The literal air gap is largely fiction. The business network is hacked, then some vulnerability exploited in the bridge points or routers (it's a network of networks!). Why connect the SCADA to the business network at all? To get the data out to do reports, send email alarms etc. in theory this data exporting should be secure. Problem is that who is hacking your SCADA system? It's not the usual suspects; there is no money in it and the barrier of entry is too high for the script kiddies. It's other countries wanting to perform espionage. How the hell do you protect against that? Look at stuxnet, I mean really look at how that took down the centrifuges. Governments have resources that the average hacking group simply doesn't (or SCADA group). They also have no reason to reveal a compromised system. There could be sleeper, targeted, custom malware sitting on every SCADA server in the US, just waiting for the a time where it will be useful to activate. It's a brave new world!

      • The question then becomes: Why the hell are mission-critical systems connected to business networks that are themselves connected to the Internet? Surely the government could quit blowing our taxes on dumb, pointless shit for just long enough to get any hardware and technicians required to completely segregate these critical systems from any "business" network that may be connected to the Internet or vulnerable to malware hopping from some employee's USB flash stick? The idea that such critical systems co

        • by aaarrrgggh (9205)

          The alternative to a SCADA system is basically a bunch of stand-alone PLCs. They are heavily used for sub-process control (black-box control), and also for validating the proper operation of the SCADA system. But, to think you can do everything a modern SCADA system does with hard-wired PLCs is disingenuous. The controls become the network.

          It all comes down to economics. You can make a system extremely robust, but you have to start from the smallest component which increases cost by an order of magnitud

          • Like I said, if the government would just quit wasting our tax dollars, the cost would really not be much of a problem. Just look at all the money going into funding a massively failed drug war and an overpopulated prison system, and the money that goes into funding recycling worthless things like plastic and paper. Not to mention our outsourcing of... well, almost everything, out to places like China. And does the government care? Nah... doesn't seem like it. They're only just now starting to open up

            • by cusco (717999)
              Why in the world do you think 1) the private corporations that run our energy infrastructure should get our tax dollars to fix their security issues, or 2) the private corporations that run our energy infrastructure would allow new laws to enforce security to be passed? Even if 2) happens, who is going to enforce it?
        • They're not government owned in the US...
        • by firewrought (36952) on Wednesday February 20, 2013 @03:11AM (#42953367)

          Why the hell are mission-critical systems connected to business networks that are themselves connected to the Internet?

          Because the functioning of the business relies integrally on both.

          Look... I sympathize with the "air gap" argument, but it's not the mid-90's anymore. Business has been transformed by the ability to connect industrial systems with centralized command centers with payment systems with other companies. It's not for execs to have bullshit ipad dashboards... it's for the business to make operational decisions that will take effect in the upcoming hours/minutes/seconds, to meet contractual and legal obligations, to feed customer- and billing-related systems (no point in running a business if you can't cut a bill, eh?).

          The world's not going back... VPN's, firewalls, segregated networks, etc., etc., but "air gap" won't do it anymore. Data is the lifeblood of business.

      • by thegarbz (1787294)

        They aren't, at least, not directly.

        In some cases they actually are, though strictly speaking it's more of a VPN type system. The lines between SCADA, PLC, DCS, etc are extremely blurred these days. You'll find a large portion of SCADA systems aren't actually SCADA at all but rather remotely controlled and managed control systems. They rely a lot of security by obscurity.

        I work at a plant where we control several devices via SHDSL links. The modems have internet facing IP addresses and are connected to a SCADA system at one end and to our dis

    • by simishag (744368)
      I can appreciate your sentiment, but I think it's wishful thinking. We can certainly argue that these devices SHOULD not be connected to the Internet, but the simple fact is that a great deal of them ARE connected, and many that are not "intended" to be connected will end up connected, and those systems need to be designed with that possibility in mind. They are currently designed with no more security than my pull-start lawn mower.
    • by west (39918)

      ... why are mission critical devices connected to the internet

      Because being connected to the internet saves a *lot* of money. Instead of having to have an entire emergency team on site at all hours, you can get away with a minimal team at nights/weekends, and workers who can, in an emergency, connect from home.

      It takes a very capable manager who can persuade the higher ups that its necessary to continue spending a few millions dollars in wage costs every year to avoid what (at least until very recently) se

    • Because mission critical devices may not be manned. This is a rising trend in remote asset management. It's used extensively in upstream processing and pipelining that is slowly working it's way to downstream.

      Heck one large gas ... manufacturer (though it's hard to call air separation "manufacturing") in our country runs all plants remotely. Sure there are staff there, but no one in the control room, no one in front of the computers. The onsite staff are used to bring the plant online and handle emergency c

  • by xiando (770382) on Tuesday February 19, 2013 @08:11PM (#42950695) Homepage Journal
    Anyone with a web-server will tell you that they are seeing dozens of penetration attempts daily, even right now. I also see this on my home ADSL line. I'm not saying the government there is doing it, but I do know that there is no other country which is attacking everything everywhere this aggressively. I don't have any web pages in Chinese and I wonder if I would be better off just using one of those iptables -j DROP lists who list all IPs in China.
    • I'm not saying the government there is doing it, but I do know that there is no other country which is attacking everything everywhere this aggressively

      I'm not saying that you're contradicting yourself, but you're contradicting yourself.

      • by drinkypoo (153816)

        I'm not saying the government there is doing it, but I do know that there is no other country which is attacking everything everywhere this aggressively

        I'm not saying that you're contradicting yourself, but you're contradicting yourself.

        I'm not saying that your reading comprehension skills are for shit, but seriously?

        • Arrgghhh, I hate having to explain myself.

          I'm not saying the government there is doing it

          You say that, and then immediately follow with this:

          there is no other country which is attacking everything everywhere this aggressively

          So either you're contradicting what you just said, you're attempting to imply that the mountains, rivers, fields and rocks of China are attacking other countries or you're drawing no distinction between the people living in China and the 'country' of China. And why would any given person in China be 'China'? Could I fairly categorize all Americans as bone-headed uneducated sloths with 150 TV channels who love the

    • by PPH (736903)

      Call me when the USA stops digging around in everyone's financial records.

    • by DamonHD (794830)

      I've seen at least one attack per minute since I took my ISP on-line on the NSF-managed Internet ~1993 (in those days Chile and .vz were the main source IIRC) whenever I've looked. And I still get upwards of ~10,000 SPAM attempts on my mail accounts per day, at least when I could last be bothered to waste the CPU cycles and Flash write cycles to count them. Attacks across the Net are not new. I was the only UK ISP even attempting to protect my own systems with an firewall (which I wrote and we nominally

      • by cusco (717999)
        It's currently fashionable to blame the Chinese government for every attempted hack across the Internet these days, but people generally forget that "China" encompasses 1/4 of the world's population. That's a frack of a lot of individuals who might be carrying out their own attacks, and a heck of a lot more who haven't secured their computer (or the computers in their Internet cafe or grade school computer lab) and are participating in botnets. At least they're not blaming the phantasmagorical "army" of N
  • by Baron_Yam (643147) on Tuesday February 19, 2013 @08:20PM (#42950765)

    China benefits from a functional United States. So long as the benefits outweigh any prize that would remove them in the taking, Americans are fairly safe from Chinese attack.

    • Re: (Score:2, Interesting)

      by Anonymous Coward

      1: Does china control their military any better than the USSR did?

      2: Mapping out US electrical utilities is a big deal because if you want to disable your opponents energy infrastructure you need to know where all the substations are at. Those are far more vulnerable than the power stations themselves.

      3: Also there are trade secrets to acquire as well as contracts. If you know who they do business with, and you can copy their technology, then you can sell to those companies and make buko bucks doing so.

      • by drinkypoo (153816)

        Mapping out electrical utilities is not a big deal, it is trivial. It is perfectly legal to drive around the country following power lines and they can find agents who blend in and can claim to be on vacation, looking for property or whatever. If there were a serious danger of attack on us via our infrastructure someone would have done it already because it is so very unprotected.

        • by camperdave (969942) on Tuesday February 19, 2013 @10:49PM (#42952029) Journal

          Mapping out electrical utilities is not a big deal, it is trivial. It is perfectly legal to drive around the country following power lines and they can find agents who blend in and can claim to be on vacation, looking for property or whatever. If there were a serious danger of attack on us via our infrastructure someone would have done it already because it is so very unprotected.

          Drive around the country? Google Maps, my friend. You can follow power lines all over the place from the comfort of your living-room.

          • by silanea (1241518)
            Following power lines on Google Maps? OpenStreetMap [openstreetmap.org], my friend. Some people have already gone to ridiculous levels of detail in mapping things that formerly would have gotten you a quick visit from your friendly domestic intelligence service. With many countries opening up their data this is only going to get worse, or better, depending on your point of view. And it is all - how fitting for the topic discussed in TFA - readily available on the Internet.
        • by drinkypoo (153816)

          Well, no, this ties in with my "someone would have done it already" idea, because they would have done it even before google maps.

        • by cusco (717999)
          If you're in the industry you just go to FERC or one of the industry groups and buy your maps directly from them. They even show things like the capacity of different stretches of power lines, output of power plants, interconnections, map out distribution routes, and the like for you.
    • Re: (Score:3, Informative)

      by kheldan (1460303)

      China benefits from a functional United States

      "Functional" is a very broad term. Everything could be "functional" and still be wired for demolition (in the virtual sense) at the push of a button halfway around the world, and furthermore laced with failsafes so that any attempt to tamper with it blows it all up in our faces. It could be that way right now and nobody knows it (or is telling us about it). Change the names around and think about it a moment: Someone infiltrates Iran's industrial control infrastructure in this way, and once it's completely

      • by lennier (44736)

        Someone infiltrates Iran's industrial control infrastructure in this way, and once it's completely irrevocable, issue what amounts to a blackmail notice. If it all worked as designed then Iran has no choice but to give in to any demands made, or have irrecovable damage done to their country.

        Ah, I see you've lived through a few Microsoft product upgrade cycles.

    • China benefits from a functional United States. So long as the benefits outweigh any prize that would remove them in the taking, Americans are fairly safe from Chinese attack.

      This was my first thought. What possible motive would the Chinese have for wanting to bring down the US power grid? Do they want the value of the dollar to plummet? The US and China are economically interdependent. Sure, they want to spy on us. But everyone spies on everyone these days. This supposed threat is just fear mongering.

  • by Anonymous Coward

    The problem comes from the previous generation of smart meter addressing which included broadcast groups and whose keys were managed by the utilities via HSMs. The tech is solid, but when you are dealing with utilities whom have very little real sophistication on the IT side dealing with crypto technologies they don't understand, bad things can and will happen.

    Get access to the HSM at the provider, or the smart cards they've backed up keys onto, and you can forge a packet that will trigger a significant nu

  • ...over this bullshit? How many times do we have to hammer into managers and security teams alike that this shit is serious? When do we just start replacing ineptitude with people who give a shit?

  • Mr. President, we must not allow... an Internet air gap!

  • And I'll give you two guesses where the original coding work was outsourced to...

    • by cusco (717999)
      For the SCADA systems? Generally Germany, Italy and the US. The last I heard even India bought SCADA software from the US.
  • Firewalls are pretty vulnerable. In order to really defend a network, you can never make a mistake. And everybody makes a mistake from time to time And once they are in, they are hard to get out.

    Much focus needs to be made on things like well made interfaces and quality documentation that has no ambiguities or errors. Many times mistakes are made because something just wasn't clear enough, and it was interpreted to be something other than what it really is. Security itself is hard in part because of so many parameters and settings. For example some value being entered might be unclear whether it is the name of something, or is being used to search for something, or is being used as a match expre

  • by Anonymous Coward

    By definition a SCADA Master system is connected to any number of remote units spread all over geography. The physical links that make up this network are also extremely vulnerable as encrypted SCADA protocols are as yet fairly rare. For a cash strapped utility with a limited number of technicians, being able to remotely connect to equipment in difficult to reach places becomes very attractive.

  • by Anonymous Coward

    are already compromised? Set up new security using compromised systems; it's the long con.

  • Don't put the systems that control the power grid on the internet! Or if you do, make them read-only. If they have to be networked, ever heard of a VLAN? Hell no they haven't because they hire outsourced 3rd party contractors to write this stuff and they don't have to officially deal with it after the check clears. They don't necessarily have to sit there and manage it and deal with the software and control systems on a daily basis. And they certainly aren't the best or brightest programmers, they're ju
    • by cusco (717999)
      Oh, yes, a VLAN! Security by obscurity, how very novel!

      The SCADA systems themselves are almost never actually on the Internet unless they're running through a VPN tunnel. They frequently are connected to the company's business network though, which for obvious reasons IS on the Internet, which is often the problem. Limiting and securing the link between the SCADA and business network should be the focus, not running around crying "The sky is falling! I need tax dollars to fix my private infrastructu
  • by fazey (2806709)
    Well, before now it wasnt affordable. All the money is in politics. =x

    I also wonder about the quality of engineer they hire when they design these systems. Why arent they behind firewalls with ACLs that only allow a what NEEDS to communicate with it. Why arent there secondary boxes, so they can be patched as 0days come out? Why are they running software that has been reported to be riddled with vulnerabilities?

    if you find yourself patching THAT often... its time to find new software written by security

COBOL is for morons. -- E.W. Dijkstra

Working...