Utilities Racing To Secure Electric Grid 113
FreeMichael61 writes "In the latest episode of Spy vs. Spy, China rejects accusations it's hacking U.S. companies to steal IP or bring down the grid. But there's no doubt the grid can be hacked, CIO Journal's Steve Rosenbush and Rachael King report. Industrial control networks are supposed to be protected from the Internet by an air gap that, it turns out, is largely theoretical. Internal security is often lax, laptops and other devices are frequently moved between corporate networks and control networks, and some SCADA systems are still directly connected to the internet. What security standards actually exist are out of date and don't cover enough, and corporations often use questionable supply chains because they are cheaper."
Best Nerdcore Band Name Ever (Score:4, Funny)
Theoretical Air Gap!
Re:Best Nerdcore Band Name Ever (Score:4, Funny)
Well, I, uh, don't think it's quite fair to condemn a whole program because of a single slip-up, sir.
Re: (Score:3)
Thank you, General Turgidson, that will be all.
"Racing to secure Electric Grid" (Score:2)
I picture some arrogant fool at a race, who spends several minutes fast walking backwards around the track. When the race is almost won, then he starts running in the right direction.
With luck, the utilities will be back at the starting line before the competition crosses the finish line.
it always baffles me (Score:5, Insightful)
... why are mission critical devices connected to the internet
sure we know that the weakest link is the meatware, not the hardware, but still...
Re: (Score:1)
TFA answered this. It stated:
"The production systems are supposed to be kept off line... so they aren’t vulnerable to viruses distributed via the Internet." followed by, "...these air gaps can be hopped if the two systems use common computer peripherals such as... USB sticks..."
He is saying not connecting these to the internet is not enough.
Re: (Score:2)
fair enough... ... no connectivity either, or at least tightly controlled connectivity.
Simple solution... (Score:4, Interesting)
Re: (Score:2)
TFA answered this. It stated:
"The production systems are supposed to be kept off line... so they aren’t vulnerable to viruses distributed via the Internet." followed by, "...these air gaps can be hopped if the two systems use common computer peripherals such as... USB sticks..."
He is saying not connecting these to the internet is not enough.
It's amazing what security enhancements can be made with a little dollop of silicone glue.
"Bit o' Silastic in the USB port when you're finished, George..."
Re: (Score:2)
as Marvin would say (Score:2)
It will just end in tears.
At some point... (Score:2)
At some point, someone needs to interact with the system, and the system needs to interact with the devices it controls. So, in theory, these systems cannot be completely isolated.
Also, consider that if these systems were isolated with a pure vacuum, it may make the universe fly apart!
http://science.slashdot.org/story/13/02/19/2151238/does-the-higgs-boson-reveal-our-universes-doomsday [slashdot.org]
Re: (Score:1)
Re:it always baffles me (Score:5, Interesting)
They aren't supposed to be online, no. What you have though is the desire to do remote monitoring. One of the SCADA systems I used had an email module so you could get an email when things got all fucked up. That's a super awesome feature to have on a mission critical device.
"Hey, Beardo, it's Loader 1. Probably nothing to worry about, but sensors picked up a fluctuation in the output. Last time this happened the system crashed hard. Yeah, I know you're in a movie. Come check on meeee."
Now if this was up to me, and I know it's not, I'd build that module with an optoelectronic relay so it can send messages but be physically incapable of receiving them. Of course that does limit the usefulness, I can't send back messages, but I could call the place and let the night crew know there's a problem (if they aren't already aware) and how to mediate it.
Re:it always baffles me (Score:4, Interesting)
If the SCADA system is architected properly, remote monitoring is done via a Historian server that does not have the ability to affect the control systems.
I helped setup a Honeywell system to run a power plant in central California. My job was to architect the network piece of it. The hardware itself was completely mirrored in a typical master / slave relationship so that if the master failed, the slave was completely synchronized and could pick up the load.
There was a hardware firewall in between the production network and the Historian. The connection between the two was one way so that the it could report historical data for reporting purposes.
The corporate network connected to the historian via an IPSEC/AES-256 VPN connection. The switch fabric was redundant and the firewall used dual-homed, active/passive connections to mitigate against the potential of a switch failure.
Re:it always baffles me (Score:4)
There was a hardware firewall in between the production network and the Historian. The connection between the two was one way so that the it could report historical data for reporting purposes.
What I'm seeing you say is what I'd like to hear, but unfortunately what I'm reading is you were depending on a piece of software on a firewall to be vulnerability-free in order to provide your one-way communications.
Re: (Score:2)
I am relying on the firewall. If someone is hacking a Cisco ASA that is set to default deny, I have bigger problems.
Re: (Score:1)
Re: (Score:2)
Without knowing anything at all about SCADA except that it is a thing (or group of things) that exists:
Real, solid 1-way data connections are entirely possible. As a basic and slow example, RS-232 with only TXD and ground connected will only allow data to go in one direction.
Re: (Score:2)
Real, solid 1-way data connections are entirely possible. As a basic and slow example, RS-232 with only TXD and ground connected will only allow data to go in one direction.
Yes, this is precisely the line along which I am thinking. This is an interface that, barring strange quirks in the serial driver, you're not going to be hacked through. When someone says "it's got a one-way network connection" I say "bullshit". If they say it's got a one-way serial connection or something like that, I can buy that. Even then it's not impossible, merely astronomically improbable that someone can successfully pull off an attack through that channel. And if all you have is the TX connected to
Re: (Score:2)
Re: (Score:2)
That's fine, until the device that you need to monitor is 27 miles away. Even RS-485 is only good for 4000 feet.
There are optical data-diodes (yes that is basically the industry term for a hardware unidirectional network connection).
the whole mess is located at a power dam that's an hour's drive away and has been snowbound for the last three months (real world example, BTW).
Most plants are going to manned. Which makes a manually-enabled 2-way network an option. Only pjhysically power it on for maintenance with a VPN and you've narrowed the vulnerability window down to practically nothing.
Re: (Score:2)
Huh? Where? Not in North America.
Re: (Score:2)
Show me someone hacking through a default deny policy on a Cisco ASA and I will start to worry. Until then, this is going into the "mountains out of molehills" category.
The other side of the connection (the remote monitoring side, not the SCADA side) is a VPN tunnel with a single IP endpoint and a policy that only allows it to connect to the historian server. So first you'd have to hack the VPN concentrator and convince it to allow you to route traffic to a host that is not defined by the policy, and then
Re: (Score:2)
Even easier. Pull out a RS232 cable to run between a internet connected computer and the SCADA system.
You can still send detailed alerts and possibly even remotely monitor the site. Just keep it read only and use a teeny tiny bit of quality code and its safe.
Its using stuff like IP which is the stupid part.
Re: (Score:2)
Re: (Score:2)
TX isn't necessarily bad with a serial line either for situations that require it.
You just need a decent chunk of good code to authenticate everything and validate it.
Plugging it directly in to the internet isn't the right solution pretty much ever.
Mind you thats only slightly worse than a serial line with dodgy code controlling it.
Re: (Score:2)
The SCADA system that I have been most exposed to covered an entire very large county's utility district.
Re: (Score:2)
Erm you can connect the serial to the internet or private internal network or VPN.
The point of the serial connection is to limit the attack area.
No ports, no other services running, nothing excessive which isn't needed to communicate the raw data.
That was my point.
Re: (Score:2)
Only connect one data link and ground, and it will be secure. Run a balanced pair if you are feeling fancy. If the receiver can keep up, you don't even need flow control lines. Problem solved.
You could just set a led that blinks when there is something wrong and have a 2 dollar webcam monitor it. If someone manages to force data through a led that isn't configured for such activity, they have won the internet and deserve access.
Re: (Score:2)
That's what I said, an optoelectronic relay.
Re: (Score:1)
Re: (Score:2, Informative)
Cut the crap.
Thereare millions of mission critical things that are online for good reasons.
Just do it right.
Assuming,you don't have to do it right,because there's a air gap or anything else the sales guy would say when explaining why you don't have to hire an expensive network security guy will just get you in trouble.
It's like trusting a car salesman that this car is cheaper because it uses full synthetic oil so you never have to change it again.
Re: (Score:2)
There are millions of IT folks that would love to do that, but, inexplicably, they are not the people who make the decisions as to what technology to use. Those decisions are frequently made by MBAs/C-level morons who haven't seen a line of code in twenty years, or by the bean counters in Accounting who can barely open Excel. The IT folks just have to clean up after the tremendously bad decision-making that is a result.
Re: (Score:2)
Would be better if it just said "loader on fire".
Re: (Score:2)
Or they could just use, oh I don't know... let's say some other sort of dedicated communication device. We could call it a "telephone."
Re: (Score:2)
I've done those too. Hooked up an output signal to an autodialler to call myself.
"Hey Beardo, that loader is all fucked up again. Pick up bananas on the way home."
The only downside is you can't get data on the fly, just a generic error.
Happens all the time (Score:5, Informative)
Do you think that the energy industry is any easier on IT folks than anybody else?
Big dollar consultants instead of trained employees, given full unescorted access because the manager doesn't want to have to sit in the datacenter and escort them to the restrooms and such.
My SCADA datacenter still allows a cleaning crew in unescorted.
And electricians, and HVAC contractors and so on.
I found out they were PAINTING my datacenter the day that my storage started freaking out with heat alarms. Went running downstairs to find the facilities team had left a painting crew in the datacenter to cover all of my cabinets (and vented tiles) with tarps.
So these devices might not start connected to the internet, but a USB key here, a rogue cellular wi-fi bridge there, and some wild stuff can happen.
I've heard of other shops that had their SCADA people upset that they couldn't work from home, so they set up "secret" networks that only they knew about so they could still get in. Secret to their co-workers/management, but easy to find for the people who do that for a living.
Going anon for good reason.
Not all companies are equally bad (Score:4, Interesting)
One of my clients is a large electric utility. Their security, both physical and for IT systems, is top notch. None of their SCADA systems are online, they do routine and regular audits of all security, and even 'trusted' people like myself have to jump through hoops to get into the Data Center, and are always escorted.
They have really cool doors to get in too. They are like decontamination booths. You step into a vertical tube and wait to be cleared then the tube rotates and opens the other side.
On the other hand, I've done work for other utilities where yes, the cleaning crew goes in through what amounts to an open door, without an escort.
Re: (Score:3)
not just work from home but more remote plants swt (Score:2)
not just work from home but more remote plants switching / sub stations. Also the control centers need to be able to control all of that or do want to have some at each mid size to big substation 24/7 tied to a phone and control bank? As well a ready to go on call linemen who will drive out to the smaller ones to filp the switches?
power lines have reclosers some kind of wireless links on them.
Re:it always baffles me (Score:5, Informative)
... why are mission critical devices connected to the internet
sure we know that the weakest link is the meatware, not the hardware, but still...
They aren't, at least, not directly. They are however generally connected at various points to the "business" network which is connected to the Internet (people gotta email). The literal air gap is largely fiction. The business network is hacked, then some vulnerability exploited in the bridge points or routers (it's a network of networks!). Why connect the SCADA to the business network at all? To get the data out to do reports, send email alarms etc. in theory this data exporting should be secure. Problem is that who is hacking your SCADA system? It's not the usual suspects; there is no money in it and the barrier of entry is too high for the script kiddies. It's other countries wanting to perform espionage. How the hell do you protect against that? Look at stuxnet, I mean really look at how that took down the centrifuges. Governments have resources that the average hacking group simply doesn't (or SCADA group). They also have no reason to reveal a compromised system. There could be sleeper, targeted, custom malware sitting on every SCADA server in the US, just waiting for the a time where it will be useful to activate. It's a brave new world!
Re: (Score:2)
The question then becomes: Why the hell are mission-critical systems connected to business networks that are themselves connected to the Internet? Surely the government could quit blowing our taxes on dumb, pointless shit for just long enough to get any hardware and technicians required to completely segregate these critical systems from any "business" network that may be connected to the Internet or vulnerable to malware hopping from some employee's USB flash stick? The idea that such critical systems co
Re: (Score:2)
The alternative to a SCADA system is basically a bunch of stand-alone PLCs. They are heavily used for sub-process control (black-box control), and also for validating the proper operation of the SCADA system. But, to think you can do everything a modern SCADA system does with hard-wired PLCs is disingenuous. The controls become the network.
It all comes down to economics. You can make a system extremely robust, but you have to start from the smallest component which increases cost by an order of magnitud
Re: (Score:2)
Like I said, if the government would just quit wasting our tax dollars, the cost would really not be much of a problem. Just look at all the money going into funding a massively failed drug war and an overpopulated prison system, and the money that goes into funding recycling worthless things like plastic and paper. Not to mention our outsourcing of... well, almost everything, out to places like China. And does the government care? Nah... doesn't seem like it. They're only just now starting to open up
Re: (Score:2)
Re: (Score:3)
Re:it always baffles me (Score:4, Insightful)
Why the hell are mission-critical systems connected to business networks that are themselves connected to the Internet?
Because the functioning of the business relies integrally on both.
Look... I sympathize with the "air gap" argument, but it's not the mid-90's anymore. Business has been transformed by the ability to connect industrial systems with centralized command centers with payment systems with other companies. It's not for execs to have bullshit ipad dashboards... it's for the business to make operational decisions that will take effect in the upcoming hours/minutes/seconds, to meet contractual and legal obligations, to feed customer- and billing-related systems (no point in running a business if you can't cut a bill, eh?).
The world's not going back... VPN's, firewalls, segregated networks, etc., etc., but "air gap" won't do it anymore. Data is the lifeblood of business.
Re: (Score:2)
They aren't, at least, not directly.
In some cases they actually are, though strictly speaking it's more of a VPN type system. The lines between SCADA, PLC, DCS, etc are extremely blurred these days. You'll find a large portion of SCADA systems aren't actually SCADA at all but rather remotely controlled and managed control systems. They rely a lot of security by obscurity.
I work at a plant where we control several devices via SHDSL links. The modems have internet facing IP addresses and are connected to a SCADA system at one end and to our dis
Re: (Score:1)
Re: (Score:3)
Because being connected to the internet saves a *lot* of money. Instead of having to have an entire emergency team on site at all hours, you can get away with a minimal team at nights/weekends, and workers who can, in an emergency, connect from home.
It takes a very capable manager who can persuade the higher ups that its necessary to continue spending a few millions dollars in wage costs every year to avoid what (at least until very recently) se
Unmanned. (Score:3)
Because mission critical devices may not be manned. This is a rising trend in remote asset management. It's used extensively in upstream processing and pipelining that is slowly working it's way to downstream.
Heck one large gas ... manufacturer (though it's hard to call air separation "manufacturing") in our country runs all plants remotely. Sure there are staff there, but no one in the control room, no one in front of the computers. The onsite staff are used to bring the plant online and handle emergency c
Re: Fuck off (Score:1)
he was downmodded for the attitude, not the information.
if a person can't keep their cool during a discussion they annoy me. I suspect you don't see it very differently.
Re: (Score:2, Interesting)
Half of these articles don't even hide the fact that they're written and promoted by people that are looking for government money to secure infrastructure. Often it's even infrastructure that they own and that they're responsible for. One such person is even named in the first sentence of this article.
We're all in danger! Quick better make some new laws, imprison a few more people, and find a hero that can protect us!
Re: (Score:1)
Well, yes, and apparently, telling them to fuck off is uncouth. Who'da thunk... I guess everybody out there is saying, "Just give 'em the money" Somehow that is more "civilized".
China tries to crack everything, news at 11 (Score:5, Interesting)
Re: (Score:2)
I'm not saying the government there is doing it, but I do know that there is no other country which is attacking everything everywhere this aggressively
I'm not saying that you're contradicting yourself, but you're contradicting yourself.
Re: (Score:1)
I'm not saying the government there is doing it, but I do know that there is no other country which is attacking everything everywhere this aggressively
I'm not saying that you're contradicting yourself, but you're contradicting yourself.
I'm not saying that your reading comprehension skills are for shit, but seriously?
Re: (Score:1)
I'm not saying the government there is doing it
You say that, and then immediately follow with this:
there is no other country which is attacking everything everywhere this aggressively
So either you're contradicting what you just said, you're attempting to imply that the mountains, rivers, fields and rocks of China are attacking other countries or you're drawing no distinction between the people living in China and the 'country' of China. And why would any given person in China be 'China'? Could I fairly categorize all Americans as bone-headed uneducated sloths with 150 TV channels who love the
Meh. (Score:1)
Call me when the USA stops digging around in everyone's financial records.
Re: (Score:1)
they explicitly do not dig around in nearly any financial records at all.
Oh really [wikipedia.org]? If you think this is really aimed at stopping terrorism, keep in mind that the entire 9/11 attack could have easily been funded by some Saudi Prince diverting funds from a good weekend in Las Vegas. The CIA's surveillance won't pick that sort of thing up. This is financial espionage, plain and simple.
You are coming off as someone that uses bias and calls it reason.
Sometimes one has to, to make a point. Look at the USA's espionage program. Enter the country with a laptop and you risk having it searched. Funny. Terrorists coming in would have empty laptops. The
Re: (Score:2)
I've seen at least one attack per minute since I took my ISP on-line on the NSF-managed Internet ~1993 (in those days Chile and .vz were the main source IIRC) whenever I've looked. And I still get upwards of ~10,000 SPAM attempts on my mail accounts per day, at least when I could last be bothered to waste the CPU cycles and Flash write cycles to count them. Attacks across the Net are not new. I was the only UK ISP even attempting to protect my own systems with an firewall (which I wrote and we nominally
Re: (Score:2)
The best defence is interdependence (Score:3, Interesting)
China benefits from a functional United States. So long as the benefits outweigh any prize that would remove them in the taking, Americans are fairly safe from Chinese attack.
Re: (Score:2, Interesting)
1: Does china control their military any better than the USSR did?
2: Mapping out US electrical utilities is a big deal because if you want to disable your opponents energy infrastructure you need to know where all the substations are at. Those are far more vulnerable than the power stations themselves.
3: Also there are trade secrets to acquire as well as contracts. If you know who they do business with, and you can copy their technology, then you can sell to those companies and make buko bucks doing so.
Re: (Score:2)
Mapping out electrical utilities is not a big deal, it is trivial. It is perfectly legal to drive around the country following power lines and they can find agents who blend in and can claim to be on vacation, looking for property or whatever. If there were a serious danger of attack on us via our infrastructure someone would have done it already because it is so very unprotected.
Re:The best defence is interdependence (Score:4, Informative)
Mapping out electrical utilities is not a big deal, it is trivial. It is perfectly legal to drive around the country following power lines and they can find agents who blend in and can claim to be on vacation, looking for property or whatever. If there were a serious danger of attack on us via our infrastructure someone would have done it already because it is so very unprotected.
Drive around the country? Google Maps, my friend. You can follow power lines all over the place from the comfort of your living-room.
Re: (Score:2)
Re: (Score:2)
Well, no, this ties in with my "someone would have done it already" idea, because they would have done it even before google maps.
Re: (Score:2)
Re: (Score:3, Informative)
China benefits from a functional United States
"Functional" is a very broad term. Everything could be "functional" and still be wired for demolition (in the virtual sense) at the push of a button halfway around the world, and furthermore laced with failsafes so that any attempt to tamper with it blows it all up in our faces. It could be that way right now and nobody knows it (or is telling us about it). Change the names around and think about it a moment: Someone infiltrates Iran's industrial control infrastructure in this way, and once it's completely
Re: (Score:2)
Someone infiltrates Iran's industrial control infrastructure in this way, and once it's completely irrevocable, issue what amounts to a blackmail notice. If it all worked as designed then Iran has no choice but to give in to any demands made, or have irrecovable damage done to their country.
Ah, I see you've lived through a few Microsoft product upgrade cycles.
Re: (Score:1)
Oups! it would not satisfy the 21st century God called Economy.
it's funny how so many americans think so highly of their "economy"
won't be long and the american economy will be back to the barter system, and there won't be any need to secure the electricity grid because it won't be running because nobody will be able to afford to pay for it
maybe i'm a kook, but everyone who calls out economic bubble bursts in advance are kooks before it happens
Re: (Score:2)
China benefits from a functional United States. So long as the benefits outweigh any prize that would remove them in the taking, Americans are fairly safe from Chinese attack.
This was my first thought. What possible motive would the Chinese have for wanting to bring down the US power grid? Do they want the value of the dollar to plummet? The US and China are economically interdependent. Sure, they want to spy on us. But everyone spies on everyone these days. This supposed threat is just fear mongering.
China shouldn't be the concern (Score:2, Interesting)
The problem comes from the previous generation of smart meter addressing which included broadcast groups and whose keys were managed by the utilities via HSMs. The tech is solid, but when you are dealing with utilities whom have very little real sophistication on the IT side dealing with crypto technologies they don't understand, bad things can and will happen.
Get access to the HSM at the provider, or the smart cards they've backed up keys onto, and you can forge a packet that will trigger a significant nu
So when do people lose their jobs... (Score:2)
...over this bullshit? How many times do we have to hammer into managers and security teams alike that this shit is serious? When do we just start replacing ineptitude with people who give a shit?
bailout baby. bailout! (Score:2)
bailout - its the New Capitalism.
Internet Air Gap (Score:1)
Mr. President, we must not allow... an Internet air gap!
Outsourcing...what could possibly go wrong? (Score:1)
And I'll give you two guesses where the original coding work was outsourced to...
Re: (Score:2)
Mistakes happen (Score:2)
Firewalls are pretty vulnerable. In order to really defend a network, you can never make a mistake. And everybody makes a mistake from time to time And once they are in, they are hard to get out.
Much focus needs to be made on things like well made interfaces and quality documentation that has no ambiguities or errors. Many times mistakes are made because something just wasn't clear enough, and it was interpreted to be something other than what it really is. Security itself is hard in part because of so many parameters and settings. For example some value being entered might be unclear whether it is the name of something, or is being used to search for something, or is being used as a match expre
SCADA Systems Are Connected For a Reason (Score:1)
By definition a SCADA Master system is connected to any number of remote units spread all over geography. The physical links that make up this network are also extremely vulnerable as encrypted SCADA protocols are as yet fairly rare. For a cash strapped utility with a limited number of technicians, being able to remotely connect to equipment in difficult to reach places becomes very attractive.
Existing Systems... (Score:1)
are already compromised? Set up new security using compromised systems; it's the long con.
lazy and stupid (Score:2)
Re: (Score:2)
The SCADA systems themselves are almost never actually on the Internet unless they're running through a VPN tunnel. They frequently are connected to the company's business network though, which for obvious reasons IS on the Internet, which is often the problem. Limiting and securing the link between the SCADA and business network should be the focus, not running around crying "The sky is falling! I need tax dollars to fix my private infrastructu
lol (Score:1)
I also wonder about the quality of engineer they hire when they design these systems. Why arent they behind firewalls with ACLs that only allow a what NEEDS to communicate with it. Why arent there secondary boxes, so they can be patched as 0days come out? Why are they running software that has been reported to be riddled with vulnerabilities?
if you find yourself patching THAT often... its time to find new software written by security
Re: (Score:2)
Then where are we going to get cheap stuff?