Slashdot is powered by your submissions, so send in your scoop

 



Forgot your password?
typodupeerror
×
United Kingdom The Almighty Buck

$600k Fine Over Data Center Death (datacenterdynamics.com) 169

judgecorp writes: UK contractors Balfour Beatty and Norland have been fined £380,000 ($580k) after an electrician was electrocuted while working on a data center owned by finance firm Morgan Stanley. The fine follows mounting concern that safety is being compromised because of the need for data centers to remain online non-stop. This leads to pressure for contractors to work on live power supplies.
This discussion has been archived. No new comments can be posted.

$600k Fine Over Data Center Death

Comments Filter:
  • by dwywit ( 1109409 ) on Thursday October 29, 2015 @04:44AM (#50823449)

    How about a mandatory downtime for the data centre of say, 24 hours?

    Hit 'em in the hip pocket - which is what a fine is supposed to do, but rarely, in the case of corporations, achieves its desired affect.

    • Downtime? That would be when the work doesn't automatically switch to the other data centers, right?

      • by dwywit ( 1109409 )

        Yep, but an unscheduled 24-hour outage is still going to look bad on *someone's* quarterly report.

        The workload might switch elsewhere, but fixed costs don't.

      • If they are doing high-frequency trading, the extra milliseconds that the other datacenter is taking will kill their profits.

    • Re: (Score:2, Insightful)

      by Anonymous Coward

      Someone made the call to have live circuits worked on, despite this being illegal. That person should be tried for manslaughter. It's about time people stopped hiding behind the corporate veil and are made accountable for their decisions, just as you or I would be.

      • by Anonymous Coward

        There's a degree of incompetence from the engineer's side too. The first rule of electrical engineering is "turn it off". If it absolutely positively can't be turned off, don a Faraday suit and wear isolating gloves.

        • by AlecC ( 512609 )

          But to know to turn it off, you have to know it is on. And that appears to be the problem here - people didn't know what was on.

          • The second rule is "treat every circuit as live, even if you know it is not" So yes the electrician was not taking the proper precautions if he was not treating the circuit as live.

            • by vlad30 ( 44644 )
              Ditto and for $2 you can get a live wire electrical tester $5 will get you a contact-less version
            • by Z00L00K ( 682162 )

              Assuming that there's no asshat turning on the power while you are working because they needed the power NOW.

              • by Anonymous Coward on Thursday October 29, 2015 @07:01AM (#50823913)

                This is why (at least in the US), standard (and required) practice is to place a lock on the source of energy, and retain the only key to said lock. If multiple workers are all working on the same circuit, each worker has his (or her) own lock on the circuit.

                It's called lockout/tagout, and there are SEVERE fines for removing somebody else's lock (and if somebody gets killed, due to your removing his lock, that would be considered manslaughter)

                • This is why (at least in the US), standard (and required) practice is to place a lock on the source of energy, and retain the only key to said lock. If multiple workers are all working on the same circuit, each worker has his (or her) own lock on the circuit.

                  It's called lockout/tagout, and there are SEVERE fines for removing somebody else's lock (and if somebody gets killed, due to your removing his lock, that would be considered manslaughter)

                  It's mandatory in the UK too. Often ignored by independent electricians working by themselves, but hopefully less so by those working for large companies....... UK Health and Safety Executive page on the topic: http://www.hse.gov.uk/safemain... [hse.gov.uk]

                • Unfortunately, the case appears to be improper Energized Electrical Work (EEW) Google EEW hotwork. There are many cases where EEW is performed. Common applications include linemen replacing cutouts, changing insulators, installing new cutouts for new home construction, etc. EEW hotwork requires special permits and tools and protective clothing.

                  Death and resulting fines was due to the failure of following proper proceedures for EEW hotwork. LOTO is preferred over EEW, but there are reasons to do EEW.

                • by Z00L00K ( 682162 )

                  Well, it works for the main switch in a computer hall, but often power is only cut off on a circuit breaker for the circuits where the work is done, not the whole data center.

                  Unfortunately way too common. Often because the installation shall be cheap.

                • by RobinH ( 124750 )
                  You speak the truth, and yet I've worked with a lot of maintenance people and electricians who just don't lockout/tagout. They all think they're immune, I guess. Most electricians are used to working on live equipment.
              • by flink ( 18449 )

                You should be able to lock the breaker out and take the key with you so they can't turn the power on.

              • Or a BOFH or his PFY thinking the worker is in fact a member of Management or perhaps Accounting ...

              • Assuming that there's no asshat turning on the power while you are working because they needed the power NOW.

                That should be controlled by appropriate Lockout-Tagout [wikipedia.org], hard to turn something on when the switch is padlocked and the key is in your pocket.

              • Assuming that there's no asshat turning on the power while you are working

                This is what lockout-tagout isolators are for. YOU turn off the power on the circuit which YOU are going to be working on, and YOU put a padlock through the isolator locking it isolated ; and that padlock is normally a combination lock which YOU and ONLY you know the number for.

                This is not rocket science. It is designing safety into the system from day one. These days it is actually getting hard to find isolators which don't have the

            • The second rule is "treat every circuit as live, even if you know it is not" So yes the electrician was not taking the proper precautions if he was not treating the circuit as live.

              Coincidentally, last night I was talking to my father who worked as an electrician/electrical engineer for over 30 years. He said that back in the day, it was quite common for sparkies to work on live kit....they knew the cables were live and knew how to respect them and what precautions to take. This wasn't just restricted to the standard 240V supply (c.f. the 110V used on the other side of the Pond) but also to distribution kit, running well into the kilovolt range. For example, bare end of armoured ca

        • by Z00L00K ( 682162 )

          Turn it off, then do a short circuit to ground at the work position as well so if someone turns on the power then the fuse will blow. If they are unlucky then the main fuse will blow and that's going to make a mark in someone's report.

          If it's really bad the UPS will die - and that will definitely make a mark in the budget for that year.

        • by Anonymous Coward

          The official rules in Switzerland are:

          - turn it off
          - secure against accidently turning it on (e.g. a lock on the switch)
          - test for voltage
          - ground the wires
          - work

          Its a pain in the ass and a lot of workes disrespect the rules.

          But advice like "treat every wire as live" is pretty useless. What do you do with a live wire?

        • Standard Operating Procedure is to turn it off at the breaker box, then Red Tag the switches so that nobody turns it back on again. Anybody other than the person that red tagged it turning it back on risks getting their ass kicked for endangering someone else's life. As a matter of general principle, I'd probably throw a meter across the circuit to make sure it's not live before touching anything too, but that's not required, and there should be plenty of indicator lights anyway. I'd love to know what the t
      • Someone made the call to have live circuits worked on, despite this being illegal. That person should be tried for manslaughter. It's about time people stopped hiding behind the corporate veil and are made accountable for their decisions, just as you or I would be.

        No, companies are people now, better to charge the company with manslaughter and figure out how to send it to prison.

        • by Anonymous Coward

          Just require it to shut down all operations. However, people will still be entitled to their paychecks while the corporation is in prison, just as we still require child support payments from fathers in prison who are unable to work.

          If the corporation can survive being completely shut down for 2-3 years while paying its employees, creditors, lawsuits from breached contracts, etc from savings, then it can resume operations afterwards. I'm not sure many could survive that though, outside of maybe Apple and

        • by Z00L00K ( 682162 )

          To paraphrase what happens when programs do something bad: "The company has executed an illegal operation and will be terminated".

        • Someone made the call to have live circuits worked on, despite this being illegal.

          In the UK - where this happened - it's not illegal to work on live circuits. The HSE has this in the FAQ:

          When is it safe to work on live electrical equipment?

          It is never absolutely safe to work on live electrical equipment. There are few circumstances where it is necessary to work live, and this must only be done after it has been determined that it is unreasonable for the work to be done dead. Even if working live can be just

      • by AlecC ( 512609 )

        Reading the article, it appears that it is not that "somebody made the call", it is that communications between the teams working on the project, from two companies, was so bad that one crew didn't realist that people would be working in the area, and the other crew didn't realise it was live. Incompetence, not risk-taking.

        • Yes, and a competent electrician always assumes a circuit is live, even if he knows otherwise. You never know when some idiot is going to flip the switch back on that you turned off to kill the circuit, or if you simply flipped the wrong switch or clear forgot. If you're working on an electrical circuit you *always* assume it is live, no exceptions!

          • In your mind that might be easy.

            However how do you actually connect a life power conduct to a machine?

            You don't. You switch the power off.

            So how exactly do you now do the 'I connect it, but assume it is still life' when actually connecting a life power line is impossible?

            Sorry ... some of the posters repeating that myth are: morons.

      • by vlad30 ( 44644 )
        Not illegal everywhere but precautions must be taken when doing live work.
      • by jabuzz ( 182671 ) on Thursday October 29, 2015 @07:00AM (#50823905) Homepage

        It happened in the United Kingdom and live line working is perfectly legal. If you read the article you will see the death occurred due to a lack of communication causing the people to be unaware they where working in the vicinity of line equipment.

        Here is a link to the Health and Safety Executive's press release on the subject which has more details. The full judgement does not seem to have hit the judiciary web side yet. At least my searches are coming up blank.

        http://press.hse.gov.uk/2015/e... [hse.gov.uk]

        Basically the fines where from not operating proper health and safety systems. If someone has enough patience you should be able to dig out the full judgement from the

    • by Lumpy ( 12016 )

      $10X the "fine" must be paid directly to the victims family, Plus a mandatory Data-center full downtime of 48 hours while the "investigation" takes place.

      if they find ANY problems the fines and time down multiply.

      Make a death like this capable of completely crushing the company.

    • Well . . . I was wondering what the dollar value of a human life is these days. So now, I guess that it is about $600K . . . in the UK. The value of human lives might vary in other countries.

      You might want to consider that when planning where to put your data center . . .

      • by moeinvt ( 851793 )

        That might not be the full value. Note that this was a fine imposed for violations of the "Health and Safety Work Act". Maybe the UK is different, but in the USA, you could be certain that a civil lawsuit was in the works.

    • Mandatory downtime would not be punishing the DC, it would be punishing the customers of the DC would would loose far more money from loss of business than the DC itself would.

    • by jabuzz ( 182671 )

      Live line working is standard industry practice for power distribution.

      https://en.wikipedia.org/wiki/... [wikipedia.org]

      I see no reason why similar procedures cannot be developed for any electrical work.

      In general there is fewer incidents in live line working that dead line working. Mostly because if you know it is live then you will be treating with the utmost respect.

    • How about a mandatory downtime for the data centre of say, 24 hours?

      Hit 'em in the hip pocket - which is what a fine is supposed to do, but rarely, in the case of corporations, achieves its desired affect.

      Because you'd be punishing the customers, not only the DC.

  • Working live is recognized as an acceptable risk in circumstances where it is not practicable to turn stuff off (UPS's and backup-generators may have made this a logistics nightmare even if you DGAF about uptime.) There are qualifications you can get to prove your competence to work live, and there are some very comprehensive procedures to follow that make live working a reasonably safe undertaking (even for the boys doing live joints on HT cables)

    If the guy was a "competent person" in the eyes of the law,

    • by Pinky's Brain ( 1158667 ) on Thursday October 29, 2015 @06:11AM (#50823707)

      From reading TFA it seems he didn't know it was live because there were 2 companies at work without any single person being responsible for coordination.

    • When I read in the HSE publication [hse.gov.uk] that the hapless person "was electrocuted when his forehead made contact with the 415V live terminals of the second unit", I immediately facepalmed. The apparent lack of safety awareness truly boggles my mind.

      If he was not a competent person then he had no business attempting the work.

      The sad truth is, that the incompetent don't realise that they're not competent at the job.

  • by Anonymous Coward

    With Load balancing, fail over clusters, hot sites, all the thing that can make part of a site go offline for a while without serious impact.

  • by ledow ( 319597 ) on Thursday October 29, 2015 @06:33AM (#50823777) Homepage

    How much downtime was caused by ensuring the circuit was safe and removing the body?

    If zero, SUE THEM INTO OBLIVION. Risking either this contractor unnecessarily (you could have just switched it off) or other workers and emergency workers (because you didn't switch it off after it had demonstrably killed someone).

    If some downtime, then why couldn't you have done that to do the work?

    Sorry, but I fail to see how the risk of a death and possible short-circuits, joined phases etc. because of working with the terminals live in any way "secures" uptime any more than scheduling proper downtime and having properly redundant systems.

    You are just ask likely to bridge the WRONG circuit while working live, or causing a short, which will cause more damage and more downtime than just switching things off to do the work. And you guys have redundant power with UPS that you can bypass to work on the UPS, etc. if necessary? If not, that downtime isn't all that important to you anyway.

    There's no excuse for this, hence the court fine. And you've got to be an idiot to knowingly let people work on a live multi-phase system. Hell, even a fused, RCD'd, single-phase can be bad enough.

  • From my experience (Score:5, Interesting)

    by l0n3s0m3phr34k ( 2613107 ) on Thursday October 29, 2015 @06:40AM (#50823819)
    In working in data centers, I can totally see how this happened. Reading the actual source article, it reads like they had already connected the first circuit, and he got popped while working on the second. I would assume they had shut off both, installed the first PSU, then probably someone turned them BOTH back on instead of just the first one. When he went to connect the second PSU...

    These are the kinds of accidents proper "change control" is supposed to stop, it seems no one working there really knew the over-all implementation plan. At our local data center, we have actual licensed electricians for high DC stuff, they know to "never trust always test". Even though we contract all that out too, we try to make sure the people on the site are aware of these things via bright stickers, lock-outs, etc. I have no idea if they have required licensing and training for their "cable jointer" positions in the UK.
  • Management doesn't understand the difference between telling them it's technically possible to do live maintenance and that it's a challenge like the rest of our technical feats. I feel for the guy and his survivors. I've seen the same pressures play out on my data centers, but thankfully we were able to arrange that type of work as semi-regular full DR test and the place I was working at simply didn't have the same uptime demands when push came to shove. From a safety angle, I can't help but think that mo
    • DC is just safer to work with

      That depends...if you touch a live DC wire with your palm, your hand will naturally contract and grip the wire. Same thing with an AC wire, and your hand will be repelled. With enough power, both can still hurt like hell, but the latter is the better option.

      • The DC bus is likely at a lower voltage than typical AC circuits, therefore less able to cause electrocution.

        • The DC bus is likely at a lower voltage than typical AC circuits, therefore less able to cause electrocution.

          It's not the voltage that kills, it's the current. As little as 100mA is fatal if it crosses the heart. And if you want to work with lower voltage for the same job that means the current must be higher.

          • Ohm's law, though. For the same resistance (a human body), lower voltage == lower current.

          • The DC bus is likely at a lower voltage than typical AC circuits, therefore less able to cause electrocution.

            It's not the voltage that kills, it's the current. As little as 100mA is fatal if it crosses the heart. And if you want to work with lower voltage for the same job that means the current must be higher.

            The higher current goes to the equipment the powerlines are connected to. But that current will not affect YOU in any way. However, if you happen to touch a low voltage line, the current through your body will be lower (Ohm's law) than when you touch a high voltage line. Hence, low voltage is safer.
            Try taking the poles of a car battery (12V) in your hands and prodding your electrical outlet (110/230V) with two metal rods to feel the difference. Don't do it the other way round, chances are you won't be able

  • I feel bad for him, and I hope his family gets a decent settlement*, but have to ask why didn't he check the live line first? When I work on electrical stuff at home, I always check AND DOUBLECHECK that the breakers are off and that no juice is running. And, that's with voltages that won't necessarily kill me. Working with this level of voltage? Holy crap.

    * As stated by some in the UK, it's not likely to be a big settlement. Too bad in this case.

  • by tomhath ( 637240 ) on Thursday October 29, 2015 @10:41AM (#50825557)
    As I read the article the accident was caused by a screw up in communications by the contractor doing the work. They tried to blame it on the data center pressuring them, but the judge apparently didn't accept that argument and fined the contractor. It seems submitter is looking for evil where there is really just incompetence.
    • by ledow ( 319597 )

      When your customer kicks up a fuss for you to "just get it done", it's not licence to just ignore your training, skills and safety procedures. Even if they threaten to cut your business, you still can't just say "Oh, let's forget all that stuff this one time because we might lose future work".

      That's not how the law works, and not how it SHOULD work - the contractor shouldn't just cut corners because they're being pressured by the datacentre. They should say "We do it safely, or not at all". They didn't,

  • I don't understand why you would need to work on a live rain. Seems to me like a very badly designed system ..

A committee takes root and grows, it flowers, wilts and dies, scattering the seed from which other committees will bloom. -- Parkinson

Working...