University of Twente NOC Destroyed 483
JanJoost writes "Around 08.00 CET today the University of Twente Network Operations Center, which amongst other things hosts a SURFnet PoP as well as security.debian.org and non-us.debian.org, caught fire.
The UT, which hosted the HAL in august last year is completely unreachable and is not likely to come back up any time soon. The fire department has given up every hope on protecting the server area and is now trying to protect the surrounding buildings.
More information can be found at the Telegraaf,
Planet Internet and Twentsche Courant.
Pictures can be found here and here.
It's a shame to see a great infrastructure go down in flames like this."
Well Damn... (Score:3, Insightful)
After that, I wish them luck getting back online.
now the engineers come out... (Score:5, Insightful)
A good reminder.. (Score:5, Insightful)
Priorities (Score:5, Insightful)
If not, was anyone hurt?
If not, do they have insurance?
If they do... well, I'm sure someone just lost their masterpiece pr0n directory, but otherwise, things like this happen. (ask Hemos) You have to make it through such things. In this case, it was a commercial (educational) building and no one is homeless, so it's less of a tragedy than usual. Let's hope that they rebuild with something better and newer.
That said, I get the feeling that those plumes of smoke really are millions of dollars floating away in the wind...
Re:now the engineers come out... (Score:3, Insightful)
I had no less than 1000 companies call asking how to handle licensing and DRA's.
I'd be surprised if they didn't have one, it's pretty much standard practice comercially....
Yo Grark
- Canadian Bred with American Buttering
Re:A good reminder.. (Score:5, Insightful)
Oh Dear.... (Score:2, Insightful)
Is there anything that could cause this naturally? I mean, judging from the pictures it looked fairly large and out of control...I hope no one caused this purposefully. I've never had to deal with a catastrophe like this, luckily...I send my best wishes of luck and hope to those involved and pray no one got hurt. =(
Re:So much for server areas never burning down (Score:2, Insightful)
I.E. "See, if they would have had a fire suppression system, this would have never happened."
PGP Keyserver root was hosted by SURFNet (Score:5, Insightful)
Re:So much for server areas never burning down (Score:5, Insightful)
BACKUP!!! (Score:5, Insightful)
BACKUPS BACKUPS BACKUPS Off-site! I've had enough of people who are talking about RAID-5 because 5TB tape drive arrays are too slow. Always keep your BACKUPS!
About as good as it gets with only two sites... (Score:5, Insightful)
The lesson should be: Primary back-up is a very good start, but secondary/tertiary back-up is the thing if it's that critical.
Kjella
WARNING: Above post a GOATSE! (Score:1, Insightful)
If you want to read about the fire, go here [theregister.co.uk]. Apparently UT was a major node for KazAA, and'a primo source for warez and pr0n.
Re:Halon dumps? (Score:3, Insightful)
Most scenario would only require a less than 8% of concentration to take out the fire. Under 10% and you can still breath.
Problem of Halon is when over 900 degree C, it breaks down into hydrogen fluoride, hydrogen bromide and bromine - stuff that are toxic. So, run!
Re:now the engineers come out... (Score:1, Insightful)
And six months later, 80% of them scrapped or seriously cut their disaster recovery plans because they were too expensive.
Most people in chare of creating disaster recovery plans were told something like 'We need to be able to be back in busness 100% within 24 hours, but we can't spend any money (or can only spend a woefully indadaquate amount of money) on new servers, redundant network lines, improved backups, offsite tape storage, etc.'
Re:Disaster Recovery plans (Score:5, Insightful)
Do full backups weekly, store copies offsite. Incremental backups daily, copies offsite also. If you can afford it (or can't afford any downtime), have emergency backup hardware (enough for minimal operations) in an offsite storage facility. Old hardware that would otherwise be thrown out is good for this (remember, it's for an emergency). Have a supplier who can get replacement hardware to you in a hurry (so you can get off of those old 90 MHz Pentium servers).
The most vital part of the plan, after backups, is good insurance. If the building burns to the ground Monday morning, you want to be able to call the insurer Monday Noon, and have the check in hand Tuesday morning at the latest.
These recommendations do not cover disasters such as 767s flying into the building and killing all the sysops. Earthquakes dropping the building on the same. Etc. The people are the most important part of any company and, if too many of them are lost at once, the company probably is lost too.
Unless you have really good (and expensive)insurance which can provide enough funds for you to hire new people, get them trained, and keep the company solvent while you do so.
Re:Vunerability (Score:4, Insightful)
It is a shame that a building hosting so many good initiatives should be the one to go, but as always: there is no excuse for not have a backup.
Uhhh, yes there is... I suspect you either know nothing about IT or are fresh out of college. DRP (disaster recovery planning) factors in things such as criticality of data, cost, and acceptable downtime. A university payroll system may need to be back up within 12 hours of a major incident, so in addition to tape backups you might have a failover site. Contrary to your simplistic post, even the richest corporations rarely have failover sites of their own. They simply contract out to a DRP vendor who have these types of machines lying dormant in a glass room, waiting to cut over. On the other hand a university FTP site is probably classified as low risk, low impact. So you would rely on off-site backup tapes and perhaps only restore when you've arranged for an alternate site and taken delivery of new servers. You don't pay millions of dollars to have two glass rooms just so you can have uninterrupted FTP service...
Re:Vunerability (Score:5, Insightful)
I just hope they're well insured....poor colleagues...
On the upside: they may get a squeaky-clean start when this blows over
Re:About as good as it gets with only two sites... (Score:1, Insightful)
It apparently can happen, though.
In any case, the chances of two buildings standing side by side being taken down is much greater than two random buildings on either coast.
Keep in mind - there's enough nuclear weaponry out there to make the United States into a glass desert. What if that goes off? I hope you had secondary+ backups in Canada and Mexico!
(Frankly, I wouldn't be concerned with backups at that point.)
The point is that backup is simply another game of risk.
Re:About as good as it gets with only two sites... (Score:3, Insightful)
Really, not having your backups in close proximity to the data center IS good policy.
Re:now the engineers come out... (Score:2, Insightful)
Even the obvious like "lie low as smoke rises and there is more oxygen at ground level" and to "try and cover your mouth & nose". At least, this is what I learned from that incident, sadly after the fact.
Re:Vunerability (Score:2, Insightful)
Re:About as good as it gets with only two sites... (Score:3, Insightful)
I'm sure that there's been lots of study of this, but I suspect that a good distance is 1-2 hours drive. Far enough to avoid most of the impact from things like a big chunk of the city being shut down for weeks, close enough to get to (with some inconvenience) if necessary.
In the Chicago area it might be something like downtown Chicago and Schaumburg, Naperville, Aurora or even Rockford (at the 2-hour mark).
You mean HiveCache (formerly MojoNation)? (Score:3, Insightful)
Re:now the engineers come out... (Score:2, Insightful)
So, when does the pledge drive start? (Score:2, Insightful)
Re:In other news (Score:2, Insightful)
Umm, so? Outgoing traffic is almost invariably from a random port > 1024. That's how TCP connections are generally done.
Re:About as good as it gets with only two sites... (Score:3, Insightful)
The only [gag] good [retch] reasons to have the backup in the other tower are spelled "lazy" and "stupid".
They Needed Low-Tech Fire Protection (Score:4, Insightful)
Critical stuff is spread between the file rooms, with metal conduit, etc. protecting the few small holes they added for wiring.
Steel & reinforced concrete aren't quite obsolete.
Re:Halon systems aren't illegal, but.... (Score:3, Insightful)
Why does it need to provide oxygen? The air provides oxygen. How come you're not worried that air is about 80% nitrogen, which also provides no oxygen? See, humans don't need to breathe 100% oxygen... we do fine with much less. The advantage of Halon over CO2 is that it does not extinguish a fire by displacing oxygen. It will put out a fire [tpub.com] at concentrations of about 5%, leaving plenty of oxygen to breathe.
the burned Halon/air/diesel mixture produces some really nasty toxic gasses.
And a fire doesn't? If you have an undersized system installed, you're gonna have problems in a fire anyways. With a proper system, the small quantity of toxic gasses produced by the Halon decomposition before the fire is extinguished (which is a fraction of a second--Halon systems have been used for explosion suppression) is much preferable to the large quantity of toxic gasses and heat produced by a fire.
Re:now the engineers come out... (Score:2, Insightful)
I also asked them what happens if a plane crashes on takeoff from Dulles airport (only a few miles away and where the plane that hit the Pentagon took off from) and hits the datacenter, and they said they had the flight path altered when they built the datacenter to minimize the chance of such a thing. They didn't admit to it, but I bet that that building is hardened, too.
Re:now the engineers come out... (Score:5, Insightful)
But if you breathe that air out and breathe oxygen-less air in, you will pass out very fast. Most people have the misconception that you can do without oxygen for a minute or two before dying. That's not true - your body needs to consume fresh oxygen at a continuing rate just to function at all, it's just that your lungs can HOLD a small supply of oxygen to supply this need for a minute or two. Get rid of that oxygen by breathing it out and replacing it with oxygenless air, and you're going to pass out in just a few seconds, and be dead shortly thereafter.
And the worst part is you won't FEEL like anything is wrong. Your body is unable to measure the level of oxygen in your lungs. Instead your body senses the level of carbon dioxide in your lungs. As the by-product of normal breathing, when carbon dioxide has built up enough, that indicates you've converted a lot of oxygen and it's time for another breath. This is what triggers the automatic involuntary breathing that takes over when you stop thinking about it. This is also what triggers the panic feeling that you get when you know you need air. Your lymph nodes detect too much carbon dioxide and start sending the panic signal to your mind. What this all means is that if your body isn't exchanging oxygen for carbon dioxide, your body doesn't even realize it's asphixiating. If there's no oxygen in your lungs to start with, then there won't be any carbon dioxide building up in the lungs, and you will feel no sensation of needing a breath at all. You'll feel just fine for a few seconds and then *poof* you're gone as the blood going to your brain runs out of oxygen and your brain activity just plain stops.
So if you're ever in a halon gas system when it goes off - DO NOT BREATHE. Just hold whatever breath happens to already in your lungs and get out. The instinct is to hold your breath by first inhaling your lungs full and THEN holding it, but that's the worst thing you could do, as explained above. The tricky part is remembering to override that instinct.
Re:Nice Moderation (Score:2, Insightful)
After that, I wish them luck getting back online.
Someone explain how the FUCK that can be considered insightful?
Yet again we see the need for moderators to pass an IQ test.