University of Twente NOC Destroyed 483
JanJoost writes "Around 08.00 CET today the University of Twente Network Operations Center, which amongst other things hosts a SURFnet PoP as well as security.debian.org and non-us.debian.org, caught fire.
The UT, which hosted the HAL in august last year is completely unreachable and is not likely to come back up any time soon. The fire department has given up every hope on protecting the server area and is now trying to protect the surrounding buildings.
More information can be found at the Telegraaf,
Planet Internet and Twentsche Courant.
Pictures can be found here and here.
It's a shame to see a great infrastructure go down in flames like this."
Vunerability (Score:5, Interesting)
This shows the vunerability of putting all computers in one building. To have a safe network one needs to spread (duplicate) the information over several computer at several locations. How far apart these locations has to be is depending on how important you data is.
It is a shame that a building hosting so many good initiatives should be the one to go, but as always: there is no excuse for not have a backup. By that I don't only mean that tape that always seems to go missing when needed, but multiple sites (or at least buildings) that provide redundancy.
So much for server areas never burning down (Score:5, Interesting)
What a shame.
last syslog messages from SURFnet routers (Score:5, Interesting)
few seconds later on the local side:
lo0.cr2.amsterdam2.surf.net 1146: Nov 20 07:20:56.458 UTC: %CLNS-5-ADJCHANGE: ISIS: Adjacency to ar5.enschede1 (POS2/0) Down, interface deleted(non-iih)
Halon dumps? (Score:4, Interesting)
I hope Debian practices good management principles by having offsite backup.
Re:now the engineers come out... (Score:5, Interesting)
These are for small fires that can be contained within a data centre, ie. a computer catching fire or emitting smoke. Really good systems are very localised (racks or cage specific). A big fire just isn't going to be stopped by such a system.
I haven't read too much into this particular incident (ie. not at all), but my initial thought was that something more serious must have happened (well, duh!), perhaps a fire outside of the main suppression system (outside of the raised floor area?). Or perhaps the paint on the walls/carpet wasn't fire resistant and just took hold very quickly.
Or a large initial fire (gas leak?) that just didn't die down when the supression system kicked in. Maybe the type of fire (again, gas? oil?) didn't die down because Data Center supression systems presumably focus on electrical fires.
A well, just my 2p's worth.
Re:now the engineers come out... (Score:5, Interesting)
Personally, I'm not too sure of this route. I can understand it, from an environmental and human-safety perspective (the gasses eat the ozone layer, and you *really* don't want to inhale stuff that ties up oxygen at those rates), but if it ever happens here, it's going to be an awfully hurried mess to get everything back up and running in a reasonable timespan, even if the fire is only a small one that doesn't destroy the building.
Warning Sounds (Score:1, Interesting)
In that environment an old fashion siren would have been a much better solution.
Re:A good reminder.. (Score:3, Interesting)
Halon systems aren't illegal, but.... (Score:4, Interesting)
However, I wouldn't want to have to fill out all the paperwork involved with a discharge! We had an accidental discharge (a leak, I believe) and they decided it was enough impetus to remove the system.
I think they are using CO2 now. The advantage of Halon is that you can breathe quite comfortably in an atmosphere that will not sustain fire. CO2 works just about as effectively but will not sustain life.
OTOH, more recent studies have shown that just because you can still breath in a Halon infiltrated environment doesn't mean that there are no health effects!
I expect there are more CO2 systems going in now, with lots of alarms to make sure people get out before the atmosphere gets unlivable.
Michael
Re:Vunerability (Score:2, Interesting)
Re:now the engineers come out... (Score:3, Interesting)
Well, they came in to do the test, everything but actually spraying water, and managed to flip the breaker. It was interesting to note, that only a few of the machines actually powered off. Some are on smaller single machine UPSs, so those stayed running. The most expensive piece of equipment in there, a Nortel Telephony Switch which handles 6 PRI lines (almost 150 ISDN lines) is on battery backup, and kept running.
All in all very little powered down. In the even of a real fire, we would have had sparks flying everywhere. What did management do when I mentioned this to them? That's right, nothing.
Re:Halon systems aren't illegal, but.... (Score:2, Interesting)
Strangely enough, there are no servers in there, just hubs, routers and comms gear.
Re:now the engineers come out... (Score:3, Interesting)
When I worked for DEC they actually had sprinkler systems in their data centers (I couldn't verify first hand that every data center had sprinkler, but I was told so).
The reasoning was, that even if they was one big, bad, evil, wicked water damage in a data center, that the company considered this to be better then one killed employee by gas extuinguishing systems.
That was in the early 90ties
Re:Vunerability (Score:1, Interesting)
The first kept their AS400 farm running the majority of their critical applications in a building near their chemical plant (*ding*). Normally this would only be considered a minor risk, however they were in a low-lying area of Louisiana. We were actually looking UP at the Mississippi River (well, we would have without the dykes). They never seemed to grasp why we were so appalled that we could see a WATERLINE from a previous flood about 6 feet above groundlevel!
Another had a backup site and stored their tape backups at a site that was DIRECTLY in the path of one of the busiest runways at an extremely busy airport. We actually had to pause conversations when anything jet-powered was taking off. They didn't consider that to be a risk either.
Re:A good reminder.. (Score:3, Interesting)
Details on the recovery:
http://www.baselinemag.com/article2/0,
Some information on the scale of what eSpeed handles:
http://wstonline.com/story/mag/WST20011
Even Cantor Fitzgerald mentions it in their Q3/01 results summary:
http://www.cantor.com/articles/article1
Re:In other news (Score:2, Interesting)
In total, about 20.000 rooms for individual students in the Netherlands are responsible for over 1 Gbps of traffic. This is non-stop (i.e. during the evening it is closer to 2 Gbps), and over 90+ % is outgoing traffic from 'unknown ports'.
I guess forcing the RIAA and the MPAA to change their business models the hard way counts as innovation nowadays
Re:last syslog messages from SURFnet routers (Score:1, Interesting)
FYI- 61 degrees C = 141 degrees F.
Sprinkler systems kick in around 150 to 160 degrees; temperature at the top of the room was probably higher.
My bet is that router died(along with everything else) when the sprinkler system came on and power was cut to the room(most fire suppression systems in datacenters, water or gas, cut power to the entire room.)
It dated back to the earliest systems- "big red switches" usually have what are called "Molly guards", which are supposedly named for the daughter of a Univ. of Calif. programmer whose daughter hit the button 3 times in one day.
The same jargon dictionary mentions that in the 'early dates', said BRS fired a shell into the mains bar of the mainframe to sever it.
That said, 60 degrees is about the max operating temperature for most electronics. So take your pick
Re:So much for server areas never burning down (Score:3, Interesting)