University of Twente NOC Destroyed 483
JanJoost writes "Around 08.00 CET today the University of Twente Network Operations Center, which amongst other things hosts a SURFnet PoP as well as security.debian.org and non-us.debian.org, caught fire.
The UT, which hosted the HAL in august last year is completely unreachable and is not likely to come back up any time soon. The fire department has given up every hope on protecting the server area and is now trying to protect the surrounding buildings.
More information can be found at the Telegraaf,
Planet Internet and Twentsche Courant.
Pictures can be found here and here.
It's a shame to see a great infrastructure go down in flames like this."
More info (Score:5, Informative)
Photo's (Score:5, Informative)
Re:why does this matter? (Score:4, Informative)
"which amongst other things hosts a SURFnet PoP as well as security.debian.org and non-us.debian.org "
security.debian.org and non-us.debian.org are GONE (well, burning, hopefully data can be recovered). Yes there's mirrors, but it still sucks.
Debian Security mirror (Score:5, Informative)
You can use debian.essentkabel.com to download the latest security updates (in case you haven't already). Please note this is NOT an official mirror.
More Pictures from the webcam (Score:5, Informative)
Also, here's what seems to be the only close up I could find of the fire. pics [traserv.com]
Halon is probably illegal today (Score:4, Informative)
Re:More info (Score:5, Informative)
He's now probably busy setting up klecker.debian.org as the next security.debian.org host. Don't get yourself trojaned, please people, don't panic and just wait for the official Debian announcement that everything has been fixed again. Or play around with inofficial mirrors like [tu-clausthal.de] these [xs4all.nl], and there are more. But I feel a bit stupid myself, because - unlike Wichert - I have done nothing myself except forwarding the news and act like a karma whore.
Update from SARA (SURFnet NOC) (Score:5, Informative)
Re:Priorities (Score:5, Informative)
The university doesn't insure itself agains this kind of disaster because they are able to set aside enough money to cover the cost themselves and don't have to pay the insurer's profit.
Now how they are going to deal with this and their financial troubles (they're almost broke) is another issue...
LFS mirror affected too (Score:5, Informative)
Last news is that HP (Who supplied most of the UT backbone equipment) is on its way with emergency equipment to have things up and running somewhere tomorrow.
Re:Halon dumps? -- not if the whole building is in (Score:5, Informative)
Now, I would *love* to see a halon system capable of stopping that...
Owh, and the fire seems to be under control by now, as evidenced on http://webcam.traserv.com/
(which you can contrast to http://www-infstud.sci.kun.nl/~arthurvl/ispy.jpg (taken at about 09:40 CET this morning)).
Re:Halon is probably illegal today (Score:2, Informative)
security updates mirrored worldwide. (Score:5, Informative)
Actually, Debian security updates are usually also added to the proposed-updates section, which means that they are available on nearly all Debian mirrors worldwide.
Something like /etc/apt/sources.list should work well.
deb ftp://ftp.XX.debian.org/pub/debian/ proposed-updates main contrib non-free
deb ftp://ftp.XX.debian.org/pub/debian-non-US/ proposed-updates/non-US main contrib non-free
(replace XX with your ISO country code) in your
You can find mirrors on Debian [debian.org]'s website.
Translation of the Twentsche Courant Tubantia (Score:3, Informative)
The burning building contains the IT department and a part of the faculty of Business Administration [Closest I could come up with: Bestuurskunde]. The building has three storeys.
The university fears the loss of its network facilities and is trying to save the main computer. According to a spokesman this network is amongst the fastest in Europe. Most classes are expected to resume as normal today. [According to other sources, this is an exam week, meaning few classes anyway. This is also a reason that few students were around so chances were that this also reduced potential casualties]
The fire department is fighting the fire with 25 firefighters and expects to need the entire day to extinguish the fire. No dangerous materials have been released by the fire at this point. At the moment nothing is known about the cause of this fire.
Re:No Halon? (Score:3, Informative)
Moreover, Halon system are no longer installed (globally) since 1987 (Montreal Protocol) due to its CFC damaging effect. Most systems already installed are replaced by FM-200. Water, Argon, FE-13, Inergen and a few others are all possible replacement.
Just a few translations (Score:5, Informative)
There is a complete backup of the data, so no ireplaceable data-loss. Still, several tens of millions of euros worth of damage.
An emergency network will be constructed, that will take several days. Please do not post a link to the new network on slashdot, the emergency solution will not have a very high capacity...
The exams for today were mostly cancelled, internet and e-mail for students and teacher will be down untill the emergency network is in place.
Kind regards,
Maarten Sneep from Amsterdam
PS. This is the same town that a few years ago was struck by a fireworks disaster. See www.volkskrant.nl [volkskrant.nl] for pictures.
Information from the University site (Score:2, Informative)
The info was posted at 11 am NL time, 3 hours after the start of the fire.
The fire was still burning but under control. There were no (real live) victims. According to measurements done by the fire department no dangerous chemicals were released during the fire. The building was recently renovated and all asbestos was removed.
The part of the building where all the computer infrastructure is housed is a total loss. There are no test setups or labs in the building. There are backups, but the information on the backups is from last weekend. UT is working on restoring the network, but this work can take days.
The part of the building that burned down contained project rooms for students and some class rooms. It also contained some rooms of the applied communications science department.
Sjaak.
Re:now the engineers come out... (Score:5, Informative)
I swear... (Score:1, Informative)
slashdot out of boredom because there seemed
to be a network problem.
[screenshot]
http://flaez.ch/scratch/twente_bu
Re:now the engineers come out... (Score:2, Informative)
This hasn't been a real problem since the original Halon systems. There is generally a long protracted countdown (15-20secs is a long time when talking about fires.) even on the newer systems such as inergen. Potentially dangerous gases have breathing apparatus in the protected area just in case. (ie: someone is electrocuted, fire, can't move person, grab breathing mask, put on their head (or grab two and move them) and leave room.)
Though these systems are very expensive. We have the largest data center (by floorspace) in Queensland (multi-storey). When someone "mistook" the "release without interrupt" button for the exit door release, it cost AUS$10-12 thousand to replace the gas(inergen). Of course now it's _much_ harder to make this mistake. (Involving a hammer and a glass cover).
But then again, the person in question was known to play indoor cricket between the mainframes. I'm sure that a fast moving ball had absolutely nothing to do with tripping an inset switch on the wall.
Debian is working on recovering (Score:5, Informative)
At around 8 this morning (local time) a fire started in the computing facilities of Twente University. This affects Debian, since one of our servers (satie) is hosted there. At this moment it seems very likely that the machine can not be recovered from the fire.
The following services are currently down as a result of this:
security.debian.org
non-us.debian.org
nm.debian.org
qa.debian.org
We are working to restoring these services on another machine and hope to have things in mostly working order by tomorrow. Security advisories are still available at http://www.debian.org/security/
Wichert.
Re:Halon systems aren't illegal, but.... (Score:4, Informative)
Both gasses extinguish fire by excluding oxygen... which isn't good for people either.
The difference is that one cannot comfortably breath CO2, an involuntary physiological reaction makes it impossible. (next time you open a bottle of Coke, try sniffing the little cloud of CO2 that forms in the top of the bottle)
Halon, otoh, is perfectly comofortable to breathe in and out, but will provide no oxygen. This is similar to breathing helium (recommended as the nicest way to commit suicide, after N20) in that one would be quite comfortable right up until one passed out from hypoxia. Death follows soon after.
Also, i've read in boating magazines that undersized Halon systems used in engine compartments can be dangerous. Apparently, if the diesel engine is running when the system fires and there isn't enough Halon to kill the engine, the burned Halon/air/diesel mixture produces some really nasty toxic gasses.
Anyhow.. enough rambling!
Re:Vunerability (Score:3, Informative)
1) Sending tapes off-site too early -- TSM allows you to have multiple "storage pools" within one library. Basically, you backup your data into one storage pool (set of tapes) and then duplicate it to the off-site copies. Then the DRM (Disaster Recovery Manager) scripts automatically eject the tapes from the library for packaging to send off-site. You don't ship off your on-site copies, so when you need to do a restore, it's always here.
2) Partly full tapes -- TSM has a concept of "migrations" where it moves data between tapes to better utilize them. The internal DB automatically knows where the data is, so you don't have to worry about which tape has which days' backups -- it doesn't matter.
3) Full backupset -- TSM 4.2 introduced "portable backup sets" -- basically, you backup your data into the TSM server as normal, then "generate" a backupset -- takes all the files for a particular node and writes them to a series of tapes which you then eject and store off-site. When you need to recover from those tapes, the server component isn't needed -- locally attached tape on the server and the client piece of TSM is all you need --- speeds up recovery by a few hours!
TSM's a great product. Does lots and lots of great things that other vendors are only now trying to figure out.
Sure, it's pricy, but how much is your data worth?
Re:In other news (Score:4, Informative)
Re:last syslog messages from SURFnet routers (Score:2, Informative)
PING lo0.ar5.enschede1.surf.net (145.145.255.14): 56 data bytes
--- lo0.ar5.enschede1.surf.net ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
Breathe Quite Comfortably? (Score:5, Informative)
You've obviously never been in a Halon dump. Either that or you consider burning in your lungs to be "quite comfortable". Not to mention that if you're standing under one of the discharge nozzles at a dump you can get a nasty case of frostbite.
I've personally been through two 1211 dumps and had to enter a computer room and drag staff out after an FM200 dump. It takes about two days to completely stop coughing.
Re:I predict... (Score:3, Informative)
It's not as if the debian project didn't have the domain properly setup with 3 diffrent nameservers in 3 completely different locations.
They may even have a working mirror...
Re:now the engineers come out... (Score:2, Informative)
Even with scheduling regular business continuity plan tests & sending people offsite for a week or so, these plans are cheaper than maintainig a 2nd data center.
Re:last syslog messages from SURFnet routers (Score:4, Informative)
Re:Debian Security mirror (Score:3, Informative)
Debian haven't released a security advisory since yesterday, so it's deeply unlikely you'll need an update before they get a new security.debian.org online tomorrow.
translation.. (Score:3, Informative)
"University network down from fire"
"ENSCHEDE - A fire in the computing center from the university of Twente (UT) in Enschede has caused a few 10's of millions of euros damage on Wednesday.
The fire, which broke out about 8:00am, has disabled the universities network. There were no injuries. The fire department has cleared out several buildings in the nearby facinity. The IT departments of the UT and a part of the business studies department were housed in the building where the fire broke out. The building was three stories high.
The fire department fought the fire with several dozen people and estimated that they would need the whole day to get the fire under control. There are, until now, no dangerous substances released. The cause of the outbreak is not yet known.
According to Van Vught, a backup of the network's data has been made. "All data have been safely stored". Because of exams, there were no classes at the time. There were also few students on the campus on wednesday. Exams on wednesday have been cancelled.
Staff at the UT want to set up a temporary network, which can take several days. Thousands of staff and students can not make use of the campus network until a temporary network has been installed.
One on-looking business studies student is shocked (?): "incredibly sad. It means that I can't do my assignments. Doubtless, a temporary building will need to be set-up to help the situation". The Saxion Polytechn in Enschede has made room available.
The web-site from the Enschede city council is not available because of the fire. The council used the servers from the UT. The Saxion Polytech is also without internet access for the same reason."
Okay, this translation is a wee-bit too literal, but it's understandable.
Michael.
Re:From "The Register" (Score:2, Informative)
It's a bitch fast connection, btu as you can see, not that stable.
Re:Vunerability (Score:2, Informative)
more information (Score:2, Informative)
frontpage.fok.nl [frontpage.fok.nl] (a dutch news site) reports the following, well at least not the article itself but in a reaction to it: "The D-wing was recently renovated to make it more 'fireproof', so there are no harmful gasses and such.
Every morning a backup is made from all the data. Unfortunetely, before this backup could be taken, it was already destroyed, which means data from yesterday morning is lost.
Tonight (CET of course) there will probably be an emergency network running at 10kb/s and the server will probably be online in 3 days. Until then Teletop (some kind of information system for communication between tutors and students) and the utwente email will be down. Something about the building: The TWRC building some kind of monument which included that the outside of the building cannot be changed. This means the renovation plans have to be the same as the orignal building."
At least I hope they got rid of the air-problem, because it really stinked in that building anyways. Too bad everything to lost. They were just about to move all the servers to another location.
I was just printing my report which I had to hand in at 12:00 CET when I heard it. I worked the last 24 hours to complete the damn thing and then this.
Lucky for me I don't have internet from the university, but it still sucks bigtime.
Re:Class IV construction (Score:3, Informative)
Do NOT stay at ground level with HALON! (Score:4, Informative)
Halon (or at least most of the Halons used for fire suppression; not sure about all of them) is non-toxic, though it'll get you a bit high, like nitrous oxide or most solvents, so being stuck in the stuff won't injure you quickly (except from flying objects that were blown around by the gas pressure.) However, it's no substitute for oxygen, and you'll probably be wanting some oxygen real soon now. If you can remember not to breathe in the stuff, try not to breathe deeply, because there's more oxygen left in your lungs than the stuff you'll be breathing in, and unfortunately, while your body can generally figure out not to breathe in water or hot smoke, it's not as good at realizing that near-room-temperature inert gases aren't very useful. Mostly, don't worry about it - find a safe door to run for and run for it, and do some deep breathing once you're outside, and try to close the door behind you.
Things are being restored slowly (Score:3, Informative)
Debian is restoring the lost services on klecker. At this moment qa.debian.org is up and running and the non-US and security archives are available as well, although their backend systems have not been restored yet.
Valuable lessons have been learned though: it is very useful to have machines on standby where you can switches services to when needed. Having backups of important data is also really useful (and we could have done a bit better at that. UTwente apparently has good off-site backups of its own data though). And having good insurance is also definitely useful.