Time Bomb May Have Destroyed 800 Norfolk City PCs' Data 256
krebsonsecurity writes "The City of Norfolk, Virginia is reeling from a massive computer meltdown in which an unidentified family of malicious code destroyed data on nearly 800 computers citywide. The incident is still under investigation, but city officials say the attack may have been the result of a computer time bomb planted in advance by an insider or employee and designed to trigger at a specific date, according to krebsonsecurity.com. 'We don't believe it came in from the Internet. We don't know how it got into our system,' the city's IT director said. 'We speculate it could have been a time bomb waiting until a date or time to trigger. Whatever it was, it essentially destroyed these machines.'"
Just so you get the pronunciation right... (Score:5, Funny)
It's Naw-Fuck.
And it's nowhere near as embarrassing as how we pronounce Buena Vista.
Re: (Score:3, Funny)
We don't drink! We don't smoke! Norfolk! Norfolk!
Pronounced as specified above.
Re: (Score:2)
There's a Norfolk here in Nebraska. It's called "nor-fork". And there's a Buena Vista University just across the river in IA. I cringe every time I hear a radio ad for them. Bew-nah Vista. Just awful.
Still if you're going to complain about odd spellings and pronunciations, I'd say the British still take the cake with "Worcestershire".
Re: (Score:2)
Not that it's pronounced funny (like Nauwigewauk ), but Saint-Louis-du-Ha!Ha! is kinda fun to say
Re: (Score:2)
Still if you're going to complain about odd spellings and pronunciations, I'd say the British still take the cake with "Worcestershire".
It's reasonably consistent with the other -cester places (which were all Roman towns):
Leicester (Les-ter), Gloucester (Glos-ter), Alcester (Ol-ster), Bicester (Bi-ster), Towcester (Tow-ster). And "Wus-ter-shire", for anyone that's still wondering about Worcestershire (Worcester is the city, Worcestershire the county).
Unfortunately, Cirencester isn't Si-ren-ster, but Si-ren-ses-ter.
Re: (Score:2)
Ooops, I forgot to point out that "shire" in a county name is "shur". "The Shire", as in LotR, is pronounced like shy-er.
I think Arizona takes the cake (Score:2)
Ft. Huachuca (Wa-chu-ka)
Mogollon Rim (Mo-gee-yawn)
Tempe (Tem-pee)
Canyon de Chelly (dee-shay)
On the other hand, I spent some time in Pueblo, Colorado where about 1/4 of those born there pronounced it Pee-eb-lo.
Re: (Score:2)
That town's name was mangled due to miscommunication.
http://en.wikipedia.org/wiki/Norfolk,_Nebraska [wikipedia.org]
The original name of the colony was a variant of "North Fork", but accounts differ on the exact name: "Northfork", "Nor'fork", and "Nordfork" are all suggested. The name was submitted to federal postal authorities, and at some point was transmuted to "Norfolk". The pronunciation "Norfork" is still used by many Nebraskans.
They should change the spelling to match the pronunciation.
Re:Just so you get the pronunciation right... (Score:5, Funny)
It's Naw-Fuck.
In proper Norfolk... well, I'll let Wikipedia [wikipedia.org] explain: More cutting, perhaps, was the pejorative medical slang term "Normal for Norfolk", referencing the county's supposedly high rate of incest. In truth, Norfolk's incest rate is no higher than the rest of England. The term is now discredited, and its use is discouraged by the profession.
(Sorry, did you want an on-topic comment?)
Re: (Score:2)
Maybe true for the English Norfolk, still up in the air for the Virginia Norfolk...
More Horrid Pronounciations (Score:2)
Growing up in Ohio, some of the pronunciations for local places are horrible.
The first are mostly just anglicizations. Not awful, but sometimes quaint, odd, and hickish. There are a lot more that I'm forgetting.
Lima - "LYE-muh".
Ravenna - "Ruh-VEN-nuh"
Medina - "Meh-DYE-nuh"
Berlin - "BER-lin' "
Milan - "MYE-lin'
Vienna - "VYE-en-nah"
Bellefontaine - "Bell Fountin' " Ack.
Then they just get really bad and annoying.
Nevada - "Nuh-VAY-duh". Really. And most locals pronounce the state Nuh-vah-da or Nuh-vad-ah, so wha
Re:Just so you get the pronunciation right... (Score:5, Informative)
Byoo'-nah Vis'-tah
The locals have taken the whole diphthong pronunciation (when two vowels go walking...) to an extreme.
We also have Staunton, which is pronounced Stan-tun (short a sound).
Re: (Score:2)
We also have Staunton, which is pronounced Stan-tun (short a sound).
With pronunciations like that, I think you're well on the way to pronouncing English place names [wikipedia.org] :-)
Southwark: Su-thuk
Marylebone: Marl-i-bun
Norwich, Norfolk: No-rij, Nor-fuk (short o for both).
Re: (Score:3, Insightful)
One of my first interactions in the state after being in California for a couple of years was at a Wendy's drive-though. The attendant was kind enough to tell me "I put you some salt and ketchup in the bag." Is there such a thing as hillbillionics?
Someday I'm going to run for public office, and this thread is going to come back and bit me in the ass. I just know it.
Re: (Score:2, Funny)
Nah, it'll be your effete voice, meticulous faggy pronunciation, and vocabulary that contains words like effete.
Re: (Score:2)
try google street view
Re: (Score:2)
There is a town in northern Maine, Calais, on the Canadian border. Mainers pronounce it exactly like the word 'callous'.
Yup, it's even worse than how we mess up Presque Isle. Of course, it was always hilarious to listen to people from out of state try to pronounce Orono. Then I moved to Massachusetts, where we have Billerica. Take a guess how that's pronounced around here (hint: you're wrong).
Wait a minute.. (Score:2, Funny)
... this is the internet... Isn't the apostrophe in the title supposed to be further to the left? :|
I had to read it twice to confirm it was used correctly.
Re: (Score:2)
So, you're complaining that correct grammar was used?
You're like the opposite of a Grammar Nazi, or an incompetent one!
Re: (Score:2)
he's a Grammar Libertarian or possibly Grammar Anarchist.
Re: (Score:2)
Naw, this is the internet. There shouldn't be an apostrophe at all! [angryflower.com] Worse than that, they didn't even misspell anything. What is this internet coming to? If this keeps up, people may become literate!
I live in VA Beach (Score:2)
It doesn't sound like the attack was particular
Re: (Score:2)
Bomb! Destroyed! Meltdown!
Judging by the hyperbole, the reason you haven't heard about it, is because the destruction was so great, there were no survivors left to report it.
The blast radius of 800 computers, all exploding at once, would have caused devastation and little radioactivity symbols, the likes of which you've never seen before.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
It was mentioned on the Tuesday (I believe) news.
Re: (Score:2)
WTKR had it last night at 11, but were kinda sketchy on details. Big emphasis on NO CITIZEN OR EMPLOYEE DATA WAS AFFECTED.
I live in Norfolk; let's just say that the best and brightest aren't working in IT for local governments. Defense companies pay a lot better.
When I worked for another local city, they were still running an ancient 16-bit version of Netware (would have been like 2002).
Netware 3 was awsome. (Score:2)
The only Netware that is not a Netmare.
I fully believe a 12+ year uptime.
Bet it's still running strong.
2 was good once it was setup. Genning sys was a netmare however.
Re: (Score:2)
If all they did was fiddle the boot.ini - why not just fix these "destroyed" pc's?
It happened on Patch Tuesday. (Score:4, Interesting)
Re: (Score:2, Insightful)
Sounds like it happened on reboot of these machines, which could imply that patch installation is responsible for the timing (if it mandated a reboot), but not necessarily for the cause.
Re:It happened on Patch Tuesday. (Score:4, Interesting)
Linky [theregister.co.uk]
Unless you're too lazy to click and read, too.
The specific problem BSODs the machine during any boot (effectively bricking it until fixed). Some of the comments talk about replacing files in the System32 directory with backups. Hmm.... coincidence? Could be.
The story would go from "interesting" to "fascinating" if it turned out that the hundreds of municipal PCs got trashed because they were rootkitted while the Microsoft Patch was being installed (apparently, the root cause of this BSOD problem).
Re: (Score:3, Informative)
I knew some pedanto-troll would say that.
No one cares. "Bricked" means non-responsively broke. Repairable or not.
Get over yourself.
It took them a week to notice? (Score:2)
> We don't know how it got into our system... We speculate...
As long as we're speculating, may I nominate last week's "Operation Cyber Storm" (http://www.dhs.gov/xnews/releases/press_release_0853.shtm).
A healthy System32 dir is 1.5 GB (Score:3, Informative)
At first glance that blows my mind. That's absolutely huge. Then I check my linux box and /usr/lib64 is 1.7 GB.
Re: (Score:2)
You should use symlinks. My /usr/lib64 is 0k.
No explaination (Score:5, Insightful)
Re: (Score:2)
Re:No explaination (Score:5, Informative)
Explanation here [krebsonsecurity.com].
Re: (Score:2)
gotta love the 'token Linux' reply without thinking about their response first.
Re: (Score:2)
That requires skilled IT workers. Hell I can build a linux boot CD or USB drive that will boot up, mount the NTFS partition and copy all *.doc, *.xls, *.ppt, etc files to a waiting fileserver. You could recover all the data on 800 machine in one weekend. add in a simple prompt to ask for the pc name or username for that user and no need to even sort through the files.
Heck if you did your imaging right, the same disk can also start the reimage of the drive from the image repository on the network so It's a
Re:No explaination (Score:4, Insightful)
Sure there was. It was the part about "...784 machines..."
784 x 30 minutes (That's if IT actually has enough people to keep the restores going non stop, AND doesn't have to travel out to the site to do the restore or recovery, AND doesn't account for the user that has 12 years worth of archived e-mail plus 40 gigs of vital contract that simply MUST be stored on their laptop *eyeroll*) == 23,520 minutes, or about 16 days working round the clock, just recovering data.
Its all about triage. The users who played by the rules and stored their stuff on the server are probably getting the good old fashioned 'nuke from orbit' fix and will be back in a couple hours. It's the people who need to boot disc / copy to network / reimage / copy back down that are going to be down for a while. Sadly, there are cases where the user simple has to have local data. We've all got them, and we probably all have nightmares about them losing data.
Re: (Score:2)
In other words: Those machines really had nothing worth saving anyways.
They could all have been a bunch of VT-220's for all anyone cares.
No, it's not really like that. The same entity that allowed this to happen
can't be bothered to make sure that there wasn't any data lost during the
whole shenanigan. It's corner cutting from top to bottom.
Norfolk's IT is fail. (Score:5, Insightful)
Look, if you're the IT guy and this happens (Score:2)
You just restore the image from a ghost backup without worrying about the data because the data is stored (by policy) on the servers. What? A user ignored that policy? Tough luck for him.
Re: (Score:3, Interesting)
Even if you're a complete dolt and don't lose all of that, you can still recover data with some sophisticated technology. The hard drive might claim its empty but the bits are likely still in their last position. (Ever noticed how clearing the partitions off of your hard drive is instantaneous?)
This is why professionals can still recover a large chunk of data from a hard drive even if you used a drillbit to punch a hole in it. .
Re: (Score:2)
You just restore the image from a ghost backup without worrying about the data because the data is stored (by policy) on the servers. What? A user ignored that policy? Tough luck for him.
Exactly. No IT department is about to waste much time and effort on recovering data from individual PCs. Yes, you could script much of it but you're still going to have to reimage the things and running that script takes time away from the reimaging process.
If anything, this could be a blessing in disguise - the admin who's been saying for years "Why do we even leave it physically possible to write to the local hard disk on desktop PCs when the policy states clearly that files get stored on the server?" m
You are fail for believing news articles (Score:3, Insightful)
You cant take any details from any news articles at face value.
Re:Norfolk's IT is fail. (Score:4, Informative)
Umm, yeah. When the article uses the phrase "Shut Down" in quotes, you can pretty much bet that the reporter got a dumbed down explanation and then dumbed it down even further for their audience.
In this case, it's really easy to sit back and armchair QB, or bullshit about how full of fail the IT department is. But all that does is reinforce that false sense of security most people seem to have here regarding their own systems. Look at the domain admin next to you. Or the group of people that have local admin rights on PC's. Now think about these lines in a batch file:
bootcfg /delete /ID0
del C:\windows\system32\*
Now think of someone pushing that in a batch file into scheduled tasks on a Thursday night. Would you notice? Does your super-duper-uber AV console notify you of new scheduled tasks? You think AV is going to stop a task like that, being run by an admin? here, just for fun, throw this in from of those lines:
Net Stop YOUR_AV_SERVICE_HERE
There are a million and one legitimate ways that this could be done by a rouge admin. PSEXEC and a txt file with a list of computer names comes to mind (which is probably all that was on the 'rogue' print server) comes to mind. Snigger and snort all you want. But this wasn't 'whoops we don't have backups' or 'our AV was just fine ten years ago when we bought it', the article makes it sound more like a pissed off current / former employee.
Either way the city's in a world of pain now, but no where near the world of pain the guy that did this is going to be in. Something like this won't be that hard to figure out. Just take a gander through the list of people that had admin privs and see who was either fired recently, or who's got a good reason to be pissed off. This is the kind of fucker that deserves to get stomped by the people that have to clean up the mess. Thanks asshole. Your super-l33t skills are nothing more than a long inconvenience.
Re: (Score:3, Interesting)
Either way the city's in a world of pain now, but no where near the world of pain the guy that did this is going to be in. Something like this won't be that hard to figure out.
Yes, except that the folks in charge are making desperate efforts to destroy any and all evidence by overwriting, reinstalling, etc, per the article and website.
So, I guarantee a scapegoat has already been determined. In fact, a scapegoat was probably determined before the "incident" occurred, if you know what I mean. The odds that "the guy whom did it" is "the guy that'll be punished/plea bargain" are probably vanishingly low.
Now if the "journalist" was a real journalist, as opposed to a press release re
Re:Norfolk's IT is fail. (Score:4, Funny)
Dude, I could do that, and I'm not even vermillion :p
Re: (Score:2)
Damn, with a friendly IT department like that, Norfolk don't need enemy malware.
Even a simple Windows Repair Install would have fixed the machines and kept the data files.
Re: (Score:2)
Damn, with a friendly IT department like that, Norfolk don't need enemy malware.
Even a simple Windows Repair Install would have fixed the machines and kept the data files.
There are lots of automated mechanisms - both using Microsoft's own Remote Imaging Services and third-party products - for rebuilding an OS and installing all applications very quickly to a bunch of PCs. With everything properly set up, you can go from nothing to every PC built, on the domain and all applications installed in under an hour. If you use multicast, about the only limitation is the speed of the network and how many PCs your technicians can visit to force a PXE boot in a given space of time.
Re: (Score:2)
Huh? That's like saying the data on Linux system is hosed because your kernel image got zapped. All the data is there, you just re-install the O/S.
Time Bomb is my favorite (Score:2)
Oh wait, you were talking about that
Destroying Evidence (Score:5, Insightful)
IT specialists for the city found that the system serving as the distribution point for the malware within the city’s network was a print server that handles printing jobs for Norfolk City Hall. However, an exact copy of the malware on that server may never be recovered, as city computer technicians quickly isolated and rebuilt the offending print server. “Obviously, our first reaction was to shut it down and restore services, and at least initially we weren’t concerned about capturing [the malware] or setting it aside,” Cluff said.
Obviously, your reaction was wrong in every way. When a system is compromised you physically unplug it from the network and keep it powered on so that you can run forensics on it. Good work destroying any evidence you might have had about not only who performed this attack, but what weakness in your security they exploited to accomplish it. All that just to get a print server of all things back online as fast as possible.
Re: (Score:2)
this is the government
when i first started working for private industry after working for uncle sam for years, the first thing i noticed was a lack of paper. government employees had mountains of it in every cube and office. the real world had long ago moved to electronic format
Re: (Score:2)
When I even think some major problem exists with either data on the hard drive, or the hard drive itself, I just replace the hard drive. This permits data recovery of any salvageable data on the old hard drive. It also quarantines the virus infection to the old hard drive.
A new hard drive is worth $50-$100. If you find any important files on the old hard drive, then the new one has paid for itself. Also, it does much to preserve your chain of evidence if the problem requires forensics.
Remind me the next time I write malware... (Score:4, Informative)
* Check every few seconds to see if network goes down
* Write a bogus entry in the log files that points to some oddball behavior, like a disk-read error or something
* If network is down freeze screen so it looks like computer just locked up
* Ignore all input
* Wipe key parts of disk so forensic recovery is impossible or at least very difficult
* Wipe key parts of memory so forensic recovery is impossible or at least very difficult
* Wipe key parts of cache so forensic recovery is impossible or at least very difficult
* Force or fake a BSOD screen so a casual user will think his computer crashed and blame any resulting data loss on the crash
Re: (Score:2)
Isn't the best thing to do image it, rebuild it, get it running, restore the image on duplicate test hardware then do forensics?
Re: (Score:2)
Overtime? (Score:2)
How many machines can you reimage in a day? Even if you only do one at a time, I imagine you could do 4 or 5 in a working day. If you have an entire office full, ready connected up to the network, you just have to pop in a CD (if you even need one) start the PC and move on. A couple of dozen people could do that lot in a weekends worth of overtime.
Most of the time I spend on rolling out a new PC is delivery, connection and admin. Where's the problem here?
Re: (Score:2)
I can reimage hundreds of computers in a few hours. It all depends on their uniformity and operating systems. Windows has to be imaged on similar hardware or they will BSOD even if they have been sysprepped, for Linux and Mac any image will work on any machine (given you have a fairly standard modular kernel and the architecture stays the same).
Re: (Score:2)
Not true, I had a XP image that will work across a Lot of variations. you need to have ALL drivers for all variations in the image and have the image as a OEM install image. It add's time but it runs the final driver installs and setup on first reboot.
You can do it, you need the OEM tools. I really hope that Windows 7 can do the same.
Re: (Score:2)
I can re-image 60 in a day myself if I stop and talk to people, screw around throwing nerf darts, and riding the electric moped around the office looking for cold pizza.. more if I had more USB sticks or time to make more Boot CD's. I think the network here will eat it's self if I try to re-image 200 or more, the Image server is only 100Bt.
Pop in CD or USB drive, reboot, click yes, go to next one, repeat, go to next one, repeat. DO about 10 go back and walk past them to make sure everything is running, s
Re: (Score:2)
How many machines can you reimage in a day? Even if you only do one at a time, I imagine you could do 4 or 5 in a working day.
It shouldn't take more than 30 minutes to re-image a machine, unless the image is far larger than it really should be.
With a DL DVD-R, you can store about a 15GB image (using compression) along with the bootloader and imaging software. Pop in the disc, boot up and maybe click a few wizard "Next >" buttons.
While one tech starts re-imaging, another can burn extra copies of the imaging DVD-R if there aren't enough to do the job quickly. Then, just hand disks to every employee as they come in and let them
Re: (Score:2)
$20 says... (Score:2)
Twenty bucks says that they never figure out what happened.
Feh. (Score:3, Interesting)
If lil' ol' me can spend a few hundred dollars on enough hard drives stuffed into external enclosures the have two complete backups of all ~1.5TB of data in my system, surely a municipal government can spend a few thousand dollars to do it too.
What the hell, who runs systems that important without backups? Management teams named Shirley?
Re: (Score:2)
"What the hell, who runs systems that important without backups?"
The government, 'nuff said.
Re:Feh. (Score:4, Informative)
From TFA:
Re: (Score:2)
That has been something of a constant battle in my case. I was dumb enough to think backing up the Documents folders from the roaming profiles was good enough. Then I discovered just how non-roaming roaming profiles can be, and that the smallest issue can basically make a roaming profile effectively limited to that computer.
So I went to folder redirection, redirecting the whole Documents tree to the file server. Works great, except people save data to their desktops. So I decide "Okay, I'll redirect the
Re: (Score:2)
It's the only way to do it. Why the hell should 99%+ of the company literally make your job take longer, cost more and be more frustrating?
Re: (Score:2)
That's basically what I said to my manager. I explained that people just storing data anywhere they wanted was making my job a nightmare. Our tape backups were often getting dangerously full, which ultimately would mean a pretty expensive upgrade to a new higher capacity tape drive.
So far I've stopped short of strict quotas, but I see a few people don't know how to or don't want to delete email, meaning the Exchange server database is eating more space, which means more space on the tapes.
People often see
Really? (Score:2)
By corrupting the Windows System32 folder install they lost their own files? Did the malware delete some key file that prevents Window's from hosing the disk and crushing the MFT and/or MBR? I doubt it. The OS installs may be unrecoverable, but the article / spokes people seem to jump the gun by stating such generalizations like "destroyed data" and "essentially destroyed these machines". I imagine that actual "data" of importance is still recoverable vi
Re: (Score:2)
Re: (Score:2)
"simply replace the machines with new ones"
It usually isn't simple. They have to be specifically configured for their usage context, possible configured for the domain, shares, print servers (lol), software installations, blah blah blah. I don't think simply getting new machines is the answer. Why not just use backed-up images and reformat? Purchasing a new machine is hardware cost, and the hardware wasn't destroyed by the virus. And also, pur
Dealing w/ something similar at work (Score:2)
A similar thing happened where I work (uni campus), although due to config errors, not a timebomb.
400 machines got imaged and we're scrambling to collect drives, install new ones, reimage and then run recovery on the old orig drives.
Microsoft really needs to add the ability to set user profiles on a different partition, as you can w/ UNIX.
Re: (Score:3, Informative)
Um, they're called 'roaming profiles' and have been around for some time. You can store users' profiles anywhere you want...different drive, or even a remote server.
Re: (Score:2)
So, because you experienced roaming profiles with an administrator that didn't know how to configure the clients, it's therefore a bad idea?
Firstly, you can limit the cache size. Secondly, you can change the cache location to move it out of the profile.
Roaming profiles aren't necessarily the solution here - though they could be.
Re: (Score:2)
I'm not sure what you mean? It's straightforward, if not trivial, to change the profile location. Two minutes with Google will show you how for your version of Windows.
no major problems (Score:3, Informative)
Re-worked summary of TFA:
- All that has been damaged is the System32 folder of user machines.
- 'Destroyed' I imagine is an IT staff trying to dumb down his language to his perception of the level of the reporter's IT knowledge
- Their IT may have done quite well, the only 'damage' is to PCs that were shut down in the 1 hour window between the attack starting and IT containing it
- Employees were supposed to save to the network. The only issue stated is that some staff were breaking the rules and saved things to their own PC.
All they need to do with the affected machines is to boot from a Windows or Linux CD, copy the files to memory stick and throw their standard "new install" image on. No data loss. No network down time. All they're looking at is some hassle for the ~ 18% of users affected and a very busy IT department. Provided the affected users have other machines to work on (or however not losing much productivity) they're not far off having the best scenario any It department can realistically hope for (well, I'd like to say it's reasonable to hope for not having pissed off employees). Sure, no doubt a dozen IT managers can post their "perfect" system, and another dozen IT managers can show how they could destroy it.
Re:Essentially destroyed? (Score:4, Informative)
Re:Essentially destroyed? (Score:5, Insightful)
if they were running backups, they wouldn't be scratching their heads and behaving completely ignorant of what exactly it was or when it was put in. They obviously lost everything, which I'm sorry but I find some darwinism/justice in that. If you don't even have a backup to look at to see what it was sitting on the hard drive waiting to blow up, you're just beyond help. Maybe better luck next time.
But too many out there simply must learn their lessons the hard way. That will never change.
Re:Essentially destroyed? (Score:5, Insightful)
We've instituted offsite backups, both over the tubes and physically taking images of our servers (all virtualized of course) offsite to a bank safety deposit box. If, for whatever reason, the whole damned building explodes tomorrow, we've got the data sitting on servers in two other geographically distant locations. But if we can't get to those, we have the VM images, so as long as we can get our hands on a server capable of running Linux KVM, we could be up and running in short order (I estimate 3-4 hours, including host OS installation).
The days when a physical or digital attack can fuck the whole organization are gone. There are enough traditional and newer backup schemes out there that even long downtimes aren't necessary.
Re: (Score:2)
You must have a pretty small site if all of your data is contained within the .vmdk files and you can restore an entire datacenter (from bare metal) in 3-4 hours (including OS install time).
Re: (Score:2)
It's relatively small, but we're actually backing images up to hard drives, not to tape or over the wire. The files themselves are both backed up to tape, and use DFS and some other mechanisms (like robocopy replication) to our remote servers. In a worst case scenario, I could pretty much drive the 100 miles, grab the remote domain controller and file servers from one of our satellite sites and drop them in the main office. The guys out there might not be happy that they were accessing everything through
Re: (Score:2)
You must have a pretty small site if all of your data is contained within the .vmdk files and you can restore an entire datacenter (from bare metal) in 3-4 hours (including OS install time).
If you use any of the various wizards that create an install script based on your actual VM host config, you can usually re-install a host in less than 10 minutes.
Then, if you have a good backup of the actual running config of the host (i.e., the VM database, the virtual disk files, etc.), it's just a matter of getting the data to where it belongs.
For most, the biggest issue would definitely be acquistion of the hardware (the hosts, all the network hardware, SAN, etc.), which would generally take a lot long
Re: (Score:2)
(I estimate 3-4 hours, including host OS installation).
I've done this in some small VMWare setups: using snapshot feature on FS (LVM works) plus a few very large external drives (those USB to SATA cradles work great), automate a backup of the snapshots of the OS and VM partitions once every X days take the drive offsite and use another one. With 3 drives, you can rotate them and always keep one offsite. What you now have is essentially a fully working drive you can insert into another server and just turn on, no OS install, no fiddling with VMWare install and v
Re: (Score:2)
The basic idea behind storing snapshots is simply to allow faster recovery of operations even in the case of absolute disaster. We still have nightly differential backups, a weekly full backup, plus Server 2003 DFS and some scripted replication (via robocopy) of file servers. Nothing replaces a good backup scheme, a major pain in the ass to develop, and sometimes a pain to maintain. When we formulated the project, the basic notion was "If a fire/meteor/other disaster takes out one of our offices, how can
Re: (Score:2)
Just to clarify, I was talking about host's file system snapshots (think LVM), and not VMWares's guest snapshots. FS snapshots will let you get a consistent backup of host OS and all the VMs. (if you backup a running VM without a FS snapshot you likely end up with a useless corrupted file)
This way you can grab the off-site backup drive, install it into fresh hardware, turn it on and have a fully functional system in matter of seconds.
And no, it does not replace file-level backups - its just for emergency re
Re:Essentially destroyed? (Score:5, Insightful)
You got it. it's also a great example of how incompetent most City's IT staff are, Hey municipalities... you get what you pay for. How's those $25,000 a year IT staff working out for ya?
Re: (Score:3, Insightful)
But whoever hated them enough to install the timebomb would obviously have sabotaged the backups. Maybe that was what the delay was all about.
Re: (Score:3, Insightful)
It's not, except for the insane or people who aren't able or willing to use a reasonable imaging and app distribution system.
It appears that people who didn't RTFA or who work at tiny tiny sites are criticizing these guys without knowing what the hell they're talking about.
No one does workstation backups because it's costly, risky, inefficient, and generally doesn't work. The only way to make it work is to say "put all the documents you need to backup here" and here is better off being a network drive anywa
Re:I bet they just got Religion (Score:4, Informative)
From working in the backup industry for years, I'm sure they have backups, the problem is that they never tried to verify or restore them. but is there really isn't any data there, compression is great when you just "tar cv * > /dev/null" ...
Heck one time I had a guy who was getting Parity Errors decide that the best way to solve them was to just shut off Parity Checking... Ignorance is bliss I suppose.
Seriously I can't count the number of times I tried to help someone restore their backups after a critical loss that turned out to never have actually verified that they worked in the first place. Just as bad as when I worked in a photo shop and someone said they couldn't get their film out... put the camera in the light locked compartment, stuck my hands in, just to find that he had taken 36 'priceless vacation pictures' on the back of the camera body instead of film.
Re: (Score:2)
Maybe with tapes this is a reasonable expectation.
However, users and IT folk alike copy files to and from CD, to and from the internet, across networks, from drive to drive, from USB to hard drive and back and they don't run into parity errors.
So it's not unreasonable to assume that software and hardware designed to be backup tools wouldn't fail as often as they do.
When my drives fail, it's almost always VERY OBVIOUS, not some subtle creeping error.
I think most of the time the problem is not data c
'if the data will be in a usable form' (Score:2)
When I set up my backup system, one of my concerns was that my loved ones might need to restore my backups, should I get hit by a truck someday.
I realized that if my filesystem is a rat's nest when I back it up, then my backups will be rat's nests as well, as would any restored data. So I have spent several months scrupulously organizing all of my filesystems on all of my computers.
That simplifies my backups,
sort and compress makes small backups (Score:2, Funny)
When you sort the bits first compressed backups are really small.
Re: (Score:2)
Re: (Score:2)
Like the Stupid IT directory that went websurfing as administrator on the Print server?
dollarsto doughnuts that the whole thing is not a planned attack but simply an idiot move that infected a machine and it spread.