Time Bomb May Have Destroyed 800 Norfolk City PCs' Data 256
krebsonsecurity writes "The City of Norfolk, Virginia is reeling from a massive computer meltdown in which an unidentified family of malicious code destroyed data on nearly 800 computers citywide. The incident is still under investigation, but city officials say the attack may have been the result of a computer time bomb planted in advance by an insider or employee and designed to trigger at a specific date, according to krebsonsecurity.com. 'We don't believe it came in from the Internet. We don't know how it got into our system,' the city's IT director said. 'We speculate it could have been a time bomb waiting until a date or time to trigger. Whatever it was, it essentially destroyed these machines.'"
Re:Essentially destroyed? (Score:5, Insightful)
if they were running backups, they wouldn't be scratching their heads and behaving completely ignorant of what exactly it was or when it was put in. They obviously lost everything, which I'm sorry but I find some darwinism/justice in that. If you don't even have a backup to look at to see what it was sitting on the hard drive waiting to blow up, you're just beyond help. Maybe better luck next time.
But too many out there simply must learn their lessons the hard way. That will never change.
No explaination (Score:5, Insightful)
Norfolk's IT is fail. (Score:5, Insightful)
Re:Essentially destroyed? (Score:5, Insightful)
We've instituted offsite backups, both over the tubes and physically taking images of our servers (all virtualized of course) offsite to a bank safety deposit box. If, for whatever reason, the whole damned building explodes tomorrow, we've got the data sitting on servers in two other geographically distant locations. But if we can't get to those, we have the VM images, so as long as we can get our hands on a server capable of running Linux KVM, we could be up and running in short order (I estimate 3-4 hours, including host OS installation).
The days when a physical or digital attack can fuck the whole organization are gone. There are enough traditional and newer backup schemes out there that even long downtimes aren't necessary.
Re:It happened on Patch Tuesday. (Score:2, Insightful)
Sounds like it happened on reboot of these machines, which could imply that patch installation is responsible for the timing (if it mandated a reboot), but not necessarily for the cause.
Destroying Evidence (Score:5, Insightful)
IT specialists for the city found that the system serving as the distribution point for the malware within the city’s network was a print server that handles printing jobs for Norfolk City Hall. However, an exact copy of the malware on that server may never be recovered, as city computer technicians quickly isolated and rebuilt the offending print server. “Obviously, our first reaction was to shut it down and restore services, and at least initially we weren’t concerned about capturing [the malware] or setting it aside,” Cluff said.
Obviously, your reaction was wrong in every way. When a system is compromised you physically unplug it from the network and keep it powered on so that you can run forensics on it. Good work destroying any evidence you might have had about not only who performed this attack, but what weakness in your security they exploited to accomplish it. All that just to get a print server of all things back online as fast as possible.
Re:Essentially destroyed? (Score:5, Insightful)
You got it. it's also a great example of how incompetent most City's IT staff are, Hey municipalities... you get what you pay for. How's those $25,000 a year IT staff working out for ya?
You are fail for believing news articles (Score:3, Insightful)
You cant take any details from any news articles at face value.
Re:Just so you get the pronunciation right... (Score:3, Insightful)
One of my first interactions in the state after being in California for a couple of years was at a Wendy's drive-though. The attendant was kind enough to tell me "I put you some salt and ketchup in the bag." Is there such a thing as hillbillionics?
Someday I'm going to run for public office, and this thread is going to come back and bit me in the ass. I just know it.
Re:No explaination (Score:4, Insightful)
Sure there was. It was the part about "...784 machines..."
784 x 30 minutes (That's if IT actually has enough people to keep the restores going non stop, AND doesn't have to travel out to the site to do the restore or recovery, AND doesn't account for the user that has 12 years worth of archived e-mail plus 40 gigs of vital contract that simply MUST be stored on their laptop *eyeroll*) == 23,520 minutes, or about 16 days working round the clock, just recovering data.
Its all about triage. The users who played by the rules and stored their stuff on the server are probably getting the good old fashioned 'nuke from orbit' fix and will be back in a couple hours. It's the people who need to boot disc / copy to network / reimage / copy back down that are going to be down for a while. Sadly, there are cases where the user simple has to have local data. We've all got them, and we probably all have nightmares about them losing data.
Re:Essentially destroyed? (Score:3, Insightful)
But whoever hated them enough to install the timebomb would obviously have sabotaged the backups. Maybe that was what the delay was all about.
Re:Essentially destroyed? (Score:3, Insightful)
It's not, except for the insane or people who aren't able or willing to use a reasonable imaging and app distribution system.
It appears that people who didn't RTFA or who work at tiny tiny sites are criticizing these guys without knowing what the hell they're talking about.
No one does workstation backups because it's costly, risky, inefficient, and generally doesn't work. The only way to make it work is to say "put all the documents you need to backup here" and here is better off being a network drive anyway.
Re:Essentially destroyed? (Score:2, Insightful)
VMware Data Recovery is a piece of shit that rarely works the way you want it. Try reading the forums sometime to see how much grief it gives others.