Forgot your password?
typodupeerror
Security United States IT

Time Bomb May Have Destroyed 800 Norfolk City PCs' Data 256

Posted by timothy
from the philadelphia-experiment dept.
krebsonsecurity writes "The City of Norfolk, Virginia is reeling from a massive computer meltdown in which an unidentified family of malicious code destroyed data on nearly 800 computers citywide. The incident is still under investigation, but city officials say the attack may have been the result of a computer time bomb planted in advance by an insider or employee and designed to trigger at a specific date, according to krebsonsecurity.com. 'We don't believe it came in from the Internet. We don't know how it got into our system,' the city's IT director said. 'We speculate it could have been a time bomb waiting until a date or time to trigger. Whatever it was, it essentially destroyed these machines.'"
This discussion has been archived. No new comments can be posted.

Time Bomb May Have Destroyed 800 Norfolk City PCs' Data

Comments Filter:
  • by Overzeetop (214511) on Wednesday February 17, 2010 @03:23PM (#31174150) Journal

    It's Naw-Fuck.

    And it's nowhere near as embarrassing as how we pronounce Buena Vista.

    • Re: (Score:3, Funny)

      by wintercolby (1117427)
      Yes, and their Highschool cheer is:
      We don't drink! We don't smoke! Norfolk! Norfolk!

      Pronounced as specified above.
    • by Hatta (162192)

      There's a Norfolk here in Nebraska. It's called "nor-fork". And there's a Buena Vista University just across the river in IA. I cringe every time I hear a radio ad for them. Bew-nah Vista. Just awful.

      Still if you're going to complain about odd spellings and pronunciations, I'd say the British still take the cake with "Worcestershire".

      • by JustOK (667959)

        Not that it's pronounced funny (like Nauwigewauk ), but Saint-Louis-du-Ha!Ha! is kinda fun to say

      • by xaxa (988988)

        Still if you're going to complain about odd spellings and pronunciations, I'd say the British still take the cake with "Worcestershire".

        It's reasonably consistent with the other -cester places (which were all Roman towns):
        Leicester (Les-ter), Gloucester (Glos-ter), Alcester (Ol-ster), Bicester (Bi-ster), Towcester (Tow-ster). And "Wus-ter-shire", for anyone that's still wondering about Worcestershire (Worcester is the city, Worcestershire the county).

        Unfortunately, Cirencester isn't Si-ren-ster, but Si-ren-ses-ter.

        • by xaxa (988988)

          Ooops, I forgot to point out that "shire" in a county name is "shur". "The Shire", as in LotR, is pronounced like shy-er.

      • I think that Arizona, with its odd mix of Indian, Spanish, English and who-knows-what takes the cake with odd spellings and pronunciations.

        Ft. Huachuca (Wa-chu-ka)
        Mogollon Rim (Mo-gee-yawn)
        Tempe (Tem-pee)
        Canyon de Chelly (dee-shay)


        On the other hand, I spent some time in Pueblo, Colorado where about 1/4 of those born there pronounced it Pee-eb-lo.
      • That town's name was mangled due to miscommunication.

        http://en.wikipedia.org/wiki/Norfolk,_Nebraska [wikipedia.org]

        The original name of the colony was a variant of "North Fork", but accounts differ on the exact name: "Northfork", "Nor'fork", and "Nordfork" are all suggested. The name was submitted to federal postal authorities, and at some point was transmuted to "Norfolk". The pronunciation "Norfork" is still used by many Nebraskans.

        They should change the spelling to match the pronunciation.

    • by xaxa (988988) on Wednesday February 17, 2010 @04:04PM (#31174858)

      It's Naw-Fuck.

      In proper Norfolk... well, I'll let Wikipedia [wikipedia.org] explain: More cutting, perhaps, was the pejorative medical slang term "Normal for Norfolk", referencing the county's supposedly high rate of incest. In truth, Norfolk's incest rate is no higher than the rest of England. The term is now discredited, and its use is discouraged by the profession.

      (Sorry, did you want an on-topic comment?)

    • Growing up in Ohio, some of the pronunciations for local places are horrible.

      The first are mostly just anglicizations. Not awful, but sometimes quaint, odd, and hickish. There are a lot more that I'm forgetting.
      Lima - "LYE-muh".
      Ravenna - "Ruh-VEN-nuh"
      Medina - "Meh-DYE-nuh"
      Berlin - "BER-lin' "
      Milan - "MYE-lin'
      Vienna - "VYE-en-nah"
      Bellefontaine - "Bell Fountin' " Ack.

      Then they just get really bad and annoying.

      Nevada - "Nuh-VAY-duh". Really. And most locals pronounce the state Nuh-vah-da or Nuh-vad-ah, so wha

  • by VMaN (164134)

    ... this is the internet... Isn't the apostrophe in the title supposed to be further to the left? :|

    I had to read it twice to confirm it was used correctly.

    • by travdaddy (527149)
      ... this is the internet... Isn't the apostrophe in the title supposed to be further to the left? :|

      So, you're complaining that correct grammar was used?

      You're like the opposite of a Grammar Nazi, or an incompetent one!
    • by mcgrew (92797) *

      Naw, this is the internet. There shouldn't be an apostrophe at all! [angryflower.com] Worse than that, they didn't even misspell anything. What is this internet coming to? If this keeps up, people may become literate!

  • I live in VA Beach, which is the next city down the road (I live a few blocks from exist 20 264, and down-town Norfolk is exit 13ish), and I work in a security-related position, so we tend to keep up on news like this, but this is the first I'm hearing of it, though it looks to have gone down last week (apparently the boot.ini files were modified between 16:30 and 17:30 on 9 February, and only the computers which rebooted during that time period were affected).

    It doesn't sound like the attack was particular
    • Bomb! Destroyed! Meltdown!

      Judging by the hyperbole, the reason you haven't heard about it, is because the destruction was so great, there were no survivors left to report it.
      The blast radius of 800 computers, all exploding at once, would have caused devastation and little radioactivity symbols, the likes of which you've never seen before.

      • by bsDaemon (87307)
        I don't know, I used to play with all disasters turned on in SimCity 2000, and then try and cause them.... shooting the nuclear plant with the microwave beam from the power satellite and stuff. Plenty of radiation symbols when that got done.
      • As someone who lives across the river in Newport News (yes, that's the name of the city), I can tell you that we don't need no destruction to see little radioactivity symbols. We just need to count the ships. (Hint: Norfolk Naval Base and Newport News Shipbuilding).
    • (I also don't watch local TV news, so I don't know if they mentioned it)

      It was mentioned on the Tuesday (I believe) news.
    • by idiotnot (302133)

      WTKR had it last night at 11, but were kinda sketchy on details. Big emphasis on NO CITIZEN OR EMPLOYEE DATA WAS AFFECTED.

      I live in Norfolk; let's just say that the best and brightest aren't working in IT for local governments. Defense companies pay a lot better.

      When I worked for another local city, they were still running an ancient 16-bit version of Netware (would have been like 2002).

      • The only Netware that is not a Netmare.

        I fully believe a 12+ year uptime.

        Bet it's still running strong.

        2 was good once it was setup. Genning sys was a netmare however.

    • I hope they have good backup policies.

      If all they did was fiddle the boot.ini - why not just fix these "destroyed" pc's?

  • by gimmebeer (1648629) on Wednesday February 17, 2010 @03:27PM (#31174246)
    I wonder if there is any correlation between the number of PCs that crashed and the number of PCs set to automatically download and install patches...
    • Re: (Score:2, Insightful)

      by Chrutil (732561)

      I wonder if there is any correlation between the number of PCs that crashed and the number of PCs set to automatically download and install patches

      Sounds like it happened on reboot of these machines, which could imply that patch installation is responsible for the timing (if it mandated a reboot), but not necessarily for the cause.

  • > We don't know how it got into our system... We speculate...

    As long as we're speculating, may I nominate last week's "Operation Cyber Storm" (http://www.dhs.gov/xnews/releases/press_release_0853.shtm).

  • by caseih (160668) on Wednesday February 17, 2010 @03:33PM (#31174348)

    At first glance that blows my mind. That's absolutely huge. Then I check my linux box and /usr/lib64 is 1.7 GB.

  • No explaination (Score:5, Insightful)

    by HotNeedleOfInquiry (598897) on Wednesday February 17, 2010 @03:33PM (#31174350)
    As to why they couldn't just boot to linux or a recovery CD and salvage the data....
    • by gurutc (613652)
      You're right. This is big enough to spend some money on data recovery. Especially the mechanism so they can identify the perp.
    • Re:No explaination (Score:5, Informative)

      by wiredog (43288) on Wednesday February 17, 2010 @03:50PM (#31174642) Journal

      Explanation here [krebsonsecurity.com].

    • by berwiki (989827)
      ...because that sounds ridiculously slow/annoying/complex for 800 PCs.

      gotta love the 'token Linux' reply without thinking about their response first.
    • by Lumpy (12016)

      That requires skilled IT workers. Hell I can build a linux boot CD or USB drive that will boot up, mount the NTFS partition and copy all *.doc, *.xls, *.ppt, etc files to a waiting fileserver. You could recover all the data on 800 machine in one weekend. add in a simple prompt to ask for the pc name or username for that user and no need to even sort through the files.

      Heck if you did your imaging right, the same disk can also start the reimage of the drive from the image repository on the network so It's a

    • Re:No explaination (Score:4, Insightful)

      by Darth_brooks (180756) <clipper377@NoSpAm.gmail.com> on Wednesday February 17, 2010 @05:00PM (#31175894) Homepage

      Sure there was. It was the part about "...784 machines..."

      784 x 30 minutes (That's if IT actually has enough people to keep the restores going non stop, AND doesn't have to travel out to the site to do the restore or recovery, AND doesn't account for the user that has 12 years worth of archived e-mail plus 40 gigs of vital contract that simply MUST be stored on their laptop *eyeroll*) == 23,520 minutes, or about 16 days working round the clock, just recovering data.

      Its all about triage. The users who played by the rules and stored their stuff on the server are probably getting the good old fashioned 'nuke from orbit' fix and will be back in a couple hours. It's the people who need to boot disc / copy to network / reimage / copy back down that are going to be down for a while. Sadly, there are cases where the user simple has to have local data. We've all got them, and we probably all have nightmares about them losing data.

  • by castironpigeon (1056188) on Wednesday February 17, 2010 @03:34PM (#31174366)
    So the data is wiped because the System32 folder is fucked up? Uh-huh... guess they have to throw out all those computers and order new ones. Looks like the data's gone forever.
    • You just restore the image from a ghost backup without worrying about the data because the data is stored (by policy) on the servers. What? A user ignored that policy? Tough luck for him.

      • Re: (Score:3, Interesting)

        Even if you're a complete dolt and don't lose all of that, you can still recover data with some sophisticated technology. The hard drive might claim its empty but the bits are likely still in their last position. (Ever noticed how clearing the partitions off of your hard drive is instantaneous?)

        This is why professionals can still recover a large chunk of data from a hard drive even if you used a drillbit to punch a hole in it. .

      • by jimicus (737525)

        You just restore the image from a ghost backup without worrying about the data because the data is stored (by policy) on the servers. What? A user ignored that policy? Tough luck for him.

        Exactly. No IT department is about to waste much time and effort on recovering data from individual PCs. Yes, you could script much of it but you're still going to have to reimage the things and running that script takes time away from the reimaging process.

        If anything, this could be a blessing in disguise - the admin who's been saying for years "Why do we even leave it physically possible to write to the local hard disk on desktop PCs when the policy states clearly that files get stored on the server?" m

    • You cant take any details from any news articles at face value.
       

    • by Darth_brooks (180756) <clipper377@NoSpAm.gmail.com> on Wednesday February 17, 2010 @04:32PM (#31175382) Homepage

      Umm, yeah. When the article uses the phrase "Shut Down" in quotes, you can pretty much bet that the reporter got a dumbed down explanation and then dumbed it down even further for their audience.

      In this case, it's really easy to sit back and armchair QB, or bullshit about how full of fail the IT department is. But all that does is reinforce that false sense of security most people seem to have here regarding their own systems. Look at the domain admin next to you. Or the group of people that have local admin rights on PC's. Now think about these lines in a batch file:

      bootcfg /delete /ID0

      del C:\windows\system32\*

      Now think of someone pushing that in a batch file into scheduled tasks on a Thursday night. Would you notice? Does your super-duper-uber AV console notify you of new scheduled tasks? You think AV is going to stop a task like that, being run by an admin? here, just for fun, throw this in from of those lines:

      Net Stop YOUR_AV_SERVICE_HERE

      There are a million and one legitimate ways that this could be done by a rouge admin. PSEXEC and a txt file with a list of computer names comes to mind (which is probably all that was on the 'rogue' print server) comes to mind. Snigger and snort all you want. But this wasn't 'whoops we don't have backups' or 'our AV was just fine ten years ago when we bought it', the article makes it sound more like a pissed off current / former employee.

      Either way the city's in a world of pain now, but no where near the world of pain the guy that did this is going to be in. Something like this won't be that hard to figure out. Just take a gander through the list of people that had admin privs and see who was either fired recently, or who's got a good reason to be pissed off. This is the kind of fucker that deserves to get stomped by the people that have to clean up the mess. Thanks asshole. Your super-l33t skills are nothing more than a long inconvenience.

      • Re: (Score:3, Interesting)

        by vlm (69642)

        Either way the city's in a world of pain now, but no where near the world of pain the guy that did this is going to be in. Something like this won't be that hard to figure out.

        Yes, except that the folks in charge are making desperate efforts to destroy any and all evidence by overwriting, reinstalling, etc, per the article and website.

        So, I guarantee a scapegoat has already been determined. In fact, a scapegoat was probably determined before the "incident" occurred, if you know what I mean. The odds that "the guy whom did it" is "the guy that'll be punished/plea bargain" are probably vanishingly low.

        Now if the "journalist" was a real journalist, as opposed to a press release re

      • by u38cg (607297) <calum@callingthetune.co.uk> on Wednesday February 17, 2010 @07:23PM (#31178050) Homepage

        There are a million and one legitimate ways that this could be done by a rouge admin.

        Dude, I could do that, and I'm not even vermillion :p

    • Damn, with a friendly IT department like that, Norfolk don't need enemy malware.

      Even a simple Windows Repair Install would have fixed the machines and kept the data files.

      • by jimicus (737525)

        Damn, with a friendly IT department like that, Norfolk don't need enemy malware.

        Even a simple Windows Repair Install would have fixed the machines and kept the data files.

        There are lots of automated mechanisms - both using Microsoft's own Remote Imaging Services and third-party products - for rebuilding an OS and installing all applications very quickly to a bunch of PCs. With everything properly set up, you can go from nothing to every PC built, on the domain and all applications installed in under an hour. If you use multicast, about the only limitation is the speed of the network and how many PCs your technicians can visit to force a PXE boot in a given space of time.

    • by murdocj (543661)

      Huh? That's like saying the data on Linux system is hosed because your kernel image got zapped. All the data is there, you just re-install the O/S.

  • I think Time Bomb is the best of all the No-Heroics superheros. http://www.youtube.com/watch?v=JLaXUTdybjc [youtube.com]
    Oh wait, you were talking about that ....
  • by Reason58 (775044) on Wednesday February 17, 2010 @03:48PM (#31174604)
    From the article:

    IT specialists for the city found that the system serving as the distribution point for the malware within the city’s network was a print server that handles printing jobs for Norfolk City Hall. However, an exact copy of the malware on that server may never be recovered, as city computer technicians quickly isolated and rebuilt the offending print server. “Obviously, our first reaction was to shut it down and restore services, and at least initially we weren’t concerned about capturing [the malware] or setting it aside,” Cluff said.

    Obviously, your reaction was wrong in every way. When a system is compromised you physically unplug it from the network and keep it powered on so that you can run forensics on it. Good work destroying any evidence you might have had about not only who performed this attack, but what weakness in your security they exploited to accomplish it. All that just to get a print server of all things back online as fast as possible.

    • by alen (225700)

      this is the government

      when i first started working for private industry after working for uncle sam for years, the first thing i noticed was a lack of paper. government employees had mountains of it in every cube and office. the real world had long ago moved to electronic format

    • by Cassini2 (956052)

      When I even think some major problem exists with either data on the hard drive, or the hard drive itself, I just replace the hard drive. This permits data recovery of any salvageable data on the old hard drive. It also quarantines the virus infection to the old hard drive.

      A new hard drive is worth $50-$100. If you find any important files on the old hard drive, then the new one has paid for itself. Also, it does much to preserve your chain of evidence if the problem requires forensics.

    • by davidwr (791652) on Wednesday February 17, 2010 @04:35PM (#31175422) Homepage Journal

      * Check every few seconds to see if network goes down
      * Write a bogus entry in the log files that points to some oddball behavior, like a disk-read error or something
      * If network is down freeze screen so it looks like computer just locked up
      * Ignore all input
      * Wipe key parts of disk so forensic recovery is impossible or at least very difficult
      * Wipe key parts of memory so forensic recovery is impossible or at least very difficult
      * Wipe key parts of cache so forensic recovery is impossible or at least very difficult
      * Force or fake a BSOD screen so a casual user will think his computer crashed and blame any resulting data loss on the crash

    • by epp_b (944299)

      When a system is compromised you physically unplug it from the network and keep it powered on so that you can run forensics on it.

      Isn't the best thing to do image it, rebuild it, get it running, restore the image on duplicate test hardware then do forensics?

    • The last company I worked for the owners wanted systems back online as fast as possible. Once they were back online, I could troubleshoot to my heart's content to figure out what caused the problem. I suspect that the decision makers in this situation had the same priorities. Computers are there to get work done. Get them back up and working as soon as possible so that people can get work done. Once everybody who relies on those computers can work again, only then do you start trying to figure out what went
  • How many machines can you reimage in a day? Even if you only do one at a time, I imagine you could do 4 or 5 in a working day. If you have an entire office full, ready connected up to the network, you just have to pop in a CD (if you even need one) start the PC and move on. A couple of dozen people could do that lot in a weekends worth of overtime.

    Most of the time I spend on rolling out a new PC is delivery, connection and admin. Where's the problem here?

    • by guruevi (827432)

      I can reimage hundreds of computers in a few hours. It all depends on their uniformity and operating systems. Windows has to be imaged on similar hardware or they will BSOD even if they have been sysprepped, for Linux and Mac any image will work on any machine (given you have a fairly standard modular kernel and the architecture stays the same).

      • by Lumpy (12016)

        Not true, I had a XP image that will work across a Lot of variations. you need to have ALL drivers for all variations in the image and have the image as a OEM install image. It add's time but it runs the final driver installs and setup on first reboot.

        You can do it, you need the OEM tools. I really hope that Windows 7 can do the same.

    • by Lumpy (12016)

      I can re-image 60 in a day myself if I stop and talk to people, screw around throwing nerf darts, and riding the electric moped around the office looking for cold pizza.. more if I had more USB sticks or time to make more Boot CD's. I think the network here will eat it's self if I try to re-image 200 or more, the Image server is only 100Bt.

      Pop in CD or USB drive, reboot, click yes, go to next one, repeat, go to next one, repeat. DO about 10 go back and walk past them to make sure everything is running, s

    • by nabsltd (1313397)

      How many machines can you reimage in a day? Even if you only do one at a time, I imagine you could do 4 or 5 in a working day.

      It shouldn't take more than 30 minutes to re-image a machine, unless the image is far larger than it really should be.

      With a DL DVD-R, you can store about a 15GB image (using compression) along with the bootloader and imaging software. Pop in the disc, boot up and maybe click a few wizard "Next >" buttons.

      While one tech starts re-imaging, another can burn extra copies of the imaging DVD-R if there aren't enough to do the job quickly. Then, just hand disks to every employee as they come in and let them

  • Twenty bucks says that they never figure out what happened.

  • Feh. (Score:3, Interesting)

    by Pojut (1027544) on Wednesday February 17, 2010 @04:05PM (#31174882) Homepage

    If lil' ol' me can spend a few hundred dollars on enough hard drives stuffed into external enclosures the have two complete backups of all ~1.5TB of data in my system, surely a municipal government can spend a few thousand dollars to do it too.

    What the hell, who runs systems that important without backups? Management teams named Shirley?

    • "What the hell, who runs systems that important without backups?"

      The government, 'nuff said.

    • Re:Feh. (Score:4, Informative)

      by mcgrew (92797) * on Wednesday February 17, 2010 @04:14PM (#31175036) Homepage Journal

      From TFA:

      Cluff said the malicious software appears to have been designed to trash vital operating files in the Windows\System32 folder on the infected machines. Cluff said a healthy, functioning System32 directory weighs in at around 1.5GB, but the computers infected with this as-yet-unidentified malware had their System32 folders chopped down to around a third of that size, rendering them unbootable. Cluff added that city employees are urged to store their data on file servers, which were largely untouched by the attack, but he said employees who ignored that advice and stored important documents on affected desktop computers may have lost those files.

  • by cosm (1072588)
    "destroyed data on nearly 800 computers citywide".

    By corrupting the Windows System32 folder install they lost their own files? Did the malware delete some key file that prevents Window's from hosing the disk and crushing the MFT and/or MBR? I doubt it. The OS installs may be unrecoverable, but the article / spokes people seem to jump the gun by stating such generalizations like "destroyed data" and "essentially destroyed these machines". I imagine that actual "data" of importance is still recoverable vi
    • If it costs them $1 Mil in labor to recover the machines vs. $0.8 Mil to simply replace the machines with new ones, then the machines are "destroyed."
      • by cosm (1072588)
        Perhaps destroyed by an economist or bureaucrats standards.

        "simply replace the machines with new ones"

        It usually isn't simple. They have to be specifically configured for their usage context, possible configured for the domain, shares, print servers (lol), software installations, blah blah blah. I don't think simply getting new machines is the answer. Why not just use backed-up images and reformat? Purchasing a new machine is hardware cost, and the hardware wasn't destroyed by the virus. And also, pur
  • A similar thing happened where I work (uni campus), although due to config errors, not a timebomb.

    400 machines got imaged and we're scrambling to collect drives, install new ones, reimage and then run recovery on the old orig drives.

    Microsoft really needs to add the ability to set user profiles on a different partition, as you can w/ UNIX.

    • Re: (Score:3, Informative)

      by Itninja (937614)

      Microsoft really needs to add the ability to set user profiles on a different partition, as you can w/ UNIX.

      Um, they're called 'roaming profiles' and have been around for some time. You can store users' profiles anywhere you want...different drive, or even a remote server.

    • by Albanach (527650)

      Microsoft really needs to add the ability to set user profiles on a different partition, as you can w/ UNIX.

      I'm not sure what you mean? It's straightforward, if not trivial, to change the profile location. Two minutes with Google will show you how for your version of Windows.

  • no major problems (Score:3, Informative)

    by DaveGod (703167) on Wednesday February 17, 2010 @05:21PM (#31176256)

    Re-worked summary of TFA:
    - All that has been damaged is the System32 folder of user machines.
    - 'Destroyed' I imagine is an IT staff trying to dumb down his language to his perception of the level of the reporter's IT knowledge
    - Their IT may have done quite well, the only 'damage' is to PCs that were shut down in the 1 hour window between the attack starting and IT containing it
    - Employees were supposed to save to the network. The only issue stated is that some staff were breaking the rules and saved things to their own PC.

    All they need to do with the affected machines is to boot from a Windows or Linux CD, copy the files to memory stick and throw their standard "new install" image on. No data loss. No network down time. All they're looking at is some hassle for the ~ 18% of users affected and a very busy IT department. Provided the affected users have other machines to work on (or however not losing much productivity) they're not far off having the best scenario any It department can realistically hope for (well, I'd like to say it's reasonable to hope for not having pissed off employees). Sure, no doubt a dozen IT managers can post their "perfect" system, and another dozen IT managers can show how they could destroy it.

This screen intentionally left blank.

Working...