Atlanta Still Struggles To Recover From Ransomware Attack (reuters.com) 91
An anonymous reader quotes Reuters:
Atlanta's top officials holed up in their offices on Saturday as they worked to restore critical systems knocked out by a nine-day-old cyber attack that plunged the Southeastern U.S. metropolis into technological chaos and forced some city workers to revert to paper... Police and other public servants have spent the past week trying to piece together their digital work lives, recreating audit spreadsheets and conducting business on mobile phones in response to one of the most devastating "ransomware" virus attacks to hit an American city. Three city council staffers have been sharing a single clunky personal laptop brought in after cyber extortionists attacked Atlanta's computer network with a virus that scrambled data and still prevents access to critical systems. "It's extraordinarily frustrating," said Councilman Howard Shook, whose office lost 16 years of digital records...
City officials have declined to discuss the extent of damage beyond disclosed outages that have shut down some services at municipal offices, including courts and the water department. Nearly 6 million people live in the Atlanta metropolitan area... Atlanta police returned to taking written case notes and have lost access to some investigative databases, department spokesman Carlos Campos told Reuters... Meanwhile, some city employees complained they have been left in the dark, unsure when it is safe to turn on their computers. "We don't know anything," said one frustrated employee as she left for a lunch break on Friday.
"Our data management teams are working diligently to restore normal operations and functionalities to these systems," said a spokesperson for the police department, adding that they "hope to be back online in the very near future."
City officials have declined to discuss the extent of damage beyond disclosed outages that have shut down some services at municipal offices, including courts and the water department. Nearly 6 million people live in the Atlanta metropolitan area... Atlanta police returned to taking written case notes and have lost access to some investigative databases, department spokesman Carlos Campos told Reuters... Meanwhile, some city employees complained they have been left in the dark, unsure when it is safe to turn on their computers. "We don't know anything," said one frustrated employee as she left for a lunch break on Friday.
"Our data management teams are working diligently to restore normal operations and functionalities to these systems," said a spokesperson for the police department, adding that they "hope to be back online in the very near future."
They should all be sacked. (Score:5, Insightful)
They should all be sacked.
Backups. Backups. Backups.
Simple. Known process.
Not done = sacked.
Re: (Score:1, Interesting)
This is the inevitable outcome anyone could have foreseen would arise from letting people who don't care about security sell millions of computers to people who don't understand security. If there had been a backup server I guarantee it would have just been hacked too.
Re: (Score:1)
This is the inevitable outcome anyone could have foreseen would arise from letting people who don't care about security sell millions of computers to people who don't understand security.
Next to stupidity, greed is one of humans' finest traits.
Has anyone considered that this might have been an "inside job" . . . ? Like, a disgruntled city employee purposely starting the infection for a small fee from the ransomers . . . ?
It would be interesting to see if they can trace the spread back to a "patient zero".
Re: (Score:3)
Never underestimate the power of human stupidity.
Anyway - this also highlights the need to really segment your data nets so that an intrusion don't propagate easily.
And backups are also important of course. CD-ROMs are decent for short term archiving, but for long term archiving we need something better. SD cards also have a little "lock" switch, but it's in reality telling the computer that the device is read only so it's not proof against extreme hacks.
Re: (Score:1)
The problem is the plastic those disks are made of has a minimum guaranteed life span of about 12 years. Sure, you might be able to keep them for 20 or 30 if you keep them out of direct sunlight and don't read them too much, but even then the rotational speeds they have to endure makes the risk of one shattering in the drive an inevitability over the long term. Stored in sunlight or exposed to too much temperature change and even the dye they use for the writable disks can break down, leading to data loss
Re: (Score:3)
Never underestimate the power of human stupidity.
To fail to do so would be stupid.
Re: (Score:2)
The part that bugs me is that the effect is completely indistinguishable from the equally likely probability of completely accidental cross-contamination from an employee's personal USB device.
Re: They should all be sacked. (Score:3)
Re:They should all be sacked. (Score:4, Informative)
What I can't understand is why these high profile ransomware attacks haven't prompted a rush to adopt copy-on-write filesystems. It's not like ZFS is exactly new.
I understand that because of cost places like Atlanta try to run their networks with the least expertise they can get away with, but projects like FreeNAS make it really easy. I have a cheap server running at home and have background tasks scheduled to rsync changes to it. It's like it's not even there, but if I need to I can mount the NAS box and right click on a file in Windows and access the all the previous versions.
Re: (Score:2)
You are suggesting a Unix approach to a Windows problem.
Sure it works fine, and you will be better off in the long run and it would be cheaper too. But that isn’t was Windows was designed. You need to buy multiple thousands of dollars on a app to do this. That is often overkill for the need at hand. Because having that app on the resume looks better then a tiny little tool that does the trick.
Re: (Score:2)
Re:They should all be sacked. (Score:4, Insightful)
If you think of security exclusively in terms of prevention you are in deep trouble.
Re:"Unknown User" (Score:5, Informative)
Nonsense! 100% daily backups of systems, using a suite of tools kept offline except during backups activity is ALWAYS a solution....simply because an attack starts at a particular time; anything you've kept offline prior to that time is a resource to be used to recover. Yes, there is the problem of recapturing the lost data in that time interval, but it's a LOT better than having to start redesigning software from scratch AFTER the attack has occurred!
100% daily backups, with recycling of media over a period of a few weeks is a MANDATORY requirement for every computer under my management. Since I started doing that in 2001, I have never had (nor has any client had) an unrecoverable loss of data.
The other trick is keeping data separated from executables. My mantra is "C: is for Code, D: is for Data". The idea that everything should be on the same logical drive is simply WRONG.
There are no perfectly secure systems, and perfection is a fools game. But, simple strategies, unerringly repeated over time, can make recovery from assaults (or hard-disk failure) a straight-forward solution.
Re: (Score:2)
My work windows machine keeps close to 100% immediate backups. As soon as I change a file, it's saved over the network and there are a few weeks of all file changes available for recovery.
An infection is identified pretty quickly, the affected machine(s) isolated and rolled back. Pretty much the only thing you can lose (unless you are really trying) is what you've just typed into the editor.
Real work is done on Linux machines we VNC into, where I understand things are more tightly backed up.
This is not hard
Re: (Score:2)
There's so many ways storage and backups and could be leveraged to mitigate this, although I would imagine for most sites with Windows it would revolve around existing storage platform snapshots or VM backups to offline storage or in isolated security realms, not "zfs" by itself.
I have clients that run hourly incrementals of all VMs and their backup runs in an isolated security realm (firewalled, some physical network isolation and no shared or trust-related Windows security relationship).
Re: (Score:3)
You can have rsync only send updated blocks. Or you could simply use a COW file system mounted by iSCSI on your windows servers, if you prefer simplicity to features.
Re: (Score:1)
This is the inevitable outcome anyone could have foreseen would arise from letting people who don't care about security sell millions of computers to people who don't understand security. If there had been a backup server I guarantee it would have just been hacked too.
If your backups can be corrupted by someone hacking into your server, then they aren't really backups.
Tape is still widely used for backups for a good reason. We rotate ours offsite, and our contingency plan if we do get hacked is to rebuild the backup server from scratch and restore from backup, we have 4 weekly, 4 monthly, and 8 quarterly sets of backup tapes off site, so we can go back 2 years if needed to get a good backup.
Re: They should all be sacked. (Score:1)
There's a serious lack of accountability at all levels when it comes to IT security.
One of the biggest benefits of the industry 's move to Cloud, imo, is to remove certain classes of vulnerabilities from the hands of many organizations since I don't expect real accountability to ever increase.
Yes and no. (Score:3, Insightful)
Yes, they should all be sacked.
No, not the IT guys. The beancounters and managers who ignored their advice and failed to foresee the need for a proper backup management strategy for the city. IT knows this crap can happen, and IT tells Management about the need for proper backups, daily, weekly, monthly, on-site, off-site, and tape. We tell them RAID is not a backup strategy. WE tell them without backups their necks are in the noose when, not if, the shit hits the fan.
Well, 9 days ago, the fan got crushed u
Re: (Score:1)
Depends. If they had any kind of audits done, how in the hell did they pass? Who performed them?
Remember the big credit card breach at Target? They were audited -- and passed -- just a couple of months earlier.
Every company -- EVERY COMPANY -- that has suffered a major breach in the last few years was audited and certified Grade A Okey Dokey a few months before the breach.
Audits are completely useless and meaningless because:
(a) The auditors are just as stupid and incompetent as the people they are auditing
(2) If the auditors start flunking people and telling their clients that they are going to have
Re: (Score:2, Insightful)
Audits are completely useless and meaningless because:
(a) The auditors are just as stupid and incompetent as the people they are auditing
(2) If the auditors start flunking people and telling their clients that they are going to have to fix their shitty broken systems --- i.e., spend a lot of money -- they will quickly lose all their business. So, everyone passes their audits with flying colors.
This, this, THIS. I briefly worked for a company that provided lockbox services for a number of banks, running the
Re: (Score:2)
Audits usually check for the bare minimum. And they may fail some aspects and pass others. So they get a good enough score to stay certified.
Re: (Score:2)
>Audits usually check for the bare minimum.
Audits check for compliance -- to ISO 9000, webtrust, NIST, whatever.
The specs are useless because they can't be too prescriptive.
Re: (Score:3)
It's been quite a few years since I went through an ISO 9001 audit, but I recall thinking what a load of crap it was at the time. I'm no expert on it, but what I was told was that it's primary function was to verify that you followed your processes. But, it did nothing to ensure that those processes were worth a damn. Then, what the fuck good is it? Maybe I was mislead...I've never bothered to look it up.
Re: (Score:1)
This. This is why ISO isn't useful for anything but PR that you have processes in place.
I worked at a ba
Re: (Score:2)
Bahahah! That's a good one. The managers will simply fire their IT and give themselves bonuses for doing so now that the troublemakers are gone. Problem solved.
We all have seen Office Space right? The bobs seem to LOVE management everytime and they seem to think they are invaluable unlike I.T. Sadly they are right as HR won't touch them as they have too much power over HR's job.
Some organizations even have a rule that only good ideas come from management and lowly employees need to shut up and take notes in
Re: (Score:2)
There's a serious lack of accountability at all levels when it comes to IT security.
One of the biggest benefits of the industry 's move to Cloud, imo, is to remove certain classes of vulnerabilities from the hands of many organizations since I don't expect real accountability to ever increase.
Yep. I hold my head high as a competent techy, but when it comes to my wife's business, it's cloud all the way. Pay someone else to do the storage security, serving, email and access control. It's cheaper than hiring and a lot easier.
Re:They should all be sacked. (Score:5, Insightful)
Unless you are working for a government agency with bosses who don’t want to fund your department.
Backups cost money. Redundant off site hot failover systems cost more.
Please explain to the general public on why the city should have computers running in hope you don’t need to use them. When they can use that money to feed the poor.
I have done years of consulting and working across many agencies. And for nearly every agency the tech workers are not incompetent, I may disagree with their methods, but they know what they are talking about. The bosses on the other hand especially ones without technical background, see the IT departments as a cost center. So will invest the minimum necessary to keep it running. They don’t realize that their equipment is being attacked constantly and it is only matter of time until something gets across.
Current I work in healthcare and luckily management invest a lot into IT. So when spyware hit we only suffered minor damage and had it restored running with 15 minuets of missing data. After that incident we in the IT area was livid, and doubled our efforts to stop it again.
Re: (Score:1)
No - They where probably running outdated Java based web servers, that where hacked using an open source tool.
And those web servers can run on any OS, so it has nothing to do with Linux.
Re: (Score:2)
WannaCry on Linux, nice try troll.
Re: (Score:3)
Why not? The first thing every Linux installation does is enable interoperability with Windows networking. Wanacry very quickly spreads to SMB shares. If they are writable then a remote client can happily encrypt your shit. Or if you want, https://www.samba.org/samba/se... [samba.org] gives you your own Linux special flavour of Wanacry.
Now yes the GP is a troll, and it most likely wasn't the case. But security is about dealing with the possible, and just running Linux doesn't make you immune from anything, especially n
Re: (Score:2)
Re: (Score:2)
but properly rotated backups (in my case daily, weekly, monthly and yearly) limit any potential impact, for Linux or any other kind of server.
Can we get a "Here! Here!" in here!
I always like to point to two examples of Wannacry which happened in the largest port of Europe:
DHL: Got hit by Wannacry: Completely ceased all delivery operations. The entire business went down for 3 days. Warehouses filled with undelivered packages.
Port of Rotterdam: Got hit by Wannacry: Picked up pen and paper and kept on processing containers at the same rate as before. IT in the background recovered and spent a bit of money on importing the paper trail back into the e
Re: (Score:2)
Why not? The first thing every Linux installation does is enable interoperability with Windows networking. Wanacry very quickly spreads to SMB shares. If they are writable then a remote client can happily encrypt your shit. Or if you want, https://www.samba.org/samba/se... [samba.org] gives you your own Linux special flavour of Wanacry.
Now yes the GP is a troll, and it most likely wasn't the case. But security is about dealing with the possible, and just running Linux doesn't make you immune from anything, especially not user stupidity.
I've actually stopped setting up Windows networking by default on my Linux systems, especially my servers. It's easier to install FileZilla or WinSCP on Windows.
Danny Droptables hits Atlanta? (Score:4, Insightful)
We complain bitterly about problems with industrial espionage and yet we still cheap out and use crapware swiss cheese .Net garbage that hackers in China and Russian can drive a truck through.
What's with blaming the Russians and Chinese? (Score:1, Redundant)
Great post until you mentioned them. Lots of other possibilities - and unless it was government sanctioned, it's especially pointless to mention those particular nations.
Re: (Score:1)
Someone else suggested it first but I'd also place my bets on "disgruntled employee" as the primary threat vector. Furthermore, I'd even go so far as to postulate "a few free days to goof off without any work to do" as the motive.
Re: (Score:1)
It really can't, at this point, or they'd have at least been able to fix it by now.
And ... LOL... I'm no Microsoft shill. If you're that bad at reading the context cues maybe you should stop telling yourself you know fuck-all about computers.
Re: Danny Droptables hits Atlanta? (Score:1)
I was with you until you confused VB6 and .Net so apparently you have a knowledge problem here.
Re: (Score:1)
I was with you until you confused VB6 and .Net so apparently you have a knowledge problem here.
No confusion involved. A butt tonne of old VB code is also still out there working so that the look and feel of the interfaces remain static as well as antiquated data base interface requirements that do not get updated because of rewrite and retraining costs. Then the problem becomes leaving network visible vulnerabilities there ignored for a very long time just waiting for a criminal hacker to spot them. The point is how and where in the front end did the attack happen?
Dollars to donuts the attack was do
Inside job (allegedly) (Score:5, Interesting)
Re: (Score:2)
From the previous article: "working diligently with support from Microsoft to resolve the issue". So I'm assuming a Windows-based attack vector. Hey let's connect everything to Active Directory and let everything authenticate everywhere. Authorization, yeah, we authenticate everyone through Active Directory.
Re: The SOUTH Shall Rise Again! (Score:1)
Atlanta isn't really the South. Atlanta is to the South as Austin is to Texas. The all-hat, no-cows cowboy.
kill the windows servers, and do backups (Score:2, Insightful)
throw the windows servers in the trash.
and do backups.
OK, so it's April 1 & all, but still.... (Score:3)
Re: (Score:2)
You are looking at this backwards. As explained by the program "Yes Minister", losing files can be very convenient:
I live in metro ATL. Non-issue. (Score:1)
It isn't like the other 4.5M people are impacted by what the 500K people in Atlanta do or their incompetent city govt.
City of Atlanta is just 500K people. Hardly the entire metro area of 5+M people. Many of those non-Atlanta govts are efficient, capable, and smart. A few county govts merge their services to save money overall. These aren't tiny rural counties. Metro Atlanta has about 20 nearby counties. Fulton is where Atlanta is.
Proof! (Score:1)
Stupid is as stupid does.
"Three city council staffers have been sharing a single clunky personal laptop brought in after cyber extortionists"
And no one sees this as an issue? Only in government could people exceed to this level of incompetence and still know they have jobs.
Question: (Score:1)
Re:Question: (Score:4, Funny)
So has Georgia actually passed a law that will effectively make the investigation of this ransomware attack illegal? That would be both stupid and highly amusing.
They don't know. All the laws were on the servers.