The FBI Accessed and Repaired 'Hundreds' of Hacked Microsoft Exchange Servers (csoonline.com) 86
America's top law enforcement agency "obtained a court order that allowed it to remove a backdoor program from hundreds of private Microsoft Exchange servers that were hacked through zero-day vulnerabilities earlier this year," reports CSO. (Thanks to detritus. (Slashdot reader #46,421) for sharing the news...)
Earlier this week, the Department of Justice announced that the FBI was granted a search and seizure warrant by a Texas court that allows the agency to copy and remove web shells from hundreds of on-premise Microsoft Exchange servers owned by private organizations. A web shell is a type of program that hackers install on hacked web servers to grant them backdoor access and remote command execution capabilities on those servers through a web-based interface.
In this case, the warrant targeted web shells installed by a cyberespionage group dubbed Hafnium that is believed to have ties to the Chinese government. In early March, Microsoft reported that Hafnium has been exploiting previously unpatched vulnerabilities in Microsoft Exchange to compromise servers. At the same time, the company released patches for those vulnerabilities, as well as indicators of compromise and other detection tools, but this didn't prevent other groups of attackers from exploiting the vulnerabilities after they became public. In its warrant application, dated April 13, the FBI argues that despite the public awareness campaigns by Microsoft, CISA and the FBI itself, many servers remained infected with the web shell deployed by Hafnium. While the exact number has been redacted from the unsealed warrant, the DOJ said in a press release that it was "hundreds."
The FBI asked for, and received court approval, to access the malicious web shells through the passwords set by the original attackers and then use that access against the malware itself by executing a command that will delete the web shell, which is essentially an .aspx script deployed on the server. The FBI was also allowed to make a copy of the web shells first because they could constitute evidence.
The warrant states that it "does not authorize the seizure of any tangible property" or the copying or alteration of any content from the servers aside from the web shell themselves, which are identified in the warrant by their unique file paths. This means the FBI was not granted permission to patch the vulnerabilities to protect the servers from future exploitation or to remove any additional malware or tools that hackers might have already deployed...
The FBI sent an email message from an official email account, including a copy of the warrant, to the email addresses associated with the domain names of the infected servers.
An official statement from the Department of Justice is already using the past tense, announcing that U.S. authorities "have executed a court-authorized operation to copy and remove malicious web shells from hundreds of vulnerable computers in the United States. They were running on-premises versions of Microsoft Exchange Server software used to provide enterprise-level email service."
In this case, the warrant targeted web shells installed by a cyberespionage group dubbed Hafnium that is believed to have ties to the Chinese government. In early March, Microsoft reported that Hafnium has been exploiting previously unpatched vulnerabilities in Microsoft Exchange to compromise servers. At the same time, the company released patches for those vulnerabilities, as well as indicators of compromise and other detection tools, but this didn't prevent other groups of attackers from exploiting the vulnerabilities after they became public. In its warrant application, dated April 13, the FBI argues that despite the public awareness campaigns by Microsoft, CISA and the FBI itself, many servers remained infected with the web shell deployed by Hafnium. While the exact number has been redacted from the unsealed warrant, the DOJ said in a press release that it was "hundreds."
The FBI asked for, and received court approval, to access the malicious web shells through the passwords set by the original attackers and then use that access against the malware itself by executing a command that will delete the web shell, which is essentially an .aspx script deployed on the server. The FBI was also allowed to make a copy of the web shells first because they could constitute evidence.
The warrant states that it "does not authorize the seizure of any tangible property" or the copying or alteration of any content from the servers aside from the web shell themselves, which are identified in the warrant by their unique file paths. This means the FBI was not granted permission to patch the vulnerabilities to protect the servers from future exploitation or to remove any additional malware or tools that hackers might have already deployed...
The FBI sent an email message from an official email account, including a copy of the warrant, to the email addresses associated with the domain names of the infected servers.
An official statement from the Department of Justice is already using the past tense, announcing that U.S. authorities "have executed a court-authorized operation to copy and remove malicious web shells from hundreds of vulnerable computers in the United States. They were running on-premises versions of Microsoft Exchange Server software used to provide enterprise-level email service."
Responsible parties (Score:4, Interesting)
Shouldn't these server operators be held responsible for their negligence? This coming from a person who thinks people should be required to have a basic "operators license" to even access the internet.
Re: (Score:3)
Re: Responsible parties (Score:3)
Re: (Score:1)
Re: (Score:3)
Maybe the FBI had access because they were removing their own stuff :)
Re: (Score:2, Offtopic)
Anyway, let's not argue over The Queen's English while The Queen is grieving!
Re: (Score:1)
Re: (Score:2)
Re: (Score:1)
Responsible parties-freedom. (Score:2)
This coming from a person who thinks people should be required to have a basic "operators license" to even access the internet.
Runs smack up against any kind of notion of "freedom" as well as Utopian dreams of what the internet is.
Given the quality of Exchange CUs (Score:4)
Given Windows has HTTP.SYS with userland support through IIS to allow port-sharing, it should be trivial to provide side-by-side installs with seamless handover to make patching an easy, automatic, zero-downtime affair.... what am I saying? This is Microsoft we're talking about!
Re: (Score:1)
FOSS equivalents
There are no FOSS equivalents to Exchange. There might be alternatives, but in no way, shape, or form are they equivalent. They do not have near the features and compatibility that Exchange has. You can argue that Exchange is shitty and that it's patch management is horrible and you would be correct. But let's not pretend that FOSS has anything that comes remotely close to the features of Exchange. If it did, Exchange would not exist.
Re: (Score:2)
Exchange links calendar functions to email functions effectively, which is its selling point. For SMTP, there are many excellent SMTP servers and have been for decades: it's why many if not most large scale services put a non-Exchange SMTP server in _front_ of Exchange, for security of a vital exposed service. For client access, there are many effective webmail services, and IMAP still works well.
Like Microsoft Word for documents, MS Exchange is used because it's a standard, not because it's the best.
Re: (Score:3)
That would cost a lot of money, and what do you gain from it? The mythical libertarian porn where if you punish enough people, everyone magically stops making mistakes or poor decisions?
Re: (Score:3, Insightful)
The mythical libertarian porn where if you punish enough people
You have a fundamentally unsound understanding of the term 'libertarian.'
Re: (Score:3)
I understand the ideology, but this is the way libertarians want it practiced. Because at the end of the day, it's a preference to holding people accountable from doing stupid things, not preventing them from doing stupid things in the first place because "freedoms"
Re: (Score:1)
Re: (Score:2)
These server operators are the victims of a crime.
Their "negligence" is installing Exchange.
Re: (Score:2)
Perhaps MS Exchange servers could be considered an "attractive nuisance"?
Re: (Score:1)
Re: (Score:1)
Microsoft makes money by fixing sloppy software. (Score:5, Interesting)
One of the reasons, over many years, has been that Microsoft would charge a full price for new versions of software. So Microsoft would make money for fixing former sloppiness.
That is my understanding.
Re: (Score:2)
There will always be bugs. Nobody can produce perfect software (especially not at an affordable price). So I can forgive Microsoft for the occasional bug and patch.
However, most of the reason people choose Microsoft is because of their popularity. People assume that popularity implies quality. If THIS is the product that most people use, then it must be best-in-class, otherwise, why would so many people use it? It is a classic bandwagon fallacy [wikipedia.org], and it allows Microsoft to keep its position of market do
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
To use another: Linux is like one of those Jeeps you used to buy as a kit.
The Jeep can be fixed. Repairing the Volare is inviting more misery.
Re: (Score:2)
There's also the problem that Exchange gives the company an all-in-one package for email, calendar, and a few other features. There are alternatives, but they're separate services, which presumably means higher maintenance costs - you're monitoring and patching 4 services instead of one.
That appeals to the money side of a company, and since that money side typically also under-staffs IT, it appeals to many overworked sysadmins.
Re: (Score:2)
Yep.
Those fools at Verizon just switched company-wide from Outlook to GMail.
Everyone hates the fuck out of GMail, but on the bright side, some C-level schmuck earned a bonus for crippling the company.
I've never been so glad to leave a place in all my years (and not just because of that).
Re: (Score:2)
I've seen quite a few companies discard Outlook in favor of Gmail. There have been complaints, but in general it's been far more reliable, effective, and scalable.
Office 365 has been trying to compete with Gmail, with mixed success. But not having to maintain Exchange servers is nearly _always_ a fiscal benefit for every size of organization. It's especially useful as a step to prying the Active Directory maintainers out of their niche.
Re: (Score:2)
The up and coming wave of employees is fine with GMail and a diverse if often incompatible mix of web delivered services.
This is the overlooked value proposition of eliminating Exchange, the howls of those used to its integrated features will eventually be drowned out by younger generations who just accept less integration and incompatibility as expected norms which serve the interests of feature diversification.
Re: (Score:2)
... will eventually be drowned out by younger generations who just accept less integration and incompatibility as expected norms which serve the interests of feature diversification.
Yeah, I'm not so sure about that. I doubt you could find a single VZ employee who didn't want to go back to Outlook/Exchange, regardless of age.
If anything, in my experience people expect more functionality and interoperability as time passes, not less. And I think that's true regardless of age.
I can state for a fact that dislike of the Outlook-to-Gmail change was met with universal condemnation regardless of age. No one, literally no one said, "Yippee, fewer features and crappier integration, woo hoo!"
So,
Re: (Score:2)
It's not that popularity implies quality, it's that popularity implies more people know how to use it already and more other software/services will be designed to work with it.
Re: (Score:2)
Microsoft transformed Exchange in the 2013 version, mostly I think because they wanted the code to be cloud-focused, but since then they have done minimal improvement because they're trying to shove world+dog into Office 365 for perpetual rental income.
Sure, they keep "patching" Exchange as they find new problems, but they're not looking hard nor are they considering meaningful new improvements for the benefit of on premise installation. I expect them to continue to keep Exchange on premise around in its a
Great initiative, hoping it will be enough (Score:3, Informative)
When hackers get access, they tend to move laterally to consolidate their presence, ie increase their access with multiple backdoors, spread to other hosts, etc.
In my experience, many times, simply removing a single backdoor is not enough to secure the host. It's difficult to trust a host once it has been compromised once
I hope they were doing more than just removing that piece of malware or the hackers might be able to get right back in
Re: (Score:1)
Ah, not to put too fine a point on it, but what they were actually doing here, is removing the competition.
Your concerns regarding overall system integrity? Weeell...
Re: Great initiative, hoping it will be enough (Score:1)
ON PREMISES.
Premises is the word garndammit. Premises is singular and plural. It's like "fish"
Re: (Score:2)
Presumably, it would be up to the sysadmins of the victims to do a more thorough cleaning. Not the FBI.
Re: Great initiative, hoping it will be enough (Score:2)
Reported that alerts went out to the community, MS offered the patch, but these particular servers were never upgraded/fixed.
It was the failure of the sysadmins to patch their servers that led the FBI to resort to this extreme act by the FBI.
Re: (Score:2)
Re: (Score:2)
FTA, the FBI supposedly sent emails to the impacted parties. No telling if those emails were ever read but the effort to communicate was made.
More to the point, if there are still unpatched Exchange servers connected to the Internet then I doubt that the so-called admins of those servers have any situational awareness or skillsets needed to remediate whatever post-exploit activities the "bad actors" performed on their networks.
Instead of this half-assed measure by the Fan Belt Inspectors I would argue that
Re: (Score:2)
I'm curious how they determined which e-mail address to send a message. Is there a service in Exchange that retrieves an admin e-mail address for that server?
Re: (Score:2)
I'm curious how they determined which e-mail address to send a message. Is there a service in Exchange that retrieves an admin e-mail address for that server?
Well, the current SMTP RFC (5321) requires a "postmaster" address to be defined and accepting mail: https://tools.ietf.org/html/rf... [ietf.org]
Seems pretty trivial to use reverse DNS to find a domain name and prepend "postmaster@" to it.
I know that the RFC texts are pretty dry but if someone is paying you to run a MTA you should at least be aware of the basics of the job.
Re: (Score:2)
Most "postmaster" and "abuse" addresses are not ever read or acted on, much like the FBI accepting reports on spam or fraudulent calls.
Re: (Score:2)
I did some remediation on Hafnium hacked Exchange servers and nearly all of them showed no persistence or lateral movement at all.
It was kind of baffling considering the broad success of the exploit. The theories we dreamed up were:
That nobody expected it to work that well, it was like a pop machine that spits out not just one free can, but 100, and you don't have a container for all of them. There were just too many exploited servers to capitalize on most of them.
The other one is that this was a programm
Re: (Score:2)
Interesting insight, thanks for sharing. Assuming nothing was done being beyond installing the initial shell which was detected, I guess it is one less thing to worry about albeit the fact the "door is technically open", ie the servers could in theory still be vulnerable
With that said some of the oddballs I have encountered replace compromised utilities/libraries to hide running processes and even had specific routines to evade sandbox detection. In one case, one admin I know thought he kicked out the intru
FBI shouldn't do that. (Score:1)
Re:FBI shouldn't do that. (Score:4, Insightful)
Re: (Score:2)
Re: (Score:2)
If you care enough to apply for a warrant and you establish your credibility to the judge and then do only what the warrant authorizes you to do, then sure, you can fix hacked servers too. You can't be charged with unauthorized access when you get your access authorized first.
Hang On... Legal Precedent... (Score:4, Interesting)
The relevant law in this case would most probably be the Computer Fraud and Abuse Act [wikipedia.org], for which the criminal offences under the act begin as follows:-
"(a) Whoever—
(1) having knowingly accessed a computer without authorization...
So now the question becomes, did the order issued to the FBI by a Texas court convey sufficient authority to supersede the provisions of the CFAA? Which has the greater authority and jurisdiction, a Texas Court or a Federal Law?
Reading through the paper trail and documents linked to the articles, it sort-of looks as though the authorization was granted by the Southern District of Texas. Does the Southern District of Texas have the authority to make a ruling with national impact? Does it have the authority to make a ruling that is binding on other states?
Whilst the Department of Justice clearly thinks it does, I'd have to say that it certainly shouldn't.
Even more interestingly, if you read the PDF of the partially un-sealed warrant [justice.gov], it is clearly and unambiguously written as a search warrant and not as a blanket approval to make changes to the impacted servers.
The now-granted request includes the following specific language,
"Therefore, I make this affidavit in support of an application for a warrant under Federal Rule of Criminal Procedure 41(b)(6)(B) to use remote access techniques to search certain Microsoft Exchange Servers located in the United States, further identified in Attachment A, and to seize or copy electronically stored information that constitutes evidence and/or instrumentalities of unauthorized access and damage to protected computers, further described in Attachment B."
See anything in there which says that the FBI are seeking the authority to modify the machines they want to search? No. Me neither.
This is Fourth Amendment territory. Let them show that Texas has the authority to grant the right to search nationally. Let them show that the granted motion included an approved request to modify systems so searched. Let them show that the Texas Court has the authority to over-rule standing legislation.
As the old but important saying goes, "The law may upset reason, but reason may not upset the law."
Whether the motives in this case were honorable or not, this stinks.
Re: (Score:1)
You may want to read up on the US Federal Court system.
https://www.justice.gov/usao/justice-101/federal-courts
The Southern District of Texas is a court that deals with issues of US federal law, not Texas laws.
Re:Hang On... Legal Precedent... (Score:4, Informative)
Someone who was making a good-faith inquiry as to what statutory authority under which this action was was done might have looked at the partially unsealed warrant that they themselves linked to to see what statutory authority the warrant requestor claimed.
For those who thought ytene was asking in good faith, the relevant statutory authority claimed is in the section "Statutory Authority". Paragraph 7, immediately after the heading, claims "7.Federal Rule of Criminal Procedure 41(b)(6)(B) provides that “a magistrate judge with authority in any district where activities related to a crime may have occurred has authority to issue a warrant to use remote access to search electronic storage media and to seize or copy electronically stored information located within or outside that district if . . . (B) in an investigation of a violation of 18 U.S.C. 1030(a)(5), the media are protected computers that have been damaged without authorization and are located in five or more districts.”"
A seizure as distinct from a copy would not leave the material behind, thus the warrant requestor can claim statutory authority to request and grant a warrant with the end result that the web shells are removed.
Secondly, someone who was making a good-faith inquiry as to what statutory authority under which this action was taken would not have claimed "Does the Southern District of Texas have the authority to make a ruling with national impact? Does it have the authority to make a ruling that is binding on other states?...I'd have to say that it certainly shouldn't." when the same paragraph 7 from the partially unsealed warrant that they themselves link to states "...a magistrate judge with authority in any district where activities related to a crime may have occurred has authority to issue a warrant to use remote access to search electronic storage media and to seize or copy electronically stored information located within or outside that district..."
Thirdly, someone who was making a good-faith inquiry as to what statutory authority under which this action was taken would not have claimed that the warrant requestor did not request this action, to wit "See anything in there which says that the FBI are seeking the authority to modify the machines they want to search? No. Me neither. ", when the partially unsealed warrant that they themselves link states, in paragraph 20, "This warrant authorizes the United States to seize and copy from Microsoft Exchange Servers located in the United States the web shells identified in Attachment A, and to delete the web shells from those servers. "
Lastly, someone who was making a good-faith inquiry as to what statutory authority under which this action was was done might have titled that inquiry "Hang on... Statutory Authority" instead of titling it "Hang on... Legal Precedent" when they were not inquiring about legal precedent at all.
Re: (Score:3)
A seizure of computer material usually involves removing the physical computers, not modifying the information therein, and given that it's criminal evidence, they use things like write blocks to avoid accidental modification and preserve the chain of custody to avoid spoilation.
This is weirder than you're making it out to be. I want to get rid of the hacked web shells too, but I'm not convinced that giving the FBI the right to hack random servers is the right way to go about this and inasmuch as they're g
Re: Hang On... Legal Precedent... (Score:1)
Re: (Score:2)
"This is weirder than you're making it out to be. "
I make no claims as to whether it is good or bad that such statutory authority exists. I am refuting ytene's claims that no reference to statutory authority was used in the warrant request.
"inasmuch as they're going around modifying servers, it should complicate any prosecutions based on this evidence."
Note that in Attachment A they had to establish an exact list of servers that would be affected, thus they must already have some evidence to identify those
Re: Hang On... Legal Precedent... (Score:1)
Article is slightly wrong (Score:2)
In its warrant application, dated April 13...
The warrant was issued and carried out earlier this year. On April 13th the US Attorney petitioned the court to unseal the warrant because the action had been concluded.
FBI should take over Windows patching. (Score:4, Funny)
Re: (Score:2)
I drive a 10+ year old car. Should I be forced to buy a newer car with all the bells and whistles? Should any car which is an antique be forbidden to drive on the road? Perhaps we should have a law which says you must replace any piece of machinery, computer, etc every four years. Would that be acceptable?
Yes, completely acceptable. Good idea you have there.
Re: (Score:2)
If your car is a significant danger to other people, due to poor maintenance or emissions, then you will be forbidden to drive it. Been that way for ages.
Now, if instead of taking away my car when it fails inspection the FBI would like to come over and repair it for me for free, I will welcome that.
Re: (Score:2)
if instead of taking away my car when it fails inspection the FBI would like to come over and repair it for me for free, I will welcome that.
Is it still welcomed if they install a GPS tracker and a cabin microphone at the same time?
Re: FBI should take over Windows patching. (Score:2)
Now, if instead of taking away my car when it fails inspection the FBI would like to come over and repair it for me for free, I will welcome that.
Where the fuck do you live that the government seizes your car if/when it fails inspection? Here in Texas you simply fail inspection and are free to fix it, when I lived in NJ they would put a failed inspection sticker and send you on your way.
Re: FBI should take over Windows patching. (Score:2)
FFS Microsoft released a patch to both remove and correct operation of server, these admins failed to/choose not to apply those patches to their servers.
Captain America (Score:1)
I'm pretty sure that I'd like Captain America to stay the fuck out of my servers. Maybe I'm a researcher. Maybe I'm running a honeypot. Maybe as a private citizen what I run is my own business and not the business of Captain America.
Good thing I'm not actually running Exchange.
Re: (Score:1)
If the server you are running happens to be hosting malware/hackers then it certainly *is* the business of Captain America. Because you do *not* have the right to do whatever you want when that may be harmful to others.
CISA (Score:2)
Good. My only complaint would be that CISA should have been the one give permission by the court, not the FBI.
Nice move, but they should go for the jugular (Score:1)
As in, charge Microsoft for their time and effort fixing a problem that should never exist. There is precedent I believe in that PDâ(TM)s can charge people for such as persistent nuisance calls to recover costs.
Re: Nice move, but they should go for the jugular (Score:2)
MS released a patch to correct it, these sysadmins chose not to patch it.
FBI competence being legendary (Score:1)
Microsoft cannot fix Exchange server (Score:1)
WTF? (Score:3)