Server Snafu Makes Microsoft Beg For CA Audit Data From Its Partners (softpedia.com) 115
An anonymous reader writes: Microsoft, just like Google, Apple, and Mozilla, is part of the CA/BForum, an organization of web browser vendors and certification authorities (CAs). As a browser vendor, Microsoft maintains a list of authorized CAs and their respective root certificates. According to a message on the CA/BForum, there was an error on the server that was running a CRM application that managed this list of trusted certificates and the adjacent details regarding each certificate and CA. The data is lost forever and Microsoft is now asking CAs to resend their most recent audits. Currently a lot of certs are broken in Edge and IE. Microsoft says that it lost audit data for 147 root certificates, which resulted in many SSL/TLS certificates showing errors inside the company's products.
wtf (Score:5, Interesting)
Re:wtf (Score:5, Insightful)
Seriously. No backup?
Maybe they used Azure for their backup ...
Re:wtf (Score:5, Insightful)
This might be the correct explanation. I have seen the technology management to actually trust on their "the cloud is the backup" fairytale. And then we lost data multiple times thanks to software or administration errors which deleted the data from all replicates. After fourth data loss the dumb ass management started to plan a real write-only backup system. Thankfully I don''t work on that company anymore as the management is still there planning for their next failures.
Re:wtf (Score:4, Insightful)
There are fallbacks, backups, and disaster recovery mechanisms. They are three different things, with three different purposes, and managers love to confuse them.
Re: (Score:1)
All these technical terms confused Microsoft management, and it all cost more money, so they checked the 'no' box.
Re: wtf (Score:3)
Manually.
Backups should never be read by the server to ensure it has no dependency on the data.
Backup should never be overwritten by the server to protect the backup.
Backups should be independent verified for completeness because servers and engineers do unexpected things.
I just made that up, but it sounds about right.
Re: (Score:3)
This bit doesn't sound right:
Backups should never be read by the server to ensure it has no dependency on the data.
If you never read your backups, how do you verify that the data was successfully backed up? I've seen dual-backup systems fail because, after several years of apparent backups, when the data was needed it turned out that nothing (copy #1) and the wrong data (copy #2) had been backed up.
Re: (Score:3, Informative)
Actually, what seems to have happened is that they _did_ have a backup. But, they had to roll back to an old one.
Re: (Score:2)
Actually, what seems to have happened is that they _did_ have a backup. But, they had to roll back to an old one.
Sounds like the excuse I'd give if I was worrying about keeping my job.
Re: (Score:1)
"Our CRM system suffered a data loss, and it looks like it rolled back to an old backup. As a result, we lost audit data for about 147 roots."
see: https://cabforum.org/pipermail... [cabforum.org]
Re:wtf (Score:4, Insightful)
... rolled back to an old backup. As a result, we lost audit data for about 147 roots.
How the fuck are there that many changes for root CA's withing the period of one backup?
Re: (Score:1)
These are audit records for public CA roots. Every one of these 147 is a public CA _root_ certificate. Not an intermediary, not a site certificate, not some bozo's SMIME cert, a public _root_ that every Windows user trusts to sign any non-EV certificate.
Some Certificate Authorities manage several CA roots, particularly the oldest ones because they issued their initial certs when nobody knew how any of this would work, there wasn't a CA/B to decide any rules, it was the Wild West. But even today a new CA mig
Re: (Score:1)
How the fuck are there that many changes for root CA's withing the period of one backup?
They rolled back to an old backup. Not necessarily the most recent. Perhaps all of the more recent backups were borked, and that was the most recent unborked backup...
Re: (Score:1)
The files may have been open so they weren't able to back them up so it wasn't within the period of one backup. DOS/Windows isn't like UNIX. You can't work with open files. That's why Windows has to crash completely for even minor updates.
Re: wtf (Score:2)
Re: (Score:1)
The files may have been open so they weren't able to back them up so it wasn't within the period of one backup. DOS/Windows isn't like UNIX. You can't work with open files. That's why Windows has to crash completely for even minor updates.
I am not a real Windows Admin, but that just isn't true. Modern backups of Windows servers take advantage of a snapshot-ting capability (I think it's called VSS) so that all files can be backed-up. I have no idea how it actually works, but I know that it does.
Re: (Score:2)
... rolled back to an old backup. As a result, we lost audit data for about 147 roots.
How the fuck are there that many changes for root CA's withing the period of one backup?
Edge is involved, Win10 is a different beast, if one has the proper certs they can bypass the windows firewall. Every since Windows supplied a firewall with their OS that's been the way it has worked.
Re: (Score:1)
... rolled back to an old backup. As a result, we lost audit data for about 147 roots.
How the fuck are there that many changes for root CA's withing the period of one backup?
Because they only backed up the system once, and then never actually started the backups running on their regular schedule, I'll bet.
Re: (Score:2)
"Our CRM system suffered a data loss, and it looks like we were too fucking stupid to have a recent backup."
Re: (Score:2)
It should be on their OneDrive... you know, in the cloud
Really, just how brittle is this "Internet"? And how will Microsoft verify these certificates? Hmmm?
Re:wtf (Score:4, Interesting)
It's Microsoft. Data loss from lack of backups is has happened to them before. [roughlydrafted.com] Unfortunately they didn't learn from past mistakes.
Re: (Score:2)
Seriously. No backup?
"We're Microsoft, trust us with your data, hurr durr."
Re: (Score:1)
Seriously. No backup?
I know. And this is the company that has been one of the most aggressive about pushing their products into "the Cloud".
Re: (Score:2)
Re: (Score:2)
Or the first and second-level managers were the ones who laid out an effective plan, and their subordinates (whom you'd promote) didn't bother to implement it correctly.
Perhaps it'd be better to investigate the whole situation first, rather than jump to any knee-jerk response.
Re: (Score:2)
Re: wtf (Score:2, Insightful)
Microsoft like many other tech companies has lots of problems with middle management. Good managers get promoted quickly to more senior roles because there's constant growth and new projects; this means that what's left in middle management ris mediocre lifers or total noobs who haven't shown their potential yet. It's a dead layer with zero potential for improvement unless the company goes stale like IBM. Promoting insiders to middle management doesn't fill the void, it accelerates the spiral.
This is one of
What a joke (Score:1)
I wonder if these are the same people making gui design decisions for windows 10.. I bet the same department head signs both teams' checks.
Re: (Score:2)
I wonder if these are the same people making gui design decisions for windows 10.. I bet the same department head signs both teams' checks.
They have 118,000 employees. Blaming them all is like blaming the army when you don't get your social security check.
Looking Back (Score:2)
I'd hate to be in the Retrospective meeting for THAT iteration.
You're supposed to deliver a releasable product, not release all your products (obscure Objective-C reference counting joke).
chrome (Score:3)
Re: (Score:2, Informative)
They aren't missing the certificate data, but rather the audit data associated with those certificates which is NOT stored with the certificates that are on computers everywhere. The summary is a bit misleading on that point.
Re: (Score:3)
stories such as this make me smirk but also check if my backups are working properly. they are. back to smirking.
but seriously, how often do people normally back up? my /home directory is on a NAS with ZFS and keeps 24 hourly snapshots, 7 daily snapshots, 4 weekly snapshots and 6 monthly ones. this gets automatically synced to my secondary (backup) NAS and once a week i manually sync it to a nas at my parents' house. i lost all my data in the late 90s and never want to go through that experience again.
Re: (Score:2)
but seriously, how often do people normally back up? my /home directory is on a NAS with ZFS and keeps 24 hourly snapshots, 7 daily snapshots, 4 weekly snapshots and 6 monthly ones. this gets automatically synced to my secondary (backup) NAS and once a week i manually sync it to a nas at my parents' house. i lost all my data in the late 90s and never want to go through that experience again.
Mine is very similar. I can roll back quite a way, and it has come in very helpful.
But the answer to your question is: Most regular people simply don't back up at all. And professional setups aren't always a whole lot better.
Re: (Score:3, Informative)
It a load of rubbish from the original author. There's no reason whatsoever that loss of this data would cause problems in IE or Edge. Removing roots from MS's program doesn't happen without human input.
Time to double check my own backups (Score:2)
If Microsoft can perpetrate something like this, I think I had better set aside some time to verify that I do not have omissions in my own backup and disaster recovery procedures.I cannot imagine having to report something like this to top management.
Re: (Score:2)
Please define the acronyms in the summaries so those of us who aren't experts in a particular topic can follow along.
This should be at a +5. My directors always stopped presenters at dry runs every time they made a Alphabet soup statement. All it takes is giving the letters, then what they stand for, and after that people follow it just fine. And in a multi- skillset place like /., its pretty helpful.
And be careful calling anything an acronym around here, the pedants will jump on you like crocodiles on a wildebeest. Then we'll have 50 posts on what an acronym is or isn't.
Re: (Score:2)
IT'S AN ACRONYM IF YOU PRONOUNCE IT AS A WORD, LIKE NASA
IT'S AN INITIALISM IF YOU READ THE LETTERS, LIKE CIA OR FBI
BLARGARGLARGLARGLARGLARGLARGLARGLARGLARGL!!!!!
Filter error: Don't use so many caps. It's like YELLING. Filter error: Don't use so many caps. It's like YELLING. Filter error: Don't use so many caps. It's like YELLING.
Ya gotta stop after the third espresso!
Re: (Score:2)
Double click on the word - this will highlight it. Right click on the word and select search, this will open search in a new browser tab. You can even set up a variety of search engines as you go, they'll make it much easier for you.
If you don't mind my asking... (Score:2)
How many root certificates does Microsoft hold and how long did it take to recover the 147 that were lost? Tech news posted to Slashdot tends to be a little skeletal and runs on the principle of "better late than never."
Microsoft says that it lost audit data for 147 root certificates, which resulted in many SSL/TLS certificates showing errors inside the company's products.
I am curious as well about how often these certificates change. How old a backup is too old?
Re: (Score:1)
Due to a weird design decision, Windows management tools only show currently cached root certificates, not the full list (currently 343).
More info at
http://hexatomium.github.io/2015/08/29/why-is-windows/
http://trax.x10.mx/apps.html
Re: (Score:1)
How many root certificates does Microsoft hold and how long did it take to recover the 147 that were lost? Tech news posted to Slashdot tends to be a little skeletal and runs on the principle of "better late than never."
343 total, and they're required to be audited annually. It doesn't take a mathematician to see how old thier tarball was!
How long.. (Score:1)
Passive voice to the rescue (Score:4, Funny)
Re: (Score:2)
automated since win10
The realy SNAFU ist another one. (Score:1)
A system crashing and having to restore from an "older" backup is something that could happen to almost anybody.
The one thing that got me in the article:
"As many of you may have just noticed, our system just generated a bunch of emails informing many of you that you are subject to removal because Microsoft does not have evidence of a qualifying audit on file,"
And that they then asked them to re-send the data....
1) If I restore from an older backup, and know I may have (for example) lost payment data, I don'
Re: (Score:1)
No, not if the system handles something really important (and/or highly visible like this). A system will occationally break, so you use sufficient redundancy. RAID avoids loss from disk breakage. Backups avoid loss from destruction of complete systems (fire) or griveous admin mistakes. (delete wrong database...) Logging transactions on another server makes sure you don't loose what happened between the last backup and the disaster.
I do all that. But in the event that a plane crashes right between our two server rooms which are ~500 metres apart (thus loosing all the RAID and Online-replication backups) I might still have to go back to an off-site backup, where the transaction log replication happens only every 10 minutes, so the backup might be "10 minutes old" in that case.
Which would prompt me to start up the system (that is, after I somehow got hold of new hardware, and if me and my co-workers didn't go up in the same ball of fire
Comedy of errors (Score:2)
exaggerated FUD (Score:2)
Where is the evidence of any SSL/TLS certificates showing errors? Seems like total conjecture based on poor reading of this audit data request made by Microsoft.
This is AUDIT data, not the actual cert info. Read the details of the audit requirements here: http://social.technet.microsoft.com/wiki/contents/articles/31635.microsoft-trusted-root-certificate-program-audit-requirements.aspx
This just means that Microsoft lost the documentation showing that the Certificate Authorities had performed their annual aud
Re: (Score:2)
Everybody seems to know what you're talking about, but I've got no idea. Was is spam e-mail or what? (Or was it actually a Bellevue exercise studio? The first page of a Google search didn't list that, and I'd think it would.)