Mozilla To Track Infrastructure Time-Bombs in Wake of Recent Firefox Armagadd-on (zdnet.com) 123
In the wake of the mass disablement of Mozilla Firefox's add-on ecosystem last weekend, Mozilla has committed to improving its asset tracking and developing a mechanism that can quickly push updates to users when needed. From a report: Due to an intermediate certificate expiring on May 4 at 1AM UTC, users found their browser add-ons were switched off and could not be re-enabled. Thanks to timezones and the rotation of the planet, users on the western side of the Pacific were the first hit. Writing in a blog post, Firefox CTO Eric Rescorla detailed some initial thoughts and announced a formal post-mortem would be published next week. "First, we should have a much better way of tracking the status of everything in Firefox that is a potential time bomb and making sure that we don't find ourselves in a situation where one goes off unexpectedly. We're still working out the details here, but at minimum we need to inventory everything of this nature," Rescorla wrote. "Second, we need a mechanism to be able to quickly push updates to our users even when -- especially when -- everything else is down.
Why would a UTC be timezone dependant? (Score:4, Interesting)
Isn't the point of a UCT Time stamp, is that effected at the same point of time (minus speed of light, and gravitational differences) everywhere.
Re:Why would a UTC be timezone dependant? (Score:5, Informative)
Re: (Score:2)
Where were they after that though?
Re: (Score:1)
It's almost like people think you can renew certs instantly.
Even if you have new ones queued up and ready to go, if your infrastructure has any complexity at all, or rather if it has any isolation (redundancy) at all, it's going to take a while. Somewhere OTOO an hour to a few hours.
That's assuming you immediately identify the problem as a bad cert or certs, and that may take most of an hour or perhaps even longer depending how indirect the root causing path is.
Re: Why would a UTC be timezone dependant? (Score:1, Informative)
Are you serious? Do you even understand the concept of time zones?
Of course the certificate expired at the same time globally. The problem is that it happened at different local times. For some users it was at night or early in the morning. For others it was mid-day. In California, where many of Firefox's developers reside, it would have been the early evening.
It's harder to fix problems that arise in the evening or at night. You have to start locating people who are busy doing something else (if in Califor
Re: (Score:2)
Maybe just keep track of expiration dates? (Score:5, Insightful)
Seems to me that the expiration of the certificate that broke many extensions should have by default monitored and renewed long before a crisis. Yeah its great to find a quick push for emergency updates, but how about doing better to prevent them in the first place. That would be priority one in my book, just saying Mozilla/Firefox.
Re:Maybe just keep track of expiration dates? (Score:4, Insightful)
Seems to me that the expiration of the certificate that broke many extensions should have by default monitored and renewed long before a crisis.
Basic professionalism. Heck, I'd look down on a hobby project that screwed up something this simple.
However, it's nice to see they're taking the mistake seriously, rather than downplaying it. Seems they got their wake-up call.
Re: (Score:2)
Re:Maybe just keep track of expiration dates? (Score:5, Insightful)
Re:Maybe just keep track of expiration dates? (Score:4, Informative)
You're basically saying that you expect everyone to be perfect all the time.
Professionalism in software is making sure things work despite human failings. You don't rely on people just happening to remember to do something important! If practical, you automate it. If not, you automate adding a team calendar entry to remind people.
The reality is that people leave, email addresses change, systems are re-provisioned and eventually its going to get missed.
Sure, if your hobby project isn't one you put much effort into. But if you're going to make a professional effort, not just messing around to kill time, you get on top of certs from the beginning, just like you figure out logging and test automation and deployment and your security model and all those boring things.
Re:Maybe just keep track of expiration dates? (Score:4, Informative)
Professionalism in software is making sure things work despite human failings. You don't rely on people just happening to remember to do something important! If practical, you automate it.
Root certificates are kept on disconnected systems. This isn't automatable
If not, you automate adding a team calendar entry to remind people.
Sure, if your hobby project isn't one you put much effort into. But if you're going to make a professional effort, not just messing around to kill time, you get on top of certs from the beginning, just like you figure out logging and test automation and deployment and your security model and all those boring things.
This isn't Mozilla's first root certificate, you can safely assume they've had some process of reminders that worked to this point.One can assume they'll fix this problem, but eventually something will breakdown and in some years something else will break.
Re: (Score:1)
Good engineers don't let important things break. Even if that's hard. They rest is excuses for failure. If Mozilla can screw up something this obvious, I really have to wonder about the security of FF. Sadly there's no practical alternative for banking etc. that's not Google-tainted, but this doesn't exactly fill me with confidence.
Re: (Score:1)
Re: (Score:2, Insightful)
Name one thing that has never or could never break.
The anti-lock brake system controller in your car. If it breaks it doesn't brake, and if it brakes it doesn't break. Obviously.
But, seriously, do you really think engineers for somehting important should have the attitude of "mistakes happen, what can you do?" For the bridge you drive over? For MCAS? For the elevator you ride in? For the next x-ray machine you use? [wikipedia.org]
The point is, you make a professional effort to do important things right. Keeping a root cert renewed is important, and is a very simple
The calendar system they'll buy in three years (Score:4, Insightful)
Certainly they SHOULD have found a way to avoid this type of problem. And expiring certificates have been a problem for many organizations, including Microsoft. Why?
The problem with "automate adding a calendar entry to remind people" is that three or five years down the road, you may very well be using a completely different calendar system. You had better be using a new version, which likely won't work quite the same. Whose calendar, by the way? It's entirely possible that the team that handles certs today won't exist, especially under the same name, a few years from now.
Absolutely organizations need a way to ensure that several years later, an entirely different group of people, in a re-structured organization, perform a task. That's actually non-trivial though.
Consider just with the example of Mozilla, at one point they would have needed to renew / exchange certificates that Netscape bought. How would a Netscape ops person add a calendar entry for the Mozilla foundation to do something before the foundation even existed?
Re: (Score:2)
The problem with "automate adding a calendar entry to remind people" is that three or five years down the road, you may very well be using a completely different calendar system.
So you forsaw a problem. That's great. Do you need me to explain how you mitigate the problem? Like, when your calendar-interaction automation breaks every time it runs, since it can't talk to the new system, how you know it's on fire? Monitoring? Alarming? Basic professionalism?
It's entirely possible that the team that handles certs today won't exist, especially under the same name, a few years from now.
So you forsaw a problem. That's great. I bet you can think of a way to mitigate it, if you stop explaining why it can't be done and instead just do it.
Absolutely organizations need a way to ensure that several years later, an entirely different group of people, in a re-structured organization, perform a task. That's actually non-trivial though
Never claimed it was trivial. Engineers get paid to solve the non-trivia
Re: (Score:2)
I didn't say it can't be done, I said a number of well-staffed, generally professional organizations have missed it.
> When the Tower of London locks up at night, there's a ceremony every night involving 4 people and an elaborate call-and-response ritual.
If you do it every night, it's not hard to remember (especially of you have four people doing it). If it only needs to be done once every five years, it's a lot harder to train the new guy, who won't be hired for another three years.
Not impossible, but t
Re: (Score:2)
Security Certs in seem like a sloppy process.
They don't really do that great of a job proving you are who you say you are, and they are relativity expensive to implement, and when they fail, it breaks down a whole slue of things.
Re: (Score:2)
In YOUR countries' english.
Re: (Score:2)
This is assuming someone who was responsible didn't think this was a great way to finally force all those pre-quantum holdouts to upgrade.
Re: (Score:1)
No, it's not great to allow a push upgrade. No. No. No.
The upgrade should be a pull by the user. It's fine to tell the user that they need to upgrade, but doing it behind their back is not only itself nefarious, it's an invitation to even more nefarious deeds.
Re: (Score:2)
People and companies do. In general, everyone is aware of the issue. It's surprisingly hard to get right all of the time. At scale, due to the number and variety of certificates and use cases, it's one of these "sanitation" tasks that requires meticulous attention. No matter how hard you try, at a certain size, you'll start having embarrassing outages due to expirations anyway, due to weird intersections of causes. Hopefully not many.
Re: (Score:2)
"addon armageddon" seems to retain pizazz (superfluous or not) while being less retarded
Mozilla doesn't like intermediates anyways (Score:1)
We use Venafi where I work and our one browser that refuses to play nice is Firefox. They never accept the Venafi intermediate and I have been told that's by design.
Entertaining how that same thing has bit them in the butt!
Re: (Score:2)
Straw man argument.
I was talking about Mozilla not accepting internal intermediate certificates by design. The other browsers get it. Firefox is "differently developmentally abled" in this department.
Well, I'd hope so. (Score:5, Insightful)
Re: (Score:3)
Re: (Score:1)
Too busy playing around (Score:4, Insightful)
Re: (Score:1)
You're free to move to one of the forks if you don't like it.
Alas, that won't stop you from bitching about it in forums like this.
Re: (Score:1)
Yeah! When the people who are now in charge of a project are running it into the ground, making asinine decisions, and have lost market share to the point of the project being irrelevant you need to shut up and like it. Never mind that you may have been a contributor, or financial supporter, or just a devoted fan of the project.
"Move on to a fork" is a stupid retort on so many levels. What do you suggest people do when the project decision makers are failing in such a dramatic fashion? Let them? There shou
I thought it was a feature... (Score:2)
And here I naively thought they had found a way to disable Pocket.
Perhaps the wrong message? (Score:5, Insightful)
The real question is why should unmodified plugins that had a valid certificate chain at the time of installation become invalid later?
The plugin didn't change. Nothing changed except the passage of time.
Re:Perhaps the wrong message? (Score:4, Interesting)
The real question is why should unmodified plugins that had a valid certificate chain at the time of installation become invalid later?
Simple, programs have no sense of time. Specifically, it runs and exits repeatedly and is completely unaware of modifications made to the profile.
This could be exploited by publishing a malware extension with the expectation that it will be quickly revoked. Once you have a valid signature, you can forcibly install your extension using a different piece of malware and replacing the central cert with an expired one. The result of Firefox not checking at every startup would be that it would allow the malware extension to be enabled.
Re: (Score:1)
You have absolutely no idea how code signing works [sslshopper.com], do you?
Not that Mozilla does either in this case, but you seem to be even more lost on the subject. I'm not even sure you understood the OP's original statement.
Re: (Score:2, Interesting)
At a guess: because Firefox needs to validate the addon when it starts, in case it was modified by some malware while firefox wasn't running?
Re: (Score:3)
The real question is why should unmodified plugins that had a valid certificate chain at the time of installation become invalid later?
Wouldn't such a record of already installed, no need to check plug-ins be a prime target for hackers? Install malware, flip the "you already checked me" bit and you're in the clear. This way you have to circumvent or disable the whole system to get around it.
Re: (Score:1)
And the malware will replace the hash... then why not sign the hash... uhm, but then the private key should not be on the PC*... Ah, I get it... The private key should be on a HW signing module kept offline at Mozilla, that is used to sign an intermediate certificate, that is used to sign that add-on.
* The problem initially was that while installing another SW (as Admin) an hostile add-on was also installed.
Re: (Score:1)
Re: (Score:2)
If you're suggesting that the malware is able to modify Firefox configuration in general somehow before installing the malware-extension, then I'm betting it is able to do many worse things than simply setting some particular flag in Firefox configuration. For example replace the Firefox links in desktop with a malware-provided one.
sooo... (Score:5, Insightful)
Re: (Score:1)
Re: (Score:2)
Every day that goes by, we edge closer to the idea that when software is "unsupported", it will die instantly and never, ever work again. Because software goes bad over time, of course.
Re: (Score:1)
It removed my add-ons too. I've been a FF user since beta but with the other add-on breakages and the stupid ass "studies" they like to sneak in I'm about to jump ship, which makes me sad.
I don't have much faith (Score:2)
...a mechanism to be able to quickly push updates (Score:2)
"Second, we need a mechanism to be able to quickly push updates to our users even when -- especially when -- everything else is down."
Sigh, yet another thing that will I'm sure be on by default, with either no way or a very difficult way to opt out of, and gives Mozilla persistent control into your browser way past the initial download. Why can't they learn, once I download the package I should be able to do whatever I want with it, for as long as I want, without ever having to talk to their servers again.
"And" ? (Score:2)
Thanks to timezones and the rotation of the planet, ...
Someone doesn't understand how Time Zones work.
The real question is ... (Score:2)
A little late... (Score:2)
Cert expiration tracking is for real ... (Score:2)
I work at a certain well known tech company where certs have 1 year expirations and your dashboards better contain tracking metrics along with alarms lest you be embarrassed during a presentation, or worse, have to present a discussion of an event due to an expiration.
There are a whole lot of certs out there, and there are many reasons to change/renew them, and they live in many different environments. Some can be automagically renewed by automated systems. Some not so much. Some have to be updated on one h
BS problem caused by verisign (Score:1)
Verisign wanted re-occuring income every year, so they limited their certificates to 1 year. All the normal BS arguments came out - this is reasonable, you can do this it's good for a whole year, etc. Try maintaining 3 thousand or more sites. Now it's a chore every day. Especially since people depend on those sites and don't want it to be down for even a minute. Coordination, etc. Then if someone screws it up? Oh my God. The complaints come in.
Real world complaint - "Hillary was so far up my ass she spray p
Firefox ain't done till Dissenter won't run (Score:2)
It's the modern-day version of "DOS ain't done till Lotus won't run". If the certificates had expired 6 months ago, this would not have happened. The series of events goes like so...
* Various websites moderate non-leftist comments out of existence, or shut down comments altogether, or point comments to Facebook forums, or Disqus.
* Free speech website gab.com puts out a browser extension that provides a comment functionality similar to Disqus. This was popular with people who were banned from Facebook and Tw
Re: (Score:2)
Lol the Firefox PC police are out in force tonight.
Re: (Score:2)
It's complete validation of our statements.
Truth hurts. I hate to see Mozilla die like this. They were my first browser.
God Speed Netscape.
Re: (Score:2)
Their new logo is such trash. I like this Ars Technica forum member's suggested logo more than any of the trash Mozilla came up with: m-scape [harmless.de]