IE9, FF4 Beta In Real-World Use Face-Off 358
An anonymous reader writes "Most browser benchmarks are isolated, artificial tests that can be gamed by browser vendors optimizing those specific cases. With only those benchmarks to go on, the folks at LucidChart were skeptical that the IE9 beta would actually outperform other modern browsers in real-world applications. To separate hype from reality, they built their first browser benchmarking tool, based in LucidChart itself. This benchmark is to SunSpider what a Left4Dead 2 benchmark is to 3Dmark Vantage. Product specs don't matter, only real-world performance on a real-world application. The results were surprising. IE9 held its own pretty well (with a few caveats), and the latest Firefox 4 beta came in dead last."
Frames per Second? (Score:2, Insightful)
Frames per second seems like pretty much the opposite of "real-world" for how 99% of users use their browsers.
Speaking as someone that switched to OS X (Score:4, Insightful)
from Linux this month after using Linux since 1993, I think this applies to all of FOSS.
Somehwere around 2000-2006 FOSS was basically head-and-head with commercial software in practical usability and maintainability, with its own distinct advantages and a relatively small learning curve.
Then there was this veer into "if you ever want all the Windows users to switch..." thinking, and in an effort to eliminate the learning curve FOSS threw away pretty much all of its advantages as well. If FOSS is just Windows/Mac OS/IE by another name, why choose FOSS?
Particularly when Windows/Mac OS/IE win on the polish, compatibility, and accessibility fronts by virtue of their being cathedral-built software?
With Firefox slow and cumbersome, Thunderbird choking on Gmail IMAP continuously while Apple's Mail.app sails along happily, and KDE4/GNOME3 being emblematic of the many ways in which FOSS has lost its way, I just decided I'd had enough of the nonsense. I'm ready to be able to walk into Best Buy, purchase any device, and expect that it will work seamlessly with the current generation of computing devices, without options, without Bugzilla (and condescendingly dismissive developer retorts), and without lots of consulting Google to find out how the gconf infrastructure has changed in the last two years or how HAL has been replaced by DeviceKit or policies moved from /etc tree A to uneditable dynamic filesystem B (but just use this easy command line management tool to set options...)
It just plain saves me a boatload of time and headache to use something else, like OS X plus Google apps plus Chrome. The pending desktopization of FOSS has fizzled thanks to the politics of the bazaar.
"Real World"? (Score:5, Insightful)
For me, "real-world" means: is gmail fast enough? is opening a new tab fast? is image rendering fast enough? is html video fast enough? is the occasional embellished html5 animation fast enough? is typing into the address bar fast enough?
I'm sure their diagramming app is cool and everything, but I don't think I've ever seen anyone use anything like it, so I'm not sure what is "real world" about using it for a benchmark.
They even said that they altered the test in the middle to fix IE's performance problem. Come on.
Figure of Speech (Score:2, Insightful)
I love how a difference of a few milliseconds (looks to be 5ms) means a browser "tanks" and its position, when compared to other browsers, can be described as "dead last." Oh no, we're not painting a bias picture here.
Re:Speaking as someone that switched to OS X (Score:5, Insightful)
Hear, hear. I'm right there with you.
As an example along similar lines, a user at my office reported a bug in Thunderbird to me. I tested and found it was definitely a Thunderbird bug. I made a test case file and submitted it to Bugzilla. A few days later my reported bug is deleted, to be merged with the same bug report from *2005*
Nobody who works on Thunderbird felt like working on the bug. It's not a sexy bug, probably doesn't hit too many people, and has work arounds...so it's stayed in the software for ~6 years.
And yeah I know, I should go in and fix it myself. Maybe one day I will. In the meanwhile I'll keep using Mail.app and I'll move more users over to new versions of Outlook that actually seem somewhat decent, and we'll go from there.
Is there any quality email app for Windows??
Broken time measurements of the inter-frame time (Score:5, Insightful)
The way this benchmark measures "intra-frame time" is broken. In particular, it uses a setInterval with a 1ms delay. No browser actually respects that 1ms. Chrome clamps it to 5ms; others clamp it to 10ms, all to avoid the website thrashing the CPU pointlessly.
The upshot is that Chrome's interframe delay in the graph is about 5ms and Firefox 3.6's interframe delay is aboug 10ms. Which this particular benchmark can't tell apart from "no delay at all", given its methodology.
Firefox 4 beta, IE9 beta, Safari, and Opera seem to have delays greater than 10ms, so they're clearly doing some work they can't finish in 10ms.... or have slightly buggy timer implementations. Or both.
Of course in practice frame rates above 60fps or so are pointless since the screen doesn't redraw that often. ;)
On the other hand, on Mac, on modern hardware, I get 4.5fps in Chrome 7 dev on a random trial document I just tried, with JS render tiems on the order of 7ms (with a 7ms standard deviation) and "intra-frame time" of 224ms with a 900ms standard deviation (yes, those numbers are nuts). Firefox 4 beta comes in at about 11s for the JS (with 3ms stddev) and 125ms for the "intra-frame time" (with a claimed stddev of 0, which looks really suspicious).
It'd be nice if there were non-obfuscated source for this benchmark so its number-crunching could be evaluated; that 0 stddev is ... highly improbable.
Re:dev IE9 and dev FF vs release Chrome? (Score:5, Insightful)
Me too.
Its odd how both the summary above and the linked article sort of over look the fact that Chrome just blew the doors off of every other browser and the compared the production version to the latest and greatest of the others.
Chrome really deserves top billing, but the story is about the who is going to come in dead last.
Yawn.
Bullshit Slashvertisment (Score:5, Insightful)
This benchmark can be run by anyone in LucidChart. First, sign up for a free account here.
Nuff said
Re:"Real World"? (Score:2, Insightful)
For me, "real-world" means: is gmail fast enough? is opening a new tab fast? is image rendering fast enough? is html video fast enough? is the occasional embellished html5 animation fast enough? is typing into the address bar fast enough?.
I use gmail in basic HTML mode. It takes away some of the things I never use and, in my opinion looks much cooler (none of that modern looking nonsense). I also barely ever type into the address bar.
Odd definition of "dead last." (Score:5, Insightful)
Re:dev IE9 and dev FF vs release Chrome? (Score:5, Insightful)
Re:dev IE9 and dev FF vs release Chrome? (Score:5, Insightful)
They forever screwed themselves because they used to charge people to use it. When there's a ton of competition, and it's free? Yeah.
So did Netscape, which eventually turned into Firefox. I don't think that has anything to do with it. Opera has great technology, and no understanding of what most users want in a (default) user interface. They also seem to have no clue that most users never change default settings, so it doesn't matter how configurable it is.
Re:Speaking as someone that switched to OS X (Score:5, Insightful)
How have Apple and Microsoft handled your bug reports for Mail.app and Outlook? Did the handle them like Mozilla, where you enter the bug directly in their internal bug databases, monitor the progress, participate in discussions with the developers, and even contribute development yourself? Or do you have no idea what the status is, no influence on the outcome, and no ability to contribute at all? Were the bugs even submitted to development? Were you able to find a way to submit them to Apple and Microsoft at all -- could you communicate with anyone beyond level 1 end user support technicians?
Every application has bugs as old as its first release-- have you seen the age of some Windows security vulnerabilities, going back over a decade? -- and your particular concern won't necessarily get fixed. But if you compare the experience of handling end user bugs at Mozilla with the same thing at Apple or Microsoft, well, there really is no comparison.
Re:dev IE9 and dev FF vs release Chrome? (Score:4, Insightful)
Absolutely. I find it humorous how I will occasionally hear coworkers cursing:
1) The speed of their browsers. "Render, god damn it!" echos down the halls.
2) The ability to quickly switch tasks/tabs within the browser (ie responsiveness vs. speed). "Fucking flash!"
3) The stability of the browser. I don't really care so much if a single tab crashes; I'll just reload it. Someone with 40+ tabs in firefox, however, is stuck waiting a minute or so while whatever they were doing crawls back from the dead. (Users who don't have session management in their browser are even less fortunate.)
Meanwhile, I sit there contentedly working away, not distracted by such things, due to using Chrome and a lightweight window manager on Linux. I only start noticing a slow down when I'm being inefficient, anyway - IE, doing too much at once, getting distracted, and not getting anything done.
Of course, the slow users don't complain all that much, either. Seems they can't quite keep up with much of anything. :P
A little speed in the right places makes a huge difference.
Re:Fuck this shit (Score:5, Insightful)
Re:dev IE9 and dev FF vs release Chrome? (Score:2, Insightful)
Well, some people are not "most of the people", and i like that there is one browser who caters to that group.
Re:Speaking as someone that switched to OS X (Score:5, Insightful)
Let me see if I understand your complaint - a rare bug with workarounds is given extremely low priority and that's indicative of a general problem with FOSS software? What do you propose as a change to the FOSS model of development to improve the engineering?
Well, first here we have a user. He sure as hell wouldn't have found the workaround. Then it gets reported to an administrator, who can't find any known bug (assuming he searched, but I would first) who then spends time building a test case. Then it goes to a bug triager who knows about five year old bugs and can identify this as a dupe and provide a workaround.
1. 99% of users would never find that workaround
2. After that effort by three people, nothing will really be done to fix it
Even better are the projects that drive down bug count by attrition, WINE is rather notorious for this I've noticed. Every few WINE versions - who are on a biweekly schedule - they'll ask you to retest even though there's been no patches towards the bug, so when you grow tired of that shit the bug is "solved".
As for rarity, any bug that doesn't happen on a developer's machine for a developer's use case is by assumption rare unless a real shit storm of complaints prove otherwise. Don't get me wrong, it's pretty much central to OSS that people fix the stuff they wanted fixed, no bug is too obscure if you're willing to fix it yourself. But it also very much creates an A-list and a B-list of bugs to get fixed, and you go in B. It's actually slightly easier with commercially paid support who don't really have an agenda of their own, what the customers most want fixed is their priority.
Of course no, a single anecdote from single project doesn't prove anything. Sometimes I've had good experiences, but also many bad. Most annoying ar those where a bug report only leads to more and more work until it far outweighs my interest in fixing the bug. I'm guessing that's often the case for the OSS developer on the other side of the table too, which is why he is pushing 90% of the work right back at me. Often the choices then end up: a) keep using buggy software, b) spend way too much time getting it fixed or c) buy something that works.
I wish there was more of an organized bounty system, you'd pledge towards some bug fix or feature request, then developers could take it on. The pledges would be kept by a trust until the developer claims to be done (or withdraw), then the pledgers vote to confirm / claim incomplete / reject that it has been done. You'd probably need to have some sort of arbitration system to deal with formal disputes, who could take a processing fee off the pledge. Combine that with an eBay-like reputation system and I think it should work out well without too much hassle. I know it sounds a little like rent-a-coder and I don't mean it like that, more of an organized way of doing small custom work inside existing projects.
... and even older Opera? (Score:5, Insightful)
I guess the actual selection of versions shows how the point of the article was more about bashing FF4 compared to IE9 (in which it also failed, given the very small difference between them) rather than doing a honest comparison of all of the browsers.
Re:Translated from Redmondese this means... (Score:2, Insightful)
Re:Too late for a film at 11 joke... (Score:4, Insightful)
I'm not so quick to forget why I dropped IE in the first place. To get away from ActiveX and general apathetic browser security. Where's that represented in benchmark?
Re:Fuck this shit (Score:4, Insightful)
Instant updates to all your users at exactly the same time. No more worrying about if your users are using old, insecure, incompatible versions.
For times when users need to be connected to eachother, having everybody go through a standard HTTP server is the easiest way to getting rid of networking problems, not having to worry about firewalls, and not having to expose your users computers to the web.
Just try submitting patches (Score:3, Insightful)
and seeing how many of these fruits of your labor actually end up in future releases. (Hint: none, even if the bug does things like solve massive NTFS corruption or critical on-screen corruption in the ATI 2D driver, both of which are real attempts of mine.)
They say "U Want? U Fix!"
Then you do and they say it doesn't:
Fit with project goals
Adhere to project style or standards
Offer regression data about other use cases
Solve a big enough problem to justify effort to include
Or they'll just ignore the hell out of you and eventually (as you noted) mark the bug as "solved" simply because bug submitters stop responding to repeated nonsense bug labor/reply requests after 2-3 years... even if there are replies (SOMETIMES DOZENS OF THEM) in the Bugzilla threads linking to WORKING PATCHES.
God I've had it with having to rebuild half of my packages from .src.rpm each release using hacked and rehacked patches, version after version, from Bugzilla discussions that never, ever seem to make it into the code year after year and release after release because arrogant maintainers have their heads up their asses.
THIS is why I'm done with FOSS. Just done. I loved it and the community in the '90s. Now it's mostly arrogant young hotshots with no particular interest in getting actual work done apart from the work of coding for coding's sake, implementing new experimental unstable (if not useful) features at the expense of old, stable, useful ones that most users rely on.
After all, u want... u fix! (But not in my project -- build your own codebase from scratch!)
Re:Fuck this shit (Score:3, Insightful)