Vorbis And Musepack Win 128kbps Multiformat Test

technology is sexy writes "After 11 days of collecting results Roberto Amorim today announced the results of his 2nd Multi-Format listening test: Vorbis fork AoTuV scored the highest and ranks as the winner together with open source contender Musepack closely followed by Apple's AAC implementation and LAME MP3, which improved markably since last year thanks to further tunings of its VBR model done by Gabriel Bouvigne. Sony's ATRAC3 format ranks last after WMA on the third place. The suprising success of AoTuV (compared to last year's performance of Xiph.org's reference implementation) shows the potential of Vorbis and possible room for further tuning and improvments. Take a look at the detailed results and their discussion at Hydrogenaudio.org."

Good.

[Cyno01]

Good to know MP3 is still improving. Yes vorbis and others are great, but i know every software and hardware player out there plays MP3. I'll be ripping all my cds to high quality MP3 befor i go to college, not because its the absolute best, but because its a standard. Standards aren't always the most efficient, but their strength lies that you cant change them on a week to week basis. Whatever hologrphic storage based finger sized half terabyte 24th generation iPod i buy ten years from now will probably still play my 128 and 256 MP3s.

best vs popular

[trs9000]

i realize the geeks of the populace want the highest quality encoding to win. naturally. and it helps when something such as vorbis is rated so highly; it gives it even more geek cred.

however: as someone who studied music and audio, i am constantly surprised at what people will listen to. my friends (well some of them) have no problem cranking low quality mp3s of 50 cent, while i drop my jaw at the poor audio quality as a result of lost information. one time i even remarked to my dad "oh its an mp3" when he was playing something i had given to him which had been apparently later encoded. he wasnt sure (he didnt do the encoding) but doublechecked and yes it was mp3 (probably 160 kbps). he was impressed, when to me the timbral change in the cymbals was a dead giveaway. another time i asked a friend of mine if he was using aac to import all his cds in to itunes when he had been recently doing so. he looked at me blankly and said "whats aac?". which meant, yes he was.

i apologize for rambling, this is what im arriving at:
despite early adoption influence etc that geeks hold, how much does all of this really matter. most people dont care what format its in as long as they can listen to it. and often they cant discern loss of quality unless its extreme. so while i applaud these efforts, im simply wondering if -- aside from research -- they arent futile.

Re:But does it matter?

[Liquid-Gecka]

I don't think we have to "get the word out." Most cool tech innovations make it into the mainstream if they are really good enough. I remember buying a CD-RW (4x2x2) for $400. Everybody thought I was stupid for spending so much on a piece of hardware. Later I spend $300 on a 64MB MP3 player. The guy at the desk told me that I shouldn't get a MP3 player because changing the media was really hard. Yet most people now have both of these gadgets. If Vorbis is license free and simple enough to put on a hardware chip then it will slowly gain support and slowly people will begin to see it.

iTunes AAC encoding problems

[lotsofno]

It is interesting that the note that they used the AAC encoder in iTunes 4.2 instead of the newer 4.5 because of "quality" concerns.

Apparently there's some "high frequency ringing" going on [hydrogenaudio.org].

Better stick to something else for now, if planning to rip to AAC.

Re:mp3 still defacto standard

[Roland Piquepaille]

This might be of interest to musicians but the proverbial "jane doe" will keep using mp3 for quite a while

Actually it's not that simple. Jane and Joe Doe will start using Ogg, AoTuV or other TLA and ETLA compression schemes when their favorite music players feature them. In the case of Ogg, it's not going to happen anytime soom because:

1 - There's an entrenched MP3 market, as you said

2 - It's an open-source format, i.e. it reeks of piracy and hackers in the minds of music player manufacturers and of the public

3 - It doesn't have the backing of major industry players, being seen as a "maverick" effort to undermine other potentially money-making closed-source formats

4 - It certainly doesn't have the backing of the RIAA, because it doesn't have DRM and other in-the-customer's-face copyright protection schemes

In short, people using Ogg will be opensource-aware and advocates for a long time to come. As for other Apple customer-unfriendly sort of schemes, I'm not convinced the general populace has bought into the idea of paying for music tracks that can become unplayable at the next Apple format-change-du-jour, because they're copyright-protected and therefore impossible to convert to another standard (in theory).

So yes, you're right, MP3 will stay around for a long time. I certainly won't convert my collection anytime soon...

Open source wins: really cool, but ...

[eatmadust]

it isn't everything. Microsoft still has enough cash to fight this. Our local radio (Switzerland) still broadcasts in .asx. I sent them an e-mail asking them why. They said because their server is sponsored by Microsoft. Now I listen to virgin radio, they broadcast in broadband ogg [virginradio.co.uk]

Inaccurate test, big bitrate differences

[Echnin]

Quoting post on second page of discussion:

This particular test should be called, "The 128 kbps test for iTunes/WMA, and the low-130 test for AC3 and LAME, and the close-to-160 test for MPC/Vorbious.

Leahy iTunes MPC Vorbis Lame WMA Atrac3
bitrate 128 155 149 133 128 132
Score 4.34 4.41 4.68 4.11 4.37 3.76

That really doesn't look very fair to me! MPC and Vorbis using about 20% more bits than Lame and iTunes AAC.

Re:But does it matter?

[Liquid-Gecka]

At the time there where many large format media types available. Tape, Jazz, Zip, portable HD all where in there prime. CD-RW discs couldn't be read in most CD-R drives and most earlier audio systems wouldn't read CD-R discs. There wasn't a big advantage to buying a CD-RW drive over other solutions. Zip drives where $100 and the discs where $2, when I bought my CD-RW discs where like $20 and CD-R discs where $.50, or $1.25 with a jewel.

Re:Good.

[m0rbidini]

OK, you have a point. But check VorbisHardware [xiph.org] for hardware with Ogg Vorbis support. Also, though Lame did well, MP3 is known to have some limitations. But if you have to use MP3, experiment --alt-preset standard in Lame. It was made to offer very good sound quality in bitrates that average below 200 kbps in most cases.

Regarding the results... It's a bit surprising that this third party tuning/tweak of Vorbis did so well. Which is great and I think Xiph should think about incorporating this work on their official encoder as soon as possible, in order to take advantage of its potential. You may be surprised about the relative low performance of AAC. This is partially due to the fact that the chosen AAC encoder was a CBR only encoder (because it was the best AAC encoder at this bitrate on a previous test - Nero encoder is also a good one and offers VBR encoder). With a good implementation of VBR AAC, it should be possible to get a better performance.

While most of the tested codecs/formats showed good performance at 128 kbps, this test alone shows that none can give transparency ( transparency == unability to distinct from the original source for most people and under good conditions) at this bitrate, contrary to what many think. People who think this is important should demand higher quality files from famous online music services (like iTunes Music Store).

People interested in lossy audio encoding should also try Musepack (file extension .mpc). It is considered by many of the hydrogenaudio enthusiasts as the best format at medium/high bitrates, offering transparency with bitrates normally lower (with standard preset ~170 kbps, typical 142 ... 184 kbps) than what is possible with other formats/codecs. It's now open source (LGPL, iirc). Its biggest disadvantage is the lack of support in portable players (though decoding musepack is faster than decoding the other formats in this test). There are plugins for almost every software player and foobar2000 [foobar2000.org] (which I consider the best one) has native support for it

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:Good.

[Malc]

I can't say that I'm that worried about it. It's just a matter writing a short script to iterate over my FLAC archive and re-encode. I anticipate doing that anyway as encoders improve at the same bit-rate. In fact, I'm already thinking of doing that with anyway to change some of my options...

Re:best vs popular

[trs9000]

You can critise the 'timbral change in the cymbals' but why do we worry so much about this when, in many cases came off a synthesizer anyway.

no, im afraid you may have misunderstood. what i was trying to convey is that while i may have a personal preference for a more 'accurate' representation, that doesnt really matter because most people dont seem to care too much. it has nothing to do with the synthesizing of it. i make electronic music so i tend to be quite fond of that, personally. (^_^)

it can be considered personal preference and if someone would rather listen to an downsampled, 8bit version of my music, well by all means, let them.

but in the pursuit of standards and codecs, etc, i think we should strive for accuracy -- but keeping in mind that it may not be adopted: whatever means you give the majority that is readily available and working, i believe *that* is what they will use.

Re:Striving for innovation

[Anonymous Coward]

The Microsoft-published PC version of Halo uses Ogg Vorbis for all of its audio - so it's not as if there hasn't been a precendent. :-)

Re:More vorbis content is needed

[Petronius]

archive.org [archive.org] has lots of *free* content in OGG format.

Vorbis and MP3 formats from a techie view point

[tiger_omega]

Having designed and written a mp3 decoder and now working on a vorbis decoder I can't say I'm that suprised by vorbis coming out on top.

From a technological standpoint the Vorbis codec has 10 years of audio compression R&D in it since MP3 was invented.

MP3 is a subband DCT based codec using fixed window length. Vorbis is also DCT based but encodes an approximation to the orginal frame's spectral curve and also uses variable length window length.

In using the source from the vorbis library and the decoder specification to help guide its development I have to say it is a real joy to code. The people at xiph.org have really done a first class job and have approached some of the problems of audio codec design with some of the best lateral thinking that I have ever seen.

Believe me! Coming from me that is very rare praise.

Re:How much of this is just OGG fans voting?

[13Echo]

I was always fond of LAME encodings in a high quality VBR mode. It was always my favorite method of storing my music on my hard drive, since the quality was quite good. Over time, I decided that I would really start comparing it to some other formats for long-term archiving. I wanted to settle with one format, once and for all. I had originally been a BladeEnc user, but LAME seemed far superior to me.

When I first enconded some of my music in the Vorbis format, I was a bit underwhelmed when comparing it to LAME. It didn't really sound the same. Then, I compared the Vorbis files to the raw WAV rips. Surprisingly, the Vorbis files sounded more true to the original WAV rips. I was very surprised. All this time, my ears had tuned to the LAME acoustic model, which wasn't as accurate as I had once thought. After comparing a large portion of my CD collection in both LAME and Vorbis encodings, I made a decision...

I decided to start using FLAC. That way, I could listen to al of my music without any concern for quality. Sure, each CD takes up about 300 MB of space (50%-60% average compression), but it sounds so sweet.

If quality is a concern, maybe LAME MP3/AAC/Ogg Vorbis aren't the the right choices. Hard drive limitations aren't so much of an issue anymore. I guess that I cna see a point in having lower quality files for easy web transmission and low storage capacity, but the quality difference is just too noticable for me to ignore, when comparing any of these formats to a lossless format like FLAC. That's also one of the reasons that I like Magnatune so much, since I can buy music online that is already compressed in lossless FLAC format.

Re:Expensive earbuds and MP3 players

[kryptkpr]

I *really* get a kick out of it when people buy an MP3 player and a pair of high-end earbuds. It's just plain inane

Ever heard of --alt-preset-extreme?

Sure.. stuff I download will continue to sound crappy (I don't even keep anything below 192kbit anymore).. but stuff I encode myself sounds quite good. I'm not audiophile, but I cannot tell the difference between an --alt-preset-extreme'd recording and the original.

Re:FLAC?

[lovemayo]

according to this [hydrogenaudio.org], 54 results was discarded because they ranked the reference file, instead of the encoded file. If flac was to be included in such a test, I'm sure it would have won, but im also sure it wouldn't have scored a perfect 5, even if it should have.

What if?

[Anonymous Coward]

"what if people liked the 128kbs mp3 version better than the cd recording!"

People like to pierce all parts of their body too. All we can learn from this practice is that 50% of the population is below average.

And people listening to 128kb lossy music and saying "they sound just like the CD" are in the lower part of that bell curve.

That's hardly insulting, its just reflects the statistical model here.

Interesting MPC outlier

[mcg1969]

The MPC codec was neck-and-neck with Vorbis most of the time, except for one song by Debussy. What is interesting though is that it only encoded at 91kbps for that song---suggesting that perhaps if it were forced to use more bits it might have scored higher. It seems the heuristics it uses to determine how many bits it needs didn't quite work for that song.