Slashdot Log In
After 4 Years, HydrogenAudio Opens New 128kbps Listening Test
Posted by
timothy
on Sun Nov 09, 2008 03:02 PM
from the my-tin-ears-expect-pure-gold dept.
from the my-tin-ears-expect-pure-gold dept.
kwanbis writes "After more than four years, a new MP3@128kbps listening test is finally open at HydrogenAudio.org! The featured encoders are: LAME 3.97, LAME 3.98.2, iTunes 8.0.1.11, Fraunhofer IIS mp3surround CL v1.5, and Helix v5.1 2005.08.09. The low anchor is l3enc 0.99a. The purpose of this test is to find out which popular MP3 VBR encoder outputs the best quality on bitrates around 128 kbps. All encoders experienced major or minor updates that should improve audio quality or encoding speed, and we have a totally new encoder on board. Note that you do not have to test all samples — it is a great help even if you test one or two. The test is scheduled to end on November 22nd, 2008."
Related Stories
[+]
2nd Multi-Format 128kbps Public Listening Test 316 comments
technology is sexy writes "Roberto Amorim has launched his latest public listening test evaluating the performance of different audio codecs at 128kbps, among them Apple's AAC implementation (used in iTunes), LAME, Ogg Vorbis fork auTuV, WMA, Musepack and even Sony's Atrac3 format, which is soon to be used in their own music store. Read more on Hydrogenaudio and check out the results of prior tests. As opposed to most evaluations of audio codecs, this is a scientific test adhering to ITU-R BS.1116-1 as much as possible while still allowing everybody to participate."
[+]
Audio Compression Primer 236 comments
Hack Jandy writes "For those of you with a little extra time this afternoon, check out Sudhian's primer to all things concerning audio compression. The article details everything from DRM to CRC matrixes (with a healthy dosage of Ogg)."
[+]
Apple: Music Listeners Test 128kbps vs. 256kbps AAC 428 comments
notthatwillsmith writes "Maximum PC did double-blind testing with ten listeners in order to determine whether or not normal people could discern the quality difference between the new 256kbps iTunes Plus files and the old, DRM-laden 128kbps tracks. But wait, there's more! To add an extra twist, they also tested Apple's default iPod earbuds vs. an expensive pair of Shure buds to see how much of an impact earbud quality had on the detection rate."
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
use the cans, luke (Score:5, Insightful)
good headphones are a must for such close listening tests. you'll only be able to hear really major differences with most speakers.
Re:use the cans, luke (Score:4, Interesting)
Good headphones are nice in so far as they block ambient noise and allow you to hear any artifacts easier, but since MP3 is a perceptual encoder it is actually more likely that artifacts are audible on "defective" hardware.
If a cheap speaker or cheap headphone's frequency response is bad enough to mess with the model's idea of masking, for example, poor quality reproduction can actually make the 'tricks' of MP3 apparent.
Parent
Re:use the cans, luke (Score:4, Insightful)
Good headphones seldom block ambient noise. Some of the very best headphones out there are open, including (but not limited to):
AKG 701/702
Grado (all models)
Beyer DT880
Sennheiser HD600/650
They're made, not to isolate you from the environment and prevent sound from escaping either way, but to replace loudspeakers with the best sound possible, with no regard to (or for) the environment.
My beef with this test is that they use 128 kbps VBR.
These days, space isn't as such a premium as it was a few years ago, and few use 128 kbps anymore, unless it's the default for their encoder or they haven't bothered changing.
I would think that 160, 192, 224, 256 and even 320 kbps are more common than 128 kbps these days.
Then there's VBR. And VBR in itself is by many considered evil -- yes, you cram in more data that way, but you end up with a sound stream that switches back and forth between different qualities, which is more apparent to the ear than if it was all at the lowest quality. It's like listening to a radio where the FM stereo kicks in every now and then. Yes, that is quantifiably a better quality than listening to it in mono, but I still prefer switching to mono to get a worse, but stably worse, sound.
The same piano key hit multiple times can end up sounding different with VBR. First you get an awesome 224 or 320 kbps note, then another, but then omgwehaveusedupallthebandwidth you get a 80 kbps note that just doesn't sound similar. Overall, the quality has gone up, but the net effect is that it sounds jarring.
Personally, I have enough disk space, and use FLAC when I can. When I can't, I use 224 kbps CBR, because at that high bitrates, I can't really tell any difference, and I avoid the whole VBR bitrate-changing problem.
Parent
Re:use the cans, luke (Score:5, Insightful)
Actually, the point of VBR is to keep quality close to constant, as some audio frames are more easily compressed than others. Constant bitrate actually gives you variable quality. Variable bitrate gives you near constant quality. If you "hear" the quality changing in a VBR recording, theres something wrong with the encoder.
Parent
Re: (Score:3, Interesting)
The same piano key hit multiple times can end up sounding different with VBR. First you get an awesome 224 or 320 kbps note, then another, but then omgwehaveusedupallthebandwidth you get a 80 kbps note that just doesn't sound similar.
ahh, audiophiles. they'll make up anything to try and sound like they hear better than the rest of us, even when what they've described it entirely NOT how VBR works. it's nothing to do with available bandwidth, it's about what is required to represent that frame of sound. if encoding silence at 224kbps makes you happy, by all means go on believing that shit sounds better than 320kbps vbr files that will come out smaller in size.
personally i just use apple lossless for everything so i dont have to concern
Re:use the cans, luke (Score:4, Informative)
Parent
Re: (Score:2)
While headphones may give better sound for the money spent isn't expensive speakers better than expensive headphones?
Re: (Score:3, Interesting)
Depends if you can isolate outside noise as well. If you live like a hermit, certainly(no neighbours making noise while testing, etc...
Good headphones do that for you, and isolate ambient noise better. You can't noise-cancel on speakers either, not practically.
Re: (Score:3, Insightful)
Assuming equal quality speakers (which indeed cost a lot more than headphones) The difference is your room. Many people don't enjoy the sound of being in an isolation chamber with your headphones, An effect that would be mimicked listening to speakers in a sound dampened room. A lot of people enjoy the effects of sound unfolding like in a music hall.
I'm sure the best sound in the world comes from those ridiculously expensive listening rooms you see, baffled and shaped like an opera house. But if you put the
Re:use the cans, luke (Score:5, Funny)
And I'd be even more surprised to see a moron who paid $200,000 for fuckin' speakers to admit otherwise.
Parent
Re: (Score:3, Insightful)
it really depends on the application. for music, headphones are probably a much better value and give a better listening experience. but if you're building a home theater setup for watching movies, obviously speakers are the way to go. you just can't properly enjoy 5.1 surround sound with stereo headphones. but at the same time, most recording studios probably work with headphones far more than they work with speakers.
and in this day and age you shouldn't have to spend $200 to get a decent pair of headphone
Re:use the cans, luke (Score:5, Funny)
Then imagine how good $200,000 headphones would be. They'd include an extra large driver to place on your torso for deep bass.
Parent
Re: (Score:3, Interesting)
Personally, I like my music to kick me in the guts a bit, you know?
Re:use the cans, luke (Score:5, Insightful)
Parent
Re:use the cans, luke (Score:4, Insightful)
I call horseshit. I only care about the differences *I* can hear with the speakers/headphones *I* have. Isn't that the whole point? The shortcuts I can take without noticing a difference...
It is a heck of a lot easier to upgrade your equipment than it is to re-encode your audio, assuming you even have the original sources around.
What sounds fine today on your current system may sound poor on your next system tomorrow.
Parent
Re:use the cans, luke (Score:4, Insightful)
Parent
ugh (Score:5, Insightful)
Wow, what a mess. Download this package. Now download fourteen more packages (DownThemAll is the only reason I didn't give up right then). Y'know, I'm kinda interested in this subject, as I have no trouble hearing artifacts in most 128kbps CBR MP3s, but this is just a huge pain in the ass. Wouldn't a simple Flash app have made things so much easier?
Re:ugh (Score:5, Interesting)
Agreed.
I went into the trouble of trying to run this under Linux.
the supplied batch files didn't work - it was missing files due to bad paths. the java application required a HUGE meddling around, choosing the settings, creating tests... I gave up. I'm not *that* motivated to help.
If you're trying to design a public test, the goal is to make it as simple as possible. An online application is an absolute must here.
I would be surprised if there will be anymore than a few hundred responses to this, all from a very specific demographic, Hardly a representative sample of the general population.
Parent
Re:ugh (Score:5, Funny)
Wow, what a mess. Download this package. Now download fourteen more packages (DownThemAll is the only reason I didn't give up right then). Y'know, I'm kinda interested in this subject, as I have no trouble hearing artifacts in most 128kbps CBR MP3s, but this is just a huge pain in the ass. Wouldn't a simple Flash app have made things so much easier?
I gave up at step 3751 "Buy Monster cables". ;-)
Parent
Re:ugh (Score:5, Informative)
Parent
Re:ugh (Score:4, Insightful)
That hardly solves the problem. The applet should be embedded in the web page and download all the samples automatically, on demand. Why the stupid rigmarole of doing everything yourself? It's a ridiculously complex process. I gave up when I discovered that "OS X users are asked to handle decoding of samples themselves" what does decoding the samples involve? I haven't a fucking clue, because that's all it tells me.
Parent
Re: (Score:3, Funny)
All this work for some LAME encoding...
What kind of music is involved (Score:5, Interesting)
Re:What kind of music is involved (Score:5, Informative)
You are asserting MP3 has faults it does not have.
"Overtones" are not an issue, nor do I think you could point out a 'problem sample' which fails due to the presence (or lack) of "overtones."
Popular music, in fact, is often harder to encode efficiently as it tends to have the dynamics compressed out of it (see loudness war), full of distortion, and therefore be closer to random data.
Temporal smearing is clearly a problem with MP3, and is evident in music such as harpsichord, but that is not the claim you make.
Do you have any ABX tests to back your claim?
Parent
Re: (Score:3)
No he's right I'm afraid. On sounds with hundreds of overtones, even a rate of 256 kbps (in stereo) isn't enough. I know because I experimented with image transmission over the sound by synthesising an image into a sound by turning each horizontal line into a modulated sine at a specific frequency. Here's an example [sourceforge.net] with the input and output image transmitted over a 256 kbps (mono!) MP3.
Long story short, when you've got over 500 overtones simultaneously, you need a much higher bitrate. In the aforementioned
Re:What kind of music is involved (Score:5, Insightful)
Parent
Re: (Score:3, Interesting)
No it's not the point. The point is, in such kinds of music as "spectralist music" there's a much higher density of "sound information" due to the shear number of overtones and it requires higher encoding bitrates.
As for the point of the experiment I linked to, the point isn't to actually store images in MP3s but to show how images can be transmitted over sound with a good quality, furthermore in an intuitive format (i.e. the image's 2 space dimensions are 'mapped' to the sound's time and frequency dimensio
Re: (Score:3, Insightful)
you didn't say anything about the effect on sound (Score:3, Insightful)
You said that if you look at a spectrum as an image you can spot degradation in a lossy encoder. That's entirely irrelevant, since mp3 is a perceptual encoder; it deliberately loses information that, according to its predictive model, listeners cannot actually hear. To show that it fails to do this, you need to conduct a blind listening test with the encoded and un-encoded version, and actually reliably (i.e. significantly more than 50% of the time) pick out the encoded one.
Re:What kind of music is involved (Score:5, Interesting)
"Overtones" are not an issue
Incorrect. One of the ways MP3 achieves lower bitrates is by removing overtones of a fundamental frequency when the overtones are reasonably quieter. If, for example, you pluck an "A" string tuned to 440hz, the string would also resonate at 880Hz, 1320Hz, 1760, and so on. An MP3 encoder would remove these overtones if they were significantly quieter than the original 440Hz tone, since research has shown that the human ear doesn't really notice them if the fundamental is much louder. The problem arises, as the parent noted, in some niche music; however anyone should be able to notice this in things like cymbals, where the most basic sound and timbre of the instrument is defined entirely by the overtones it produces. You can hear this as an almost flanger-esque quality to the cymbals in sub-128Kb/s encoded MP3s. Any drummer will tell you that this drives them up a wall, and the way the psychoacoustic model of MP3 compression handles overtones is the culprit.
Parent
Re: (Score:3, Interesting)
Re:What kind of music is involved (Score:5, Funny)
I'm not too worried about the quality of my music. Since I mostly listen to noize, industrial and EBM, the occasional scratching, pop, siren, explosion, grinding metal and screaming only accentuate the already apparent awesomeness of what I'm burning holes in my ear drums with.
Parent
Re: (Score:3, Interesting)
I've noticed that besides classical music, the music that is hardest to encode is 70s/80s underground punk music (the hard kind), because it's often recorded on VERY bad equipment with lots of background amplifier humming, distortion, recorded on a cheap cassette 4-track porta studio in someones garage, and no mastering what so ever. The encoder have a very hard time to keep up with all the "extras" that are usually mastered away. At 128 kbps, hihats and cymbals sound like "pssh" instead of "tss" and the gu
Painfully painful test (Score:2, Insightful)
You know what, I thought I'd be nice and give this a shot, but the amount of effort involved just isn't worth it. If it isn't 'click on this link, listen, rate', it's too much work. Download x, install x, email x - way, way, way too much work for what is being given in return.
mono encoding (Score:2)
I am deaf in one ear, so I won't take the test since I don't think I can do it justice.
I know that mono encoding saves relatively little space since joint stereo minimizes redundancy between the channels, but is there anything else I should be aware of as someone who transcodes everything to mono before I copy it to my mp3 player ?.
Out of phase (Score:5, Informative)
is there anything else I should be aware of as someone who transcodes everything to mono before I copy it to my mp3 player ?
Some songs are recorded with parts out of phase between the stereo channels. This means that the left and right channels, instead of being up/up and down/down, are up/down and down/up, which creates directional effects in stereo (especially on a surround receiver) but cancels itself out in the conversion to mono. For instance, "Happiness in Slavery" on Broken by Nine Inch Nails loses the snare in mono, and the quality of the snare drum in the remix of Coburn's "We Interrupt This Program" used with the NEDM meme drifts back and forth between clap-like and snare-like.
Parent
Outdated? (Score:5, Insightful)
Umm... 128 Kbps? Seriously? And no Ogg Vorbis, AAC etc... If you're bothering to set up a listening test, why limit yourself to 128 Kbps MP3?
Also, this should really be set up as a blind test, you get to listen to two clips, and have to choose which is better. The clips are randomized, of course... I could go on, but I'd just make myself sound even more arrogant. ;)
Re:Outdated? (Score:4, Funny)
Parent
Re: (Score:3, Insightful)
this should really be set up as a blind test, you get to listen to two clips, and have to choose which is better.
I agree entirely. They should also include different bitrates - do many people still use 128kbps? - and versions which aren't compressed at all. Hopefully the results might shut up the audiobores who keep insisting that MP3 isn't good enough for their precious ears.
Re:Outdated? (Score:4, Interesting)
MP3 is plenty good enough, it just requires more bits. Why have 192kbps MP3 when you can save room with an equally good 160kbps Vorbis?
Why have 160 kbps Vorbis when hard drives are growing in capacity and dropping in price?
I used to encode things in 192 kbps, then VBR, and now I want to smack myself over the head for doing so; blank CDs weren't so cheap back then, and I wanted to save a little bit of money. Looking back, it sure as hell wasn't worth it - I have crappy, lossy mp3 encodings of rare albums that I cannot obtain anymore, and a hard drive that could easily hold 2000 albums encoded with FLAC.
Sure, there was a time when storage was a premium, but now it isn't. Save room for WHAT? Five years from now, when you will be able to cheaply have 10 TB of storage space in your computer, are you going to regret having 160 kbps Vorbis instead of FLAC encodings? I know I would be, so now I'm encoding every CD I still have in lossless. If I were interested in HD video (which I'm not), I'd have no intention of re-encoding it to smaller sizes, because I *know* there will be a time when I'd regret it. Of course, YMMV.
Parent
Re:Outdated? (Score:5, Insightful)
Indeed, after reripping my entire collection for like the 3rd or 4th time because it got corrupted randomly, I switched to just ripping everything to lossless. If I need a copy for multiple MP3 players and such or change my mind about what compression rate or type I want it's a task my computer can handle without me swapping tons of discs.
It's always been easier to encode to a lower quality than to a higher quality. And in a strict sense the latter isn't really even possible.
Parent
Re: (Score:3, Insightful)
For the same reason you do performance testing on slow machines. It makes it easier to detect differences in sound quality (or slow code in performance testing) and the results scale smoothly upwards.
By limiting the bitrate to 128, you're more likely to get good data instead of just guesses.
Re: (Score:3, Informative)
Re: (Score:3, Informative)
Also, this should really be set up as a blind test, you get to listen to two clips, and have to choose which is better. The clips are randomized, of course...
Glad you took the time and checked how they do it.
The test is done using the ABC/HR blind listening utility, which does pretty much what you suggest.
It is a blind test; they've tested aac, ogg, etc (Score:3, Informative)
HydrogenAudio has done all kinds of listening tests [hydrogenaudio.org] over the years with various different codecs, including multiple encoders and rates for all of the major lossy formats. Their public tests are well designed blind tests-they give you the various samples but don't tell you which is from which encoder, and you're asked to use ABC/HR which is a program designed specifically for blind testing of audio.This one just happens to be 128 kbps MP3.
This is particularly of interest to a lot of people because
Why are we still focusing on 128kb? (Score:3, Insightful)
It was 10 years ago when I bought a Rio PMP-300, the first readily available flash-based MP3 player. It came with 32MB of internal memory, and would accept a single SmartMedia card, 32MB max in size (which I quickly went out and bought).
Back then, the size of your MP3 files mattered a whole lot more. At 128K CBR, I could fit 6 to 9 songs on each bank, depending on how long they were. The artifacts were noticeable to even my poor hearing. So I then stepped up to 160kb CBR and then LAME -remix (VBR, average ~ 190K ) encoding setting. I will make a note here that not all MP3 encoders are created equal - there is no fixed encoding standard, just for decoding.
With the VBR files, I could only fit 3-6 songs per bank of the Rio, so yea, it mattered then. If I wanted a specific CD to take to the gym with me, I had to think about what I put on the Rio. Often I couldn't fit the whole CD on the device or I had to swap play order to better use the slack space in each memory partition.
Can you even buy a MP3 player with less than 1GB of internal flash memory today? Skip past something like the iPod shuffle or equivalent at 1 and 2 GB, and you are quickly looking at 4GB, 8GB, 16GB or more.
I just encoded my copy of Linkin park's Minutes to Midnight CD that I bought with LAME 3.97 high quality VBR and it came out to 77.6 MB for the whole thing with the average bit rates in the 230kb/s to 270kbs. It wouldn't fit on the RIO at this quality. On the cheapest iPod Shuffle, I could fit 13 similarly sized CDs at this quality encoding. On the cheapest iPod Nano, around 100 similarly sized and encoded CDs.
My point??
128Kbs is sooo 1990s.. We've moved on. Storage, be it flash or Hard drives, has gotten order(s) of magnitude cheaper and bigger. So why aren't we moving our mindset about default MP3 quality UP to reflect the change? Make very High quality VBR the default and raise the average quality bar.
Pro sound tech chiming in (Score:3, Interesting)
MP3 is optimized for best performance at 256kbps. MPEG-4 AAC is optimized for 128kbps. Trying to determine which MP3 codec works best at 128kbps is like figuring out whether Jimi Hendrix or Jimmy Page would be better if they lost two fingers off their left hands. Similarily, MP2 is optimized for 384kbps, and beats MP3 at bitrates beyond 256, which is why it is widely used on DVD's at 384kbps.
Here's how it plays out:
Lossless codecs obviously are best when bandwidth isn't an issue
MP2 (MPEG-1 layer 2) is best from 320kbps upwards
MP3 (MPEG-1 layer 3) is best from 160 - 256 kbps
AAC (specified initialy in MPEG-2, finalized in MPEG-4 [they skipped MPEG-3 not to be confused with MP3]) best from 128kbps downwards
MP2's at 384kbps sound better than MP3's at 256kbps, which sound better than AAC at 128kbps. None of the codecs sound any better at higher-than-optimal bitrates, i.e. an MP3 at 320kbps cannot sound better than a 256kbps MP3 encoded from the same source.
Simply put, it's the codec that determines the optimal bitrate. Given a 128kbps bitrate, who cares how an inappropriate codec performs?
Direct link (Score:3, Informative)
Only up half an hour and already slashdotted.
Here is a direct link to the download site: http://www.listening-tests.info/mp3-128-1/ [listening-tests.info]
Re: (Score:3, Informative)
Must be a really small 'entire music collection'. 1/2 of a gig is 512MiB. Average FLAC encoded song I have are anywhere from 8MiB to 25MiB per song.
I hope you meant a half TB. That would make it much bigger... but again, you could still probably have your entire collection stored in WAV. I have about 1,300 songs.. all stored in various formats (MP3, OGG, FLAC) and it's under 10GiB.
Re: (Score:3, Funny)