This is stolen from here mostly, since only few things have changed...
----------------------------------------------------------------------

The purpose of this thread is to (finally) test LAME 3.96 throughly enough to make it the new recommended LAME version and enhance communication between LAME developers and the HA.org community. The recent confusion about the different LAME versions and compiles (-Qrcd trouble etc.) have led to the decision that moving along with LAME development has been postponed for long enough and LAME 3.90.X can hopefully be declared dead very soon.
Please test as many samples as possible according to the following guidelines and post any results here.


1. Use the following LAME compiles (last updated on 2004/04/13):
lame3.90.3
lame3.96 final

2. The focus of the test should be --alt-preset/--preset standard, since it will allow us to make conclusions regarding the overall performance of the 'code level tweaked' VBR presets. Other VBR/ABR/CBR presets are interesting too, but not as important. If problems with --alt preset standard are detected feel free to compare extreme and insane too. Test the following combinations please:

VBR/ABR
(~256kbps) 3.96 --preset extreme vs. 3.90.3 --alt-preset extreme
(~210kbps) 3.96 --preset standard vs. 3.90.3 --alt-preset standard
(~160kbps) 3.96 -V 4 vs. 3.96 --preset 160 vs. 3.90.3 --alt-preset 160*
(~128kbps) 3.96 -V 5 vs. 3.96 --preset 128 vs. 3.90.3 --alt-preset 128*

CBR
(320kbps) 3.96 --preset insane vs. 3.90.3 --alt-preset insane
3.96 --preset cbr <bitrate> vs. 3.90.3 --alt-preset cbr <bitrate>

If you want, you can additionally test VBR/ABR vs. CBR at comparable bitrates

3. You may use any sample you want, as long as you upload a losslessly compressed version of that (or provide a working link), so others can verify your results. The upload thead is here. Using samples that have been used for testing 3.96beta is a good idea to show progress/regressions.

4. Your test results have to include the following:

• ABX results for
  3.90.3 vs. Original
  3.96 vs. Original
  3.96 vs. 3.90.3
• ABC/HR results are appreciated especially at lower bitrates, but shouldn't be considered a requirement.
• (Short) descriptions of the artifacts/differences

Notes:

*
Since ABR presets decrease gain as workarround against audible clipping on decoding of loud tracks, please use replaygain (trackgain) for decoding to .wav / ABXing, ideally using foobar2000 (apply to the original as well to get matching volume, of course!). On quiet tracks --scale 1 can be added after --(alt-)preset <bitrate> on encoding alternatively. Details see here

There won't be a separate discussion thread for now. To make it easy to find posts containing results, please start such posts (only!) with "RESULT". Results matching the requirements that have been posted elsewhere will be added to the "Results" post.

-----------------------------------------------
Edit 1 [2004/04/13]: typo fixed (3.96b2 -> 3.96), thanks to Lyx
Edit 2 [2004[04[14]: Replaygain/--scale related comment added. (-> "*")

This post has been edited by tigre: Apr 14 2004, 12:03

Test results so far:


VBR/ABR


~ 256kbps
----------------------- 3.96 --p extreme vs. 3.90.3 --ap exteme
3.96 --p extreme > 3.90.3 --ap extreme :: drone_short :: freakngoat :: 0x verified so far
---------------------------------------------------
3.96b1 --p extreme < 3.90.3 --ap extreme :: Lazy_Jones :: owowo :: 0x verified so far

----------------------- 3.96 --p extreme vs. 3.96 -V 1
3.96 --p extreme < 3.96 -V 1 :: drone_short :: freakngoat :: 0x verified so far + missing ABX results*



~ 210kbps
------------- 3.96 --p standard vs. 3.90.3 --ap standard
3.96b1** --p standard > 3.90.3 --ap standard :: spahm :: Pio2001 :: 1x verified by Wombat
3.96b1** --p standard > 3.90.3 --ap standard :: Birds :: Wombat :: 0x verified so far
3.96b1** --p standard > 3.90.3 --ap standard :: hokuscaredpiano :: Moitah :: 0x verified so far
3.96b1** --p standard > 3.90.3 --ap standard :: Lazy_Jones :: owowo :: 1x verified by freakngoat
3.96 --p standard > 3.90.3 --ap standard :: fatboy :: High Fidelity :: 1x verified 2Bdecided
3.96 --p standard > 3.90.3 --ap standard :: Drone_short :: freakngoat :: 0x verified so far
3.96 --p standard > 3.90.3 --ap standard :: awe32_20sec :: 2Bdecided :: 0x verified so far
---------------------------------------------------
3.96b2 --p standard = 3.90.3 --ap standard :: 41_30sec :: ViPER1313 :: 0x verified so far
---------------------------------------------------
3.96b2 --p standard < 3.90.3 --ap standard :: myf_4sec :: LoFiYo :: 0x verified so far
3.96b2 --p standard < 3.90.3 --ap standard :: Hustlejet :: harashin :: 0x verified so far
3.96b2 --p standard < 3.90.3 --ap standard :: Chanchan1 :: tigre :: 0x verified so far
3.96b2 --p standard < 3.90.3 --ap standard :: Hosokawa___Atem_lied :: harashin :: 0x verified so far
3.96 --p standard < 3.90.3 --ap standard :: 41_30sec :: Gecko :: 0x verified so far

--------- 3.96 --p fast standard vs. 3.90.3 --ap fast standard
3.96 --p fast standard > 3.90.3 --ap fast standard :: myf_4sec :: LoFiYo :: 0x verified so far


~ 160kbps
---------------------------------------------------
3.96b2 -V 4 < 3.90.3 --ap 160 :: myf_4sec :: LoFiYo :: 0x verified so far


~ 128kbps
-------------------- 3.96 -V 5 vs. 3.96 --p 128
3.96 -V 5 > 3.96 --p 128 :: Atrain :: ff123 :: 2x verified by tigre, [proxima]
3.96 -V 5 > 3.96 --p 128 :: BachS1007 :: tigre :: 0x verified so far
3.96 -V 5 > 3.96 --p 128 :: FloorEssence :: ff123 :: 2x verified by tigre, [proxima]
3.96 -V 5 > 3.96 --p 128 :: Layla :: ff123 :: 2x verified by tigre, [proxima]
3.96 -V 5 > 3.96 --p 128 :: MidnightVoyage :: ff123 :: 1x verified by tigre
3.96 -V 5 > 3.96 --p 128 :: Waiting :: ff123 :: 2x verified by [proxima], tigre
3.96 -V 5 > 3.96 --p 128 :: Blackwater :: tigre :: 1x verified by [proxima]
3.96 -V 5 > 3.96 --p 128 :: LifeShatters :: tigre :: 0x verified so far
---------------------------------------------------
3.96 -V 5 < 3.96 --p 128 :: BeautySlept :: tigre :: 0x verified so far
3.96 -V 5 < 3.96 --p 128 :: Blackwater :: ff123 :: 0x verified so far + missing ABX results*
3.96 -V 5 < 3.96 --p 128 :: TheSource :: ff123 :: 1x verified by tigre
3.96 -V 5 < 3.96 --p 128 :: LisztBMinor :: tigre :: 1x verified by [proxima]
3.96 -V 5 < 3.96 --p 128 :: BachS1007 :: [proxima] :: 0x verified so far

-------------------- 3.96 -V 5 vs. 3.90.3 --ap 128
3.96 -V 5 > 3.90.3 --ap 128 :: fatboy :: FatBoyFin :: 1x verified by High Fidelity
3.96b2 -V 5 > 3.90.3 --ap 128 :: Its_me :: tigre :: 0x verified so far
3.96b2 -V 5 > 3.90.3 --ap 128 :: entierren con rumba :: tigre :: 0x verified so far
3.96 -V 5 > 3.90.3 --ap 128 :: Atrain :: ff123 :: 1x verified by tigre
3.96 -V 5 > 3.90.3 --ap 128 :: BeautySlept :: tigre :: 0x verified so far
3.96 -V 5 > 3.90.3 --ap 128 :: FloorEssence :: ff123 :: 1x verified by tigre
3.96 -V 5 > 3.90.3 --ap 128 :: Layla :: ff123 :: 2x verified by tigre, [proxima]
3.96 -V 5 > 3.90.3 --ap 128 :: LifeShatters :: ff123 :: 1x verified by tigre
3.96 -V 5 > 3.90.3 --ap 128 :: MidnightVoyage :: ff123 :: 1x verified by tigre
3.96 -V 5 > 3.90.3 --ap 128 :: Thear1 :: ff123 :: 0x verified so far + missing ABX results*
3.96 -V 5 > 3.90.3 --ap 128 :: Waiting :: ff123 :: 2x verified by [proxima], tigre
3.96 -V 5 > 3.90.3 --ap 128 :: Blackwater :: tigre :: 0x verified so far
---------------------------------------------------
3.96 -V 5 < 3.90.3 --ap 128 :: BachS1007 :: tigre :: 1x verified by [proxima]
3.96 -V 5 < 3.90.3 --ap 128 :: Blackwater :: ff123 :: 1x verified by [proxima]
3.96 -V 5 < 3.90.3 --ap 128 :: TheSource :: ff123 :: 1x verified by tigre
3.96 -V 5 < 3.90.3 --ap 128 :: LisztBMinor :: tigre :: 1x verified by [proxima]
3.96 -V 5 < 3.90.3 --ap 128 :: Atrain :: [proxima] :: 0x verified so far
3.96 -V 5 < 3.90.3 --ap 128 :: FloorEssence :: [proxima] :: 0x verified so far

-------------------- 3.96 --p 128 vs. 3.90.3 --ap 128
3.96 --p 128 > 3.90.3 --ap 128 :: Atrain :: ff123 :: 1x verified by tigre
3.96 --p 128 > 3.90.3 --ap 128 :: BeautySlept :: tigre :: 0x verified so far
3.96 --p 128 > 3.90.3 --ap 128 :: FloorEssence :: ff123 :: 0x verified so far + missing ABX results*
3.96 --p 128 > 3.90.3 --ap 128 :: Layla :: ff123 :: 0x verified so far + missing ABX results*
3.96 --p 128 > 3.90.3 --ap 128 :: LifeShatters :: ff123 :: 0x verified so far + missing ABX results*
3.96 --p 128 > 3.90.3 --ap 128 :: MidnightVoyage :: ff123 :: 1x verified by tigre
3.96 --p 128 > 3.90.3 --ap 128 :: Thear1 :: ff123 :: 0x verified so far + missing ABX results*
3.96 --p 128 > 3.90.3 --ap 128 :: Waiting :: ff123 :: 1x verified by tigre
3.96° --p 128 > 3.90.3 --ap 128 :: applaud :: [proxima] :: 0x verified so far + missing ABX results*
3.96° --p 128 > 3.90.3 --ap 128 :: campestre :: [proxima] :: 0x verified so far + missing ABX results*
3.96° --p 128 > 3.90.3 --ap 128 :: fatboy :: [proxima] :: 0x verified so far + missing ABX results*
3.96° --p 128 > 3.90.3 --ap 128 :: Preecho1 :: [proxima] :: 0x verified so far + missing ABX results*
---------------------------------------------------
3.96 --p 128 < 3.90.3 --ap 128 :: Waiting :: ViPER1313 :: 1x verified by [proxima]
3.96 --p 128 < 3.90.3 --ap 128 :: BachS1007 :: tigre :: 1x verified by Atrain :: [proxima]
3.96 --p 128 < 3.90.3 --ap 128 :: FloorEssence :: tigre :: 1x verified by Atrain :: [proxima]
3.96 --p 128 < 3.90.3 --ap 128 :: Layla :: tigre :: 1x verified by [proxima]
3.96 --p 128 < 3.90.3 --ap 128 :: LifeShatters :: tigre :: 0x verified so far
3.96 --p 128 < 3.90.3 --ap 128 :: LisztBMinor :: tigre :: 1x verified by [proxima]
3.96 --p 128 < 3.90.3 --ap 128 :: Blackwater :: [proxima] :: 0x verified so far
3.96° --p 128 < 3.90.3 --ap 128 :: ct_reference :: [proxima] :: 0x verified so far + missing ABX results*
3.96° --p 128 < 3.90.3 --ap 128 :: fall :: [proxima] :: 0x verified so far + missing ABX results*
3.96° --p 128 < 3.90.3 --ap 128 :: rebel :: [proxima] :: 0x verified so far + missing ABX results*
3.96 --p 128 < 3.90.3 --ap 128 :: Atrain :: [proxima] :: 0x verified so far

---------- 3.96 -V 5 --athaa-sensitivity 1 vs. 3.96 -V 5
3.96 -V 5 --a-s 1 > 3.96 -V 5 :: Atrain :: [proxima] :: 0x verified so far + missing ABX results*
3.96 -V 5 --a-s 1 > 3.96 -V 5 :: Blackwater :: [proxima] :: 0x verified so far + missing ABX results*
3.96 -V 5 --a-s 1 > 3.96 -V 5 :: BachS1007 :: [proxima] :: 0x verified so far + missing ABX results*
3.96 -V 5 --a-s 1 > 3.96 -V 5 :: LisztBMinor :: [proxima] :: 0x verified so far + missing ABX results*
3.96 -V 5 --a-s 1 > 3.96 -V 5 :: Bayle___Jeita_or_Waters___Murmur___Etching :: [proxima] :: 0x verified so far
3.96 -V 5 --a-s 1 > 3.96 -V 5 :: Track01__Avison___Concerto_Grosso_30_sec :: [proxima] :: 0x verified so far



CBR


320kbps
no results so far


160kbps
no results so far


128kbps
---------------------------------------------------
3.96 --p cbr 128 < 3.90.3 --ap cbr 128 :: 41_30sec :: ViPER1313 :: 1x verified by SometimesWarrior, Brainchild4010
3.96 --p cbr 128 < 3.90.3 --ap cbr 128 :: Waiting :: ViPER1313 :: 0x verified so far
3.96 --p cbr 128 < 3.90.3 --ap cbr 128 :: Layla :: ViPER1313 :: 0x verified so far


* If a result doesn't match the minimum requirements (-> 4. in 1st post), the result is 'greyed out' until the missing data is provided or someone else confirms the results. If your result isn't included in this list at all, we're either too slow, or there's too much missing (ABX results, link to the sample, clear statment which version is better, information about lame version/setting used etc.). If you want to provide missing information, do it in a new post to ensure that we notice it.

** Results where 3.96 beta 1 --preset standard outperformed 3.90.3 --alt-preset standard were kept as the only difference between 3.96 beta1 and 3.96 final is the increased minimum bitrate, so no regression should be possible.

° Results by [proxima] with past-beta-2 internal version that should give results identical with 3.96 final


------------------------------------------------------------
Edit: All results from other threads I could find that belong here are added. - Tell me, if I've missed something, please.

This post has been edited by tigre: Apr 21 2004, 08:25

The latest version is 3.96, not 3.96b2.

RESULT


I did a short ABX with fatboy.wav
for preset standard and --preset 128

My results:

3.96 --preset standard vs. 3.90.3 --alt-preset standard

3.90.3 vs. Original
Total: 10/10 (0.1%)
Comment: Details and voice sounded a bit brighter (?) or more open - the original (I guess at least) softer but a bit more full

3.96 vs. Original
Total: 7/10 (17.2%)
Comment: Only minimal details in the background seemed a bit softer.
Hardly to ABX! Very good!

3.96 vs. 3.90.3
Total: 10/10 (0.1%)
Comment: Soft details in the background a little bit different, but hard to ABX.


---------------------------

3.96 --preset 128 vs. 3.90.3 --alt-preset 128

3.90.3 vs. Original
Total: 10/10 (0.1%)
VERY easy to distinguish metallic and distorted sound

3.96 vs. Original
Total: 10/10 (0.1%)
Comment: still easy to distinguish from the original, but very much better than 3.90.3! less distorsion, sounds higher than original

3.96 vs. 3.90.3
Total: 10/10 (0.1%)
Comment: 3.96 is the winner!!!

---------------


At least for fatboy.wav 3.96 is my favourite.

This post has been edited by tigre: Apr 15 2004, 16:27

RESULT

My settings:

3.90.3 --alt-preset 128 --scale 1
3.96 --preset 128 --scale 1
3.96 -V 5

Using the 12 samples I used for the first 64 kbs test



abchr results files at http://ff123.net/export/lame3.96test.zip
No ABX, though.



Looks like 3.90.3 ap 128 is the clear loser for me on these 12 samples

ff123

Thanks for the results so far.

I've performed a quick wave substraction test to find out for which settings the encoding results for beta1/beta2/final are identical. Results:


So the results from beta test thread are still valid for
3.90.3 --alt-preset 128 vs. 3.96 -V 5 (same for ap160 vs V4, but no results submitted)
3.90.3 --alt-preset standard vs. 3.96b2 --preset standard

edit: extreme/insane results added

This post has been edited by tigre: Apr 14 2004, 06:31

The results post should be up-to-date now. Only the ABC/HR test by ff123 misses because I'm not sure if/how to add those results. I'd add them greyed out until they're verified with ABX results (by someone else) and only use the results with differences in ABC/HR ratings... Or should I add ABC/HR results like these as a completely separated section to results post?

Your opinion, ff123?

Hmm, I don't know. I guess it's up to you. I'd like to see somebody else try all of those samples as well, though.

BTW, you'll notice that I added --scale 1 to the ABR command lines. That's because they include some gain reduction in them that the VBR settings don't.

ff123

I am unable to reproduce the dropout problem in 41_30sec.wav in 3.96final using

--preset cbr 128

I have uploaded my encode to: http://ff123.net/export/41_30sec_cbr128.mp3

I can distinguish from the original, but it doesn't sound worse than plain artifacting to me.

ff123

There is a problem at 7.11s

Are you talking about the clicking sounds? I hear them in the reference too. Did you download my encode to see if it's the same?

ff123

I didn't notice the gain reduction until know, probably because I used foobar2000 with replaygain (track gain) enabled for decoding. Given that fb2k decodes gapless on replaygain scanning, this should be a decent sollution to avoid volume-related problems as well.

Related questions: I assume that the gain reduction of ABR presets is used to reduce problems related to clipping on decoding of loud tracks.
- Is there some other reason like a clipping problem with encoding comparable to musepack?
- Based on this: Is using --scale 1 for the test 100% save or can it cause problems?
I'd like to modify the test recommendations for ABR presets according to the answers to these questions: Recommend using replaygain for decoding (including original file) and/or using --scale 1 for ABR.

Back when Dibrom was tuning the cbr/abr presets, he noticed a clicking problem caused by clipping at low bitrates (I think the sample was badvilbel.wav). So he put in a gain reduction which is greater at lower bitrates and progressively gets smaller as the bitrate increases.

I don't know if this problem still exists or not, but at 128 kbit/s, it was just barely audible, if I remember.

--scale 1 is better than using replaygain upon decoding because there is a resolution issue when using replaygain on mp3's.

ff123

Thanks for the explanation, ff123.


Could you please explain? AFAIK foobar2000 decodes to 32bit float, applies DSPs (e.g. volume change) and truncates/dithers to fixedpoint (e.g. 16bit for use with ABC/HR). Maybe I've too many different things to focus on right now, but I don't see a resolution issue.

I've added your results to the post BTW. I hope this motivates people to repeat the test with these samples (-> ABX ... ).

Oh, ok then. I guess I was thinking of mp3gain, which can only adjust in 0.3 dB increments. If fb2k applies replaygain subsequent to decoding, then everything is fine.

Edit:

Regarding testing of the 12 samples: I think it's more important that 1 or 2 people verify the entire suite (even if ABX results are not supplied) than it is for several people to ABX several samples out of the 12. So it would look more like the tests performed to find the best vorbis encoder. However, beggars can't be choosers, so any verification at all is welcome.

ff123

This post has been edited by ff123: Apr 14 2004, 11:21