Optimizing decoding results selection

[hoonlight]

e786e26

In the above commit, we changed the logic for selecting decoding results to the following

1. find the result that satisfies compression_ratio_threshold with the highest avg_logprob.

2. if no such result exists, find the result that satisfies the highest avg_logprob among "all" results.

After applying this, there was a noticeable improvement.

So we can assume that @guillaumekln's idea that when choosing a decode_result, it is better to consider both avg_logprob and compression_ratio, rather than just choosing the result with the highest avg_logprob value, is valid.

If so, instead of jumping directly from step 1 above to step 2, we could also consider the following modified logic.

1. find the decode_result that satisfies compression_ratio_threshold with the highest avg_logprob.

2. if no such result exists:
   Instead of finding the highest avg_logprob value from all results,
   Find the value of highest avg_logprob while gradually increasing compression_ratio_threshold.

A simple example code is shown below.

if compression_ratio_threshold is not None:
    compression_ratio_threshold = options.compression_ratio_threshold
    rate = 1

    while True:
        threshold_results = [
            result
            for result in all_results
            if result[3] <= compression_ratio_threshold * rate
            ]
    
        if threshold_result:
            decode_result = max(threshold_results, key=lambda x: x[1])
            break
        
        rate += 0.2

else:
   decode_result = max(all_results, key=lambda x: x[1])

return decode_result

In my tests on a variety of sources, the higher the compression_ratio was above the default threshold of 2.4, the more likely it was to be a badly transcribed sentence. When the compression_ratio was in double digits, it was very unlikely to be a correctly transcribed sentence.

By applying this alternative, we can adaptively choose the final result by increasing the bounds of the compression ratio threshold.

What do you think @guillaumekln, if you think this is helpful, I'll submit a PR.

[Purfview]

I couldn't come to conclusion about 356 PR effects, on every run output is different in those problematic audio areas.

[hoonlight]

https://github.com/archive-r/faster-whisper/tree/compare-results

This is a branch that adds fallback result logging to results.txt so that multiple results can be compared.

non_fallback_result: The result before fallback.
highest_logprob_result: The result selected by considering only avg_log_prob.
below_compression_threshold_result: e786e26
adaptive_result: The result selected based on the suggestions in this discussion.
openai_whisper_result: The Result selected by the openai/whisper method (choose the highest temp)

@Purfview, would you mind testing this when you have a chance?

Edit: I made some modifications to the proposed logic.

[Purfview]

Thx for the test code. I'll check it later.

PS
Atm, I'm trying to extend #376 bugfix as it only partially address the problem.

[Purfview]

Looks like results from below_compression_threshold_result and adaptive_result are best, but I don't see that adaptive_result is better from these two.

What statistics you got from your tests?

[hoonlight]

Hi @Purfview, thanks for sharing your test results.
I'm still testing, and in many cases the below_compression_threshold_result and the adaptive_result were the same, and in rare cases they were different. I haven't confirmed which is better yet. Once I have a little more data, I'll see which result has a lower word error rate.

[hoonlight]

After a week of testing, I saw slight improvements in English, Korean, and Japanese. I haven't tested it on other languages.
Since it was only a slight improvement, I'm closing this discussion.