About the issue of long subtitle texts without any punctuation

[gkngkngkn]

Just as mentioned before in the subtitledit project, I encounter situations where there are no punctuation marks in a large portion of the text when transcribing some video files using Fast-whisper. I use large v2 model to transcribe only english video. the only parameter I add is “--sentence" to keep putting out compelet sentence to translate. All other parameters are set to default.

I have tried adding intitial_prompts such like --intitial_prompt "Hello, this is the first sentence. And this is the second one. A little pause ... and we are back."instructing it to add punctuation marks, but this problem still persists.

What methods or parameters need to be added to force consistent punctuation throughout the entire transcription?

[despairTK]

Try different --compute_type, --beam_size, --best_of, --initial_prompt. It is impossible for a set of parameters to be suitable for all audios. When encountering audios without punctuation marks, you can only try to adjust the parameters to achieve better results.

[despairTK]

Try --initial_prompt ",.", it is best to use the large-v2 model --compute_type to automatically select. If using --initial_prompt does not produce punctuation, disable this and keep other parameters unchanged before transcribing.

[Purfview]

Try --initial_prompt

Similar prompt is already on by default.

[gkngkngkn]

yeah, I tried it sir, the effect improved significantly but not stable in some files, I speculate that this issue is related to the underlying logic of Whisper.

[despairTK]

Transcribing audio cannot only rely on the same parameters. Different audio languages, different speaking speeds, pause intervals and other factors will affect the final result, which requires a long period of experimentation.

[gkngkngkn]

yeah,i see sir.

[Purfview]

My advice: don't touch --intitial_prompt.
Can you share the JSON file with these issues? [made with defaults, without --intitial_prompt] . Use -f=all to get it.
Btw, what model do you use and do you change other settings?

[Purfview]

What if you use default prompt and add --prefix="Okay, so the one hour is very key."?

Btw, how it looks with just -prompt=None?

[gkngkngkn]

[despairTK]

I actually have a rather interesting solution. Generally, there are two situations in which audio without punctuation and sentences that cannot be segmented normally occurs:

One is that there is no sound or accompaniment in the first few seconds of the audio. In this case, you need to use editing software to edit directly to the beginning of the first sentence, cut out the extra few seconds of accompaniment at the beginning, and then transcribe the processed audio.

Second, if after editing out the beginning accompaniment part, the transcription still has no punctuation and the sentences cannot be separated normally, then edit again to the beginning of the second sentence and the punctuation marks and sentence separation will be displayed normally.

I don't know what the rationale is for this either. . . But before I didn't use the --initial_prompt parameter, this was how I achieved normal punctuation and sentence segmentation.

[gkngkngkn]

My speculation is that Whisper automatically transcribes the current text based on the form of the preceding text (such as tone, format, punctuation, etc.), then uses the current text as the basis for the next paragraph, like a chain reaction of dominoes.

In the case of long videos where exceptionally lengthy statements without punctuation occur midway, Whisper might mistakenly assume that the standard for transcription is without punctuation, leading to the omission of punctuation in the subsequent text.

The reason why I found that a longer initial prompt yields better results in testing is likely because it consistently provides sufficient lengthy text as a basis, allowing the entire text to be stably outputted in accordance with the format.

In the situation you described, I speculate that the reason is that Whisper treats the "no sound or accompaniment" part as equivalent to text without punctuation, as the foundational basis for the entire transcription. This is also why an initial prompt is necessary I guess.

[despairTK]

However, if a sentence is used as the initial prompt, it will affect the correct segmentation of some sentences in the audio. For example, a word that originally appeared at the end of the previous sentence was split to the beginning of the next sentence. Although these two sentences are independent sentences, changing the position of a word may change its meaning. This is the result of testing a lot of audio, so now when I use the initial prompt, I only use two simple symbols ",." to produce an effect. If it still doesn't work, I will choose another one-- compute_type, if it doesn’t work, I will cut the beginning of the audio to a certain extent.

[Purfview]

Isn't punctuation degrading after some time with your custom prompt?

[despairTK]

English audio of more than 1 hour does have a greater chance of triggering no punctuation, but Asian languages are more likely to trigger no punctuation. . . Even if the audio is only less than ten minutes, it will be easily triggered. For example, out of 10 videos in Asian languages, 4 or more will have no punctuation marks. In order to deal with this situation before, I tried many tips, as shown in the screenshot.
I even combined several sentences from the same language. Even so, Asian languages are still much better than English without punctuation. Another point is that Asian languages do not have punctuation marks like English. For example, the first sentence has punctuation marks, the second sentence has no punctuation marks, and the third sentence has punctuation marks again. Or maybe a certain sentence suddenly has punctuation marks. Here I also provide a few screenshots for reference. If you need the corresponding json file, I can also provide it.

[Purfview]

@despairTK

but Asian languages are more likely to trigger no punctuation

But I by default did nothing for those Asian languages. Let's try something new...
Does Chinese vs Korean vs Japanese punctuations differ?

[despairTK]

Yes, I know this is not your fault. I just guess that this situation may be the same as the reason why some English audio transcriptions have no punctuation marks, just like this feedback from [gkngkngkn].

The punctuation marks and usage habits of these three languages are different.

Chinese punctuation marks:，。！？

Japanese punctuation marks: 、。！？

Korean punctuation marks: ,.!?

Japanese does not use，during pauses, but uses 、. Korean punctuation marks are the same as English.

[gkngkngkn]

@despairTK bro, You can try using a article perfected with punctuation in these languages as initial prompts, as long as possible. In my attempts, this method has proven to be effective, the overall situation with punctuation is good, although some issues still arise as I metioned.

[despairTK]

@gkngkngkn
I have tried, but the effect is not good. If too many sentences are used as initial prompts in Asian languages, it will affect the sentence segmentation of Asian languages, and the unexpected situation I mentioned here will occur. ##130 (reply in thread)

[Purfview]

Ok, here you go, the promised test build -> https://we.tl/t-wBSWVD8hQy

New hardcoded initial_prompt presets:
-prompt=default - this should be better than previous default.
-prompt=default2 - this is maybe similar to "custom", I think maybe to make this as the default preset in the new release.
-prompt=default_zh
-prompt=default_ja

--reprompt=2 - This will re-add initial prompt to every window. So initial prompt never disappears from the prompt.

So, I would want feedback on how new prompt defaults compares to the previous version, then check if reprompt helps with the long audios with the punctuation problems.

[despairTK]

Does this set of parameters only test English? My plan is to put audio files in three languages: English, Korean, and Japanese into one folder for batch testing.

[Purfview]

I'm thinking of making auto for prompt then add different default prompts depending on the language set by the --language.

Is making that "Hello, this....." as default for English is good idea?
Should we translate it to other common languages and make the similar default prompts? And for other languages leave current "default2"?

Does this set of parameters only test English?

What set of parameters? default and default2 are made out of just latin punctuation marks.

English, Korean, and Japanese into one folder for batch testing

Ja has different punctuation so obviously it's not good idea.

[Purfview]

@despairTK Here try this as prompt for ja, made with the google translate:

こんにちは、最初の文です。そしてこれが2つ目です。少し休憩してください...そして戻ってきます。

[despairTK]

--model large-v2 --beam_size 10 --best_of 10 --compute_type=float16 --sentence --verbose=true --max_comma 1000 --max_gap 6000 --initial_prompt "Hello, this is the first sentence. And this is the second one. A little pause ... and we are back."

When I was just testing, there was something strange happening with one of the files. The test parameters in the previous reply are used. Enable --reprompt=2 and there will be a sudden lack of punctuation near the end of the audio. Modified to disable --reprompt=2 and re-transcribed, the punctuation at the end returned to normal.

--reprompt=2 False.zip
--reprompt=2 True.zip

[Purfview]

Look at the whole general results, not at the few sentences.

Maybe smaller prompt size would help on such files, test with 32, 64 or 128 size.
Those numbers means how many tokens [similar to words] are taken from a previous transcription into the sliding prompt window, smaller - less previous context - transcription accuracy can drop a little bit. Don't forget that initial prompt is fitting in that max size, so actual real context is a bit less.

Btw, if audio and speech is very clear maybe you can disable the context prompt. In such case reprompt has interesting effect - it would make only an initial prompt as the whole prompt for every chunk. In the reference/original code it would insert an initial prompt only to the first chunk and nothing at all to the next chunks.

[despairTK]

I tested a few dozen more audios in different languages, and also read here: https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_prompting_guide.ipynb

Then I feel that the sentences filled in by --initial_prompt should be adjusted by myself based on the transcribed audio, and there will not be a "universal" initial prompt. --initial_prompt The initial purpose should only be to affect some polyphonic words, professional nouns, names, and capitalization of the transcribed content.

The impact of --initial_prompt on punctuation and sentence segmentation may be good or bad. The good thing is that there are punctuation marks, but the bad thing is that it imitates the sentence style of the initial prompt, and then it is originally a complete sentence, but it ends up being divided into two sentences. It is also possible that the ending word of the first line of sentences appears in the second line. Start the sentence. This kind of error is usually difficult to detect. After all, the transcribed subtitles have hundreds of lines, and it is impossible to check them line by line.

This is just my personal opinion after testing, there may be better solutions. I also hope there is a good way to solve the current problem. :)

[gkngkngkn]

@despairTK
“Such initial hints can bias the transcription content toward the theme of the painting, and filling in error-prone professional terms can effectively solve the problem of selecting homonymous words when transcribing.”

Yes, providing a specialized prompt can improve accuracy in transcribing specific professional terms or ensuring correct capitalization. while I believe punctuation should be relatively consistent across different prompts. I am wondering if there is a universal prompt, at least in terms of punctuation segmentation, that can consistently ensure accurate transcription for various types of text. For example, using a passage that covers a range of standard punctuation usage as a prompt(don't need too long).
can test using this which created by chatgpt as prompt if u have time :)

As for the segmentation errors you mentioned that may arise from adding prompts, occasional mistakes are acceptable to me compared to the issue of without punctuation. :)

[despairTK]

@gkngkngkn
There is no universal initial prompt. . . . If you have tested various --initial_prompt, you should find that the shorter the sentence you provide, the sentences in the transcription result will also be divided into shorter ones. The longer the sentence you provide, the longer the sentences in the transcription result will be divided. .

--initial_prompt seems to me to be more like correcting typos and helping to transcribe some nouns in professional fields as its main function. The segmented sentences and punctuation marks may simply imitate the sentence style of the initial prompt, and cannot perfectly segment the sentences correctly.

So my first choice is to use the default --initial_prompt parameter, which has minimal impact on the overall results. If there are no punctuation marks when transcribing, I will cut out the audio and then re-transcribe it. Generally, only a small number of audios will be transcribed without punctuation marks.

It is possible that waiting for the release of better models in the future will solve these problems.

[Purfview]

What about prompt reset on no punctuation in a segment?

[despairTK]

Is it resetting from the beginning of the audio or just where there is no punctuation? Anyway, you can put this feature in and we can test it.

[Purfview]

Is it resetting from the beginning of the audio or just where there is no punctuation?

No, it would reset only after no punctuation in a segment is detected.

[Purfview]

Here is test4 -> https://we.tl/t-E0WZgFf0tR

Added --prompt_reset_on_no_end switch, should reset prompt if there is no end of sentence in a chunk/window.
Not tested, should write a log message on verbose when triggered.

Test it on audio where it after some time starts transcribing with no punctuations. Check if it triggers and if punctuation recovers after it.

Of course it should be used with reprompt.

[Purfview]

What do the sentences you add look like?

"Hello, this is the first sentence. And this is the second one. A little pause... and we are back."

I've translated it to top 30 languages.

Btw, --initial_prompt=default consist of 4 basic punctuations and it auto switches between various languages if language is set, otherwise it uses latin.

[despairTK]

In the test version you released before, the two parameters --reprompt=2 and --initial_prompt "Hello, this is the first sentence. And this is the second one. A little pause ... and we are back." are sufficient. Solve the problem of no punctuation marks in English audio.

Is the --prompt_reset_on_no_end parameter a solution to the problem of no punctuation marks in audio in other languages?

[Purfview]

In the test version you released before, the two parameters --reprompt=2 and --initial_prompt "Hello, this is the first sentence. And this is the second one. A little pause ... and we are back." are sufficient. Solve the problem of no punctuation marks in English audio.

That is same as --initial_prompt=auto --language=en.
I think to set auto by default in the next release.

Is the --prompt_reset_on_no_end parameter a solution to the problem of no punctuation marks in audio in other languages?

Yes, if you set the language or your own initial prompt.
It checks for these punctuations: s_end_chr = ['.', '。', '!', '！', '?', '？','؟']

[despairTK]

If it only transcribes English audio, is there no difference between this beta version and the official version?

[Purfview]

If it only transcribes English audio, is there no difference between this beta version and the official version?

I don't understand what you mean. "Difference" in what?

[Purfview]

r160.11 with new stuff is released.
Continue the feedback on them there -> #142