Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash after "Operation finished" #381

Closed
151SoGood opened this issue Jan 15, 2025 · 11 comments
Closed

Crash after "Operation finished" #381

151SoGood opened this issue Jan 15, 2025 · 11 comments

Comments

@151SoGood
Copy link

Whenever I run the large-v3-turbo model, once it completes (as it indicates 'Operation finished in: XX'), cmd prompt hangs & the .exe crashes ('Standalone Faster-Whisper-XXL has stopped working'), and after I click the Windows prompt 'Close program', cmd prompt unfreezes and I can enter the next command.

I don't experience this with the other models, e.g. medium & medium.en.
Any way for us to troubleshoot this? I ran it with '--verbose=true' - anything look odd to you? I can see it glitches at the end, repeating the same text ("Thanks") 6 times. Also, these warnings "Compression ratio threshold is not met with temperature"?

Not a big deal as it does the job to completion, just the crashes are annoying.

Standalone Faster-Whisper-XXL r245.1 running on: CUDA

Number of visible GPU devices: 1

Supported compute types by GPU: {'int8', 'float32', 'int8_float32'}

Note: 'large-v3' model may produce worse results than 'large-v2'!

[2025-01-15 02:47:42.740] [ctranslate2] [thread 53040] [info] CPU: AuthenticAMD (SSE4.1=true, AVX=true, AVX2=true, AVX512=false)
[2025-01-15 02:47:42.740] [ctranslate2] [thread 53040] [info] - Selected ISA: AVX2
[2025-01-15 02:47:42.741] [ctranslate2] [thread 53040] [info] - Use Intel MKL: false
[2025-01-15 02:47:42.741] [ctranslate2] [thread 53040] [info] - SGEMM backend: DNNL (packed: false)
[2025-01-15 02:47:42.741] [ctranslate2] [thread 53040] [info] - GEMM_S16 backend: none (packed: false)
[2025-01-15 02:47:42.742] [ctranslate2] [thread 53040] [info] - GEMM_S8 backend: DNNL (packed: false, u8s8 preferred: true)
[2025-01-15 02:47:42.742] [ctranslate2] [thread 53040] [info] GPU #0: NVIDIA GeForce GTX 1080 (CC=6.1)
[2025-01-15 02:47:42.742] [ctranslate2] [thread 53040] [info] - Allow INT8: true
[2025-01-15 02:47:42.743] [ctranslate2] [thread 53040] [info] - Allow FP16: false (with Tensor Cores: false)
[2025-01-15 02:47:42.743] [ctranslate2] [thread 53040] [info] - Allow BF16: false
[2025-01-15 02:47:45.588] [ctranslate2] [thread 53040] [info] Using CUDA allocator: cuda_malloc_async
[2025-01-15 02:47:45.824] [ctranslate2] [thread 53040] [info] Loaded model C:\Testing\Faster-Whisper-XXL_models\faster-whisper-large-v3-turbo on device cuda:0
[2025-01-15 02:47:45.824] [ctranslate2] [thread 53040] [info] - Binary version: 6
[2025-01-15 02:47:45.825] [ctranslate2] [thread 53040] [info] - Model specification revision: 3
[2025-01-15 02:47:45.826] [ctranslate2] [thread 53040] [info] - Selected compute type: int8_float32

Faster-Whisper's large-v3-turbo model loaded in: 3.15 seconds

Starting sequential inference to transcribe: 'C:\Testing\lorem_ipsumh.mp4'

Processing audio with duration 02:07.640

VAD [silero_v4_fw] finished in: 0:00:00.466

VAD filter removed 00:00.000 of audio

VAD filter kept the following audio segments: [00:00.000 -> 02:07.640]

Processing segment at 00:00.000
[2025-01-15 02:47:46.806] [ctranslate2] [thread 37480] [info] Loaded cuBLAS library version 12.1.3
[00:00.000 --> 00:06.980] lorem ipsum
[00:07.220 --> 00:12.400] So, lorem ipsum
[00:12.620 --> 00:18.720] In terms of where lorem ipsum
[00:19.060 --> 00:23.140] So, lorem ipsum
[00:23.340 --> 00:27.660] And lorem ipsum
Processing segment at 00:27.660
[00:27.660 --> 00:33.920] So, lorem ipsum
[00:34.540 --> 00:40.180] Afterwards, lorem ipsum
[00:40.280 --> 00:43.280] I lorem ipsum
[00:43.500 --> 00:47.860] And lorem ipsum
[00:48.240 --> 00:53.060] And lorem ipsum
[00:53.220 --> 00:55.700] If lorem ipsum
[00:56.140 --> 00:57.360] It'll lorem ipsum.
Processing segment at 00:57.660
[00:57.960 --> 00:59.920] I lorem ipsum
[01:00.140 --> 01:05.080] So lorem ipsum
[01:05.080 --> 01:06.620] before lorem ipsum
[01:06.900 --> 01:09.280] And ilorem ipsum
[01:09.640 --> 01:13.660] lorem ipsum
[01:14.320 --> 01:18.580] lorem ipsum
[01:18.740 --> 01:22.760] lorem ipsum
Processing segment at 01:22.760
[01:22.760 --> 01:28.160] lorem ipsum
[01:28.520 --> 01:33.000] lorem ipsum
[01:33.360 --> 01:36.060] lorem ipsum
Processing segment at 01:36.060
[01:36.060 --> 01:40.660] lorem ipsum
[01:40.960 --> 01:44.260] And lorem ipsum
[01:44.600 --> 01:47.380] And lorem ipsum
Processing segment at 01:47.380
[01:48.720 --> 01:55.320] And lorem ipsum
[01:55.320 --> 01:58.340] to lorem ipsum
[01:58.480 --> 02:00.120] So, lorem ipsum
[02:00.400 --> 02:01.120] This lorem ipsum
[02:01.240 --> 02:04.720] If lorem ipsum
[02:04.740 --> 02:06.820] And lorem ipsum
Processing segment at 02:06.820

    Compression ratio threshold is not met with temperature 0.0 (14.754386 > 1.900000)
    Compression ratio threshold is not met with temperature 0.2 (12.578947 > 1.900000)
    Compression ratio threshold is not met with temperature 0.4 (14.754386 > 1.900000)
    Compression ratio threshold is not met with temperature 0.6 (6.809524 > 1.900000)
    Compression ratio threshold is not met with temperature 0.8 (8.900000 > 1.900000)
    Log probability threshold is not met with temperature 0.8 (-1.429833 < -0.700000)
    Compression ratio threshold is not met with temperature 1.0 (8.669903 > 1.900000)
    Log probability threshold is not met with temperature 1.0 (-0.995539 < -0.700000)
    [02:06.920 --> 02:07.000] Thanks.
    [02:07.000 --> 02:07.100] Thanks.
    [02:07.100 --> 02:07.280] Thanks.
    [02:07.280 --> 02:07.440] Thanks.
    [02:07.440 --> 02:07.620] Thanks.
    Reset prompt. prompt_reset_on_temperature threshold is met 1.000000 > 0.500000

Transcription speed: 6.46 audio seconds/s

Subtitles are written to 'C:\Testing' directory.

Operation finished in: 0:00:23.732
@Purfview Purfview changed the title Crash using the large-v3-turbo model Crash after transcription is finished Jan 15, 2025
@Purfview Purfview changed the title Crash after transcription is finished Crash after "Operation finished" Jan 15, 2025
@Purfview
Copy link
Owner

Whenever I run the large-v3-turbo model, once it completes (as it indicates 'Operation finished in: XX'), cmd prompt hangs & the .exe crashes ('Standalone Faster-Whisper-XXL has stopped working'), and after I click the Windows prompt 'Close program', cmd prompt unfreezes and I can enter the next command.

It's known issue for some users, but reason is unknown and no solution.
It was reported by a user there: #282 (comment).
Then in the newer version I just added "del model" at very end and he said that he didn't reproduced crash anymore, probably that thing was unrelated as there were many libs updated...

You could post your Windows version, GPU model and error from "Windows Event Viewer".

Self-note:
There (#282 (comment)) was this pair of errors from Windows Event Viewer:

Faulting application name: faster-whisper-xxl.exe, version: 192.3.4.0, time stamp: 0x6626da66
Faulting module name: KERNELBASE.dll, version: 10.0.17763.6054, time stamp: 0xc9a93043
Exception code: 0xe06d7363
Fault offset: 0x0000000000041b39

Faulting application name: faster-whisper-xxl.exe, version: 192.3.4.0, time stamp: 0x6626da66
Faulting module name: ucrtbase.dll, version: 10.0.17763.1490, time stamp: 0x48ac8393
Exception code: 0xc0000409
Fault offset: 0x000000000006e77e

Reports at faster-whsiper & ctranslate2:
Windows process crashes when the GPU model is unloaded
Python process crashes on exit under Windows with CUDA

@Purfview
Copy link
Owner

Purfview commented Jan 15, 2025

Btw, is that paste from console is same like you see it in console?
Because after Processing segment at 02:06.820 line formatting looks weird, it should look like this:

[00:47.740 --> 00:48.940]  Some text.
  Processing segment at 00:53.480
* Compression ratio threshold is not met with temperature 0.0 (9.054054 > 2.400000)
* Compression ratio threshold is not met with temperature 0.2 (14.565217 > 2.400000)

And there should be two spaces in front of "Processing segment".

Also, these warnings "Compression ratio threshold is not met with temperature"?

That's normal, not warnings, just an info. Don't use verbose if you don't want to see it.

I can see it glitches at the end, repeating the same text ("Thanks") 6 times.

That's "normal" too AKA hallucination. Try better VAD like --vad_method pyannote_v3 it may help.

@151SoGood
Copy link
Author

151SoGood commented Jan 15, 2025

Thanks for looking into it! Same pair of errors for me:

Event Viewer logs:

5:41:05PM:
Faulting application name: faster-whisper-xxl.exe, version: 245.1.0.0, time stamp: 0x67859210
Faulting module name: KERNELBASE.dll, version: 10.0.19041.1741, time stamp: 0xe9b4a91b
Exception code: 0xe06d7363
Fault offset: 0x0000000000034fd9

5:41:07PM:
Faulting application name: faster-whisper-xxl.exe, version: 245.1.0.0, time stamp: 0x67859210
Faulting module name: ucrtbase.dll, version: 10.0.19041.789, time stamp: 0x2bd748bf
Exception code: 0xc0000409
Fault offset: 0x000000000007286e

In your 2nd link, there seems to be users who reported a workaround here: SYSTRAN/faster-whisper#71 (comment)

  • however not sure if that can be implemented as a fix in your version?

@151SoGood
Copy link
Author

151SoGood commented Jan 15, 2025

Btw, is that paste from console is same like you see it in console? Because after Processing segment at 02:06.820 line formatting looks weird, it should look like this...

You are right - might have been a copy+paste formatting which messed it up (I think formatting / tab characters got messed up when I copied from my reply in the other issue).

Thanks for also clarifying on the other points, and also providing the suggestion for a better VAD, will give it a go!

@Purfview
Copy link
Owner

Purfview commented Jan 15, 2025

Faulting module name: KERNELBASE.dll

Exactly same fault as for another user, it's in Windows' API module, that can mean anything...

To debug further you could share minidump created for a crash:

  1. Select Start > Control Panel.
  2. Double-click System, and then select Advanced system settings > Advanced.
  3. Under Startup and Recovery, select Settings.
  4. In the Write debugging information list, select Small memory dump (256k):

alt text

Note: Don't select "Automatically restart".
After a crash minidump should be created at "C:\Windows\Minidump".
Zip it and drag & drop it here into a post.

In your 2nd link, there seems to be users who reported a workaround here: SYSTRAN/faster-whisper#71 (comment)

And users who reported that this workaround doesn't work for them...
One of the workarounds posted there you can try yourself -fallback None, but quality of results may degrade a bit.

@Purfview
Copy link
Owner

Purfview commented Jan 15, 2025

Test this "test1" exe: https://fromsmash.com/.x_pTRmmXT-bt
Copy exe to "Faster-Whisper-XXL" folder and check if crash can be reproduced with it.

@151SoGood
Copy link
Author

Thanks @Purfview. Tested it with test1.exe and it crashed in the same way. Confirmed however there was no minidump created after the crash.

@151SoGood
Copy link
Author

151SoGood commented Jan 15, 2025

And users who reported that this workaround doesn't work for them... One of the workarounds posted you can try yourself -fallback None, but quality of results may degrade a bit.

Can confirm this works running -fallback=None avoids the crashing.

@Purfview
Copy link
Owner

Confirmed however there was no minidump created after the crash.

Did you enabled minidump like I wrote? Try to restart Windows.

@151SoGood
Copy link
Author

151SoGood commented Jan 15, 2025

Confirmed however there was no minidump created after the crash.

Did you enabled minidump like I wrote? Try to restart Windows.

I did. I will restart and see how that goes.
Update: Will do that later tomorrow when back on the PC.

@Purfview
Copy link
Owner

Closing. Will be fixed when CTranslate2/Nvidia/Windows fix it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants