Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SpeechLM Update #12430

Draft
wants to merge 161 commits into
base: main
Choose a base branch
from
Draft

SpeechLM Update #12430

wants to merge 161 commits into from

Conversation

stevehuang52
Copy link
Collaborator

@stevehuang52 stevehuang52 commented Feb 28, 2025 •
edited
Loading

Important

The Update branch button must only be pressed in very rare occassions.
An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.

What does this PR do ?

A couple of important updates in SpeechLM

Collection: [speechlm]

Changelog

  • Unified various input format into multimodal conversation, where audio and text are interleaved.
  • Added Whisper encoder support.
  • Fixed issues with PEFT not saving batch norm stats.
  • Fixed loading PEFT ckpt for inference.
  • Context parallel support in LLM. Speech encoder still cannot.
  • Various improvements and refactoring.

stevehuang52 and others added 30 commits September 16, 2024 10:38
@stevehuang52
fix type bugs
d967c51 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
Merge branch 'main' of https://github.com/NVIDIA/NeMo into main
84eaa59
@stevehuang52
Merge remote-tracking branch 'origin/main' into slm_v2
22217d0
@stevehuang52
Merge remote-tracking branch 'origin/main' into slm_v2
799a5ec
@stevehuang52
Update mixin.py
568f073 
add type hint

Signed-off-by: He Huang (Steve) <[email protected]>
@stevehuang52
Apply isort and black reformatting
f91030d 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
Update mixin.py
c9256ca 
Signed-off-by: He Huang (Steve) <[email protected]>
@stevehuang52
Apply isort and black reformatting
e41e4f9 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
Update mixin.py
2a70bd3 
Signed-off-by: He Huang (Steve) <[email protected]>
@stevehuang52
Merge branch 'main' into slm_v2
725fd88
@stevehuang52
add datamodule
de70200 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
Merge remote-tracking branch 'origin/main' into slm_v2
98a2673
@stevehuang52
add speechlm peft train, continue train, validation and misc
a00e5dd 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
Merge remote-tracking branch 'origin/main' into heh/speechlm_nemo2.0
52a1b0e
@stevehuang52
resolve merge confict
9d57bc6 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
update datamodule
ae3c5be 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
Merge remote-tracking branch 'origin/main' into heh/speechlm_nemo2.0
c2ef987 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
add script
dc951e2 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
fix speechlm inference
248f4e2 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
update
c42e17e 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
mergin origin/main
9ec5a96 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
Merge remote-tracking branch 'origin/main' into heh/speechlm_nemo2.0
90dcb6a
@stevehuang52
fix tp support
0a93523 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
Apply isort and black reformatting
5c87470 
Signed-off-by: stevehuang52 <[email protected]>
@artbataev
Apply isort and black reformatting
934b0db 
Signed-off-by: artbataev <[email protected]>
@stevehuang52
Merge remote-tracking branch 'origin/main' into heh/speechlm_nemo2.0
e7af9a7
@stevehuang52
Merge remote-tracking branch 'origin/main' into heh/speechlm_nemo2.0
0c32fa5
@stevehuang52
update
9d98e89 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
refactor
b659912 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
Merge remote-tracking branch 'origin/main' into heh/speechlm_nemo2.0
3653320
@stevehuang52
added multimodal conversation
095c24e 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
update
66d3ece 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
debug
3901303 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
debug
d9b29c4 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
update cfg
07755a4 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
clean up
3b3ae9e 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
Merge remote-tracking branch 'origin/main' into heh/speechlm_dev
1dcbd6c 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
add cp and fix tp
1fc1ffd 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
add missing __init__
b93bcf0 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
fix
144e39f 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
fix import ckpt
c0040d3 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
update io
c6791b0 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
fix hf tokenizer remove_special_tokens
74a04be 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
refactor
957e729 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
comment out lhotse assert
dd725e7 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
update cfg
04c6e77 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
update cfg
902b0f6 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
refactor and update inference
d05cbd4 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
update infer
b2a853f 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
update
c0da97f 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
update cfg
f2f288a 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
fix peft trainable params and update
fd6de0d 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
add support for whisper encoder
26bb5a4 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
Merge remote-tracking branch 'origin/main' into heh/speechlm_dev
601364e 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52
update
b88f0c0 
Signed-off-by: stevehuang52 <[email protected]>
@stevehuang52 stevehuang52 self-assigned this Feb 28, 2025
github-advanced-security[bot]
github-advanced-security bot found potential problems Feb 28, 2025
View reviewed changes
nemo/collections/speechlm/data/dataset/audio_text_lhotse_dataset.py
input_ids = [sample.input_ids for sample in samples]
context_ids = [sample.context_ids for sample in samples]
context_lengths = [sample.context_length for sample in samples]
answer_ids = [sample.answer_ids for sample in samples]

Check warning

Code scanning / CodeQL

Variable defined multiple times Warning

This assignment to 'answer_ids' is unnecessary as it is
redefined
Loading
before this value is used.
nemo/collections/speechlm/modules/asr_module.py
if self.target_module is not None:
model = get_nested_attr(asr_model, self.target_module)

model = HFWrappedEncoder(model)

Check failure

Code scanning / CodeQL

Potentially uninitialized local variable Error

Local variable 'model' may be used before it is initialized.
nemo/collections/speechlm/utils/text_generation/audio_text_generation_utils.py
elif isinstance(inputs, tuple) and len(inputs) == 4:
context_tokens_tensor, context_length_tensor, audio_signal, audio_signal_length = inputs
elif isinstance(inputs, tuple) and len(inputs) == 6: # multi-audio
has_multi_audios = True

Check notice

Code scanning / CodeQL

Unused local variable Note

Variable has_multi_audios is not used.

Copilot Autofix AI 2 days ago

The best way to fix the problem is to remove the unused variable has_multi_audios. This will clean up the code and eliminate the unnecessary assignment. Specifically, we need to remove the line where has_multi_audios is assigned a value and any related code that references it.

Suggested changeset 1
nemo/collections/speechlm/utils/text_generation/audio_text_generation_utils.py

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/nemo/collections/speechlm/utils/text_generation/audio_text_generation_utils.py b/nemo/collections/speechlm/utils/text_generation/audio_text_generation_utils.py
--- a/nemo/collections/speechlm/utils/text_generation/audio_text_generation_utils.py
+++ b/nemo/collections/speechlm/utils/text_generation/audio_text_generation_utils.py
@@ -195,3 +195,3 @@
     context_start_idx = None
-    if has_multi_audios:
+    if num_audios is not None:
         num_audios = torch.empty(batch_size, dtype=torch.int64, device=torch.cuda.current_device())
@@ -367,3 +367,2 @@
     tokenizer = model.tokenizer
-    has_multi_audios = False
     num_audios = None
@@ -377,3 +376,2 @@
     elif isinstance(inputs, tuple) and len(inputs) == 6:  # multi-audio
-        has_multi_audios = True
         (
EOF
@@ -195,3 +195,3 @@
context_start_idx = None
if has_multi_audios:
if num_audios is not None:
num_audios = torch.empty(batch_size, dtype=torch.int64, device=torch.cuda.current_device())
@@ -367,3 +367,2 @@
tokenizer = model.tokenizer
has_multi_audios = False
num_audios = None
@@ -377,3 +376,2 @@
elif isinstance(inputs, tuple) and len(inputs) == 6: # multi-audio
has_multi_audios = True
(
Copilot is powered by AI and may make mistakes. Always verify output.
Positive Feedback
Negative Feedback

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

Please select one or more of the options
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@stevehuang52 stevehuang52

Labels
common Multi Modal skip-linting
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@stevehuang52 @artbataev @ko3n1g