[lora]feat: use exclude modules to loraconfig. #11806
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
sayakpaul
changed the title
| pipe_cp, _ = self.check_if_adapters_added_correctly( | ||
| pipe_cp, text_lora_config=text_lora_config, denoiser_lora_config=denoiser_lora_config | ||
| ) | ||
| denoiser_exclude_modules = self._get_exclude_modules(pipe_cp) |
There was a problem hiding this comment.
I'm afraid I don't fully understand how this test works but that shouldn't be a blocker. If this tests the exact same condition we're facing in the FusionX lora, then it should be good :)
There was a problem hiding this comment.
No, not the cleanest test this one. I added a description in the test, see if that helps?
Currently, we don't have:
If this tests the exact same condition we're facing in the FusionX lora
I will think of a way to include a test for that, too.
| string_to_replace = f"{adapter_name}." if adapter_name else "" | ||
|
|
||
| for name in model_state_dict.keys(): | ||
| if string_to_replace: |
There was a problem hiding this comment.
maybe the if-statement is not needed here because string_to_replace will be an empty string, but no problem keeping as micro optimization
| exclude_modules = _derive_exclude_modules(model_state_dict, peft_state_dict, adapter_name) | ||
| if exclude_modules: | ||
| if not is_peft_version(">=", "0.14.0"): | ||
| msg = "It seems like there are certain modules that need to be excluded when initializing `LoraConfig`. Your current `peft` version doesn't support passing an `exclude_modules` to `LoraConfig`. Please update it by running `pip install -U peft`." |
There was a problem hiding this comment.
Just wondering: The exclude_modules are derived from all modules - modules of peft_state_dict, right? So in practice, exlcude_modules would never be empty. Thus, users with old PEFT versions will always see this warning, even though in the vast majority of cases, there is nothing to worry about. I wonder exclude_modules should just be ignored without any warning in that case.
There was a problem hiding this comment.
Or a debug() info with this message with an amendment saying "For majority use cases, this should be okay. But if you notice anything unexpected, please file an issue."
What does this PR do?
If we try to do:
It prints:
Unfold
Loading adapter weights from state_dict led to missing keys in the model: vace_blocks.0.proj_out.lora_A.default_0.weight, vace_blocks.0.proj_out.lora_B.default_0.weight, vace_blocks.1.proj_out.lora_A.default_0.weight, vace_blocks.1.proj_out.lora_B.default_0.weight, vace_blocks.2.proj_out.lora_A.default_0.weight, vace_blocks.2.proj_out.lora_B.default_0.weight, vace_blocks.3.proj_out.lora_A.default_0.weight, vace_blocks.3.proj_out.lora_B.default_0.weight, vace_blocks.4.proj_out.lora_A.default_0.weight, vace_blocks.4.proj_out.lora_B.default_0.weight, vace_blocks.5.proj_out.lora_A.default_0.weight, vace_blocks.5.proj_out.lora_B.default_0.weight, vace_blocks.6.proj_out.lora_A.default_0.weight, vace_blocks.6.proj_out.lora_B.default_0.weight, vace_blocks.7.proj_out.lora_A.default_0.weight, vace_blocks.7.proj_out.lora_B.default_0.weight.It happens because
target_modulesinLoraConfigtreats the values intarget_modulesas suffixes. In this case,proj_outis a part oftarget_moduleswhich is whyvace_blocks_*get targeted here (and hence the missing warning).LoraConfigallows us to specifyexclude_modules, too. This PR introduces support to add it when initializing theLoraConfig.@apolinario does it work for you?