Hotswap allow different alpha scalings and ranks

[BenjaminBossan]

WIP

See huggingface/diffusers#9453 for context.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[BenjaminBossan]

not stale

[BenjaminBossan]

@sayakpaul I had some time to brush up this PR. How do we proceed? As you can see, I have added diffusers tests, but since huggingface/diffusers#9453 is not merged, it won't work yet (i.e. there are recompiles). That PR is awaiting this PR, so there is a mutual dependency.

Also, for now Conv2d is not fully implemented, but to confirm that it works, we could test with only Linear, right?

[sayakpaul]

Also, for now Conv2d is not fully implemented, but to confirm that it works, we could test with only Linear, right?

That should work for now. I would add a note about this somewhere.

@sayakpaul I had some time to brush up this PR. How do we proceed? As you can see, I have added diffusers tests, but since huggingface/diffusers#9453 is not merged, it won't work yet (i.e. there are recompiles).

To make progress and help unblock this PR, how about we merge this PR with sufficient testing (which I believe we already have) and mark the diffusers test with an xfail?