Add NearSwap

[popbyte]

NearSwap Algorithm

NearSwap retains most of the weights of the base model, but when a weight is similar between the two, it is interpolated to the secondary model value. A parameter t specifies the sameness threshold. When the distance between two values is below t, the weight from the secondary model is used.

This PR implements the NearSwap algorithm from here

[popbyte]

CI should now pass, please review

[ElliotStein]

Thanks for the PR! I’ve made a couple of changes:
1. I swapped out NumPy for PyTorch to avoid unnecessary CPU-GPU data transfers.
2. I restored the lweights = t / lweights line, which I believe is necessary since the t parameter wasn’t being used otherwise.

This looks like a solid implementation of the algorithm described at QuartetAnemoi-70B-t0.0001.

To clarify the algorithm (Given base_model and secondary_model):
• Use weights from secondary_model directly when they are close to those in base_model (within threshold t).
• Interpolate between base_model and secondary_model otherwise, with t/abs(V0-V1) as the scaling parameter.
• The further apart the weights are, the more the interpolation favours the base_model

Is that understanding correct?

[popbyte]

2. I restored the lweights = t / lweights line, which I believe is necessary since the t parameter wasn’t being used otherwise

Oops, that was accidentally removed.

Is that understanding correct?

Yes.

[cg123]

Thanks for the PR! I've updated it to work with the new merge method registry.