Add a "drop_similar" argument to the TableVectorizer

[GaelVaroquaux]

It would be great to make it really easy drop the redundant columns in the TableVectorizer (Using the DropSimilar transformer inside the TableVectorizer), by a simple additional argument.

This would have improvements both is speed/memory and maybe in statistical performance.

I realize that it makes the TableVectorizer even more a swiss-army knife than it currently is, but honestly, it's sooo useful and we use it everywhere, even as an element in complex pipelines.

[Neilblaze]

@GaelVaroquaux is this up for grabs? If yes, I'd like to contribute! :)
I have experience with skrub, but this will be my first time contributing to the project itself, so I may ping you in this issue thread itself if I hit any roadblocks. Thanks!

[Neilblaze]

@GaelVaroquaux does this change look good? Tests are passing although I can see UserWarning warnings.

[GaelVaroquaux]

Hi @Neilblaze ,

This is still up for grabs, and the change that you suggest are totally going in the right direction.

However, unless I am wrong, we don't have the DropSimilar transformer yet. It is issue #1001 , the parent issue. That issue would need to be tackled first, and I suggest that we go through the whole process of tackling it and merging it before moving on with this one. Understanding the process will then make you more productive.

Thanks!!

[Neilblaze]

@GaelVaroquaux Makes sense and thanks for the update! I'll keep track of #1001 for now, and hopefully when it gets merged into the main, we can continue with this.

[GaelVaroquaux]

There is no PR for #1001 for now. I'm not sure when we'll get it done. We'd love to, but we're shuffling so many things 😁

[Neilblaze]

Yep, I saw that. So for the meantime, I'll try to grab some other open issues. Will ping here/on discord if I get stuck anywhere, and thanks! 😃