Adding Principal Covariates Classification (PCovC) Code #248

rvasav26 · 2025-05-14T17:38:19Z

Adding PCovC code, examples, and testing suite. Adding a PCov base class that both PCovR and PCovC inherit from.

Contributor (creator of PR) checklist

Tests updated (for new features and bugfixes)?
Documentation updated (for new features)?
Issue referenced (for PRs that solve an issue)?

For Reviewer

CHANGELOG updated if important change?

📚 Documentation preview 📚: https://scikit-matter--248.org.readthedocs.build/en/248/

…rom it

New PR for exclusively adding PCovC code

rosecers

Overall this is looking very good. There's some standardizing tasks I noted and a couple style/organization questions I'd like discussed before approving.

src/skmatter/decomposition/_kernel_pcovr.py

src/skmatter/decomposition/_pcov.py

examples/pcovc/PCovC-BreastCancerDataset.ipynb

rosecers · 2025-05-15T18:31:04Z

src/skmatter/decomposition/_pcovc.py

+        W : numpy.ndarray, shape (n_features, n_properties)
+            Classification weights, optional when classifier=`precomputed`. If
+            not passed, it is assumed that the weights will be taken from a
+            linear classifier fit between X and Y


There should be a note here or above about the relationship between X, Y, Z, and W.

Left a more detailed note in the "Notes" section of the docstring

rosecers · 2025-05-15T18:31:45Z

src/skmatter/decomposition/_pcovc.py

+            classifier = LogisticRegression()
+            if W is None:
+                W = LogisticRegression().fit(X, Y).coef_.T
+                W = W.reshape(X.shape[1], -1)


Codecov says this is not covered

Should be covered now - added test case to test_pcovc.py's test_precomputed_classification

docs/src/references/index.rst

docs/src/references/pcovr_decomposition.rst

src/skmatter/decomposition/_pcovr.py

src/skmatter/utils/_pcovc_utils.py

PicoCentauri

Thanks for this nice new feature!

I took a very quick look. Once the example is a clean Python file I can check more carefully.

PicoCentauri · 2025-05-22T10:05:36Z

examples/pcovc/PCovC-BreastCancerDataset.ipynb

This has to be python example file as the others so be rendered properly on the website. Take a look at the example folder how we do format a file. I can also help if there are issues.

Should be working now! I think we will go in and polish these up in a future update but for now we have two brief examples

PicoCentauri · 2025-05-22T10:08:25Z

src/skmatter/decomposition/_pcov.py

+        self,
+        mixing=0.5,
+        n_components=None,
+        svd_solver="auto",
+        tol=1e-12,
+        space="auto",
+        iterated_power="auto",
+        random_state=None,
+        whiten=False,


Did we decide if we are starting to use type hints or not?

This code is taken from the original _pcovr.py, @rosecers or @cajchristian may be able to better answer this

I think we can have a later conversation about type hints, but the general "scikit-learn does not do it" points to no at this time.

PicoCentauri · 2025-05-22T10:09:23Z

src/skmatter/decomposition/_pcov.py

+        self.fit_svd_solver_ = self.svd_solver
+        if self.fit_svd_solver_ == "auto":
+            # Small problem or self.n_components_ == 'mle', just call full PCA
+            if max(X.shape) <= 500 or self.n_components_ == "mle":


Why the value of 500? Does it make sense to hardcode this?

See above comment

PicoCentauri · 2025-05-22T10:10:08Z

src/skmatter/decomposition/_pcov.py

+        S_sqrt = np.diagflat([np.sqrt(s) if s > self.tol else 0.0 for s in S])
+        S_sqrt_inv = np.diagflat([1.0 / np.sqrt(s) if s > self.tol else 0.0 for s in S])


Maybe move the list comprehension as extra variable above?

See above comment

src/skmatter/decomposition/_pcov.py

PicoCentauri

Without checking the math carefully the overall design looks good.

Can you update the CHANGELOG file in the root of the repo with the addition you made. Thanks

PicoCentauri · 2025-05-27T19:22:54Z

docs/src/references/decomposition.rst

@@ -1,5 +1,5 @@
-Principal Covariates Regression (PCovR)
-=======================================
+Principal Covariates Regression (PCovR) and Classification (PCovC)


Maybe find a shorter name. This is making the sidebar in the docs super larger

How about :

Suggested change

Principal Covariates Regression (PCovR) and Classification (PCovC)

Hybrid Mapping Techniques (PCovR and PCovC)

PicoCentauri

Thanks LGTM. If @rosecers is also happy we can merge.

cajchristian · 2025-05-28T20:18:43Z

docs/src/bibliography.rst

@@ -45,3 +45,9 @@ References
    Michele Ceriotti, "Improving Sample and Feature Selection with Principal Covariates
    Regression" 2021 Mach. Learn.: Sci. Technol. 2 035038.
    https://iopscience.iop.org/article/10.1088/2632-2153/abfe7c.
+
+.. [Jorgensen2025]
+    Christian Jorgensen, Arthur Y. Lin, and Rose K. Cersonsky,


Missing an author!!!

Suggested change

Christian Jorgensen, Arthur Y. Lin, and Rose K. Cersonsky,

Christian Jorgensen, Arthur Y. Lin, Rhushil Vasavada, and Rose K. Cersonsky,

cajchristian · 2025-05-28T20:28:11Z

examples/pcovc/PCovC-BreastCancerDataset.py

+print(load_breast_cancer().DESCR)
+
+# %%
+#
+# Scale Feature Data
+# ------------------
+#
+# Below, we transform the Breast Cancer feature data to have a mean of zero
+# and standard deviation of one, while preserving relative relationships
+# between feature values.
+


Can probably delete these lines

Suggested change

print(load_breast_cancer().DESCR)

# %%

#

# Scale Feature Data

# ------------------

#

# Below, we transform the Breast Cancer feature data to have a mean of zero

# and standard deviation of one, while preserving relative relationships

# between feature values.

cajchristian · 2025-05-28T20:30:18Z

examples/pcovc/PCovC-BreastCancerDataset.py

+# coding: utf-8
+
+"""
+PCovC with the Breast Cancer Dataset


Suggested change

PCovC with the Breast Cancer Dataset

Comparing PCovC with PCA and LDA

cajchristian · 2025-05-28T20:32:20Z

examples/pcovc/PCovC-BreastCancerDataset.py

+# We use Principal Component Analysis to reduce the Breast Cancer feature
+# data to two features that retain as much information as possible
+# about the original dataset.


Can delete this too

Suggested change

# We use Principal Component Analysis to reduce the Breast Cancer feature

# data to two features that retain as much information as possible

# about the original dataset.

cajchristian · 2025-05-28T20:33:11Z

examples/pcovc/PCovC-BreastCancerDataset.py

+# Here, we use Linear Discriminant Analysis to find a projection
+# of the feature data that maximizes separability between
+# the benign/malignant classes.


Same

Suggested change

# Here, we use Linear Discriminant Analysis to find a projection

# of the feature data that maximizes separability between

# the benign/malignant classes.

cajchristian · 2025-05-28T20:39:02Z

examples/pcovc/PCovC-BreastCancerDataset.py

+
+fig, axis = plt.subplots()
+axis.scatter(-T_lda[:], np.zeros(len(T_lda[:])), c=y)
+


Similar to the sections above, let's add a section where we perform solely the PCovC projection of the data. Then we can get rid of the lower for loop.

cajchristian · 2025-05-28T20:41:36Z

examples/pcovc/PCovC-IrisDataset.py

+# coding: utf-8
+
+"""
+PCovC with the Iris Dataset


Suggested change

PCovC with the Iris Dataset

PCovC Hyperparameter Tuning

cajchristian · 2025-05-28T20:42:17Z

examples/pcovc/PCovC-IrisDataset.py

+print(load_iris().DESCR)
+
+# %%
+#
+# Scale Feature Data
+# ------------------
+#
+# Below, we transform the Iris feature data to have a mean of zero and
+# standard deviation of one, while preserving relative relationships
+# between feature values.
+


Can delete

Suggested change

print(load_iris().DESCR)

# %%

#

# Scale Feature Data

# ------------------

#

# Below, we transform the Iris feature data to have a mean of zero and

# standard deviation of one, while preserving relative relationships

# between feature values.

cajchristian · 2025-05-28T20:44:05Z

examples/pcovc/PCovC-IrisDataset.py

+# We use Principal Component Analysis to reduce the Iris feature
+# data to two features that retain as much information as possible
+# about the original dataset.


delete

Suggested change

# We use Principal Component Analysis to reduce the Iris feature

# data to two features that retain as much information as possible

# about the original dataset.

cajchristian · 2025-05-28T20:52:35Z

examples/pcovc/README.rst

Need to also include a description of PCovC here - this shows up in the "getting started" page of the docs

cajchristian · 2025-05-28T20:54:52Z

src/skmatter/decomposition/__init__.py


-__all__ = ["pcovr_covariance", "pcovr_kernel", "PCovR", "KernelPCovR"]


Is this functionality correct? Are pcovr_covariance and pcovr_kernel still callable?

cajchristian · 2025-05-28T21:02:30Z

src/skmatter/decomposition/_pcovc.py

+    classifier: {`LogisticRegression`, `LogisticRegressionCV`, `LinearSVC`, `LinearDiscriminantAnalysis`,
+        `RidgeClassifier`, `RidgeClassifierCV`, `SGDClassifier`, `Perceptron`, `precomputed`}, default=None
+        classifier for computing :math:`{\mathbf{Z}}`. The classifier should be one of
+        `sklearn.linear_model.LogisticRegression`, `sklearn.linear_model.LogisticRegressionCV`,
+        `sklearn.svm.LinearSVC`, `sklearn.discriminant_analysis.LinearDiscriminantAnalysis`,
+        `sklearn.linear_model.RidgeClassifier`, `sklearn.linear_model.RidgeClassifierCV`,
+        `sklearn.linear_model.SGDClassifier`, or `Perceptron`. If a pre-fitted classifier
+        is provided, it is used to compute :math:`{\mathbf{Z}}`.


This is not showing up correctly in the docs

cajchristian · 2025-05-28T21:13:27Z

src/skmatter/decomposition/_pcovc.py

+    def decision_function(self, X=None, T=None):
+        """Predicts confidence scores from X or T."""


Should touch this up further. Here's a start

Suggested change

def decision_function(self, X=None, T=None):

"""Predicts confidence scores from X or T."""

r"""Predicts confidence scores from X or T.

.. math::

\mathbf{Z} = \mathbf{T} \mathbf{P}_{TZ}

= \mathbf{X} \mathbf{P}_{XT} \mathbf{P}_{TZ}

Parameters

----------

X : ndarray, shape(n_samples, n_features)

Original data for which we want to get confidence scores, where n_samples is the number of samples and n_features is the number of features.

T : ndarray, shape (n_samples, n_components)

Projected data for which we want to get confidence scores, where n_samples is the number of samples

and n_components is the number of components.

Returns

--------

Z

cajchristian

We're almost there. I think the docs still need some work, but most of my changes are nitpicky. Also, I'm not seeing the examples show up in the section navigation dropdown.

cajchristian · 2025-05-28T21:22:49Z

Should we also change the PCovR section of the landing page to something hybrid learning-esque?

rvasav26 and others added 5 commits May 13, 2025 19:00

Adding PCovC code along with base class, modifying PCovR to inherit f…

1c3fab9

…rom it

Merge pull request #246 from rvasav26/adding-pcovc-new

513241b

New PR for exclusively adding PCovC code

Finalizing/touching up docs

7f24a7d

Fixing linting

0c841dd

Minor changes to examples, formatting

f807237

rvasav26 marked this pull request as ready for review May 14, 2025 17:39

rvasav26 requested review from rosecers and ceriottm May 14, 2025 17:40

Fixing docstrings to address docs build errors

c651e23

rvasav26 requested a review from cajchristian May 14, 2025 18:48

Fixing whitespace for linter

c0a16aa

rosecers force-pushed the adding-pcovc-new branch from e302dbd to c0a16aa Compare May 14, 2025 19:57

rosecers and others added 2 commits May 14, 2025 15:23

Adding pcovc to docs

f66aa9d

Making PCovC accessible via API reference on docs for now.

df8fa2e

rvasav26 force-pushed the adding-pcovc-new branch from 3995e16 to df8fa2e Compare May 15, 2025 02:34

rosecers reviewed May 16, 2025

View reviewed changes

rosecers requested a review from PicoCentauri May 16, 2025 16:41

Implementing Rosy's suggestions to code

f56ea06

rvasav26 requested a review from rosecers May 19, 2025 18:56

PicoCentauri reviewed May 22, 2025

View reviewed changes

Adding PCovC to docs examples

78e7051

PicoCentauri reviewed May 27, 2025

View reviewed changes

Updating CHANGELOG, changing PCovC fit() note

12a2c39

rvasav26 force-pushed the adding-pcovc-new branch from ddee0d5 to 12a2c39 Compare May 27, 2025 23:45

PicoCentauri approved these changes May 28, 2025

View reviewed changes

cajchristian reviewed May 28, 2025

View reviewed changes

		S_sqrt = np.diagflat([np.sqrt(s) if s > self.tol else 0.0 for s in S])
		S_sqrt_inv = np.diagflat([1.0 / np.sqrt(s) if s > self.tol else 0.0 for s in S])

	Principal Covariates Regression (PCovR) and Classification (PCovC)
	Hybrid Mapping Techniques (PCovR and PCovC)

	Christian Jorgensen, Arthur Y. Lin, and Rose K. Cersonsky,
	Christian Jorgensen, Arthur Y. Lin, Rhushil Vasavada, and Rose K. Cersonsky,

	PCovC with the Breast Cancer Dataset
	Comparing PCovC with PCA and LDA

	# We use Principal Component Analysis to reduce the Breast Cancer feature
	# data to two features that retain as much information as possible
	# about the original dataset.

	# Here, we use Linear Discriminant Analysis to find a projection
	# of the feature data that maximizes separability between
	# the benign/malignant classes.


		fig, axis = plt.subplots()
		axis.scatter(-T_lda[:], np.zeros(len(T_lda[:])), c=y)


		__all__ = ["pcovr_covariance", "pcovr_kernel", "PCovR", "KernelPCovR"]

		def decision_function(self, X=None, T=None):
		"""Predicts confidence scores from X or T."""

-    def decision_function(self, X=None, T=None):
-        """Predicts confidence scores from X or T."""
+r"""Predicts confidence scores from X or T.
+.. math::
+    \mathbf{Z} = \mathbf{T} \mathbf{P}_{TZ}
+                      = \mathbf{X} \mathbf{P}_{XT} \mathbf{P}_{TZ}
+        Parameters
+        ----------
+        X : ndarray, shape(n_samples, n_features)
+             Original data for which we want to get confidence scores, where n_samples is the number of samples and n_features is the number of features.
+        T : ndarray, shape (n_samples, n_components)
+            Projected data for which we want to get confidence scores, where n_samples is the number of samples
+            and n_components is the number of components.
+        Returns
+        --------
+        Z

Adding Principal Covariates Classification (PCovC) Code #248

Are you sure you want to change the base?

Adding Principal Covariates Classification (PCovC) Code #248

Uh oh!

Conversation

rvasav26 commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contributor (creator of PR) checklist

For Reviewer

Uh oh!

rosecers left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PicoCentauri left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rvasav26 May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

PicoCentauri left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PicoCentauri left a comment

Choose a reason for hiding this comment

Uh oh!

cajchristian May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cajchristian May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rvasav26 commented May 14, 2025 •

edited

Loading

rvasav26 May 23, 2025 •

edited

Loading

cajchristian May 28, 2025 •

edited

Loading

cajchristian May 28, 2025 •

edited

Loading

cajchristian left a comment •

edited

Loading