Euclid: new method to retrieve scientific LE3 products #3313

cosmoJFH · 2025-05-07T10:02:18Z

Dear Astroquery team,

we would like to include a new method in the Euclid project to retrieve scientific LE3 products. This method is similar (in scope) to the following methods:

get_product_list: It retrieves an Astropy Table object with a list of .FITS file names (i.e., products) and other relevant metadata that can be used to download the individual FITS files.
get_product: It retrieves a product (i.e., a .FITS file), with data that can be analysed.
get_observation_products: It retrieves a product (i.e., a .FITS file), with data that can be analysed.

We have also developed new tests.

cc @esdc-esac-esa-int

jira: EUCLIDMNGT-1275

… Pre-recon', into 'SEL Wide'

…lity

codecov · 2025-05-07T11:06:41Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.97%. Comparing base (79f30f8) to head (f9bed5d).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3313      +/-   ##
==========================================
+ Coverage   69.87%   69.97%   +0.09%     
==========================================
  Files         232      232              
  Lines       19757    19818      +61     
==========================================
+ Hits        13806    13867      +61     
  Misses       5951     5951

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

bsipocz

Overall looks good with some minor, mostly documentation-related comments.

Also, I would think a rebase would be great to remove the merge commit as well as the unrelated changes in e.g. the changelog commit.

bsipocz · 2025-05-07T15:11:32Z

astroquery/esa/euclid/core.py

+        product_type : str, optional, default None
+            Available product types per category and group:
+
+                #. Internal Data Products


This ~200 lines feels to be a lot, maybe we should instead refer to a docs URL in this docstring?

The Euclid documentation is pretty complex, so we tried to provide a simple description of the available parameters that could be used. Although this information was extracted from the Euclid documentation, it is not straight forward to extract the category/group/product_type for all the LE3 types.

In any case, I will ask the Euclid team at ESAC.

Just to be clear, my suggestion is to put this into the narrative docs and refer to that from the docstring.

(given the overall complexity, I think that's a cleaner place for the users to learn more about the options, also it's more possible to give more context about the options there)

We have moved the information of this method to the documentation of the Euclid module. We have included a table at the end of the document and included a reference to it in the method.

bsipocz · 2025-05-07T15:15:21Z

docs/esa/euclid/euclid.rst

+    >>> le3_product_list = Euclid.get_scientific_product_list(tile_index=102034414)
+    >>> print("Found", len(le3_product_list), "results")
+    Found 12 results
+    >>> print(le3_product_list)


add the table printout here, or remove the print line

Done. The documentation was updated with the output of the functions.

bsipocz · 2025-05-07T15:15:39Z

docs/esa/euclid/euclid.rst

+    >>> results = euclid.get_scientific_product_list(category='Weak Lensing Products', group='2PCF', product_type='DpdTwoPCFWLShearShear2D')
+    >>> print("Found", len(le3_product_list), "results")
+    Found 12 results
+    >>> print(le3_product_list)


same, either remove the print line, or add an output

Done. The documentation was updated with the output of the functions.

bsipocz · 2025-05-07T15:23:30Z

astroquery/esa/euclid/core.py

@@ -1276,5 +1277,411 @@ def get_datalinks(self, ids, *, linking_parameter='SOURCE_ID', verbose=False):

        return self.__eucliddata.get_datalinks(ids=ids, linking_parameter=linking_parameter, verbose=verbose)

+    def get_scientific_product_list(self, *, observation_id=None, tile_index=None, category=None, group=None,


This is a small side note from someone not exactly familiar with Euclid data products and use cases, so I could easily be in the wrong:

I'm not sure about the naming, isn't science products returned with get_product_list, too? maybe swap scientific to highlevel or something else? Or do you expect the users mostly using this method to access data?

I will ask to the Euclid team at ESAC.

Eventually, we may revise the names of the methods in the EuclidClass to e.g., homogenize the method names and reduce their length - but that is well beyond the scope of this task. So we would leave the present name of this method.

OK, that sounds good.

bsipocz · 2025-05-07T15:28:31Z

docs/esa/euclid/euclid.rst

@@ -240,7 +240,30 @@ To get the list of products associated with a given Euclid observation_id or til
  EUC_MER_BGSUB-MOSAIC-DES-Z_TILE102018211-83C32F_20241018T143526.104818Z_00.00.fits               1398  102018211           DECAM     DECAM_z  SCIENCE         SKY 57.9990741   -51.5     IMAGE


-The method returns a list of products as an astropy.table.
+The method returns a list of products as an astropy.table. It is also possible to search by observation_id, but not by both parameters simultaneously.


Suggested change

The method returns a list of products as an astropy.table. It is also possible to search by observation_id, but not by both parameters simultaneously.

The method returns a list of products as an `~astropy.table.Table`. It is also possible to search by observation_id, but not by both parameters simultaneously.

Documentation updated.

bsipocz · 2025-05-07T15:30:55Z

astroquery/esa/euclid/core.py

+
+        Returns
+        -------
+        The list of products (astropy.table)


I would suggest avoid using 'list' to describe a table as it's slightly misleading

Suggested change

The list of products (astropy.table)

The products in an astropy.table.Table

Changes committed to the branch.

…duct_list to the documentation

Jorge Fernandez Hernandez added 19 commits March 23, 2025 08:41

EUCLIDMNGT-1275 New method get_scientific_data_product_list

f294440

EUCLIDMNGT-1275 New method get_scientific_data_product_list

b021e33

EUCLIDMNGT-1275 Update example

f89022b

EUCLIDMNGT-1275 Update product types and documentation

27313a6

EUCLIDMNGT-1275 Update product types and documentation

5105576

EUCLIDMNGT-1275 New category PHZ

0cc0da1

EUCLIDMNGT-1275 New category PHZ

3d135dd

EUCLIDMNGT-1275 sort categories in the help

aa5864e

EUCLIDMNGT-1275 rename parameter dataset to dataset_release

3b24cd3

EUCLIDMNGT-1275 rename parameter dataset to dataset_release

0e7e4d0

EUCLIDMNGT-1275 Update help info

c6dd7e5

EUCLIDMNGT-1275 Remove wrong files

e0278bc

EUCLIDMNGT-1275 Update implementation

19b9c05

EUCLIDMNGT-1275 Update help info

5b47cbe

EUCLIDMNGT-1275 Update variable names

bd5f70c

EUCLIDMNGT-1275 Update comments for Q1 data release

a2ce748

EUCLIDMNGT-1275 merge categories 'SEL 'Wide Post-recon' and 'SEL Wide…

c2374d2

… Pre-recon', into 'SEL Wide'

EUCLIDMNGT-1275 final changes from the archive scientists

70ffeff

EUCLIDMNGT-1275 include PR reference

8c96881

cosmoJFH force-pushed the ESA_euclid_EUCLIDMNGT-1275_LE3_search_capability branch from 5ec7e07 to 8c96881 Compare May 7, 2025 10:34

cosmoJFH and others added 4 commits May 7, 2025 12:35

Merge branch 'main' into ESA_euclid_EUCLIDMNGT-1275_LE3_search_capabi…

e4ed3d2

…lity

EUCLIDMNGT-1275 Fix codestyle: trailing whitespace

ff97448

EUCLIDMNGT-1275 Include PR reference

c470938

EUCLIDMNGT-1275 Remove duplicated information

f89cdba

cosmoJFH force-pushed the ESA_euclid_EUCLIDMNGT-1275_LE3_search_capability branch from 3a2b771 to f0ee810 Compare May 7, 2025 11:24

EUCLIDMNGT-1275 Fix indentation

ea52f13

cosmoJFH force-pushed the ESA_euclid_EUCLIDMNGT-1275_LE3_search_capability branch from 92cdb46 to ea52f13 Compare May 7, 2025 11:47

bsipocz added the esa.euclid label May 7, 2025

bsipocz added this to the v0.4.11 milestone May 7, 2025

bsipocz reviewed May 7, 2025

View reviewed changes

Jorge Fernandez Hernandez added 3 commits May 7, 2025 20:31

EUCLIDMNGT-1275 Include changes suggested by the astroquery team

c861abc

EUCLIDMNGT-1275 Fix bug to retrieve data by group

229c74d

EUCLIDMNGT-1275 move the description of the method get_scientific_pro…

f9bed5d

…duct_list to the documentation

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Euclid: new method to retrieve scientific LE3 products #3313

Euclid: new method to retrieve scientific LE3 products #3313

cosmoJFH commented May 7, 2025 •

edited

Loading

codecov bot commented May 7, 2025 •

edited

Loading

bsipocz left a comment

bsipocz May 7, 2025

cosmoJFH May 7, 2025

bsipocz May 7, 2025

bsipocz May 7, 2025

cosmoJFH May 9, 2025 •

edited

Loading

bsipocz May 7, 2025

cosmoJFH May 7, 2025

bsipocz May 7, 2025

cosmoJFH May 7, 2025

bsipocz May 7, 2025

cosmoJFH May 7, 2025

cosmoJFH May 9, 2025

bsipocz May 10, 2025

bsipocz May 7, 2025

cosmoJFH May 7, 2025

bsipocz May 7, 2025

cosmoJFH May 7, 2025

		@@ -1276,5 +1277,411 @@ def get_datalinks(self, ids, *, linking_parameter='SOURCE_ID', verbose=False):

		return self.__eucliddata.get_datalinks(ids=ids, linking_parameter=linking_parameter, verbose=verbose)

		def get_scientific_product_list(self, *, observation_id=None, tile_index=None, category=None, group=None,

	The method returns a list of products as an astropy.table. It is also possible to search by observation_id, but not by both parameters simultaneously.
	The method returns a list of products as an `~astropy.table.Table`. It is also possible to search by observation_id, but not by both parameters simultaneously.

	The list of products (astropy.table)
	The products in an astropy.table.Table

Euclid: new method to retrieve scientific LE3 products #3313

Are you sure you want to change the base?

Euclid: new method to retrieve scientific LE3 products #3313

Conversation

cosmoJFH commented May 7, 2025 • edited Loading

codecov bot commented May 7, 2025 • edited Loading

Codecov Report

bsipocz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cosmoJFH May 9, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cosmoJFH commented May 7, 2025 •

edited

Loading

codecov bot commented May 7, 2025 •

edited

Loading

cosmoJFH May 9, 2025 •

edited

Loading