Enable TAP for ESO module

[juanmcloaiza]

Addresses #3138

Changes:

• query_main
• list_instruments
• query_instruments
• list_surveys
• query_surveys
• query_apex_quicklooks
• deprecate open_form and cache arguments
• beta test

[pep8speaks]

Hello @juanmcloaiza! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2025-01-17 14:13:03 UTC

[codecov]

Codecov Report

Attention: Patch coverage is 81.40351% with 53 lines in your changes missing coverage. Please review.

Project coverage is 75.19%. Comparing base (fd08e21) to head (6b86480).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
astroquery/eso/core.py	73.22%	49 Missing ⚠️
astroquery/eso/utils.py	96.03%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3141      +/-   ##
==========================================
+ Coverage   69.87%   75.19%   +5.31%     
==========================================
  Files         232        3     -229     
  Lines       19770      520   -19250     
==========================================
- Hits        13815      391   -13424     
+ Misses       5955      129    -5826     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bsipocz]

One comment for now regarding the deprecation, otherwise is looking good so far (I didn't dive into the details, but had a quick look

[juanmcloaiza]

@bsipocz, hi, I'd like to ask for your feedback regarding deprecated functionality.

This PR's scope is to switch the back-end of the ESO module from WDB to TAP. In doing so, many of the functions will be renamed and their signatures changed (i.e. completely new functions are being written, while others are being removed).

If I understand correctly, astroquery requires that deprecated features are handled via @deprecated decorators. However, in this particular case, the changes are rather large --to some extent a @deprecated decorator would need to be added to the whole ESO module (take this last phrase with a grain of salt, I'm exaggerating).

The question I'd like to ask you as an astroquery maintainer is the following: Would it be admissible for this time to avoid adding deprecated decorators to each of the functions and arguments that we are discontinuing? My impression is that the @deprecated benefits will be small in comparison to the illegibility of the code they'll produce. I would prefer instead to make it very explicit in the documentation that whatever functionality available in past versions may not be available from this version on.

Please let us know what you think (cc @szampier). Thanks in advance for your feedback ;)

[bsipocz]

Given this is a full refactoring and switching out what backend to use on the server side, it should be OK to do it without decorating with deprecations and remove methods that don't make sense any more.
However, the ones that used to be around should issue deprecation for keywords that got removed, etc. Similarly, methods that got renamed should maybe stay around for a release cycle as an alias of the new methods.

[bsipocz]

Also let me know when I should come in and do a first round of review. I already noticed a few things that are odd and would preferably done differently, e.g. the custom caching functions (e.g. it would be nice to either reuse what we already have, or generalize these and put it into a more central location). Also, I see some workaround for the tests that preferably should be resolved, e.g. don't filter warnings but if they are legit, then check if they are raised, also, empty tests should not be around for the sake of fooling codecov (you can just ignore the report for now, it doesn't report on the remote tests)

[juanmcloaiza]

Hi @bsipocz , thanks for the feedback. I reply each of your points here:

• deprecation warnings - all clear, I'll do it as you suggest.
• fooling codecov - You are completely right. I wanted to understand how codecov worked and forgot to remove those tests. The nice thing I learned is that codecov does actually tell you, line by line, whether a line was executed by your test suite, so no way to fool it by writing empty tests - https://docs.codecov.com/docs/about-code-coverage. Really nice tool, it did help me to write new real and legit offline tests and now the report looks much better.
• filtering warnings - I'm afraid that what you suggest was my original intention, but for some function calls inside loops, the warning would be raised, while for some others it wouldn't. To do things properly, I have created specific tests to check that the warning is raised when it should, and not raised when it shouldn't.
• custom caching functions - I didn't find a way to reuse the caching functions available, but the ones I wrote are largely based on them. Since we want to keep this PR minimal, I will remove cache functionality for the time being. However, these functions to cache TAP queries could be useful. Should I open a new PR aiming to add them to a more central location?

I will ping you again when we're ready with our internal feedback rounds, so you can go through it more thoroughly. Thanks again for your preliminary feedback! 👍

[juanmcloaiza]

@bsipocz , hi, one quick question. Preparing for the future, we are thinking on developing ESO submodules, so that we can query ESO data in the following way:

import astroquery.eso.observations as observations # Current Pull Request
import astroquery.eso.catalogues as catalogues # To be developed in the future

raw_datasets = observations.list_raw_instruments()
# >>> ['MUSE', 'VVV', 'SPHERE', ... ]

# Query raw data:
raw_data = observations.Raw() # Raw data from table dbo.raw
my_astropy_table = raw_data.query(instrument="MUSE")

# Query raw data from an IST:
muse_data = observations.Raw("MUSE") # Raw data from table ist.muse
my_astropy_table = muse_data.query(...)

# Query Phase 3 data:
processed_data = observations.Processed() # Processed data from table ivoa.ObsCore
my_astropy_table = processed_data.query(collections=["MUSE"])

Would this be OK for astroquery standards?

[keflavich]

I'd recommend that some of what you're proposing should be keywords. e.g. observations.list_instruments(processed_status='raw') instead of list_raw_instruments.

If you're going to use observations.Raw("MUSE"), then keep the symmetry and use observations.Processed("MUSE") too.

Other archives generally split their queries by Observation first, Processed State next (e.g., MAST). Is there a structural reason you need to split by processed state first, or could you also support queries that return everything associated with a given observation, then filter from there? If you can support both ways, I think that's better - the average user will not want raw data, so your approach looks possibly better.

[bsipocz]

I would really like to keep the facility name in the name of classes, etc, so while submodules are fine I think it would be better to keep calling them something ESO.

import astroquery.eso.observations as EsoObservations # Current Pull Request
import astroquery.eso.catalogues as EsoCatalogues # To be developed in the future

(I know that mast is an outlier in this sense, and then gemini copied their approach, but I haven't given up on getting the renamed done as outlined in #1748)

[keflavich]

Strongly agree @bsipocz . What you've suggested is a documentation (suggested usage) change, right? i.e., import astroquery.eso.Observations as EsoObservations is how we'd recommend users import the module, but it wouldn't require any code changes?

[bsipocz]

i.e., import astroquery.eso.Observations as EsoObservations is how we'd recommend users import the module, but it wouldn't require any code changes?

No, I would strongly recommend to rename the class itself.

[juanmcloaiza]

Hi @bsipocz and @keflavich , thanks for the attention given to my question. My takeaway is as follows:

• Keep the facility name in the name of classes, etc; submodules are fine --> Cool, thanks! 👍

To quickly answer your questions:

• Is there a structural reason you need to split by processed state first? --> Yes
• Could you also support queries that return everything associated with a given observation? --> No

Not to deviate the discussion, I'm not going into more detailed answers, but am happy to discuss further elsewhere if you're interested. For the rest of your comments, I don't have a clear answer now but they gave important insights as to what external users expect. We will discuss internally and come back with a solution. Thanks a lot! 👍

[juanmcloaiza]

@bsipocz, @keflavich, hi, thanks again for the discussion last time.

We will implement a new interface, but to streamline the review process, we won't do it in this pull request. I will open a new one shortly.

@bsipocz, could you please start giving a more thorough look at this pull request? @szampier is also looking at it in parallel, but we don't expect large changes.

Thanks in advance, looking forward to your feedback!

[juanmcloaiza]

I have some minor suggestions on the docs. I didn't review the main code in detail - I just assume it works.

A question, though: Why did you change the docstring format from numpy format to the current one? I haven't checked the docs build, but I'm not sure these will render correctly. If they do, no problem from my end (though I will let @bsipocz say if the style clash is too great), but I'd like to verify that.

@keflavich , thanks for the comments. About the docstrings:

I asked you-know-who to format them in "reStructuredText Sphinx style" as opposed to NumPy or Google styles because we are using Sphinx (https://sphinx-rtd-tutorial.readthedocs.io/en/latest/docstrings.html), so it renders nicely (perhaps a bit packed for some tastes), but I must admit I didn't pay attention to the convention followed in the rest of the modules beforehand. I am adding here two screenshots with the renders of the docstrings written in the two different styles:

reStructuredText Sphinx style -- compact, with bullet points and links to datatypes docs

NumPy style -- not so cluttered; each parameter is a "subsection" rather than a bullet point.

I prefer the Sphinx style, but have no strong opinions at all, so we can push whatever style you and @bsipocz decide (changing the docstring style is basically 1-click away from my side), just let me know.

[keflavich]

The way the docstrings get formatted in the end should be independent of how they're input, but you've shown that's not the case.

In any case, my concern is that the docstrings are consistently formatted internally because mixing formats in a package is a bad idea (doubles the possibility of bugs, and is definitely unexpected for the end user). So please use the format that looks like this:

        Parameters
        ----------
        survey : string or list
            Name of the survey(s) to query.  

[bsipocz]

First round without looking into the content, it's just about what unrelated changes need to be removed, and some data file bookkeeping comments.

[bsipocz]

Please don't add any changes to the CI or other config files, and remove all these changes from the history.

[bsipocz]

please don't store data files as pickles, write them out to a data file format (anything you would expect the server to return), but never a pickle.

If possible remove these from the history, too.

[bsipocz]

remove these changes from the history. You can run just ESO test with pytest out of the box as pytest -P eso -R which will pick up only the ESO tests.

(this is also mentioned in the docs, but we may need to add a bit more explanation to that if you think it would have helped you to find the correct command: https://astroquery.readthedocs.io/en/latest/testing.html#running-only-the-remote-data-tests)

[bsipocz]

remove these changes from the history

[bsipocz]

yes, please follow numpydoc formatting. Also, I wonder if this is ready for review, as there are commits still coming in.

Also, I wonder if you would consider reorging a commits with a bit of cleanup would be useful as there are a lot going on through the 160+ commits (see my comments about changes to the CI and configs, none of those should be in this PR or in fact left in the history).