Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify the increasingly convoluted selection mechanism #363

Open
JSKenyon opened this issue Feb 25, 2025 · 0 comments
Open

Simplify the increasingly convoluted selection mechanism #363

JSKenyon opened this issue Feb 25, 2025 · 0 comments
Assignees

Comments

@JSKenyon
Copy link
Collaborator

The reopening of #226 has made it clear that the current approach to selecting DDIDs and fields is somewhat error prone. This is due, in part, to the fact that it xds_from_table doesn't have an internal filtering mechanism (if we ignore TAQL). This means that in order to provide chunking information to those calls, we need to supply the chunking for all the resulting datasets, rather than just the ones we care about. This is not a huge problem, but the current code is somewhat sloppy and will likely be a problem again in the future.

There are two options to consider:

  • Start a PR on dask-ms to include some sort of filter/callable which deselects certain DDIDs/fields in the xds_from_table call.
  • Modify QuartiCal's chunking code to be smarter e.g. provide dummy chunks for deselected fields/DDIDs. QuartiCal should also be much more explicit about the mapping from xarray dataset to chunk specification. Currently, this is positional and will likely be the source of future errors.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant