Implement gmt xarray BackendEntrypoint #3919

weiji14 · 2025-04-17T02:14:01Z

Description of proposed changes

Allow 'gmt' to be used as an engine in xarray.open_dataarray and xarray.open_dataset for decoding raster NetCDF (grid) or GeoTIFF (image) files!

TODO:

Initial implementation supporting kind="grid"
Let pygmt.io.load_dataarray() use engine="gmt" by default
Support raster_kind="image"
Documentation improvements

References:

Implements idea in #3673 (comment)

Preview: https://pygmt-dev--3919.org.readthedocs.build/en/3919/api/generated/pygmt.GMTBackendEntrypoint.html

Reminders

Run make format and make check to make sure the code follows the style guide.
Add tests for new features or tests that would have caught the bug that you're fixing.
Add new public functions/methods/classes to doc/api/index.rst.
Write detailed docstrings for all functions/methods.
If wrapping a new module, open a 'Wrap new GMT module' issue and submit reasonably-sized PRs.
If adding new functionality, add an example to docstrings or tutorials.

Slash Commands

You can write slash commands (/command) in the first line of a comment to perform
specific operations. Supported slash command is:

/format: automatically format and lint the code

Initial implementation of a 'gmtread' xarray BackendEntrypoint for decoding NetCDF files! Following instructions at https://docs.xarray.dev/en/v2025.03.1/internals/how-to-add-new-backend.html on registering a backend. Only a minimal implementation for now to read kind=grid data.

So that GMTDataArray accessor knows the path to the original NetCDF file to retrieve correct metadata.

Use gmtread as default engine instead of netcdf4.

Default is still kind='grid', by allow use of kind='image' too.

Users will need to set the 'kind' parameter explicitly to 'grid' or 'image'.

Also set `kind='grid'` argument in _load_earth_relief_holes

Need to set a default value of None for 'decode_kind' it seems, from reading https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html#open-dataset

Because xr.core.types.ReadBuffer is not available until xarray 2024.11.0, xref pydata/xarray#9787

Partially reverts b9e3eb4

weiji14 · 2025-04-17T06:39:00Z

Probably best to split this into two PRs so that we have two changelog entries:

(feature) Implement gmtread xarray BackendEntrypoint
- Changes to pyproject.toml, xarray_backend.py and tests/test_xarray_backend.py
**Breaking** Set load_dataarray's default engine to engine="gmtread", and require decode_kind parameter to be set (e.g. to decode_kind='grid')
- Changes to io.py, helpers/testing.py, datasets/samples.py, etc

But please take a look at the current implementation, and suggest comments first.

seisman · 2025-04-17T06:57:15Z

Probably best to split this into two PRs so that we have two changelog entries:

(feature) Implement gmtread xarray BackendEntrypoint

Changes to pyproject.toml, xarray_backend.py and tests/test_xarray_backend.py

Breaking Set load_dataarray's default engine to engine="gmtread", and require decode_kind parameter to be set (e.g. to decode_kind='grid')

Changes to io.py, helpers/testing.py, datasets/samples.py, etc

Sounds good to me.

But please take a look at the current implementation, and suggest comments first.

Here are my questions:

Why not just use "gmt" as the engine name?
The load_dataarray function is almost the same as the xr.load_dataarray function, except that load_dataarray also get the GMT accessor information. With the "gmtread" engine added, xr.load_dataarray("xxx.grd", engine="gmtread", decode_kind="grid") works exactly the same as load_dataarray("xxx.grd", decode_kind="grid"). So maybe we can deprecate load_dataarray?

weiji14 · 2025-04-17T07:16:51Z

Why not just use "gmt" as the engine name?

Yes, we could do that actually, and engine="gmt" will make more sense if we want to implement writing to a NetCDF driver later via xr.DataArray.to_netcdf.

The load_dataarray function is almost the same as the xr.load_dataarray function, except that load_dataarray also get the GMT accessor information. With the "gmtread" engine added, xr.load_dataarray("xxx.grd", engine="gmtread", decode_kind="grid") works exactly the same as load_dataarray("xxx.grd", decode_kind="grid"). So maybe we can deprecate load_dataarray?

True, the GMT accessor loading is already in the BackendEntrypoint, so we might as well deprecate load_dataarray. Ok to do it in 2 releases (remove in v0.18.0) or 4 releases (remove in v0.20.0)?

seisman · 2025-04-17T08:01:43Z

True, the GMT accessor loading is already in the BackendEntrypoint, so we might as well deprecate load_dataarray. Ok to do it in 2 releases (remove in v0.18.0) or 4 releases (remove in v0.20.0)?

4 releases sounds good to me.

pygmt/xarray_backend.py

seisman · 2025-04-17T08:15:55Z

pygmt/xarray_backend.py

+            ext = Path(filename_or_obj).suffix
+        except TypeError:
+            return False
+        return ext in {".grd", ".nc", ".tif"}


What about .tiff?

Suggested change

return ext in {".grd", ".nc", ".tif"}

return ext in {".grd", ".nc", ".tif", ".tiff"}

Ok, added .tiff. Any other known extensions to add? E.g. based on https://docs.generic-mapping-tools.org/6.5/reference/file-formats.html#grid-files?

pygmt/tests/test_xarray_backend.py

Co-authored-by: Dongdong Tian <[email protected]>

seisman · 2025-04-18T00:20:23Z

pygmt/io.py

@@ -2,10 +2,14 @@
 PyGMT input/output (I/O) utilities.
 """

+from typing import Literal


Since we're going to deprecate load_dataarray(xref: #3919 (comment)), we should revert the changes in this file.

Do we want to change the default engine in load_dataarray to engine="gmt" or no?

Changing the default engine to engine="gmt" means that users have to set decode_kind, so it's a breaking change. It makes little sense to introduce a breaking change to a deprecated function, so my answer is no.

Ok, I've reverted changes to io.py in commit f722b47. So keeping default engine behaviour as before for the load_dataarray function that will be deprecated in #3922.

seisman · 2025-04-18T00:21:17Z

pygmt/helpers/testing.py

@@ -154,7 +154,7 @@ def load_static_earth_relief():
        A grid of Earth relief for internal tests.
    """
    fname = which("@static_earth_relief.nc", download="c")
-    return load_dataarray(fname)
+    return load_dataarray(fname, decode_kind="grid")


Again, with load_dataarray deprecated, we should use xr.load_dataarray(fname, engine="gmt", decode_kind="grid") instead, but it can be done in a separate PR.

Started draft PR at #3922, will cherrypick things across.

seisman · 2025-04-18T00:32:49Z

doc/api/index.rst

@@ -173,6 +173,7 @@ Input/output
 .. autosummary::
    :toctree: generated

+    GMTBackendEntrypoint


I'm wondering if it makes more sense to have a separate category like "Xarray Integration" and put both GMTBackendEntrypoint and GMTDataArrayAccessor in the category. Also, if #3854 is implemented, GMTDataArrayAccessor no longer makes sense in the "Metadata" category.

Yeah, I was wondering about this too. I can put GMTBackendEntrypoint in a new "Xarray Integration" section as you suggested, and then we'll remove the I/O section in a separate PR.

Move GMTDataArrayAccessor to the "Xarray Integration" section in this PR?

Move GMTDataArrayAccessor to the "Xarray Integration" section in this PR?

I think do that in #3854. Hopefully this PR can be done before that.

seisman · 2025-04-23T02:26:28Z

This PR almost looks good, except that the doctest doesn't look good (what's ivar?):

weiji14 · 2025-04-23T03:21:14Z

This PR almost looks good, except that the doctest doesn't look good (what's ivar?):

Appears to be a sphinx thing. I found mention of a napoleon_use_ivar setting set here (added in #383) that converts the .. attribute:: directive, and because the docstring output has Attributes:, sphinx converts it to ivar 😅

seisman · 2025-04-23T06:20:49Z

This PR almost looks good, except that the doctest doesn't look good (what's ivar?):

Appears to be a sphinx thing. I found mention of a napoleon_use_ivar setting set here (added in #383) that converts the .. attribute:: directive, and because the docstring output has Attributes:, sphinx converts it to ivar 😅

The xarray documentation also uses napoleon, but the xarray repr looks correct (e.g., https://docs.xarray.dev/en/stable/generated/xarray.DataArray.html#xarray.DataArray), but I don't find any special settings in the conf.py file (https://github.com/pydata/xarray/blob/main/doc/conf.py).

seisman · 2025-04-23T06:23:42Z

pygmt/xarray_backend.py

+    Attributes:
+        Conventions:   CF-1.7
+        title:         Produced by grdcut
+        history:       grdcut @earth_relief_01d_p -R-55/-47/-24/-10 -Gstatic_eart...
+        description:   Reduced by Gaussian Cartesian filtering (111.2 km fullwidt...
+        actual_range:  [190. 981.]
+        long_name:     elevation (m)


A workaround for issue #3919 (comment)

Suggested change

Attributes:

Conventions: CF-1.7

title: Produced by grdcut

history: grdcut @earth_relief_01d_p -R-55/-47/-24/-10 -Gstatic_eart...

description: Reduced by Gaussian Cartesian filtering (111.2 km fullwidt...

actual_range: [190. 981.]

long_name: elevation (m)

...

I put the ellipisis after the colon in 'Attributes:' line, and it seemed to remove the :ivar from showing up (see commit f2446fc):

Only downside is the ellipsis shows as Attributes:..., but should be acceptable.

pygmt/xarray_backend.py

seisman · 2025-04-23T14:35:55Z

pygmt/xarray_backend.py

+        # `chunks` and `cache` DO NOT go here, they are handled by xarray
+    ) -> xr.Dataset:
+        """
+        Backend open_dataset method used by Xarray in :py:func:`~xarray.open_dataset`.


With the fix in #3927, sphinx-autogen will generate a stub file for this method. The method documentation is not ideal, at least one "Parameters" section is needed.

weiji14 · 2025-04-24T00:59:26Z

pygmt/xarray_backend.py

Considering moving this file to pygmt/xarray/backend.py, and then potentially in #3854, we can move pygmt/accessors.py to pygmt/xarray/accessors.py? And if we do pandas accessors (xref #3854 (comment)) in the future, we could put it under pygmt/pandas/accessor.py?

Sounds good

Xref #2620

Also cross-referencing to https://docs.generic-mapping-tools.org/6.5/reference/features.html#grid-file-format

pygmt/xarray/backend.py

seisman · 2025-04-24T01:53:07Z

pygmt/xarray/backend.py

+
+    Internally, GMT uses the netCDF C library to read netCDF files, and GDAL for GeoTIFF
+    and other raster formats. See also
+    :gmt-docs:`reference/features.html#grid-file-format`.


Maybe add a few sentences explaining why users may want to read netCDF/GeoTIFF files via GMT, rather than netcdf4/rasterio, something like:

Compared to the "netcdf4"/"rasterio" engines, the "gmt" engine can read GMT remote files (file names starting with @) directly and provides the GMTDataArray accessor .gmt for easy access to GMT-specific features and metadata.

Ok, updated the description in commit acab8b2, please review. I didn't mention the netcdf4/rasterio engines directly, but just mentioned that the GMT engine works better with GMT remote files and features in general.

Co-authored-by: Dongdong Tian <[email protected]>

seisman · 2025-04-24T02:56:53Z

Looks pretty good to me.

weiji14 · 2025-04-24T04:32:07Z

Will leave this up for review until end of the week-ish.

Xref 6a4e1a4 in #3919

weiji14 added the feature Brand new feature label Apr 17, 2025

Add source encoding to raster xarray.DataArray

2b98a65

So that GMTDataArray accessor knows the path to the original NetCDF file to retrieve correct metadata.

seisman mentioned this pull request Apr 17, 2025

Add pygmt.gmtread to read a dataset/grid/image into pandas.DataFrame/xarray.DataArray #3673

Draft

7 tasks

weiji14 added 11 commits April 17, 2025 14:30

Set load_dataarray's default engine to gmtread

f41a812

Use gmtread as default engine instead of netcdf4.

Support reading GeoTIFF images via the kind='image' argument

b71dcf5

Default is still kind='grid', by allow use of kind='image' too.

[skip ci] Satisfy type checking

60c6726

Make 'kind' a required kwarg with no default value

bf09a39

Users will need to set the 'kind' parameter explicitly to 'grid' or 'image'.

Add .grd to set of supported extensions in guess_can_open

e622878

Also set `kind='grid'` argument in _load_earth_relief_holes

Rename 'kind' parameter to 'decode_kind' and mypy fix

b9e3eb4

Need to set a default value of None for 'decode_kind' it seems, from reading https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html#open-dataset

Rename parameter kind to decode_kind in load_dataarray

3a109ac

Specify decode_kind="grid" in load_ functions

d764f78

[skip ci] format

98910f8

Use type: ignore[override]

126560e

Because xr.core.types.ReadBuffer is not available until xarray 2024.11.0, xref pydata/xarray#9787

Don't set default decode_kind in open_dataset

436f0fc

Partially reverts b9e3eb4

weiji14 added the needs review This PR has higher priority and needs review. label Apr 17, 2025

Rename engine="gmtread" to engine="gmt"

328b6ab

weiji14 changed the title ~~Implement gmtread xarray BackendEntrypoint~~ Implement gmt xarray BackendEntrypoint Apr 17, 2025

Add GMTBackendEntrypoint to doc/api/index.rst

06aa39d

seisman reviewed Apr 17, 2025

View reviewed changes

Apply suggestions from code review

c79e50a

Co-authored-by: Dongdong Tian <[email protected]>

seisman reviewed Apr 18, 2025

View reviewed changes

weiji14 added 4 commits April 20, 2025 08:26

Add test for xr.load_dataarray with a GRD grid

c0ccd4b

Update GMTBackendEntrypoint description to say NetCDF C library is used

c5ebf9f

lint

17fd9d7

Move GMTBackendEntrypoint to a new "Xarray Integration" API section

1c8af3a

[skip ci] lint

2e5fb6b

seisman reviewed Apr 23, 2025

View reviewed changes

seisman mentioned this pull request Apr 23, 2025

Add 'toctree' option to autosummary directive to generate stub pages #3927

Merged

seisman reviewed Apr 23, 2025

View reviewed changes

Merge branch 'main' into backendentrypoint/gmtread

b136e26

weiji14 commented Apr 24, 2025

View reviewed changes

weiji14 added 5 commits April 24, 2025 13:08

NetCDF to netCDF

19c5252

Xref #2620

Add Parameters section to docs of open_dataset

3bb1f44

Also cross-referencing to https://docs.generic-mapping-tools.org/6.5/reference/features.html#grid-file-format

Move pygmt/xarray_backend.py to pygmt/xarray/backend.py

8984c1c

Workaround sphinx Attributes ivar using ellipsis

f2446fc

Remove extra blankspace

52c0b7e

seisman reviewed Apr 24, 2025

View reviewed changes

weiji14 and others added 2 commits April 24, 2025 14:08

Apply suggestions from code review

143a375

Co-authored-by: Dongdong Tian <[email protected]>

Describe more on why one should choose the gmt engine

acab8b2

weiji14 marked this pull request as ready for review April 24, 2025 04:18

weiji14 added final review call This PR requires final review and approval from a second reviewer and removed needs review This PR has higher priority and needs review. labels Apr 24, 2025

seisman approved these changes Apr 24, 2025

View reviewed changes

seisman merged commit ecb933c into main Apr 26, 2025
30 of 32 checks passed

seisman deleted the backendentrypoint/gmtread branch April 26, 2025 01:21

seisman removed the final review call This PR requires final review and approval from a second reviewer label Apr 26, 2025

This was referenced Apr 26, 2025

Reorganize GMTDataArrayAccessor to the pygmt.xarray module #3928

Merged

Remove the dependency of netCDF4 #3643

Merged

weiji14 added a commit that referenced this pull request Apr 27, 2025

Rename decode_kind to raster_kind

6430bb2

Xref 6a4e1a4 in #3919

weiji14 mentioned this pull request Apr 29, 2025

Allow passing region to GMTBackendEntrypoint.open_dataset #3932

Draft

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement gmt xarray BackendEntrypoint #3919

Implement gmt xarray BackendEntrypoint #3919

weiji14 commented Apr 17, 2025 •

edited

Loading

weiji14 commented Apr 17, 2025 •

edited

Loading

seisman commented Apr 17, 2025

weiji14 commented Apr 17, 2025

seisman commented Apr 17, 2025

seisman Apr 17, 2025

weiji14 Apr 17, 2025

weiji14 Apr 17, 2025 •

edited

Loading

seisman Apr 18, 2025

weiji14 Apr 19, 2025

seisman Apr 20, 2025

weiji14 Apr 22, 2025

seisman Apr 18, 2025

weiji14 Apr 19, 2025

seisman Apr 18, 2025

weiji14 Apr 19, 2025 •

edited

Loading

seisman Apr 20, 2025 •

edited by weiji14

Loading

weiji14 Apr 22, 2025

seisman commented Apr 23, 2025

weiji14 commented Apr 23, 2025 •

edited

Loading

seisman commented Apr 23, 2025

seisman Apr 23, 2025

weiji14 Apr 24, 2025 •

edited

Loading

seisman Apr 23, 2025 •

edited

Loading

weiji14 Apr 24, 2025

seisman Apr 24, 2025

seisman Apr 24, 2025

weiji14 Apr 24, 2025

seisman commented Apr 24, 2025

weiji14 commented Apr 24, 2025

	return ext in {".grd", ".nc", ".tif"}
	return ext in {".grd", ".nc", ".tif", ".tiff"}

Implement gmt xarray BackendEntrypoint #3919

Implement gmt xarray BackendEntrypoint #3919

Conversation

weiji14 commented Apr 17, 2025 • edited Loading

weiji14 commented Apr 17, 2025 • edited Loading

seisman commented Apr 17, 2025

weiji14 commented Apr 17, 2025

seisman commented Apr 17, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weiji14 Apr 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weiji14 Apr 19, 2025 • edited Loading

Choose a reason for hiding this comment

seisman Apr 20, 2025 • edited by weiji14 Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seisman commented Apr 23, 2025

weiji14 commented Apr 23, 2025 • edited Loading

seisman commented Apr 23, 2025

Choose a reason for hiding this comment

weiji14 Apr 24, 2025 • edited Loading

Choose a reason for hiding this comment

seisman Apr 23, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seisman commented Apr 24, 2025

weiji14 commented Apr 24, 2025

weiji14 commented Apr 17, 2025 •

edited

Loading

weiji14 commented Apr 17, 2025 •

edited

Loading

weiji14 Apr 17, 2025 •

edited

Loading

weiji14 Apr 19, 2025 •

edited

Loading

seisman Apr 20, 2025 •

edited by weiji14

Loading

weiji14 commented Apr 23, 2025 •

edited

Loading

weiji14 Apr 24, 2025 •

edited

Loading

seisman Apr 23, 2025 •

edited

Loading