[WIP] Multiprocessing interpolation and reprojection #661
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #648.
Description
This PR introduces raster interpolation and reprojection using multiprocessing to optimize memory usage and improve performance.
Code changes
multiproc_interp_points
leverages multiprocessing to create a separate process for each point to be interpolated. For each point, the function calculates the smallest possible raster window to open based on the interpolation method and point coordinates, and use theraster.crop
method to open the window. Theraster.interpolate
method is then applied to the cropped raster window, allowing for efficient interpolation without loading the full raster into memory.multiproc_reproject
utilizes raster tiling (see Implement Tiling for Multiprocessing #652) to perform multiprocessing-based reprojection. A separate process is created for each tile of the raster. Each process opens the corresponding tile using theraster.icrop
method, calculates the bounds of the reprojected tile, snaps the bounds to the target reprojected grid, and reprojects the tile using theraster.reproject
method. The reprojected tiles are written separately to disk, with safeguards in place to prevent data overwriting.Tests
multiproc_interp_points
andmultiproc_reproject
. The test results are compared against the behavior of theraster.interpolate
andraster.reproject
methods to ensure consistency.Note:
Currently, there are some tile alignment issues when performing reprojection operations that involve more than just translation. Further investigation is required to address these challenges.
Difference between tiled reprojection and classic reprojection (exploradores_aster_dem):