0.13.0

What's Changed

• Introduce finalize_dag method in Plan that caches DAG optimization by @tomwhite in #340
• Add VirtualInMemoryArray that keeps small arrays in memory by @tomwhite in #336
• Fix Modal test following #336 by @tomwhite in #341
• Change the DAG to have separate nodes for operations and arrays by @tomwhite in #337
• Allow more control over optimizations applied by @tomwhite in #342
• Handle the case where an array is created by calling map_blocks with no input arrays by @tomwhite in #343
• Add a test for visualizing the quad means computation by @tomwhite in #345
• Show nbytes for each array and total on visualization by @tomwhite in #344
• Fuse multiple inputs by @tomwhite in #346
• Upgrade to sphinx-notes/pages v3 by @tomwhite in #347
• Minor improvements to DAG visualization by @tomwhite in #348
• Remove deprecated code by @tomwhite in #349
• Add generated doc for primitive and core ops to design page by @tomwhite in #351
• Support arbitrary block index functions in blockwise by @tomwhite in #350
• Refactor pipeline to reduce coupling between runtime and primitive layers by @tomwhite in #352
• Implement stack using general_blockwise by @tomwhite in #353
• Implement reshape_chunks using general_blockwise by @tomwhite in #354
• More control over operation fusion by @tomwhite in #355
• Minor memory fixes by @tomwhite in #358
• Use Dask's 'broadcast trick' to save memory for single-valued arrays by @tomwhite in #359
• Add a show_hidden option to visualize by @tomwhite in #360
• Reinstate slow tests by @tomwhite in #361
• Add function_nargs to BlockwiseSpec by @tomwhite in #363
• Add missing api doc by @tomwhite in #364
• Implement partial_reduce and tree_reduce using generalized blockwise by @tomwhite in #365
• Quad means validation test by @tomwhite in #367
• Take allowed_mem into account when fusing primitive operations by @tomwhite in #366
• Fuse operations with different numbers of tasks by @tomwhite in #368
• Show number of tasks prominently on graph visualization by @tomwhite in #371
• Introduce event objects for all callback methods by @tomwhite in #373
• Minor Modal improvements by @tomwhite in #375
• Run tests using JAX as the backend array API (on CPU) by @tomwhite in #374
• Introduce max_total_num_input_blocks as a heuristic to control fusion by @tomwhite in #376
• Rename array_name to name in runtime following #337 by @tomwhite in #377
• Enable backup tasks by default for all executors that support them by @tomwhite in #378
• Remove unused array_names parameter in executors by @tomwhite in #380
• Introduce compute_id for each computation by @tomwhite in #382
• Add a Rich progress bar by @tomwhite in #383
• Change timeline visualization default format to SVG by @tomwhite in #384
• Preserve laziness in fuse_multiple to avoid materializing array blocks in _partial_reduce by @tomwhite in #386
• Fix Modal unit tests following change in #382 by @tomwhite in #387
• Add debug log statements to optimizer to see why nodes are fused by @tomwhite in #388
• Allow optional path in from_zarr and to_zarr by @tomwhite in #391
• Remove CUBED_MODAL_FORCE_BUILD env var … by @tomwhite in #393
• Add introductory slides to docs by @tomwhite in #394
• Use donfig to control default spec by @tomwhite in #392
• small changes to coiled/aws examples by @rsignell in #369
• Skip vecdot test due to zero-size array failures by @tomwhite in #399
• Show both operation ID and name in visualization by @tomwhite in #396
• Add support for minimum_workers in Coiled by @tomwhite in #398
• Create a single set of example scripts that can run on any executor by @tomwhite in #395

New Contributors

• @rsignell made their first contribution in #369

Full Changelog: 0.12.0...0.13.0

0.12.0

What's Changed

• Improve separation between layers by @tomwhite in #314
• Fix Modal deprecation of f.call(...) by @tomwhite in #320
• Disable the Hypthosis deadline when running Array API tests. by @tomwhite in #325
• Allow dtype to be passed to blockwise and reduction functions by @tomwhite in #321
• Array API backend by @tomwhite in #317
• Update examples to use Python 3.9 by @tomwhite in #330
• Re-write arange to use map_direct, which avoids writing a Zarr array by @tomwhite in #332

Full Changelog: 0.11.0...0.12.0

0.11.0

What's Changed

• Introduce VirtualEmptyArray and hide virtual arrays in the DAG visualization by @tomwhite in #290
• Introduce RetryingFunctionExecutor by @tomwhite in #292
• Add support for batching inputs to avoid overwhelming the backend service by @tomwhite in #291
• Support version 2022.12 of the Array API by @tomwhite in #293
• Fix array __abs__ method for complex inputs by @tomwhite in #294
• Be explicit when opening Zarr arrays by @tomwhite in #296
• Fix bug when running with Kerchunk by @tomwhite in #297
• Remove limited implementation of __setitem__ from Array by @tomwhite in #302
• Introduce VirtualFullArray by @tomwhite in #303
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #305
• Fix case where shape is an int for empty by @tomwhite in #306
• Update Coiled example by @tomwhite in #309
• Add nansum and nanmean functions by @tomwhite in #308
• Add moveaxis by @tomwhite in #313
• Update minimum Python version to 3.9 by @tomwhite in #298

Full Changelog: 0.10.0...0.11.0

0.10.0

What's Changed

• Scaling docs by @TomNicholas in #252
• Remove PipelineExecutor classes by @tomwhite in #253
• Surface rechunk intermediates in DAG by @tomwhite in #257
• Add type hints for runtime by @tomwhite in #258
• Fix modal import by @tomwhite in #261
• Add compute_arrays_in_parallel option to Modal and Python async executors by @tomwhite in #263
• Use new merge_chunks function in reduction by @tomwhite in #256
• Remove vendored rechunker API, since we only use the algorithm by @tomwhite in #264
• Add unit tests for virtual_offsets function by @tomwhite in #265
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #266
• Lithops compute arrays in parallel by @tomwhite in #259
• Remove duplicate copy of projected_mem and num_tasks from DAG by @tomwhite in #271
• Allow step size > 1 by @tomwhite in #270
• Fix reserved_mem in create_lazy_zarr_arrays by @tomwhite in #273
• Dask tests by @tomwhite in #269
• Coiled Functions executor by @TomNicholas in #260
• Ensure runtime memory is not less than allowed by @tomwhite in #274
• In-process Python executor should not retry by @tomwhite in #275
• Refactor pipeline mappable by @tomwhite in #272
• Generalise async executor code by @tomwhite in #278
• Dask distributed async executor by @tomwhite in #279
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #283
• Dask check_runtime_memory by @tomwhite in #281
• Fixes for converting strings to bytes by @TomNicholas in #289

Full Changelog: 0.9.0...0.10.0

0.9.0

What's Changed

• Fix Modal deprecated API by @tomwhite in #214
• Modal GCP by @tomwhite in #215
• Add html array repr by @TomNicholas in #216
• Add to pre-commit.ci by @tomwhite in #218
• Skip brew update when setting up GraphViz on mac CI to improve run time by @tomwhite in #227
• Set 'reraise' to True for retries by @tomwhite in #225
• Run mypy in CI by @tomwhite in #229
• Print irregular chunking pattern when error raised by @TomNicholas in #231
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #232
• Allow rechunk to take a dict for chunks by @tomwhite in #226
• Improve Modal backups implementation by @tomwhite in #238
• Test that plan generation scales to very large workloads by @tomwhite in #240
• Don't materialize input iterables to map_unordered by @tomwhite in #242
• Lithops retries and backups by @tomwhite in #217
• Add type hints for primitive functions by @tomwhite in #236
• Add links to Xarray integration and blog post to docs by @tomwhite in #243
• Account for reserved_mem when creating arrays by @tomwhite in #245
• Use np.unravel_index rather than iterating over np.ndindex by @tomwhite in #246
• Add matmul example that validates result by @tomwhite in #244
• Document strong consistency by @tomwhite in #250
• Re-enable rechunker's consolidate_reads by @tomwhite in #249
• Introduce VirtualOffsetsArray that is never materialized by @tomwhite in #248
• Avoid divide by zero from projected mem by @tomwhite in #251

New Contributors

• @pre-commit-ci made their first contribution in #232

Full Changelog: 0.8.0...0.9.0

0.8.0

What's Changed

• Lithops gcf example by @TomNicholas in #183
• Name Modal Stub objects to avoid warning by @tomwhite in #189
• Convert str to bytes by @TomNicholas in #188
• Reinstate Lithops AWS examples that were accidently removed in #174 by @tomwhite in #185
• Improvements and fixes to Modal examples by @tomwhite in #190
• More examples on GCF by @tomwhite in #198
• Fix measure reserved by @tomwhite in #200
• Include 'cubed' in the directories created in the working directory by @tomwhite in #202
• Add user guide and break out getting started into separate pages by @tomwhite in #203
• Show array variable name in visualization by @tomwhite in #192
• Runtime refactoring by @tomwhite in #206
• Make Zarr array creation lazy by @tomwhite in #207
• Make executors create Zarr arrays by @tomwhite in #209
• Make Lithops retries apply per-task by @tomwhite in #211
• Move Modal examples to 'aws' directory by @tomwhite in #212
• Publish packages to PyPI using GitHub Actions by @tomwhite in #213

Full Changelog: 0.7.0...0.8.0

0.7.0

What's Changed

• History analysis by @tomwhite in #170
• Rename memory settings by @tomwhite in #180
• Docs on installing from conda + optional dependencies by @TomNicholas in #178
• Rename peak_memory to peak_measured_mem in line with other memory settings by @tomwhite in #182

Full Changelog: 0.6.3...0.7.0

0.6.3

What's Changed

• Update examples by @tomwhite in #174
• Run tests on macOS and Windows by @tomwhite in #176

Full Changelog: 0.6.2...0.6.3

0.6.2

What's Changed

• Make lithops optional for running tests by @tomwhite in #169
• Optional dependency specification and pyproject.toml cleanup by @TomNicholas in #168
• Use requirements file from Array API tests repo by @tomwhite in #172
• Fix modal tests (env must not contain beam or lithops dependencies) by @tomwhite in #173

Full Changelog: 0.6.1...0.6.2

0.6.1

What's Changed

• Remove dask from pip by @tomwhite in #159

Full Changelog: 0.6.0...0.6.1