0.10.0

[jaheba]

Overview

Arrow based datasets

We have added support for Parquet-files, as well as Arrow's binary format. This is an opt-in feature, requiring pyarrow to be installed. Use pip install 'gluonts[pro]' or pip install 'gluonts[arrow]' to ensure the correct version is installed.

FileDataset has been reworked to support .parquet and .arrow files. Previously, it had assumed all files to use jsonlines. To continue using jsonlines ensure that the the files use one of the .json, .jsonl, .json.gz, jsonl.gz suffixes.

Depending on the dataset size and shape, Arrow can be much faster than the json variant. In more extreme cases we saw speedups of more than 100x when using arrow vs jsonlines (see #2003 for some examples).

To convert a given dataset into arrow, you can use the gluonts.dataset.arrow utility:

python -m gluonts.dataset.arrow write </path/to/dataset> my-dataset.arrow

PandasDataset

We have added support for pandas.DataFrame and pandas.Series as well. You can now directly model data given in a DataFrame using gluonts.dataset.pandas.PandasDataset. In this tutorial we describe in depth how you can use PandasDataset to speed up modelling using GluonTS.

Changelog

New Features

• Add TimeLimitCallback to mx/trainer callbacks. #1631 - Add TimeLimitCallback to mx/trainer callbacks. (by @yx1215)
• adding MQF2 (Multi-horizon) #1780 - adding MQF2 (Multi-horizon) (by @KelvinKan)
• Added QuarterlyBegin time feature #1903 - Added QuarterlyBegin time feature (by @kashif)
• Porting SimpleFeedForwardEstimator to PyTorch #1924 - Porting SimpleFeedForwardEstimator to PyTorch (by @lostella)
• DeepAR PyTorch: make samplers configurable #1925 - DeepAR PyTorch: make samplers configurable (by @lostella)
• added support for pandas dataframes #1935 - added support for pandas dataframes (by @rsnirwan)
• Add support for beta-NLL loss #1962 - Add support for beta-NLL loss (by @kashif)
• Add Uber-TLC dataset to dataset repository. #1982 - Add Uber-TLC dataset to dataset repository. (by @Hongqing-work)
• Add info cli. #1990 - Add info cli. (by @jaheba)
• Add HP tuning example with Optuna #1987 - Add HP tuning example with Optuna (by @npnv)
• Add arrow-based dataset. #2000 - Add arrow-based dataset. (by @vafl, @lostella, @jaheba)
• add ND for item_metrics #2002 - add ND for item_metrics (by @melopeo)
• Added support of "long" RTS, making short RTS be "past_feat_dynamic_real" #2006 - Added support of "long" RTS, making short RTS be "past_feat_dynamic_real" (by @zoolhasson)
• Add DatasetWriter. #2061 - Add DatasetWriter. (by @jaheba)
• Add support for second frequency. #2074 - Add support for second frequency. (by @kashif)

Breaking Changes

• Breaking: Fix return types of features #1917 - Breaking: Fix return types of features (by @lostella)
• Breaking: Update dependency fbprophet -> prophet #1941 - Breaking: Update dependency fbprophet -> prophet (by @lostella)
• Breaking: Split incremental quantile output into separate class #1946 - Breaking: Split incremental quantile output into separate class (by @lostella)
• Breaking: reorg torch package, shorten import paths #1965 - Breaking: reorg torch package, shorten import paths (by @lostella)
• Use pd.Period instead of pd.Timestamp. #1980 - Use pd.Period instead of pd.Timestamp. (by @jaheba)
• Remove freq argument from Forecast. #1997 - Remove freq argument from Forecast. (by @kashif)
• Remove dct_reduce. #2011 - Remove dct_reduce. (by @jaheba)
• Remove mandatory freq attribute of Predictor. #2017 - Remove mandatory freq attribute of Predictor. (by @kashif)
• Remove multiprocessing dataloader. #2018 - Remove multiprocessing dataloader. (by @jaheba)
• Rework FileDataset. #2019 - Rework FileDataset. (by @jaheba)
• Add dataset_writer to get_dataset. #2053 - Add dataset_writer to get_dataset. (by @Hongqing-work)
• Add jsonl.encode_json, remove serialize_data_entry. #2070 - Add jsonl.encode_json, remove serialize_data_entry. (by @jaheba)

Bug Fixes / Minor Improvements

• Settings._let will pop element it added instead of just the last one. #1704 - Settings._let will pop element it added instead of just the last one. (by @jaheba)
• Fix typing issues in torch estimators, update base estimators docstrings #1905 - Fix typing issues in torch estimators, update base estimators docstrings (by @lostella)
• Fix the use of the scaling parameter in Transformer model #1909 - Fix the use of the scaling parameter in Transformer model (by @StanislasGuinel)
• Fix AddTimeFeatures transformation for multiples of base frequencies #1916 - Fix AddTimeFeatures transformation for multiples of base frequencies (by @lostella)
• Fix: use broadcast_lesser in place of comparisons in ISQF #1920 - Fix: use broadcast_lesser in place of comparisons in ISQF (by @vincentqb)
• Fix dummy estimator #1931 - Fix dummy estimator (by @canerturkmen)
• Fix Pytorch Lightning tutorial. #1933 - Fix Pytorch Lightning tutorial. (by @jaheba)
• Fixed autograd inplace operations error in Transformed Distribution #1938 - Fixed autograd inplace operations error in Transformed Distribution (by @shubhamkapoor)
• Fix: Hard threshold positive distribution parameters #1950 - Fix: Hard threshold positive distribution parameters (by @lostella)
• Fix forecast keys (quantiles) output by TemporalFusionTransformer #1952 - Fix forecast keys (quantiles) output by TemporalFusionTransformer (by @lostella)
• Fix: use of num_parallel_samples in deepAR #1968 - Fix: use of num_parallel_samples in deepAR (by @kashif)
• Fix: torch DeepAR observed indicator in multivariate case #1969 - Fix: torch DeepAR observed indicator in multivariate case (by @kashif)
• use FieldName #1975 - use FieldName (by @kashif)
• Documentation: add docstrings for torch-based models #1983 - Documentation: add docstrings for torch-based models (by @lostella)
• Fix OffsetSplitter for negative offsets #1986 - Fix OffsetSplitter for negative offsets (by @lostella)
• Pin protobuf version. #1989 - Pin protobuf version. (by @jaheba)
• Remove packaged pytorch-ts from gluonts.nursery.SCott #1991 - Remove packaged pytorch-ts from gluonts.nursery.SCott (by @lostella)
• Documentation: fix and speed up tutorials #1999 - Documentation: fix and speed up tutorials (by @lostella)
• Refactor splitter assertion and add error message #2004 - Refactor splitter assertion and add error message (by @rsnirwan)
• Rework itertools, add col-to-row and row-to-col functions. #2005 - Rework itertools, add col-to-row and row-to-col functions. (by @jaheba)
• Re-add cache for parsing 'pd.Period'. #2008 - Re-add cache for parsing 'pd.Period'. (by @jaheba)
• Update website template, clean up homepage and tutorials #2013 - Update website template, clean up homepage and tutorials (by @lostella)
• Expose Estimator, Predictor, Forecast in gluonts.model. #2014 - Expose Estimator, Predictor, Forecast in gluonts.model. (by @jaheba)
• Fix mean in AffineTransformedDistribution #2015 - Fix mean in AffineTransformedDistribution (by @stailx)
• Fix torch affine transformed distribution #2016 - Fix torch affine transformed distribution (by @lostella)
• Remove unnecessary files from docs folder, update gitignore #2020 - Remove unnecessary files from docs folder, update gitignore (by @lostella)
• Update references to dev branch. #2021 - Update references to dev branch. (by @lostella)
• Fix README. Use DataFramesDataset. #2024 - Fix README. Use DataFramesDataset. (by @jaheba)
• Make HP tuning tutorial more accurate #2025 - Make HP tuning tutorial more accurate (by @jaheba)
• Re-add support for Python 3.6 #2028 - Re-add support for Python 3.6 (by @jaheba)
• Add support for nan values in Rotbaum #2029 - Add support for nan values in Rotbaum (by @zoolhasson)
• Simplify lag values computation in torch DeepAR #2035 - Simplify lag values computation in torch DeepAR (by @lostella)
• Minor improvements to the hierarchical model #2036 - Minor improvements to the hierarchical model (by @rshyamsundar)
• Make Quantile derive from pydantic.BaseModel. #2047 - Make Quantile derive from pydantic.BaseModel. (by @jaheba)
• Add concepts section to docs. #2050 - Add concepts section to docs. (by @jaheba)
• Add tutorial on DataFramesDataset #2051 - Add tutorial on DataFramesDataset (by @rsnirwan)
• Add optional parameter time_axis to forecast_start. #2057 - Add optional parameter time_axis to forecast_start. (by @melopeo)
• Fix type annotations for predict_to_numpy #2062 - Fix type annotations for predict_to_numpy (by @lostella)
• Always pass freq explicitly to pd.period_range. #2066 - Always pass freq explicitly to pd.period_range. (by @kashif)
• Docs: simplify call to evaluator #2068 - Docs: simplify call to evaluator (by @lostella)
• Fix: DistributionLoss not encodable. #2092 - Fix: DistributionLoss not encodable. (by @jaheba)
• Add Airtraffic dataset. #2098 - Add Airtraffic dataset. (by @jaheba)
• Fixup trainer in case of non-finite loss. #2108 - Fixup trainer in case of non-finite loss. (by @jaheba)
• Change default behavior for TrainDatasets overwrite #2121 - Change default behavior for TrainDatasets overwrite (by @nklingen)

---

This discussion was created from the release 0.10.0.

[kashif]

Awesome! Note, another breaking change in this release is #2017 and in the improvement/fixes section I also updated the codeql #2040 and fixed the potential pandas period issue #2066

[lostella]

Updated — I just left out the codeql one since that’s less about the release, and more about development cycle

[kashif]

ah no worries! thanks!