01_Data_Preparation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "\n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Preparation\n",
    "---\n",
    "\n",
    "This repository uses simulated orange juice sales data from [Azure Open Datasets](https://azure.microsoft.com/services/open-datasets/) to walk you through the process of training many models and forecasting on Azure Machine Learning. \n",
    "\n",
    "This notebook walks you through all the necessary steps to configure the data for this solution accelerator, including:\n",
    "\n",
    "1. Download the sample data\n",
    "2. Split in training/forecasting sets\n",
    "3. Connect to your workspace and upload the data to its Datastore\n",
    "\n",
    "### Prerequisites\n",
    "If you have already run the [00_Setup_AML_Workspace](00_Setup_AML_Workspace.ipynb) notebook you are all set.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.0 Download sample data\n",
    "\n",
    "The time series data used in this example was simulated based on the University of Chicago's Dominick's Finer Foods dataset, which featured two years of sales of 3 different orange juice brands for individual stores. You can learn more about the dataset [here](https://azure.microsoft.com/services/open-datasets/catalog/sample-oj-sales-simulated/). \n",
    "\n",
    "The full dataset includes simulated sales for 3,991 stores with 3 orange juice brands each, thus allowing 11,973 models to be trained to showcase the power of the many models pattern. Each series contains data from '1990-06-14' to '1992-10-01'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll need the `azureml-opendatasets` package to download the data. You can install it with the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%pip install azureml-opendatasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll start by downloading the first 10 files but you can easily edit the code below to train all 11,973 models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_maxfiles = 10 # Set to 11973 or 0 to get all the files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from azureml.opendatasets import OjSalesSimulated\n",
    "\n",
    "# Pull all of the data\n",
    "oj_sales_files = OjSalesSimulated.get_file_dataset()\n",
    "\n",
    "# Pull only the first `dataset_maxfiles` files\n",
    "if dataset_maxfiles:\n",
    "    oj_sales_files = oj_sales_files.take(dataset_maxfiles)\n",
    "\n",
    "# Create a folder to download\n",
    "target_path = 'oj_sales_data' \n",
    "os.makedirs(target_path, exist_ok=True)\n",
    "\n",
    "# Download the data\n",
    "oj_sales_files.download(target_path, overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.0 Split data in two sets\n",
    "\n",
    "We will now split each dataset in two parts: one will be used for training, and the other will be used for simulating batch forecasting. The training files will contain the data records before '1992-5-28' and the last part of each series will be stored in the inferencing files.\n",
    "\n",
    "Finally, we will upload both sets of data files to the Workspace's default [Datastore](https://docs.microsoft.compython/api/azureml-core/azureml.core.datastore(class))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scripts.helper import split_data\n",
    "\n",
    "# Provide name of timestamp column in the data and date from which to split into the inference dataset\n",
    "timestamp_column = 'WeekStarting'\n",
    "split_date = '1992-05-28'\n",
    "\n",
    "# Split each file and store in corresponding directory\n",
    "train_path, inference_path = split_data(target_path, timestamp_column, split_date)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.0 Upload data to Datastore in AML Workspace\n",
    "\n",
    "In the [setup notebook](00_Setup_AML_Workspace.ipynb) you created a [Workspace](https://docs.microsoft.com/python/api/azureml-core/azureml.core.workspace.workspace). We are going to register the data in that enviroment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.workspace import Workspace\n",
    "\n",
    "ws = Workspace.from_config()\n",
    "\n",
    "# Take a look at Workspace\n",
    "ws.get_details()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will upload both sets of data files to your Workspace's default [Datastore](https://docs.microsoft.com/azure/machine-learning/how-to-access-data). \n",
    "A Datastore is a place where data can be stored that is then made accessible for training or forecasting. Please refer to [Datastore documentation](https://docs.microsoft.com/python/api/azureml-core/azureml.core.datastore(class)) on how to access data from Datastore."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Connect to default datastore\n",
    "datastore = ws.get_default_datastore()\n",
    "\n",
    "# Upload train data\n",
    "ds_train_path = target_path + '_train'\n",
    "datastore.upload(src_dir=train_path, target_path=ds_train_path, overwrite=True)\n",
    "\n",
    "# Upload inference data\n",
    "ds_inference_path = target_path + '_inference'\n",
    "datastore.upload(src_dir=inference_path, target_path=ds_inference_path, overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### *[Optional]* If data is already in Azure: create Datastore from it\n",
    "\n",
    "If your data is already in Azure you don't need to upload it from your local machine to the default datastore. Instead, you can create a new Datastore that references that set of data. \n",
    "The following is an example of how to set up a Datastore from a container in Blob storage where the sample data is located. \n",
    "\n",
    "In this case, the orange juice data is available in a public blob container, defined by the information below. In your case, you'll need to specify the account credentials as well. For more information check [the documentation](https://docs.microsoft.com/python/api/azureml-core/azureml.core.datastore.datastore#register-azure-blob-container-workspace--datastore-name--container-name--account-name--sas-token-none--account-key-none--protocol-none--endpoint-none--overwrite-false--create-if-not-exists-false--skip-validation-false--blob-cache-timeout-none--grant-workspace-access-false--subscription-id-none--resource-group-none-)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "automl"
    ]
   },
   "outputs": [],
   "source": [
    "blob_datastore_name = \"automl_many_models\"\n",
    "container_name = \"automl-sample-notebook-data\"\n",
    "account_name = \"automlsamplenotebookdata\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "automl"
    ]
   },
   "outputs": [],
   "source": [
    "from azureml.core import Datastore\n",
    "\n",
    "datastore = Datastore.register_azure_blob_container(\n",
    "    workspace=ws, \n",
    "    datastore_name=blob_datastore_name, \n",
    "    container_name=container_name,\n",
    "    account_name=account_name,\n",
    "    create_if_not_exists=True\n",
    ")\n",
    "\n",
    "if 0 < dataset_maxfiles < 11973:\n",
    "    ds_train_path = 'oj_data_small/'\n",
    "    ds_inference_path = 'oj_inference_small/'\n",
    "else:\n",
    "    ds_train_path = 'oj_data/'\n",
    "    ds_inference_path = 'oj_inference/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.0 Register dataset in AML Workspace\n",
    "\n",
    "The last step is creating and registering [datasets](https://docs.microsoft.com/azure/machine-learning/concept-data#datasets) in Azure Machine Learning for the train and inference sets.\n",
    "\n",
    "Using a [FileDataset](https://docs.microsoft.com/python/api/azureml-core/azureml.data.file_dataset.filedataset) is currently the best way to take advantage of the many models pattern, so we create FileDatasets in the next cell. We then [register](https://docs.microsoft.com/azure/machine-learning/how-to-create-register-datasets#register-datasets) the FileDatasets in your Workspace; this associates the train/inference sets with simple names that can be easily referred to later on when we train models and produce forecasts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.dataset import Dataset\n",
    "\n",
    "# Create file datasets\n",
    "ds_train = Dataset.File.from_files(path=datastore.path(ds_train_path), validate=False)\n",
    "ds_inference = Dataset.File.from_files(path=datastore.path(ds_inference_path), validate=False)\n",
    "\n",
    "# Register the file datasets\n",
    "dataset_name = 'oj_data_small' if 0 < dataset_maxfiles < 11973 else 'oj_data'\n",
    "train_dataset_name = dataset_name + '_train'\n",
    "inference_dataset_name = dataset_name + '_inference'\n",
    "ds_train.register(ws, train_dataset_name, create_new_version=True)\n",
    "ds_inference.register(ws, inference_dataset_name, create_new_version=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.0 *[Optional]* Interact with the registered dataset\n",
    "\n",
    "After registering the data, it can be easily called using the command below. This is how the datasets will be accessed in future notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "oj_ds = Dataset.get_by_name(ws, name=train_dataset_name)\n",
    "oj_ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also possible to download the data from the registered dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "download_paths = oj_ds.download()\n",
    "download_paths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load one of the data files to see the format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "sample_data = pd.read_csv(download_paths[0])\n",
    "sample_data.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Steps\n",
    "\n",
    "Now that you have created your datasets, you are ready to move to one of the training notebooks to train and score the models:\n",
    "\n",
    "- Automated ML: please open [02_AutoML_Training_Pipeline.ipynb](Automated_ML/02_AutoML_Training_Pipeline/02_AutoML_Training_Pipeline.ipynb).\n",
    "- Custom Script: please open [02_CustomScript_Training_Pipeline.ipynb](Custom_Script/02_CustomScript_Training_Pipeline.ipynb)."
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "roastala"
   }
  ],
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}