lab1/lab1-intro_to_numpy_scikitlearn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 1: Revision of Jupyter Notebooks & NumPy and an Introduction to Scikit-learn\n",
    "\n",
    "This lab assumes you have used Python and Jupyter Notebooks before and offers a recap of some of the basics and key libraries (such as NumPy and Matplotlib) as well as introducing scikit-learn, a popular machine learning library. For a refresher on Python, see the [Introduction to Python lab](https://github.com/UoB-COMS21202/lab_sheets_public/tree/master/lab_1) or the University of Bristol [Beginning Python](https://milliams.gitlab.io/beginning_python/) course.\n",
    "\n",
    "We strongly encourage you to use Python 3 as opposed to Python 2. Specifically Python 3.6 or newer is recommended.\n",
    "\n",
    "The following libraries will be used throughout the unit:\n",
    "\n",
    "- [NumPy](http://docs.scipy.org/doc/numpy/index.html), for scientific computation\n",
    "- [Pandas](https://pandas.pydata.org/docs/reference/index.html), for data analysis\n",
    "- [Matplotlib](http://matplotlib.org/contents.html), to plot any kind of data\n",
    "- [Scikit-learn](https://scikit-learn.org/stable/user_guide.html), for machine learning\n",
    "\n",
    "The libraries above have complete and very good documentation which can be used to learn other features of the libraries or for questions and examples. The documentation is available either online (links above) or via Python itself, e.g. `help(numpy.array)` in the Python interpreter.\n",
    "\n",
    "The following libraries are required for this lab so make sure these are installed using pip3 or Anaconda (recommended to use a virtual environment):\n",
    "- numpy\n",
    "- pandas\n",
    "- matplotlib\n",
    "- seaborn\n",
    "- scikit-learn\n",
    "- scikit-image\n",
    "\n",
    "For example, to install scikit-learn in a new conda environment, run\n",
    "```\n",
    "$ conda create -n COMS30035_labs\n",
    "$ conda activate COMS30035_labs \n",
    "$ conda install scikit-learn\n",
    "```\n",
    "For further help see the installation guides on the libraries documentation.\n",
    "\n",
    "## Jupyter Notebook\n",
    "\n",
    "This modules labs will be run on [Jupyter Notebook](http://jupyter.org/), an interactive coding environments embedded in a webpage supporting various programing languages (Python, R, Lua, etc.) through the concept of kernels.  \n",
    "\n",
    "It allows you to enrich your code with complex comments formatted in Markdown and $\\LaTeX$, as well as to place the results of your computation right below your code.\n",
    "\n",
    "Notebooks are organised in cells which can contain either code (in our case, this will be Python code) or text, which can be easily and nicely formatted using the Markdown notation. \n",
    "\n",
    "To edit an already existing cell simply double-click on it. You can use the toolbar to insert new cells, edit and delete them (or use keyboard shortcuts which are very handy to speed up coding). \n",
    "\n",
    "Cells can be run, by hitting `shift+enter` when editing a cell or by clicking on the `Run` button at the top. Running a Markdown cell will simply display the formatted text, while running a code cell will execute the commands executed in it. \n",
    "\n",
    "**Note**: when you run a code cell, all the created variables, implemented functions and imported libraries will be then available to every other code cell. However, it is commonly assumed that cells will be run sequentially in terms of prerequisites. To reset all variables and functions (for debugging) simply click `Kernel > Restart` from the Jupyter menu.\n",
    "\n",
    "#### A bit on Markdown language (and a bit of LaTeX and HTML) if you're interested \n",
    "\n",
    "Markdown cells allow you to write fancy and simple comments: all of this is written in Markdown - double click on this cell to see the source. Introduction to Markdown syntax can be found [here](https://daringfireball.net/projects/markdown/syntax).\n",
    "\n",
    "As Markdown is translated to HTML upon displaying it also allows you to use pure HTML: more details are available [here](https://daringfireball.net/projects/markdown/syntax#html).\n",
    "\n",
    "Finally, you can also display simple $\\LaTeX$ equations in Markdown thanks to `MathJax` support. For inline equations wrap your equation between `$` symbols; for display mode equations use `$$`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing the libraries\n",
    "\n",
    "Before we start this lab we need to import the aforementioned libraries, using the `import` keyword and bind the libraries to the `np`, `pd` and `plt` etc namespaces with the `as` keyword."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from skimage import io\n",
    "from sklearn.datasets import load_boston\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.metrics import mean_squared_error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) NumPy \n",
    "\n",
    "NumPy is designed for scientific computing. NumPy defines its own multidimensional array which can be created with:\n",
    "\n",
    "``` Python\n",
    "np.array([[1, 2], [3, 4], [5, 6]])\n",
    "```\n",
    "- For more details, type `help(np.array)` in your Python console or visit online help [here](http://docs.scipy.org/doc/numpy/reference/generated/numpy.matrix.html). \n",
    "\n",
    "#### 1.1) Array operations\n",
    "\n",
    "create two arrays, `A` and `B`:\n",
    "\n",
    "``` Python\n",
    "A = np.array([[2, 3], [4, -1], [5, 6]])\n",
    "B = np.array([[5, 2], [8, 9], [2, 1]])\n",
    "```\n",
    "\n",
    "and perform the following operations, printing the array C:\n",
    "\n",
    "- $C = 3A$\n",
    "- $C = A + B$\n",
    "- $C = AB^T$ (dot product or inner product)\n",
    "- $C = A \\odot B$ (Hadamard product or elementwise product)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2) More array operations\n",
    "\n",
    "Calculate now the *sum*, *mean*, and *variance* of the matrix `A`, using `NumPy` functions/array properties `mean`, `sum`, `var`.\n",
    "\n",
    "Hint: `help(np.sum)` or look [here](http://docs.scipy.org/doc/numpy/reference/generated/numpy.sum.html).  \n",
    "Hint: `help(np.mean)` or look [here](http://docs.scipy.org/doc/numpy/reference/generated/numpy.mean.html).  \n",
    "Hint: `help(np.var)` or look [here](http://docs.scipy.org/doc/numpy/reference/generated/numpy.var.html#numpy.var).\n",
    "\n",
    "Afterwards, calculate the *sum* of the rows and then the columns of `A`. Hint, specify the parameter `axis`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3) Implement the sigmoid function using numpy. \n",
    "\n",
    "The sigmoid function is a non-linear function used in machine learning (logistic regression) and also deep learning (as an activation function). \n",
    "\n",
    "$$sigmoid(x) = \\frac{1}{1+e^{-x}}$$\n",
    "\n",
    "where $x$ could now be either a real number, a vector, or a matrix.\n",
    "\n",
    "Implement the sigmoid function by defining a function called `sigmoid` which takes 1 argument $x$, a scalar or numpy array of any size and outputs the $sigmoid(x)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the sigmoid of the array:\n",
    "``` Python\n",
    "A = np.array([-5, 0, 5])\n",
    "```\n",
    "Plot the sigmoid curve for $ x \\in [-5, 5] $. Hint, use numpy [arange](https://numpy.org/doc/stable/reference/generated/numpy.arange.html) or [linspace](https://numpy.org/doc/stable/reference/generated/numpy.linspace.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.4) Standardise columns using numpy\n",
    "\n",
    "A common technique used in machine learning is to standardise the data to ensure all features have values that lie on a comparable scale. Standardisation helps visualise data but also helps with convergence and to achieve high predictive performance for some machine learning algorithms.\n",
    "\n",
    "To standardise a dataset we center the data by subtracting the mean of each feature, then scale by dividing by the standard deviation of the feature. Assuming the data is arranged with features in columns and training instances in rows, standardisation will result in each column vector of the data matrix having a mean of 0 and standard deviation of 1.\n",
    "\n",
    "Implement a `standardiseCols(x)` function to standardise the columns of a numpy array.\n",
    "\n",
    "Note, in Python you are able to perform mathematical operations between arrays of different shapes (such as substracting the row vector of means from a matrix) due to broadcasting, for more information read [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Standardise the columns of the array below by calling the `standardiseCols(x)` function:\n",
    "``` Python\n",
    "x = np.array([\n",
    "    [0, 3, 5],\n",
    "    [1, 6, 4],\n",
    "    [3, -2, 8],\n",
    "    [-1, 1, 10]\n",
    "])\n",
    "```\n",
    "Print the mean and standard deviation of the columns of the standardised array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.5) Reshaping numpy arrays \n",
    "\n",
    "The attribute [np.shape](https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.shape.html) and function [np.reshape()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.reshape.html) are commonly used in machine learning:\n",
    "- X.shape is used to get the shape (dimension) of a matrix/vector X. \n",
    "- X.reshape(...) is used to create a new array containing the elements of X with the provided shape. \n",
    "\n",
    "For example, in computer vision, an image is represented by a 3D array of shape $(length, height, colour)$ where the colour represents the three RGB (red, green, blue) channels. Lets first load and plot the image. In order for the image to be given as an input into a machine learning algorithm, the 3D array needs to be reshaped to a vector of shape $(length*height*3, 1)$, that's your task below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "image = io.imread('flower.png')\n",
    "io.imshow(image)\n",
    "image.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reshape the $image$ array to vector $v$ and print the shape of the created vector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Note on array dimensions\n",
    "The array $a$ below is a 1-dimensional array which has some slightly non-intuitive effects, such as the transpose is the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.arange(5)\n",
    "print(a)\n",
    "print(a.shape)\n",
    "print(a.T)\n",
    "print(a.T.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1-D arrays should be avoided and instead column or row vectors should be used which can be formed from 1-D arrays using reshape. Note the double square bracket.\n",
    "\n",
    "The row vector of $a$ is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = a.reshape(1,-1)\n",
    "print(a)\n",
    "a.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The column vector of $a$ is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = a.reshape(-1,1)\n",
    "print(a)\n",
    "a.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can check the dimensions are what you want by using the assert command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert(a.shape == (5,1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Note on function and object property\n",
    "As Python is an object oriented language, the difference between *function* and *object property* should be understood.  \n",
    "An object instance, e.g. NumPy array `A = np.array([[1, 2], [3, 4], [5, 6]])` inherits all the functions from the class `numpy.array`. Therefore, to sum all elements of array `A` we can choose two approaches:\n",
    "\n",
    "- `A.sum()`, or\n",
    "- `np.sum(A)`.\n",
    "\n",
    "the first one is advisable.\n",
    "\n",
    "Moreover, some objects have *properties* (e.g. shape of an array). Instead of calling the shape *function*, an array object has the shape *property*, i.e.:\n",
    "\n",
    "- `A.shape`\n",
    "- `np.shape(A)`\n",
    "\n",
    "the first one is advisable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Scikit-learn\n",
    "Scikit-learn is an open source machine learning library that supports supervised and unsupervised learning. It provides various tools for model fitting, data preprocessing, model selection and evaluation as well as many other utilities. The exercises takes you through a very simple workflow for training and evaluating machine learning models.\n",
    "\n",
    "### Scikit-learn basics\n",
    "\n",
    "#### 2.1) Datasets\n",
    "Scikit-learn can be used to import [datasets](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.datasets) using the dataset loader to load small standard, or 'toy', datasets (such as iris classification or boston house pricing) or the  dataset fetcher to download and load larger dataset from the ‘real world’.\n",
    "\n",
    "Firstly, load the Boston dataset and print the number of examples and features and feature names in the dataset. Note, [load_boston](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html#sklearn.datasets.load_boston) has already been imported from sklearn.datasets. \n",
    "\n",
    "Then using seaborn create a [pairplot](https://seaborn.pydata.org/tutorial/axis_grids.html) (or scatterplot matrix) to show feature joint relationships and individual feature distributions. Note, the data must be in a pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pairplots are a quick and effective way to perform exploratory data analysis (EDA) to find patterns, relationships and anomalies to guide subsequent analysis. A pairplot allows us to see both the distribution of single variables and relationships between two variables. As you can see above, a pairplot of 14 features is crowded and difficult to interpret.\n",
    "\n",
    "If you were wondering, the feature names abbreviations stand for:   \n",
    "1. CRIM      per capita crime rate by town\n",
    "2. ZN        proportion of residential land zoned for lots over 25,000 sq.ft.\n",
    "3. INDUS     proportion of non-retail business acres per town\n",
    "4. CHAS      Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)\n",
    "5. NOX       nitric oxides concentration (parts per 10 million)\n",
    "6. RM        average number of rooms per dwelling\n",
    "7. AGE       proportion of owner-occupied units built prior to 1940\n",
    "8. DIS       weighted distances to five Boston employment centres\n",
    "9. RAD       index of accessibility to radial highways\n",
    "10. TAX      full-value property-tax rate per \\$10,000\n",
    "11. PTRATIO  pupil-teacher ratio by town\n",
    "12. B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town\n",
    "13. LSTAT    \\% lower status of the population\n",
    "\n",
    "The target variable MEDV is Median value of owner-occupied homes in $1000's."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2) Preprossing\n",
    "The scikit-learn has a [preprocessing](https://scikit-learn.org/stable/modules/preprocessing.html) package which provides several common functions to change the raw data into something more suitable for the machine learning algorithm. In general, learning algorithms benefit from standardisation of the dataset or if some outliers are present then robust scalers or transformers are more appropriate.\n",
    "\n",
    "Standardise the Boston training dataset loaded in the previous exercise using [StandardScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScaler). Check the features have zero mean and unit variance/standard deviation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.3) Train test split\n",
    "Training and evaluating a model on the same dataset will lead to overfitting and poor performance on unseen data. To measure the generalisation ability of a model, it is common practice to hold out part of the available data as a test set. Scikit-learn provides a [train_test_split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html#sklearn.model_selection.train_test_split) function which performs a random split into training and testing sets.\n",
    "\n",
    "When evaluating different hyperparameters for machine learning models then there is still a risk of overfitting on the test set because the parameters can be tweaked until the model performs optimally. To solve this problem, yet another part of the dataset can be held out as a so-called validation set which is used for evaluating hyperparameter values before final evaluation can be done on the test set. However, this can drastically reduce the number of samples which can be used for model training and the performance depends on the training and validation splits. To get round this problem cross-validation can be used which trains multiple models on \"folds\" of the training set, read about cross-validation in scikit-learn [here](https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation).\n",
    "\n",
    "Your task is to split the Boston (scaled) training data set so 70% of the data is for training and the remaining 30% to test.\n",
    "\n",
    "How many training examples are there in the training and test sets?\n",
    "\n",
    "Should the `random_state` parameter be specified for the `train_test_split` function?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.4) Model fitting\n",
    "Scikit-learn provides many built-in machine learning algorithms and models, called estimators. An estimator is any object that learns from data; it may be a classification, regression or clustering algorithm or a transformer that extracts/filters useful features from raw data. Each estimator can be fitted to some data using its fit method, as was done on the previous exercise to standardise the raw data.\n",
    "\n",
    "Fit a simple [linear regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html) model to the training data from the train-test split.\n",
    "\n",
    "Print the linear model coefficients. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.5) Model evauluation\n",
    "Scikit-learn supports simple and quick evaluation of machine learning models using the [metrics](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics) module which quantifies the quality of predictions.\n",
    "\n",
    "Calculate the root mean squared error (RMSE) of both the training and test sets for the linear model. Is the model overfitting?\n",
    "\n",
    "Also, plot the model predictions on a [scatter plot](https://matplotlib.org/3.2.2/api/_as_gen/matplotlib.pyplot.scatter.html?highlight=scatter#matplotlib.pyplot.scatter) alongside the real values for the test data. All the data points would lie on a diagonal (prediction = real) if the model was 100% accurate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This lab provides some very basics of scikit-learn, more to come in future labs!\n",
    "\n",
    "## Wrap up\n",
    "\n",
    "That's it for lab 1, a revision of Jupyter Notebooks and NumPy as well as an introduction to Scikit-learn. We have covered:\n",
    "- The interactive coding environment of Jupyter Notebooks which allows Markdown and $\\LaTeX$ to be added to code. Notebooks show the results of your computation right below your code which is great for quick coding experiments and debugging. These features allow Notebooks to be used to create human-readable documents containing visualiations of results and analysis.\n",
    "- The NumPy package performing a range of array operations, standardised the columns of an array and reshaped an image array into a column vector.\n",
    "- The  machine learning library Scikit-learn which can be used to import datasets, perform data preprocessing and then train and evaluate models. We went through a basic machine learning pipeline with the Boston housing dataset.\n",
    "\n",
    "### References\n",
    "- COMS30035 Machine Learning lecture notes 1.\n",
    "\n",
    "#### Materials used to create the lab\n",
    "- University of Bristol's Symbols, Patterns and Signals course\n",
    "- Andrew Ng's Neural Networks and Deep Learning course on Coursera"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}