examples/rl/policy-iteration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dynamic Programming\n",
    "### Policy Iteration\n",
    "Policy Iteration is a Dynamic Programming method to estimate the value function $V_\\pi(s)$ of a policy $\\pi(a|s)$ and improve the policy so that the actions selected lead to better return.\n",
    "\n",
    "Here, the Markov Decision Process (MDP) is a simple grid environment where the goal state is 2 rectangles. The states are the x,y positions in the grid and the possible actions are going north, south, west or east.  \n",
    "\n",
    "The goal is to find the optimal policy so that when placed in a location, an agent can select the best action to find itself in one of the goal states. The policy is initially random (actions are selected with equal probability) and its value function is estimated. For value estimation, the Policy Evaluation algorithm is used. To improve the policy, the Policy Improvement algorithm is used. By itarating Policy Evaluation and Policy Improvement, the policy converges to the optimal one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-20T11:55:29.529577Z",
     "start_time": "2020-04-20T11:55:29.508579Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>.container {width: 98%}</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<style>.container {width: 98%}</style>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-20T11:55:30.432586Z",
     "start_time": "2020-04-20T11:55:29.534579Z"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-20T11:55:32.091643Z",
     "start_time": "2020-04-20T11:55:30.436575Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "import sys\n",
    "sys.path.insert(0, \"../../\")\n",
    "from fromscratch.rl.mdp import GridWorld, State\n",
    "from fromscratch.rl.dp import PolicyEvaluation, PolicyImprovement\n",
    "from fromscratch.rl.policy import GenericPolicy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grid World\n",
    "Grid environment of n rows and m columns and 2 goal states  \n",
    "Allowed actions are going north, south, west or east, one step at a time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-20T11:55:32.635622Z",
     "start_time": "2020-04-20T11:55:32.094620Z"
    }
   },
   "outputs": [],
   "source": [
    "grid = GridWorld(6, 9, [State([0, 0]), State([4, 6])])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-20T11:55:33.514616Z",
     "start_time": "2020-04-20T11:55:32.641619Z"
    }
   },
   "outputs": [],
   "source": [
    "policy = GenericPolicy(grid, gamma=0.9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-20T11:55:34.235630Z",
     "start_time": "2020-04-20T11:55:33.517622Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWEAAAEICAYAAACOBEVFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deZyN5f/H8dfnzD7DGNKCGTshX1soWdKihNK3tGiRVJOQpZKyNMZapEQqpEUpRFSIFkna7MpW37KHZDebmTnn+v0xx/yGMMs5x3XP+DwfDw8zt3Ou+23mnve55j7nXLcYY1BKKWWHy3YApZQ6n2kJK6WURVrCSillkZawUkpZpCWslFIWaQkrpZRFWsLKsUTkDREZdJZ/NyJS9Vxm8u63s4gsO8u/LxGRh89lJlV4aQmrc0JE7haRn0UkWUT2eT/uJiJypvsYY7oaY4YWYF8dRWTjKdu+PMO2Z/I7vlL+pCWsAk5EngReAUYDlwAXA12BpkDoGe4T5MMuvwVqisiF3rGCgbpA5CnbmgBL8zOw935K+Y2WsAooESkBDAG6GWNmGWOOmSxrjDH3GmOOe2/3joi8LiILRCQZuMa7bViOsfqKyB4R2S0iXc60T2PMbmAL0MK7qQGwgaxyzrnNBawUkRIiMlVE/hGR7SIyUERc3n12FpHvReRlETkIDD7N/7GViGwWkSMi8ipwxtm9UqfSElaB1gQIAz7Jw23vAYYDxYGTzrmKSGvgKaAVUA24PpexlvL/hdsC+M47Zs5tPxlj0oHxQAmgMnA10Al4MMdYV5BV6hd58+XMVRqYDQwESgN/kjXDVypPtIRVoJUG9htjMk9sEJEfROSwiKSKSIsct/3EGPO9McZjjEk7ZZw7gbeNMeuNMcmcZkZ6ipyz3uZklfB3p2z71nva4y7gWe8sfRswBrg/x1i7jTHjjTGZxpjUU/bTBtjoneVnAGOBvblkUyqblrAKtANA6ZznUo0xVxljYrz/lvMY3HmWccqe8u/bc9nvUqCOiJQErgR+NMZsBsp4tzXz3qY0Weelc463HShXkFwma0Wss91eqZNoCatA+xE4DrTPw23PtqTfHiAux+flzzqQMVuA3UA8sMMYk5QjTzxQDPgJ2A9kABVOGfuvguTyvtoj7sw3V+pkWsIqoIwxh4FE4DUR6SAixUTEJSL1gKh8DDUT6CwitUQkEkjIw32+A57w/n3CMu+2lcaYVGOM2zv2cBEpLiIVvP/+fh5zzQcuE5HbvLP9nmS9AkSpPNESVgFnjBlFVrE9DewD/gYmAv2AH/I4xudknW9dDPzh/Ts335L1ZFrOJ/m+827L+dK0x4Fksp58WwZ8ALyVx1z7gTuA58k6vVIN+D4v91UKQHRRd6WUskdnwkopZZGWsFJKWaQlrJRSFmkJK6WURQFZjKR06QtMxfJnfRmn8tHf+/7h0OFD1Khe3XYUR3K73Wza/Bvl4+KIji5uOw4A//vjT6KiIilbpoztKMqCbTt2sH//gX+tKxKQEq5Yvjwrly0JxNAKSE5OpkyVS3Fnukno34+b29xkO5LjDB05isQNGwkJDWHFd99wlhUzz4nvvv+BG265jcwjR/l1+Y9ccEEpq3nUudewWcvTbtfTEYXQ+Dcm4XZ7SDt+nKeeHYi+zPBkR48eZdTLr+B2u9m+YweLvvzadiT69h9EWloaHuPh+Zdeth1HOYiWcCGTmZnJiFFjSElJAeCPLVv48utvLKdyljemvE1qWtb6P8nJKTybkGg1z8rVa1i+chUAaWnHGf/aRJKTk61mUs6hJVzIBAcHM3bUSEJDs9ZCTxzYn0aXN7Ccylnat23D4AFZF8y46YZWDBnU32qeWjUuZWjCwOzPX3phBJGRkRYTKScJyDvmGjaob/SccGB1fOAhps+ajUk+bDuKY0lUDHNnTKN9u7a2owBZeRpd3oDlS/PyjmtV1DRs1pKVq9f868kJnQkrpZRFWsJKKWWRlrBSSlmkJayUUhZpCSullEVawkopZZGWsFJKWaQlrJRSFmkJK6WURXkqYRFpLSK/icgfIvJMoENNmxFCxRrFcRWLpmKN4kybERLoXSqllBW5lrCIBAETgJuAWkBHEakVqEDTZoQQ3z2C7TtdGCNs3+kivnuEz0WclpbGuNfeIL5HLz8lVUqdasWq1dze8X7Wb9hoO0qhkZf1hBsDfxhjtgCIyHSgPRCQr/KAhHBSUk9+e3VKqjAgIZx778rI93hpaWlMnPI2iSNe4Pjx43g8HsaOGlmgbLroytkZY6yv26vyx1/fsxWrVtN3wHOsWLmKjMwMWl13DZUrVcz3OKGhoQQHB2SZ8wI5F8d0Xv635YCdOT7fBVxx6o1EJB6IBygfF1fgQDt2nf4/fKbtuZny7nv0fvrZk7ZFXVi2QGMtX7rYMSuW3X7rLaSnp9uOkW3v3r+JrV6LzKMHbEfJdmm1qlxarZrtGNm6dLqPKxs3sh0j2+S332X+wkXMnfGBz2M1u741GRkZ2WtbP9brCR7r9US+x3HaAkfFL45l/YofqFihQsD2kZcSPl37/WvpNWPMJGASZK2iVtBA5WMN23f+e5flYws2ZHyXzkRERNA/IZFjScl4PB5SD+wtaDzH6PDf9nT4b3vbMbIdOHgQt9ttO8ZJNq9daTvCSaa8/qrtCCeZOXsOX32zxC9jrVq2hGeeS2Txt9+SmZnJxHFjebDTfX4Z26bk5GT+2r0noCWclyfmdgE5p7axwO7AxIHhiWlERpxcuJERhuGJaQUaLyQkhC6d7mPn7xsZP2YUXR960B8xlVI51L6sFvNmz2D5t4vpeMcdNLmise1IhUZeZsIrgGoiUgn4C7gbuCdQgU6c9x2QEM6OXUL52KwCLsj54JxOlLFSKnBqX1aLdye/bjtGoZJrCRtjMkWkB7AICALeMsZsCGSoe+/K8Ll0lVKqMMjT05DGmAXAggBnUUqp846+Y04ppSzSElZKKYu0hJVSyiItYaWUskhLWCmlLNISVkopi7SElVLKIi1hpZSySEtY+ezT+Z8zfNSLAPQbmMDylassJ1JnY4zhxVfGZy/e029QAocPH7YbymFWr11Hv4EJALzw0ljmfDovYPvSElY+mzBxEh9+NBuAF18ZH9ADVvnu+PHjDB72/2tqj355HJt++91iIuf5bMHnjBn3qvfjhYyd8FrA9qUlrHw2IjGByIgIAMLDwniyZw/LidTZhIeH0++J3kR4v2cN69fTVc9O0atbV8LDwgCIjIhgZGJCwPalJax8dnn9elzRuCEul4tu8Q9RuvQFtiOpXPTq3pUgl4vwsDBGjxhqO47jxMTE0LtHN1wuF/Xq1uGqK/91HQu/cWQJezwe4nv0Yu26X2xHUXk0evhQqlWtQr8n+tiOAoDb7abLo93ZuGmz7SjZhj0/mo8/+cx2DACio6NJHNSfa1tezdXNm9mO40hP9epBjerVeDHAD1Jy4nIk/tSwQX2zctmSAt9/zqef0eHeB2je9CqWLNTziyr/pk2fSadHutK61fXM/3im7Ths2bqNmvUbER1dnL/+2ExoaKjtSOoca9isJStXr/nXZYMcNxP2eDz07f8cHo+HFStXsWLVatuRVCHjdrvpNygBj8fD4m+XOuLKvwMTh+IxhtS040yZ+r7tOMpBHFfCn8ybz45duwBISU2l74DnLCdShc30j2bzz/79QNYrAfoNGmw1z5at25g5ew6ZmZkkJyczKHGYoy7SquxyXAmXKFGC+nXrAFAsKoraNWtaTqQKm5iYEtSrk3UMXVj6AmpeWt1qntDQEBo3apj9+RWNGgb8Muqq8HDkOWEAiYph8oRxPNy5k39CqfOORMUwd8Y02rdrazsKkJXHaZd0V+dOoTknrJRS5xMtYaWUskhLWCmlLNISVkopi7SElVLKIi1hpZSySEtYKaUs0hJWSimLtISVUsoiLWGllLIo1xIWkbdEZJ+IrD8XgYoat9vNlHff488tW21HyTZ/4SK++/6Hf21PSUlh0lvvsHfv3xZSqYLYuWsXE6e8TUZGhu0oACQlJfHGm2/xzz/7bUcpNPIyE34HaB3gHEXWkSNHeaR7T2o3akLHB7o4oowf6/UE17drzxUtruW7738gJSWFF18ZT9kqNejW+0lmf/Kp7YgqFzt37eKhx3pQve7ldO/zFNt37LSaJykpiRGjx1C2Sg0ef7Iv8z5faDVPYZKnBXxEpCIwzxhTOy+D+msBn/mzZ9Km9Q0+jeMv8z5fyM0d7vZpjKCgIIzHw7KvF/l8Ta8dO3dSocZ/fBoDQIDixYuTmpZGRkYGLpcLj8dToLFuvbktc6ZP8zmTv0hUDMu+WkjTJlfajgJA9bqXc+P11zJ+zGifxtm4aTN1rmiKiJCZmenTWFPffIP7O/p2XAOUjqvM0WPHfD6GOt1zN+9OfsPnPP7y+JNPkzjgWUqVKunzWGdawCfY55G9RCQeiAcoHxfn83iHd2+nRIkSPo/jLy2bN2PsqJH5PriOHjvG4OHPExoaisvlokune6lR3felFeNiY3ln4mscPHQo3/d94pkBiAjh4WG0aNqUFk2v4pXX3iAlJYW048e56soruO2Wdvket6XDLpPjtGNo1bIlREZG+jxO+bhY+jzejdcmvklwcDBpaWk88Xh3YsuVzfdYbW+80ec8AMMSBjJoyHBS09JIT0/n2qtb0ObGVvke5/prWvolj7+8+sYk7u5wW0AfyB07Ey4qUlJSqHNFU268/loG9XuaSy652HYkbu5wFxkZmbwwdDB162TNpjMyMpj6wXSGjBzFpPFjubHVdZZTqtzs33+AF156mVlzP2X5t4u58MLSVvOkp6czZer7PP/iS0yd/EaRuHadP3+bOtNMWEtYKaXO4FyUsL5ETSmlLMrLS9Q+BH4ELhWRXSLyUOBjKaXU+SHXJ+aMMR3PRRCllDof6ekIpZSySEtYKaUs0hJWSimLtISVUsoiLWGllLJIS1gppSzSElZKKYv8toCPUkoVFf/8s5+Vq9cAsOS7ZVxYujTVq1UNyL50JqyUUqd4afwE2t91DwBDRo7i/ocfDdi+tISVUuoUD3fuRFBQEAChISH06dEtYPvSElZKqVNUqVyJW9u1xeVyUbJkSe647daA7UtLWCmlTmPocwPweDwkPNsve1YcCFrCSil1Glu2bgPgjy1bArofLWGllDqFMYan+g8EYPzrEzl06HDA9qUlrJRSp/jy62/40zsTdns8jB47LmD70hJWSqlTrFm3jvT0dADS0tJYs+6XgO1LS1gppU7R78k+ZBzZD8Cyrxby+dxZAduXlrBSSlmkJayUUhZpCSullEVawkopZZGWsFJKWaQlrJRSFmkJK6WURVrCSillkZawUkpZpCWslFIW5VrCIhInIt+IyCYR2SAivfI6+J49ewO6+pAqmL927+bw4cLxfdn1118cOXLEdoxsSUlJbN+xw3aMbG63m82//W47hnXHjh1jx86dtmMUSF5mwpnAk8aYmsCVQHcRqXW2O2RkZPBYzz6Ur1GbO+5/wB85lR9dfWNbylSpwYDBQx1bxjt37aJL1+5UrFmHTo90tR2HpKQkRoweQ5nKl9Kw+TW24+B2u5n+0Wwq1apLzQaN2bptm+1IVj3SoxeVatWl08NdHfUgmRdijMnfHUQ+AV41xnx5ptu4XC4TEhKSvQpRQbRs3ozP584iPDy8wGP405q16yhZMoaKFSrYjgLAgQMHaXrdjfz2v/8VeIyI8HDEJbw5YTwd7+zgUx63282CRV9wc5ubfBoHIHHE8zz/4st4jPHpGGrd6jo+mfkhoaGhPuVZsOgL7u7UBbfbTUpqaoHHKVe2LMuXfk3ZMmV8yrNz1y6aXdeag4cOkZSc7NNYSxbO4+rmzXwaw18yMzO5/Z5OfDp/QYHHCAkJIcjlos/j3RmR+JzPmSQqhh8Wf0GTKxr7PFbDZi1ZuXqNnLo9X5e8F5GKQH3g59P8WzwQ7/34pMuBlC59Qf7SknXABgfnK15ANWh6NQ3q1WXV99/ajgJAWFgocbHlOHDoYL7vu3//AQBcQS7CQsOIjo72Oc/m337nljs6YpJ9n1lfUKoUQUFBiMeTva0gx1DZMmX8clma4sWKERkZQXJyik95KpSPIzzM90lFcFAwpUqV5MDB///eiwgXXFAqX+NERUYSXby4z3n8xeVyUa5smQJ9bU8c00EuFy6XK99fizP5/utFXNGooV/GOpM8z4RFpBjwLTDcGPPx2W5bv24dc+vNbRn10iu0bNGM+R9/5IeodklUDBUrlGfrxsCtK3qu1GrQmL379jFkYH8eefABwsLCfB5zw8ZN1G7UxC8lDHDo0GFefGU8L417lZvbtGbm++/6ZdyCysjI4L0PZ9A/IZGgoCD++mOz1TzGGBYvWcpT/Qfyy/oNbN24jvJxcVYz2fTAI12ZMXsOPbs9Sr8+vf1Wwv50pplwnkpYREKAecAiY8xLue6sQX2zctkSjh49SkhICBEREQUK7SRFqYQPHz5MRESEX8r3BH+X8AmByOqLjIwMjh1LolSpkrajAFllvG/fP1x88UW2o1iVnp5OcnIKJUvG2I5yRgU+HSEiAkwBNuWlgHPyx6+5yv9iYpx7oJ7KaVlDQkIcU8CQdRrifC9ggNDQUJ/P/duSl1dHNAXuB64VkbXeP20CnEsppc4Luc6EjTHLgH9NoZVSSvlO3zGnlFIWaQkrpZRFWsJKKWWRlrBSSlmkJayUUhZpCSullEVawkopZZGWcB6kpGQt3HL8eDqpPqyiVVRlZGRkry6WnJyM2+22nEipwkNLOBcbN22m2EXlANizdy8xZStw8OAhy6mcpe3td9K4xbUAFL84lv4JQywnUqrw0BLORZXKlSiZY/2CqpUrOXqREBtat7qeqMhIACIjImh1rf1Fz5UqLLSEcxEWFkbiwGeJiowkKiqKF0cMI2tNI3XCYw93ITgk6x3wVatU5rprrracSKnCQ0s4Dx558AFCw0KpEBdL6xuutx3HcSIiInju2X4AjnqQ2r5jh09X5ijqjh07xt9/77MdI1tmZibbtm+3HeOc0xLOg7CwMJZ8Po9PZn7omIJxmsce7sL82TMdMwv+c8tWKl9WjzfefMt2FMfq/Gg3ajdq4pgHqremvk/ly+qddxcu1RLOozr/qU3VKpVtx3CsiIgI2rS+wTEPUgMThwEwePjzpKWlWU7jPL/9/j8WLPqSlNQUpkx933Yc0tPTGZg4FID+g8+vJ3a1hFWR8+eWrcydNx+Px0N6ejqT3nrHdiTH6T94SNZLC1NSeW7IMOuz4Xfe/4CU1DSMMXz+xVfn1WxYS1gVOdM/mk1mZiYA6RnpvPnue5YTOUtaWhrzP1+U/XruAwcP8f2PP1nNNPntd7MfCNyZmUybUfivS5lXWsKqyOnVvSs/fvMlAAn9n2HBxzMtJ3KW8PBw1v38PVUqVwJg+dLF1i97P3fGNMaMzDqF9MM3X/J0n55W85xLWsKqyClWrBgNG9QHoHatmsSWK2c5kfNcWr0alSpUAKBhg/q4XHaroFzZsvznslrZeYoXL241z7mkJayUUhZpCSullEVawkopZZGWsFJKWaQlrJRSFmkJK6WURVrCSillkZawUkpZpCWslFIW5VrCIhIuIstFZJ2IbBCRxHMRzBcej8d2BEfzeDwYY2zHKLT0+FL+lJeZ8HHgWmNMXaAe0FpErgxsrIL78eflxJStwNMDn+PAgYO24zjS1Te25fKmV7N4ybdaxvnw48/LaXrdjdS70u46C6poCc7tBibrpzTJ+2mI90/Af3JXr1nLxAIsQfjzipUcT0tj/OuTmPDGZLp3fYSRiQkEBQX5lOfgwUNERIQTERHh0zj+4na7GfbCaHbv2Zvv+y774UcAbrmjI9WrVWXyhHFcXr+ez5l279lD2TJlfB7HaTZu2swjPXqx9pdfSEnJuqr0o4/3zvc4F5QqxeABzxAaGurviAVie72IwuBcHNO5ljCAiAQBq4CqwARjzM+nuU08EA9QPi7O52DzF33h2zqwGRmEh4fz1rvvM3TQAJ9LuPHV1/Kfy2oxZ/o0n8bxlz179zJ4+PM+jZGalsa6X9ez7IcffS7hP/7cQrU6DTDJh30ax59GjxhK86uu8nmc1WvX8dPyFSctWF/QY7PzffdQvVpVnzP5w5iRw/hl/QbbMbI1qFeXkUMSbMc4SbmqNVn9/bfUr1c3YPuQ/Pw6KiIxwBzgcWPM+jPdrmGD+mblsiW+pyuAT+bN5/aO9xNdIpohA/vzcOdOhIeH+zyuRMVQsUJ5tm78xQ8p7apQoza79+zlphuuZ2RiApfVqunzmBs2bqJ2oyaOKmF/+uPPLQxIHMrcT+cRGRnJod3n37XQzkcSFcOyrxbStInvZ2AbNmvJytVr/nXpmTzNhE8wxhwWkSVAa+CMJWzTDdddy4z33qZt6xv9Ur5F0Yypb1O8WDG/lO/5omqVysyY+jZ//LmFv3bvth1HFSG5lrCIXAhkeAs4ArgeeCHgyQooIiKC229tbzuGo13ZuJHtCIVW1SqV9VqDyq/yMhMuA7zrPS/sAmYaY+YFNpZSSp0f8vLqiF+A+ucgi1JKnXf0NSpKKWWRlrBSSlmkJayUUhZpCSullEVawkopZZGWsFJKWaQlrJRSFmkJK6WURVrCudi5axc16me9zXfb9h00uKoFSUlJudzr/NLrqX5c2+ZmAKrUrsfrk6dYTqSUb96e+j5VametLNju9rvo2rNPwPalJZyL8LBwtm7dlv35jp27HLMerFNkujM5eChr9bS/du/Rr48q9EJDQ/lr9x4AkpKTycjICNi+tIRzceGFpekW/zBhYWFERUUxNGGglswpBvV7muDgrHfAl4iOptM9d1tOpJRv7r7jdkpfUAqA4OBgBg94JmD70hLOg/59n0SAiPAwHup0n+04jnPJJRfTpdO9iAgjEp8jJCTEdiSlfBIUFMQLQxMREe65swNxsbEB25cjSzg5OZkKNWrz0cdzbUcBsmbDE8aO4d3JbzhmFvz8mJep36S5Yy46Oajf0/Tp0U1nwWdgjOHqG9rQt/8g21FUHt19x+082asHiQOfDeh+8nVljbzy9coaz495mUFDhlO2zCVs2bDO50sTFTVHjx6lXNWaZLrdTHtrEre1v8V2JJWLb75dStvb7wQD2zb9ykUXXWg7kjrHznRlDcfNhJOTkxkxegyZmZkcPHiIWXM+sR3JcV6Z8AZuj4e0tDT69n/OMbNhdXrGGJ7qP4jU1DQ8xjBs1Iu2IykHcVwJv/3eNFJSUnG5XKSmpTFo6HDbkRwlMzOTUS+/Qnp6Oi6Xi207dvDFV4ttx1JnsWLVatas+4WgoCAyMzOZ+OZbJCcn246lHMJxJdy+XRtGDUvE4/HQoulVjH1hpO1IjhIcHMykV18hMjISj8fD6OFDaHKFXq7IyerUvowxI4fhdrtxu91MHD+WqKgo27GUQzjynDBkXeV08oRxPNy5k39CFTEdH3iI6bNmF9mrGxdFEhVDo8sbsHyp/uZyPio054SVUup8oiWslFIWaQkrpZRFWsJKKWWRlrBSSlmkJayUUhZpCSullEVawkopZZGWsFJKWZTnEhaRIBFZIyLzAhlIKRV402aEULFGcVzFoqlYozjTZuga0LbkZybcC9gUqCCBlpSUxIjRY7i3yyO2oyhl1bQZIcR3j2D7ThfGCNt3uojvHlGoi3jKO1O5p/ND/Lllq+0o+RaclxuJSCzQFhgOPBHQRH6WlJTEuNcn8vyYl8nMyCQjM5MRg/O/sLaIEBcbi8i/3vqtvDweDy6Xc85wOS2PUwxICCcl9eTjOCVVGJAQzr13Be5aanmxf/8BklPyv8LcBzNnseS7Zcz5bD63tmvDsIRBVKlcyec85+IYylMJA2OBp4HiZ7qBiMQD8QDl4+J8DnZ186Y0alDf53He+3AGAwYPPWlbxZp1CjTW8qWLaXR5A58z+cM9d3UgLMwZV/kA2LNnL2Wr1nDUgkIRF1zCiqWLqfOf2rajAPD4Y4/SsH492zHYsev0E4kzbT9X0tPTubBCFZ/GSEtLY+bHc/l73z8s/vwznzMFFS/Fn+vXUrlSRZ/HOpNcS1hE2gH7jDGrRKTlmW5njJkETIKsVdR8DbZk4XxfhwAgvktnSpUsSd8Bgzhw8BAej4fUA3v9MrZNN7e5iZvb3GQ7RraDhw7ZjvAv6enpbN2+3TElPO7FF2xHAKB8rGH7zn8XbvlY/6+omB+hoaEFfhC/veN9fLpgIWGhofTu0Y2nevXwW649e/faLWGgKXCLiLQBwoFoEXnfGFMorngZFBTEXR1uo8N/2zNrzif8un6D7UhKWTU8MY347hEnnZKIjDAMT0yzmMo3D9x7D7Vq1uTJnt2JiYmxHSdfci1hY8yzwLMA3pnwU4WlgHM6UcZ3dbjNdhSlrDpx3ndAQjg7dgnlY7MK2Pb5YF/c0q4Nt7RrYztGgeT1nLBSqgi5966MQl26RUm+StgYswRYEpAkSil1HtLX7yillEVawkopZZGWsFJKWaQlrJRSFmkJK6WURVrCSillkZawUkpZpCWslFIWaQkrn82YNZv+CUMAeKxnH7774UereVavXcdjPfsAMHzUGN6a+r7VPKrw+Wn5iuxjKGHYCKZNnxmwfenblpXPpn4wnQWLvgRg8jtTKV26NM2vamItz9Lvf2DyO1MBWLFqNS6Xiy6dCt1yJ8qir5d8m30Mfb1kKcbAvXffGZB96UxY+WzE4OeIiAgHIDwsjCce7241z6NdOlO8WDEAoiIjeWHoYKt5VOHzeNf47GM6IiKC5wN4DGkJK5/VrfMfWjRtisvlome3rpQsaXcpwYiICAY+05egoCBq1biUq5s3s5pHFT7R0dH07d0Tl8tF44aXB/RiDo4sYY/Hw70PPsKKVattR3Gs+QsX0fOpfrZjZOv5WDzFixWj+6POuIZf/IMPEB0dTTeH5HGi2XM/5alnB9qO4Vi9uz9Gg7p1eXHE0Nxv7ANHlvCsOZ8wfdZsej/9rO0ojuR2u3n08d68PnmKYx6o3p8+k2NJSXww8yPbUQCYNfdTjhw5wgcznJHHadLT03msVx/GvT6R9Rs22gDHQiQAAAlJSURBVI7jSNHR0axY9g0N/XCZtbNxXAl7PB76DhiEx+Nh7S+/8MNPP9uO5DjTP5rNkSNHcbvd9B3wnO04bNm6jTmfzcfj8TD8hRdJSUmxmiczM5P+CYl4PB6W/fgTa9aus5rHid55/wNSUtNwu90889xg23HOa44r4VlzPmHP3r8BSElJdUTJOInb7abfoASSkpMxxvDd9z9Ynw0PTBxKeno6ACmpqUyY9KbVPO99OIMDB7OueZeamkq/QYOt5nGa9PR0BiYOJTk5GY/Hw+dffKWzYYscV8JlylxCsyZXAlCyZEz2xyqLy+Xi+muvyf68ZfNmlCpZ0mIiqF+vLrVr1QKgWZMrqVyxotU8sWXLZh83ceXK0aRxI6t5nCYoKOikY+iaFs2JiSlhMdH5TYzx/xVWGzaob1YuW+LTGBIVw+QJ43i4cyf/hCpiOj7wENNnzXbMJeY3bNxE7UZNHJMHso6huTOm0b5dW9tRHKlVu1v56psljvqeFWUNm7Vk5eo1/7rMteNmwkopdT7RElZKKYu0hJVSyiItYaWUskhLWCmlLNISVkopi7SElVLKIi1hpZSySEtYKaUs0hJWSimL8lTCIrJNRH4VkbUisjLQoZRS/y8jI4OXX32N337/n+0oKo9279nDmHGvcvhw7m8Jz89M+BpjTD1jTMOCR1NK5dff+/bRt/8g6jVpzu333K9lXAgsWPQlzwwaTGy1WgxMHHbWMnb06YjysbG2I2R7bfIUPlvwue0Y2RrUq8slF19kO0a24sWL2Y5wWhdf5Jyv0QczPkKiYvL9J676ZbjdbtLS0pj72XxqNmjMqjVrfc7T5IpGVK1S2Q//M//Yt+8fOsc/ZjvGSW7reF+BvmePdO+Jx+MhOSWFF14aS5XaZ14YPk+rqInIVuAQYICJxphJp7lNPBAPUD4u7vLtm38t6P8bgOTkZKKionwaw58kKoZKFSuwZYMzFgg3xpCamkpkZKTtKNlSUlI0z1kkJSUx97P5+b7f3/v28VT/QUSEh+NyuejdoxvPPNmbYsV8e+Bz2jG0ZOl3XHPTzY5a1e33//3B8pWr8n2/OZ/N47MFCwkLDaVEiWhGDRvCS+MnnHYVtbxe8r6pMWa3iFwEfCkim40xS3PewFvMkyBrKct8pz6Fkwr4hEAs+1lQIuKYH54TNM/ZFStWjPs63pXv+x05coS335vGf2+5mSd7dicmxj8XUnXiMeQ01atVpXq1qvm+X/m4WDZu2szAfn25+47bCQoK4qXxE0572zyVsDFmt/fvfSIyB2gMLD37vZRS/lCiRAnWr/zJdgyVDy2aNWXTmhV5um2u54RFJEpEip/4GLgBWO9TQqWUUkDeZsIXA3NE5MTtPzDGLAxoKqWUOk/kWsLGmC1A3XOQRSmlzjuOfomaUkoVdVrCSillkZawUkpZpCWslFIWaQkrpZRFWsJKKWWRlrBSSlmkJZwLj8fDnE8/A2Db9h0s+vJry4mUKnp+Xb+Bhd6frdlzP2Hv3r8tJzp3tIRz8cuv67mt4/0EuVwEBwdz0387cPDgIduxlCpSunTtzsuvvgZAx84PM2HSZMuJzh0t4VzUrfMfal9WC7fHgzGG9u3aUKpUSduxlCpS+vbpRWhICABBQUHEd+lsN9A5pCWcCxFhzIhhREVGEhISwsjEBNuRlCpyOvy3PaVKlcTlcnHPnR2Ic9AFHQJNSzgPWl13DZUrVaR1q+uocWl123GUKnJcLhejhw/FJcLgAc/YjnNO5enKGvkeVOQfYLsfhioN7PfDOP6iec7OaXnAeZk0z9k5LQ/4L1MFY8yFp24MSAn7i4isdNKFRTXP2TktDzgvk+Y5O6flgcBn0tMRSillkZawUkpZ5PQS/tdVnS3TPGfntDzgvEya5+yclgcCnMnR54SVUqqoc/pMWCmlijQtYaWUssiRJSwirUXkNxH5Q0Ssv3JbRN4SkX0ist52FgARiRORb0Rkk4hsEJFelvOEi8hyEVnnzZNoM88JIhIkImtEZJ7tLAAisk1EfhWRtSKy0gF5YkRklohs9h5LTSxmudT7dTnx56iI9LaVx5upj/d4Xi8iH4pIeED247RzwiISBPwOtAJ2ASuAjsaYjRYztQCSgKnGmNq2cuTIUwYoY4xZLSLFgVXArba+RiIiQJQxJklEQoBlQC9jzE828uTI9QTQEIg2xrSzmcWbZxvQ0BjjiDcjiMi7wHfGmDdFJBSINMYcdkCuIOAv4ApjjD/e9FWQDOXIOo5rGWNSRWQmsMAY846/9+XEmXBj4A9jzBZjTDowHWhvM5AxZilw0GaGnIwxe4wxq70fHwM2AeUs5jHGmCTvpyHeP1Yf3UUkFmgLvGkzh1OJSDTQApgCYIxJd0IBe10H/GmrgHMIBiJEJBiIBHYHYidOLOFywM4cn+/CYsE4nYhUBOoDP1vOESQia4F9wJfGGKt5gLHA04DHco6cDPCFiKwSkXjLWSoD/wBve0/ZvCkiUZYznXA38KHNAMaYv4AXgR3AHuCIMeaLQOzLiSUsp9nmrHMmDiEixYDZQG9jzFGbWYwxbmNMPSAWaCwi1k7biEg7YJ8xZpWtDGfQ1BjTALgJ6O49zWVLMNAAeN0YUx9IBpzw/EsocAvwkeUcJcn6DbwSUBaIEpH7ArEvJ5bwLiAux+exBOjXgMLMe+51NjDNGPOx7TwneH+lXQK0thijKXCL9xzsdOBaEXnfYh4AjDG7vX/vA+aQderNll3Arhy/scwiq5RtuwlYbYyxfWmN64Gtxph/jDEZwMfAVYHYkRNLeAVQTUQqeR8V7wY+tZzJUbxPhE0BNhljXnJAngtFJMb7cQRZB/BmW3mMMc8aY2KNMRXJOn4WG2MCMovJKxGJ8j6JivfX/hsAa6+2McbsBXaKyKXeTdcB1p78zqEjlk9FeO0ArhSRSO/P23VkPffid8GBGNQXxphMEekBLAKCgLeMMRtsZhKRD4GWQGkR2QUkGGOmWIzUFLgf+NV7HhagvzFmgaU8ZYB3vc9qu4CZxhhHvCzMQS4G5mT9PBMMfGCMWWg3Eo8D07yTnS3AgzbDiEgkWa+KetRmDgBjzM8iMgtYDWQCawjQ25cd9xI1pZQ6nzjxdIRSSp03tISVUsoiLWGllLJIS1gppSzSElZKKYu0hJVSyiItYaWUsuj/AIFBMJUKoSstAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "grid.plot(policy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initially, the policy chooses actions randomly. The value function of this policy shows which states are better for an agent to be when selecting actions randomly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Policy Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-20T11:55:34.626615Z",
     "start_time": "2020-04-20T11:55:34.239615Z"
    }
   },
   "outputs": [],
   "source": [
    "evaluator = PolicyEvaluation(grid, policy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Policy Evaluation iteratively uses the Bellman equation as an update rule to obtain better estimations of the true value function of the policy\n",
    "\n",
    "$\n",
    "u_{k+1}(s) = \\sum\\limits_{a}\\pi(a|s) \\sum\\limits_{s^\\prime, r} p(s^\\prime, r|s, a)\\left[r + \\gamma u_{k}(s) \\right]\n",
    "$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-20T11:55:37.951614Z",
     "start_time": "2020-04-20T11:55:34.632625Z"
    }
   },
   "outputs": [],
   "source": [
    "for _ in range(20):\n",
    "    evaluator.evaluate_once()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-20T11:55:38.664601Z",
     "start_time": "2020-04-20T11:55:37.954616Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWEAAAEICAYAAACOBEVFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deZyN5f/H8dfnzGLMQtnXQRIlWUL6KpUk2lUqISprZSlLspPI0qK0EUpkSZaS0ipabKFUSrIv2bcZs51zrt8fc8xvrLPc57hufJ6Ph0fmdu7rvJs58577XOfMdYkxBqWUUnZ4bAdQSqkLmZawUkpZpCWslFIWaQkrpZRFWsJKKWWRlrBSSlmkJaxcS0TeFpF+Z/h3IyKXns1MgfttLSI/nOHfF4pIm7OZSZ27tITVWSEiD4nIUhFJFJHdgb8/ISJyunOMMR2MMc/n4r6aicifJxz76jTHeuV0fKWCSUtYhZyIdANGAyOBYkBRoANQF4g8zTlhDu7ye+ByESkcGCscqApEn3DsWmBRTgYOnKdU0GgJq5ASkfzAYOAJY8xMY8wRk26VMaa5MSYlcLv3ROQtEZkvIonATYFjQzKN1UNEdorIDhF57HT3aYzZAWwA6gUO1QD+IL2cMx/zACtEJL+ITBKRPSKyWUT6iogncJ+tReRHEXlFRPYDA0/x/3iLiPwlIodEZAxw2qt7pU6kJaxC7VogDzA3G7d9GHgBiAOOm3MVkUZAd+AWoALQIIuxFvH/hVsPWBwYM/OxJcaYVOB1ID9wCXAD8AjwaKaxriG91IsE8mXOVQj4GOgLFAL+Jf0KX6ls0RJWoVYI2GuM8R47ICI/ichBEUkSkXqZbjvXGPOjMcZvjEk+YZwHgInGmN+NMYmc4or0BJmveq8nvYQXn3Ds+8C0x4PAc4Gr9E3AS0DLTGPtMMa8bozxGmOSTrif24A/A1f5acCrwH9ZZFMqg5awCrV9QKHMc6nGmP8ZYy4K/Fvmx+DWM4xT4oR/35zF/S4CrhKRi4E6wM/GmL+A4oFj1wVuU4j0eenM420GSuYml0lfEetMt1fqOFrCKtR+BlKAu7Nx2zMt6bcTKJ3p4/gzDmTMBmAH0A7YYoxJyJSnHRALLAH2AmlAmRPG3p6bXIF3e5Q+/c2VOp6WsAopY8xBYBDwpojcLyKxIuIRkWpATA6GmgG0FpErRCQaGJCNcxYDzwT+e8wPgWMrjDFJxhhfYOwXRCRORMoE/n1yNnN9BlQWkXsDV/udSX8HiFLZoiWsQs4YM4L0YusJ7AZ2Ae8AzwI/ZXOMz0mfb/0WWB/4b1a+J/3FtMwv8i0OHMv81rROQCLpL779AHwITMhmrr1AU+BF0qdXKgA/ZudcpQBEF3VXSil79EpYKaUs0hJWSimLtISVUsoiLWGllLIoJIuRRImYOBf1e3yJArYjnETyuyxTxCnX0cm2xKNH2bB5K5UqlCciPAgPK4+zx4/P52Pt3/8QX6ok+fLFOc8DOF0S4p9/NxATHU2J4kF6B5vrVqhwXSBX2bRlC3v37jvpkxSSEo7Dw31Eh2LoXBnzxD22I5zEc/tDtiMcR4qVdXR+vbsfYt2GTdxwbW1GD+nvPFAeZ4+f50e+wpo/XyYiIoJl33zOGVbMzL6wiFyfuvinn7m1yQOkHT7Mb0sWUbBAEH4IO/xBFXwuLOFgfN2DpNb1N53yuNu+iuoctOSXVaxc8wd+v593p0xn9959VvMcPnyEka+9gc/nY/PWbSz4ZqHVPAA9+w0kOTkZ4/cz/JXXbMdRLqIlrBzrN/wVkpLT19tJTU1j1JvjrOZ5570PSEpOAdKnSXo/P9RqnhUrV7NsxUoAklNSeO3tcSQmJlrNpNxDS1g51vPJdjzWrCkAA7p3ofl92VkmInTuatyQAc92A6Bxg/oMeq6H1TxXVLqMwX2fy/j45aHPEx3tnuk6ZVdIfmOusIQZV80JD3HX/Cucf3PCf/y9jio33oZ/5/rgBHI4JwzgKVCC2ZMncPdtjYIQCEdzwgCe/IWpVaM6S7/7Mjh5dE44ay6bE16xctVJgdz2VVRKqQuKlrBSSlmkJayUUhZpCSullEVawkopZZGWsFJKWaQlrJRSFmkJK6WURVrCSillUbZKWEQaicjfIrJeRHqFOtQ/NGUKf/AOh5jCH/xD01DfpVJKWZFlCYtIGPAG0Bi4AmgmIleEKtA/NGURY0ggHvCQQDyLGHPOFnGK18fD07/hw9Xr8fn9tuMA0Of18Tw/9gMOJ+giMip3uj3XlxdfetU1CxFNmT6D9p26sn3HjuOOb922jTZPdGbqjJmWkmUtO+sJ1wbWG2M2AIjINOBu4M9QBFrGQLwnrDvhJZplDKQCH4XiLrPFGENSmi/H5+1PSmHW75v4/O+tPLdgGcNurc2DV10SlHmgpOQUcrP2xwfzvmLvwUOMen8G3Vs9QJeH7yV/EPKoc09SUlKuHkOTpk4nMSGRF196lV7dutKpQ1tiYmId50lJScHny/n32Uez5jLviwV8MG0GzR9oSttHH+Gd8ROZOvNjUlPTOHjwAHffcVuOxw0LCyNPnjw5Pi8nslPCJYGtmT7eBlxz4o1EpB3QDiDWwUIeCZTK0fGz5aPfN9B8+ne5Pj8h1UtCqpdWMxdSOCaKWx3m2bxjF5fc3tzRGCmpaQx8630SjiYx4kVnyz0WLliQwgXdtVvIZZeWp+Kl5W3HyPBYy4e5pmZN2zEyrPtnPZVqnPStnCPJKSn0Hvg8qamp9H/uWUdjpaamkrdQcWd5kpOZOHkKH82ZQ0JCIv7As89Zn8xjVpGSuRoz5cBuIiKcLd50Jtkp4VM16kk/Oo0xY4GxkL6KWm4DxbItMBVx8nGbHqhSngeq5Pwbev/RZEoMm0LeiDCKxOZlRKNruLl87h4MmZUpURTfqq9zd26jZuw5cJCoyEj6tWtJh6Z3Os5TpFBBdv2+zPE4wfTXssW2Ixzn3TGjbUc4zmUVLsV/JHcL8BctdxlHEhKJzpuXQX160aZ1S8d5IiMj8R/Zn6tz72v+CJ989jmRkZG0e7QVbVu3YuzE9xj33iRSU1O59+47mT5pouOMoZCdEt4GlM70cSlgx2lu61htBrKIMcdNSYRzlNoMDNVdhlRsZAQtq1fgjkrx3FmpDB6P/aX12t53OzFRUXRoeid5o0L7VEudn9o92opiRYvSpnXLkD9dz46H7r+XMqVL81y3pylcuBAAr44YRu8e3Rg26mWur/s/ywlPL8v1hEUkHFgH3AxsB5YDDxtj/jjdOU7XE/6HpixjIAmUIpZt1HY4H6zrCWfN6XrCQReE9YSDzuF6wkGn6wln7RxYTzjLK2FjjFdEngIWAGHAhDMVcDBU4COrL8IppdTZkq3dlo0x84H5Ic6ilFIXHLc9n1FKqQuKlrBSSlmkJayUUhZpCSullEVawkopZZGWsFJKWaQlrJRSFmkJK6WURVrCSillkZawUkpZpCWslFIWaQkrpZRF2VrAJ6fiSxRgzBP3hGLoXJG69W1HOIkUKGY7wvEi7K8Je5ywkDw0nXG4LOKQES9RuVJFmtx1R5ACqayIi5ayPB0XPtKVOv9s2LiJIcNHkS9fHLc3akhkZKTtSMoldDpCqbOg7/ND8RtDUnIKEz6YYjuOchEtYaVCbMPGTXw0aw5er5fExET6PT+U1NRU27GUS2gJKxVikZGR1K55dcbH19S8+pyYq1Rnh5awUiFWqmQJfvz6cwBq1ajOvJnTQrqFujq3aAkrpZRFWsJKKWWRlrBSSlmkJayUUhZpCSullEVawkopZZGWsFJKWaQlrJRSFmkJK6WURVrCSillUZYlLCITRGS3iPx+NgKF0ur1m5j+7U+2Y2TYuWsP70yeQUqKLuaizi1z583n56XLbcfIsOrX3/ho9hz8fv9xx30+H9NnzuLXNWssJctadq6E3wMahThHSK1ev4lG3YdS94kBtBn+ju047Nx3gI69B1P++kY81f8F/t6w0XYkpXKkXaenuem2u7j+lttcUcZDRoyi+WPtuOTKasyYNZu0tDSmzfyYcldcRYvH2/HCiJdsRzwtMcZkfSORssA8Y8yV2Rn06pKFzVKHO2v0WbCcdrUrUebiOEfjfLh6Pa1mLkREyM7/65ksHzuU6hXKOhpjw45dVH6kO0YEr9fraKy3hvanffMHHI2RISa/o9OPHEmg58AXeOulYcHJE4SdPrr16U+Pzk9SrGjRIAQCxNnsXcXqtWl4c31eH/VikPI4W4lt/b8buKxaLYcR0r+vXh0xlM4dOzgaKzU1lZgiJfH5fI7GCQsLo1DBAuzbtx+vw7Hy5MnDkV3bgrLgUs3rbmTFylUnfdGCtrOGiLQD2gHE5491PN6Ixb9yScE4Hq9ZydE415ctRtOb6vDpj7+Q5vXh8/sZ9USLHI+TN08kVS4p7SgLQPGCF9P9oTsYPetLIsLDSElNo33zB7i0bHyOx2py682O8wTLlu3beef9ycEr4SB4Zczb1Pvftdx9e2PbUQBYsegboqOjbcfIcEm5sowbM5rDR47k+Nxuz/XF4/GQJzKSm2+6gdsa3uI4T2RkJJ/M+JC/1v2T43NfGPkS+/cfICYmmnJlytC6xcNMmDSFzVu3kpiYSOFCBXmu+zM5HvfyihVDvuJd0ErYGDMWGAvpV8LBGtep0hfFMrV/Z9Zv/49+42ewYcduuja9zVqevHkieb7Ng3R/ticvjX2PKbM/o/dTbSlZLEhXa8q14uKcPasLNo/Hw+Otcn5BAjB/wZfkzZuXYYP6U/lyZxdKmTVueAuNc1HoRxISmDn7E0a+MIiGN9dHRHj6qSf48pvv6N6nHw/dfy9Pd3oyaDmD6YLZY+7SksWY2r+z7RgZLs6fnyE9ujCkRxfbUZTKsa8+nW07wnH69+pJ/149jzsmIjRq2IBGDRtYSpU9+hY1pZSyKDtvUZsK/AxUFJFtIvJ46GMppdSFIcvpCGNMs7MRRCmlLkQ6HaGUUhZpCSullEVawkopZZGWsFJKWaQlrJRSFmkJK6WURVrCSillketK+FByKl+s2wrAz1t28/uu/ZYTqays37CRbxenr9P8+dffsWv3Hqt59u8/wOdffQPA4p+WsPbvdVbzKHUmrivhSSvX0WTylwBM/XU99035ynIilZU2XXvQY8AQAJq0fJxhr46xmuet8RO5p1lLAMaMHc+DrdtYzaPUmbiuhB+qWp6IsPRYEWEeOl1b2XIilZUu7R8nMiL9ly89YR46PNrSap5WDz9EWFgYAJER4XTp2N5qHqXOxHUlXDgmL21rViLcI0SGeWjjcD1hFXp3N76VooULp69aVf9GKlW41GqeUiVL0Lzp/YSFhREdHc0jzYK08L1SIeC6EgbodWM1wkToX78GUREXzGqbOfLPxs18/t0i2zGA9HVpRwzqA8DQfr0sp0k34LkeiAgv9O8T8kW5VXAcOHCQydNmON4B51yTre2NcqrmFRXNsg/fdDTGkcSjREflyXha6YQUKOZ4jKCLdrbAd7077mfJLyvZvOpnigdjQfiIKEenG2M4fPgI+fPnc54FHG8lBOlbLsXEROPxBOlaw+9sq5ygc1ueMGcXTL0GPs+IV8fw1dyPuPmGeq7IFEy16tVnxcrVJ21v5MorYYC4mOigFPD5aMmKlaz87Xc84mHwS6NtxwHSF9AOWgEHSVxcbPAKWIXU/gMHGDN2PADd+w68oK6G9RF6Duo5cChJycmkpqXx/tSZ7Pxvl+1ISjkyYvQYfL707er/Wb+Bbxcttpzo7NESPsd4vV42bN6ScaXg8/lY9+9Gy6mUcmbVr2tISUkBIDUtjZWrf7Oc6Oxxz4SJypbw8HC2rVnGw+06MW32J6Ts/Nd2JKUcWzB7BgsX/0j9O+8lde9223HOKr0SVkopi7SElVLKIi1hpZSySEtYKaUs0hJWSimLtISVUsoiLWGllLJIS1gppSzSElZKKYu0hJVSyqIsS1hESovIdyKyVkT+EJEuZyPY+WTdhk2kpaXZjuFaR48eZdPmLaf8t23bd3Do0OGznEidTzZv2UpCQoLtGKeVnSthL9DNGHM5UAd4UkSuCG2s88fBQ4e5vP4dxNdpwITps7SMT6FLr76Ur1abZo+159+NmwDYum07jz/ZmXJVatCqw5N2A6pz0sZNm2nZpgPlq9SgY9futuOcVpYL+BhjdgI7A38/IiJrgZLAn6EM9sWPy7jh6qrkjcoTyrvJtpVr/qTOPc3wenO3kPauvfvoOvBF+owYzXfTJ1LpqqqO8uTL52xR+GDz+Xy0aNuR6bPm5nqMjz+Zx9zPPqdqlcqsXvMHxvjx+Xx8Mv8LPPkL53i8WxvUZ+7UD4iMjMx1pmBa9stKShYvTskSxW1HAWDX7j1s2LyZa2vVtB0FIKhrP/caMJjX3n4Xr8+H3+9nyvSPmDL9oxyP0+TO25k+aQLh4aFb6yxHI4tIWaA6sPQU/9YOaAcQX7yI42C3P9WbcQO68dg9jR2PFQyxMdFcEl+a/QcP5ei8o0nJHE1Kyvj4ovz5iIx0vt3OqIF96Ni6heNxgsXj8VC8WDEKFSyY43P37tsHQFhYGOIRChUoSFhYGMYvQPozh9yMW6JYMVdtDFCnwW3cfMP1fDUn52UQCh279WLO/C/w73PHqmXXXXsNPyyYF5SxChUsiHg8hJO+/OuxYzlVvHixkG8MkO0SFpFY4GOgqzHmpEk6Y8xYYCykb28UjHB+v3tW17/skrL89V3OHyCHDh+hULXruLRsPKP6dOe2+vUQOWmHkxyLjY2h6pXumRUSEV4eOpiXhw7O8blPde/FuPcn0+GxVvTu1oXChQpx4OBhXnr9DV4e8xZ3NL6VGe+PD0Hqs+/wkSO2I2RISEy0HeE4Ho+H/11TKyhjde/8JI+2aMbw0W/w2ltjaf7g/Yx/8/WgjB1s2SphEYkgvYCnGGNmhTbS+SV/vji2L/uWwgULBKV8z0evvjiEQb17UrBAgYxjF198EUP696F756fIm9fZ/nfqwlSwQAFGDBlEr2e6EhcXazvOaWVZwpLeHOOBtcaYl0Mf6fxTpFDOnwZdSMLDw48r4Mwuuij/WU6jzjcFClxsO8IZZWeyoy7QEqgvIqsDf24LcS6llLogZOfdET8A+jxaKaVCQH9jTimlLNISVkopi7SElVLKIi1hpZSySEtYKaUs0hJWSimLtISVUsoi15Ww3+8nMbDgTXJKCqm69KM6DyQG1mlISUklOTnZchpITk4mJTUVgMTEo5bTXNhcV8LjPv6M/HXvAqDLiDeo2ayj5URKOfPL6l+JK1UegN/++JOC5SpllLINycnJFLnsKhb/nL4YYlx8BX5attxangud60r4uhpViAos9ZgnMoKbalWznEgpZypVuJR8mRaQqXx5JWJiYqzliYqKolqVyhkLSsXERFO5UkVreS50rivhyuXLcvM1NfCI4BGhT9vmtiMp5UhMTAzPPdOFvHmjiImOZuTz/W1HYsTAvuSNiiJvVBTdn+xA/nz5bEe6YLmuhAGGdm6D3xha33UrRVy+ApJK5/P52LBpk+0YrvVU28cIDwvn8oqXcUPd/9mOQ51aV1OjahVEhK4d29qOc0ELzZ4dEZFIsXK5Pv0ifxRhHg8FS5ZxNE6GSHdskXSccHdsuZNBnP08nvbRLFq27cgvi7+letWrnOfxeZ2PEWzJuZ/HjQmDhbM+5KL8+RyNk5k5uNvR+ZOG9GLPgYPkSz6IST7oOI9clPMtqEIuT7TtBP/PnHqTCldeCQ966TXEI4x+d6LrVv9XJ/P5fPTqn76jxrP9B1lO417Vq1SmXHxp2zEylClZnJpXXm47xgXPdSW8fed/TPl4Ll6vD5/Px+vj37cdSWVh2sxZHDqcvuPVj0uWsurX3ywnUurc4boSnjnvc7yBp6KpaV4mTptpOZHKyrj3PiA1Lf09p16vj0lTp1tOpNS5w3Ul3Lb5Q/z0aXrxdnq8FV/PmGw5kcrK1IljmRDYRPHHr+YzoFdPy4mUOneE5oU5B6Kj81KzWvoLO5dXKE98qRKWE6msFC9WjKpVrgSgZo3qltModW5x3ZWwUkpdSLSElVLKIi1hpZSySEtYKaUs0hJWSimLtISVUsoiLWGllLJIS1gppSzSElZKKYuyLGERiRKRZSLyq4j8ISLn9DJZfr/fdoTjuC2P2/j9fsxplgBU6fQxdGZu//xk50o4BahvjKkKVAMaiUid0MYKvuUrV3PTXU2pVLue7SgAbN22ncef7ErBspexectW23Fcx+fzMe3jOZS7qiadevS2HceVNm3fSatnB1H42kb8t2ef7Tiu4/V6+WD6TOKvvJpeA4fYjnNaWa4dYdIvQxICH0YE/pwzlyb/bNhIu579WL5yNUnJyRhjaP/MszkeJ3++OAb36k5UVJSjPEeOJNC1zwCmfjwXv99Hamoa7bt0o0wO15kVEXo93ZmyZeId5XGj+Qu+omPXbhw4eIiExETeHD+RNG/OF3mvftWVdHisVQgS5s6evfvIny+OyEhnC/ofPHSYzn0GMXPefLw+H16vj/YDXqR44YI5GscjHvp0bE3JokUc5Qmm7Tv/o2TxYo7HmT3/Szr1HsjhhEQSEhMZ8dqbHDh0OMfj1KlZg0ebP+Q4z5lkawEfEQkDfgEuBd4wxiw9xW3aAe0A4ks6X3TnlcF9uevWmx2Ps2bt3yz+eSkejyfjae2496fkaqyWD9xHlSucLYJ94NAhZsz+FGMMqalpAHz57cJcjVW3zjWuKeFyZeJ5vu9zQRlr1ifz2LlrN95MxTvu/Q9yPE6hggVo/fCDjn9wBkvlG27l9gY3MXH0SEfj7N67j5mffY4x6UuHAsz77odcjVW/ztXc38j591kwLF/9G9c0vhf/zvWOx5o+9zP27N9PWlrmx1DOV2T88tuFtHjgPiIiIhxnOh3JyXybiFwEzAY6GWN+P93talatYpYvmBuEeMGx6b/d9B82io/mfIrH4yFxu/MvshOHDx/h1bETGPnamyQlJbFy8TdcdWVlq5mcbm8UbN8vWkTPfoNZ+etv3FL/RuZ/9KHtSI63JfIUL0+talex9PPZjqMcOHiIka+8ymsfzCA5JZW186dxaRm7u3Y43d5o4U9LqH9fi6CUsDGGb3/4me6DX2TNn39x3123M33CO47HdaJW/UasWPWrnHg8RyUMICIDgERjzKjT3cZtJXxsj7lNW7by78bN3HzDdZYDAeGRHD58hG++X8SdjW8lPNzyqqIuK+Fje8wt+vFnihcrSoXyl1gOhKtKGNL3mDtw6DDfL1/FXfWvx+Ox+zV0UwkfYyLz8t3iHylXJp5ylp81nq6Es/zOF5HCQJox5qCI5AUaAMNDkDHkysaXpqyL9vjKly+OJnfebjuGq9Wre63tCK52cf583NPgBtsxXEtEqF/PBRddZ5Cdy6/iwPuBeWEPMMMYMy+0sZRS6sKQnXdH/AbodglKKRUCLpsIVEqpC4uWsFJKWaQlrJRSFmkJK6WURVrCSillkZawUkpZpCWslFIWaQkrpZRFWsJKhdi/mzZTsW76SmXpK4U1ITk5xXIqd2nfvQ9NWncE4NI6NzFpxizLic4eLWGlQiwqTx42btmW8fH2/3YRHh5mMZH7pKSmknD0KJD++cmTJ4/lRGePlrBSIVayeDFaP3AfkRERxMZEM6x3D/ur5rnMoB5diQh8TooWKsT9dzSynOjs0RJW6iwY0L0zAPliY2nW5E7LadynTOmSPHDXbYgIw/s9S1jYhfNMITQ/jj0eiIrO9emJiUepXKceI58fQNN7gvCADQvdqvi55rb1e922Y5U/59sZZXbw0CGu/F993n55OHfc2iAokcz+Hbk+t3ikIb54Eapdfhlhh3cH5bPtf2NYEEYJHk/Hno7Ov+Xqynw873NurFQas3db1idkgxQpE5RxguI0a7e7rQkAeH3seHb8t4se/Qbh8/lsx1HnoJfeeIdde/bSrd8gV+zWvHDZKrbv2stnC39i974DtuO4jjGG4eOnkJySykvvTbcd56xyXQknJh5l2Muv4fV62X/gIDPn6tLFKmcOHjrEq2+Nw+fzseO/Xcxb8LXVPMYYeox8g6SUFPzGzwvvvG81jxt99fMKNm7bic/v541pc9h38JDtSGeN60p44ofTOJqUhMfjISk5if5DR9iOpM4x77w3meSUVDweD8lJyfQf6mxjTaeWr1nL6rXrCPN48Hp9jJ0xl8SjSVYzuc2AMRPSv2YipKZ5GfOhvkXNmrsb38rwQX3x+/3U+9+1vDJ0sO1I6hzT9O47GD6wD36/n7saN2TEoL5W81xVsTyjej6Fz+/H5/fz1oAexETntZrJbV7o0pZmtzfAbwzDn27Pw7ffYjvSWeO6Ei5dqiTPPNkBgGb3N+G2hu7YjludOy4pW4ZnnmgHwCMPNeWWm+pZzROVJw9dWz0IQK0rL6d1k9us5nGj+tfU4NEmjQF4utUDVChTynKis8d1JayUUhcSLWGllLJIS1gppSzSElZKKYu0hJVSyiItYaWUskhLWCmlLNISVkopi7SElVLKomyXsIiEicgqEdEVdZQ6x01ddwPlJ00g8s1PKT9pAlPX3WA70gUrJ1fCXYC1oQoSagkJCQwb9QotHm9vO4p1Tz/7HD37DmDfvv3HHf9pyTIaN2nKshW/WEqmzoap626gw8LObEkoisHDloSidFjYWYvYkmwt6i4ipYDbgReAZ0KaKMgSEhJ5fdxEXnx5NF6vlzSvlxcG5HxBFxGhdKmSiEgIUubO9h078Hpzvt7yhzNmcujQYd4c+y5PtGvDTfWu54URo1i9Zg2pKanUuroGRYsUyfG4F1+Un3z58uX4PHV29V3SiqPeqOOOHfVG0XdJK5pd9r2lVO7k9/vxeEI7a5vdnTVeBXoCcae7gYi0A9oBxJcq6TjYDXWvpVb1ao7HmTxjJn0Gv3DcsXJXVs/VWEu/+5JaV9dwnCkYtmzdRtkrrnI0RmpaGqNGv87rb71DmjcNv9/g8XgYMnwUQ4aPyvF4VSpfwa9LFjvKFEy1r67OFRUr2I6R4anm93F15Uq2Y7A1oXCOjp8tl5QqQdWK5a1mOFF48fKsX/o9l5SND3AXsvcAAAwWSURBVN19ZHUDEbkD2G2M+UVEbjzd7YwxY4GxADWrV3W8lcF384KznmjbVi24uEBBevYbyL79+/H7DUd3B2frFJviS5fCf2R/1jc8hTKVqrBn715iY2MZ0r8PN9a7jiHDRzJzzid4vV5GjxzOk+3aBDnx2bfky09tRzjO6N5dbUcAoHTsHrYkFD3lcZviixdl5czxVjOcys7du+2WMFAXuEtEbgOigHwiMtkY0yJkqYIoLCyMB+9rwv333MXMOZ+w5o8/bUeyrvMT7YmLjaV1i4eJjIwEYNK4txnctzejRo+hfr3rLSdUoTSkzvt0WNj5uCmJ6PBkhtTRHT9syLKEjTHPAc8BBK6Eu58rBZzZsTJ+8L4mtqNY163zU6c8XrZMPGNe1p1MznfH5n37LmnF1oTClI7dw5A67+t8sCWh2W1ZKeVqzS77XkvXJXJUwsaYhcDCkCRRSqkLkP7GnFJKWaQlrJRSFmkJK6WURVrCSillkZawUkpZpCWslFIWaQkrpZRFWsJKKWWRlrBSSp1gyS+r6NgzfcnbASNeYcrMOSG7L/21ZaWUOsE3i37k3cnTAPh28U9gDM3vvyck96VXwkopdYJObVqRNyp9lbm8eaMY1vfZkN2XlrBSSp0gX1wcPZ5si8fjoXb1qtSqXjVk9+XKEvb7/bRo9yTLV662HSXD4BdHMuGDKbZjZPjsiy/p0qOX7RgZNmzcxL0PP8LRo0dtR1EqKLq0e4zqVSozckDvkN6PGON4E4yT1KxezSz//utcnz9j1hwefrw9dWpdzQ9fznceKMzZ1PfmLVuoVL02kRERbF+/ltjYWMeRnOxV5/P5KFPpSnbt3sNP3wZnyyXj9zs6/+FH2zBj1hxeHDyA7l06Oc5DWrLzMYIt4YDtBMfx79hgO8JxPMXL2o5wsqjT7sh21tVqeCcrVv920je+666E/X4/PfsNwu/3s/q33/lp6TLbkRgwZBh+vx+fz8frb71jOw7TPvqYQ4cO4/P56NGnv+04bNi4iTnz5uP3+3lhxEt6NaxUDriuhGfO+YSdu/4D4GhSEj37DbSaZ/OWLUyZPoO0tDSOJiUxdNQrJCQkWMvj8/l4tt8AEhITMcaw+MefWP7LSmt5APoOHkJqaiqQ/jV7c5z79glTyq1cV8LFixXjujp1ALj4oou47tprrOaJjYnl5htvzPi44c03ZezLZoPH46FB/ZsyPr7x+usocPHF1vIA1KhalSuvuByA666tQ7myZazmUepc4so5YQBP/sKMfe1l2rRq6TyQwzlhAE/sxZQtE8+GP351ngdnc8IAzVo9zrSZH2MSDwYlj9M54T/WrqVK7bq53gH6JDonnCWdE84GnRNWSil1JlrCSillkZawUkpZpCWslFIWaQkrpZRFWsJKKWWRlrBSSlmkJayUUhZpCSullEVawkopZVG2SlhENonIGhFZLSIrQh1KKbdatHQFH3z8ie0YGdZv3cGrH84hJTXNdhQADh46zMvj3mfnrj22o1i1479dvPzWOA4eOpTlbXNyJXyTMaaaMaZm7qMpdW5a9NMSat/5II1atKVz/6G247B+02aa9R1OteadePaNiWy1XHoHjyTQd+RrlLqmAb1efIUvvv/Bah7b5n/9Hb2GvEjpanXoN2zUGcvY1Rt9xpcqZTtChrrX1uHyihVtx8hQo1pVFi5ebDtGhrhY9yyUckz7rj3o3/MZSpYo7micN8e/x1M9/n93heSUVDzxlXM11pqv5lK54qWO8vz+9z9Ua9gERPAHFl6q1LR9rsZ6t28XWt/RwFEegPJN2pCYnIzX60NEeLxHfx7vkfO1rpve3pDpb77kOE+w3Pdoe2bPX5Crcz0iJB5NYviYt3jrvQ8oVyb+lLfL1ipqIrIROAAY4B1jzNhT3KYd0A4gvnSpqzf9vipXwY9JTEwkJibG0RgZgrCKWnJyMhEREYSFhQUhkPNV1IwxJCUlER0dHZQ8TldRAzh69GjQ8gRjFTVPgRLMnjyBu29r5Gicbdt30Pv5YcycOw+v14vX52PSqy/meJy8UXlo0qgBHo+zl2KOJiUx6u2JjHx7PH6/n+TUNJ7v0JLSRQrleKw7rq9N/ljn32eTP/+W3m9P5vCRRFLSUrmnYX3ubHBjjse5pvpVVCgXxKVQHa6itu7fDSxblfOVE+fMX8CnC74mT55I8sfFMbx/b155591TrqKW3RIuYYzZISJFgK+ATsaYRae7fTCWsgyqIJRwsDkt4WALRgkHlYtK+JhNa3+n/0uvs2vPXhZMeTcoYzpx8J/fGT11LtO/XszCt4dR6KL8VvP4C5di2ief88KYcbw7fBB1a1W3mgewtpTlop+X0r57b/o+3YmHmtxJWFjYaZeyzPF6wiIyEEgwxow63W20hLOmJZwFF5awrid8Zrqe8Jnlej1hEYkRkbhjfwcaAr8HP6JSSl14snOJWBSYHbhyCwc+NMZ8EdJUSil1gciyhI0xG4CqZyGLUkpdcPQ35pRSyiItYaWUskhLWCmlLNISVkopi7SElVLKIi1hpZSySEtYKaUs0hJW553de/by8SefAfDVwkWsXqO/4KncS0tYnXfGTZrMw206AvDupA95pEMny4mUOj0tYXXeebzFw4SFp/8yaGREOD06P2k5kVKnpyWszjvFihbhseYPEhYWRr64OJrdd4/tSEqdlpawOi/17f40YR4Pwwb0ITzcfUuZKnVMjtcTztagInuAzUEYqhCwNwjjBIvmOTO35QH3ZdI8Z+a2PBC8TGWMMYVPPBiSEg4WEVnhpo1FNc+ZuS0PuC+T5jkzt+WB0GfS6QillLJIS1gppSxyewmftKuzZZrnzNyWB9yXSfOcmdvyQIgzuXpOWCmlznduvxJWSqnzmpawUkpZ5MoSFpFGIvK3iKwXkV4uyDNBRHaLiCtWghGR0iLynYisFZE/RKSL5TxRIrJMRH4N5BlkM88xIhImIqtEZJ7tLAAisklE1ojIahFZ4YI8F4nITBH5K/BYutZiloqBz8uxP4dFpKutPIFMTwcez7+LyFQRiQrJ/bhtTlhEwoB1wC3ANmA50MwY86fFTPWABGCSMeZKWzky5SkOFDfGrBSROOAX4B5bnyMRESDGGJMgIhHAD0AXY8wSG3ky5XoGqAnkM8bcYTNLIM8moKYxxhW/jCAi7wOLjTHvikgkEG2MOeiCXGHAduAaY0wwfukrNxlKkv44vsIYkyQiM4D5xpj3gn1fbrwSrg2sN8ZsMMakAtOAu20GMsYsAvbbzJCZMWanMWZl4O9HgLVASYt5jDEmIfBhROCP1Z/uIlIKuB1412YOtxKRfEA9YDyAMSbVDQUccDPwr60CziQcyCsi4UA0sCMUd+LGEi4JbM308TYsFozbiUhZoDqw1HKOMBFZDewGvjLGWM0DvAr0BPyWc2RmgC9F5BcRaWc5yyXAHmBiYMrmXRGJsZzpmIeAqTYDGGO2A6OALcBO4JAx5stQ3JcbS1hOccxdcyYuISKxwMdAV2PMYZtZjDE+Y0w1oBRQW0SsTduIyB3AbmPML7YynEZdY0wNoDHwZGCay5ZwoAbwljGmOpAIuOH1l0jgLuAjyzkuJv0ZeDmgBBAjIi1CcV9uLOFtQOlMH5ciRE8DzmWBudePgSnGmFm28xwTeEq7EGhkMUZd4K7AHOw0oL6ITLaYBwBjzI7Af3cDs0mferNlG7At0zOWmaSXsm2NgZXGmF2WczQANhpj9hhj0oBZwP9CcUduLOHlQAURKRf4qfgQ8InlTK4SeCFsPLDWGPOyC/IUFpGLAn/PS/oD+C9beYwxzxljShljypL++PnWGBOSq5jsEpGYwIuoBJ72NwSsvdvGGPMfsFVEKgYO3QxYe/E7k2ZYnooI2ALUEZHowPfbzaS/9hJ0rlto1RjjFZGngAVAGDDBGPOHzUwiMhW4ESgkItuAAcaY8RYj1QVaAmsC87AAvY0x8y3lKQ68H3hV2wPMMMa44m1hLlIUmJ3+/Uw48KEx5gu7kegETAlc7GwAHrUZRkSiSX9XVHubOQCMMUtFZCawEvACqwjRry+77i1qSil1IXHjdIRSSl0wtISVUsoiLWGllLJIS1gppSzSElZKKYu0hJVSyiItYaWUsuj/AIVkuTke9fJ3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "grid.plot(policy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, the states that are close to the goal have high value and the states far from the goal have all 0 value.  \n",
    "This basically tells us that when we make random moves, if we are close to the goal we are likely to reach it. But if we are a little farther away, chances are that we will not get to the goal.  \n",
    "\n",
    "This intuitive result shows that the value function exponentially decreases as we get far away from the goal since there is about $(\\frac{1}{4})^n$ probability to reach the goal when we are $n$ steps away from it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Policy Iteration\n",
    "Policy Evaluation and Policy Improvement are executed sequentially to find the optimal policy.  \n",
    "At every step, the value function estimation is updated and based on it the best action at every state is calculated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-20T11:55:39.228187Z",
     "start_time": "2020-04-20T11:55:38.668601Z"
    }
   },
   "outputs": [],
   "source": [
    "policy.reset()\n",
    "improver = PolicyImprovement(grid, policy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-20T11:55:43.459279Z",
     "start_time": "2020-04-20T11:55:39.232168Z"
    }
   },
   "outputs": [],
   "source": [
    "for _ in range(10):\n",
    "    evaluator.evaluate_once()\n",
    "    improver.improve_once()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-20T11:55:44.286165Z",
     "start_time": "2020-04-20T11:55:43.462167Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWEAAAEICAYAAACOBEVFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3dd3iUVfrG8e8zk07oICAgRZAigtIExYIFsfddxdXV37pgw7oKCioquIq9i0hRqdIsWFAp0qSDNJHeayiBtKnn90eGbGghyfsOZzDP57q4JMPMmVuY3PPOO5PniDEGpZRSdnhsB1BKqZJMS1gppSzSElZKKYu0hJVSyiItYaWUskhLWCmlLNISVjFLRD4WkWcL+HMjIvVOZKbI/d4tItML+PMpInLvicykTl5awuqEEJHbRGS2iGSKyM7I7x8QETnWbYwx9xljXirGfd0uIssPu+znY1zWvajrK+UmLWEVdSLyBPAO8BpQFagC3AecDyQc4zZeB3f5K9BIRCpH1ooDmgEph13WFphalIUjt1PKNVrCKqpEpCzwIvCAMWa0MeaAybXQGHOHMcYXud5gEflIRL4XkUygfeSy3vnWelJEtonIVhH5v2PdpzFmK7AWuDByUXNgGbnlnP8yDzBPRMqKyOcisktENohITxHxRO7zbhGZISJvicgeoNdR/h8vF5EVIpIuIu8Dxzy6V+pwWsIq2toCicDXhbhuJ6APUBo45JyriHQE/gNcDtQHLjvOWlP5X+FeCEyLrJn/slnGGD/wHlAWqAtcBNwF3JNvrXPJLfVTIvny56oEjAF6ApWANeQe4StVKFrCKtoqAWnGmODBC0RkpojsE5FsEbkw33W/NsbMMMaEjTE5h63zN2CQMWapMSaToxyRHib/Ue8F5JbwtMMu+zVy2uPvwNORo/T1wBvAnfnW2mqMec8YEzTGZB92P1cByyNH+QHgbWD7cbIplUdLWEXbbqBS/nOpxpjzjDHlIn+W/zG4qYB1Tj3szzcc536nAk1FpDzQBvjNGLMCqBa5rF3kOpXIPS+df70NQPXi5DK5E7EKur5Sh9ASVtH2G+ADri/EdQsa6bcNqJnv69MKXMiYtcBWoDOw0RiTkS9PZyAVmAWkAQGg1mFrbylOrsinPWoe++pKHUpLWEWVMWYf8ALwoYjcIiKpIuIRkbOBUkVY6kvgbhFpLCIpwPOFuM004PHIfw+aHrlsnjEm2xgTiqzdR0RKi0ityJ8PKWSu74AzReSmyNH+w+R+AkSpQtESVlFnjOlLbrE9BewEdgD9gG7AzEKu8QO551snAasj/z2eX8l9My3/m3zTIpfl/2haVyCT3DffpgPDgIGFzJUG3Aq8Qu7plfrAjMLcVikA0aHuSilljx4JK6WURVrCSillkZawUkpZpCWslFIWRWUYSZKIKR1D/V7WGztZDkqOdzKfxn3xyfG2IxxCko4618cqSUyyHeFQCYm2Exwq1vIAeGPncb1+02bSdu85Yq5IVEq4NB5uJiUaSxdLxzJF+TjqiXFWldK2IxyiWpPY+mhrQqM6tiMcQU4/w3aEQ9U63XaCQ0j1Ez7a+bik7Cm2I+Rp1eHao14ee4eISilVgmgJK6WURVrCSillkZawUkpZpCWslFIWaQkrpZRFWsJKKWWRlrBSSlmkJayUUhZpCSullEVawkopZZGWsFJKWaQlrJRSFhWqhEWko4j8KSKrRaR7tEOt4laGsox+pDOUZazi1mjfpVJKWXHcEhYRL/ABcCXQGLhdRBpHK9AqbmUq75PBaYCHDE5jKu87LmK/MYzPyeTDzHR3gjpkjGHigUwe2LyNfaGQ7TgA/L4vg06//cHCvRm2owCw7UAWD34/i4ELV9mOAkCWz8+bP0znkS/G244CQDgcZuyvc/jb82+xPzPLdhwA5i5ezi0PPcXSlWtsRwFg05at3PtYN4aP/dp2lGMqzDzh1sBqY8xaABEZAVwPLI9GoDn0InjYLOIgKcyhF/UZVeT1ghi+zcnky5xM/MZggH+llClWtkQ5Yh5zkRljmBvw8djaPewMBgliWO8P0CCx6GsniuBxIdOyHB+dpy7lt937CYQNbSuWoUHp5CKvE+8R4j3Oz3Bty/bT9/tZfP77WgKhMGtqZ3Bbk6LPF/aKkBjnfHh+ViDIR/P+5L8zvsQfDBHv9fLfv3Uo1lopic6H1YfDYb6a/wfdnn6XXfsO4AsE2LAjjdNPrVLktZIS4vG48G82d8Uaund7nbmLlxMIBrns/HOpW7N6kddJiI8jLs75mPNN23bQ691+jBj/M/5AgH3p6Vzfsej/Zl6vh8TE6A6rP+6W9yJyC9DRGHNv5Os7gXONMQ8ddr3OQGeAVKTFHRRvkHo/0jn6AXqYLpQt8npL8TMDf7GyHK5v6QrUj3M2qX9TKMjD+3e7kufxShV4oFIFx+ucvXINmeHcJygPEC7mOnVLJbHkypaO89wwbSkTd+4jXPBDs1AOPN2JJIdF/PrMpTw9caHzMMCi3g/RpEbRyzK/eeu20OaFj13J89ZDd9H15o6O10m5/C4CoRDH65PjadW0MbNGD3ac56Lb/83MBYsJO8wD4Nu8ivh45zt0tOpwLfMWLT7iqKkwJXwrcMVhJdzaGNP1WLepLF5T3J01hrIsciriUKls5A7OLPJ6IQyJyR6G5mSSbcIYYGR5Z98EThhjWBIMMCKQyXp/gKAxjKtdk4ZJ9raGWePz84Evg5+27yNowvRtVpcu9U61lme3L8A7aQf4cO6fBMJhLq1TjfGdLrWWxx8KMXjRGp6buoQsn5+EOC+7P+ppLY8xhglLVvPU2Ils2J6GPxhkyeDXqFfd3u4oS9dtosfn3zBp1jyCwRAfvfQ099x89J0kToSdu/fw8keD+PTLr/EHAtx0dUdG9v/AWh44dgkX5rh/M1Az39c1gK1uBTtca3oxlfcPOSURRxat6VWs9bwIlyWlcHFiMlN82WwM2z3/KiI0jU+gU/UKzMrKZkz6AU6Nj8ouU4V2emICw1s05s/9Wbzx52bOr1z0VxxuqpgYzyuXteDJ85rw2sylNK1S3mqeBK+Xzi3O4J6bOjJ46gLWpe21mkdE6Ni0PldccwUT5i5m5MSZVClv99+sSZ2afPPJWyxduYY3BnxB27PPsprnlIoVeLvnE/R46kn+++4HXHBua6t5ClKYI+E4YCVwKbAFmAt0MsYsO9ZtnBwJQ+6bc3PoRQY1SGUzrYt5PvigjuV1j7nj0T3mjk/3mCuY7jFXsGIfCRtjgiLyEDAB8AIDCypgN9RnlKPSVUqpk0WhXgcbY74Hvo9yFqWUKnH0J+aUUsoiLWGllLJIS1gppSzSElZKKYu0hJVSyiItYaWUskhLWCmlLNISVkopi7SElVLKIi1hpZSySEtYKaUs0hJWSimLojLItqzXQ8cyxR8fGTaGj7MOcGViMnUc7mQB7oyN7L97L1Xi4riurDsjKJ2Ojpy4Yy9Tdqbz0lm1XcnjdHTkxvRMnp20kH7XtnW8kwU4HxsZCofpPPArnriyHY2ruzTO0OHoyD5fjKNxrerceKE7s22djo4cP3kacxcv54VHuriTx+HYyFVr19Hnrffp/+YrruxkASCp5Yp9W7/fz78ffJjnnu7G6XVdGK3qOfr3RUweCc8J+PjFn82A7AO2owCwPRDkrbQ9PL9jF9nh4m7+456wMTw4fzXvrNzMohjZlLPXlEUMX7qO/vNX2o4CwMhZS/hixiK6jZxgOwoAa7fuoPfnY7nvzQH4A0HbcQgGg3Tp8TKv9BvMijXrbccB4Jk+fRkyaiyffznWdhQABg8ZxpDhI3nm+Reiej8xV8JhY/gsOwMDrAoGWBUM2I7Eu2l7MMYQNIYhe+3v1jxucxp7fAHCBp5dst52HDbsy2DU8g0Y4MWpi8kJ2t29JBQO033UBMLGMPmPtSzdvMNqHoBnB3yJMYYcn59B30+2HYchX/9ARlYWxsAzb9jd9gdg5Zq1fP/LZMLG0OPlvgQCdr/v/X4/PV94ibAxfPP9j6xeszZq9xVzJTwn4GNXZAsiP/CZ5aPh7YEgY9L3EwCyjeGD3XusHg2HjaHHkvVkhnL3y/t11z7rR8Mv/Po7gVDu30lWIGj9aHjkrCWk7c/dAt4XDPH0lz9ZzbN26w5GTZlFMBQmM8fHcwNHWT0aDgaD9HjjQzKysgmFQnw7aZr1o+EeL7+Gz+8DYF/6futHw4OHDCN9f273+P1+evR6MWr3FXMlnCJCXW/uqeokhNO8dvdfixc4Ozkp7+uzk5Ks/6W1LJ+a9/vm5VNJ9tpNVLd8aeqWzz1X3qxKeSql2Nu0FKBcShLNauWec6+cmkKjUytbzZMQH0frRv87f9u6UT3kiE1uTqzWzf63aW7rpo1JSkywmAbq161D9aq5/2ZnN2lMhXJ298yrUL48ZzfN3SevRvVTqX969LaSOu4ec8VRLy7evF6moqM1bty7gwdSSnN5YvH3qjvIjTfm6q1YTY34OKacXtvxWuD8jbl/zl7B6E1pZN7SzpU8Tt+YW7ZzH2f3+5bAs3e6kseN/dzi7n6WsQ934rrmjVxIhOM35uLad6Jlw7rM+qi3K3GcvjHX4e6HmDhzDqGVc9zJ4/CNuSkzfuOSm24nvGO9K3nA2RtzAFKqHNN/+ZHz27ZxnKVlu4uZt2DhEU+/tg/qlFKqRNMSVkopi7SElVLKIi1hpZSySEtYKaUs0hJWSimLtISVUsoiLWGllLJIS1gppSzSElZKKYuOW8IiMlBEdorI0hMRSCmlSpLCHAkPBjpGOYdSSpVIxy1hY8xUYM8JyHKECseYRG9Dk6REWiUn246R58wypSjtwg4WbimVYHfa3bFULO18AJRb6teoyrmNnA3dcVPzMxtQvYpLu464oFzZMrYjHFXZMtHNVagpaiJSGxhvjGlSwHU6A50BKns8LT4p62x8YKYJU0rcOWXtxhS1rHCYBBHiXJpB6HSKmjGGA8EQZeLdKT+nU9QA0nP8lE1yZySiG1PU0rNyKJuSdPwrFpbDKWoHsrJJSUzE69LoUadT1MLhMJlZ2ZROLf5WZIfkcThFDSB9/35XS8/pFLX09HTKlnVnrGbUp6gZYz4xxrQ0xrQs40J5ulXAbknxeFwrYDeIiGsF7Ba3CtgtrhawC0qnJLtWwG7weDyuFbBbon3UWVRuFXBBYucRoZRSJZCWsFJKWVSYj6gNB34DGojIZhH5V/RjKaVUyXDck4rGmNtPRBCllCqJ9HSEUkpZpCWslFIWaQkrpZRFWsJKKWWRlrBSSlmkJayUUhZpCSullEVawkopZZGWsFJKWaQlrJRSFmkJK6WURTFbwtN92ewNh2zHyDM7M4s/cny2Y+RZfSCbCdusbHhyVAd8AT5btJrCbBJwogz/7XfSDmTajpFn0vylLFu32XYMFWOiMhU8Od7raDeLpTk5vLF+B7eULc0rVao4zuN0F4vMYIgu4+dQKSGOJVe2xOPCcHenO1k8MPhHZm9JY+3DN1HNhS18nO5k8e7Xk+n17W9UrFuHa89p5DiP010spi1ewZ39RvPQTVfwdtd/Os+Ds50s9u0/wPU976FOjeos/m444sYGARn7nK/hpmTnO9i4zfhjaLC/CR/14pg8En5l524E+HZ/BlsDAdtx6Ld6KyFj2OUP8M2W3bbjMGvzLhZu34MHofe0xbbjcCDbx+vfTwfgyRETYuJouNvHwwAYMH4SO/emW04Dbw4cijGwYes2JkybZTuOiiExV8JLc3JYmJ2DAULG8G6a3ZfcmcEQfVdsJjsUJjMYpsfidYQtl0z3X+aTHQjhD4f5/Pe1bDuQZTXPOz/NJBR5lt+6bz/jF62wmmfa4hUsWbsRgLAxvDzkK6t59u0/wNuDhpHj85GZlc2Tr7wTE09UKjbEXAmvyPERijxAg8CfPr/VPFuyffjD/3sZsScQJD1g71x1MBxm3b4MDn4Lh8JhVu05YC0PwIL1W8n2BwHwB0PMX7fVap7fV28gEMz9N/IFgvy+eoPVPBu2bCMQDOZ9vX1XGlnZORYTqVhSqN2Wi+qs5CTzVe2ajtaot2I1fapW5u/lnG+05/ScMECp0dOplZLI8qtaOV4LnJ8T/sfYaYxctp7As3e6ksfpOeFlW3bQrMf7BAe/5Eoep+eEAeLad2LsS49zXbuWLgRyvrux94zWtDqrMbPGDHYlT6ydE5bKzr7noyK1vO0EeVq178C8hb9Hb7dlpZRSRaclrJRSFmkJK6WURVrCSillkZawUkpZpCWslFIWaQkrpZRFWsJKKWWRlrBSSlmkJayUUhYdt4RFpKaITBaRP0RkmYg8ciKCKaVOTn+sWkM4fPSxjTZs2LiJjIzYmSt9uMIcCQeBJ4wxjYA2wIMi0ji6sZRSJ5uZ8xbS7oZOnHnJNQwd963tOKzbuJk7uzzI6eecy/2PP2U7zjEdd6i7MWYbsC3y+wMi8gdQHVge5WzEuzH42iVJHg/VkhNsx8hTJjHedoRDCLHzb5WfxxtbZ9wqVYidgTI7du9l7eZttG3m/Jjqon89zoIVq8nKziEuzss/H+3OPx/tXuR1Xn+2G493vttxnu4vv867A78gGAoTDocZOmoMQ0eNKfI6N15zFSMHfUJcXFT2vwCKuLOGiNQGzgFmH+XPOgOdAU51IfCoWjU4KynR8TpumX35OZSJ99qOkafv5S3o0sLZ5DM3NTq1Mr/2uNd2jENMe78XrRs6m3zmpqXfj6RKpQq2Y+S5/+V3+HryTEILfnK8VpUK5Qkbg9frzXtCLs4TTuWK7jxJVapQHhEPcV4hGBkjWqli0f/uq1WtgscT3SfyQo+yFJFU4FegjzFmbEHXdWOUpZvcGGXpNqejLN3mdJSl61wYZek2p6MsXedwlGWH+7sxcfZCV0oY4M/0AD1efYuvJkxk2Puv8/frrnJl3eLavXcvr/b/gnf7DeCOW29iwPtvW81zrFGWhTpkFZF4YAww9HgFrJQqmRrWq8uY/u+xY1cap1SqaDsOFcuXp++Lz9P9sYcpnZpqO84xHbeEJXdHwgHAH8aYN6MfSSl1MqtSuZLtCIeoUD52zsMfTWFOdpwP3AlcIiKLIr/svs5QSqm/iMJ8OmI6xOhb30opdZKLrc/vKKVUCaMlrJRSFmkJK6WURVrCSillkZawUkpZpCWslFIWaQkrpZRFWsJKlUA5Pj9+fwCAzOxsy2lKNi1hpUqYHJ+fKpfeyrSFSwEoc/71zPx9meVUJZeWsFIlTFJiAs3OOB2JzOsulZzEmXVr2w1VgmkJK1UC9X303yQnJpCcmMATd95C2dKlbEcqsWK2hLcGAvjChZt1fCLs8gXY5w/ajpEnwx9g24Es2zHyhMJh1u7cYzvGITZs34U/EDv/ZrGkTdNGnNOwPiLCo3fcbDtOiRaVPTvik+MdDVLfkuXjkh/m0vPMWjzZ0PlweKcD1MPGcOF746hdLpWJd3VwnAecD1F/uP8Yvlmwgk1vP0lKogvbLjkcoj7y5+nc9fKHzP3kZc6pX9txHKcD1Nds3EzDTo/yZo/H6HrXbY7zAI6HqLvNpG11dPvPn7ibXekHKONLx/jSnQdKTHa+hsskLoa2ATvG5qcxeSTcZ/lGAF5fsZmMYMhyGhizfAO7s3zM3ZLG3K1ptuOwIW0fo+YsxRcM8vGkObbjEAqF6f7JcACe7jfMcppcz771EYjwwrv9yfH5bMeJSbWqVKLlGbG1w0tJFHMlvDXbx8iNuwgaCBnDR6udPds7FTaG7hMXkBkIkhMM0f2XBVbzAPQaN5FQOExOIEjvb6aQ5fNbzTNy0kz2Z+Z+zGnG0pUsXLXeap41Gzfz9S+/Eg6H8QcC9B85zmoepQoScyU8bnMawci+d4Gw4Yt1O6zmWbxjLxvTMwEwwLQNO9iTbe/IKhgKMXbeMoKh3Jc2GTl+Jv+xzloegP7fTco79xoMhfhiwlSreUaM/4lgKPcVlD8QZMCob6zmUaogMVfC99SpyuT2TQG4v141vruoidU8zaqUZ/a9uRuJeEVY2OVaKiTb2wU6zutl3gsP0qJOdQBmPteFjk3rW8sDMOzZrgzo1gWAae+/wHN3232j55F/3saMkQMAeK7rvYzv/5bVPEoVJOZKOCXOS/MKpQFoWCaFmilJVvOICM2r5W5aWLNsCmeeUs5qHoD6VSty+im523e3rFMdb5S35D6eahXL0/T0Wrl5GtSlXKrdjzullkqh5VmNATiz/unUqFrFah6lChJzJayUUiWJlrBSSlmkJayUUhZpCSullEVawkopZZGWsFJKWaQlrJRSFmkJK6WURVrCSill0XFLWESSRGSOiPwuIstE5IUTEUwpdXIKH2Nkoy2xludwhTkS9gGXGGOaAWcDHUWkTXRjKaVONvszMnnxw0FUbHsVE6bPth2HYDDIF6PGcVrzdnTv3dd2nGM67lB3Y4wBMiJfxkd+xc6WFydQgsdrO0KeyPZg6iSya+8+yqaWIiE+NgaNB4JB9h7I4pTyZRyv1XvIV7w+egKhcJhgMETX3m9xSZsWRV7njms7cEGLZo7zjPtlKl17v8mBrGwyMrPo+8En7E3fX+R12rQ4m3tuu8VxnoIUamcNEfEC84F6wAfGmCOe5kSkM9AZoGaK8yljrzatw1XVKjhexy1j/3Yxp6TaHSaU33M3XEKHJnanp+VXp1plXvy/W23HOETfbg9zQcuzbcfI0+Tme7nqgnMZ9MKTtqMA8PhHw/jsp+ns//YTx2t9+sNUgqEQOT4/Ho+HNZu2sGbTliKvk5QQ70oJf/nDRNL2phMI/m97q/5DRhR5nZ9+ncY/br6e+Cg+cYoxhT+oFZFywDigqzFm6bGu17xCaTP90th58Dvd3iganG5v5DqH2xu5zen2RlHhcHsjb/MOtDqzAbO+eM+VOE63N7qi22tMXLiM4E+DHWcJBIMMmbeKHu98QtredD558SnuufFqx+sWlzGGSbPn8+Qb/ViyYiU3X92RkZ+8ay0PQKsrbmDe70uOeA1bpD3mjDH7RGQK0BE4ZgkrpUqW+Lg4/u/ma7jzuo6M/3Uml7VtaTWPiHBpm5Ys+OUaJs+YRZ3TaljNU5DjlrCIVAYCkQJOBi4DXo16MqXUSSc+Po4bL7vQdow8IsIl7drajlGgwhwJVwM+i5wX9gBfGmPGRzeWUkqVDIX5dMRi4JwTkEUppUoc/Yk5pZSySEtYKaUs0hJWSimLtISVUsoiLWGllLJIS1gppSzSElZKKYu0hJVSyiItYaWibM2mrTS84R4A5i77kzZ3diXH57eWxx8I0u7R3kxcuAyAhnc/xYqNzoYBqeLTElYqypISE1i3ZXve11t2phHntTebOs7rYWva3ryv1+9IIykhNmYcl0RawkpFWfVTKvHP6zqQEB9HanISL3f9F3Fx9krY4/Hw6r//TmpSInFeD3+/uA21q1a2lqek0xJW6gR4vsudAJROLcXtHdtbTgM3X9CSCmVSAXjhnzdaTlOyFWmecGFJUoKjQeqZ/gBNP/qWVy9vwS2NaznP48IA9ate/4y6p1Tg/buudbwW4HiIet9h3zBy8m/M7dcHj8f5c6nTIerT5y3iloe6seDrIZxaxYWjKocD1KPByRD1Uz3w1v2dqFe9Kt59O9zZH2z75mLf1AMMuvcGlm3aQW18jtY6yCTEzs4zeWIpUzh01Itj8kj4g7l/sjUji6d+nk8oBnZKnbV6E9NWrmfwtAVs2p1uOw77M7N4ecg4/ty4la+nz7MdB4AnX32H3fvS6f3hANtRYlaXay7h0nMa246R56KGdXjgct2z17aYK+FMf4BXpi8lGDbszfEx5o+NtiPRbeSP5PiDhMOGF7+aZDsO7475kVDYkOMP8NTHw6xv6T1t7kKWrlxDOBzms7HfsW1nmtU8Sp1MYq6EP1u0hqxAEI9AdiBErymLrOZZsmk7M1ZtxCNCyIT5fMZC9mRkWcsTDIV4fcR4/IEgHo+wYfsufpq3xFoegBfe64/P58cjQiAY5M0BQ6zmUepkEnMlfG2Dmvz3suaEDVxQ6xTe6NDKap6G1SrzZqerCBlDKGz44K7rKF8q2VqeOK+Xj5+4l5SkRMJhw6v3daJtY7u7Lj/X9V7uv+MWwsbw6pMPcc+t11nNo9TJJOZKuGbZUjzWJve82W1N6nBl/epW88THeXm4Q+4eVbUrlePei1sicsSGqSfUbZeex1VtcnezfuxvV1M2NcVqngtbNafzbbnvsD/2f3fQuF5dq3mUOpnEXAkrpVRJoiWslFIWaQkrpZRFWsJKKWWRlrBSSlmkJayUUhZpCSullEVawkopZZGWsFJKWVToUZYi4gXmAVuMMddEL5JSKtqGzWxMz1Ht2bS7LDUrptP71sl0Om+57VglUlGOhB8B/ohWEKVixbI16+n4wNNMmrMQY1yZ/OtIRnYO1/R4g0E/TiUQDDpeb9jMxtw38Go27i6HQdi4uxz3DbyaYTOdj9n8ddFyrvzPyyxatd7xWm5YvWETtz/yDINHf2s7yjEV6khYRGoAVwN9gMejmkgpl2RkZbN73/4i3+6n3+bx67zfmfn7Ms44rTp9H+vMJa3PcZwnGAqxJd/eboW1dfc+fp6/jOlLVtJz0Gh633ML/7jsPIq7K1zPUe3J8iccclmWP4Geo9oX+2h46op1PPXyIJav30wgGOTHOYsoX7pUkdepUCaV0inOB2St3rydnp+OYPxvC/D5A+zdv5/2bVsWeZ1SyclUqlDOcZ6CFPZ0xNvAU0DpY11BRDoDnQFOK1v0v/zDXVirCi2rVXS8jlu6XX0BtSqVtx0jz22XnkdifFQ2RimWKhUrckad02zHOMTt3fvw/fQ5xb69Pxhk0cq1XHF/dwLzJzjOM3TiTP71evGH3mfk+MjI8fHvNwfSoGY1zqtYvLLatLtskS4vjCv7foY/FMIYgwA9+4+gZ/8RRV7nwmaNmPTO88XOcdC/+37MjCV/Eo68kvl5+mzqXly86X45y38jPorfa3K8l1sicg1wlTHmARG5GPjP8c4Jtzi1opl979XupXTIje2NXOdweyO3Od3eyHUWtzca/uNk7nnuNbxeD1ee35o+D91Dg9o1HW1v5MTmXTM/Qe8AAAuoSURBVHuod9d/SIyPo371qrzW5XYubtYQ2bGlWOvVfexBNu4+8ujutIr7WPvWB8Vac/WO3Tz7/Sy+nTGPUCjMR//5N3dfeXGx1nLD3gMZvDlyPO+M+RG/P8CNHdoz4t3/WssD0PrGu5i3ZPkRIxgLU+/nA9eJyFVAElBGRIYYY/7hdkilYkHbpo34140defj2G2lQu6btOFQsk0rnq9tzY7uWueXrcJRq71snc9/Aqw85JZGS4Kf3rZOLvWa9KhUZ/vwjrN68nbdHfUe7sxo6yuhU+dKpvHTvbTzxaFfeHDiUNs2aWM1TkOMeCR9yZT0Sdo8eCRfsL7bRZ1Q42JwzKp+OOC3GHkOAVI+d7zMnR8JKqb+YTuct14+kxYgilbAxZgowJSpJlFKqBNKfmFNKKYu0hJVSyiItYaWUskhLWCmlLNISVkopi7SElVLKIi1hpZSySEtYKaUs0hJWSimLtISVUsoiLWGllLJIS1gppSyKyRIOG8Nd46Yzd2ua7Sh5XvpqMoOmzrcdI893vy3k0fc+sx0jz9qNW7j5wSfJys6xHQWAYDDEbd16s3jlWttR8vQcOJoRk2bZjpFn7NylPDX8B9sx8vyxYQt/f/4t/AHn++i5wefzc+tD3fhz7fqo3k9URllKYpKjGb5j5yxh5LL1rM8JMq1nZ+eBHM7u3bB9F698N42EeC+33nIdqclJjiM5md8bCoW4/51H2Jm2h390up1WTZ1v0ChlT3F0+2c/eIlvfpnKR2N+4D8PdnGch+Rj7qRVKEO+HMuYidPZmxPkp2HF31LoEInF3/ts5fqNvDlmAqWSk7jxb7eQlJjoOI5JKP7j0B8I8sDnr5KemcVdt95Ak7rOh9c7nd37zIsf8t20OQyauYT7Ot3iOA+AlKlc7Nt+OuAzvvp5CmFPHGMHf+I8jPfodRtzR8LhcJhuIycQNobfN25n5qqNtiPRa9BowuEwoVCY98b8aDsOI777mf0HMgiFw3Tr+67tOKxdv5Gvvv+JsDH0eet9srKyreYJBoP0eOUtwuEwM+YuYOFS+3Nze7z9CaFwGH8gQP9R39iOw2c/TiHb5ycUDvPMJ8Nsx2H5qrX8MmM24bDhubf74fcHrObx+Xz06pv7GPpx0q+sWLU6avcVcyU8Zt4ytu07AECWP0C3kXZLb8P2XQz7ZQaBUIgsn59Xhn5NhsWX3KFQiO6vvUdGVjbGGKbNW8TcxXZLpucrr+EP5H7TZOXk8OGgz63m+WLsN+zel7szR3ZODt1efsNqnpXrN/LVxKmEQiEys3Po9cFAcnw+a3n8gSDPfjqSzBwf4bDhx9mLWLp2k7U8AM+8/gE5fj8A+zMyGDj6a6t5Ph06ggOZGQD4A36e6dM3avcVcyVcrVxpzj+jFgDlSyXTrn4tq3lSk5O4pPmZeV9f3qopCXH2NiTxeDxcfl7rvK8vat2cCuXKWMsD0PysJjRp2ACAdq1bUqeW3X3ZalStQrtWLQCoWa0qbVs0s5qnTKlStG/dIu/ry89rRZzXay2P1+PhspZn5X198TlnUi41xVoegFZNG3P6adUBaNfibE6rVtVqntOqn0q71q0AqFenDq3Ojt5jqEh7zBVWyzrVzexe9ztaI+7uZ+l3z/X866KWzgO5sJ9bXPtO1K5amdXD33GeB+d7unV6rCcjv/uJ0Mrib+l+SB6H54SXrVjJWRd1ILxjvSt58Ds/peGp2Yhxn77P9Vdc6kIgMPudvVHsPfMCWjVpxKyRLpxfBMyu4u22fNAVT/Rh4vwlBKcUfWv6o3F6TnjK7Plc+o/7CK2a60oecHZOGMBTpTbTvh3N+a2d91CrDtcyb9HiI/aYi7kjYaWUKkm0hJVSyiItYaWUskhLWCmlLNISVkopi7SElVLKIi1hpZSySEtYKaUs0hJWSimLtISVUsqiQg1BEJH1wAEgBASNMS78LLFSSqmiTKJpb4yJnSnrSin1FxDTpyNqVixnO0Ke85s04LIWTWzHyNP8zAZUrVTRdow8pVNL2Y5wVFUqx87fUdMzTqddi6a2Y+Rpc2Z96lW3O60sv4rlytqOcFTly0Y3V6GmqInIOmAvYIB+xpgjxkCJSGegM8BpFcu2WPvGfxwFy/T5KZWY4GiNPC5MUcvx+4n3xuH1uvO85XSKmjGG7BwfKS7s8gHOp6gBZGVlk5JS/N0nDuHCFLWs7GxSkl3Kg/Mpatk5PhIT4vF43HkMOZ2iZowh2+cnJcn5Lh/gfIoaQGZWNqXcegzhfIqam4/pY01RK+zpiPONMVtF5BTgZxFZYYyZmv8KkWL+BHJHWToN7FoBuyQpIbbyiIhrBewW1wrYJW4WsBuSXSo7t4iIawXsFjcL2A0n4jFdqKdkY8zWyH93AuOA1gXfQimlVGEct4RFpJSIlD74e6ADsDTawZRSqiQozOmIKsA4ETl4/WHGGPu7XSql1F/AcUvYGLMWsLtJl1JK/UXF9EfUlFLqr05LWCmlLNISVkopi7SElVLKIi1hpZSySEtYKaUs0hJWSimLtISVUsoiLWGllLJIS1gppSzSElZKKYu0hJVSyqJC7axR5EVFdgEbXFiqEhBL+9ppnoLFWh6IvUyap2Cxlgfcy1TLGHPEVh9RKWG3iMi8WNrZWfMULNbyQOxl0jwFi7U8EP1MejpCKaUs0hJWSimLYr2Ej9jV2TLNU7BYywOxl0nzFCzW8kCUM8X0OWGllPqri/UjYaWU+kvTElZKKYtisoRFpKOI/Ckiq0WkewzkGSgiO0Vkqe0sACJSU0Qmi8gfIrJMRB6xnCdJROaIyO+RPC/YzHOQiHhFZKGIjLedBUBE1ovIEhFZJCLzYiBPOREZLSIrIo+lthazNIj8vRz8tV9EHrWVJ5LpscjjeamIDBeRpKjcT6ydExYRL7ASuBzYDMwFbjfGLLeY6UIgA/jcGNPEVo58eaoB1YwxC0SkNDAfuMHW35GICFDKGJMhIvHAdOARY8wsG3ny5XocaAmUMcZcYzNLJM96oKUxJiZ+GEFEPgOmGWM+FZEEIMUYsy8GcnmBLcC5xhg3fuirOBmqk/s4bmyMyRaRL4HvjTGD3b6vWDwSbg2sNsasNcb4gRHA9TYDGWOmAntsZsjPGLPNGLMg8vsDwB9AdYt5jDEmI/JlfOSX1Wd3EakBXA18ajNHrBKRMsCFwAAAY4w/Fgo44lJgja0CzicOSBaROCAF2BqNO4nFEq4ObMr39WYsFkysE5HawDnAbMs5vCKyCNgJ/GyMsZoHeBt4CghbzpGfAX4Skfki0tlylrrALmBQ5JTNpyJSynKmg24DhtsMYIzZArwObAS2AenGmJ+icV+xWMJylMti65xJjBCRVGAM8KgxZr/NLMaYkDHmbKAG0FpErJ22EZFrgJ3GmPm2MhzD+caY5sCVwIOR01y2xAHNgY+MMecAmUAsvP+SAFwHjLKcozy5r8DrAKcCpUTkH9G4r1gs4c1AzXxf1yBKLwNOZpFzr2OAocaYsbbzHBR5STsF6GgxxvnAdZFzsCOAS0RkiMU8ABhjtkb+uxMYR+6pN1s2A5vzvWIZTW4p23YlsMAYs8NyjsuAdcaYXcaYADAWOC8adxSLJTwXqC8idSLPircB31jOFFMib4QNAP4wxrwZA3kqi0i5yO+TyX0Ar7CVxxjztDGmhjGmNrmPn0nGmKgcxRSWiJSKvIlK5GV/B8Dap22MMduBTSLSIHLRpYC1N7/zuR3LpyIiNgJtRCQl8v12KbnvvbguLhqLOmGMCYrIQ8AEwAsMNMYss5lJRIYDFwOVRGQz8LwxZoDFSOcDdwJLIudhAZ4xxnxvKU814LPIu9oe4EtjTEx8LCyGVAHG5X4/EwcMM8b8aDcSXYGhkYOdtcA9NsOISAq5n4rqYjMHgDFmtoiMBhYAQWAhUfrx5Zj7iJpSSpUksXg6QimlSgwtYaWUskhLWCmlLNISVkopi7SElVLKIi1hpZSySEtYKaUs+n8uYFwtTXqxigAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "grid.plot(policy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-18T09:05:00.612483Z",
     "start_time": "2020-04-18T09:05:00.027478Z"
    }
   },
   "source": [
    "We see that the value function of the new policy is much more informative than the one of the random policy. Even for states far from the goal there it is a clear which direction to follow (basically doing gradient descent on the value function). Also, the policy has been optimized to point to states with higher value function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Windy Grid World\n",
    "A variation of the Grid World environment adds a \"wind\" in the state space such that every action is affected by an additional vertical move"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-20T11:55:44.789167Z",
     "start_time": "2020-04-20T11:55:44.292170Z"
    }
   },
   "outputs": [],
   "source": [
    "windy_grid = GridWorld(7, 10, goal=[State([3, 7])], wind=np.array([0, 0, 0, -1, -1, -1, -2, -2, -1, 0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-20T11:55:45.197166Z",
     "start_time": "2020-04-20T11:55:44.793182Z"
    }
   },
   "outputs": [],
   "source": [
    "policy = GenericPolicy(windy_grid, gamma=0.9)\n",
    "\n",
    "evaluator = PolicyEvaluation(windy_grid, policy)\n",
    "improver = PolicyImprovement(windy_grid, policy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-20T11:56:04.090169Z",
     "start_time": "2020-04-20T11:55:45.203164Z"
    }
   },
   "outputs": [],
   "source": [
    "for _ in range(30):\n",
    "    evaluator.evaluate_once()\n",
    "    improver.improve_once()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-20T11:56:06.179166Z",
     "start_time": "2020-04-20T11:56:04.095171Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVIAAAEICAYAAAANwHx+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3dd5hTVf7H8fc3fRq9SVVsgAVBRLBgWX+CLta1oIgNxYKIFRELgiJiWcW1rN11cS1gQ8SKIIIUQURAinSQ3qbPZJKc3x8JiMgwuffO3Awz39fz+DiEnPs5YSafnNxMTsQYg1JKKfs8qZ6AUkrt77RIlVLKIS1SpZRySItUKaUc0iJVSimHtEiVUsohLVKVMiLybxF5YB9/b0TkEDfnlMi9WkSm7OPvJ4nIdW7OSVVuWqSqXIhIDxGZISL5IrIp8fXNIiKljTHG3GiMedhG1mUi8usel31dymUDrR5fKau0SJVjInInMBJ4AmgENARuBE4EAqWM8TqI/A5oLSL1E8fyAW2B9D0u6wxMtnLgxDilLNEiVY6ISE1gKHCzMWaMMSbXxM0xxvQ0xhQnrvemiLwoIuNFJB84LXHZI7sd624RWS8i60Tk2tIyjTHrgOVAl8RF7YEFxAt298s8wCwRqSkib4nIZhFZJSL3i4gnkXm1iEwVkadFZBvw0F5u4/+JyCIRyRaR54BSV9mqetIiVU51BoLAJ0lc93JgGJAF/OkcpIh0A+4C/g84FDijjGNN5o/S7AJ8nzjm7pdNN8aEgX8BNYGWwCnAlcA1ux3reOLF3CAxv93nVQ/4ALgfqAcsI77SVmoXLVLlVD1gizEmsvMCEflBRHaISKGIdNntup8YY6YaY2LGmKI9jnMJ8IYxZr4xJp+9rAz3sPvq82TiRfr9Hpd9lziFcClwb2K1vBJ4Cui127HWGWP+ZYyJGGMK98g5G/g1sdouAZ4BNpQxN1XNaJEqp7YC9XY/t2iMOcEYUyvxd7v/jK3Zx3Ea7/H3q8rInQwcLSK1gU7ANGPMIuCAxGUnJa5Tj/h52t2PtwpoYmdeJr7Lz76ur6ohLVLl1DSgGDgvievua6ux9UCz3f7cfJ8HMmY5sA7oA6w2xuTtNp8+QCYwHdgClAAt9jj273bmlfgthGalX11VR1qkyhFjzA5gCPCCiFwkIpki4hGRY4AMC4d6H7haRNqISDowOIkx3wN3JP6/05TEZbOMMYXGmGji2MNEJEtEWiT+flSS8/oMOEJELkysum8l/psJSu2iRaocM8Y8TrycBgCbgI3AS8A9wA9JHuNz4ucfvwWWJv5flu+Iv0C0+wtX3ycu2/3XnvoB+cRfUJoC/A94Pcl5bQEuBh4jfqriUGBqMmNV9SG6sbNSSjmjK1KllHJIi1QppRzSIlVKKYe0SJVSyqEK2aChXt265sDmTSvi0GVL5WtnJpbC7BTecBNNXXY0hdkA0UjZ16mK2ZHUZZtISUpyV23PZUt+0V73WaiQIj2weVN+nPhVRRy6bLEU/nCVhFOYvec7Lt1jCvPKvlJFZeduT1k2ADlbU5e9I4XZWzalLNps3piS3OOf/7DUv9On9kop5ZAWqVJKOaRFqpRSDmmRKqWUQ1qkSinlkBapUko5pEWqlFIOaZEqpZRDWqRKKeWQFqlSSjmkRaqUUg4lVaQi0k1EFovIUhEZWNGTsiIcTt3728PhMKn6hIGSkpKUZUciEWKx1GzQEo1GiaRowwxjDCUlqcsOpygbSG12JMUb0yShzCJNfC7488BZQBvgMhFpU9ETS1bzI9vT6W9n8f0P013PPuKE02nX5Uy+njjZ9VLrfPaFtD7xDD79aoLr2V17XsfBJ57J6HFfuF6ol956Lwedei7//Xi864V64yPP0Kzb5bzy4XjCJe7uQHTfK+/S+MIbeXbMFxS5vHh4+pMJNLhyICM++Jr8omJXs/8zdS71+j/BQ59MIrsgdRvzlKXMz2wSkc7AQ8aYrok/3wtgjBle2pgO7doaO7s/rVy9mkPadbJ15wyFQnQ5oRNfjE72wyH/bMPGTbQ6vgs5udZ3MkpLC3F0m9ZM++wDW9nbd2Rz1CldWbfR+o466WlptGjamAXffmorO7+ggOP+fgmLli63ld2gXh0WjH+XUDBoeXw4XMKpPfswY+4Cy2Mz0kLUrJHF3HdfpHaNLMvjo9Eo3W99gK+nz7aeHQqRnhZk1v9eoElor7uq7ZMxhp4PP8f7E6dZHpseDBD0+5n6wlAOywpYHg/Q7+X3efHz78u+4h7SAn58Xg/fPtKfdjWsf78BBn88kWGfTSn7irsRIOSPb1T3Wf/LOblOyFb2u3OX0r1VCzKDflvjj3/+Q2av3Wx7G70mwJrd/rwWOH7PK4lIH+KfJ07zpvb2Iq1Xpy4XnnM2a35fl/SYGbN+IhgM4hGhU4f2tnIBatWswbndzuS35SuSz549h4DfDwY6H3es7eyszAzO/r/TmffrouSzf/oZn89HLBajs4PbnRYK0fWUk6iZlZl89pxf8Ho88ez2bfF6vLay/X4fZ5x4PPG7SpLZc+cjIkRjhuOOahP/97fB6/Vy2nFtyc7LT3rMzPmLECBmYrRrdQhpwQBgfWUqIpx01OGs3LA5+eyFS3d9feTBzchKDwH2ng10PLQFs5etKfuKO7OXrATi827VtBG1M9MBe0+3j2neiI4HNUk+e8XvAHg8Qsv6tWlQI8N2dq/3vmXUpadzadtDbI3fl2RWpBcDXY0x1yX+3AvoaIzpV9oYuytSO/7vvItpf8xRDOh/C3Xr1HF1P9Lzr7iW5k2acN+dt9KwQX1X9yPt1fd2goEAg+/qT7MmjV3dj7TvoKHsyMnh4bv707JFM1f3I73vqedZuGwlj97Zl1YHH+jqfqRP/mc03/44h+G39qbtYQfHL3RpP9JXx33LOxOmMrzPZXRsnSgCl/YjHTP1J0Z+Oolhvc6lyxGJbJf2I/1qwTIGfzyJoRecxhmtD0JEbO9H6h/0sqMi3deKtFI9tS8XurGz63Rj5xTRjZ0tqcgiTeZV+x+BQ0XkIBEJAD2AsbZmopRSVVCZ50iNMRERuQX4EvACrxtjrL86oJRSVVRSn9lkjBkPjK/guSil1H5J39mklFIOaZEqpZRDWqRKKeWQFqlSSjmkRaqUUg5pkSqlqrx5G7YBsGRLNjlF5f/GGS1SpVSV9vO6LbR/dgwAQyfMZuiEWeWeoUWqlKrSjmpUh6Y1MwBI83u5/JjDyj1Di1QpVaV5PR6Gdzsej0Cn5g1p36ReuWck9c4my8SDBOztGeiUKS5ISS4AYn1vynIMT2F06rIlpf/mYCSFa5FU3nZPCrM3bLA85OL6GXx5eFNua3ewrfEA7GMz74opUqWUqkS8HuGNM+3v21sWfWqvlFIOaZEqpZRDWqRKKeWQFqlSSjmkRaqUUg5pkSqllENapEop5ZAWqVJKOaRFqpRSDmmRKqWUQ1qkSinlUJlFKiKvi8gmEZnvxoSsisVimu0yYwzGmGqZXV2/56nNTs3324pkVqRvAt0qeB62HXzkMXQ77x/M/WWe69ntupzBqd0vZObsOa5nn3LepXTqdj7fT5/peva519xEu64X8PXkqa6X2lV3DabNmRfx6YTJrmff8eSLHHLOlYz+erLrxfLwf8bQ4uK+jPpqMpFI1NXsF8d/R5OrB/HKl1MIl0RczX53xnwa3fYkz349gyKXs62QZH4YReRAYJwx5shkDtqhfTsza8oky5P5fd06OnY5nXXrrW1z5fF4CAYCdD+7G++9+oLlXIAtW7fS+cxzWLZipaVxIkIoFOTkTsfzxbtv2srOyc3llPMuZe6ChZbHpqelcVTrw5n2yTu2sgsLi+h2xXV8P3O25bEZ6Wkc3KI500e/TjAYsDy+pCTCBTfdyeff/WAru0nD+kx78xlqZWVaHh+Lxeh132O8++Uky2Mz00LUrVWTqW8+Q6Og9e3kjDHc8vRrvDT2G8tjM9KC1EhP57t/PUTLdK/l8QD3/3csj33wlfXsUIC0QIBvHr6VIzPtbRz3xOdTuXfMBEtjBEgP+PH7vIzrfznH52+xlf35yo2c1rQeIZ+9f7fj3/2O2Rt37PUbXm7b6IlIH6APQPNmzWwdIz0tnZYHHUQ0mvyj/cZNm/B6vRigUcOGtnIBQsEQLVs0Jy8/30L2ZrxeDyZmOKCR/eyAP8BBzZuxYdPm5LM3b8Hj8RAzMQ5o2MB2tt/vo0XTxixZscpStogQi8U4oEF923uCer0emjc+gIb16iSfvSX+2TuxmKFRvbp4PfZO84sITRvWp2Hd2slnb90ezzbQoE4tfD4vYH1lKiI0rleHhrVrJp+9PTv+hYF6NbPw2ywDgEa1a9CwVlby2TtyE18JtTPTCfrt10bDGpk0rJGRfHZO/P4oAjXTgqQH/ZD8XfRPzh07g7e7HcslhzWxd4B9qFQrUjuOO+lU2h59FA/dN5CmTZq4urHz6edeRONGDRk6aAAtD2wBJcWuZZ9/5fV4PB4evW8ArQ49GMJFrmVf2X8Am7du57FBd9K2TStMUZ5r2f2HPskvi5Yw4p7+dGx7BORudy37oX//hy+mzuLx266ny7FHA2BytrmSPXL0eN764jseu6knZxx7VPzBa4e9lZlVoybOYMSHX/Nor/PoftyR8eytm1zJ/vTnxQx4/2sevuB0Ljy2NR6PYOb+bOtY/mfHMqrbsVxqs0j3tSLd74t0TyndId/FIv0LF4t0T24W6V+4WKR741aR7pVLRbpXLhXp3lTGItVff1JKKYeS+fWnd4BpwOEislZEelf8tJRSav9R5lljY8xlbkxEKaX2V/rUXimlHNIiVUoph7RIlVLKIS1SpZRySItUKaUc0iJVSlV5a3ILAViXV0RhBWz6okWqlKrSftmSTcs3vgZgwJQFDJ62qNwztEiVUlVa69pZNEwPApDu83L+IQeUe4YWqVKqSvN7PTzSuRVegbb1a3DCAcnvNpascttGb08mVTtq29zSTbOdZKfw8TiV2dWY2bQxZdnFa6xv1nJRZhofNKzNPYc1tTUewIRL31i6wopUKaUqC7/Hw5jOrSvs+PpwrpRSDmmRKqWUQ1qkSinlkBapUko5pEWqlFIOaZEqpZRDWqRKKeWQFqlSSjmkRaqUUg5pkSqllENapEop5VAyn2vfTEQmishCEVkgIv3dmJhSSu0vklmRRoA7jTGtgU5AXxFpU7HTSl7bTifR89rrWb5ipevZJ3Y9hwuvuIZFS35zPfvsHldx1qVXMXf+r65n97jpdk69qBcz5/zievZNDzxK5wuv5Psf57iefd+/XuOYS/rw9fTZGGNczf7nu59yxJV38OkP7me/+c00DrthMGOm/kTM5V3dPl6wgoNHvM2on5YQiaZoR7kkiNVvioh8AjxnjPm6tOt0aN/O/Dj5W8uT2bJlK13P/we/Llqc9Jji4mJ8Ph8+r5del/fgpX8+ZjkXIDs7h79fcgWz5yZfDsXFxXi9Xvx+P+ed1ZV3XhppK7ugoJALrurD5OkzLGSHERFCwSCnnHA84996yVZ2OBzm8r538tm331nKBkgPhehwzFF8+fq/CAT8lrOj0Si9Bw7h/c9K/VH6a3Y4kZ0W4ohDD+ar5x+lRmaG5WxjDP1HPMerH423kF0CQEZaiJZNG/PVv0dQ32v9YyuMMTz42vv8871xyWeXJLJDQRrXq8OXT91H86DlaACe/OgbBv/PSnZ8+7jMUJC6WRmMf6gvh+VsspX9ysxfuf3TH5LPjkQRID3go0YwwCdXn0WbtettZU/dksPxdbLweextOXnyxLn8tD1vr4MtbaMnIgcC7YC/3ONFpA/QB6B5s6aWJwkQiUbIzc2juLjY0jhjDIhQWFhoKxcgZmLk5edbzo7FYghQWFRkPzsWI68gf1dBJcsYg4izbGMgr6DAcjYAIhQVFdleIRljKCgs2lWOlqIRioqLnWUXFe8qR6uKw2GcLAwLiop2laMVIkK4pCSxMrT3EkdRuGRXOVoVjkaJOliVFpVEKLb5mUklsRhRB6vSrt/P562Oh3Fhk3q2j1GapFekIpIJfAcMM8Z8uK/r2l2R2tGq3XEc0boVwwY/QKvDD4OItSJ0osOpZ9Kgfj0eG3w/Rx/ZBkrcyz79gh7EYjEeHzyIju2PgbD9MrXqgt59Wb9xE4/fP4AunY7DFOW7lt37niHMnv8rT9x7O2eceDySt8O17AFPv8TYSdN44o4b6N6lEyKCyba3SbBVj436mFfGTWDEjT25sEtHPB4P7HAn+9Uvp/Lwe+MZ1us8enQ5Fp/Xi1k4z5XsMfOWcdvYqTz0fx24sv3hBHxeimfay8786AfeOO4wLm5qr0j3tSJNqkhFxA+MA740xvyzrOu7WaTxVdlut83FIv1LtotF+pdsF4t0z2w3i3Tnz+uu/NztrmXvzP/TbXepSPeW7VaR7i3brSLdW3ZlLNIyn9pL/Ba8BixMpkTdJin8iA3Nrl7Zqc7X7MormZMsJwK9gNNF5OfEf2dX8LyUUmq/UeaK1BgzBaj8DwlKKZUi+s4mpZRySItUKaUc0iJVSimHtEiVUsohLVKllHJIi1QpVeXlJN4Sm1cSJRIr/01ftEiVUlXagpx8moybCUC/n5cx5NfV5Z6hRaqUqtIOyUijdiD+K/PpXg9nNKxV7hlapEqpKi3o9XB/62Z4BQ7LTKNLvRrlnmFpGz1LUvb+WH0Tluv2g/dCV5gU3va+5w9MWfZz/f+WsuzslVstjzkfH++kpzGwQR1yVm2zlRstLn3rwYorUqWUqiSCHg8fH9a8wo6vT+2VUsohLVKllHJIi1QppRzSIlVKKYe0SJVSyiEtUqWUckiLVCmlHNIiVUoph7RIlVLKIS1SpZRySItUKaUcKrNIRSQkIjNFZK6ILBCRIW5MTClV/n7jYt5mAS+Rzdss4DcuTvWUqoRkVqTFwOnGmLbAMUA3EelUsdNK3slndKPfnXezYeNG17PPuugyrut3O2vW/u569iW9b6bnDbeyfGX5b1Jbluvvup8Le9/CoqXLXc++e/jTnH1NX+YuXOJ69vDX/sdpve9g5vxFrme/+NGXnHjTfXw/d6HtY/zGxUzmOfJoDnjIozmTea5Sl+nnKzbS/u2JjFu+AWPKf2f78iJWJici6cAU4CZjzIzSrtehfTvz4/cTLU8mJyeHy67uzQoL5bBoyRICgQBej4dbbuzDiMGDLOcCFBQUcOWN/fh1cfJ30EVLfsPv9+H1eLnq8kt5ccRQW9nhcJje/Qcw+5d5yWf/tgyv14vf5+PCv3dj1LMjbGVHo1H63jeUydN/TD576XI8Hg+BgJ9up5zMuyOH4ff7LWcbYxgw/BnGT/o++exlKxERQsEAJx/XntGPDSIzPc1W9tCX3uL9r75LPntF/OcyLRTkuDaHM+apwdTxlL612r48/d44Xhs3Ifns1esASA8FOeLApox55C4evehuS5lvsyBRon+WyWp6coSlY9ndRu/9Jb/z8PTkH4gWbc9DgHS/l8YZIUZ370j9BetsZc8vKKJNWhCPza0Pz1q0krkFRXsdnNQ2eiLiBWYDhwDP761ERaQP0AegebOmtiaanZPD1GkzyMnNtTSupKQEbzDIoiX2Vyl5+flMm/kj6zduspgdwZfmZ6GFAt5TQWEhM376maUrVloaF41GCQYCLFj8m+3s4nCYWXPnWV5dxmIxvOLh19+WEo3FsF6j8X+7nxYsZNGylZbGGWPwiIcly1dRErFXZLFYjLmLl+8qRys8Iixd8zvF4RIIWb9TGmOYv3z1rnK0QoCVGzZTWBy2PDaPvd8vS7u8Iizdkcei7XmWxxljWJ9fxI7iEurbzO66eBWvHtSYs2pl2TxC6ayuSGsBHwH9jDHzS7ue3RWpHc0ObcPBLQ/i8WFD6djhWCgpdiUXoHXHk6iZlcXjQx+ky4mdoaTItexOXc+jsKiIJ4fczxmnnIS4eLu7Xn4tq9au44kHBtD9jNMgXOhado9+9zBz7nxGDLyNf3T7G568Ha5l3/zoSMZO+oHht17HZd1Ox+fzYnKsbzJsx4OvvsvLY7/h4et7cFW3Uwn4ffQ9pbelY1SGFalVoxau4dZJvzCww6Hc3LYlmQEfmybbO7XSZM5iXjjwAM6rbW+H/H2tSC0VKYCIDAbyjTFPlnYdN4s0Eong8+22sHaxUP6a7V6RRqNRPB4PsvNpSjh12aa4wLXsWCwGgMeTOL2fY2+3czuMMfGVuNf7x2UuFSlAJBLF5/sj22qR7jxHGiF912U+CujCLRzKaEvHcnOH/Egshs/zx8s5lbFIk3nVvn5iJYqIpAFnAO6fbS/Fn4qsGmV7vd4/SrQaZXs8nj9K1GUi8qcSddvuJWrHoYymC7eQyWogRiarbZWo23wp+n5bkUwTHAD8J3Ge1AO8b4wZV7HTUkpVhEMZXemLc39UZpEaY34B2rkwF6WU2i9V/jWzUkpVclqkSinlkBapUko5pEWqlFIOaZEqpZRDWqRKKeWQFqlSqkr7raiYpnMWA3DzyvU8tm5zuWdokSqlqrSmAT+ZiXdHpXuEjhnWdwsrixapUqpKS/N46N+oLl6gid/PaTUyyj1Di1QpVeVdXb8Wh4QCPNS0QYXsE1Fhu26kalMLk6JcAKprdipV19sNdKlZ/k9Rk/X+mz+kLLtlKGhr3Js1G0AerM7LtzU+XBIr9e90RaqUUg5pkSqllENapEop5ZAWqVJKOaRFqpRSDmmRKqWUQ1qkSinlkBapUko5pEWqlFIOaZEqpZRDWqRKKeVQ0kUqIl4RmSMi+pn2Sim1Gysr0v7AwoqaiF3d/3EJgx8ZTnZ2tuvZPa7pw4AHh7B12zbXs6+//R76DXyQDRs3uZ59x0PDue7OQaz5fb3r2UNG/puetw1i+eq1rmePHPUBF97+IItWrHY9+63PJ9F9wHDmLl3pevbebIhGGJmfzdgiexuAOLE+GmFo9jZeyN3henZpxBhT9pVEmgL/AYYBdxhjuu/r+h3atzOzpkyyPJmCggL63TmA9Rs2Jj3m86++Ji0tDY/Hw/0D7uKe/jdbzgUIh8PcPuhBVq5K/k7y+dcTCAYDeD1e+t/Uh2H33mErOxqNMmDIoyxasjT57AmT8Pt9+Lxeel/Rg2eHDLKVbYzhwSdG8tO8BclnfzsZn9eLz+fj8gu688JDA/D7/bayH3/pTSbP/CnpMV98NxWPx0PA7+P8M0/nlUH9SE8LWc4GeO6dj/h8yszks6fOREQIBvx0PeE4Xh8ygJqmyFb2m+MnMmbS9OSzZ/wMQFowwMltW/PGvTcz6er7bWVviEZp5PXaGltoYrxckMucSJgIYIC2vkDS44/0+Tk7mG4ru8QYXi3MZWZJMREgBnQOJP+9P9If4JqMLNs7012zdSMLS8J7HZzsNnrPAAOArNKuICJ9gD4AzZs1szpHADZs3MTrb42yPK6wsJBQKMSX30ywXaRbtm7jxVffsDyuuDhMWijEF99MsF2kO7JzeG3Uu+Tk5lkaV1ISwefzMf6bibaLNL+ggLdGf8SadRssjYtEowQCfr6cNIVo7C6s1yiEwyW8/fF4Fvy2zNK4WCyGRzxM+GEGReEbbBVpNBpl9FeTmDJnvqVxxhhEhO9mzSW3oICaadZfZjDG8Mn3P+4qR6t+mLeY7bn2V4J35m3jxay61PBYn3uxgV+jJQjxEgWYGwknPT4nFqNrIA2vjTKLYJgXCSMIsUT6tHDyD2RroiVcnpGFvYfdfStzRSoi3YGzjTE3i8ipwF0VtSK1o16zljQ+oBFPDR/GGaefCha+qU61PLoDAb+fJx55iO7dzkQixa5lH3NKV7Jzc3l88H3845yz8Lh4u0+5oCfLVq1m+KA7uez87nhdvN3n97mNaXN+4ZE7+nLVP84lUGztwceJax98nE8mTmXwjVfS56JzCAUDmOwtrmQPeOG/vDz2G+7tdQF9L+hGZnqI9869xdaxemZv5vmsutSyUaQQXxlODBcypqiAo/wB+qXXsHUcOyLGsNhEeDEvh0P9fkbWru9a9r5WpMkU6XCgFxABQkAN4ENjzBWljXGzSAsKCkhLS9u1XDcl7t2pd66Edz1VKLH3NM+OoqIiAoEAnp13BguPzE4VF4fxej34fPEnNMbF82SRSIRYzBAIJNbAudtdy45Go5REooSCfzyVdatIjTEUhUtI2y07VUW6U4kxeAGPy5trtwwFiRiDAfwuZu+rSMv8lzTG3GuMaWqMORDoAXy7rxJ1W3p6esp249+9wN0WCoX+KFGXBYOBXSXqNp/P90eJuszr9f6pRN0kIn8q0crAL+J6ie7kE3G1RMuiv0eqlFIOWVpWGGMmAZMqZCZKKbWf0hWpUko5pEWqlFIOaZEqpZRDWqRKKeWQFqlSSjmkRaqUUg5pkSpVTSyIhLkmezMA/XO38oOL74ar6rRIlaomGni8xBJfC3CAzR2g1F9pkSpVTdT3eOnkDyLAYV4/B3lT81bbqkiLVKlq5KJQBpkiXJaWkeqpuMoYwwu5O1gTKamQ46dm5wmlqgi7OzCVh5ahoPUxwFfp9jZWrixaNMm0POarHbmM2pTHZj+83qKprdxAbuk7femKVClVpRljGPr7JgwwMSef34rKf6tNLVKlVJU2v7CY5cXxp/QlxjBqc/l/1pMWqVKqSjsyLcj/Dol//NFV9WtxS6O65Z6hRaqUqtJEhFNrxF9cOy4jnfr+8n9pSItUKaUc0iJVSimHtEiVUsohLVKllHJIi1QppRzSIlVKKYe0SJVSyqGkfqFKRFYCuUAUiBhjOlTkpJRSan9iZUV6mjHmmMpWor169+HZF/5NUZH7m9TeeNvdjHjmX+Tn57uefdeDjzDk8afJzslxPXvIU//inmFPsHXbdtez//nqf+k/ZAQbt2x1Pfu1D8dz/ZAnWbtxs+vZ08NFvFCQw6ZY1PXsvdkcjTIiZztj8vNSPZVKQYwxZV8pviLtYIwpffuT3XRo387MmjLJ8mTC4TBDhz/Otu3J30FffOU10tPTCQb8PDpkMDdc1dNyLkA0GuXRp0ayfuPGpMf8+7U3CYVCBPx+7h9wB3fdeK2tbGMMTz7/EitWr0k++41RBAnlPeAAAAydSURBVAMBfD4fd958PQ/dfrPt7OfffJtflyxNPvutd/D7ffh9Pm6+uifDbrsBn8/eu0Vee/9jfpq/MPnst0fj83rx+3xce8l5PH7LNYSCAVvZ/xs/galz5iefPXosXo8Hv8/HZWefztN39+Wzs+19z2eVFDMvEk76+hMSu9n7gA7+IFeFMmmbnmYre0s0Sj2bmzoXmhjP5mYzvjCfCBADLrSwJd8R/gBnO9zCz87uTwCNf1rECwc25vw6NWyN77ZoJXPzC2Vvf5dska4AtgMGeMkY8/JertMH6APQvFmzY1ctmmd5okt+W8rhx9hb8AaDQY46og0zv/3C1vg1a3+nxZHtbWc3b9qExdMn2hq/Zes2GrRqZ2tsKBikVs0arJszxdb43Lw86h/ViXDY+j6NoWCQ9LQQq6d+TlooZHl8cXGYg7qczcYt22xkB/D7/Cwd91/q1a5peXw0GqXtxdezcPkqy2ODAT9ej4d5H7zO9MtvszzeGMPD+TtYHI1YHrvz4eqRzNqckmGvkDpvXMvn9Q+glsd6mW6ORrl06waixpD8w8AfDvB4eb9eI3yy1z5Kit0ifXHDVi6pV5O6Nh/0y6NIGxtj1olIA+BroJ8xZnJp17e7IrWjRsOm1KlTmyeGPcw/zj8XiVbMxq170+TwoxGB4Q89wGUXXYDPWL9j2NW682ls3b6DRwbdzdU9LiKw60MkKl6nv1/MkuUrGHxnP264ogdB3LvdZ119Mz/M/oVBfa+lb68eZMbKf0u00lw+8BHGTvyBu666hNt7XUTNrEzePaWHK9lvFuYyMVxE10Aa5wTTyfJ4bO1HCvEiHVfvAOraXJXmx2K8W5DHqPxcOgaDjKhVz9Zx7LJbpE45LtI/DRB5CMgzxjxZ2nXcLNKtW7dRu3YtPJ746V5T4t4da/uOHWRlZv7xtLbEvfO02Tk5pIVCBAKJp7UufpBZXn4+Pq+PUOKObIrcO0dcWFRENBojMyOxOXGue+dpwyUlFBYVUzPrjzuyW0UaNYYiDBnyx8saqSrSnfJjMfwiBBysLu2ojEVa5hpXRDIAjzEmN/H1mcDQcp6jbXXr1klZdu1atVKWXbOGvfM85SHT5lPK8mDnFEJ5Cfj9BPyp+ZwjrwgZuFtYZcnw6G9P7pTMyYKGwEcSf9TxAf8zxtg7EamUUlVQmUVqjFkOtHVhLkoptV/StblSSjmkRaqUUg5pkSqllENapEop5ZAWqVJKOaRFqpRSDmmRKlVNzA0Xc9amdQD02LqB74oKUjyjqkOLVKlqorbHQ56J78lQbAy1bWxaovZOi1SpaqK5z8/JwTQEaOUPcHTA3nv11V9pkSpVjVyXkYUAvTOyXM+OGcMbeTlsqySbU5cnexvzqVKkclOJyrWhhVtmdr0qpfl2d2AqD8uLrO90NqG4kBgwubCQ+sb+OsrO7f62uJCX83PYHotyR43atrMrI12RKlVNRIxhdHF8y8PPigsoMO7tYRszhudzswEYW5jPtmjVWpVqkSpVTcyJhMlL7D9cAkx1cQ/bOSXFbEw8pY8AnxS6/zlnFUmLVKlq4ihfgL5p8XOjvUKZdPa7t7fr0f4gQ2rG9w6+IiOLcx1+blNlo0WqVDUREqFzIF6eHf1BMl3cmNkvwv+F4p9qcFIg5Hh3/spGi1QppRzSIlVKKYe0SJVSyiEtUqWUckiLVCmlHNIiVUoph7RIlVLKoaSKVERqicgYEVkkIgtFpHNFT0wppfYXya5IRwJfGGNaEf+M+4UVNyVrbr1zAKPeeY9IJOJ69sDBD/PKf0ZRUlLievbDTz7Dsy+/TlGRe2/z2+mfL73OiOdfJr/A/Y2BX3nnA4aM/DfZubmuZ39VWMBzuTvIroK7F+1P8qJRnl6/hdc3bUv1VHYRk3jvbalXEKkBzAVamrKunNChfTsza8oky5OJRqO8/Pqb5ObmJT3mngcGk5GeTq2aNXly+CNcen53y7kAxhhe++/bbNu+I+kxAwc/TFpaGpkZ6Tw6+D5697jIdvao0R+yfuOm5LOHDicUDBIKhRh89230v6anrWyA0Z9+zoo1a5PPHvYEgUCAUCDAwH43cNc1PfDafKfK2G++Y/GyFUlf/76nnscjQiDg547evTjjvc8IiL2dr6YXF7E0kvyD4Ov5OYSNwYdwUXoGvTNrkCb2zo7lxmJkOXxnkZ3dnwB6Zm/m+ay61HKQb3fXq84b1/JK7focaWMv1Igx/Dc/l7cLcwkbQ4mB+xrXT3r8QcEAZ9e2v31gt0UrmZtfuNcftmSK9BjgZeBX4qvR2UB/Y0z+HtfrA/QBaN6s2bGrFs2zPNGFixbT5tjjLY8D8Pv9HHbIIcybNsnW+FWr13DQ0R1sjQ34/TRs0IBVP0+zNX7zlq00bN3OdnZ6ehrbfp1la3xuXh41D29vO9vv97Fh5jekp6VZHl9cHCbzyBOIxazvQuT3+/B6PHxQoy51bOz0HjWGc7esZ5uN7J17T75TtxFNfdZ3oiwxhi6bfmdaw6aWx+7ObpG+VpDLlWmZ+G0+AIH9In0mZzu9M2vaehDJjcU4Z/N6YhIvUTuWHXMYaTYfQJwWaQdgOnCiMWaGiIwEcowxD5Q2xu6K1I70uo3IzMzgkcEPcPUVl+MXm//CNjQ8uA0lkRIGD7ybG665kpDXvT1BD+lwEpu2bGXQ7bdwS++ryQy4997l9meex+JlK7jrxt7cccO11Ai4d7v/dnkffvhpLrdefRkDbriaZX+/0rXse3ds4fviIs5Py+CajBq23y9ebAynprBIy0Oq9mHNjsUY5ynilU3bOTI9yKeHH+hattMibQRMN8YcmPjzycBAY8zfSxvjZpGuWbuWhg0aEAgEADAl7v1wrVu/gTq1axEKJXbRcTF746bNZKSnk5mZ2EXHxS3Rtm7bjs/no2aN+NMkU5T8qRinsnNziUSi1K1dC4CZJ5zrWnahiVEYM9RxuOGGFqkzLZpksj0SRYBaPvcWEPsq0jKflxhjNojIGhE53BizGPgb8af5lUKzps5+GJ1ofECjlGU3bJD8uaHyVrdO6nY3r5nl/kdk7JQmHtKq1qZF+63aLhZoMpI9wdMPeFtEAsBy4JqKm5JSSu1fkipSY8zPgL1XYpRSqorTdzYppZRDWqRKKeWQFqlSSjmkRaqUUg5pkSqllENapEop5ZAWqVIumlVcxGVbNgBw0Zb1/BJO3buTVPnRIlXKRX4RNie24dsQjRJ0sHGIqjy0SJVyUdtAkFb++L4Q7fxBDk98rfZvWqRKueyWzJrx/2fVTPFMVHkpc/cnWwcV2Qyssjm8HrClHKezv9DbXb1U19sN++9tb2GM2etuQRVSpE6IyCxjTLV7X7/e7uqlut5uqJq3XZ/aK6WUQ1qkSinlUGUs0pdTPYEU0dtdvVTX2w1V8LZXunOkSim1v6mMK1KllNqvaJEqpZRDlapIRaSbiCwWkaUiMjDV83GDiDQTkYkislBEFohI/1TPyU0i4hWROSIyLtVzcYuI1BKRMSKyKPF975zqOblBRG5P/IzPF5F3RCSU6jmVl0pTpCLiBZ4HzgLaAJeJSJvUzsoVEeBOY0xroBPQt5rc7p36AwtTPQmXjQS+MMa0AtpSDW6/iDQBbgU6GGOOBLxAj9TOqvxUmiIFOgJLjTHLjTFh4F3gvBTPqcIZY9YbY35KfJ1L/E7VJLWzcoeINAX+Drya6rm4RURqAF2A1wCMMWFjzI7Uzso1PiBNRHxAOrAuxfMpN5WpSJsAa3b781qqSaHsJCIHAu2AGamdiWueAQYAsVRPxEUtgc3AG4lTGq+KSEaqJ1XRjDG/A08Cq4H1QLYx5qvUzqr8VKYi3dt+YtXmd7NEJBP4ALjNGJOT6vlUNBHpDmwyxsxO9Vxc5gPaAy8aY9oB+UCVfz1ARGoTf4Z5ENAYyBCRK1I7q/JTmYp0LdBstz83pQot/fdFRPzES/RtY8yHqZ6PS04EzhWRlcRP45wuIqNSOyVXrAXWGmN2PusYQ7xYq7ozgBXGmM3GmBLgQ+CEFM+p3FSmIv0ROFREDhKRAPET0WNTPKcKJyJC/HzZQmPMP1M9H7cYY+41xjQ1xhxI/Hv9rTGmyqxQSmOM2QCsEZHDExf9Dfg1hVNyy2qgk4ikJ37m/0YVepHNl+oJ7GSMiYjILcCXxF/Re90YsyDF03LDiUAvYJ6I/Jy4bJAxZnwK56QqVj/g7cSCYTlwTYrnU+GMMTNEZAzwE/HfVJlDFXqrqL5FVCmlHKpMT+2VUmq/pEWqlFIOaZEqpZRDWqRKKeWQFqlSSjmkRaqUUg5pkSqllEP/DwD0YGFtC0YcAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "windy_grid.plot(policy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most interesting observation is that the improved policy learned that in order to get to the goal, it shouldn't follow the most direct path to it. But instead \"follow the wind\" and transition to states that while are farer from the goal, the influence of the wind will take the agent to the it.   \n",
    "\n",
    "An example is the state just to the right of the goal. The optimal action is going to the opposite direction and sort of circle around the goal and go to the state under it. In this state the optimal action is again going opposite (downwards). But due to the wind the agent will then be placed directly on the goal state."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}