lab_reinforcement_en.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| | |\n",
    "|-|-|\n",
    "| ![gym](images/gym.png) | ![img](images/pytorch.png) |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hometask\n",
    "\n",
    "The task is complementary for the bayesian bandits task, the maximum is 10 points for both.\n",
    "\n",
    "1. [5 points] Implement memory registry-driven nn-based agent for a chosen Atari game.\n",
    "\n",
    "2. [5 points] Implement screenshots-driven nn-based agent for a chosen Atari game.\n",
    "\n",
    "Supplementary materials with episode lengths and total points recieved plots are expected. \n",
    "\n",
    "Game replay videos should be also attached."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # The typical imports\n",
    "# import gym\n",
    "# import numpy as np\n",
    "# import matplotlib.pyplot as plt\n",
    "# \n",
    "\n",
    "# # Imports specifically so we can render outputs in Jupyter.\n",
    "# from matplotlib import animation\n",
    "# from IPython.display import display\n",
    "\n",
    "\n",
    "# def display_frames_as_gif(frames):\n",
    "#     \"\"\"\n",
    "#     Displays a list of frames as a gif, with controls\n",
    "#     \"\"\"\n",
    "#     #plt.figure(figsize=(frames[0].shape[1] / 72.0, frames[0].shape[0] / 72.0), dpi = 72)\n",
    "#     patch = plt.imshow(frames[0])\n",
    "#     plt.axis('off')\n",
    "\n",
    "#     def animate(i):\n",
    "#         patch.set_data(frames[i])\n",
    "\n",
    "#     anim = animation.FuncAnimation(plt.gcf(), animate, frames = len(frames), interval=50)\n",
    "#     display(anim, default_mode='loop')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use [openai](https://openai.com/systems/) `gym` + `atari_py` for making environments and `pytorch` for training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\"requirements.txt\":\n",
    "\n",
    "```\n",
    "atari_py==0.2.6\n",
    "gym==0.15.4\n",
    "pyglet==1.3.2\n",
    "torch==1.3.1\n",
    "torchvision==0.4.2\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reinforcement Learning Recap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Agent interacts with the environment through actions (**A**), changing its state (**S**) and getting the reward (**R**).\n",
    "\n",
    "The final goal is to maximize a total reward.\n",
    "\n",
    "![](images/recap.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cart-Pole\n",
    "\n",
    "![](images/cartpole.png)\n",
    "\n",
    "* **Task** - keep the pole vertical as long as possible\n",
    "* **State** - angle, rotation speed, position, velocity\n",
    "* **Action** - horizontal force, applied to the cart\n",
    "* **Reward** - 1 for each moment with almost-vertical pole (e.g., 85-95 degrees)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Atari\n",
    "\n",
    "![](images/atari.png)\n",
    "\n",
    "* **Task** - get as many points as possible\n",
    "* **State** - game screen (screenshots)\n",
    "* **Action** - various buttons\n",
    "* **Reward** - is defined by a particular game"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Doom\n",
    "\n",
    "![](images/doom.png)\n",
    "\n",
    "* **Target** - kill 'em all\n",
    "* **State** - game screen (screenshots)\n",
    "* **Action** - various buttons\n",
    "* **Reward** - +1 for killing an enemy, -N for dying"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dsicounted reward\n",
    "\n",
    "It is common to use a discount factor $\\gamma$ to give higher weights for closer rewards.\n",
    "\n",
    "Without the dicounted factor a total reward for all the states after $t$ can be defined as\n",
    "\n",
    "![](images/nodiscount.png)\n",
    "\n",
    "With the discounted factor we focus on the current rewards as the state can change in the future:\n",
    "\n",
    "![](images/discount.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Value function\n",
    "\n",
    "We want to know how good (valuable) each state is. It would help us to choose the best state to go.\n",
    "\n",
    "The value function represent how good is a state for an agent to be in. It is equal to expected total reward for an agent starting from state s. The value function depends on the policy by which the agent picks actions to perform. So, if the agent uses a given policy $\\pi$ to select actions, the corresponding value function is given by\n",
    "\n",
    "![](images/vpolicy.png)\n",
    "\n",
    "Among all possible value-functions, there exist an **optimal value function** that has higher value than other functions for all states:\n",
    "\n",
    "![](images/voptimal.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q-функция\n",
    "\n",
    "Even having optimal value functions we can't just choose the best state, only to choose an action `a`.\n",
    "\n",
    "For better actions choice there is a quality-function Q defining the effectiveness of such actions.\n",
    "\n",
    "$Q^\\pi(s, a)$ is defined as an expected reward for making action `a` and following $\\pi$ afterwards.\n",
    "\n",
    "Just like with the value function, there is an optimal $Q^*(s, a)$\n",
    "\n",
    "Since $V^*(s)$ is the maximum expected total reward when starting from state `s`, it will be the maximum of $Q^*(s, a)$ over all possible actions. Therefore, the relationship between $Q^*(s, a)$ and $V^*(s)$ is easily obtained as:\n",
    "\n",
    "![](images/VQ.png)\n",
    "\n",
    "The optimal strategy is therefore derived from the optimal $Q$:\n",
    "\n",
    "![](images/pioptimal.png)\n",
    "\n",
    "As the result **the task** of optimal strategy search for an agent **is** reduced to **defining $V^*$ и $Q^*$**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q-learning\n",
    "\n",
    "$Q(s, a)$ is defined recursevly through the Bellman equation as\n",
    "\n",
    "![](images/bellman.png)\n",
    "\n",
    "The Q-learning idea is to estimate Q iteratively w.r.t. the Bellman equation:\n",
    "\n",
    "![](images/qiter.png)\n",
    "\n",
    "The initial approximations will be random but the more the agent knows the closer we are to $Q^*$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \"Handmade\" environment\n",
    "\n",
    "![](images/zombie.png)\n",
    "\n",
    "We want to find an icecream and not to be eaten by zombie at the same time.\n",
    "\n",
    "Our action is chosen from 4 directions to go from the current cell.\n",
    "\n",
    "Initial state:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "i *\n",
      "z c\n"
     ]
    }
   ],
   "source": [
    "ZOMBIE = \"z\"\n",
    "CAR = \"c\"\n",
    "ICE_CREAM = \"i\"\n",
    "EMPTY = \"*\"\n",
    "\n",
    "grid = [\n",
    "    [ICE_CREAM, EMPTY],\n",
    "    [ZOMBIE, CAR]\n",
    "]\n",
    "\n",
    "for row in grid:\n",
    "    print(' '.join(row))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "State class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class State:\n",
    "    def __init__(self, grid, car_pos):\n",
    "        self.grid = grid\n",
    "        self.car_pos = car_pos\n",
    "        \n",
    "    def __eq__(self, other):\n",
    "        return isinstance(other, State) and self.grid == other.grid and self.car_pos == other.car_pos\n",
    "    \n",
    "    def __hash__(self):\n",
    "        return hash(str(self.grid) + str(self.car_pos))\n",
    "\n",
    "    def __str__(self):\n",
    "        return f\"State(grid={self.grid}, car_pos={self.car_pos})\"\n",
    "\n",
    "    def __repr__(self):\n",
    "        return self.__str__()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Actions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "UP = 0\n",
    "DOWN = 1\n",
    "LEFT = 2\n",
    "RIGHT = 3\n",
    "\n",
    "ACTIONS = [UP, DOWN, LEFT, RIGHT]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initial state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "State(grid=[['i', '*'], ['z', 'c']], car_pos=[1, 1])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "start_state = State(grid=grid, car_pos=[1, 1])\n",
    "start_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Action functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from copy import deepcopy\n",
    "\n",
    "def act(state, action):\n",
    "    def new_car_pos(state, action):\n",
    "        p = deepcopy(state.car_pos)\n",
    "        if action == UP:\n",
    "            p[0] = max(0, p[0] - 1)\n",
    "        elif action == DOWN:\n",
    "            p[0] = min(len(state.grid) - 1, p[0] + 1)\n",
    "        elif action == LEFT:\n",
    "            p[1] = max(0, p[1] - 1)\n",
    "        elif action == RIGHT:\n",
    "            p[1] = min(len(state.grid[0]) - 1, p[1] + 1)\n",
    "        else:\n",
    "            raise ValueError(f\"Unknown action {action}\")\n",
    "        return p\n",
    "\n",
    "    p = new_car_pos(state, action)\n",
    "    grid_item = state.grid[p[0]][p[1]]\n",
    "    \n",
    "    new_grid = deepcopy(state.grid)\n",
    "    \n",
    "    if grid_item == ZOMBIE:\n",
    "        reward = -100\n",
    "        is_done = True\n",
    "        new_grid[p[0]][p[1]] += CAR\n",
    "    elif grid_item == ICE_CREAM:\n",
    "        reward = 1000\n",
    "        is_done = True\n",
    "        new_grid[p[0]][p[1]] += CAR\n",
    "    elif grid_item == EMPTY:\n",
    "        reward = -1\n",
    "        is_done = False\n",
    "        old = state.car_pos\n",
    "        new_grid[old[0]][old[1]] = EMPTY\n",
    "        new_grid[p[0]][p[1]] = CAR\n",
    "    elif grid_item == CAR:\n",
    "        reward = -1\n",
    "        is_done = False\n",
    "    else:\n",
    "        raise ValueError(f\"Unknown grid item {grid_item}\")\n",
    "\n",
    "    return State(grid=new_grid, car_pos=p), reward, is_done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import random\n",
    "\n",
    "random.seed(42)\n",
    "\n",
    "N_STATES = 4\n",
    "N_EPISODES = 20\n",
    "\n",
    "MAX_EPISODE_STEPS = 100\n",
    "\n",
    "MIN_ALPHA = 0.02\n",
    "\n",
    "alphas = np.linspace(1.0, MIN_ALPHA, N_EPISODES)\n",
    "gamma = 1.0\n",
    "eps = 0.2\n",
    "\n",
    "q_table = dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def q(state, action=None):\n",
    "    if state not in q_table:\n",
    "        q_table[state] = np.zeros(len(ACTIONS))\n",
    "\n",
    "    if action is None:\n",
    "        return q_table[state]\n",
    "\n",
    "    return q_table[state][action]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def choose_action(state):\n",
    "    if random.uniform(0, 1) < eps:\n",
    "        return random.choice(ACTIONS) \n",
    "    else:\n",
    "        return np.argmax(q(state))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 1: total reward -> 999\n",
      "Episode 2: total reward -> 998\n",
      "Episode 3: total reward -> 997\n",
      "Episode 4: total reward -> 997\n",
      "Episode 5: total reward -> 999\n",
      "Episode 6: total reward -> 999\n",
      "Episode 7: total reward -> 998\n",
      "Episode 8: total reward -> -100\n",
      "Episode 9: total reward -> -101\n",
      "Episode 10: total reward -> 999\n",
      "Episode 11: total reward -> 999\n",
      "Episode 12: total reward -> 999\n",
      "Episode 13: total reward -> 999\n",
      "Episode 14: total reward -> 999\n",
      "Episode 15: total reward -> 999\n",
      "Episode 16: total reward -> 998\n",
      "Episode 17: total reward -> 999\n",
      "Episode 18: total reward -> 999\n",
      "Episode 19: total reward -> 999\n",
      "Episode 20: total reward -> 999\n"
     ]
    }
   ],
   "source": [
    "for e in range(N_EPISODES):\n",
    "    state = start_state\n",
    "    total_reward = 0\n",
    "    alpha = alphas[e]\n",
    "\n",
    "    for _ in range(MAX_EPISODE_STEPS):\n",
    "        action = choose_action(state)\n",
    "        next_state, reward, done = act(state, action)\n",
    "        total_reward += reward\n",
    "\n",
    "        q(state)[action] = q(state, action) + \\\n",
    "                alpha * (reward + gamma *  np.max(q(next_state)) - q(state, action))\n",
    "        state = next_state\n",
    "        if done:\n",
    "            break\n",
    "    print(f\"Episode {e + 1}: total reward -> {total_reward}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{State(grid=[['i', '*'], ['z', 'c']], car_pos=[1, 1]): array([998.9999565 , 225.12936017, -85.10182825, 586.19245204]),\n",
       " State(grid=[['i', 'c'], ['z', '*']], car_pos=[0, 1]): array([ 895.94526316,  842.8767095 , 1000.        ,  967.10727091]),\n",
       " State(grid=[['ic', 'c'], ['z', '*']], car_pos=[0, 0]): array([0., 0., 0., 0.]),\n",
       " State(grid=[['i', '*'], ['zc', 'c']], car_pos=[1, 0]): array([0., 0., 0., 0.])}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q_table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "State(grid=[['i', '*'], ['z', 'c']], car_pos=[1, 1])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "start_state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "State(grid=[['i', 'c'], ['z', '*']], car_pos=[0, 1])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "state_1 = act(start_state, choose_action(start_state))[0]\n",
    "state_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "State(grid=[['ic', 'c'], ['z', '*']], car_pos=[0, 0])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "state_2 = act(state_1, choose_action(state_1))[0]\n",
    "state_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Openai Gym"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "\n",
    "env = gym.make('CartPole-v0')\n",
    "observation = env.reset()\n",
    "for t in range(500):\n",
    "    screen = env.render(mode='rgb_array')\n",
    "    action = env.action_space.sample()\n",
    "    observation, reward, done, info = env.step(action)\n",
    "    if done:\n",
    "        observation = env.reset()\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7fd702a6e6a0>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD8CAYAAAB9y7/cAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEqJJREFUeJzt3XGs3eV93/H3p5hAlmQ1hAtybTOTxmtDp8XQO+KIaaKQtsC6mUpNBZsaFCFdJhEpUaOt0ElrkIbUSmvYonUobqFxpiyEkaS4iDZlDlGVPwKxE8excShO4sS39rBZgCSLxmby3R/3ucmJOb73+N57fH0fv1/S0fn9nt9zfuf7wOFzf/e5v4eTqkKS1J+fWu4CJEnjYcBLUqcMeEnqlAEvSZ0y4CWpUwa8JHVqbAGf5PokzyQ5kOTOcb2PJGm4jOM++CTnAH8D/DIwDXwRuKWqnl7yN5MkDTWuK/irgANV9Y2q+r/Ag8CWMb2XJGmIVWM671rg0MD+NPC2k3W+6KKLasOGDWMqRZJWnoMHD/L8889nMecYV8APK+on5oKSTAFTAJdeeik7d+4cUymStPJMTk4u+hzjmqKZBtYP7K8DDg92qKqtVTVZVZMTExNjKkOSzl7jCvgvAhuTXJbkNcDNwPYxvZckaYixTNFU1fEk7wE+A5wDPFBV+8bxXpKk4cY1B09VPQY8Nq7zS5Lm5kpWSeqUAS9JnTLgJalTBrwkdcqAl6ROGfCS1CkDXpI6ZcBLUqcMeEnqlAEvSZ0y4CWpUwa8JHXKgJekThnwktQpA16SOmXAS1KnDHhJ6pQBL0mdWtRX9iU5CHwPeAU4XlWTSS4EPgFsAA4Cv1lVLyyuTEnSqVqKK/hfqqpNVTXZ9u8EdlTVRmBH25cknWbjmKLZAmxr29uAm8bwHpKkeSw24Av4qyS7kky1tkuq6ghAe754ke8hSVqARc3BA1dX1eEkFwOPJ/naqC9sPxCmAC699NJFliFJOtGiruCr6nB7Pgp8GrgKeC7JGoD2fPQkr91aVZNVNTkxMbGYMiRJQyw44JO8LskbZreBXwH2AtuBW1u3W4FHFlukJOnULWaK5hLg00lmz/Pfquovk3wReCjJbcC3gXcuvkxJ0qlacMBX1TeAtw5p/1/AdYspSpK0eK5klaROGfCS1CkDXpI6ZcBLUqcMeEnqlAEvSZ0y4CWpUwa8JHXKgJekThnwktQpA16SOmXAS1KnDHhJ6pQBL0mdMuAlqVMGvCR1yoCXpE4Z8JLUKQNekjo1b8AneSDJ0SR7B9ouTPJ4kmfb8wWtPUk+lORAkj1Jrhxn8ZKkkxvlCv4jwPUntN0J7KiqjcCOtg9wA7CxPaaA+5amTEnSqZo34Kvqr4HvnNC8BdjWtrcBNw20f7RmfAFYnWTNUhUrSRrdQufgL6mqIwDt+eLWvhY4NNBvurW9SpKpJDuT7Dx27NgCy5AkncxS/5E1Q9pqWMeq2lpVk1U1OTExscRlSJIWGvDPzU69tOejrX0aWD/Qbx1weOHlSZIWaqEBvx24tW3fCjwy0P6udjfNZuCl2akcSdLptWq+Dkk+DlwDXJRkGvg94PeBh5LcBnwbeGfr/hhwI3AA+AHw7jHULEkawbwBX1W3nOTQdUP6FnDHYouSJC2eK1klqVMGvCR1yoCXpE4Z8JLUKQNekjplwEtSpwx4SeqUAS9JnTLgJalTBrwkdcqAl6ROGfCS1CkDXpI6ZcBLUqcMeEnqlAEvSZ0y4CWpUwa8JHVq3oBP8kCSo0n2DrR9IMnfJtndHjcOHLsryYEkzyT51XEVLkma2yhX8B8Brh/Sfm9VbWqPxwCSXA7cDPxCe81/SXLOUhUrSRrdvAFfVX8NfGfE820BHqyql6vqm8AB4KpF1CdJWqDFzMG/J8meNoVzQWtbCxwa6DPd2l4lyVSSnUl2Hjt2bBFlSJKGWWjA3wf8LLAJOAL8YWvPkL417ARVtbWqJqtqcmJiYoFlSJJOZkEBX1XPVdUrVfVD4I/58TTMNLB+oOs64PDiSpQkLcSCAj7JmoHdXwdm77DZDtyc5LwklwEbgacWV6IkaSFWzdchyceBa4CLkkwDvwdck2QTM9MvB4HbAapqX5KHgKeB48AdVfXKeEqXJM1l3oCvqluGNN8/R/97gHsWU5QkafFcySpJnTLgJalTBrwkdcqAl6ROGfCS1CkDXpI6Ne9tklIPdm29fWj7L059+DRXIp0+XsFLUqcMeEnqlAEvSZ0y4CWpUwa8JHXKgJekThnwktQpA16SOmXAS1KnDHidFVyxqrORAS9JnZo34JOsT/JEkv1J9iV5b2u/MMnjSZ5tzxe09iT5UJIDSfYkuXLcg5AkvdooV/DHgfdX1VuAzcAdSS4H7gR2VNVGYEfbB7gB2NgeU8B9S161JGle8wZ8VR2pqi+17e8B+4G1wBZgW+u2DbipbW8BPlozvgCsTrJmySuXJM3plObgk2wArgCeBC6pqiMw80MAuLh1WwscGnjZdGs78VxTSXYm2Xns2LFTr1ySNKeRAz7J64FPAu+rqu/O1XVIW72qoWprVU1W1eTExMSoZUiSRjRSwCc5l5lw/1hVfao1Pzc79dKej7b2aWD9wMvXAYeXplxJ0qhGuYsmwP3A/qr64MCh7cCtbftW4JGB9ne1u2k2Ay/NTuVIkk6fUb6y72rgt4CvJtnd2n4X+H3goSS3Ad8G3tmOPQbcCBwAfgC8e0krliSNZN6Ar6rPM3xeHeC6If0LuGORdUmSFsmVrJLUKQNekjplwEtSpwx4ndV2bb19uUuQxsaAl6ROGfCS1CkDXpI6ZcBLUqcMeEnqlAEvSZ0y4CWpUwa8JHXKgJekThnwktQpA16SOmXAS1KnDHhJ6pQBr7PGL059eLlLkE6rUb50e32SJ5LsT7IvyXtb+weS/G2S3e1x48Br7kpyIMkzSX51nAOQJA03ypduHwfeX1VfSvIGYFeSx9uxe6vqPwx2TnI5cDPwC8DPAP8jyd+vqleWsnBJ0tzmvYKvqiNV9aW2/T1gP7B2jpdsAR6sqper6pvAAeCqpShWkjS6U5qDT7IBuAJ4sjW9J8meJA8kuaC1rQUODbxsmrl/IEiSxmDkgE/yeuCTwPuq6rvAfcDPApuAI8AfznYd8vIacr6pJDuT7Dx27NgpFy5JmttIAZ/kXGbC/WNV9SmAqnquql6pqh8Cf8yPp2GmgfUDL18HHD7xnFW1taomq2pyYmJiMWOQJA0xyl00Ae4H9lfVBwfa1wx0+3Vgb9veDtyc5LwklwEbgaeWrmRJ0ihGuYvmauC3gK8m2d3afhe4JckmZqZfDgK3A1TVviQPAU8zcwfOHd5BI0mn37wBX1WfZ/i8+mNzvOYe4J5F1CVJWiRXskpSpwx4SeqUAS9JnTLgJalTBrwkdcqA11lv19bbl7sEaSwMeEnqlAEvSZ0y4CWpUwa8JHXKgJekThnwktQpA16SOmXAa8VLMvJjnOeQzjQGvCR1apQv/JC68ueHp35i/5/9zNZlqkQaL6/gddY7MfClXhjwOqsY5jqbjPKl2+cneSrJV5LsS3J3a78syZNJnk3yiSSvae3ntf0D7fiG8Q5BGp3TMTqbjHIF/zJwbVW9FdgEXJ9kM/AHwL1VtRF4Abit9b8NeKGq3gzc2/pJZ4TJ218d8Ia+ejXKl24X8P22e257FHAt8C9a+zbgA8B9wJa2DfAw8J+TpJ1HWnZ33z35k/vLVIc0biPdRZPkHGAX8Gbgj4CvAy9W1fHWZRpY27bXAocAqup4kpeANwLPn+z8u3bt8v5irRh+VrVSjBTwVfUKsCnJauDTwFuGdWvPwz79r7p6TzIFTAFceumlfOtb3xqpYOlEpztw/WVUp8Pk5OT8neZxSnfRVNWLwOeAzcDqJLM/INYBh9v2NLAeoB3/aeA7Q861taomq2pyYmJiYdVLkk5qlLtoJtqVO0leC7wD2A88AfxG63Yr8Ejb3t72acc/6/y7JJ1+o0zRrAG2tXn4nwIeqqpHkzwNPJjk3wNfBu5v/e8H/muSA8xcud88hrolSfMY5S6aPcAVQ9q/AVw1pP3/AO9ckuokSQvmSlZJ6pQBL0mdMuAlqVP+74K14nmTljScV/CS1CkDXpI6ZcBLUqcMeEnqlAEvSZ0y4CWpUwa8JHXKgJekThnwktQpA16SOmXAS1KnDHhJ6pQBL0mdMuAlqVOjfOn2+UmeSvKVJPuS3N3aP5Lkm0l2t8em1p4kH0pyIMmeJFeOexCSpFcb5f8H/zJwbVV9P8m5wOeT/EU79q+r6uET+t8AbGyPtwH3tWdJ0mk07xV8zfh+2z23Peb6hoUtwEfb674ArE6yZvGlSpJOxUhz8EnOSbIbOAo8XlVPtkP3tGmYe5Oc19rWAocGXj7d2iRJp9FIAV9Vr1TVJmAdcFWSfwDcBfw88I+AC4Hfad0z7BQnNiSZSrIzyc5jx44tqHhJ0smd0l00VfUi8Dng+qo60qZhXgb+FLiqdZsG1g+8bB1weMi5tlbVZFVNTkxMLKh4SdLJjXIXzUSS1W37tcA7gK/NzqsnCXATsLe9ZDvwrnY3zWbgpao6MpbqJUknNcpdNGuAbUnOYeYHwkNV9WiSzyaZYGZKZjfwr1r/x4AbgQPAD4B3L33ZkqT5zBvwVbUHuGJI+7Un6V/AHYsvTZK0GK5klaROGfCS1CkDXpI6ZcBLUqcMeEnqlAEvSZ0y4CWpUwa8JHXKgJekThnwktQpA16SOmXAS1KnDHhJ6pQBL0mdMuAlqVMGvCR1yoCXpE4Z8JLUKQNekjo1csAnOSfJl5M82vYvS/JkkmeTfCLJa1r7eW3/QDu+YTylS5LmcipX8O8F9g/s/wFwb1VtBF4AbmvttwEvVNWbgXtbP0nSaTZSwCdZB/xT4E/afoBrgYdbl23ATW17S9unHb+u9ZcknUarRuz3H4F/A7yh7b8ReLGqjrf9aWBt214LHAKoquNJXmr9nx88YZIpYKrtvpxk74JGcOa7iBPG3olexwX9js1xrSx/L8lUVW1d6AnmDfgkvwYcrapdSa6ZbR7StUY49uOGmaK3tvfYWVWTI1W8wvQ6tl7HBf2OzXGtPEl20nJyIUa5gr8a+OdJbgTOB/4uM1f0q5Osalfx64DDrf80sB6YTrIK+GngOwstUJK0MPPOwVfVXVW1rqo2ADcDn62qfwk8AfxG63Yr8Ejb3t72acc/W1WvuoKXJI3XYu6D/x3gt5McYGaO/f7Wfj/wxtb+28CdI5xrwb+CrAC9jq3XcUG/Y3NcK8+ixhYvriWpT65klaROLXvAJ7k+yTNt5eso0zlnlCQPJDk6eJtnkguTPN5W+T6e5ILWniQfamPdk+TK5at8bknWJ3kiyf4k+5K8t7Wv6LElOT/JU0m+0sZ1d2vvYmV2ryvOkxxM8tUku9udJSv+swiQZHWSh5N8rf239valHNeyBnySc4A/Am4ALgduSXL5cta0AB8Brj+h7U5gR1vlu4Mf/x3iBmBje0wB952mGhfiOPD+qnoLsBm4o/27Weljexm4tqreCmwCrk+ymX5WZve84vyXqmrTwC2RK/2zCPCfgL+sqp8H3srMv7ulG1dVLdsDeDvwmYH9u4C7lrOmBY5jA7B3YP8ZYE3bXgM807Y/DNwyrN+Z/mDmLqlf7mlswN8BvgS8jZmFMqta+48+l8BngLe37VWtX5a79pOMZ10LhGuBR5lZk7Lix9VqPAhcdELbiv4sMnPL+TdP/Oe+lONa7imaH616bQZXxK5kl1TVEYD2fHFrX5Hjbb++XwE8SQdja9MYu4GjwOPA1xlxZTYwuzL7TDS74vyHbX/kFeec2eOCmcWSf5VkV1sFDyv/s/gm4Bjwp21a7U+SvI4lHNdyB/xIq147suLGm+T1wCeB91XVd+fqOqTtjBxbVb1SVZuYueK9CnjLsG7teUWMKwMrzgebh3RdUeMacHVVXcnMNMUdSf7JHH1XythWAVcC91XVFcD/Zu7byk95XMsd8LOrXmcNrohdyZ5LsgagPR9t7StqvEnOZSbcP1ZVn2rNXYwNoKpeBD7HzN8YVreV1zB8ZTZn+Mrs2RXnB4EHmZmm+dGK89ZnJY4LgKo63J6PAp9m5gfzSv8sTgPTVfVk23+YmcBfsnEtd8B/EdjY/tL/GmZWym5f5pqWwuBq3hNX+b6r/TV8M/DS7K9iZ5okYWbR2v6q+uDAoRU9tiQTSVa37dcC72DmD1sremV2dbziPMnrkrxhdhv4FWAvK/yzWFX/EziU5Oda03XA0yzluM6APzTcCPwNM/Og/3a561lA/R8HjgD/j5mfsLcxM5e5A3i2PV/Y+oaZu4a+DnwVmFzu+ucY1z9m5te/PcDu9rhxpY8N+IfAl9u49gL/rrW/CXgKOAD8d+C81n5+2z/Qjr9puccwwhivAR7tZVxtDF9pj32zObHSP4ut1k3AzvZ5/DPggqUclytZJalTyz1FI0kaEwNekjplwEtSpwx4SeqUAS9JnTLgJalTBrwkdcqAl6RO/X/XK4VpC88vZgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(screen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.09530588, -0.71450394, -0.01925967,  0.60931127])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "observation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "done"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{}"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def demo(name, n_episodes=10):\n",
    "    env = gym.make(name)\n",
    "    try:\n",
    "        for i_episode in range(n_episodes):\n",
    "            observation = env.reset()\n",
    "            for t in range(100):\n",
    "                env.render()\n",
    "                action = env.action_space.sample()\n",
    "                observation, reward, done, info = env.step(action)\n",
    "                if done:\n",
    "                    print(\"Episode finished after {} timesteps\".format(t+1))\n",
    "                    break\n",
    "    finally:\n",
    "        env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "demo('Pong-v0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Discrete(2)\n"
     ]
    }
   ],
   "source": [
    "print(env.action_space)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Box(4,)\n"
     ]
    }
   ],
   "source": [
    "print(env.observation_space)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4.8000002e+00 3.4028235e+38 4.1887903e-01 3.4028235e+38]\n"
     ]
    }
   ],
   "source": [
    "print(env.observation_space.high)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-4.8000002e+00 -3.4028235e+38 -4.1887903e-01 -3.4028235e+38]\n"
     ]
    }
   ],
   "source": [
    "print(env.observation_space.low)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gym import spaces\n",
    "space = spaces.Discrete(8) # Set with 8 elements {0, 1, 2, ..., 7}\n",
    "x = space.sample()\n",
    "assert space.contains(x)\n",
    "assert space.n == 8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "demo('MountainCar-v0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "demo('Breakout-v0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "import math\n",
    "import random\n",
    "import numpy as np\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "from collections import namedtuple\n",
    "from itertools import count\n",
    "from PIL import Image\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "from torch.autograd import Variable\n",
    "import torchvision.transforms as T\n",
    "\n",
    "env = gym.make('CartPole-v0').unwrapped\n",
    "\n",
    "# set up matplotlib\n",
    "is_ipython = 'inline' in matplotlib.get_backend()\n",
    "if is_ipython:\n",
    "    from IPython import display\n",
    "\n",
    "plt.ion()\n",
    "\n",
    "# if gpu is to be used\n",
    "use_cuda = False  # torch.cuda.is_available()\n",
    "FloatTensor = torch.cuda.FloatTensor if use_cuda else torch.FloatTensor\n",
    "LongTensor = torch.cuda.LongTensor if use_cuda else torch.LongTensor\n",
    "ByteTensor = torch.cuda.ByteTensor if use_cuda else torch.ByteTensor\n",
    "BoolTensor = torch.cuda.BoolTensor if use_cuda else torch.BoolTensor\n",
    "Tensor = FloatTensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "Transition = namedtuple('Transition',\n",
    "                        ('state', 'action', 'next_state', 'reward'))\n",
    "\n",
    "\n",
    "class ReplayMemory(object):\n",
    "\n",
    "    def __init__(self, capacity):\n",
    "        self.capacity = capacity\n",
    "        self.memory = []\n",
    "        self.position = 0\n",
    "\n",
    "    def push(self, *args):\n",
    "        \"\"\"Saves a transition.\"\"\"\n",
    "        if len(self.memory) < self.capacity:\n",
    "            self.memory.append(None)\n",
    "        self.memory[self.position] = Transition(*args)\n",
    "        self.position = (self.position + 1) % self.capacity\n",
    "\n",
    "    def sample(self, batch_size):\n",
    "        return random.sample(self.memory, batch_size)\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.memory)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DQN(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(DQN, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 16, kernel_size=5, stride=2)\n",
    "        self.bn1 = nn.BatchNorm2d(16)\n",
    "        self.conv2 = nn.Conv2d(16, 32, kernel_size=5, stride=2)\n",
    "        self.bn2 = nn.BatchNorm2d(32)\n",
    "        self.conv3 = nn.Conv2d(32, 32, kernel_size=5, stride=2)\n",
    "        self.bn3 = nn.BatchNorm2d(32)\n",
    "        self.head = nn.Linear(448, 2)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = F.relu(self.bn1(self.conv1(x)))\n",
    "        x = F.relu(self.bn2(self.conv2(x)))\n",
    "        x = F.relu(self.bn3(self.conv3(x)))\n",
    "        return self.head(x.view(x.size(0), -1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "\n",
    "env = gym.make('CartPole-v0').unwrapped\n",
    "observation = env.reset()\n",
    "for t in range(500):\n",
    "    env.render()\n",
    "    action = env.action_space.sample()\n",
    "    observation, reward, done, info = env.step(action)\n",
    "    if done:\n",
    "        observation = env.reset()\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD8CAYAAAB9y7/cAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEppJREFUeJzt3XGs3eV93/H3p5hAlmQ1hAtybDOTxltDp8WwO+KIaaKQtsDWmUrNBJsaFCFdJhEpUaOt0ElrIg2pldawRetQ3ULjVFkII8nwEGvKHKIqfwRiJ45j41BuEie+tYfNAiRZNDaT7/64zw1n5vje43vu9fV9eL+ko/P7Pef5/e73gcPn/u5zfg8nVYUkqT8/s9IFSJKWhwEvSZ0y4CWpUwa8JHXKgJekThnwktSpZQv4JNcneTrJdJI7l+vnSJKGy3LcB5/kHOAvgV8CZoCvALdU1VNL/sMkSUMt1xX8VcB0VX27qv4P8ACwbZl+liRpiDXLdN71wOGB/RngnafqfNFFF9WmTZuWqRRJWn0OHTrEc889l3HOsVwBP6yo/28uKMkUMAVw6aWXsnv37mUqRZJWn8nJybHPsVxTNDPAxoH9DcCRwQ5Vtb2qJqtqcmJiYpnKkKTXruUK+K8Am5NcluR1wM3AzmX6WZKkIZZliqaqTiR5P/B54Bzg/qo6sBw/S5I03HLNwVNVjwKPLtf5JUnzcyWrJHXKgJekThnwktQpA16SOmXAS1KnDHhJ6pQBL0mdMuAlqVMGvCR1yoCXpE4Z8JLUKQNekjplwEtSpwx4SeqUAS9JnTLgJalTBrwkdcqAl6ROjfWVfUkOAT8EXgZOVNVkkguBTwObgEPAP6mq58crU5J0upbiCv4Xq2pLVU22/TuBXVW1GdjV9iVJZ9hyTNFsA3a07R3ATcvwMyRJCxg34Av48yR7kky1tkuq6ihAe754zJ8hSVqEsebggaur6kiSi4HHknxz1APbL4QpgEsvvXTMMiRJJxvrCr6qjrTnY8DngKuAZ5OsA2jPx05x7PaqmqyqyYmJiXHKkCQNseiAT/KGJG+a2wZ+GdgP7ARubd1uBR4et0hJ0ukbZ4rmEuBzSebO85+q6s+SfAV4MMltwPeA94xfpiTpdC064Kvq28A7hrT/T+C6cYqSJI3PlayS1CkDXpI6ZcBLUqcMeEnqlAEvSZ0y4CWpUwa8JHXKgJekThnwktQpA16SOmXAS1KnDHhJ6pQBL0mdMuAlqVMGvCR1yoCXpE4Z8JLUKQNekjplwEtSpxYM+CT3JzmWZP9A24VJHkvyTHu+oLUnyceSTCfZl+TK5SxeknRqo1zBfxy4/qS2O4FdVbUZ2NX2AW4ANrfHFHDv0pQpSTpdCwZ8Vf0F8P2TmrcBO9r2DuCmgfZP1KwvA2uTrFuqYiVJo1vsHPwlVXUUoD1f3NrXA4cH+s20tldJMpVkd5Ldx48fX2QZkqRTWeoPWTOkrYZ1rKrtVTVZVZMTExNLXIYkabEB/+zc1Et7PtbaZ4CNA/02AEcWX54kabEWG/A7gVvb9q3AwwPt721302wFXpybypEknVlrFuqQ5FPANcBFSWaA3wF+F3gwyW3A94D3tO6PAjcC08CPgfctQ82SpBEsGPBVdcspXrpuSN8C7hi3KEnS+FzJKkmdMuAlqVMGvCR1yoCXpE4Z8JLUKQNekjplwEtSpwx4SeqUAS9JnTLgJalTBrwkdcqAl6ROGfCS1CkDXpI6ZcBLUqcMeEnqlAEvSZ0y4CWpUwsGfJL7kxxLsn+g7cNJ/irJ3va4ceC1u5JMJ3k6ya8sV+GSpPmNcgX/ceD6Ie33VNWW9ngUIMnlwM3AL7Rj/mOSc5aqWEnS6BYM+Kr6C+D7I55vG/BAVb1UVd8BpoGrxqhPkrRI48zBvz/JvjaFc0FrWw8cHugz09peJclUkt1Jdh8/fnyMMiRJwyw24O8Ffg7YAhwFfr+1Z0jfGnaCqtpeVZNVNTkxMbHIMiRJp7KogK+qZ6vq5ar6CfBHvDINMwNsHOi6ATgyXomSpMVYVMAnWTew+2vA3B02O4Gbk5yX5DJgM/DkeCVKkhZjzUIdknwKuAa4KMkM8DvANUm2MDv9cgi4HaCqDiR5EHgKOAHcUVUvL0/pkqT5LBjwVXXLkOb75ul/N3D3OEVJksbnSlZJ6pQBL0mdMuAlqVMGvCR1yoCXpE4Z8JLUqQVvk5Rei/Zsv/1VbX936g9XoBJp8byCl6ROGfDSSYZdvUurkQEvjcDpGa1GBrwkdcqAl6ROGfCS1CkDXpI6ZcBLUqcMeEnqlAEvSZ0y4CWpUwsGfJKNSR5PcjDJgSQfaO0XJnksyTPt+YLWniQfSzKdZF+SK5d7EJKkVxvlCv4E8KGqejuwFbgjyeXAncCuqtoM7Gr7ADcAm9tjCrh3yauWJC1owYCvqqNV9dW2/UPgILAe2AbsaN12ADe17W3AJ2rWl4G1SdYteeWSpHmd1hx8kk3AFcATwCVVdRRmfwkAF7du64HDA4fNtLaTzzWVZHeS3cePHz/9yiVJ8xo54JO8EfgM8MGq+sF8XYe01asaqrZX1WRVTU5MTIxahiRpRCMFfJJzmQ33T1bVZ1vzs3NTL+35WGufATYOHL4BOLI05UqSRjXKXTQB7gMOVtVHB17aCdzatm8FHh5of2+7m2Yr8OLcVI4k6cwZ5Sv7rgZ+A/hGkr2t7beB3wUeTHIb8D3gPe21R4EbgWngx8D7lrRiSdJIFgz4qvoSw+fVAa4b0r+AO8asS5I0JleySlKnDHhJ6pQBL0mdMuAlqVMGvCR1yoCXpE4Z8JLUKQNekjplwEtSpwx4SeqUAS9JnTLgJalTBrwkdcqAl6ROGfCS1CkDXpI6ZcBLUqcMeEnq1Chfur0xyeNJDiY5kOQDrf3DSf4qyd72uHHgmLuSTCd5OsmvLOcApKW0Z/vtK12CtGRG+dLtE8CHquqrSd4E7EnyWHvtnqr6t4Odk1wO3Az8AvAW4L8n+ZtV9fJSFi5Jmt+CV/BVdbSqvtq2fwgcBNbPc8g24IGqeqmqvgNMA1ctRbGSpNGd1hx8kk3AFcATren9SfYluT/JBa1tPXB44LAZ5v+FIElaBiMHfJI3Ap8BPlhVPwDuBX4O2AIcBX5/ruuQw2vI+aaS7E6y+/jx46dduCRpfiMFfJJzmQ33T1bVZwGq6tmqermqfgL8Ea9Mw8wAGwcO3wAcOfmcVbW9qiaranJiYmKcMUiShhjlLpoA9wEHq+qjA+3rBrr9GrC/be8Ebk5yXpLLgM3Ak0tXsiRpFKPcRXM18BvAN5LsbW2/DdySZAuz0y+HgNsBqupAkgeBp5i9A+cO76CRpDNvwYCvqi8xfF790XmOuRu4e4y6JEljciWrJHXKgJekThnwktQpA16SOmXAS1KnDHhJ6pQBL0mdMuAlqVMGvCR1yoCXpE4Z8JLUKQNekjplwEtSpwx4vSYkGekx7vHznUM60wx4SerUKF/4Ib3m/NcjUz/d/tW3bF/BSqTFM+ClAYPBLq12TtFICzD0tVqN8qXb5yd5MsnXkxxI8pHWflmSJ5I8k+TTSV7X2s9r+9Pt9U3LOwRpeTlFo9VqlCv4l4Brq+odwBbg+iRbgd8D7qmqzcDzwG2t/23A81X1NuCe1k9aFX71LdsNdHVjlC/dLuBHbffc9ijgWuCftvYdwIeBe4FtbRvgIeA/JEk7j3RWm7x9LtxfCfmPrEwp0thG+pA1yTnAHuBtwB8A3wJeqKoTrcsMsL5trwcOA1TViSQvAm8GnjvV+ffs2eP9w+qG72WdLUYK+Kp6GdiSZC3wOeDtw7q152Hv7lddvSeZAqYALr30Ur773e+OVLC0GGcydP1jVUthcnJy7HOc1l00VfUC8EVgK7A2ydwviA3AkbY9A2wEaK//LPD9IefaXlWTVTU5MTGxuOolSac0yl00E+3KnSSvB94NHAQeB369dbsVeLht72z7tNe/4Py7JJ15o0zRrAN2tHn4nwEerKpHkjwFPJDk3wBfA+5r/e8D/jTJNLNX7jcvQ92SpAWMchfNPuCKIe3fBq4a0v6/gfcsSXWSpEVzJaskdcqAl6ROGfCS1Cn/b5J6TfBGLr0WeQUvSZ0y4CWpUwa8JHXKgJekThnwktQpA16SOmXAS1KnDHhJ6pQBL0mdMuAlqVMGvCR1yoCXpE4Z8JLUKQNekjo1ypdun5/kySRfT3IgyUda+8eTfCfJ3vbY0tqT5GNJppPsS3Llcg9CkvRqo/z/4F8Crq2qHyU5F/hSkv/WXvsXVfXQSf1vADa3xzuBe9uzJOkMWvAKvmb9qO2e2x7zfXvCNuAT7bgvA2uTrBu/VEnS6RhpDj7JOUn2AseAx6rqifbS3W0a5p4k57W29cDhgcNnWpsk6QwaKeCr6uWq2gJsAK5K8reBu4CfB/4ecCHwW617hp3i5IYkU0l2J9l9/PjxRRUvSTq107qLpqpeAL4IXF9VR9s0zEvAnwBXtW4zwMaBwzYAR4aca3tVTVbV5MTExKKKlySd2ih30UwkWdu2Xw+8G/jm3Lx6kgA3AfvbITuB97a7abYCL1bV0WWpXpJ0SqPcRbMO2JHkHGZ/ITxYVY8k+UKSCWanZPYC/7z1fxS4EZgGfgy8b+nLliQtZMGAr6p9wBVD2q89Rf8C7hi/NEnSOFzJKkmdMuAlqVMGvCR1yoCXpE4Z8JLUKQNekjplwEtSpwx4SeqUAS9JnTLgJalTBrwkdcqAl6ROGfCS1CkDXpI6ZcBLUqcMeEnqlAEvSZ0y4CWpUwa8JHVq5IBPck6SryV5pO1fluSJJM8k+XSS17X289r+dHt90/KULkmaz+lcwX8AODiw/3vAPVW1GXgeuK213wY8X1VvA+5p/SRJZ9hIAZ9kA/APgT9u+wGuBR5qXXYAN7XtbW2f9vp1rb8k6QxaM2K/fwf8S+BNbf/NwAtVdaLtzwDr2/Z64DBAVZ1I8mLr/9zgCZNMAVNt96Uk+xc1grPfRZw09k70Oi7od2yOa3X5G0mmqmr7Yk+wYMAn+UfAsarak+SaueYhXWuE115pmC16e/sZu6tqcqSKV5lex9bruKDfsTmu1SfJblpOLsYoV/BXA/84yY3A+cBfZ/aKfm2SNe0qfgNwpPWfATYCM0nWAD8LfH+xBUqSFmfBOfiququqNlTVJuBm4AtV9c+Ax4Ffb91uBR5u2zvbPu31L1TVq67gJUnLa5z74H8L+M0k08zOsd/X2u8D3tzafxO4c4RzLfpPkFWg17H1Oi7od2yOa/UZa2zx4lqS+uRKVknq1IoHfJLrkzzdVr6OMp1zVklyf5Jjg7d5JrkwyWNtle9jSS5o7UnysTbWfUmuXLnK55dkY5LHkxxMciDJB1r7qh5bkvOTPJnk621cH2ntXazM7nXFeZJDSb6RZG+7s2TVvxcBkqxN8lCSb7b/1t61lONa0YBPcg7wB8ANwOXALUkuX8maFuHjwPUntd0J7GqrfHfxyucQNwCb22MKuPcM1bgYJ4APVdXbga3AHe3fzWof20vAtVX1DmALcH2SrfSzMrvnFee/WFVbBm6JXO3vRYB/D/xZVf088A5m/90t3biqasUewLuAzw/s3wXctZI1LXIcm4D9A/tPA+va9jrg6bb9h8Atw/qd7Q9m75L6pZ7GBvw14KvAO5ldKLOmtf/0fQl8HnhX217T+mWlaz/FeDa0QLgWeITZNSmrflytxkPARSe1rer3IrO3nH/n5H/uSzmulZ6i+emq12ZwRexqdklVHQVozxe39lU53vbn+xXAE3QwtjaNsRc4BjwGfIsRV2YDcyuzz0ZzK85/0vZHXnHO2T0umF0s+edJ9rRV8LD634tvBY4Df9Km1f44yRtYwnGtdMCPtOq1I6tuvEneCHwG+GBV/WC+rkPazsqxVdXLVbWF2Sveq4C3D+vWnlfFuDKw4nyweUjXVTWuAVdX1ZXMTlPckeQfzNN3tYxtDXAlcG9VXQH8L+a/rfy0x7XSAT+36nXO4IrY1ezZJOsA2vOx1r6qxpvkXGbD/ZNV9dnW3MXYAKrqBeCLzH7GsLatvIbhK7M5y1dmz604PwQ8wOw0zU9XnLc+q3FcAFTVkfZ8DPgcs7+YV/t7cQaYqaon2v5DzAb+ko1rpQP+K8Dm9kn/65hdKbtzhWtaCoOreU9e5fve9mn4VuDFuT/FzjZJwuyitYNV9dGBl1b12JJMJFnbtl8PvJvZD7ZW9crs6njFeZI3JHnT3Dbwy8B+Vvl7sar+B3A4yd9qTdcBT7GU4zoLPmi4EfhLZudB/9VK17OI+j8FHAX+L7O/YW9jdi5zF/BMe76w9Q2zdw19C/gGMLnS9c8zrr/P7J9/+4C97XHjah8b8HeAr7Vx7Qf+dWt/K/AkMA38Z+C81n5+259ur791pccwwhivAR7pZVxtDF9vjwNzObHa34ut1i3A7vZ+/C/ABUs5LleySlKnVnqKRpK0TAx4SeqUAS9JnTLgJalTBrwkdcqAl6ROGfCS1CkDXpI69f8AEJqEm9LRAoEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import gym\n",
    "\n",
    "env = gym.make('CartPole-v0').unwrapped\n",
    "env.reset()\n",
    "screen = env.render(mode='rgb_array')\n",
    "env.step(env.action_space.sample())\n",
    "env.close()\n",
    "plt.imshow(screen)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAADWCAYAAADBwHkCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAFKhJREFUeJzt3XuwHnV9x/H3h5MLIYRcSMBAokdpuEgHAiKgWIvh0kir4NRWaSuBoV5aHMmIKOCMYmunMuXWGTuoyE1RUKMIpl4IAWqxCiQQIBAggNHEHHK4JCFcDLl8+8f+DuxzznnO85znfvZ8XjM7z/52f7v7fXaf8z37/Haf/SkiMDOzkW+XdgdgZmaN4YRuZlYQTuhmZgXhhG5mVhBO6GZmBeGEbmZWEE7o1nKSTpd0V7vj6CSSuiWFpDHtjsVGLif0gpG0RtIrkl7MDV9td1ztJulYSeuauP4LJV3frPWbVcNnA8X0voi4rd1BjDSSxkTE9nbH0QxFfm/2Op+hjyKSrpC0KFe+SNJSZaZKWizpGUkb0/isXN07JX1Z0v+ls/6fSNpT0nckvSDpXkndufoh6VOSnpL0rKT/kDTo503SgZKWSHpe0mOS/naI9zBZ0lWSeiT9IcXUVeH9TQR+BuyT+9ayTzqrXiTpekkvAKdLOlLSryVtStv4qqRxuXUenIt1g6QLJM0HLgA+lNb9QBWxdkm6OO2bp4C/rHDsPpfWsSXto+Ny67lA0pNp3nJJs3PH4CxJq4HVlfa1pPEppt+n9/Y1SRPSvGMlrZN0jqTe9J7OGCpma4OI8FCgAVgDHF9m3m7A48DpwJ8BzwKz0rw9gb9OdSYBPwB+nFv2TuAJYD9gMvBIWtfxZN/0vgVck6sfwB3ANOCNqe4/pnmnA3el8YnAWuCMtJ7DU1wHl3kPPwa+npbbC7gH+HgV7+9YYF2/dV0IbANOITu5mQC8DTg6xdINrAIWpvqTgB7gHGDXVD4qt67rhxHrJ4BHgdlpH92R9tmYQd7zAWkf7ZPK3cB+afxc4KFUR8ChwJ65Y7AkrX9CpX0NXA7ckupPAn4C/Htu/20H/gUYC5wEvAxMbfdn3kPus9LuADw0+IBmCf1FYFNu+Ghu/pHA88DvgFOHWM9cYGOufCfw+Vz5EuBnufL7gBW5cgDzc+V/Bpam8dN5PaF/CPjfftv+OvDFQWLaG9gKTMhNOxW4o9L7o3xC/2WF/bkQuCm3rfvL1LuQXEKvFCtwO/CJ3LwTKZ/Q/wToJfvnObbfvMeAk8vEFMC8XLnsvib7Z/AS6R9FmvcO4Le5/fdKPr4U09Ht/sx7eH1wG3oxnRJl2tAj4p70FX8v4Pt90yXtBlwGzAempsmTJHVFxI5U3pBb1SuDlHfvt7m1ufHfAfsMEtKbgKMkbcpNGwN8u0zdsUCPpL5pu+S3U+79DSEfI5L2By4FjiA74x8DLE+zZwNPVrHOamLdh4H7Z1AR8YSkhWT/NA6W9Avg0xGxvoqY8tsYal/PIHu/y3PxCujK1X0uStvhX2bgMbc2chv6KCPpLGA8sB74bG7WOWRf24+KiD2Ad/ctUsfmZufG35i22d9a4H8iYkpu2D0i/qlM3a3A9FzdPSLi4L4KQ7y/co8V7T/9CrKmkDlpP1zA6/tgLVmTUzXrqRRrDwP3T1kR8d2IeBdZUg7goipi6h/XUPv6WbJ/ygfn5k2OCCfsEcQJfRRJZ59fBv4B+AjwWUlz0+xJZH/QmyRNI/saXq9z08XW2cDZwPcGqbMY2F/SRySNTcPbJR3Uv2JE9AC3ApdI2kPSLpL2k/TnVby/DcCekiZXiHkS8ALwoqQDgfw/lsXAGyQtTBcQJ0k6Krf+7r4Lv5ViJfv28ClJsyRNBc4rF5CkAyTNkzQe+CPZcer71vRN4F8lzVHmEEl7lllV2X0dETuBK4HLJO2VtruvpL+osL+sgzihF9NPVHof+k3KfrByPXBRRDwQEavJzj6/nRLF5WQXzp4FfgP8vAFx3EzWXLEC+G/gqv4VImILWfvxh8nOqp8mO/scX2adpwHjyC7KbgQWATMrvb+IeBS4AXgq3cEyWPMPwGeAvwO2kCW41/4JpVhPILte8DTZnSPvSbN/kF6fk3TfULGmeVcCvwAeAO4DflQmHtK++ArZsXmarDnpgjTvUrJ/DreS/SO6iuw4DlDFvv4c2YXv36S7fm4j+9ZmI4Qi3MGFNZ6kIGu2eKLdsZiNFj5DNzMrCCd0M7OCcJOLmVlB1HWGLml++vnwE5LKXqU3M7Pmq/kMPT2T4nGyq/7rgHvJfpn3SLllpk+fHt3d3TVtz8xstFq+fPmzETGjUr16fil6JPBERDwFIOlG4GSyW7QG1d3dzbJly+rYpJnZ6COp7C+J8+ppctmX0p8Vr0vT+gfyMUnLJC175pln6ticmZkNpZ6EPthPwge030TENyLiiIg4YsaMit8YzMysRvUk9HWUPotiFoM/q8PMzFqgnoR+LzBH0puVdQDwYbJnKZuZWRvUfFE0IrZL+iTZ8yi6gKsj4uGGRWZmZsNS1/PQI+KnwE8bFIuZmdXBHVyYATtefXnAtK6x/R5aqHoeDW/WfH6Wi5lZQTihm5kVhBO6mVlBOKGbmRWEL4raqPDc478pKfeuXFpS3rlj24BlDnjfuSXlMbtObHxgZg3kM3Qzs4JwQjczKwgndDOzgnAbuo0KWzf3lpQ3r11ZUt51yhsGWcrdM9rI4jN0M7OCcEI3MyuIuppcJK0BtgA7gO0RcUQjgjIzs+FrRBv6eyLi2Qasx6xptEvpl9FdusaWzpe/rNrI50+xmVlB1JvQA7hV0nJJHxusgjuJNjNrjXoT+jERcTjwXuAsSe/uX8GdRJuZtUZdCT0i1qfXXuAm4MhGBGVmZsNXc0KXNFHSpL5x4ERg5dBLmZlZs9Rzl8vewE3KuuUaA3w3In7ekKjMzGzYak7oEfEUcGgDYzEzszr4tkUzs4JwQjczKwgndDOzgnBCNzMrCCd0M7OCcEI3MysIJ3Qzs4JwQjczKwgndDOzgnBCNzMrCCd0M7OCcEI3MyuIigld0tWSeiWtzE2bJmmJpNXpdWpzwzQzs0qqOUO/Fpjfb9p5wNKImAMsTWUzM2ujigk9In4JPN9v8snAdWn8OuCUBsdlZmbDVGsb+t4R0QOQXvcqV9GdRJuZtUbTL4q6k2gzs9aoNaFvkDQTIL32Ni4kMzOrRa0J/RZgQRpfANzcmHDMzKxW1dy2eAPwa+AASesknQl8BThB0mrghFQ2M7M2qthJdEScWmbWcQ2OxczM6uBfipqZFYQTuplZQTihm5kVhBO6mVlBOKGbmRWEE7qZWUE4oZuZFYQTuplZQTihm5kVRMVfipoVQcTOCjU0yKRBppl1MJ+hm5kVhBO6mVlB1NpJ9IWS/iBpRRpOam6YZmZWSTVt6NcCXwW+1W/6ZRFxccMjMmuC8XtMLylrl66S8s5tfxywzPaXN5eUx4yf2PjAzBqo1k6izcysw9TThv5JSQ+mJpmp5Sq5k2gzs9aoNaFfAewHzAV6gEvKVXQn0WZmrVHTfegRsaFvXNKVwOKGRWTWBOMnDd2GvmPbKwOW2davDX3Xqfs0PjCzBqrpDF3SzFzxA8DKcnXNzKw1Kp6hp06ijwWmS1oHfBE4VtJcIIA1wMebGKOZmVWh1k6ir2pCLGZmVgc/y8VGBT/LxUYD//TfzKwgnNDNzArCCd3MrCCc0M3MCsIJ3cysIJzQzcwKwgndzKwgnNDNzArCCd3MrCCc0M3MCsIJ3cysIKrpJHq2pDskrZL0sKSz0/RpkpZIWp1ey/ZaZGZmzVfNGfp24JyIOAg4GjhL0luB84ClETEHWJrKZmbWJtV0Et0TEfel8S3AKmBf4GTgulTtOuCUZgVpZmaVDasNXVI3cBhwN7B3RPRAlvSBvcos406izcxaoOqELml34IfAwoh4odrl3Em0mVlrVJXQJY0lS+bfiYgfpckb+voWTa+9zQnRzMyqUc1dLiLrcm5VRFyam3ULsCCNLwBubnx4ZmZWrWq6oDsG+AjwkKQVadoFwFeA70s6E/g98DfNCdHMzKpRTSfRdzFoh4sAHNfYcMzMrFb+paiZWUE4oZuZFYQTuplZQTihm5kVhBO6mVlBOKGbmRWEE7qZWUE4oZuZFYQTuplZQTihm5kVhBO6mVlBOKGbmRVEPZ1EXyjpD5JWpOGk5odrZmblVPP43L5Oou+TNAlYLmlJmndZRFzcvPDMzKxa1Tw+twfo6zt0i6S+TqLNzKyD1NNJNMAnJT0o6WpJU8ss406izcxaoJ5Ooq8A9gPmkp3BXzLYcu4k2sysNWruJDoiNkTEjojYCVwJHNm8MM3MrJKaO4mWNDNX7QPAysaHZ2Zm1aqnk+hTJc0FAlgDfLwpEZqZWVXq6ST6p40Px8zMauVfipqZFYQTuplZQTihm5kVhBO6mVlBOKGbmRVENbctmo18EcNfRoPd3GXWuXyGbmZWEE7oZmYF4YRuZlYQbkO3UaFr3ISSsrpKP/o7t786YJkdW19uakxmjeYzdDOzgnBCNzMriGoen7urpHskPZA6if5Smv5mSXdLWi3pe5LGNT9cMzMrp5o29K3AvIh4MXV0cZeknwGfJusk+kZJXwPOJOvFyKzj7DZtZkl57K4TS8pbN784YJmtG3tKJ3Qf1vC4zBqp4hl6ZPo+7WPTEMA8YFGafh1wSlMiNDOzqlTbBV1X6tyiF1gCPAlsiojtqco6YN8yy7qTaDOzFqgqoae+Q+cCs8j6Dj1osGpllnUn0WZmLTCs+9AjYpOkO4GjgSmSxqSz9FnA+ibEZ6PQ5s2bS8pnnHFGxTqVTBxfeu7y6fe+paQ8ebeBJxvXXntNSfnWz1wyrG0OZsGCBSXl0047re51mvWp5i6XGZKmpPEJwPHAKuAO4IOp2gLg5mYFaWZmlVVzhj4TuE5SF9k/gO9HxGJJjwA3SvoycD9wVRPjNDOzCqrpJPpBYMD9WhHxFFl7upmZdQA/y8U6zquvlj5X5bbbbhtQZ8uWLcNa57gxpR/1t8/9aEl59ylzBizzq5VfKCnffvvtw9rmYN75znfWvQ6zcvzTfzOzgnBCNzMrCCd0M7OCcEI3MysIXxS1jjOm3wXM8ePHD6gz7Iui43crKe/sml5S3q49Biyzs2vgtHqNHTu24es06+MzdDOzgnBCNzMrCCd0M7OCaGkb+rZt2+jp6alc0Ua1559/vqS8c+fOute5Y9tLJeWVv/5SSfnJ3oEPC316/UN1b7e//m3//nuwRvIZuplZQTihm5kVRD2dRF8r6beSVqRhbvPDNTOzcurpJBrg3IhYNMSyJbZv3467obNKNm7cWFJuRBv6K6/uKCkvWvrLutdZi5deKm3L99+DNVI1j88NYLBOos3MrIPU1El0RNydZv2bpAclXSZp4M/5KO0kuv+Zl5mZNU5NnURL+lPgfOBA4O3ANOBzZZZ9rZPoqVOnNihsMzPrr9ZOoudHxMVp8lZJ1wCfqbT8hAkTOOSQQ4YfpY0qmzZtKin3f7bLSDZz5sySsv8erJFq7ST6UUkz0zQBpwArmxmomZkNrZ5Oom+XNAMQsAL4RBPjNDOzCurpJHpeUyIyM7OaFKdx0gpj27ZtJeWtW7e2KZLG698Btlkj+af/ZmYF4YRuZlYQTuhmZgXhhG5mVhC+KGodZ9y4cSXlE088cUCdzZs3tyqchtp///3bHYIVmM/QzcwKwgndzKwgnNDNzArCbejWcSZPnlxSXrSo6j5UzEY1n6GbmRWEE7qZWUE4oZuZFYSyLkNbtDHpGeB3wHTg2ZZtuHaOs7FGQpwjIUZwnI3W6XG+KSJmVKrU0oT+2kalZRFxRMs3PEyOs7FGQpwjIUZwnI02UuKsxE0uZmYF4YRuZlYQ7Uro32jTdofLcTbWSIhzJMQIjrPRRkqcQ2pLG7qZmTWem1zMzArCCd3MrCBantAlzZf0mKQnJJ3X6u2XI+lqSb2SVuamTZO0RNLq9Dq1zTHOlnSHpFWSHpZ0dofGuaukeyQ9kOL8Upr+Zkl3pzi/J2lcpXW1gqQuSfdLWpzKHRenpDWSHpK0QtKyNK2jjnuKaYqkRZIeTZ/Td3RSnJIOSPuwb3hB0sJOirEeLU3okrqA/wLeC7wVOFXSW1sZwxCuBeb3m3YesDQi5gBLU7mdtgPnRMRBwNHAWWn/dVqcW4F5EXEoMBeYL+lo4CLgshTnRuDMNsaYdzawKlfu1DjfExFzc/dLd9pxB/hP4OcRcSBwKNl+7Zg4I+KxtA/nAm8DXgZu6qQY6xIRLRuAdwC/yJXPB85vZQwV4usGVubKjwEz0/hM4LF2x9gv3puBEzo5TmA34D7gKLJf4o0Z7LPQxvhmkf0BzwMWA+rQONcA0/tN66jjDuwB/JZ0s0WnxpmL60TgV50c43CHVje57AuszZXXpWmdau+I6AFIr3u1OZ7XSOoGDgPupgPjTM0YK4BeYAnwJLApIranKp1y7C8HPgvsTOU96cw4A7hV0nJJH0vTOu24vwV4BrgmNWF9U9JEOi/OPh8GbkjjnRrjsLQ6oWuQab5vcpgk7Q78EFgYES+0O57BRMSOyL7WzgKOBA4arFproyol6a+A3ohYnp88SNVO+IweExGHkzVXniXp3e0OaBBjgMOBKyLiMOAlOrTpIl0XeT/wg3bH0kitTujrgNm58ixgfYtjGI4NkmYCpNfeNseDpLFkyfw7EfGjNLnj4uwTEZuAO8na/KdI6utUpROO/THA+yWtAW4ka3a5nM6Lk4hYn157ydp8j6Tzjvs6YF1E3J3Ki8gSfKfFCdk/xvsiYkMqd2KMw9bqhH4vMCfdRTCO7CvPLS2OYThuARak8QVkbdZtI0nAVcCqiLg0N6vT4pwhaUoanwAcT3Zx7A7gg6la2+OMiPMjYlZEdJN9Fm+PiL+nw+KUNFHSpL5xsrbflXTYcY+Ip4G1kg5Ik44DHqHD4kxO5fXmFujMGIevDRciTgIeJ2tT/Xy7LyLk4roB6AG2kZ1pnEnWnroUWJ1ep7U5xneRff1/EFiRhpM6MM5DgPtTnCuBL6TpbwHuAZ4g+6o7vt3HPRfzscDiTowzxfNAGh7u+7vptOOeYpoLLEvH/sfA1E6Lk+xC/XPA5Ny0joqx1sE//TczKwj/UtTMrCCc0M3MCsIJ3cysIJzQzcwKwgndzKwgnNDNzArCCd3MrCD+H9LYDvDWJZy3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "resize = T.Compose([\n",
    "    T.ToPILImage(),\n",
    "    T.Resize(40, interpolation=Image.CUBIC),\n",
    "    T.ToTensor()\n",
    "])\n",
    "# This is based on the code from gym.\n",
    "screen_width = 600\n",
    "\n",
    "\n",
    "def get_cart_location():\n",
    "    world_width = env.x_threshold * 2\n",
    "    scale = screen_width / world_width\n",
    "    return int(env.state[0] * scale + screen_width / 2.0)  # MIDDLE OF CART\n",
    "\n",
    "\n",
    "def get_screen():\n",
    "    screen = env.render(mode='rgb_array').transpose((2, 0, 1))\n",
    "    screen = screen[:, 160:320]\n",
    "    view_width = 320\n",
    "    cart_location = get_cart_location()\n",
    "    if cart_location < view_width // 2:\n",
    "        slice_range = slice(view_width)\n",
    "    elif cart_location > (screen_width - view_width // 2):\n",
    "        slice_range = slice(-view_width, None)\n",
    "    else:\n",
    "        slice_range = slice(cart_location - view_width // 2,\n",
    "                            cart_location + view_width // 2)\n",
    "    # Strip off the edges, so that we have a square image centered on a cart\n",
    "    screen = screen[:, :, slice_range]\n",
    "    # Convert to float, rescare, convert to torch tensor\n",
    "    # (this doesn't require a copy)\n",
    "    screen = np.ascontiguousarray(screen, dtype=np.float32) / 255\n",
    "    screen = torch.from_numpy(screen)\n",
    "    # Resize, and add a batch dimension (BCHW)\n",
    "    return resize(screen).unsqueeze(0).type(Tensor)\n",
    "\n",
    "\n",
    "\n",
    "env = gym.make('CartPole-v0').unwrapped\n",
    "env.reset()\n",
    "env.step(env.action_space.sample())\n",
    "plt.figure()\n",
    "plt.imshow(get_screen().cpu().squeeze(0).permute(1, 2, 0).numpy(),interpolation='none')\n",
    "plt.title('Example extracted screen')\n",
    "plt.show()\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 128\n",
    "GAMMA = 0.999\n",
    "EPS_START = 0.9\n",
    "EPS_END = 0.05\n",
    "EPS_DECAY = 200\n",
    "TARGET_UPDATE = 10\n",
    "\n",
    "policy_net = DQN()\n",
    "target_net = DQN()\n",
    "target_net.load_state_dict(policy_net.state_dict())\n",
    "target_net.eval()\n",
    "\n",
    "if use_cuda:\n",
    "    policy_net.cuda()\n",
    "    target_net.cuda()\n",
    "\n",
    "optimizer = optim.RMSprop(policy_net.parameters())\n",
    "memory = ReplayMemory(10000)\n",
    "\n",
    "\n",
    "steps_done = 0\n",
    "\n",
    "\n",
    "def select_action(state):\n",
    "    global steps_done\n",
    "    sample = random.random()\n",
    "    eps_threshold = EPS_END + (EPS_START - EPS_END) * \\\n",
    "        math.exp(-1. * steps_done / EPS_DECAY)\n",
    "    steps_done += 1\n",
    "    if sample > eps_threshold:\n",
    "        with torch.no_grad():\n",
    "            return policy_net(state.type(FloatTensor)).data.max(1)[1].view(1, 1)\n",
    "    else:\n",
    "        return LongTensor([[random.randrange(2)]])\n",
    "\n",
    "\n",
    "episode_durations = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "def optimize_model():\n",
    "    if len(memory) < BATCH_SIZE:\n",
    "        return\n",
    "    transitions = memory.sample(BATCH_SIZE)\n",
    "    # Transpose the batch (see http://stackoverflow.com/a/19343/3343043 for\n",
    "    # detailed explanation).\n",
    "    batch = Transition(*zip(*transitions))\n",
    "\n",
    "    # Compute a mask of non-final states and concatenate the batch elements\n",
    "    non_final_mask = BoolTensor(tuple(map(lambda s: s is not None,\n",
    "                                          batch.next_state)))\n",
    "    non_final_next_states = torch.cat([s for s in batch.next_state\n",
    "                                       if s is not None])\n",
    "    state_batch = torch.cat(batch.state)\n",
    "    action_batch = torch.cat(batch.action)\n",
    "    reward_batch = torch.cat(batch.reward)\n",
    "\n",
    "    # Compute Q(s_t, a) - the model computes Q(s_t), then we select the\n",
    "    # columns of actions taken\n",
    "    state_action_values = policy_net(state_batch).gather(1, action_batch)\n",
    "\n",
    "    # Compute V(s_{t+1}) for all next states.\n",
    "    next_state_values = torch.zeros(BATCH_SIZE).type(Tensor)\n",
    "    with torch.no_grad():\n",
    "        next_state_values[non_final_mask] = target_net(non_final_next_states).max(1)[0]\n",
    "    # Compute the expected Q values\n",
    "    expected_state_action_values = (next_state_values * GAMMA) + reward_batch\n",
    "\n",
    "    # Compute Huber loss\n",
    "    loss = F.smooth_l1_loss(state_action_values, expected_state_action_values)\n",
    "\n",
    "    # Optimize the model\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    for param in policy_net.parameters():\n",
    "        param.grad.data.clamp_(-1, 1)\n",
    "    optimizer.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_durations():\n",
    "    plt.figure(2)\n",
    "    plt.clf()\n",
    "    durations_t = torch.FloatTensor(episode_durations)\n",
    "    plt.title('Training...')\n",
    "    plt.xlabel('Episode')\n",
    "    plt.ylabel('Duration')\n",
    "    plt.plot(durations_t.numpy())\n",
    "    # Take 100 episode averages and plot them too\n",
    "    if len(durations_t) >= 100:\n",
    "        means = durations_t.unfold(0, 100, 1).mean(1).view(-1)\n",
    "        means = torch.cat((torch.zeros(99), means))\n",
    "        plt.plot(means.numpy())\n",
    "\n",
    "    plt.pause(0.001)  # pause a bit so that plots are updated\n",
    "    if is_ipython:\n",
    "        display.clear_output(wait=True)\n",
    "    #     display.display(plt.gcf())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m--------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-59-11779efdcef4>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     27\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     28\u001b[0m         \u001b[0;31m# Perform one step of the optimization (on the target network)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 29\u001b[0;31m         \u001b[0moptimize_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     30\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     31\u001b[0m             \u001b[0mepisode_durations\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-57-052da663916a>\u001b[0m in \u001b[0;36moptimize_model\u001b[0;34m()\u001b[0m\n\u001b[1;32m     32\u001b[0m     \u001b[0;31m# Optimize the model\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     33\u001b[0m     \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 34\u001b[0;31m     \u001b[0mloss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     35\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mparam\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mpolicy_net\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparameters\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     36\u001b[0m         \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclamp_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.anaconda3/lib/python3.7/site-packages/torch/tensor.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(self, gradient, retain_graph, create_graph)\u001b[0m\n\u001b[1;32m    164\u001b[0m                 \u001b[0mproducts\u001b[0m\u001b[0;34m.\u001b[0m \u001b[0mDefaults\u001b[0m \u001b[0mto\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    165\u001b[0m         \"\"\"\n\u001b[0;32m--> 166\u001b[0;31m         \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgradient\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    167\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    168\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mregister_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.anaconda3/lib/python3.7/site-packages/torch/autograd/__init__.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(tensors, grad_tensors, retain_graph, create_graph, grad_variables)\u001b[0m\n\u001b[1;32m     97\u001b[0m     Variable._execution_engine.run_backward(\n\u001b[1;32m     98\u001b[0m         \u001b[0mtensors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad_tensors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 99\u001b[0;31m         allow_unreachable=True)  # allow_unreachable flag\n\u001b[0m\u001b[1;32m    100\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    101\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "num_episodes = 50\n",
    "for i_episode in range(num_episodes):\n",
    "    # Initialize the environment and state\n",
    "    env.reset()\n",
    "    last_screen = get_screen()\n",
    "    current_screen = get_screen()\n",
    "    state = current_screen - last_screen\n",
    "    for t in count():\n",
    "        # Select and perform an action\n",
    "        action = select_action(state)\n",
    "        _, reward, done, _ = env.step(action[0, 0].item())\n",
    "        reward = Tensor([reward])\n",
    "\n",
    "        # Observe new state\n",
    "        last_screen = current_screen\n",
    "        current_screen = get_screen()\n",
    "        if not done:\n",
    "            next_state = current_screen - last_screen\n",
    "        else:\n",
    "            next_state = None\n",
    "\n",
    "        # Store the transition in memory\n",
    "        memory.push(state, action, next_state, reward)\n",
    "\n",
    "        # Move to the next state\n",
    "        state = next_state\n",
    "\n",
    "        # Perform one step of the optimization (on the target network)\n",
    "        optimize_model()\n",
    "        if done:\n",
    "            episode_durations.append(t + 1)\n",
    "            plot_durations()\n",
    "            break\n",
    "    # Update the target network\n",
    "    if i_episode % TARGET_UPDATE == 0:\n",
    "        target_net.load_state_dict(policy_net.state_dict())\n",
    "\n",
    "print('Complete')\n",
    "env.close()\n",
    "plt.ioff()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "tran = memory.sample(1)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7fd693ca4e48>"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAADKCAYAAAC11LviAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADkJJREFUeJzt3W2spGV9x/Hvr8uyukhdkAdXlhQ1VCGmLJZSDE2DKGYljQ+JTSRNsy9IsIkmmphWaJNWkjbRpIp90ZhgRXlhfShqIQQfNivG2DQo4IoLK4K6lpWVI1UqioK7598Xcx89s3vOnjnzdOZc+/0kk5nrmvve+eXMnP/e55p75p+qQpK0/v3OWgeQJI2HBV2SGmFBl6RGWNAlqREWdElqhAVdkhphQZekRljQJakRIxX0JDuSPJjk4STXjiuUJGn1MuwnRZNsAL4DXAEcAL4OXFVVDyy3z+bNm2vLli1DPZ4kHa8OHjz4eFWdvtJ2J4zwGBcDD1fV9wCSfAJ4PbBsQd+yZQvXXHPNCA8pScef66+//geDbDfKkstZwCOLxge6uT5Jrklyd5K7n3rqqREeTpJ0LKMU9Cwxd9T6TVXdWFUXVdVFmzdvHuHhJEnHMkpBPwCcvWi8DXh0tDiSpGGNUtC/Dpyb5IVJTgTeDNw2nliSpNUa+k3RqjqU5G3AF4ANwE1Vdf/YkkmSVmWUs1yoqjuAO8aURZI0gpEKurReHX7ml33jE0869ehtDv2qf8LuXppxfvRfkhphQZekRljQJakRFnRJaoRviuq48Lzfv6Rv/OQPv9M3fuYXPzlqnx/t+Vzf+PnbXzv+YNIYeYQuSY2woEtSIyzoktQI19B1XNj03DP6xnXocN/44L37jtrnF3P7JxlJGjuP0CWpERZ0SWrESEsuSfYDTwKHgUNVddE4QkmSVm8ca+ivrKrHx/DvSBNT8/N94/n5/jV0fufoP1Z7fdCl9cMlF0lqxKgFvYAvJrknyTVLbWCTaEmajlGXXC6tqkeTnAHsSvLtqvrK4g2q6kbgRoAXvOAFfqG0JE3IqB2LHu2u55J8FrgY+Mqx95Km78iGFk8/ObfiPptP2zapONJEDL3kkuSkJCcv3AZeA+wdVzBJ0uqMcoR+JvDZJAv/zr9X1efHkkqStGpDF/Sq+h5wwRizSJJG4He56LhQR5x3Pv/Mr5bZ8rc2bDppUnGkifA8dElqhAVdkhphQZekRljQJakRvimq41PvdNtjKz/YrPXFI3RJaoQFXZIaYUGXpEZY0CWpERZ0SWqEBV2SGrFiQU9yU5K5JHsXzZ2aZFeSh7rrUyYbU5K0kkGO0D8K7Dhi7lpgd1WdC+zuxpKkNbRiQe9ayv3kiOnXAzd3t28G3jDmXJKkVRp2Df3MqjoI0F2fsdyGNomWpOmY+JuiVXVjVV1UVRdt3rx50g8nScetYQv6Y0m2AnTXK3fclSRN1LAF/TZgZ3d7J3DreOJIkoY1yGmLHwf+G3hJkgNJrgbeA1yR5CHgim4sSVpDK359blVdtcxdrxpzFknSCPykqCQ1woIuSY2woEtSIyzoktQIC7okNcKCLkmNsKBLUiMs6JLUCAu6JDXCgi5JjbCgS1IjLOiS1Ihhm0S/O8kPk+zpLldONqYkaSXDNokGuKGqtneXO8YbS5K0WsM2iZYkzZhR1tDfluS+bknmlOU2skm0JE3HsAX9g8CLge3AQeB9y21ok2hJmo6hCnpVPVZVh6tqHvgQcPF4Y0mSVmuogp5k66LhG4G9y20rSZqOFXuKdk2iLwNOS3IA+AfgsiTbgQL2A2+ZYEZJ0gCGbRL94QlkkSSNwE+KSlIjLOiS1AgLuiQ1woIuSY2woEtSI1Y8y0VqQrLq+xOPd7S++IqVpEZY0CWpERZ0SWqEa+g6Lvzy8QN94/nDv+4bb3zWyUfts2nL8yeaSRo3j9AlqRGD9BQ9O8mdSfYluT/J27v5U5PsSvJQd71skwtJ0uQNcoR+CHhnVZ0HXAK8Ncn5wLXA7qo6F9jdjSVJa2SQb1s8SK8rEVX1ZJJ9wFnA6+l9rS7AzcCXgXdNJKU0osNPH9H+sKp/7HnoasCqXrFJzgEuBO4CzuyK/ULRP2Pc4SRJgxu4oCd5DvBp4B1V9bNV7GeTaEmagoEKepKN9Ir5x6rqM930Ywut6LrruaX2tUm0JE3HIGe5hF6Hon1V9f5Fd90G7Oxu7wRuHX88aUyS/ovUoEE+WHQp8JfAt5Ls6eb+FngP8KkkVwP/A/z5ZCJKkgYxyFkuXwWWO6R51XjjSJKG5XlZktQIC7okNcKCLkmNsKBLUiMs6JLUCAu6JDXCgi5JjbCgS1IjLOiS1AgLuiQ1woIuSY2woEtSI0ZpEv3uJD9Msqe7XDn5uJKk5Qzy9bkLTaLvTXIycE+SXd19N1TVP08uniRpUKM0iZbWkSGaWhzZSFqacaM0iQZ4W5L7ktyU5JQxZ5MkrcIoTaI/CLwY2E7vCP59y+xnk2hJmoKhm0RX1WNVdbiq5oEPARcvta9NoiVpOlZcQ1+uSXSSrd36OsAbgb2TiSiN7ukn5/rGNX+4b5wNR/8qPPu0bRPNJI3bKE2ir0qyHShgP/CWiSSUJA1klCbRd4w/jiRpWH5SVJIaMciSi7TuzT/zq/6J5Ijh0cc2GzYd8Sa+56VrxnmELkmNsKBLUiMs6JLUCAu6JDXCN0V1fIhfzqX2eYQuSY2woEtSIyzoktQIC7okNcKCLkmNGKRJ9LOSfC3JN7sm0dd38y9McleSh5J8MsmJk48rSVrOIEfoTwOXV9UF9LoT7UhyCfBeek2izwV+Clw9uZiSpJWsWNCr5+fdcGN3KeBy4JZu/mbgDRNJKEkayKAt6DZ0zS3mgF3Ad4EnqupQt8kB4KzJRJQkDWKggt71Dt0ObKPXO/S8pTZbal+bREvSdKzqLJeqegL4MnAJsCXJwlcHbAMeXWYfm0RL0hQMcpbL6Um2dLefDbwa2AfcCbyp22wncOukQkqSVjbIl3NtBW5OsoHefwCfqqrbkzwAfCLJPwLfAD48wZySpBUM0iT6PuDCJea/R289XZI0A/ykqCQ1wu9D13HhhBzqG89X/7HMYTYctU+WPnFLmlkeoUtSIyzoktQIC7okNcKCLkmN8E1RHRfma75v/Ov+ISdu3HjUPjXfv9G8TaM14zxCl6RGWNAlqREWdElqhGvoOi7831P969+PzP2kb/ymy1521D6PzD3RN37yqWfGH0waI4/QJakRozSJ/miS7yfZ0122Tz6uJGk5gyy5LDSJ/nmSjcBXk3yuu++vq+qWY+wrSZqSQb4+t4ClmkRL68bJmzf1jc8/54y+8QP756YZR5qIoZpEV9Vd3V3/lOS+JDck2XSMf0KSNGFDNYlO8jLgOuClwB8BpwLvWmpfm0RL0nQM2yR6R1UdrJ6ngY+wTPcim0RL0nQM2yT620m2dnMB3gDsnWRQSdKxjdIk+ktJTgcC7AH+aoI5JUkrGKVJ9OUTSSRJGoqfFJWkRljQJakRFnRJaoQFXZIaYUGXpEZY0CWpERZ0SWqEBV2SGmFBl6RGWNAlqREWdElqhAVdkhphQZekRljQJakR6fWAntKDJT8GfgCcBjw+tQcenjnHaz3kXA8ZwZzjNus5f6+qTl9po6kW9N88aHJ3VV009QdeJXOO13rIuR4ygjnHbb3kXIlLLpLUCAu6JDVirQr6jWv0uKtlzvFaDznXQ0Yw57itl5zHtCZr6JKk8XPJRZIaMfWCnmRHkgeTPJzk2mk//nKS3JRkLsneRXOnJtmV5KHu+pQ1znh2kjuT7Etyf5K3z2jOZyX5WpJvdjmv7+ZfmOSuLucnk5y4ljkXJNmQ5BtJbu/GM5czyf4k30qyJ8nd3dxMPe9dpi1Jbkny7e51+opZypnkJd3PcOHysyTvmKWMo5hqQU+yAfhX4LXA+cBVSc6fZoZj+Ciw44i5a4HdVXUusLsbr6VDwDur6jzgEuCt3c9v1nI+DVxeVRcA24EdSS4B3gvc0OX8KXD1GmZc7O3AvkXjWc35yqravuj0ull73gH+Bfh8Vb0UuIDez3VmclbVg93PcDvwh8BTwGdnKeNIqmpqF+AVwBcWja8DrptmhhXynQPsXTR+ENja3d4KPLjWGY/IeytwxSznBDYD9wJ/TO+DGycs9VpYw3zb6P0CXw7cDmRGc+4HTjtibqaed+B3ge/TvTc3qzkX5XoN8F+znHG1l2kvuZwFPLJofKCbm1VnVtVBgO76jDXO8xtJzgEuBO5iBnN2yxh7gDlgF/Bd4ImqOtRtMivP/QeAvwHmu/HzmM2cBXwxyT1JrunmZu15fxHwY+Aj3RLWvyU5idnLueDNwMe727OacVWmXdCzxJyn2axSkucAnwbeUVU/W+s8S6mqw9X7s3YbcDFw3lKbTTdVvyR/BsxV1T2Lp5fYdBZeo5dW1cvpLVe+NcmfrnWgJZwAvBz4YFVdCPyCGV266N4XeR3wH2udZZymXdAPAGcvGm8DHp1yhtV4LMlWgO56bo3zkGQjvWL+sar6TDc9czkXVNUTwJfprflvSXJCd9csPPeXAq9Lsh/4BL1llw8wezmpqke76zl6a74XM3vP+wHgQFXd1Y1voVfgZy0n9P5jvLeqHuvGs5hx1aZd0L8OnNudRXAivT95bptyhtW4DdjZ3d5Jb816zSQJ8GFgX1W9f9Fds5bz9CRbutvPBl5N782xO4E3dZutec6quq6qtlXVOfRei1+qqr9gxnImOSnJyQu36a397mXGnveq+hHwSJKXdFOvAh5gxnJ2ruK3yy0wmxlXbw3eiLgS+A69NdW/W+s3ERbl+jhwEPg1vSONq+mtp+4GHuquT13jjH9C78//+4A93eXKGcz5B8A3upx7gb/v5l8EfA14mN6fupvW+nlflPky4PZZzNnl+WZ3uX/h92bWnvcu03bg7u65/0/glFnLSe+N+v8FnrtobqYyDnvxk6KS1Ag/KSpJjbCgS1IjLOiS1AgLuiQ1woIuSY2woEtSIyzoktQIC7okNeL/AegArWrKTbp4AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(tran.state.cpu().squeeze(0).permute(1, 2, 0).numpy() + 0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "614"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(memory)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}