Q-learning-cart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep $Q$-learning\n",
    "\n",
    "In this notebook, we'll build a neural network that can learn to play games through reinforcement learning. More specifically, we'll use $Q$-learning to train an agent to play a game called [Cart-Pole](https://gym.openai.com/envs/CartPole-v0). In this game, a freely swinging pole is attached to a cart. The cart can move to the left and right, and the goal is to keep the pole upright as long as possible.\n",
    "\n",
    "![Cart-Pole](assets/cart-pole.jpg)\n",
    "\n",
    "We can simulate this game using [OpenAI Gym](https://github.com/openai/gym). First, let's check out how OpenAI Gym works. Then, we'll get into training an agent to play the Cart-Pole game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "Number of possible actions: 2\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "import numpy as np\n",
    "\n",
    "# Create the Cart-Pole game environment\n",
    "env = gym.make('CartPole-v1')\n",
    "\n",
    "# Number of possible actions\n",
    "print('Number of possible actions:', env.action_space.n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid token (<ipython-input-2-8ffee8a5a70e>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-2-8ffee8a5a70e>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    [2018-01-22 23:10:02,350] Making new env: CartPole-v1\u001b[0m\n\u001b[0m           ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid token\n"
     ]
    }
   ],
   "source": [
    "[2018-01-22 23:10:02,350] Making new env: CartPole-v1\n",
    "\n",
    "\n",
    "Number of possible actions: 2\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We interact with the simulation through `env`.  You can see how many actions are possible from `env.action_space.n`, and to get a random action you can use `env.action_space.sample()`.  Passing in an action as an integer to `env.step` will generate the next step in the simulation.  This is general to all Gym games. \n",
    "\n",
    "In the Cart-Pole game, there are two possible actions, moving the cart left or right. So there are two actions we can take, encoded as 0 and 1.\n",
    "\n",
    "Run the code below to interact with the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "actions = [] # actions that the agent selects\n",
    "rewards = [] # obtained rewards\n",
    "state = env.reset()\n",
    "\n",
    "while True:\n",
    "    action = env.action_space.sample()  # choose a random action\n",
    "    state, reward, done, _ = env.step(action) \n",
    "    rewards.append(reward)\n",
    "    actions.append(action)\n",
    "    if done:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can look at the actions and rewards:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actions: [0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0]\n",
      "Rewards: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]\n"
     ]
    }
   ],
   "source": [
    "print('Actions:', actions)\n",
    "print('Rewards:', rewards)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "Actions: [0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0]\n",
    "Rewards: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The game resets after the pole has fallen past a certain angle. For each step while the game is running, it returns a reward of 1.0. The longer the game runs, the more reward we get. Then, our network's goal is to maximize the reward by keeping the pole vertical. It will do this by moving the cart to the left and the right.\n",
    "\n",
    "## $Q$-Network\n",
    "\n",
    "To keep track of the action values, we'll use a neural network that accepts a state $s$ as input.  The output will be $Q$-values for each available action $a$ (i.e., the output is **all** action values $Q(s,a)$ _corresponding to the input state $s$_).\n",
    "\n",
    "<img src=\"assets/q-network.png\" width=550px>\n",
    "\n",
    "For this Cart-Pole game, the state has four values: the position and velocity of the cart, and the position and velocity of the pole.  Thus, the neural network has **four inputs**, one for each value in the state, and **two outputs**, one for each possible action. \n",
    "\n",
    "As explored in the lesson, to get the training target, we'll first use the context provided by the state $s$ to choose an action $a$, then simulate the game using that action. This will get us the next state, $s'$, and the reward $r$. With that, we can calculate $\\hat{Q}(s,a) = r + \\gamma \\max_{a'}{Q(s', a')}$.  Then we update the weights by minimizing $(\\hat{Q}(s,a) - Q(s,a))^2$. \n",
    "\n",
    "Below is one implementation of the $Q$-network. It uses two fully connected layers with ReLU activations. Two seems to be good enough, three might be better. Feel free to try it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "class QNetwork:\n",
    "    def __init__(self, learning_rate=0.01, state_size=4, \n",
    "                 action_size=2, hidden_size=10, \n",
    "                 name='QNetwork'):\n",
    "        # state inputs to the Q-network\n",
    "        with tf.variable_scope(name):\n",
    "            self.inputs_ = tf.placeholder(tf.float32, [None, state_size], name='inputs')\n",
    "            \n",
    "            # One hot encode the actions to later choose the Q-value for the action\n",
    "            self.actions_ = tf.placeholder(tf.int32, [None], name='actions')\n",
    "            one_hot_actions = tf.one_hot(self.actions_, action_size)\n",
    "            \n",
    "            # Target Q values for training\n",
    "            self.targetQs_ = tf.placeholder(tf.float32, [None], name='target')\n",
    "            \n",
    "            # ReLU hidden layers\n",
    "            self.fc1 = tf.contrib.layers.fully_connected(self.inputs_, hidden_size)\n",
    "            self.fc2 = tf.contrib.layers.fully_connected(self.fc1, hidden_size)\n",
    "\n",
    "            # Linear output layer\n",
    "            self.output = tf.contrib.layers.fully_connected(self.fc2, action_size, \n",
    "                                                            activation_fn=None)\n",
    "            \n",
    "            ### Train with loss (targetQ - Q)^2\n",
    "            # output has length 2, for two actions. This next line chooses\n",
    "            # one value from output (per row) according to the one-hot encoded actions.\n",
    "            self.Q = tf.reduce_sum(tf.multiply(self.output, one_hot_actions), axis=1)\n",
    "            \n",
    "            self.loss = tf.reduce_mean(tf.square(self.targetQs_ - self.Q))\n",
    "            self.opt = tf.train.AdamOptimizer(learning_rate).minimize(self.loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experience replay\n",
    "\n",
    "Reinforcement learning algorithms can have stability issues due to correlations between states. To reduce correlations when training, we can store the agent's experiences and later draw a random mini-batch of those experiences to train on. \n",
    "\n",
    "Here, we'll create a `Memory` object that will store our experiences, our transitions $<s, a, r, s'>$. This memory will have a maximum capacity, so we can keep newer experiences in memory while getting rid of older experiences. Then, we'll sample a random mini-batch of transitions $<s, a, r, s'>$ and train on those.\n",
    "\n",
    "Below, I've implemented a `Memory` object. If you're unfamiliar with `deque`, this is a double-ended queue. You can think of it like a tube open on both sides. You can put objects in either side of the tube. But if it's full, adding anything more will push an object out the other side. This is a great data structure to use for the memory buffer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "\n",
    "class Memory():\n",
    "    def __init__(self, max_size=1000):\n",
    "        self.buffer = deque(maxlen=max_size)\n",
    "    \n",
    "    def add(self, experience):\n",
    "        self.buffer.append(experience)\n",
    "            \n",
    "    def sample(self, batch_size):\n",
    "        idx = np.random.choice(np.arange(len(self.buffer)), \n",
    "                               size=batch_size, \n",
    "                               replace=False)\n",
    "        return [self.buffer[ii] for ii in idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## $Q$-Learning training algorithm\n",
    "\n",
    "We will use the below algorithm to train the network.  For this game, the goal is to keep the pole upright for 195 frames. So we can start a new episode once meeting that goal. The game ends if the pole tilts over too far, or if the cart moves too far the left or right. When a game ends, we'll start a new episode. Now, to train the agent:\n",
    "\n",
    "* Initialize the memory $D$\n",
    "* Initialize the action-value network $Q$ with random weights\n",
    "* **For** episode $\\leftarrow 1$ **to** $M$ **do**\n",
    "  * Observe $s_0$\n",
    "  * **For** $t \\leftarrow 0$ **to** $T-1$ **do**\n",
    "     * With probability $\\epsilon$ select a random action $a_t$, otherwise select $a_t = \\mathrm{argmax}_a Q(s_t,a)$\n",
    "     * Execute action $a_t$ in simulator and observe reward $r_{t+1}$ and new state $s_{t+1}$\n",
    "     * Store transition $<s_t, a_t, r_{t+1}, s_{t+1}>$ in memory $D$\n",
    "     * Sample random mini-batch from $D$: $<s_j, a_j, r_j, s'_j>$\n",
    "     * Set $\\hat{Q}_j = r_j$ if the episode ends at $j+1$, otherwise set $\\hat{Q}_j = r_j + \\gamma \\max_{a'}{Q(s'_j, a')}$\n",
    "     * Make a gradient descent step with loss $(\\hat{Q}_j - Q(s_j, a_j))^2$\n",
    "  * **endfor**\n",
    "* **endfor**\n",
    "\n",
    "You are welcome (and encouraged!) to take the time to extend this code to implement some of the improvements that we discussed in the lesson, to include fixed $Q$ targets, double DQNs, prioritized replay, and/or dueling networks.\n",
    "\n",
    "## Hyperparameters\n",
    "\n",
    "One of the more difficult aspects of reinforcement learning is the large number of hyperparameters. Not only are we tuning the network, but we're tuning the simulation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_episodes = 1000          # max number of episodes to learn from\n",
    "max_steps = 200                # max steps in an episode\n",
    "gamma = 0.99                   # future reward discount\n",
    "\n",
    "# Exploration parameters\n",
    "explore_start = 1.0            # exploration probability at start\n",
    "explore_stop = 0.01            # minimum exploration probability \n",
    "decay_rate = 0.0001            # exponential decay rate for exploration prob\n",
    "\n",
    "# Network parameters\n",
    "hidden_size = 64               # number of units in each Q-network hidden layer\n",
    "learning_rate = 0.0001         # Q-network learning rate\n",
    "\n",
    "# Memory parameters\n",
    "memory_size = 10000            # memory capacity\n",
    "batch_size = 20                # experience mini-batch size\n",
    "pretrain_length = batch_size   # number experiences to pretrain the memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "mainQN = QNetwork(name='main', hidden_size=hidden_size, learning_rate=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populate the experience memory\n",
    "\n",
    "Here we re-initialize the simulation and pre-populate the memory. The agent is taking random actions and storing the transitions in memory. This will help the agent with exploring the game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the simulation\n",
    "env.reset()\n",
    "# Take one random step to get the pole and cart moving\n",
    "state, reward, done, _ = env.step(env.action_space.sample())\n",
    "\n",
    "memory = Memory(max_size=memory_size)\n",
    "\n",
    "# Make a bunch of random actions and store the experiences\n",
    "for ii in range(pretrain_length):\n",
    "\n",
    "    # Make a random action\n",
    "    action = env.action_space.sample()\n",
    "    next_state, reward, done, _ = env.step(action)\n",
    "\n",
    "    if done:\n",
    "        # The simulation fails so no next state\n",
    "        next_state = np.zeros(state.shape)\n",
    "        # Add experience to memory\n",
    "        memory.add((state, action, reward, next_state))\n",
    "        \n",
    "        # Start new episode\n",
    "        env.reset()\n",
    "        # Take one random step to get the pole and cart moving\n",
    "        state, reward, done, _ = env.step(env.action_space.sample())\n",
    "    else:\n",
    "        # Add experience to memory\n",
    "        memory.add((state, action, reward, next_state))\n",
    "        state = next_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "Below we'll train our agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 1 Total reward: 13.0 Training loss: 1.0960 Explore P: 0.9987\n",
      "Episode: 2 Total reward: 10.0 Training loss: 1.1484 Explore P: 0.9977\n",
      "Episode: 3 Total reward: 19.0 Training loss: 1.1362 Explore P: 0.9959\n",
      "Episode: 4 Total reward: 10.0 Training loss: 1.1183 Explore P: 0.9949\n",
      "Episode: 5 Total reward: 22.0 Training loss: 1.1334 Explore P: 0.9927\n",
      "Episode: 6 Total reward: 13.0 Training loss: 1.1836 Explore P: 0.9914\n",
      "Episode: 7 Total reward: 17.0 Training loss: 1.1760 Explore P: 0.9898\n",
      "Episode: 8 Total reward: 23.0 Training loss: 1.1724 Explore P: 0.9875\n",
      "Episode: 9 Total reward: 22.0 Training loss: 1.1357 Explore P: 0.9854\n",
      "Episode: 10 Total reward: 14.0 Training loss: 1.0746 Explore P: 0.9840\n",
      "Episode: 11 Total reward: 14.0 Training loss: 1.0780 Explore P: 0.9826\n",
      "Episode: 12 Total reward: 14.0 Training loss: 1.1397 Explore P: 0.9813\n",
      "Episode: 13 Total reward: 16.0 Training loss: 1.2129 Explore P: 0.9797\n",
      "Episode: 14 Total reward: 15.0 Training loss: 1.1247 Explore P: 0.9783\n",
      "Episode: 15 Total reward: 27.0 Training loss: 1.3099 Explore P: 0.9757\n",
      "Episode: 16 Total reward: 26.0 Training loss: 1.4050 Explore P: 0.9731\n",
      "Episode: 17 Total reward: 24.0 Training loss: 1.1676 Explore P: 0.9708\n",
      "Episode: 18 Total reward: 18.0 Training loss: 1.4283 Explore P: 0.9691\n",
      "Episode: 19 Total reward: 28.0 Training loss: 1.5620 Explore P: 0.9664\n",
      "Episode: 20 Total reward: 11.0 Training loss: 1.6338 Explore P: 0.9654\n",
      "Episode: 21 Total reward: 11.0 Training loss: 1.4370 Explore P: 0.9643\n",
      "Episode: 22 Total reward: 14.0 Training loss: 1.9597 Explore P: 0.9630\n",
      "Episode: 23 Total reward: 19.0 Training loss: 1.4672 Explore P: 0.9612\n",
      "Episode: 24 Total reward: 14.0 Training loss: 1.3749 Explore P: 0.9599\n",
      "Episode: 25 Total reward: 56.0 Training loss: 1.9932 Explore P: 0.9545\n",
      "Episode: 26 Total reward: 28.0 Training loss: 4.1964 Explore P: 0.9519\n",
      "Episode: 27 Total reward: 31.0 Training loss: 1.8510 Explore P: 0.9490\n",
      "Episode: 28 Total reward: 19.0 Training loss: 2.7173 Explore P: 0.9472\n",
      "Episode: 29 Total reward: 15.0 Training loss: 3.1718 Explore P: 0.9458\n",
      "Episode: 30 Total reward: 16.0 Training loss: 2.1332 Explore P: 0.9443\n",
      "Episode: 31 Total reward: 23.0 Training loss: 2.5775 Explore P: 0.9422\n",
      "Episode: 32 Total reward: 8.0 Training loss: 2.8523 Explore P: 0.9414\n",
      "Episode: 33 Total reward: 21.0 Training loss: 5.8166 Explore P: 0.9395\n",
      "Episode: 34 Total reward: 17.0 Training loss: 2.6288 Explore P: 0.9379\n",
      "Episode: 35 Total reward: 25.0 Training loss: 6.0082 Explore P: 0.9356\n",
      "Episode: 36 Total reward: 12.0 Training loss: 5.0686 Explore P: 0.9345\n",
      "Episode: 37 Total reward: 19.0 Training loss: 4.0529 Explore P: 0.9327\n",
      "Episode: 38 Total reward: 19.0 Training loss: 4.1906 Explore P: 0.9309\n",
      "Episode: 39 Total reward: 15.0 Training loss: 5.0404 Explore P: 0.9296\n",
      "Episode: 40 Total reward: 15.0 Training loss: 5.8610 Explore P: 0.9282\n",
      "Episode: 41 Total reward: 14.0 Training loss: 4.6934 Explore P: 0.9269\n",
      "Episode: 42 Total reward: 15.0 Training loss: 5.5647 Explore P: 0.9255\n",
      "Episode: 43 Total reward: 11.0 Training loss: 6.0198 Explore P: 0.9245\n",
      "Episode: 44 Total reward: 13.0 Training loss: 13.7475 Explore P: 0.9233\n",
      "Episode: 45 Total reward: 28.0 Training loss: 16.1322 Explore P: 0.9208\n",
      "Episode: 46 Total reward: 22.0 Training loss: 13.8135 Explore P: 0.9188\n",
      "Episode: 47 Total reward: 41.0 Training loss: 14.1117 Explore P: 0.9151\n",
      "Episode: 48 Total reward: 14.0 Training loss: 7.6372 Explore P: 0.9138\n",
      "Episode: 49 Total reward: 20.0 Training loss: 19.6946 Explore P: 0.9120\n",
      "Episode: 50 Total reward: 16.0 Training loss: 3.8142 Explore P: 0.9105\n",
      "Episode: 51 Total reward: 12.0 Training loss: 17.8192 Explore P: 0.9095\n",
      "Episode: 52 Total reward: 15.0 Training loss: 14.5007 Explore P: 0.9081\n",
      "Episode: 53 Total reward: 18.0 Training loss: 16.9326 Explore P: 0.9065\n",
      "Episode: 54 Total reward: 61.0 Training loss: 47.6859 Explore P: 0.9011\n",
      "Episode: 55 Total reward: 9.0 Training loss: 11.9922 Explore P: 0.9003\n",
      "Episode: 56 Total reward: 11.0 Training loss: 4.2069 Explore P: 0.8993\n",
      "Episode: 57 Total reward: 27.0 Training loss: 4.2972 Explore P: 0.8969\n",
      "Episode: 58 Total reward: 21.0 Training loss: 4.9177 Explore P: 0.8950\n",
      "Episode: 59 Total reward: 15.0 Training loss: 27.5186 Explore P: 0.8937\n",
      "Episode: 60 Total reward: 18.0 Training loss: 26.3578 Explore P: 0.8921\n",
      "Episode: 61 Total reward: 13.0 Training loss: 20.4137 Explore P: 0.8910\n",
      "Episode: 62 Total reward: 13.0 Training loss: 3.8543 Explore P: 0.8898\n",
      "Episode: 63 Total reward: 25.0 Training loss: 19.4254 Explore P: 0.8876\n",
      "Episode: 64 Total reward: 13.0 Training loss: 24.7878 Explore P: 0.8865\n",
      "Episode: 65 Total reward: 46.0 Training loss: 12.8137 Explore P: 0.8824\n",
      "Episode: 66 Total reward: 12.0 Training loss: 4.1244 Explore P: 0.8814\n",
      "Episode: 67 Total reward: 24.0 Training loss: 22.1255 Explore P: 0.8793\n",
      "Episode: 68 Total reward: 13.0 Training loss: 18.1204 Explore P: 0.8782\n",
      "Episode: 69 Total reward: 15.0 Training loss: 4.4118 Explore P: 0.8769\n",
      "Episode: 70 Total reward: 24.0 Training loss: 3.7348 Explore P: 0.8748\n",
      "Episode: 71 Total reward: 10.0 Training loss: 29.6902 Explore P: 0.8739\n",
      "Episode: 72 Total reward: 25.0 Training loss: 48.9264 Explore P: 0.8718\n",
      "Episode: 73 Total reward: 11.0 Training loss: 4.1234 Explore P: 0.8708\n",
      "Episode: 74 Total reward: 21.0 Training loss: 15.3027 Explore P: 0.8690\n",
      "Episode: 75 Total reward: 11.0 Training loss: 9.0154 Explore P: 0.8681\n",
      "Episode: 76 Total reward: 13.0 Training loss: 3.3321 Explore P: 0.8670\n",
      "Episode: 77 Total reward: 45.0 Training loss: 4.6520 Explore P: 0.8631\n",
      "Episode: 78 Total reward: 21.0 Training loss: 84.4783 Explore P: 0.8613\n",
      "Episode: 79 Total reward: 20.0 Training loss: 23.8805 Explore P: 0.8596\n",
      "Episode: 80 Total reward: 20.0 Training loss: 35.5340 Explore P: 0.8579\n",
      "Episode: 81 Total reward: 10.0 Training loss: 13.4873 Explore P: 0.8571\n",
      "Episode: 82 Total reward: 12.0 Training loss: 32.0735 Explore P: 0.8561\n",
      "Episode: 83 Total reward: 11.0 Training loss: 3.3168 Explore P: 0.8551\n",
      "Episode: 84 Total reward: 19.0 Training loss: 44.7226 Explore P: 0.8535\n",
      "Episode: 85 Total reward: 14.0 Training loss: 20.1650 Explore P: 0.8524\n",
      "Episode: 86 Total reward: 10.0 Training loss: 27.4170 Explore P: 0.8515\n",
      "Episode: 87 Total reward: 8.0 Training loss: 12.9582 Explore P: 0.8508\n",
      "Episode: 88 Total reward: 25.0 Training loss: 4.4822 Explore P: 0.8487\n",
      "Episode: 89 Total reward: 13.0 Training loss: 37.8192 Explore P: 0.8477\n",
      "Episode: 90 Total reward: 11.0 Training loss: 4.1791 Explore P: 0.8467\n",
      "Episode: 91 Total reward: 19.0 Training loss: 4.8683 Explore P: 0.8451\n",
      "Episode: 92 Total reward: 37.0 Training loss: 69.1402 Explore P: 0.8421\n",
      "Episode: 93 Total reward: 44.0 Training loss: 19.4360 Explore P: 0.8384\n",
      "Episode: 94 Total reward: 9.0 Training loss: 19.3532 Explore P: 0.8377\n",
      "Episode: 95 Total reward: 11.0 Training loss: 65.4179 Explore P: 0.8368\n",
      "Episode: 96 Total reward: 58.0 Training loss: 21.2540 Explore P: 0.8320\n",
      "Episode: 97 Total reward: 10.0 Training loss: 4.2062 Explore P: 0.8311\n",
      "Episode: 98 Total reward: 18.0 Training loss: 4.0838 Explore P: 0.8297\n",
      "Episode: 99 Total reward: 14.0 Training loss: 42.2685 Explore P: 0.8285\n",
      "Episode: 100 Total reward: 22.0 Training loss: 25.5561 Explore P: 0.8267\n",
      "Episode: 101 Total reward: 12.0 Training loss: 3.2452 Explore P: 0.8257\n",
      "Episode: 102 Total reward: 14.0 Training loss: 93.5608 Explore P: 0.8246\n",
      "Episode: 103 Total reward: 18.0 Training loss: 25.5043 Explore P: 0.8231\n",
      "Episode: 104 Total reward: 22.0 Training loss: 21.0315 Explore P: 0.8214\n",
      "Episode: 105 Total reward: 18.0 Training loss: 4.2790 Explore P: 0.8199\n",
      "Episode: 106 Total reward: 28.0 Training loss: 28.8197 Explore P: 0.8176\n",
      "Episode: 107 Total reward: 13.0 Training loss: 4.5439 Explore P: 0.8166\n",
      "Episode: 108 Total reward: 15.0 Training loss: 3.0664 Explore P: 0.8154\n",
      "Episode: 109 Total reward: 15.0 Training loss: 46.4665 Explore P: 0.8142\n",
      "Episode: 110 Total reward: 14.0 Training loss: 4.7580 Explore P: 0.8130\n",
      "Episode: 111 Total reward: 21.0 Training loss: 23.4666 Explore P: 0.8114\n",
      "Episode: 112 Total reward: 19.0 Training loss: 123.7870 Explore P: 0.8098\n",
      "Episode: 113 Total reward: 17.0 Training loss: 51.5347 Explore P: 0.8085\n",
      "Episode: 114 Total reward: 19.0 Training loss: 4.0637 Explore P: 0.8070\n",
      "Episode: 115 Total reward: 17.0 Training loss: 32.6818 Explore P: 0.8056\n",
      "Episode: 116 Total reward: 16.0 Training loss: 29.7045 Explore P: 0.8043\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 117 Total reward: 24.0 Training loss: 5.0225 Explore P: 0.8024\n",
      "Episode: 118 Total reward: 33.0 Training loss: 4.7034 Explore P: 0.7998\n",
      "Episode: 119 Total reward: 17.0 Training loss: 38.3731 Explore P: 0.7985\n",
      "Episode: 120 Total reward: 19.0 Training loss: 23.1203 Explore P: 0.7970\n",
      "Episode: 121 Total reward: 15.0 Training loss: 29.9689 Explore P: 0.7958\n",
      "Episode: 122 Total reward: 9.0 Training loss: 34.1407 Explore P: 0.7951\n",
      "Episode: 123 Total reward: 26.0 Training loss: 27.8251 Explore P: 0.7931\n",
      "Episode: 124 Total reward: 16.0 Training loss: 63.0652 Explore P: 0.7918\n",
      "Episode: 125 Total reward: 22.0 Training loss: 26.5004 Explore P: 0.7901\n",
      "Episode: 126 Total reward: 9.0 Training loss: 24.2451 Explore P: 0.7894\n",
      "Episode: 127 Total reward: 10.0 Training loss: 24.3480 Explore P: 0.7886\n",
      "Episode: 128 Total reward: 13.0 Training loss: 38.1763 Explore P: 0.7876\n",
      "Episode: 129 Total reward: 14.0 Training loss: 46.9423 Explore P: 0.7865\n",
      "Episode: 130 Total reward: 14.0 Training loss: 38.6913 Explore P: 0.7854\n",
      "Episode: 131 Total reward: 13.0 Training loss: 36.6368 Explore P: 0.7844\n",
      "Episode: 132 Total reward: 9.0 Training loss: 32.4554 Explore P: 0.7837\n",
      "Episode: 133 Total reward: 13.0 Training loss: 4.6220 Explore P: 0.7827\n",
      "Episode: 134 Total reward: 52.0 Training loss: 4.1074 Explore P: 0.7787\n",
      "Episode: 135 Total reward: 24.0 Training loss: 49.5419 Explore P: 0.7769\n",
      "Episode: 136 Total reward: 16.0 Training loss: 4.0263 Explore P: 0.7756\n",
      "Episode: 137 Total reward: 12.0 Training loss: 4.3180 Explore P: 0.7747\n",
      "Episode: 138 Total reward: 12.0 Training loss: 4.1720 Explore P: 0.7738\n",
      "Episode: 139 Total reward: 11.0 Training loss: 38.6964 Explore P: 0.7730\n",
      "Episode: 140 Total reward: 12.0 Training loss: 31.1678 Explore P: 0.7720\n",
      "Episode: 141 Total reward: 11.0 Training loss: 3.8333 Explore P: 0.7712\n",
      "Episode: 142 Total reward: 9.0 Training loss: 4.4864 Explore P: 0.7705\n",
      "Episode: 143 Total reward: 16.0 Training loss: 88.8037 Explore P: 0.7693\n",
      "Episode: 144 Total reward: 19.0 Training loss: 3.6769 Explore P: 0.7679\n",
      "Episode: 145 Total reward: 43.0 Training loss: 90.3444 Explore P: 0.7646\n",
      "Episode: 146 Total reward: 21.0 Training loss: 3.4290 Explore P: 0.7630\n",
      "Episode: 147 Total reward: 9.0 Training loss: 50.7586 Explore P: 0.7624\n",
      "Episode: 148 Total reward: 17.0 Training loss: 56.1365 Explore P: 0.7611\n",
      "Episode: 149 Total reward: 14.0 Training loss: 36.3331 Explore P: 0.7600\n",
      "Episode: 150 Total reward: 16.0 Training loss: 4.5942 Explore P: 0.7588\n",
      "Episode: 151 Total reward: 9.0 Training loss: 3.9164 Explore P: 0.7582\n",
      "Episode: 152 Total reward: 26.0 Training loss: 30.2060 Explore P: 0.7562\n",
      "Episode: 153 Total reward: 14.0 Training loss: 3.3890 Explore P: 0.7552\n",
      "Episode: 154 Total reward: 17.0 Training loss: 3.5427 Explore P: 0.7539\n",
      "Episode: 155 Total reward: 13.0 Training loss: 86.2039 Explore P: 0.7529\n",
      "Episode: 156 Total reward: 29.0 Training loss: 61.7202 Explore P: 0.7508\n",
      "Episode: 157 Total reward: 12.0 Training loss: 35.7999 Explore P: 0.7499\n",
      "Episode: 158 Total reward: 16.0 Training loss: 34.1518 Explore P: 0.7487\n",
      "Episode: 159 Total reward: 18.0 Training loss: 36.5945 Explore P: 0.7474\n",
      "Episode: 160 Total reward: 41.0 Training loss: 29.7727 Explore P: 0.7444\n",
      "Episode: 161 Total reward: 17.0 Training loss: 45.5252 Explore P: 0.7431\n",
      "Episode: 162 Total reward: 13.0 Training loss: 3.5415 Explore P: 0.7422\n",
      "Episode: 163 Total reward: 24.0 Training loss: 66.4774 Explore P: 0.7404\n",
      "Episode: 164 Total reward: 14.0 Training loss: 2.5397 Explore P: 0.7394\n",
      "Episode: 165 Total reward: 11.0 Training loss: 132.0429 Explore P: 0.7386\n",
      "Episode: 166 Total reward: 19.0 Training loss: 47.2274 Explore P: 0.7372\n",
      "Episode: 167 Total reward: 9.0 Training loss: 36.8114 Explore P: 0.7365\n",
      "Episode: 168 Total reward: 29.0 Training loss: 56.1881 Explore P: 0.7344\n",
      "Episode: 169 Total reward: 12.0 Training loss: 2.3406 Explore P: 0.7336\n",
      "Episode: 170 Total reward: 10.0 Training loss: 69.3148 Explore P: 0.7329\n",
      "Episode: 171 Total reward: 11.0 Training loss: 29.3554 Explore P: 0.7321\n",
      "Episode: 172 Total reward: 9.0 Training loss: 63.6889 Explore P: 0.7314\n",
      "Episode: 173 Total reward: 10.0 Training loss: 57.2189 Explore P: 0.7307\n",
      "Episode: 174 Total reward: 41.0 Training loss: 29.8276 Explore P: 0.7277\n",
      "Episode: 175 Total reward: 24.0 Training loss: 72.9758 Explore P: 0.7260\n",
      "Episode: 176 Total reward: 12.0 Training loss: 59.0537 Explore P: 0.7252\n",
      "Episode: 177 Total reward: 29.0 Training loss: 83.0169 Explore P: 0.7231\n",
      "Episode: 178 Total reward: 18.0 Training loss: 27.3644 Explore P: 0.7218\n",
      "Episode: 179 Total reward: 19.0 Training loss: 57.7709 Explore P: 0.7205\n",
      "Episode: 180 Total reward: 18.0 Training loss: 3.0030 Explore P: 0.7192\n",
      "Episode: 181 Total reward: 10.0 Training loss: 2.2222 Explore P: 0.7185\n",
      "Episode: 182 Total reward: 11.0 Training loss: 82.8918 Explore P: 0.7177\n",
      "Episode: 183 Total reward: 11.0 Training loss: 76.5383 Explore P: 0.7169\n",
      "Episode: 184 Total reward: 15.0 Training loss: 2.7245 Explore P: 0.7159\n",
      "Episode: 185 Total reward: 13.0 Training loss: 40.7014 Explore P: 0.7149\n",
      "Episode: 186 Total reward: 12.0 Training loss: 69.7366 Explore P: 0.7141\n",
      "Episode: 187 Total reward: 10.0 Training loss: 52.3750 Explore P: 0.7134\n",
      "Episode: 188 Total reward: 28.0 Training loss: 110.2845 Explore P: 0.7114\n",
      "Episode: 189 Total reward: 18.0 Training loss: 29.1088 Explore P: 0.7102\n",
      "Episode: 190 Total reward: 42.0 Training loss: 2.1344 Explore P: 0.7072\n",
      "Episode: 191 Total reward: 12.0 Training loss: 53.6193 Explore P: 0.7064\n",
      "Episode: 192 Total reward: 13.0 Training loss: 2.4270 Explore P: 0.7055\n",
      "Episode: 193 Total reward: 14.0 Training loss: 51.1887 Explore P: 0.7045\n",
      "Episode: 194 Total reward: 12.0 Training loss: 54.5204 Explore P: 0.7037\n",
      "Episode: 195 Total reward: 12.0 Training loss: 32.1240 Explore P: 0.7028\n",
      "Episode: 196 Total reward: 12.0 Training loss: 1.4300 Explore P: 0.7020\n",
      "Episode: 197 Total reward: 17.0 Training loss: 1.8192 Explore P: 0.7008\n",
      "Episode: 198 Total reward: 15.0 Training loss: 2.1710 Explore P: 0.6998\n",
      "Episode: 199 Total reward: 17.0 Training loss: 1.7514 Explore P: 0.6986\n",
      "Episode: 200 Total reward: 20.0 Training loss: 29.3724 Explore P: 0.6973\n",
      "Episode: 201 Total reward: 31.0 Training loss: 33.6968 Explore P: 0.6951\n",
      "Episode: 202 Total reward: 12.0 Training loss: 1.7992 Explore P: 0.6943\n",
      "Episode: 203 Total reward: 15.0 Training loss: 127.2472 Explore P: 0.6933\n",
      "Episode: 204 Total reward: 33.0 Training loss: 28.5705 Explore P: 0.6910\n",
      "Episode: 205 Total reward: 11.0 Training loss: 23.7881 Explore P: 0.6903\n",
      "Episode: 206 Total reward: 27.0 Training loss: 1.8178 Explore P: 0.6884\n",
      "Episode: 207 Total reward: 19.0 Training loss: 1.2683 Explore P: 0.6872\n",
      "Episode: 208 Total reward: 25.0 Training loss: 1.5332 Explore P: 0.6855\n",
      "Episode: 209 Total reward: 11.0 Training loss: 1.0635 Explore P: 0.6847\n",
      "Episode: 210 Total reward: 12.0 Training loss: 52.4046 Explore P: 0.6839\n",
      "Episode: 211 Total reward: 17.0 Training loss: 30.9194 Explore P: 0.6828\n",
      "Episode: 212 Total reward: 11.0 Training loss: 1.4108 Explore P: 0.6820\n",
      "Episode: 213 Total reward: 16.0 Training loss: 24.7332 Explore P: 0.6810\n",
      "Episode: 214 Total reward: 12.0 Training loss: 1.0642 Explore P: 0.6802\n",
      "Episode: 215 Total reward: 13.0 Training loss: 21.9864 Explore P: 0.6793\n",
      "Episode: 216 Total reward: 12.0 Training loss: 0.8321 Explore P: 0.6785\n",
      "Episode: 217 Total reward: 15.0 Training loss: 81.8265 Explore P: 0.6775\n",
      "Episode: 218 Total reward: 13.0 Training loss: 0.9995 Explore P: 0.6766\n",
      "Episode: 219 Total reward: 10.0 Training loss: 1.3028 Explore P: 0.6759\n",
      "Episode: 220 Total reward: 10.0 Training loss: 33.7145 Explore P: 0.6753\n",
      "Episode: 221 Total reward: 7.0 Training loss: 35.5002 Explore P: 0.6748\n",
      "Episode: 222 Total reward: 10.0 Training loss: 17.1983 Explore P: 0.6741\n",
      "Episode: 223 Total reward: 9.0 Training loss: 41.0101 Explore P: 0.6736\n",
      "Episode: 224 Total reward: 10.0 Training loss: 91.0303 Explore P: 0.6729\n",
      "Episode: 225 Total reward: 10.0 Training loss: 0.8234 Explore P: 0.6722\n",
      "Episode: 226 Total reward: 15.0 Training loss: 0.9398 Explore P: 0.6712\n",
      "Episode: 227 Total reward: 12.0 Training loss: 26.5403 Explore P: 0.6704\n",
      "Episode: 228 Total reward: 18.0 Training loss: 26.4872 Explore P: 0.6693\n",
      "Episode: 229 Total reward: 14.0 Training loss: 42.2652 Explore P: 0.6683\n",
      "Episode: 230 Total reward: 21.0 Training loss: 35.8741 Explore P: 0.6669\n",
      "Episode: 231 Total reward: 16.0 Training loss: 24.7978 Explore P: 0.6659\n",
      "Episode: 232 Total reward: 9.0 Training loss: 17.1995 Explore P: 0.6653\n",
      "Episode: 233 Total reward: 10.0 Training loss: 16.5016 Explore P: 0.6647\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 234 Total reward: 24.0 Training loss: 31.9128 Explore P: 0.6631\n",
      "Episode: 235 Total reward: 14.0 Training loss: 14.0776 Explore P: 0.6622\n",
      "Episode: 236 Total reward: 14.0 Training loss: 27.6612 Explore P: 0.6613\n",
      "Episode: 237 Total reward: 13.0 Training loss: 0.6793 Explore P: 0.6604\n",
      "Episode: 238 Total reward: 11.0 Training loss: 28.8780 Explore P: 0.6597\n",
      "Episode: 239 Total reward: 15.0 Training loss: 46.4523 Explore P: 0.6587\n",
      "Episode: 240 Total reward: 14.0 Training loss: 0.7998 Explore P: 0.6578\n",
      "Episode: 241 Total reward: 23.0 Training loss: 26.6585 Explore P: 0.6563\n",
      "Episode: 242 Total reward: 11.0 Training loss: 35.3933 Explore P: 0.6556\n",
      "Episode: 243 Total reward: 22.0 Training loss: 37.5583 Explore P: 0.6542\n",
      "Episode: 244 Total reward: 11.0 Training loss: 48.4978 Explore P: 0.6535\n",
      "Episode: 245 Total reward: 13.0 Training loss: 25.0013 Explore P: 0.6527\n",
      "Episode: 246 Total reward: 16.0 Training loss: 0.8786 Explore P: 0.6516\n",
      "Episode: 247 Total reward: 11.0 Training loss: 27.6776 Explore P: 0.6509\n",
      "Episode: 248 Total reward: 16.0 Training loss: 27.3166 Explore P: 0.6499\n",
      "Episode: 249 Total reward: 15.0 Training loss: 1.3053 Explore P: 0.6489\n",
      "Episode: 250 Total reward: 18.0 Training loss: 53.4022 Explore P: 0.6478\n",
      "Episode: 251 Total reward: 12.0 Training loss: 23.8891 Explore P: 0.6470\n",
      "Episode: 252 Total reward: 10.0 Training loss: 46.8577 Explore P: 0.6464\n",
      "Episode: 253 Total reward: 14.0 Training loss: 13.8798 Explore P: 0.6455\n",
      "Episode: 254 Total reward: 13.0 Training loss: 1.3621 Explore P: 0.6447\n",
      "Episode: 255 Total reward: 32.0 Training loss: 12.1727 Explore P: 0.6426\n",
      "Episode: 256 Total reward: 11.0 Training loss: 11.8100 Explore P: 0.6419\n",
      "Episode: 257 Total reward: 34.0 Training loss: 1.3131 Explore P: 0.6398\n",
      "Episode: 258 Total reward: 15.0 Training loss: 14.1743 Explore P: 0.6389\n",
      "Episode: 259 Total reward: 9.0 Training loss: 37.7591 Explore P: 0.6383\n",
      "Episode: 260 Total reward: 25.0 Training loss: 1.1215 Explore P: 0.6367\n",
      "Episode: 261 Total reward: 25.0 Training loss: 10.0628 Explore P: 0.6352\n",
      "Episode: 262 Total reward: 17.0 Training loss: 1.2067 Explore P: 0.6341\n",
      "Episode: 263 Total reward: 11.0 Training loss: 1.2013 Explore P: 0.6334\n",
      "Episode: 264 Total reward: 21.0 Training loss: 1.4707 Explore P: 0.6321\n",
      "Episode: 265 Total reward: 10.0 Training loss: 70.4545 Explore P: 0.6315\n",
      "Episode: 266 Total reward: 25.0 Training loss: 11.5925 Explore P: 0.6299\n",
      "Episode: 267 Total reward: 24.0 Training loss: 30.3114 Explore P: 0.6284\n",
      "Episode: 268 Total reward: 15.0 Training loss: 1.5541 Explore P: 0.6275\n",
      "Episode: 269 Total reward: 18.0 Training loss: 10.3095 Explore P: 0.6264\n",
      "Episode: 270 Total reward: 12.0 Training loss: 1.0347 Explore P: 0.6257\n",
      "Episode: 271 Total reward: 26.0 Training loss: 29.0271 Explore P: 0.6241\n",
      "Episode: 272 Total reward: 35.0 Training loss: 18.1427 Explore P: 0.6219\n",
      "Episode: 273 Total reward: 22.0 Training loss: 34.5534 Explore P: 0.6206\n",
      "Episode: 274 Total reward: 24.0 Training loss: 1.2172 Explore P: 0.6191\n",
      "Episode: 275 Total reward: 22.0 Training loss: 9.3271 Explore P: 0.6178\n",
      "Episode: 276 Total reward: 14.0 Training loss: 23.9619 Explore P: 0.6169\n",
      "Episode: 277 Total reward: 14.0 Training loss: 0.9831 Explore P: 0.6161\n",
      "Episode: 278 Total reward: 12.0 Training loss: 0.9329 Explore P: 0.6153\n",
      "Episode: 279 Total reward: 9.0 Training loss: 23.6738 Explore P: 0.6148\n",
      "Episode: 280 Total reward: 18.0 Training loss: 24.3325 Explore P: 0.6137\n",
      "Episode: 281 Total reward: 14.0 Training loss: 1.0705 Explore P: 0.6129\n",
      "Episode: 282 Total reward: 43.0 Training loss: 18.5781 Explore P: 0.6103\n",
      "Episode: 283 Total reward: 42.0 Training loss: 16.4694 Explore P: 0.6078\n",
      "Episode: 284 Total reward: 20.0 Training loss: 0.9995 Explore P: 0.6066\n",
      "Episode: 285 Total reward: 20.0 Training loss: 7.8132 Explore P: 0.6054\n",
      "Episode: 286 Total reward: 17.0 Training loss: 15.5785 Explore P: 0.6044\n",
      "Episode: 287 Total reward: 54.0 Training loss: 1.3019 Explore P: 0.6012\n",
      "Episode: 288 Total reward: 28.0 Training loss: 50.3323 Explore P: 0.5995\n",
      "Episode: 289 Total reward: 18.0 Training loss: 43.7078 Explore P: 0.5985\n",
      "Episode: 290 Total reward: 19.0 Training loss: 1.4246 Explore P: 0.5973\n",
      "Episode: 291 Total reward: 34.0 Training loss: 1.3903 Explore P: 0.5953\n",
      "Episode: 292 Total reward: 15.0 Training loss: 0.9468 Explore P: 0.5945\n",
      "Episode: 293 Total reward: 56.0 Training loss: 12.1529 Explore P: 0.5912\n",
      "Episode: 294 Total reward: 48.0 Training loss: 22.9638 Explore P: 0.5884\n",
      "Episode: 295 Total reward: 24.0 Training loss: 10.1694 Explore P: 0.5870\n",
      "Episode: 296 Total reward: 26.0 Training loss: 9.8811 Explore P: 0.5855\n",
      "Episode: 297 Total reward: 26.0 Training loss: 2.0056 Explore P: 0.5840\n",
      "Episode: 298 Total reward: 25.0 Training loss: 7.8074 Explore P: 0.5826\n",
      "Episode: 299 Total reward: 16.0 Training loss: 17.0333 Explore P: 0.5817\n",
      "Episode: 300 Total reward: 11.0 Training loss: 10.5923 Explore P: 0.5811\n",
      "Episode: 301 Total reward: 34.0 Training loss: 1.0912 Explore P: 0.5791\n",
      "Episode: 302 Total reward: 14.0 Training loss: 8.0851 Explore P: 0.5783\n",
      "Episode: 303 Total reward: 29.0 Training loss: 0.8833 Explore P: 0.5767\n",
      "Episode: 304 Total reward: 29.0 Training loss: 1.2124 Explore P: 0.5750\n",
      "Episode: 305 Total reward: 17.0 Training loss: 45.7937 Explore P: 0.5741\n",
      "Episode: 306 Total reward: 12.0 Training loss: 11.4873 Explore P: 0.5734\n",
      "Episode: 307 Total reward: 23.0 Training loss: 13.8395 Explore P: 0.5721\n",
      "Episode: 308 Total reward: 22.0 Training loss: 16.7859 Explore P: 0.5709\n",
      "Episode: 309 Total reward: 43.0 Training loss: 24.2431 Explore P: 0.5685\n",
      "Episode: 310 Total reward: 25.0 Training loss: 17.4126 Explore P: 0.5671\n",
      "Episode: 311 Total reward: 23.0 Training loss: 1.5465 Explore P: 0.5658\n",
      "Episode: 312 Total reward: 47.0 Training loss: 1.4821 Explore P: 0.5632\n",
      "Episode: 313 Total reward: 21.0 Training loss: 1.3806 Explore P: 0.5620\n",
      "Episode: 314 Total reward: 32.0 Training loss: 7.1798 Explore P: 0.5603\n",
      "Episode: 315 Total reward: 21.0 Training loss: 1.1271 Explore P: 0.5591\n",
      "Episode: 316 Total reward: 17.0 Training loss: 15.8676 Explore P: 0.5582\n",
      "Episode: 317 Total reward: 20.0 Training loss: 30.1013 Explore P: 0.5571\n",
      "Episode: 318 Total reward: 24.0 Training loss: 8.0410 Explore P: 0.5558\n",
      "Episode: 319 Total reward: 19.0 Training loss: 29.2743 Explore P: 0.5547\n",
      "Episode: 320 Total reward: 18.0 Training loss: 1.3756 Explore P: 0.5538\n",
      "Episode: 321 Total reward: 50.0 Training loss: 0.9021 Explore P: 0.5510\n",
      "Episode: 322 Total reward: 23.0 Training loss: 7.6239 Explore P: 0.5498\n",
      "Episode: 323 Total reward: 50.0 Training loss: 14.6531 Explore P: 0.5471\n",
      "Episode: 324 Total reward: 44.0 Training loss: 2.1833 Explore P: 0.5448\n",
      "Episode: 325 Total reward: 17.0 Training loss: 1.1862 Explore P: 0.5438\n",
      "Episode: 326 Total reward: 20.0 Training loss: 13.2931 Explore P: 0.5428\n",
      "Episode: 327 Total reward: 49.0 Training loss: 17.3362 Explore P: 0.5402\n",
      "Episode: 328 Total reward: 14.0 Training loss: 2.0656 Explore P: 0.5394\n",
      "Episode: 329 Total reward: 23.0 Training loss: 44.3887 Explore P: 0.5382\n",
      "Episode: 330 Total reward: 45.0 Training loss: 26.6387 Explore P: 0.5358\n",
      "Episode: 331 Total reward: 81.0 Training loss: 0.9919 Explore P: 0.5316\n",
      "Episode: 332 Total reward: 40.0 Training loss: 1.8336 Explore P: 0.5295\n",
      "Episode: 333 Total reward: 43.0 Training loss: 15.6535 Explore P: 0.5273\n",
      "Episode: 334 Total reward: 56.0 Training loss: 1.7205 Explore P: 0.5244\n",
      "Episode: 335 Total reward: 52.0 Training loss: 1.8746 Explore P: 0.5217\n",
      "Episode: 336 Total reward: 17.0 Training loss: 1.9312 Explore P: 0.5209\n",
      "Episode: 337 Total reward: 22.0 Training loss: 17.2208 Explore P: 0.5197\n",
      "Episode: 338 Total reward: 20.0 Training loss: 38.5156 Explore P: 0.5187\n",
      "Episode: 339 Total reward: 29.0 Training loss: 2.3776 Explore P: 0.5173\n",
      "Episode: 340 Total reward: 19.0 Training loss: 1.0643 Explore P: 0.5163\n",
      "Episode: 341 Total reward: 25.0 Training loss: 23.0733 Explore P: 0.5150\n",
      "Episode: 342 Total reward: 29.0 Training loss: 12.4641 Explore P: 0.5136\n",
      "Episode: 343 Total reward: 19.0 Training loss: 1.4444 Explore P: 0.5126\n",
      "Episode: 344 Total reward: 30.0 Training loss: 34.8144 Explore P: 0.5111\n",
      "Episode: 345 Total reward: 80.0 Training loss: 51.7877 Explore P: 0.5071\n",
      "Episode: 346 Total reward: 44.0 Training loss: 29.3816 Explore P: 0.5049\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 347 Total reward: 57.0 Training loss: 40.6607 Explore P: 0.5021\n",
      "Episode: 348 Total reward: 84.0 Training loss: 18.9699 Explore P: 0.4980\n",
      "Episode: 349 Total reward: 31.0 Training loss: 2.5029 Explore P: 0.4965\n",
      "Episode: 350 Total reward: 40.0 Training loss: 1.9772 Explore P: 0.4945\n",
      "Episode: 351 Total reward: 77.0 Training loss: 14.1912 Explore P: 0.4908\n",
      "Episode: 352 Total reward: 40.0 Training loss: 24.9580 Explore P: 0.4889\n",
      "Episode: 353 Total reward: 35.0 Training loss: 32.6117 Explore P: 0.4872\n",
      "Episode: 354 Total reward: 26.0 Training loss: 1.7302 Explore P: 0.4860\n",
      "Episode: 355 Total reward: 25.0 Training loss: 2.4793 Explore P: 0.4848\n",
      "Episode: 356 Total reward: 39.0 Training loss: 25.7854 Explore P: 0.4830\n",
      "Episode: 357 Total reward: 26.0 Training loss: 11.1949 Explore P: 0.4817\n",
      "Episode: 358 Total reward: 16.0 Training loss: 1.7300 Explore P: 0.4810\n",
      "Episode: 359 Total reward: 10.0 Training loss: 1.7820 Explore P: 0.4805\n",
      "Episode: 360 Total reward: 95.0 Training loss: 29.3214 Explore P: 0.4761\n",
      "Episode: 361 Total reward: 24.0 Training loss: 66.7296 Explore P: 0.4749\n",
      "Episode: 362 Total reward: 36.0 Training loss: 35.6992 Explore P: 0.4733\n",
      "Episode: 363 Total reward: 79.0 Training loss: 1.7740 Explore P: 0.4696\n",
      "Episode: 364 Total reward: 75.0 Training loss: 46.8118 Explore P: 0.4662\n",
      "Episode: 365 Total reward: 83.0 Training loss: 18.2028 Explore P: 0.4624\n",
      "Episode: 366 Total reward: 31.0 Training loss: 8.3466 Explore P: 0.4610\n",
      "Episode: 367 Total reward: 26.0 Training loss: 23.1914 Explore P: 0.4598\n",
      "Episode: 368 Total reward: 73.0 Training loss: 2.6778 Explore P: 0.4566\n",
      "Episode: 369 Total reward: 27.0 Training loss: 2.6173 Explore P: 0.4554\n",
      "Episode: 370 Total reward: 47.0 Training loss: 56.5659 Explore P: 0.4533\n",
      "Episode: 371 Total reward: 42.0 Training loss: 3.2772 Explore P: 0.4514\n",
      "Episode: 372 Total reward: 21.0 Training loss: 37.2196 Explore P: 0.4505\n",
      "Episode: 373 Total reward: 26.0 Training loss: 16.9468 Explore P: 0.4494\n",
      "Episode: 374 Total reward: 30.0 Training loss: 3.4321 Explore P: 0.4480\n",
      "Episode: 375 Total reward: 44.0 Training loss: 2.1444 Explore P: 0.4461\n",
      "Episode: 376 Total reward: 43.0 Training loss: 29.0843 Explore P: 0.4442\n",
      "Episode: 377 Total reward: 69.0 Training loss: 46.2626 Explore P: 0.4413\n",
      "Episode: 378 Total reward: 27.0 Training loss: 2.0803 Explore P: 0.4401\n",
      "Episode: 379 Total reward: 35.0 Training loss: 74.5531 Explore P: 0.4386\n",
      "Episode: 380 Total reward: 28.0 Training loss: 23.5461 Explore P: 0.4374\n",
      "Episode: 381 Total reward: 63.0 Training loss: 17.4666 Explore P: 0.4347\n",
      "Episode: 382 Total reward: 46.0 Training loss: 20.9947 Explore P: 0.4328\n",
      "Episode: 383 Total reward: 36.0 Training loss: 1.7109 Explore P: 0.4312\n",
      "Episode: 384 Total reward: 65.0 Training loss: 44.1965 Explore P: 0.4285\n",
      "Episode: 385 Total reward: 19.0 Training loss: 25.2732 Explore P: 0.4277\n",
      "Episode: 386 Total reward: 62.0 Training loss: 75.9300 Explore P: 0.4251\n",
      "Episode: 387 Total reward: 39.0 Training loss: 6.5812 Explore P: 0.4235\n",
      "Episode: 388 Total reward: 57.0 Training loss: 1.4866 Explore P: 0.4212\n",
      "Episode: 389 Total reward: 44.0 Training loss: 2.3985 Explore P: 0.4194\n",
      "Episode: 390 Total reward: 44.0 Training loss: 104.7464 Explore P: 0.4176\n",
      "Episode: 391 Total reward: 27.0 Training loss: 9.1364 Explore P: 0.4165\n",
      "Episode: 392 Total reward: 62.0 Training loss: 47.6782 Explore P: 0.4140\n",
      "Episode: 393 Total reward: 68.0 Training loss: 2.8250 Explore P: 0.4112\n",
      "Episode: 394 Total reward: 36.0 Training loss: 6.8501 Explore P: 0.4098\n",
      "Episode: 395 Total reward: 52.0 Training loss: 38.0316 Explore P: 0.4077\n",
      "Episode: 396 Total reward: 87.0 Training loss: 1.4410 Explore P: 0.4043\n",
      "Episode: 397 Total reward: 99.0 Training loss: 4.1027 Explore P: 0.4004\n",
      "Episode: 398 Total reward: 80.0 Training loss: 13.0682 Explore P: 0.3973\n",
      "Episode: 399 Total reward: 70.0 Training loss: 39.4603 Explore P: 0.3946\n",
      "Episode: 400 Total reward: 64.0 Training loss: 3.2577 Explore P: 0.3921\n",
      "Episode: 401 Total reward: 64.0 Training loss: 2.2495 Explore P: 0.3897\n",
      "Episode: 402 Total reward: 89.0 Training loss: 11.7478 Explore P: 0.3863\n",
      "Episode: 403 Total reward: 59.0 Training loss: 3.2996 Explore P: 0.3841\n",
      "Episode: 404 Total reward: 38.0 Training loss: 25.4712 Explore P: 0.3827\n",
      "Episode: 405 Total reward: 30.0 Training loss: 44.7235 Explore P: 0.3816\n",
      "Episode: 406 Total reward: 35.0 Training loss: 2.6771 Explore P: 0.3803\n",
      "Episode: 407 Total reward: 66.0 Training loss: 2.4541 Explore P: 0.3778\n",
      "Episode: 408 Total reward: 41.0 Training loss: 38.5956 Explore P: 0.3763\n",
      "Episode: 409 Total reward: 127.0 Training loss: 3.0720 Explore P: 0.3717\n",
      "Episode: 410 Total reward: 83.0 Training loss: 21.6136 Explore P: 0.3687\n",
      "Episode: 411 Total reward: 65.0 Training loss: 31.6120 Explore P: 0.3664\n",
      "Episode: 412 Total reward: 53.0 Training loss: 2.3404 Explore P: 0.3645\n",
      "Episode: 413 Total reward: 64.0 Training loss: 3.1837 Explore P: 0.3622\n",
      "Episode: 414 Total reward: 80.0 Training loss: 1.4671 Explore P: 0.3594\n",
      "Episode: 415 Total reward: 86.0 Training loss: 122.1985 Explore P: 0.3564\n",
      "Episode: 416 Total reward: 86.0 Training loss: 54.2995 Explore P: 0.3535\n",
      "Episode: 417 Total reward: 100.0 Training loss: 57.3928 Explore P: 0.3501\n",
      "Episode: 418 Total reward: 44.0 Training loss: 15.5730 Explore P: 0.3486\n",
      "Episode: 419 Total reward: 47.0 Training loss: 2.6868 Explore P: 0.3470\n",
      "Episode: 420 Total reward: 123.0 Training loss: 27.4810 Explore P: 0.3429\n",
      "Episode: 421 Total reward: 36.0 Training loss: 4.4117 Explore P: 0.3417\n",
      "Episode: 422 Total reward: 88.0 Training loss: 65.3904 Explore P: 0.3388\n",
      "Episode: 423 Total reward: 127.0 Training loss: 2.0106 Explore P: 0.3346\n",
      "Episode: 424 Total reward: 56.0 Training loss: 49.5697 Explore P: 0.3328\n",
      "Episode: 425 Total reward: 88.0 Training loss: 2.4439 Explore P: 0.3300\n",
      "Episode: 426 Total reward: 129.0 Training loss: 36.0941 Explore P: 0.3259\n",
      "Episode: 427 Total reward: 89.0 Training loss: 2.8014 Explore P: 0.3231\n",
      "Episode: 428 Total reward: 73.0 Training loss: 29.0126 Explore P: 0.3208\n",
      "Episode: 429 Total reward: 97.0 Training loss: 3.5113 Explore P: 0.3178\n",
      "Episode: 430 Total reward: 75.0 Training loss: 62.8506 Explore P: 0.3155\n",
      "Episode: 431 Total reward: 80.0 Training loss: 100.0288 Explore P: 0.3131\n",
      "Episode: 432 Total reward: 59.0 Training loss: 2.5530 Explore P: 0.3113\n",
      "Episode: 433 Total reward: 82.0 Training loss: 1.4287 Explore P: 0.3088\n",
      "Episode: 434 Total reward: 97.0 Training loss: 3.2711 Explore P: 0.3059\n",
      "Episode: 435 Total reward: 49.0 Training loss: 2.0234 Explore P: 0.3045\n",
      "Episode: 436 Total reward: 130.0 Training loss: 2.6891 Explore P: 0.3007\n",
      "Episode: 437 Total reward: 85.0 Training loss: 2.1779 Explore P: 0.2982\n",
      "Episode: 438 Total reward: 91.0 Training loss: 2.5727 Explore P: 0.2956\n",
      "Episode: 439 Total reward: 98.0 Training loss: 4.5218 Explore P: 0.2928\n",
      "Episode: 440 Total reward: 78.0 Training loss: 2.6873 Explore P: 0.2906\n",
      "Episode: 441 Total reward: 95.0 Training loss: 1.7932 Explore P: 0.2880\n",
      "Episode: 442 Total reward: 80.0 Training loss: 173.4149 Explore P: 0.2858\n",
      "Episode: 443 Total reward: 77.0 Training loss: 2.4415 Explore P: 0.2836\n",
      "Episode: 444 Total reward: 135.0 Training loss: 22.2671 Explore P: 0.2800\n",
      "Episode: 445 Total reward: 113.0 Training loss: 2.0083 Explore P: 0.2769\n",
      "Episode: 446 Total reward: 70.0 Training loss: 1.0182 Explore P: 0.2751\n",
      "Episode: 447 Total reward: 111.0 Training loss: 62.0891 Explore P: 0.2721\n",
      "Episode: 448 Total reward: 105.0 Training loss: 3.8252 Explore P: 0.2694\n",
      "Episode: 449 Total reward: 91.0 Training loss: 2.7220 Explore P: 0.2671\n",
      "Episode: 450 Total reward: 136.0 Training loss: 3.1870 Explore P: 0.2636\n",
      "Episode: 451 Total reward: 75.0 Training loss: 61.5388 Explore P: 0.2617\n",
      "Episode: 452 Total reward: 89.0 Training loss: 1.8269 Explore P: 0.2595\n",
      "Episode: 453 Total reward: 66.0 Training loss: 65.9961 Explore P: 0.2578\n",
      "Episode: 454 Total reward: 38.0 Training loss: 55.6376 Explore P: 0.2569\n",
      "Episode: 455 Total reward: 95.0 Training loss: 33.9072 Explore P: 0.2545\n",
      "Episode: 456 Total reward: 68.0 Training loss: 3.0418 Explore P: 0.2529\n",
      "Episode: 457 Total reward: 97.0 Training loss: 232.7718 Explore P: 0.2505\n",
      "Episode: 458 Total reward: 62.0 Training loss: 94.4890 Explore P: 0.2491\n",
      "Episode: 459 Total reward: 75.0 Training loss: 67.6398 Explore P: 0.2473\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 460 Total reward: 54.0 Training loss: 2.4020 Explore P: 0.2460\n",
      "Episode: 461 Total reward: 64.0 Training loss: 3.7898 Explore P: 0.2445\n",
      "Episode: 462 Total reward: 51.0 Training loss: 3.1079 Explore P: 0.2433\n",
      "Episode: 463 Total reward: 86.0 Training loss: 2.7400 Explore P: 0.2413\n",
      "Episode: 464 Total reward: 119.0 Training loss: 2.1309 Explore P: 0.2386\n",
      "Episode: 465 Total reward: 166.0 Training loss: 3.3319 Explore P: 0.2348\n",
      "Episode: 466 Total reward: 106.0 Training loss: 4.9856 Explore P: 0.2324\n",
      "Episode: 467 Total reward: 77.0 Training loss: 1.9245 Explore P: 0.2307\n",
      "Episode: 468 Total reward: 87.0 Training loss: 2.8819 Explore P: 0.2288\n",
      "Episode: 469 Total reward: 93.0 Training loss: 115.1336 Explore P: 0.2268\n",
      "Episode: 470 Total reward: 110.0 Training loss: 2.4218 Explore P: 0.2244\n",
      "Episode: 471 Total reward: 63.0 Training loss: 2.4104 Explore P: 0.2231\n",
      "Episode: 472 Total reward: 116.0 Training loss: 1.6972 Explore P: 0.2206\n",
      "Episode: 473 Total reward: 93.0 Training loss: 2.7194 Explore P: 0.2187\n",
      "Episode: 474 Total reward: 88.0 Training loss: 181.0243 Explore P: 0.2168\n",
      "Episode: 475 Total reward: 43.0 Training loss: 2.6114 Explore P: 0.2159\n",
      "Episode: 476 Total reward: 62.0 Training loss: 3.1506 Explore P: 0.2147\n",
      "Episode: 477 Total reward: 51.0 Training loss: 98.9462 Explore P: 0.2136\n",
      "Episode: 478 Total reward: 139.0 Training loss: 178.7103 Explore P: 0.2108\n",
      "Episode: 479 Total reward: 31.0 Training loss: 3.0323 Explore P: 0.2102\n",
      "Episode: 480 Total reward: 45.0 Training loss: 177.3511 Explore P: 0.2093\n",
      "Episode: 481 Total reward: 63.0 Training loss: 2.0078 Explore P: 0.2080\n",
      "Episode: 482 Total reward: 84.0 Training loss: 1.1378 Explore P: 0.2064\n",
      "Episode: 483 Total reward: 74.0 Training loss: 2.4390 Explore P: 0.2049\n",
      "Episode: 484 Total reward: 86.0 Training loss: 1.5058 Explore P: 0.2033\n",
      "Episode: 485 Total reward: 68.0 Training loss: 2.3605 Explore P: 0.2020\n",
      "Episode: 486 Total reward: 70.0 Training loss: 2.0112 Explore P: 0.2006\n",
      "Episode: 487 Total reward: 42.0 Training loss: 118.8953 Explore P: 0.1998\n",
      "Episode: 488 Total reward: 54.0 Training loss: 2.8021 Explore P: 0.1988\n",
      "Episode: 489 Total reward: 50.0 Training loss: 2.8262 Explore P: 0.1979\n",
      "Episode: 490 Total reward: 138.0 Training loss: 3.1894 Explore P: 0.1953\n",
      "Episode: 491 Total reward: 77.0 Training loss: 1.7313 Explore P: 0.1939\n",
      "Episode: 492 Total reward: 35.0 Training loss: 2.7266 Explore P: 0.1932\n",
      "Episode: 493 Total reward: 117.0 Training loss: 2.5380 Explore P: 0.1911\n",
      "Episode: 494 Total reward: 116.0 Training loss: 1.8366 Explore P: 0.1890\n",
      "Episode: 495 Total reward: 161.0 Training loss: 2.1503 Explore P: 0.1861\n",
      "Episode: 496 Total reward: 97.0 Training loss: 3.2452 Explore P: 0.1844\n",
      "Episode: 497 Total reward: 148.0 Training loss: 63.6046 Explore P: 0.1819\n",
      "Episode: 498 Total reward: 70.0 Training loss: 2.3286 Explore P: 0.1807\n",
      "Episode: 499 Total reward: 110.0 Training loss: 1.8361 Explore P: 0.1788\n",
      "Episode: 500 Total reward: 148.0 Training loss: 148.0216 Explore P: 0.1763\n",
      "Episode: 502 Total reward: 32.0 Training loss: 1.5183 Explore P: 0.1725\n",
      "Episode: 503 Total reward: 113.0 Training loss: 1.2306 Explore P: 0.1707\n",
      "Episode: 504 Total reward: 195.0 Training loss: 1.5320 Explore P: 0.1676\n",
      "Episode: 505 Total reward: 121.0 Training loss: 2.6438 Explore P: 0.1657\n",
      "Episode: 506 Total reward: 103.0 Training loss: 285.5203 Explore P: 0.1641\n",
      "Episode: 507 Total reward: 125.0 Training loss: 1.1981 Explore P: 0.1622\n",
      "Episode: 508 Total reward: 72.0 Training loss: 1.1294 Explore P: 0.1611\n",
      "Episode: 509 Total reward: 160.0 Training loss: 1.6380 Explore P: 0.1587\n",
      "Episode: 510 Total reward: 125.0 Training loss: 1.3682 Explore P: 0.1568\n",
      "Episode: 511 Total reward: 109.0 Training loss: 1.1967 Explore P: 0.1553\n",
      "Episode: 512 Total reward: 109.0 Training loss: 1.1061 Explore P: 0.1537\n",
      "Episode: 513 Total reward: 157.0 Training loss: 38.0583 Explore P: 0.1514\n",
      "Episode: 514 Total reward: 143.0 Training loss: 1.5130 Explore P: 0.1494\n",
      "Episode: 515 Total reward: 97.0 Training loss: 2.3913 Explore P: 0.1481\n",
      "Episode: 517 Total reward: 52.0 Training loss: 1.3170 Explore P: 0.1447\n",
      "Episode: 518 Total reward: 83.0 Training loss: 1.2360 Explore P: 0.1435\n",
      "Episode: 519 Total reward: 135.0 Training loss: 1.6801 Explore P: 0.1417\n",
      "Episode: 520 Total reward: 178.0 Training loss: 1.1831 Explore P: 0.1394\n",
      "Episode: 521 Total reward: 134.0 Training loss: 7.8459 Explore P: 0.1377\n",
      "Episode: 522 Total reward: 123.0 Training loss: 0.3600 Explore P: 0.1361\n",
      "Episode: 523 Total reward: 161.0 Training loss: 1.2784 Explore P: 0.1341\n",
      "Episode: 524 Total reward: 145.0 Training loss: 0.6477 Explore P: 0.1323\n",
      "Episode: 525 Total reward: 154.0 Training loss: 349.7535 Explore P: 0.1305\n",
      "Episode: 526 Total reward: 167.0 Training loss: 0.9317 Explore P: 0.1285\n",
      "Episode: 527 Total reward: 127.0 Training loss: 5.2240 Explore P: 0.1270\n",
      "Episode: 528 Total reward: 180.0 Training loss: 0.9893 Explore P: 0.1249\n",
      "Episode: 529 Total reward: 144.0 Training loss: 0.6201 Explore P: 0.1233\n",
      "Episode: 530 Total reward: 137.0 Training loss: 28.1292 Explore P: 0.1217\n",
      "Episode: 532 Total reward: 25.0 Training loss: 0.5240 Explore P: 0.1192\n",
      "Episode: 533 Total reward: 190.0 Training loss: 0.8941 Explore P: 0.1172\n",
      "Episode: 535 Total reward: 63.0 Training loss: 0.9496 Explore P: 0.1144\n",
      "Episode: 536 Total reward: 84.0 Training loss: 47.3792 Explore P: 0.1135\n",
      "Episode: 537 Total reward: 123.0 Training loss: 0.8376 Explore P: 0.1122\n",
      "Episode: 538 Total reward: 134.0 Training loss: 0.4357 Explore P: 0.1109\n",
      "Episode: 539 Total reward: 149.0 Training loss: 0.3444 Explore P: 0.1094\n",
      "Episode: 540 Total reward: 87.0 Training loss: 0.6561 Explore P: 0.1085\n",
      "Episode: 541 Total reward: 96.0 Training loss: 3.8864 Explore P: 0.1076\n",
      "Episode: 543 Total reward: 24.0 Training loss: 2.0825 Explore P: 0.1054\n",
      "Episode: 544 Total reward: 88.0 Training loss: 1.1877 Explore P: 0.1046\n",
      "Episode: 545 Total reward: 62.0 Training loss: 0.4394 Explore P: 0.1040\n",
      "Episode: 546 Total reward: 199.0 Training loss: 0.8731 Explore P: 0.1022\n",
      "Episode: 547 Total reward: 120.0 Training loss: 0.6702 Explore P: 0.1011\n",
      "Episode: 548 Total reward: 106.0 Training loss: 0.5269 Explore P: 0.1001\n",
      "Episode: 549 Total reward: 88.0 Training loss: 0.2269 Explore P: 0.0993\n",
      "Episode: 550 Total reward: 95.0 Training loss: 59.4228 Explore P: 0.0985\n",
      "Episode: 551 Total reward: 92.0 Training loss: 0.5334 Explore P: 0.0977\n",
      "Episode: 554 Total reward: 99.0 Training loss: 0.6213 Explore P: 0.0934\n",
      "Episode: 555 Total reward: 183.0 Training loss: 0.2973 Explore P: 0.0919\n",
      "Episode: 556 Total reward: 109.0 Training loss: 0.5257 Explore P: 0.0910\n",
      "Episode: 557 Total reward: 139.0 Training loss: 0.3421 Explore P: 0.0899\n",
      "Episode: 558 Total reward: 118.0 Training loss: 0.7035 Explore P: 0.0889\n",
      "Episode: 560 Total reward: 143.0 Training loss: 0.9353 Explore P: 0.0863\n",
      "Episode: 562 Total reward: 151.0 Training loss: 4.4309 Explore P: 0.0836\n",
      "Episode: 565 Total reward: 99.0 Training loss: 0.4003 Explore P: 0.0801\n",
      "Episode: 566 Total reward: 192.0 Training loss: 0.2337 Explore P: 0.0787\n",
      "Episode: 569 Total reward: 99.0 Training loss: 0.3450 Explore P: 0.0754\n",
      "Episode: 572 Total reward: 99.0 Training loss: 0.4234 Explore P: 0.0722\n",
      "Episode: 574 Total reward: 10.0 Training loss: 0.4896 Explore P: 0.0709\n",
      "Episode: 576 Total reward: 166.0 Training loss: 0.3105 Explore P: 0.0687\n",
      "Episode: 579 Total reward: 9.0 Training loss: 13.9345 Explore P: 0.0664\n",
      "Episode: 581 Total reward: 181.0 Training loss: 0.1991 Explore P: 0.0643\n",
      "Episode: 583 Total reward: 84.0 Training loss: 0.4127 Explore P: 0.0627\n",
      "Episode: 585 Total reward: 52.0 Training loss: 0.1005 Explore P: 0.0614\n",
      "Episode: 587 Total reward: 137.0 Training loss: 0.0954 Explore P: 0.0597\n",
      "Episode: 589 Total reward: 102.0 Training loss: 0.2805 Explore P: 0.0582\n",
      "Episode: 591 Total reward: 42.0 Training loss: 0.4588 Explore P: 0.0571\n",
      "Episode: 593 Total reward: 29.0 Training loss: 29.6201 Explore P: 0.0560\n",
      "Episode: 595 Total reward: 115.0 Training loss: 0.1574 Explore P: 0.0546\n",
      "Episode: 597 Total reward: 53.0 Training loss: 0.3664 Explore P: 0.0535\n",
      "Episode: 599 Total reward: 10.0 Training loss: 0.2055 Explore P: 0.0526\n",
      "Episode: 601 Total reward: 3.0 Training loss: 0.3882 Explore P: 0.0517\n",
      "Episode: 603 Total reward: 66.0 Training loss: 0.3645 Explore P: 0.0506\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 605 Total reward: 24.0 Training loss: 0.7409 Explore P: 0.0497\n",
      "Episode: 607 Total reward: 64.0 Training loss: 0.3226 Explore P: 0.0487\n",
      "Episode: 609 Total reward: 9.0 Training loss: 0.5132 Explore P: 0.0479\n",
      "Episode: 611 Total reward: 26.0 Training loss: 0.4269 Explore P: 0.0470\n",
      "Episode: 613 Total reward: 79.0 Training loss: 446.5233 Explore P: 0.0460\n",
      "Episode: 614 Total reward: 195.0 Training loss: 402.3101 Explore P: 0.0453\n",
      "Episode: 616 Total reward: 17.0 Training loss: 0.2305 Explore P: 0.0446\n",
      "Episode: 617 Total reward: 179.0 Training loss: 0.4002 Explore P: 0.0440\n",
      "Episode: 618 Total reward: 185.0 Training loss: 0.2889 Explore P: 0.0433\n",
      "Episode: 619 Total reward: 185.0 Training loss: 0.4600 Explore P: 0.0427\n",
      "Episode: 621 Total reward: 6.0 Training loss: 0.4603 Explore P: 0.0421\n",
      "Episode: 622 Total reward: 179.0 Training loss: 0.3776 Explore P: 0.0415\n",
      "Episode: 623 Total reward: 184.0 Training loss: 0.2859 Explore P: 0.0409\n",
      "Episode: 625 Total reward: 6.0 Training loss: 0.5376 Explore P: 0.0403\n",
      "Episode: 626 Total reward: 187.0 Training loss: 0.3381 Explore P: 0.0397\n",
      "Episode: 627 Total reward: 176.0 Training loss: 0.3080 Explore P: 0.0392\n",
      "Episode: 629 Total reward: 19.0 Training loss: 0.4282 Explore P: 0.0386\n",
      "Episode: 630 Total reward: 189.0 Training loss: 0.2735 Explore P: 0.0380\n",
      "Episode: 632 Total reward: 6.0 Training loss: 0.4989 Explore P: 0.0375\n",
      "Episode: 634 Total reward: 13.0 Training loss: 0.1206 Explore P: 0.0369\n",
      "Episode: 635 Total reward: 178.0 Training loss: 0.2167 Explore P: 0.0364\n",
      "Episode: 636 Total reward: 178.0 Training loss: 0.3173 Explore P: 0.0359\n",
      "Episode: 637 Total reward: 185.0 Training loss: 0.2245 Explore P: 0.0355\n",
      "Episode: 638 Total reward: 175.0 Training loss: 0.3003 Explore P: 0.0350\n",
      "Episode: 639 Total reward: 172.0 Training loss: 0.1736 Explore P: 0.0346\n",
      "Episode: 640 Total reward: 177.0 Training loss: 0.2507 Explore P: 0.0342\n",
      "Episode: 641 Total reward: 188.0 Training loss: 0.1456 Explore P: 0.0337\n",
      "Episode: 642 Total reward: 169.0 Training loss: 0.3158 Explore P: 0.0333\n",
      "Episode: 643 Total reward: 153.0 Training loss: 0.1272 Explore P: 0.0330\n",
      "Episode: 644 Total reward: 191.0 Training loss: 0.1626 Explore P: 0.0325\n",
      "Episode: 645 Total reward: 159.0 Training loss: 0.1805 Explore P: 0.0322\n",
      "Episode: 646 Total reward: 168.0 Training loss: 0.1435 Explore P: 0.0318\n",
      "Episode: 647 Total reward: 173.0 Training loss: 18.6162 Explore P: 0.0314\n",
      "Episode: 648 Total reward: 195.0 Training loss: 29.3435 Explore P: 0.0310\n",
      "Episode: 650 Total reward: 3.0 Training loss: 0.2718 Explore P: 0.0306\n",
      "Episode: 651 Total reward: 179.0 Training loss: 0.3316 Explore P: 0.0302\n",
      "Episode: 652 Total reward: 180.0 Training loss: 0.6960 Explore P: 0.0299\n",
      "Episode: 653 Total reward: 173.0 Training loss: 20.1247 Explore P: 0.0295\n",
      "Episode: 655 Total reward: 13.0 Training loss: 0.2289 Explore P: 0.0291\n",
      "Episode: 656 Total reward: 178.0 Training loss: 0.2241 Explore P: 0.0288\n",
      "Episode: 658 Total reward: 51.0 Training loss: 0.1614 Explore P: 0.0283\n",
      "Episode: 659 Total reward: 187.0 Training loss: 0.0681 Explore P: 0.0280\n",
      "Episode: 661 Total reward: 21.0 Training loss: 6.5281 Explore P: 0.0276\n",
      "Episode: 663 Total reward: 55.0 Training loss: 0.1016 Explore P: 0.0271\n",
      "Episode: 665 Total reward: 113.0 Training loss: 0.1167 Explore P: 0.0266\n",
      "Episode: 667 Total reward: 68.0 Training loss: 0.1226 Explore P: 0.0262\n",
      "Episode: 669 Total reward: 121.0 Training loss: 0.0789 Explore P: 0.0257\n",
      "Episode: 671 Total reward: 21.0 Training loss: 0.0551 Explore P: 0.0253\n",
      "Episode: 674 Total reward: 99.0 Training loss: 0.1897 Explore P: 0.0246\n",
      "Episode: 676 Total reward: 15.0 Training loss: 0.0910 Explore P: 0.0243\n",
      "Episode: 678 Total reward: 165.0 Training loss: 0.0468 Explore P: 0.0238\n",
      "Episode: 681 Total reward: 49.0 Training loss: 0.0974 Explore P: 0.0231\n",
      "Episode: 684 Total reward: 99.0 Training loss: 0.0504 Explore P: 0.0225\n",
      "Episode: 686 Total reward: 21.0 Training loss: 0.0469 Explore P: 0.0222\n",
      "Episode: 689 Total reward: 99.0 Training loss: 0.1158 Explore P: 0.0216\n",
      "Episode: 692 Total reward: 99.0 Training loss: 0.0372 Explore P: 0.0211\n",
      "Episode: 695 Total reward: 83.0 Training loss: 0.0402 Explore P: 0.0206\n",
      "Episode: 697 Total reward: 199.0 Training loss: 0.0346 Explore P: 0.0201\n",
      "Episode: 699 Total reward: 138.0 Training loss: 0.0400 Explore P: 0.0198\n",
      "Episode: 702 Total reward: 99.0 Training loss: 0.0592 Explore P: 0.0193\n",
      "Episode: 703 Total reward: 162.0 Training loss: 0.0309 Explore P: 0.0192\n",
      "Episode: 704 Total reward: 179.0 Training loss: 0.0915 Explore P: 0.0190\n",
      "Episode: 707 Total reward: 99.0 Training loss: 0.0549 Explore P: 0.0186\n",
      "Episode: 708 Total reward: 200.0 Training loss: 0.1448 Explore P: 0.0184\n",
      "Episode: 711 Total reward: 16.0 Training loss: 0.0432 Explore P: 0.0181\n",
      "Episode: 714 Total reward: 54.0 Training loss: 3.1118 Explore P: 0.0177\n",
      "Episode: 716 Total reward: 66.0 Training loss: 0.0327 Explore P: 0.0175\n",
      "Episode: 718 Total reward: 23.0 Training loss: 0.0515 Explore P: 0.0173\n",
      "Episode: 720 Total reward: 99.0 Training loss: 0.1048 Explore P: 0.0171\n",
      "Episode: 722 Total reward: 156.0 Training loss: 0.0958 Explore P: 0.0169\n",
      "Episode: 724 Total reward: 85.0 Training loss: 0.0605 Explore P: 0.0167\n",
      "Episode: 726 Total reward: 39.0 Training loss: 0.0542 Explore P: 0.0165\n",
      "Episode: 728 Total reward: 54.0 Training loss: 60.1856 Explore P: 0.0164\n",
      "Episode: 730 Total reward: 88.0 Training loss: 0.0358 Explore P: 0.0162\n",
      "Episode: 732 Total reward: 32.0 Training loss: 0.0237 Explore P: 0.0160\n",
      "Episode: 734 Total reward: 33.0 Training loss: 0.0549 Explore P: 0.0159\n",
      "Episode: 737 Total reward: 99.0 Training loss: 0.0222 Explore P: 0.0156\n",
      "Episode: 740 Total reward: 99.0 Training loss: 0.0465 Explore P: 0.0153\n",
      "Episode: 743 Total reward: 50.0 Training loss: 0.0360 Explore P: 0.0151\n",
      "Episode: 745 Total reward: 194.0 Training loss: 0.0699 Explore P: 0.0149\n",
      "Episode: 748 Total reward: 58.0 Training loss: 0.0529 Explore P: 0.0147\n",
      "Episode: 751 Total reward: 99.0 Training loss: 0.0580 Explore P: 0.0145\n",
      "Episode: 754 Total reward: 14.0 Training loss: 0.0534 Explore P: 0.0143\n",
      "Episode: 757 Total reward: 48.0 Training loss: 0.0428 Explore P: 0.0141\n",
      "Episode: 760 Total reward: 19.0 Training loss: 0.0565 Explore P: 0.0139\n",
      "Episode: 763 Total reward: 69.0 Training loss: 0.0879 Explore P: 0.0137\n",
      "Episode: 765 Total reward: 157.0 Training loss: 0.0655 Explore P: 0.0136\n",
      "Episode: 768 Total reward: 35.0 Training loss: 0.0563 Explore P: 0.0135\n",
      "Episode: 771 Total reward: 53.0 Training loss: 0.0681 Explore P: 0.0133\n",
      "Episode: 774 Total reward: 99.0 Training loss: 0.0173 Explore P: 0.0131\n",
      "Episode: 777 Total reward: 99.0 Training loss: 0.0558 Explore P: 0.0130\n",
      "Episode: 779 Total reward: 199.0 Training loss: 0.0427 Explore P: 0.0129\n",
      "Episode: 782 Total reward: 99.0 Training loss: 0.0538 Explore P: 0.0127\n",
      "Episode: 785 Total reward: 99.0 Training loss: 0.0248 Explore P: 0.0126\n",
      "Episode: 788 Total reward: 99.0 Training loss: 0.0774 Explore P: 0.0125\n",
      "Episode: 791 Total reward: 99.0 Training loss: 0.0269 Explore P: 0.0124\n",
      "Episode: 794 Total reward: 99.0 Training loss: 235.9256 Explore P: 0.0122\n",
      "Episode: 797 Total reward: 99.0 Training loss: 0.0563 Explore P: 0.0121\n",
      "Episode: 800 Total reward: 99.0 Training loss: 0.0263 Explore P: 0.0120\n",
      "Episode: 803 Total reward: 99.0 Training loss: 0.0722 Explore P: 0.0119\n",
      "Episode: 806 Total reward: 99.0 Training loss: 0.0273 Explore P: 0.0118\n",
      "Episode: 809 Total reward: 99.0 Training loss: 0.0355 Explore P: 0.0117\n",
      "Episode: 812 Total reward: 99.0 Training loss: 0.0537 Explore P: 0.0117\n",
      "Episode: 815 Total reward: 99.0 Training loss: 0.0182 Explore P: 0.0116\n",
      "Episode: 818 Total reward: 99.0 Training loss: 0.0283 Explore P: 0.0115\n",
      "Episode: 821 Total reward: 99.0 Training loss: 0.0314 Explore P: 0.0114\n",
      "Episode: 824 Total reward: 99.0 Training loss: 0.0295 Explore P: 0.0114\n",
      "Episode: 827 Total reward: 99.0 Training loss: 0.0295 Explore P: 0.0113\n",
      "Episode: 830 Total reward: 99.0 Training loss: 0.0280 Explore P: 0.0112\n",
      "Episode: 832 Total reward: 185.0 Training loss: 0.1313 Explore P: 0.0112\n",
      "Episode: 835 Total reward: 99.0 Training loss: 0.0703 Explore P: 0.0111\n",
      "Episode: 838 Total reward: 99.0 Training loss: 0.0843 Explore P: 0.0111\n",
      "Episode: 841 Total reward: 99.0 Training loss: 0.0244 Explore P: 0.0110\n",
      "Episode: 843 Total reward: 137.0 Training loss: 0.0520 Explore P: 0.0110\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 846 Total reward: 99.0 Training loss: 0.0363 Explore P: 0.0109\n",
      "Episode: 849 Total reward: 99.0 Training loss: 0.0626 Explore P: 0.0109\n",
      "Episode: 852 Total reward: 99.0 Training loss: 0.0299 Explore P: 0.0108\n",
      "Episode: 855 Total reward: 99.0 Training loss: 0.0351 Explore P: 0.0108\n",
      "Episode: 858 Total reward: 99.0 Training loss: 0.0985 Explore P: 0.0108\n",
      "Episode: 861 Total reward: 99.0 Training loss: 0.0857 Explore P: 0.0107\n",
      "Episode: 864 Total reward: 99.0 Training loss: 0.0265 Explore P: 0.0107\n",
      "Episode: 867 Total reward: 99.0 Training loss: 0.0567 Explore P: 0.0107\n",
      "Episode: 870 Total reward: 99.0 Training loss: 0.0179 Explore P: 0.0106\n",
      "Episode: 873 Total reward: 99.0 Training loss: 0.0360 Explore P: 0.0106\n",
      "Episode: 875 Total reward: 159.0 Training loss: 0.0408 Explore P: 0.0106\n",
      "Episode: 877 Total reward: 87.0 Training loss: 0.0668 Explore P: 0.0106\n",
      "Episode: 879 Total reward: 32.0 Training loss: 0.0877 Explore P: 0.0105\n",
      "Episode: 880 Total reward: 173.0 Training loss: 0.1364 Explore P: 0.0105\n",
      "Episode: 881 Total reward: 131.0 Training loss: 0.0865 Explore P: 0.0105\n",
      "Episode: 882 Total reward: 126.0 Training loss: 0.1413 Explore P: 0.0105\n",
      "Episode: 883 Total reward: 126.0 Training loss: 0.1948 Explore P: 0.0105\n",
      "Episode: 884 Total reward: 123.0 Training loss: 0.0801 Explore P: 0.0105\n",
      "Episode: 886 Total reward: 8.0 Training loss: 0.1067 Explore P: 0.0105\n",
      "Episode: 887 Total reward: 108.0 Training loss: 0.1456 Explore P: 0.0105\n",
      "Episode: 888 Total reward: 52.0 Training loss: 0.2423 Explore P: 0.0105\n",
      "Episode: 889 Total reward: 111.0 Training loss: 0.1607 Explore P: 0.0105\n",
      "Episode: 890 Total reward: 106.0 Training loss: 0.0598 Explore P: 0.0105\n",
      "Episode: 891 Total reward: 112.0 Training loss: 0.0829 Explore P: 0.0105\n",
      "Episode: 892 Total reward: 185.0 Training loss: 0.1364 Explore P: 0.0105\n",
      "Episode: 893 Total reward: 134.0 Training loss: 0.1137 Explore P: 0.0105\n",
      "Episode: 894 Total reward: 141.0 Training loss: 0.0784 Explore P: 0.0105\n",
      "Episode: 896 Total reward: 194.0 Training loss: 0.1821 Explore P: 0.0104\n",
      "Episode: 897 Total reward: 149.0 Training loss: 0.1093 Explore P: 0.0104\n",
      "Episode: 900 Total reward: 67.0 Training loss: 0.0767 Explore P: 0.0104\n",
      "Episode: 902 Total reward: 3.0 Training loss: 0.0595 Explore P: 0.0104\n",
      "Episode: 905 Total reward: 99.0 Training loss: 0.0737 Explore P: 0.0104\n",
      "Episode: 908 Total reward: 99.0 Training loss: 0.1911 Explore P: 0.0104\n",
      "Episode: 911 Total reward: 99.0 Training loss: 0.0339 Explore P: 0.0103\n",
      "Episode: 914 Total reward: 99.0 Training loss: 0.0617 Explore P: 0.0103\n",
      "Episode: 917 Total reward: 99.0 Training loss: 0.1279 Explore P: 0.0103\n",
      "Episode: 920 Total reward: 99.0 Training loss: 0.0699 Explore P: 0.0103\n",
      "Episode: 923 Total reward: 99.0 Training loss: 0.0371 Explore P: 0.0103\n",
      "Episode: 926 Total reward: 99.0 Training loss: 0.0416 Explore P: 0.0103\n",
      "Episode: 929 Total reward: 99.0 Training loss: 0.0999 Explore P: 0.0103\n",
      "Episode: 932 Total reward: 99.0 Training loss: 0.2541 Explore P: 0.0102\n",
      "Episode: 935 Total reward: 99.0 Training loss: 0.0997 Explore P: 0.0102\n",
      "Episode: 938 Total reward: 99.0 Training loss: 0.1048 Explore P: 0.0102\n",
      "Episode: 941 Total reward: 99.0 Training loss: 0.0348 Explore P: 0.0102\n",
      "Episode: 944 Total reward: 99.0 Training loss: 0.0567 Explore P: 0.0102\n",
      "Episode: 947 Total reward: 99.0 Training loss: 0.0950 Explore P: 0.0102\n",
      "Episode: 950 Total reward: 99.0 Training loss: 0.1898 Explore P: 0.0102\n",
      "Episode: 953 Total reward: 99.0 Training loss: 0.2011 Explore P: 0.0102\n",
      "Episode: 956 Total reward: 99.0 Training loss: 0.0956 Explore P: 0.0102\n",
      "Episode: 959 Total reward: 99.0 Training loss: 0.2239 Explore P: 0.0102\n",
      "Episode: 962 Total reward: 99.0 Training loss: 0.1008 Explore P: 0.0101\n",
      "Episode: 965 Total reward: 99.0 Training loss: 0.0673 Explore P: 0.0101\n",
      "Episode: 968 Total reward: 99.0 Training loss: 0.1937 Explore P: 0.0101\n",
      "Episode: 971 Total reward: 99.0 Training loss: 0.0845 Explore P: 0.0101\n",
      "Episode: 974 Total reward: 99.0 Training loss: 0.0826 Explore P: 0.0101\n",
      "Episode: 977 Total reward: 99.0 Training loss: 0.1096 Explore P: 0.0101\n",
      "Episode: 980 Total reward: 99.0 Training loss: 0.1067 Explore P: 0.0101\n",
      "Episode: 983 Total reward: 99.0 Training loss: 0.0759 Explore P: 0.0101\n",
      "Episode: 986 Total reward: 99.0 Training loss: 0.1350 Explore P: 0.0101\n",
      "Episode: 989 Total reward: 99.0 Training loss: 0.0894 Explore P: 0.0101\n",
      "Episode: 992 Total reward: 99.0 Training loss: 0.1139 Explore P: 0.0101\n",
      "Episode: 995 Total reward: 99.0 Training loss: 0.1183 Explore P: 0.0101\n",
      "Episode: 998 Total reward: 99.0 Training loss: 0.0743 Explore P: 0.0101\n"
     ]
    }
   ],
   "source": [
    "# Now train with experiences\n",
    "saver = tf.train.Saver()\n",
    "rewards_list = []\n",
    "with tf.Session() as sess:\n",
    "    # Initialize variables\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    step = 0\n",
    "    for ep in range(1, train_episodes):\n",
    "        total_reward = 0\n",
    "        t = 0\n",
    "        while t < max_steps:\n",
    "            step += 1\n",
    "            # Uncomment this next line to watch the training\n",
    "            # env.render() \n",
    "            \n",
    "            # Explore or Exploit\n",
    "            explore_p = explore_stop + (explore_start - explore_stop)*np.exp(-decay_rate*step) \n",
    "            if explore_p > np.random.rand():\n",
    "                # Make a random action\n",
    "                action = env.action_space.sample()\n",
    "            else:\n",
    "                # Get action from Q-network\n",
    "                feed = {mainQN.inputs_: state.reshape((1, *state.shape))}\n",
    "                Qs = sess.run(mainQN.output, feed_dict=feed)\n",
    "                action = np.argmax(Qs)\n",
    "            \n",
    "            # Take action, get new state and reward\n",
    "            next_state, reward, done, _ = env.step(action)\n",
    "    \n",
    "            total_reward += reward\n",
    "            \n",
    "            if done:\n",
    "                # the episode ends so no next state\n",
    "                next_state = np.zeros(state.shape)\n",
    "                t = max_steps\n",
    "                \n",
    "                print('Episode: {}'.format(ep),\n",
    "                      'Total reward: {}'.format(total_reward),\n",
    "                      'Training loss: {:.4f}'.format(loss),\n",
    "                      'Explore P: {:.4f}'.format(explore_p))\n",
    "                rewards_list.append((ep, total_reward))\n",
    "                \n",
    "                # Add experience to memory\n",
    "                memory.add((state, action, reward, next_state))\n",
    "                \n",
    "                # Start new episode\n",
    "                env.reset()\n",
    "                # Take one random step to get the pole and cart moving\n",
    "                state, reward, done, _ = env.step(env.action_space.sample())\n",
    "\n",
    "            else:\n",
    "                # Add experience to memory\n",
    "                memory.add((state, action, reward, next_state))\n",
    "                state = next_state\n",
    "                t += 1\n",
    "            \n",
    "            # Sample mini-batch from memory\n",
    "            batch = memory.sample(batch_size)\n",
    "            states = np.array([each[0] for each in batch])\n",
    "            actions = np.array([each[1] for each in batch])\n",
    "            rewards = np.array([each[2] for each in batch])\n",
    "            next_states = np.array([each[3] for each in batch])\n",
    "            \n",
    "            # Train network\n",
    "            target_Qs = sess.run(mainQN.output, feed_dict={mainQN.inputs_: next_states})\n",
    "            \n",
    "            # Set target_Qs to 0 for states where episode ends\n",
    "            episode_ends = (next_states == np.zeros(states[0].shape)).all(axis=1)\n",
    "            target_Qs[episode_ends] = (0, 0)\n",
    "            \n",
    "            targets = rewards + gamma * np.max(target_Qs, axis=1)\n",
    "\n",
    "            loss, _ = sess.run([mainQN.loss, mainQN.opt],\n",
    "                                feed_dict={mainQN.inputs_: states,\n",
    "                                           mainQN.targetQs_: targets,\n",
    "                                           mainQN.actions_: actions})\n",
    "        \n",
    "    saver.save(sess, \"checkpoints/cartpole.ckpt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Episode: 1 Total reward: 13.0 Training loss: 1.0202 Explore P: 0.9987\n",
    "Episode: 2 Total reward: 13.0 Training loss: 1.0752 Explore P: 0.9974\n",
    "Episode: 3 Total reward: 9.0 Training loss: 1.0600 Explore P: 0.9965\n",
    "Episode: 4 Total reward: 17.0 Training loss: 1.0429 Explore P: 0.9949\n",
    "Episode: 5 Total reward: 16.0 Training loss: 1.0519 Explore P: 0.9933\n",
    "Episode: 6 Total reward: 15.0 Training loss: 1.0574 Explore P: 0.9918\n",
    "Episode: 7 Total reward: 12.0 Training loss: 1.0889 Explore P: 0.9906\n",
    "Episode: 8 Total reward: 27.0 Training loss: 1.0859 Explore P: 0.9880\n",
    "Episode: 9 Total reward: 24.0 Training loss: 1.2007 Explore P: 0.9857\n",
    "Episode: 10 Total reward: 17.0 Training loss: 1.1116 Explore P: 0.9840\n",
    "Episode: 11 Total reward: 12.0 Training loss: 1.0739 Explore P: 0.9828\n",
    "Episode: 12 Total reward: 25.0 Training loss: 1.0805 Explore P: 0.9804\n",
    "Episode: 13 Total reward: 23.0 Training loss: 1.0628 Explore P: 0.9782\n",
    "Episode: 14 Total reward: 31.0 Training loss: 1.0248 Explore P: 0.9752\n",
    "Episode: 15 Total reward: 15.0 Training loss: 0.9859 Explore P: 0.9737\n",
    "Episode: 16 Total reward: 12.0 Training loss: 1.0983 Explore P: 0.9726\n",
    "Episode: 17 Total reward: 16.0 Training loss: 1.4343 Explore P: 0.9710\n",
    "Episode: 18 Total reward: 21.0 Training loss: 1.2696 Explore P: 0.9690\n",
    "Episode: 19 Total reward: 15.0 Training loss: 1.3542 Explore P: 0.9676\n",
    "Episode: 20 Total reward: 15.0 Training loss: 1.2635 Explore P: 0.9661\n",
    "Episode: 21 Total reward: 16.0 Training loss: 1.3648 Explore P: 0.9646\n",
    "Episode: 22 Total reward: 43.0 Training loss: 1.6088 Explore P: 0.9605\n",
    "Episode: 23 Total reward: 7.0 Training loss: 1.5027 Explore P: 0.9599\n",
    "Episode: 24 Total reward: 13.0 Training loss: 1.7275 Explore P: 0.9586\n",
    "Episode: 25 Total reward: 18.0 Training loss: 1.3902 Explore P: 0.9569\n",
    "Episode: 26 Total reward: 27.0 Training loss: 2.5874 Explore P: 0.9544\n",
    "Episode: 27 Total reward: 32.0 Training loss: 1.5907 Explore P: 0.9513\n",
    "Episode: 28 Total reward: 17.0 Training loss: 2.1144 Explore P: 0.9497\n",
    "Episode: 29 Total reward: 34.0 Training loss: 1.7340 Explore P: 0.9466\n",
    "Episode: 30 Total reward: 18.0 Training loss: 2.5100 Explore P: 0.9449\n",
    "Episode: 31 Total reward: 15.0 Training loss: 2.0166 Explore P: 0.9435\n",
    "Episode: 32 Total reward: 11.0 Training loss: 1.8675 Explore P: 0.9424\n",
    "Episode: 33 Total reward: 18.0 Training loss: 4.0481 Explore P: 0.9408\n",
    "Episode: 34 Total reward: 10.0 Training loss: 4.0895 Explore P: 0.9398\n",
    "Episode: 35 Total reward: 15.0 Training loss: 2.1252 Explore P: 0.9384\n",
    "Episode: 36 Total reward: 14.0 Training loss: 4.7765 Explore P: 0.9371\n",
    "Episode: 37 Total reward: 16.0 Training loss: 3.3848 Explore P: 0.9357\n",
    "Episode: 38 Total reward: 21.0 Training loss: 3.9125 Explore P: 0.9337\n",
    "Episode: 39 Total reward: 16.0 Training loss: 2.6183 Explore P: 0.9322\n",
    "Episode: 40 Total reward: 20.0 Training loss: 5.4929 Explore P: 0.9304\n",
    "Episode: 41 Total reward: 18.0 Training loss: 3.6606 Explore P: 0.9287\n",
    "Episode: 42 Total reward: 17.0 Training loss: 4.5812 Explore P: 0.9272\n",
    "Episode: 43 Total reward: 10.0 Training loss: 3.7633 Explore P: 0.9263\n",
    "Episode: 44 Total reward: 8.0 Training loss: 4.6176 Explore P: 0.9255\n",
    "Episode: 45 Total reward: 39.0 Training loss: 4.2732 Explore P: 0.9220\n",
    "Episode: 46 Total reward: 18.0 Training loss: 4.0041 Explore P: 0.9203\n",
    "Episode: 47 Total reward: 11.0 Training loss: 4.4035 Explore P: 0.9193\n",
    "Episode: 48 Total reward: 25.0 Training loss: 5.4287 Explore P: 0.9171\n",
    "Episode: 49 Total reward: 19.0 Training loss: 9.6972 Explore P: 0.9153\n",
    "Episode: 50 Total reward: 11.0 Training loss: 16.3460 Explore P: 0.9143\n",
    "Episode: 51 Total reward: 11.0 Training loss: 13.4854 Explore P: 0.9133\n",
    "Episode: 52 Total reward: 12.0 Training loss: 12.8016 Explore P: 0.9123\n",
    "Episode: 53 Total reward: 13.0 Training loss: 5.8589 Explore P: 0.9111\n",
    "Episode: 54 Total reward: 12.0 Training loss: 8.5924 Explore P: 0.9100\n",
    "Episode: 55 Total reward: 19.0 Training loss: 8.6204 Explore P: 0.9083\n",
    "Episode: 56 Total reward: 36.0 Training loss: 14.2701 Explore P: 0.9051\n",
    "Episode: 57 Total reward: 9.0 Training loss: 4.5481 Explore P: 0.9043\n",
    "Episode: 58 Total reward: 22.0 Training loss: 12.9695 Explore P: 0.9023\n",
    "Episode: 59 Total reward: 36.0 Training loss: 11.2639 Explore P: 0.8991\n",
    "Episode: 60 Total reward: 16.0 Training loss: 7.7648 Explore P: 0.8977\n",
    "Episode: 61 Total reward: 31.0 Training loss: 4.6997 Explore P: 0.8949\n",
    "Episode: 62 Total reward: 13.0 Training loss: 5.9755 Explore P: 0.8938\n",
    "Episode: 63 Total reward: 10.0 Training loss: 39.1040 Explore P: 0.8929\n",
    "Episode: 64 Total reward: 14.0 Training loss: 23.2767 Explore P: 0.8917\n",
    "Episode: 65 Total reward: 12.0 Training loss: 9.3477 Explore P: 0.8906\n",
    "Episode: 66 Total reward: 20.0 Training loss: 6.4336 Explore P: 0.8888\n",
    "Episode: 67 Total reward: 29.0 Training loss: 17.1522 Explore P: 0.8863\n",
    "Episode: 68 Total reward: 13.0 Training loss: 39.3250 Explore P: 0.8852\n",
    "Episode: 69 Total reward: 20.0 Training loss: 6.2099 Explore P: 0.8834\n",
    "Episode: 70 Total reward: 15.0 Training loss: 20.9229 Explore P: 0.8821\n",
    "Episode: 71 Total reward: 27.0 Training loss: 24.7817 Explore P: 0.8797\n",
    "Episode: 72 Total reward: 12.0 Training loss: 20.7842 Explore P: 0.8787\n",
    "Episode: 73 Total reward: 15.0 Training loss: 12.3202 Explore P: 0.8774\n",
    "Episode: 74 Total reward: 31.0 Training loss: 9.2270 Explore P: 0.8747\n",
    "Episode: 75 Total reward: 13.0 Training loss: 19.8264 Explore P: 0.8736\n",
    "Episode: 76 Total reward: 20.0 Training loss: 72.9411 Explore P: 0.8719\n",
    "Episode: 77 Total reward: 27.0 Training loss: 5.2214 Explore P: 0.8695\n",
    "Episode: 78 Total reward: 14.0 Training loss: 39.3913 Explore P: 0.8683\n",
    "Episode: 79 Total reward: 16.0 Training loss: 7.9491 Explore P: 0.8670\n",
    "Episode: 80 Total reward: 18.0 Training loss: 10.8364 Explore P: 0.8654\n",
    "Episode: 81 Total reward: 16.0 Training loss: 22.2031 Explore P: 0.8641\n",
    "Episode: 82 Total reward: 21.0 Training loss: 23.6590 Explore P: 0.8623\n",
    "Episode: 83 Total reward: 13.0 Training loss: 8.4819 Explore P: 0.8612\n",
    "Episode: 84 Total reward: 10.0 Training loss: 13.3548 Explore P: 0.8603\n",
    "Episode: 85 Total reward: 13.0 Training loss: 18.0272 Explore P: 0.8592\n",
    "Episode: 86 Total reward: 24.0 Training loss: 42.1243 Explore P: 0.8572\n",
    "Episode: 87 Total reward: 9.0 Training loss: 30.8526 Explore P: 0.8564\n",
    "Episode: 88 Total reward: 22.0 Training loss: 36.6084 Explore P: 0.8546\n",
    "Episode: 89 Total reward: 7.0 Training loss: 10.5430 Explore P: 0.8540\n",
    "Episode: 90 Total reward: 12.0 Training loss: 25.5808 Explore P: 0.8529\n",
    "Episode: 91 Total reward: 17.0 Training loss: 47.3073 Explore P: 0.8515\n",
    "Episode: 92 Total reward: 21.0 Training loss: 7.9998 Explore P: 0.8498\n",
    "Episode: 93 Total reward: 15.0 Training loss: 66.6464 Explore P: 0.8485\n",
    "Episode: 94 Total reward: 17.0 Training loss: 95.6354 Explore P: 0.8471\n",
    "Episode: 95 Total reward: 23.0 Training loss: 57.4714 Explore P: 0.8451\n",
    "Episode: 96 Total reward: 11.0 Training loss: 40.7717 Explore P: 0.8442\n",
    "Episode: 97 Total reward: 13.0 Training loss: 43.3380 Explore P: 0.8431\n",
    "Episode: 98 Total reward: 9.0 Training loss: 10.8368 Explore P: 0.8424\n",
    "Episode: 99 Total reward: 21.0 Training loss: 57.7325 Explore P: 0.8406\n",
    "Episode: 100 Total reward: 11.0 Training loss: 9.7291 Explore P: 0.8397\n",
    "Episode: 101 Total reward: 10.0 Training loss: 10.4052 Explore P: 0.8389\n",
    "Episode: 102 Total reward: 26.0 Training loss: 60.4829 Explore P: 0.8368\n",
    "Episode: 103 Total reward: 34.0 Training loss: 9.0924 Explore P: 0.8339\n",
    "Episode: 104 Total reward: 30.0 Training loss: 178.0664 Explore P: 0.8315\n",
    "Episode: 105 Total reward: 14.0 Training loss: 9.0423 Explore P: 0.8303\n",
    "Episode: 106 Total reward: 18.0 Training loss: 126.9380 Explore P: 0.8289\n",
    "Episode: 107 Total reward: 11.0 Training loss: 58.9921 Explore P: 0.8280\n",
    "Episode: 108 Total reward: 20.0 Training loss: 9.1945 Explore P: 0.8263\n",
    "Episode: 109 Total reward: 9.0 Training loss: 9.2887 Explore P: 0.8256\n",
    "Episode: 110 Total reward: 29.0 Training loss: 20.7970 Explore P: 0.8232\n",
    "Episode: 111 Total reward: 17.0 Training loss: 144.6258 Explore P: 0.8218\n",
    "Episode: 112 Total reward: 15.0 Training loss: 82.4089 Explore P: 0.8206\n",
    "Episode: 113 Total reward: 15.0 Training loss: 39.9963 Explore P: 0.8194\n",
    "Episode: 114 Total reward: 8.0 Training loss: 9.8394 Explore P: 0.8188\n",
    "Episode: 115 Total reward: 29.0 Training loss: 76.9930 Explore P: 0.8164\n",
    "Episode: 116 Total reward: 21.0 Training loss: 25.0172 Explore P: 0.8147\n",
    "Episode: 117 Total reward: 24.0 Training loss: 143.5481 Explore P: 0.8128\n",
    "Episode: 118 Total reward: 35.0 Training loss: 86.5429 Explore P: 0.8100\n",
    "Episode: 119 Total reward: 28.0 Training loss: 8.4315 Explore P: 0.8078\n",
    "Episode: 120 Total reward: 13.0 Training loss: 25.7062 Explore P: 0.8067\n",
    "Episode: 121 Total reward: 9.0 Training loss: 6.5005 Explore P: 0.8060\n",
    "Episode: 122 Total reward: 32.0 Training loss: 90.7984 Explore P: 0.8035\n",
    "Episode: 123 Total reward: 21.0 Training loss: 130.2779 Explore P: 0.8018\n",
    "Episode: 124 Total reward: 15.0 Training loss: 167.6294 Explore P: 0.8006\n",
    "Episode: 125 Total reward: 15.0 Training loss: 74.7611 Explore P: 0.7994\n",
    "Episode: 126 Total reward: 20.0 Training loss: 119.3178 Explore P: 0.7978\n",
    "Episode: 127 Total reward: 18.0 Training loss: 196.5175 Explore P: 0.7964\n",
    "Episode: 128 Total reward: 8.0 Training loss: 45.2131 Explore P: 0.7958\n",
    "Episode: 129 Total reward: 24.0 Training loss: 86.0374 Explore P: 0.7939\n",
    "Episode: 130 Total reward: 11.0 Training loss: 7.8129 Explore P: 0.7931\n",
    "Episode: 131 Total reward: 11.0 Training loss: 76.8442 Explore P: 0.7922\n",
    "Episode: 132 Total reward: 28.0 Training loss: 196.6863 Explore P: 0.7900\n",
    "Episode: 133 Total reward: 9.0 Training loss: 45.7586 Explore P: 0.7893\n",
    "Episode: 134 Total reward: 21.0 Training loss: 5.8484 Explore P: 0.7877\n",
    "Episode: 135 Total reward: 10.0 Training loss: 7.3919 Explore P: 0.7869\n",
    "Episode: 136 Total reward: 17.0 Training loss: 12.3142 Explore P: 0.7856\n",
    "Episode: 137 Total reward: 16.0 Training loss: 75.7170 Explore P: 0.7843\n",
    "Episode: 138 Total reward: 12.0 Training loss: 145.3568 Explore P: 0.7834\n",
    "Episode: 139 Total reward: 27.0 Training loss: 121.1114 Explore P: 0.7813\n",
    "Episode: 140 Total reward: 26.0 Training loss: 7.3243 Explore P: 0.7793\n",
    "Episode: 141 Total reward: 40.0 Training loss: 10.6523 Explore P: 0.7762\n",
    "Episode: 142 Total reward: 14.0 Training loss: 6.9482 Explore P: 0.7752\n",
    "Episode: 143 Total reward: 24.0 Training loss: 137.7784 Explore P: 0.7733\n",
    "Episode: 144 Total reward: 12.0 Training loss: 98.8381 Explore P: 0.7724\n",
    "Episode: 145 Total reward: 8.0 Training loss: 14.1739 Explore P: 0.7718\n",
    "Episode: 146 Total reward: 51.0 Training loss: 69.1545 Explore P: 0.7679\n",
    "Episode: 147 Total reward: 29.0 Training loss: 249.9989 Explore P: 0.7657\n",
    "Episode: 148 Total reward: 9.0 Training loss: 140.8663 Explore P: 0.7651\n",
    "Episode: 149 Total reward: 13.0 Training loss: 141.5930 Explore P: 0.7641\n",
    "Episode: 150 Total reward: 19.0 Training loss: 12.6228 Explore P: 0.7627\n",
    "Episode: 151 Total reward: 19.0 Training loss: 136.3315 Explore P: 0.7612\n",
    "Episode: 152 Total reward: 10.0 Training loss: 110.3699 Explore P: 0.7605\n",
    "Episode: 153 Total reward: 18.0 Training loss: 8.3900 Explore P: 0.7591\n",
    "Episode: 154 Total reward: 18.0 Training loss: 96.3717 Explore P: 0.7578\n",
    "Episode: 155 Total reward: 7.0 Training loss: 6.0889 Explore P: 0.7573\n",
    "Episode: 156 Total reward: 15.0 Training loss: 126.7419 Explore P: 0.7561\n",
    "Episode: 157 Total reward: 15.0 Training loss: 67.2544 Explore P: 0.7550\n",
    "Episode: 158 Total reward: 26.0 Training loss: 12.2839 Explore P: 0.7531\n",
    "Episode: 159 Total reward: 20.0 Training loss: 5.8118 Explore P: 0.7516\n",
    "Episode: 160 Total reward: 10.0 Training loss: 96.4570 Explore P: 0.7509\n",
    "Episode: 161 Total reward: 23.0 Training loss: 7.6207 Explore P: 0.7492\n",
    "Episode: 162 Total reward: 18.0 Training loss: 66.5249 Explore P: 0.7478\n",
    "Episode: 163 Total reward: 18.0 Training loss: 111.3273 Explore P: 0.7465\n",
    "Episode: 164 Total reward: 21.0 Training loss: 11.5292 Explore P: 0.7450\n",
    "Episode: 165 Total reward: 17.0 Training loss: 6.3130 Explore P: 0.7437\n",
    "Episode: 166 Total reward: 22.0 Training loss: 153.8167 Explore P: 0.7421\n",
    "Episode: 167 Total reward: 17.0 Training loss: 7.0915 Explore P: 0.7408\n",
    "Episode: 168 Total reward: 34.0 Training loss: 228.3831 Explore P: 0.7384\n",
    "Episode: 169 Total reward: 13.0 Training loss: 8.5996 Explore P: 0.7374\n",
    "Episode: 170 Total reward: 13.0 Training loss: 90.8898 Explore P: 0.7365\n",
    "Episode: 171 Total reward: 20.0 Training loss: 4.8179 Explore P: 0.7350\n",
    "Episode: 172 Total reward: 9.0 Training loss: 6.2508 Explore P: 0.7344\n",
    "Episode: 173 Total reward: 14.0 Training loss: 5.2401 Explore P: 0.7334\n",
    "Episode: 174 Total reward: 12.0 Training loss: 3.9268 Explore P: 0.7325\n",
    "Episode: 175 Total reward: 12.0 Training loss: 5.6376 Explore P: 0.7316\n",
    "Episode: 176 Total reward: 11.0 Training loss: 44.5308 Explore P: 0.7308\n",
    "Episode: 177 Total reward: 12.0 Training loss: 4.9717 Explore P: 0.7300\n",
    "Episode: 178 Total reward: 9.0 Training loss: 181.0085 Explore P: 0.7293\n",
    "Episode: 179 Total reward: 11.0 Training loss: 73.1134 Explore P: 0.7285\n",
    "Episode: 180 Total reward: 13.0 Training loss: 87.3085 Explore P: 0.7276\n",
    "Episode: 181 Total reward: 12.0 Training loss: 121.6627 Explore P: 0.7267\n",
    "Episode: 182 Total reward: 12.0 Training loss: 58.1967 Explore P: 0.7259\n",
    "Episode: 183 Total reward: 13.0 Training loss: 85.1540 Explore P: 0.7249\n",
    "Episode: 184 Total reward: 16.0 Training loss: 5.1214 Explore P: 0.7238\n",
    "Episode: 185 Total reward: 19.0 Training loss: 69.1839 Explore P: 0.7224\n",
    "Episode: 186 Total reward: 7.0 Training loss: 63.2256 Explore P: 0.7219\n",
    "Episode: 187 Total reward: 17.0 Training loss: 73.2788 Explore P: 0.7207\n",
    "Episode: 188 Total reward: 15.0 Training loss: 78.6213 Explore P: 0.7197\n",
    "Episode: 189 Total reward: 11.0 Training loss: 88.5211 Explore P: 0.7189\n",
    "Episode: 190 Total reward: 14.0 Training loss: 60.1332 Explore P: 0.7179\n",
    "Episode: 191 Total reward: 15.0 Training loss: 135.7724 Explore P: 0.7168\n",
    "Episode: 192 Total reward: 15.0 Training loss: 156.9691 Explore P: 0.7158\n",
    "Episode: 193 Total reward: 17.0 Training loss: 93.3756 Explore P: 0.7146\n",
    "Episode: 194 Total reward: 12.0 Training loss: 3.0462 Explore P: 0.7137\n",
    "Episode: 195 Total reward: 9.0 Training loss: 119.2650 Explore P: 0.7131\n",
    "Episode: 196 Total reward: 13.0 Training loss: 66.6383 Explore P: 0.7122\n",
    "Episode: 197 Total reward: 9.0 Training loss: 113.7849 Explore P: 0.7116\n",
    "Episode: 198 Total reward: 13.0 Training loss: 54.6072 Explore P: 0.7106\n",
    "Episode: 199 Total reward: 19.0 Training loss: 54.8980 Explore P: 0.7093\n",
    "Episode: 200 Total reward: 20.0 Training loss: 155.5480 Explore P: 0.7079\n",
    "Episode: 201 Total reward: 10.0 Training loss: 45.8685 Explore P: 0.7072\n",
    "Episode: 202 Total reward: 14.0 Training loss: 53.5145 Explore P: 0.7062\n",
    "Episode: 203 Total reward: 8.0 Training loss: 107.9623 Explore P: 0.7057\n",
    "Episode: 204 Total reward: 21.0 Training loss: 40.2749 Explore P: 0.7042\n",
    "Episode: 205 Total reward: 32.0 Training loss: 43.2627 Explore P: 0.7020\n",
    "Episode: 206 Total reward: 9.0 Training loss: 55.5398 Explore P: 0.7014\n",
    "Episode: 207 Total reward: 15.0 Training loss: 1.9959 Explore P: 0.7004\n",
    "Episode: 208 Total reward: 12.0 Training loss: 105.3751 Explore P: 0.6995\n",
    "Episode: 209 Total reward: 11.0 Training loss: 40.8319 Explore P: 0.6988\n",
    "Episode: 210 Total reward: 10.0 Training loss: 89.7147 Explore P: 0.6981\n",
    "Episode: 211 Total reward: 10.0 Training loss: 1.1946 Explore P: 0.6974\n",
    "Episode: 212 Total reward: 10.0 Training loss: 80.6916 Explore P: 0.6967\n",
    "Episode: 213 Total reward: 24.0 Training loss: 88.0977 Explore P: 0.6951\n",
    "Episode: 214 Total reward: 14.0 Training loss: 46.4105 Explore P: 0.6941\n",
    "Episode: 215 Total reward: 11.0 Training loss: 40.3726 Explore P: 0.6933\n",
    "Episode: 216 Total reward: 11.0 Training loss: 3.0770 Explore P: 0.6926\n",
    "Episode: 217 Total reward: 8.0 Training loss: 1.7495 Explore P: 0.6921\n",
    "Episode: 218 Total reward: 16.0 Training loss: 1.5615 Explore P: 0.6910\n",
    "Episode: 219 Total reward: 17.0 Training loss: 2.0250 Explore P: 0.6898\n",
    "Episode: 220 Total reward: 18.0 Training loss: 37.8432 Explore P: 0.6886\n",
    "Episode: 221 Total reward: 17.0 Training loss: 1.9049 Explore P: 0.6874\n",
    "Episode: 222 Total reward: 16.0 Training loss: 1.9652 Explore P: 0.6863\n",
    "Episode: 223 Total reward: 16.0 Training loss: 1.4384 Explore P: 0.6853\n",
    "Episode: 224 Total reward: 27.0 Training loss: 66.0615 Explore P: 0.6834\n",
    "Episode: 225 Total reward: 9.0 Training loss: 0.8478 Explore P: 0.6828\n",
    "Episode: 226 Total reward: 14.0 Training loss: 1.0319 Explore P: 0.6819\n",
    "Episode: 227 Total reward: 17.0 Training loss: 97.6957 Explore P: 0.6808\n",
    "Episode: 228 Total reward: 8.0 Training loss: 68.0521 Explore P: 0.6802\n",
    "Episode: 229 Total reward: 9.0 Training loss: 110.8437 Explore P: 0.6796\n",
    "Episode: 230 Total reward: 19.0 Training loss: 1.6856 Explore P: 0.6783\n",
    "Episode: 231 Total reward: 9.0 Training loss: 2.0634 Explore P: 0.6777\n",
    "Episode: 232 Total reward: 11.0 Training loss: 32.0714 Explore P: 0.6770\n",
    "Episode: 233 Total reward: 9.0 Training loss: 2.0387 Explore P: 0.6764\n",
    "Episode: 234 Total reward: 13.0 Training loss: 66.9349 Explore P: 0.6755\n",
    "Episode: 235 Total reward: 14.0 Training loss: 110.6725 Explore P: 0.6746\n",
    "Episode: 236 Total reward: 18.0 Training loss: 1.0585 Explore P: 0.6734\n",
    "Episode: 237 Total reward: 11.0 Training loss: 117.0101 Explore P: 0.6727\n",
    "Episode: 238 Total reward: 7.0 Training loss: 2.6115 Explore P: 0.6722\n",
    "Episode: 239 Total reward: 10.0 Training loss: 124.7320 Explore P: 0.6716\n",
    "Episode: 240 Total reward: 18.0 Training loss: 2.5475 Explore P: 0.6704\n",
    "Episode: 241 Total reward: 37.0 Training loss: 2.1454 Explore P: 0.6679\n",
    "Episode: 242 Total reward: 11.0 Training loss: 23.6042 Explore P: 0.6672\n",
    "Episode: 243 Total reward: 32.0 Training loss: 1.4344 Explore P: 0.6651\n",
    "Episode: 244 Total reward: 9.0 Training loss: 1.5328 Explore P: 0.6645\n",
    "Episode: 245 Total reward: 14.0 Training loss: 84.7870 Explore P: 0.6636\n",
    "Episode: 246 Total reward: 12.0 Training loss: 2.7292 Explore P: 0.6628\n",
    "Episode: 247 Total reward: 26.0 Training loss: 40.6692 Explore P: 0.6611\n",
    "Episode: 248 Total reward: 12.0 Training loss: 22.0901 Explore P: 0.6603\n",
    "Episode: 249 Total reward: 15.0 Training loss: 37.9304 Explore P: 0.6594\n",
    "Episode: 250 Total reward: 20.0 Training loss: 1.4137 Explore P: 0.6581\n",
    "Episode: 251 Total reward: 16.0 Training loss: 1.7831 Explore P: 0.6570\n",
    "Episode: 252 Total reward: 9.0 Training loss: 38.0640 Explore P: 0.6565\n",
    "Episode: 253 Total reward: 17.0 Training loss: 21.7703 Explore P: 0.6554\n",
    "Episode: 254 Total reward: 24.0 Training loss: 40.3204 Explore P: 0.6538\n",
    "Episode: 255 Total reward: 30.0 Training loss: 43.4179 Explore P: 0.6519\n",
    "Episode: 256 Total reward: 11.0 Training loss: 60.9330 Explore P: 0.6512\n",
    "Episode: 257 Total reward: 14.0 Training loss: 66.6886 Explore P: 0.6503\n",
    "Episode: 258 Total reward: 15.0 Training loss: 2.5639 Explore P: 0.6493\n",
    "Episode: 259 Total reward: 19.0 Training loss: 2.6969 Explore P: 0.6481\n",
    "Episode: 260 Total reward: 10.0 Training loss: 2.6837 Explore P: 0.6475\n",
    "Episode: 261 Total reward: 30.0 Training loss: 20.6603 Explore P: 0.6456\n",
    "Episode: 262 Total reward: 17.0 Training loss: 32.1585 Explore P: 0.6445\n",
    "Episode: 263 Total reward: 15.0 Training loss: 1.0833 Explore P: 0.6435\n",
    "Episode: 264 Total reward: 13.0 Training loss: 81.4551 Explore P: 0.6427\n",
    "Episode: 265 Total reward: 17.0 Training loss: 3.3823 Explore P: 0.6416\n",
    "Episode: 266 Total reward: 11.0 Training loss: 36.3942 Explore P: 0.6409\n",
    "Episode: 267 Total reward: 11.0 Training loss: 1.6628 Explore P: 0.6402\n",
    "Episode: 268 Total reward: 33.0 Training loss: 26.9925 Explore P: 0.6382\n",
    "Episode: 269 Total reward: 18.0 Training loss: 45.8608 Explore P: 0.6370\n",
    "Episode: 270 Total reward: 20.0 Training loss: 2.7911 Explore P: 0.6358\n",
    "Episode: 271 Total reward: 10.0 Training loss: 35.9215 Explore P: 0.6352\n",
    "Episode: 272 Total reward: 14.0 Training loss: 2.5923 Explore P: 0.6343\n",
    "Episode: 273 Total reward: 16.0 Training loss: 41.2339 Explore P: 0.6333\n",
    "Episode: 274 Total reward: 18.0 Training loss: 46.7318 Explore P: 0.6322\n",
    "Episode: 275 Total reward: 14.0 Training loss: 2.7245 Explore P: 0.6313\n",
    "Episode: 276 Total reward: 8.0 Training loss: 16.2681 Explore P: 0.6308\n",
    "Episode: 277 Total reward: 10.0 Training loss: 21.6856 Explore P: 0.6302\n",
    "Episode: 278 Total reward: 12.0 Training loss: 1.7879 Explore P: 0.6294\n",
    "Episode: 279 Total reward: 10.0 Training loss: 97.1567 Explore P: 0.6288\n",
    "Episode: 280 Total reward: 16.0 Training loss: 3.4710 Explore P: 0.6278\n",
    "Episode: 281 Total reward: 14.0 Training loss: 65.8457 Explore P: 0.6270\n",
    "Episode: 282 Total reward: 21.0 Training loss: 32.4442 Explore P: 0.6257\n",
    "Episode: 283 Total reward: 17.0 Training loss: 48.0136 Explore P: 0.6246\n",
    "Episode: 284 Total reward: 11.0 Training loss: 2.8833 Explore P: 0.6239\n",
    "Episode: 285 Total reward: 16.0 Training loss: 92.6062 Explore P: 0.6230\n",
    "Episode: 286 Total reward: 16.0 Training loss: 19.1051 Explore P: 0.6220\n",
    "Episode: 287 Total reward: 7.0 Training loss: 1.8220 Explore P: 0.6216\n",
    "Episode: 288 Total reward: 16.0 Training loss: 41.3844 Explore P: 0.6206\n",
    "Episode: 289 Total reward: 18.0 Training loss: 50.0580 Explore P: 0.6195\n",
    "Episode: 290 Total reward: 13.0 Training loss: 83.2142 Explore P: 0.6187\n",
    "Episode: 291 Total reward: 14.0 Training loss: 70.2605 Explore P: 0.6178\n",
    "Episode: 292 Total reward: 16.0 Training loss: 53.9664 Explore P: 0.6169\n",
    "Episode: 293 Total reward: 17.0 Training loss: 3.2764 Explore P: 0.6158\n",
    "Episode: 294 Total reward: 18.0 Training loss: 17.7963 Explore P: 0.6147\n",
    "Episode: 295 Total reward: 17.0 Training loss: 32.3772 Explore P: 0.6137\n",
    "Episode: 296 Total reward: 32.0 Training loss: 18.3755 Explore P: 0.6118\n",
    "Episode: 297 Total reward: 29.0 Training loss: 17.1377 Explore P: 0.6100\n",
    "Episode: 298 Total reward: 12.0 Training loss: 14.2922 Explore P: 0.6093\n",
    "Episode: 299 Total reward: 14.0 Training loss: 29.2226 Explore P: 0.6085\n",
    "Episode: 300 Total reward: 17.0 Training loss: 38.9089 Explore P: 0.6075\n",
    "Episode: 301 Total reward: 9.0 Training loss: 62.2483 Explore P: 0.6069\n",
    "Episode: 302 Total reward: 22.0 Training loss: 2.3240 Explore P: 0.6056\n",
    "Episode: 303 Total reward: 16.0 Training loss: 0.9979 Explore P: 0.6047\n",
    "Episode: 304 Total reward: 8.0 Training loss: 67.9231 Explore P: 0.6042\n",
    "Episode: 305 Total reward: 13.0 Training loss: 33.0928 Explore P: 0.6034\n",
    "Episode: 306 Total reward: 20.0 Training loss: 1.3173 Explore P: 0.6022\n",
    "Episode: 307 Total reward: 23.0 Training loss: 50.2106 Explore P: 0.6009\n",
    "Episode: 308 Total reward: 17.0 Training loss: 52.5245 Explore P: 0.5999\n",
    "Episode: 309 Total reward: 20.0 Training loss: 32.5832 Explore P: 0.5987\n",
    "Episode: 310 Total reward: 19.0 Training loss: 29.0224 Explore P: 0.5976\n",
    "Episode: 311 Total reward: 19.0 Training loss: 29.8863 Explore P: 0.5965\n",
    "Episode: 312 Total reward: 27.0 Training loss: 34.4016 Explore P: 0.5949\n",
    "Episode: 313 Total reward: 9.0 Training loss: 1.1433 Explore P: 0.5944\n",
    "Episode: 314 Total reward: 20.0 Training loss: 28.8137 Explore P: 0.5932\n",
    "Episode: 315 Total reward: 24.0 Training loss: 48.5379 Explore P: 0.5918\n",
    "Episode: 316 Total reward: 28.0 Training loss: 45.2671 Explore P: 0.5902\n",
    "Episode: 317 Total reward: 13.0 Training loss: 45.9822 Explore P: 0.5894\n",
    "Episode: 318 Total reward: 12.0 Training loss: 86.3972 Explore P: 0.5887\n",
    "Episode: 319 Total reward: 10.0 Training loss: 11.2909 Explore P: 0.5881\n",
    "Episode: 320 Total reward: 11.0 Training loss: 36.5474 Explore P: 0.5875\n",
    "Episode: 321 Total reward: 13.0 Training loss: 1.1439 Explore P: 0.5867\n",
    "Episode: 322 Total reward: 8.0 Training loss: 12.6978 Explore P: 0.5863\n",
    "Episode: 323 Total reward: 20.0 Training loss: 31.7664 Explore P: 0.5851\n",
    "Episode: 324 Total reward: 8.0 Training loss: 29.4243 Explore P: 0.5847\n",
    "Episode: 325 Total reward: 13.0 Training loss: 12.2373 Explore P: 0.5839\n",
    "Episode: 326 Total reward: 19.0 Training loss: 24.2228 Explore P: 0.5828\n",
    "Episode: 327 Total reward: 68.0 Training loss: 0.7256 Explore P: 0.5790\n",
    "Episode: 328 Total reward: 11.0 Training loss: 1.2313 Explore P: 0.5783\n",
    "Episode: 329 Total reward: 15.0 Training loss: 1.3319 Explore P: 0.5775\n",
    "Episode: 330 Total reward: 53.0 Training loss: 9.9350 Explore P: 0.5745\n",
    "Episode: 331 Total reward: 74.0 Training loss: 1.4366 Explore P: 0.5703\n",
    "Episode: 332 Total reward: 16.0 Training loss: 11.2724 Explore P: 0.5694\n",
    "Episode: 333 Total reward: 34.0 Training loss: 10.6128 Explore P: 0.5675\n",
    "Episode: 334 Total reward: 27.0 Training loss: 14.9559 Explore P: 0.5660\n",
    "Episode: 335 Total reward: 31.0 Training loss: 16.6541 Explore P: 0.5643\n",
    "Episode: 336 Total reward: 49.0 Training loss: 23.3966 Explore P: 0.5616\n",
    "Episode: 337 Total reward: 40.0 Training loss: 45.3419 Explore P: 0.5594\n",
    "Episode: 338 Total reward: 71.0 Training loss: 0.8244 Explore P: 0.5555\n",
    "Episode: 339 Total reward: 56.0 Training loss: 41.4562 Explore P: 0.5525\n",
    "Episode: 340 Total reward: 18.0 Training loss: 1.2548 Explore P: 0.5515\n",
    "Episode: 341 Total reward: 56.0 Training loss: 1.5400 Explore P: 0.5485\n",
    "Episode: 342 Total reward: 34.0 Training loss: 12.0206 Explore P: 0.5466\n",
    "Episode: 343 Total reward: 67.0 Training loss: 1.4189 Explore P: 0.5430\n",
    "Episode: 344 Total reward: 27.0 Training loss: 1.3138 Explore P: 0.5416\n",
    "Episode: 345 Total reward: 42.0 Training loss: 1.1650 Explore P: 0.5394\n",
    "Episode: 346 Total reward: 23.0 Training loss: 23.1743 Explore P: 0.5382\n",
    "Episode: 347 Total reward: 54.0 Training loss: 0.6971 Explore P: 0.5353\n",
    "Episode: 348 Total reward: 34.0 Training loss: 27.2789 Explore P: 0.5335\n",
    "Episode: 349 Total reward: 25.0 Training loss: 37.4133 Explore P: 0.5322\n",
    "Episode: 350 Total reward: 20.0 Training loss: 1.6443 Explore P: 0.5312\n",
    "Episode: 351 Total reward: 26.0 Training loss: 12.6839 Explore P: 0.5298\n",
    "Episode: 352 Total reward: 40.0 Training loss: 13.3593 Explore P: 0.5278\n",
    "Episode: 353 Total reward: 18.0 Training loss: 1.7079 Explore P: 0.5268\n",
    "Episode: 354 Total reward: 47.0 Training loss: 32.5788 Explore P: 0.5244\n",
    "Episode: 355 Total reward: 20.0 Training loss: 1.6101 Explore P: 0.5234\n",
    "Episode: 356 Total reward: 53.0 Training loss: 2.5321 Explore P: 0.5207\n",
    "Episode: 357 Total reward: 15.0 Training loss: 1.6396 Explore P: 0.5199\n",
    "Episode: 358 Total reward: 76.0 Training loss: 20.8058 Explore P: 0.5160\n",
    "Episode: 359 Total reward: 12.0 Training loss: 13.0315 Explore P: 0.5154\n",
    "Episode: 360 Total reward: 42.0 Training loss: 10.1313 Explore P: 0.5133\n",
    "Episode: 361 Total reward: 53.0 Training loss: 25.4319 Explore P: 0.5106\n",
    "Episode: 362 Total reward: 33.0 Training loss: 26.4256 Explore P: 0.5090\n",
    "Episode: 363 Total reward: 85.0 Training loss: 20.2429 Explore P: 0.5048\n",
    "Episode: 364 Total reward: 23.0 Training loss: 16.1083 Explore P: 0.5036\n",
    "Episode: 365 Total reward: 30.0 Training loss: 1.6888 Explore P: 0.5022\n",
    "Episode: 366 Total reward: 66.0 Training loss: 2.0408 Explore P: 0.4989\n",
    "Episode: 367 Total reward: 37.0 Training loss: 18.6438 Explore P: 0.4971\n",
    "Episode: 368 Total reward: 50.0 Training loss: 20.1544 Explore P: 0.4947\n",
    "Episode: 369 Total reward: 78.0 Training loss: 23.8497 Explore P: 0.4909\n",
    "Episode: 370 Total reward: 83.0 Training loss: 20.6897 Explore P: 0.4869\n",
    "Episode: 371 Total reward: 44.0 Training loss: 25.4317 Explore P: 0.4849\n",
    "Episode: 372 Total reward: 44.0 Training loss: 1.5212 Explore P: 0.4828\n",
    "Episode: 373 Total reward: 14.0 Training loss: 1.5019 Explore P: 0.4821\n",
    "Episode: 374 Total reward: 31.0 Training loss: 1.8348 Explore P: 0.4806\n",
    "Episode: 375 Total reward: 25.0 Training loss: 19.7533 Explore P: 0.4795\n",
    "Episode: 376 Total reward: 51.0 Training loss: 1.5433 Explore P: 0.4771\n",
    "Episode: 377 Total reward: 23.0 Training loss: 12.9174 Explore P: 0.4760\n",
    "Episode: 378 Total reward: 67.0 Training loss: 27.2318 Explore P: 0.4729\n",
    "Episode: 379 Total reward: 26.0 Training loss: 1.9319 Explore P: 0.4717\n",
    "Episode: 380 Total reward: 35.0 Training loss: 43.2445 Explore P: 0.4701\n",
    "Episode: 381 Total reward: 33.0 Training loss: 1.5195 Explore P: 0.4686\n",
    "Episode: 382 Total reward: 30.0 Training loss: 15.4622 Explore P: 0.4672\n",
    "Episode: 383 Total reward: 12.0 Training loss: 1.8349 Explore P: 0.4666\n",
    "Episode: 384 Total reward: 25.0 Training loss: 47.7600 Explore P: 0.4655\n",
    "Episode: 385 Total reward: 36.0 Training loss: 29.6753 Explore P: 0.4639\n",
    "Episode: 386 Total reward: 50.0 Training loss: 1.1244 Explore P: 0.4616\n",
    "Episode: 387 Total reward: 35.0 Training loss: 1.0955 Explore P: 0.4600\n",
    "Episode: 388 Total reward: 52.0 Training loss: 24.9624 Explore P: 0.4577\n",
    "Episode: 389 Total reward: 52.0 Training loss: 28.2028 Explore P: 0.4554\n",
    "Episode: 390 Total reward: 132.0 Training loss: 30.5190 Explore P: 0.4495\n",
    "Episode: 391 Total reward: 25.0 Training loss: 10.3908 Explore P: 0.4484\n",
    "Episode: 392 Total reward: 56.0 Training loss: 14.1483 Explore P: 0.4460\n",
    "Episode: 393 Total reward: 110.0 Training loss: 2.0169 Explore P: 0.4412\n",
    "Episode: 394 Total reward: 78.0 Training loss: 1.2122 Explore P: 0.4379\n",
    "Episode: 395 Total reward: 44.0 Training loss: 56.4728 Explore P: 0.4360\n",
    "Episode: 396 Total reward: 90.0 Training loss: 65.6667 Explore P: 0.4322\n",
    "Episode: 397 Total reward: 36.0 Training loss: 0.9032 Explore P: 0.4307\n",
    "Episode: 398 Total reward: 40.0 Training loss: 0.8414 Explore P: 0.4290\n",
    "Episode: 399 Total reward: 109.0 Training loss: 42.5467 Explore P: 0.4244\n",
    "Episode: 400 Total reward: 37.0 Training loss: 2.6053 Explore P: 0.4229\n",
    "Episode: 401 Total reward: 62.0 Training loss: 1.2301 Explore P: 0.4203\n",
    "Episode: 402 Total reward: 42.0 Training loss: 1.1384 Explore P: 0.4186\n",
    "Episode: 403 Total reward: 71.0 Training loss: 1.6765 Explore P: 0.4157\n",
    "Episode: 404 Total reward: 88.0 Training loss: 2.4000 Explore P: 0.4122\n",
    "Episode: 405 Total reward: 55.0 Training loss: 24.7748 Explore P: 0.4100\n",
    "Episode: 406 Total reward: 33.0 Training loss: 13.5934 Explore P: 0.4087\n",
    "Episode: 407 Total reward: 44.0 Training loss: 14.6865 Explore P: 0.4069\n",
    "Episode: 408 Total reward: 40.0 Training loss: 2.0898 Explore P: 0.4053\n",
    "Episode: 409 Total reward: 98.0 Training loss: 2.1043 Explore P: 0.4015\n",
    "Episode: 410 Total reward: 63.0 Training loss: 11.3562 Explore P: 0.3990\n",
    "Episode: 411 Total reward: 50.0 Training loss: 14.1151 Explore P: 0.3971\n",
    "Episode: 412 Total reward: 44.0 Training loss: 1.2370 Explore P: 0.3954\n",
    "Episode: 413 Total reward: 56.0 Training loss: 2.1136 Explore P: 0.3932\n",
    "Episode: 414 Total reward: 61.0 Training loss: 2.2578 Explore P: 0.3909\n",
    "Episode: 415 Total reward: 49.0 Training loss: 1.3966 Explore P: 0.3890\n",
    "Episode: 416 Total reward: 55.0 Training loss: 10.2836 Explore P: 0.3869\n",
    "Episode: 417 Total reward: 121.0 Training loss: 2.2477 Explore P: 0.3824\n",
    "Episode: 418 Total reward: 46.0 Training loss: 2.3118 Explore P: 0.3807\n",
    "Episode: 419 Total reward: 70.0 Training loss: 27.3952 Explore P: 0.3781\n",
    "Episode: 420 Total reward: 72.0 Training loss: 45.7570 Explore P: 0.3755\n",
    "Episode: 421 Total reward: 41.0 Training loss: 59.1887 Explore P: 0.3740\n",
    "Episode: 422 Total reward: 67.0 Training loss: 27.0998 Explore P: 0.3716\n",
    "Episode: 423 Total reward: 46.0 Training loss: 43.1971 Explore P: 0.3699\n",
    "Episode: 424 Total reward: 52.0 Training loss: 2.0718 Explore P: 0.3680\n",
    "Episode: 425 Total reward: 92.0 Training loss: 96.7074 Explore P: 0.3647\n",
    "Episode: 426 Total reward: 60.0 Training loss: 2.0684 Explore P: 0.3626\n",
    "Episode: 427 Total reward: 106.0 Training loss: 54.1831 Explore P: 0.3589\n",
    "Episode: 428 Total reward: 76.0 Training loss: 1.9612 Explore P: 0.3563\n",
    "Episode: 429 Total reward: 42.0 Training loss: 1.6153 Explore P: 0.3548\n",
    "Episode: 430 Total reward: 77.0 Training loss: 3.9801 Explore P: 0.3522\n",
    "Episode: 431 Total reward: 123.0 Training loss: 2.0505 Explore P: 0.3480\n",
    "Episode: 432 Total reward: 150.0 Training loss: 19.4217 Explore P: 0.3430\n",
    "Episode: 433 Total reward: 57.0 Training loss: 1.5850 Explore P: 0.3411\n",
    "Episode: 434 Total reward: 74.0 Training loss: 2.4292 Explore P: 0.3386\n",
    "Episode: 435 Total reward: 97.0 Training loss: 23.5709 Explore P: 0.3354\n",
    "Episode: 436 Total reward: 99.0 Training loss: 2.0727 Explore P: 0.3322\n",
    "Episode: 437 Total reward: 101.0 Training loss: 22.3250 Explore P: 0.3290\n",
    "Episode: 438 Total reward: 46.0 Training loss: 2.0320 Explore P: 0.3275\n",
    "Episode: 439 Total reward: 51.0 Training loss: 4.8099 Explore P: 0.3259\n",
    "Episode: 440 Total reward: 111.0 Training loss: 68.3524 Explore P: 0.3224\n",
    "Episode: 441 Total reward: 167.0 Training loss: 2.3045 Explore P: 0.3173\n",
    "Episode: 442 Total reward: 80.0 Training loss: 0.8798 Explore P: 0.3148\n",
    "Episode: 443 Total reward: 170.0 Training loss: 48.6270 Explore P: 0.3097\n",
    "Episode: 444 Total reward: 77.0 Training loss: 2.2555 Explore P: 0.3074\n",
    "Episode: 445 Total reward: 84.0 Training loss: 3.0428 Explore P: 0.3049\n",
    "Episode: 447 Total reward: 12.0 Training loss: 2.8022 Explore P: 0.2987\n",
    "Episode: 448 Total reward: 66.0 Training loss: 120.3442 Explore P: 0.2968\n",
    "Episode: 449 Total reward: 152.0 Training loss: 6.2880 Explore P: 0.2925\n",
    "Episode: 450 Total reward: 141.0 Training loss: 2.8015 Explore P: 0.2885\n",
    "Episode: 451 Total reward: 99.0 Training loss: 64.0921 Explore P: 0.2858\n",
    "Episode: 452 Total reward: 79.0 Training loss: 1.5581 Explore P: 0.2836\n",
    "Episode: 453 Total reward: 49.0 Training loss: 113.2557 Explore P: 0.2823\n",
    "Episode: 455 Total reward: 106.0 Training loss: 210.4934 Explore P: 0.2741\n",
    "Episode: 456 Total reward: 109.0 Training loss: 81.6662 Explore P: 0.2712\n",
    "Episode: 457 Total reward: 56.0 Training loss: 3.2287 Explore P: 0.2697\n",
    "Episode: 458 Total reward: 138.0 Training loss: 2.5795 Explore P: 0.2662\n",
    "Episode: 459 Total reward: 93.0 Training loss: 3.4260 Explore P: 0.2638\n",
    "Episode: 460 Total reward: 71.0 Training loss: 139.3341 Explore P: 0.2620\n",
    "Episode: 461 Total reward: 106.0 Training loss: 2.6074 Explore P: 0.2594\n",
    "Episode: 462 Total reward: 63.0 Training loss: 2.8252 Explore P: 0.2578\n",
    "Episode: 463 Total reward: 71.0 Training loss: 25.8917 Explore P: 0.2560\n",
    "Episode: 464 Total reward: 79.0 Training loss: 3.8067 Explore P: 0.2541\n",
    "Episode: 465 Total reward: 86.0 Training loss: 1.6050 Explore P: 0.2520\n",
    "Episode: 466 Total reward: 88.0 Training loss: 44.2827 Explore P: 0.2499\n",
    "Episode: 467 Total reward: 72.0 Training loss: 0.7160 Explore P: 0.2482\n",
    "Episode: 468 Total reward: 152.0 Training loss: 75.7239 Explore P: 0.2446\n",
    "Episode: 469 Total reward: 122.0 Training loss: 7.4345 Explore P: 0.2417\n",
    "Episode: 470 Total reward: 81.0 Training loss: 101.0922 Explore P: 0.2399\n",
    "Episode: 471 Total reward: 38.0 Training loss: 1.6301 Explore P: 0.2390\n",
    "Episode: 472 Total reward: 79.0 Training loss: 72.4920 Explore P: 0.2372\n",
    "Episode: 473 Total reward: 190.0 Training loss: 1.3869 Explore P: 0.2329\n",
    "Episode: 474 Total reward: 197.0 Training loss: 1.5386 Explore P: 0.2286\n",
    "Episode: 476 Total reward: 42.0 Training loss: 0.8364 Explore P: 0.2233\n",
    "Episode: 477 Total reward: 134.0 Training loss: 88.3979 Explore P: 0.2205\n",
    "Episode: 478 Total reward: 128.0 Training loss: 94.1007 Explore P: 0.2178\n",
    "Episode: 479 Total reward: 79.0 Training loss: 103.1366 Explore P: 0.2162\n",
    "Episode: 480 Total reward: 169.0 Training loss: 58.8788 Explore P: 0.2127\n",
    "Episode: 481 Total reward: 160.0 Training loss: 1.1934 Explore P: 0.2095\n",
    "Episode: 482 Total reward: 81.0 Training loss: 2.9244 Explore P: 0.2079\n",
    "Episode: 484 Total reward: 68.0 Training loss: 77.9688 Explore P: 0.2027\n",
    "Episode: 486 Total reward: 17.0 Training loss: 0.6864 Explore P: 0.1985\n",
    "Episode: 488 Total reward: 62.0 Training loss: 1.9978 Explore P: 0.1937\n",
    "Episode: 489 Total reward: 178.0 Training loss: 228.6335 Explore P: 0.1904\n",
    "Episode: 490 Total reward: 114.0 Training loss: 0.4453 Explore P: 0.1884\n",
    "Episode: 491 Total reward: 127.0 Training loss: 1.6523 Explore P: 0.1861\n",
    "Episode: 492 Total reward: 124.0 Training loss: 120.2207 Explore P: 0.1840\n",
    "Episode: 493 Total reward: 184.0 Training loss: 0.5913 Explore P: 0.1808\n",
    "Episode: 494 Total reward: 129.0 Training loss: 56.3829 Explore P: 0.1786\n",
    "Episode: 495 Total reward: 95.0 Training loss: 1.9883 Explore P: 0.1770\n",
    "Episode: 496 Total reward: 129.0 Training loss: 1.2513 Explore P: 0.1749\n",
    "Episode: 497 Total reward: 176.0 Training loss: 1.0322 Explore P: 0.1720\n",
    "Episode: 498 Total reward: 132.0 Training loss: 0.9320 Explore P: 0.1699\n",
    "Episode: 499 Total reward: 146.0 Training loss: 289.4379 Explore P: 0.1675\n",
    "Episode: 500 Total reward: 147.0 Training loss: 0.5124 Explore P: 0.1652\n",
    "Episode: 501 Total reward: 166.0 Training loss: 0.9444 Explore P: 0.1627\n",
    "Episode: 503 Total reward: 31.0 Training loss: 1.4756 Explore P: 0.1592\n",
    "Episode: 504 Total reward: 129.0 Training loss: 0.6077 Explore P: 0.1573\n",
    "Episode: 505 Total reward: 127.0 Training loss: 1.2508 Explore P: 0.1554\n",
    "Episode: 506 Total reward: 123.0 Training loss: 0.8265 Explore P: 0.1537\n",
    "Episode: 507 Total reward: 159.0 Training loss: 260.4604 Explore P: 0.1514\n",
    "Episode: 508 Total reward: 136.0 Training loss: 0.9311 Explore P: 0.1495\n",
    "Episode: 509 Total reward: 198.0 Training loss: 0.9262 Explore P: 0.1467\n",
    "Episode: 511 Total reward: 40.0 Training loss: 2.3126 Explore P: 0.1435\n",
    "Episode: 512 Total reward: 130.0 Training loss: 1.2985 Explore P: 0.1418\n",
    "Episode: 514 Total reward: 25.0 Training loss: 1.1655 Explore P: 0.1388\n",
    "Episode: 515 Total reward: 149.0 Training loss: 214.9246 Explore P: 0.1369\n",
    "Episode: 516 Total reward: 200.0 Training loss: 0.8085 Explore P: 0.1344\n",
    "Episode: 517 Total reward: 172.0 Training loss: 24.9451 Explore P: 0.1323\n",
    "Episode: 519 Total reward: 52.0 Training loss: 61.7503 Explore P: 0.1293\n",
    "Episode: 520 Total reward: 176.0 Training loss: 0.4361 Explore P: 0.1272\n",
    "Episode: 521 Total reward: 160.0 Training loss: 1.8377 Explore P: 0.1253\n",
    "Episode: 522 Total reward: 180.0 Training loss: 1.5684 Explore P: 0.1233\n",
    "Episode: 523 Total reward: 174.0 Training loss: 58.6258 Explore P: 0.1213\n",
    "Episode: 525 Total reward: 10.0 Training loss: 222.1836 Explore P: 0.1190\n",
    "Episode: 527 Total reward: 32.0 Training loss: 0.4007 Explore P: 0.1165\n",
    "Episode: 529 Total reward: 33.0 Training loss: 0.3054 Explore P: 0.1140\n",
    "Episode: 530 Total reward: 185.0 Training loss: 0.7425 Explore P: 0.1121\n",
    "Episode: 531 Total reward: 171.0 Training loss: 0.3441 Explore P: 0.1104\n",
    "Episode: 532 Total reward: 199.0 Training loss: 0.2333 Explore P: 0.1084\n",
    "Episode: 534 Total reward: 95.0 Training loss: 0.4929 Explore P: 0.1056\n",
    "Episode: 536 Total reward: 8.0 Training loss: 0.5416 Explore P: 0.1036\n",
    "Episode: 538 Total reward: 42.0 Training loss: 163.3946 Explore P: 0.1014\n",
    "Episode: 539 Total reward: 180.0 Training loss: 0.2803 Explore P: 0.0997\n",
    "Episode: 540 Total reward: 193.0 Training loss: 0.4929 Explore P: 0.0980\n",
    "Episode: 542 Total reward: 36.0 Training loss: 0.7983 Explore P: 0.0960\n",
    "Episode: 544 Total reward: 152.0 Training loss: 41.4165 Explore P: 0.0930\n",
    "Episode: 546 Total reward: 30.0 Training loss: 0.7570 Explore P: 0.0911\n",
    "Episode: 548 Total reward: 50.0 Training loss: 0.3215 Explore P: 0.0891\n",
    "Episode: 550 Total reward: 79.0 Training loss: 0.5349 Explore P: 0.0869\n",
    "Episode: 552 Total reward: 38.0 Training loss: 0.3635 Explore P: 0.0851\n",
    "Episode: 553 Total reward: 131.0 Training loss: 0.3965 Explore P: 0.0841\n",
    "Episode: 554 Total reward: 135.0 Training loss: 0.2453 Explore P: 0.0831\n",
    "Episode: 555 Total reward: 111.0 Training loss: 0.9434 Explore P: 0.0823\n",
    "Episode: 556 Total reward: 136.0 Training loss: 0.7058 Explore P: 0.0814\n",
    "Episode: 557 Total reward: 106.0 Training loss: 0.4755 Explore P: 0.0806\n",
    "Episode: 558 Total reward: 98.0 Training loss: 0.4107 Explore P: 0.0799\n",
    "Episode: 559 Total reward: 102.0 Training loss: 62.3874 Explore P: 0.0792\n",
    "Episode: 560 Total reward: 92.0 Training loss: 0.4026 Explore P: 0.0786\n",
    "Episode: 561 Total reward: 86.0 Training loss: 0.3649 Explore P: 0.0780\n",
    "Episode: 562 Total reward: 83.0 Training loss: 0.5843 Explore P: 0.0774\n",
    "Episode: 563 Total reward: 100.0 Training loss: 0.1493 Explore P: 0.0768\n",
    "Episode: 564 Total reward: 102.0 Training loss: 0.4021 Explore P: 0.0761\n",
    "Episode: 565 Total reward: 60.0 Training loss: 0.3445 Explore P: 0.0757\n",
    "Episode: 566 Total reward: 63.0 Training loss: 0.2461 Explore P: 0.0753\n",
    "Episode: 567 Total reward: 59.0 Training loss: 0.2115 Explore P: 0.0749\n",
    "Episode: 568 Total reward: 73.0 Training loss: 3.2738 Explore P: 0.0744\n",
    "Episode: 569 Total reward: 70.0 Training loss: 0.4267 Explore P: 0.0740\n",
    "Episode: 570 Total reward: 53.0 Training loss: 0.8779 Explore P: 0.0736\n",
    "Episode: 571 Total reward: 88.0 Training loss: 187.5536 Explore P: 0.0731\n",
    "Episode: 572 Total reward: 54.0 Training loss: 0.3208 Explore P: 0.0727\n",
    "Episode: 573 Total reward: 87.0 Training loss: 0.2894 Explore P: 0.0722\n",
    "Episode: 574 Total reward: 58.0 Training loss: 0.2578 Explore P: 0.0718\n",
    "Episode: 575 Total reward: 85.0 Training loss: 0.3401 Explore P: 0.0713\n",
    "Episode: 576 Total reward: 73.0 Training loss: 0.2245 Explore P: 0.0709\n",
    "Episode: 577 Total reward: 114.0 Training loss: 0.3640 Explore P: 0.0702\n",
    "Episode: 578 Total reward: 94.0 Training loss: 0.7954 Explore P: 0.0696\n",
    "Episode: 579 Total reward: 114.0 Training loss: 0.2615 Explore P: 0.0689\n",
    "Episode: 580 Total reward: 80.0 Training loss: 0.4812 Explore P: 0.0685\n",
    "Episode: 581 Total reward: 125.0 Training loss: 0.8818 Explore P: 0.0677\n",
    "Episode: 582 Total reward: 109.0 Training loss: 0.2953 Explore P: 0.0671\n",
    "Episode: 583 Total reward: 98.0 Training loss: 0.4371 Explore P: 0.0665\n",
    "Episode: 584 Total reward: 119.0 Training loss: 0.4685 Explore P: 0.0659\n",
    "Episode: 585 Total reward: 96.0 Training loss: 0.3440 Explore P: 0.0653\n",
    "Episode: 586 Total reward: 172.0 Training loss: 0.1414 Explore P: 0.0644\n",
    "Episode: 587 Total reward: 104.0 Training loss: 0.3309 Explore P: 0.0638\n",
    "Episode: 588 Total reward: 85.0 Training loss: 0.2262 Explore P: 0.0634\n",
    "Episode: 590 Total reward: 104.0 Training loss: 0.3231 Explore P: 0.0618\n",
    "Episode: 591 Total reward: 148.0 Training loss: 0.4431 Explore P: 0.0610\n",
    "Episode: 592 Total reward: 135.0 Training loss: 0.1894 Explore P: 0.0603\n",
    "Episode: 595 Total reward: 99.0 Training loss: 0.2376 Explore P: 0.0579\n",
    "Episode: 597 Total reward: 172.0 Training loss: 0.4083 Explore P: 0.0561\n",
    "Episode: 600 Total reward: 99.0 Training loss: 0.1152 Explore P: 0.0539\n",
    "Episode: 603 Total reward: 99.0 Training loss: 0.3594 Explore P: 0.0518\n",
    "Episode: 604 Total reward: 149.0 Training loss: 0.1398 Explore P: 0.0511\n",
    "Episode: 607 Total reward: 99.0 Training loss: 0.3337 Explore P: 0.0491\n",
    "Episode: 608 Total reward: 180.0 Training loss: 7.4786 Explore P: 0.0484\n",
    "Episode: 611 Total reward: 28.0 Training loss: 0.1953 Explore P: 0.0468\n",
    "Episode: 614 Total reward: 38.0 Training loss: 0.3152 Explore P: 0.0453\n",
    "Episode: 617 Total reward: 50.0 Training loss: 0.4420 Explore P: 0.0437\n",
    "Episode: 620 Total reward: 9.0 Training loss: 0.3347 Explore P: 0.0423\n",
    "Episode: 623 Total reward: 99.0 Training loss: 0.1405 Explore P: 0.0408\n",
    "Episode: 626 Total reward: 78.0 Training loss: 0.1488 Explore P: 0.0393\n",
    "Episode: 628 Total reward: 198.0 Training loss: 0.9185 Explore P: 0.0382\n",
    "Episode: 631 Total reward: 99.0 Training loss: 0.3505 Explore P: 0.0368\n",
    "Episode: 633 Total reward: 142.0 Training loss: 0.1654 Explore P: 0.0359\n",
    "Episode: 635 Total reward: 134.0 Training loss: 0.3178 Explore P: 0.0351\n",
    "Episode: 637 Total reward: 104.0 Training loss: 0.3331 Explore P: 0.0343\n",
    "Episode: 639 Total reward: 73.0 Training loss: 66.8497 Explore P: 0.0337\n",
    "Episode: 641 Total reward: 35.0 Training loss: 0.1411 Explore P: 0.0331\n",
    "Episode: 643 Total reward: 83.0 Training loss: 0.2136 Explore P: 0.0325\n",
    "Episode: 645 Total reward: 62.0 Training loss: 0.2303 Explore P: 0.0319\n",
    "Episode: 647 Total reward: 28.0 Training loss: 0.2552 Explore P: 0.0314\n",
    "Episode: 649 Total reward: 4.0 Training loss: 0.1967 Explore P: 0.0310\n",
    "Episode: 650 Total reward: 194.0 Training loss: 0.1424 Explore P: 0.0306\n",
    "Episode: 651 Total reward: 170.0 Training loss: 0.1509 Explore P: 0.0302\n",
    "Episode: 652 Total reward: 150.0 Training loss: 0.2699 Explore P: 0.0299\n",
    "Episode: 653 Total reward: 161.0 Training loss: 0.2821 Explore P: 0.0296\n",
    "Episode: 654 Total reward: 148.0 Training loss: 0.4859 Explore P: 0.0293\n",
    "Episode: 655 Total reward: 142.0 Training loss: 0.1541 Explore P: 0.0290\n",
    "Episode: 656 Total reward: 140.0 Training loss: 0.0963 Explore P: 0.0288\n",
    "Episode: 657 Total reward: 144.0 Training loss: 0.3165 Explore P: 0.0285\n",
    "Episode: 658 Total reward: 160.0 Training loss: 0.2059 Explore P: 0.0282\n",
    "Episode: 659 Total reward: 127.0 Training loss: 0.0918 Explore P: 0.0280\n",
    "Episode: 660 Total reward: 124.0 Training loss: 431.4700 Explore P: 0.0278\n",
    "Episode: 661 Total reward: 127.0 Training loss: 0.2660 Explore P: 0.0275\n",
    "Episode: 662 Total reward: 133.0 Training loss: 0.4122 Explore P: 0.0273\n",
    "Episode: 663 Total reward: 119.0 Training loss: 0.2070 Explore P: 0.0271\n",
    "Episode: 664 Total reward: 114.0 Training loss: 0.3453 Explore P: 0.0269\n",
    "Episode: 665 Total reward: 130.0 Training loss: 0.3865 Explore P: 0.0267\n",
    "Episode: 666 Total reward: 125.0 Training loss: 0.2518 Explore P: 0.0265\n",
    "Episode: 667 Total reward: 138.0 Training loss: 0.1668 Explore P: 0.0263\n",
    "Episode: 669 Total reward: 42.0 Training loss: 0.3241 Explore P: 0.0259\n",
    "Episode: 671 Total reward: 105.0 Training loss: 0.1787 Explore P: 0.0254\n",
    "Episode: 674 Total reward: 99.0 Training loss: 0.2393 Explore P: 0.0246\n",
    "Episode: 677 Total reward: 99.0 Training loss: 0.2190 Explore P: 0.0239\n",
    "Episode: 680 Total reward: 99.0 Training loss: 2.5996 Explore P: 0.0232\n",
    "Episode: 683 Total reward: 99.0 Training loss: 0.3376 Explore P: 0.0226\n",
    "Episode: 686 Total reward: 99.0 Training loss: 0.5884 Explore P: 0.0220\n",
    "Episode: 689 Total reward: 99.0 Training loss: 0.1356 Explore P: 0.0214\n",
    "Episode: 692 Total reward: 99.0 Training loss: 0.0920 Explore P: 0.0208\n",
    "Episode: 695 Total reward: 99.0 Training loss: 0.1880 Explore P: 0.0203\n",
    "Episode: 698 Total reward: 99.0 Training loss: 0.0951 Explore P: 0.0198\n",
    "Episode: 701 Total reward: 99.0 Training loss: 0.1050 Explore P: 0.0193\n",
    "Episode: 704 Total reward: 99.0 Training loss: 0.1234 Explore P: 0.0189\n",
    "Episode: 707 Total reward: 99.0 Training loss: 0.1407 Explore P: 0.0185\n",
    "Episode: 709 Total reward: 160.0 Training loss: 0.0913 Explore P: 0.0182\n",
    "Episode: 712 Total reward: 99.0 Training loss: 0.1815 Explore P: 0.0178\n",
    "Episode: 715 Total reward: 99.0 Training loss: 0.1191 Explore P: 0.0174\n",
    "Episode: 718 Total reward: 99.0 Training loss: 0.1073 Explore P: 0.0170\n",
    "Episode: 721 Total reward: 99.0 Training loss: 0.1133 Explore P: 0.0167\n",
    "Episode: 724 Total reward: 99.0 Training loss: 0.0898 Explore P: 0.0164\n",
    "Episode: 727 Total reward: 99.0 Training loss: 0.1217 Explore P: 0.0160\n",
    "Episode: 730 Total reward: 99.0 Training loss: 0.2150 Explore P: 0.0158\n",
    "Episode: 733 Total reward: 99.0 Training loss: 0.0678 Explore P: 0.0155\n",
    "Episode: 736 Total reward: 99.0 Training loss: 0.0872 Explore P: 0.0152\n",
    "Episode: 739 Total reward: 99.0 Training loss: 0.1330 Explore P: 0.0150\n",
    "Episode: 742 Total reward: 99.0 Training loss: 0.1116 Explore P: 0.0147\n",
    "Episode: 745 Total reward: 99.0 Training loss: 0.1611 Explore P: 0.0145\n",
    "Episode: 748 Total reward: 99.0 Training loss: 0.1307 Explore P: 0.0143\n",
    "Episode: 751 Total reward: 99.0 Training loss: 0.0875 Explore P: 0.0141\n",
    "Episode: 754 Total reward: 99.0 Training loss: 0.1344 Explore P: 0.0139\n",
    "Episode: 757 Total reward: 99.0 Training loss: 0.0911 Explore P: 0.0137\n",
    "Episode: 760 Total reward: 99.0 Training loss: 0.1224 Explore P: 0.0135\n",
    "Episode: 763 Total reward: 99.0 Training loss: 0.0572 Explore P: 0.0133\n",
    "Episode: 766 Total reward: 99.0 Training loss: 0.0757 Explore P: 0.0132\n",
    "Episode: 769 Total reward: 99.0 Training loss: 0.0381 Explore P: 0.0130\n",
    "Episode: 772 Total reward: 99.0 Training loss: 0.1698 Explore P: 0.0129\n",
    "Episode: 775 Total reward: 99.0 Training loss: 0.0365 Explore P: 0.0127\n",
    "Episode: 778 Total reward: 99.0 Training loss: 0.1805 Explore P: 0.0126\n",
    "Episode: 781 Total reward: 99.0 Training loss: 0.1017 Explore P: 0.0125\n",
    "Episode: 784 Total reward: 99.0 Training loss: 0.1112 Explore P: 0.0123\n",
    "Episode: 787 Total reward: 99.0 Training loss: 0.0930 Explore P: 0.0122\n",
    "Episode: 790 Total reward: 99.0 Training loss: 0.0693 Explore P: 0.0121\n",
    "Episode: 793 Total reward: 99.0 Training loss: 0.0431 Explore P: 0.0120\n",
    "Episode: 796 Total reward: 99.0 Training loss: 0.1168 Explore P: 0.0119\n",
    "Episode: 799 Total reward: 99.0 Training loss: 0.1071 Explore P: 0.0118\n",
    "Episode: 802 Total reward: 99.0 Training loss: 0.1360 Explore P: 0.0117\n",
    "Episode: 805 Total reward: 99.0 Training loss: 0.0872 Explore P: 0.0117\n",
    "Episode: 808 Total reward: 99.0 Training loss: 0.1197 Explore P: 0.0116\n",
    "Episode: 811 Total reward: 99.0 Training loss: 0.0848 Explore P: 0.0115\n",
    "Episode: 814 Total reward: 99.0 Training loss: 0.0515 Explore P: 0.0114\n",
    "Episode: 817 Total reward: 99.0 Training loss: 0.1590 Explore P: 0.0114\n",
    "Episode: 820 Total reward: 99.0 Training loss: 0.2080 Explore P: 0.0113\n",
    "Episode: 823 Total reward: 99.0 Training loss: 0.1532 Explore P: 0.0112\n",
    "Episode: 826 Total reward: 99.0 Training loss: 0.0622 Explore P: 0.0112\n",
    "Episode: 829 Total reward: 99.0 Training loss: 0.0553 Explore P: 0.0111\n",
    "Episode: 832 Total reward: 99.0 Training loss: 0.0746 Explore P: 0.0111\n",
    "Episode: 835 Total reward: 99.0 Training loss: 0.1045 Explore P: 0.0110\n",
    "Episode: 838 Total reward: 99.0 Training loss: 0.0929 Explore P: 0.0110\n",
    "Episode: 841 Total reward: 99.0 Training loss: 0.1053 Explore P: 0.0109\n",
    "Episode: 844 Total reward: 99.0 Training loss: 0.0877 Explore P: 0.0109\n",
    "Episode: 847 Total reward: 99.0 Training loss: 0.0783 Explore P: 0.0108\n",
    "Episode: 850 Total reward: 99.0 Training loss: 0.0724 Explore P: 0.0108\n",
    "Episode: 853 Total reward: 99.0 Training loss: 0.1745 Explore P: 0.0107\n",
    "Episode: 856 Total reward: 99.0 Training loss: 0.0334 Explore P: 0.0107\n",
    "Episode: 859 Total reward: 99.0 Training loss: 0.0205 Explore P: 0.0107\n",
    "Episode: 862 Total reward: 99.0 Training loss: 0.0674 Explore P: 0.0106\n",
    "Episode: 865 Total reward: 99.0 Training loss: 0.1149 Explore P: 0.0106\n",
    "Episode: 868 Total reward: 99.0 Training loss: 191.0773 Explore P: 0.0106\n",
    "Episode: 871 Total reward: 99.0 Training loss: 0.1013 Explore P: 0.0106\n",
    "Episode: 873 Total reward: 156.0 Training loss: 0.2140 Explore P: 0.0105\n",
    "Episode: 874 Total reward: 185.0 Training loss: 0.1837 Explore P: 0.0105\n",
    "Episode: 876 Total reward: 45.0 Training loss: 0.1855 Explore P: 0.0105\n",
    "Episode: 877 Total reward: 55.0 Training loss: 0.0789 Explore P: 0.0105\n",
    "Episode: 880 Total reward: 99.0 Training loss: 0.0890 Explore P: 0.0105\n",
    "Episode: 882 Total reward: 185.0 Training loss: 0.0788 Explore P: 0.0105\n",
    "Episode: 885 Total reward: 99.0 Training loss: 0.1397 Explore P: 0.0104\n",
    "Episode: 888 Total reward: 99.0 Training loss: 0.0400 Explore P: 0.0104\n",
    "Episode: 891 Total reward: 99.0 Training loss: 0.0962 Explore P: 0.0104\n",
    "Episode: 894 Total reward: 99.0 Training loss: 0.1356 Explore P: 0.0104\n",
    "Episode: 897 Total reward: 99.0 Training loss: 0.2037 Explore P: 0.0104\n",
    "Episode: 900 Total reward: 99.0 Training loss: 0.0486 Explore P: 0.0103\n",
    "Episode: 903 Total reward: 99.0 Training loss: 0.2492 Explore P: 0.0103\n",
    "Episode: 906 Total reward: 99.0 Training loss: 0.1467 Explore P: 0.0103\n",
    "Episode: 909 Total reward: 99.0 Training loss: 0.2217 Explore P: 0.0103\n",
    "Episode: 912 Total reward: 99.0 Training loss: 0.1772 Explore P: 0.0103\n",
    "Episode: 915 Total reward: 99.0 Training loss: 0.0898 Explore P: 0.0103\n",
    "Episode: 918 Total reward: 99.0 Training loss: 0.0552 Explore P: 0.0103\n",
    "Episode: 921 Total reward: 99.0 Training loss: 0.1267 Explore P: 0.0102\n",
    "Episode: 924 Total reward: 99.0 Training loss: 0.3037 Explore P: 0.0102\n",
    "Episode: 927 Total reward: 99.0 Training loss: 0.1654 Explore P: 0.0102\n",
    "Episode: 930 Total reward: 99.0 Training loss: 0.1975 Explore P: 0.0102\n",
    "Episode: 933 Total reward: 99.0 Training loss: 0.2122 Explore P: 0.0102\n",
    "Episode: 936 Total reward: 99.0 Training loss: 0.0754 Explore P: 0.0102\n",
    "Episode: 939 Total reward: 99.0 Training loss: 0.1481 Explore P: 0.0102\n",
    "Episode: 942 Total reward: 99.0 Training loss: 0.0895 Explore P: 0.0102\n",
    "Episode: 945 Total reward: 99.0 Training loss: 0.0690 Explore P: 0.0102\n",
    "Episode: 948 Total reward: 99.0 Training loss: 0.0942 Explore P: 0.0102\n",
    "Episode: 951 Total reward: 99.0 Training loss: 0.0567 Explore P: 0.0101\n",
    "Episode: 954 Total reward: 99.0 Training loss: 0.0665 Explore P: 0.0101\n",
    "Episode: 957 Total reward: 99.0 Training loss: 0.0645 Explore P: 0.0101\n",
    "Episode: 960 Total reward: 99.0 Training loss: 224.4461 Explore P: 0.0101\n",
    "Episode: 963 Total reward: 99.0 Training loss: 0.0508 Explore P: 0.0101\n",
    "Episode: 966 Total reward: 99.0 Training loss: 0.0792 Explore P: 0.0101\n",
    "Episode: 969 Total reward: 99.0 Training loss: 0.0754 Explore P: 0.0101\n",
    "Episode: 972 Total reward: 99.0 Training loss: 0.0655 Explore P: 0.0101\n",
    "Episode: 975 Total reward: 99.0 Training loss: 0.0686 Explore P: 0.0101\n",
    "Episode: 978 Total reward: 99.0 Training loss: 0.0361 Explore P: 0.0101\n",
    "Episode: 981 Total reward: 99.0 Training loss: 0.1777 Explore P: 0.0101\n",
    "Episode: 984 Total reward: 99.0 Training loss: 0.0633 Explore P: 0.0101\n",
    "Episode: 987 Total reward: 99.0 Training loss: 0.0559 Explore P: 0.0101\n",
    "Episode: 990 Total reward: 99.0 Training loss: 0.0543 Explore P: 0.0101\n",
    "Episode: 993 Total reward: 99.0 Training loss: 0.0833 Explore P: 0.0101\n",
    "Episode: 996 Total reward: 99.0 Training loss: 0.1037 Explore P: 0.0101\n",
    "Episode: 997 Total reward: 45.0 Training loss: 0.0619 Explore P: 0.0101\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing training\n",
    "\n",
    "Below we plot the total rewards for each episode. The rolling average is plotted in blue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def running_mean(x, N):\n",
    "    cumsum = np.cumsum(np.insert(x, 0, 0)) \n",
    "    return (cumsum[N:] - cumsum[:-N]) / N "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Total Reward')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzsvXecY3d57/95dNSl6WVndmd2Z3e9xd4167LGNo4hphhjCCWhcwkhJMa/OPlhILmBXHLh3t91QgqQkBCICQTDj+YACSTYxsYYXLCx1/Z6vfbWqTu9j6RRO+V7/zhFR9KRdCQdlZn5vl+vec3o6JSvNNLznKcTYwwcDofD4eTiavQCOBwOh9OccAXB4XA4HEu4guBwOByOJVxBcDgcDscSriA4HA6HYwlXEBwOh8OxhCsIDofD4VjCFQSHw+FwLOEKgsPhcDiWuBu9gGro7u5mQ0NDjV4Gh8PhbCiefvrpRcZYT6n9NrSCGBoawrFjxxq9DA6Hw9lQENG4nf24i4nD4XA4lnAFweFwOBxLuILgcDgcjiVcQXA4HA7HEq4gOBwOh2NJzRQEEQ0S0UNEdIqIXiCiD2nbO4noASI6p/3u0LYTEX2eiM4T0QkiuqJWa+NwOBxOaWppQUgAPsoYuxjANQBuI6JLAHwMwIOMsX0AHtQeA8DrAOzTfm4B8MUaro3D4XA4JaiZgmCMzTDGntH+jgI4BWAHgDcBuEvb7S4Ab9b+fhOArzOVJwC0E1F/rdbH4XBqg6IoiEQijV4GxwHqEoMgoiEAlwP4FYBtjLEZQFUiAHq13XYAuGA6bFLblnuuW4joGBEdW1hYqOWyOZy6k0gkkEqlGr2Mskkmk0gmkwCAubk5zMzMGI+dIpVKIZFIOHpOTnFqriCIKAzg+wBuZ4wVu60gi20sbwNjdzLGjjLGjvb0lKwU53A2FBMTExgbG2v0MspmfHwc4+Nqca4kSQBUS8JJFhcXMT8/7+g5OcWpqYIgIg9U5fBNxtgPtM1zuutI+63/xycBDJoOHwAwXcv1cTicjQNjDIzl3TNyakgts5gIwFcAnGKMfdb01I8AvE/7+30Afmja/ttaNtM1ANZ0VxSHw+Fw6k8tm/VdB+C9AJ4nouPatj8D8GkAdxPRBwBMAHib9tw9AG4GcB5AHMD7a7g2DqfmpNNpyLKMQCDQ0HXEYjEEAgEIglB0v1QqBUVRGr5eTvNQMwXBGHsU1nEFAHiVxf4MwG21Wg+HU29GR0cBAAcOHGjYGkRRxNTUFEKhEAYGBoruq8c+6r1euwoMQFO5mNbX1+Hz+eB2b+im2EXhldQcziZGF6iiKDZ4JdboCmxmprQ3uZmUAwBMTk5iYmKi0cuoKVxBcDgbgFQq5XjaaDPQ7AqsFBt13XbhCoLD2QCMjY0ZaaS1IB6PO37OWtzxN4sV0SzrqDVcQXA4DSSZTDZFYdyFCxdK78TZcnAFweE0kPHx8boUxuXe8Uaj0Q13F9xM622mtdQSriA4nC1GLBbD9PQ0lpaWAGBDxTa2imBuFriC4HC2AGrdqtr+Ym1tDUCmJYZVbIMxhmg0WtW1OBsfriA4nC2Afuc9OzuLWCxWcv/FxUVMT09jfX291kvbkGwVS4YrCA5nCyCKImKxWFZaJmOsoALQ93O64V61VCOYJUlyLFurUutqo8EVBIezRZiamsqKN8iyjMnJyQauKIMdwV/tXfvq6iqmpqaqOgegKoe5ubmqz7MR4AqCw2lS1tfXa3oHv1XcJDqMMUfeT1mWjb83e7yFKwgOpwkRRRGTk5O2WlAUo5gAawYFUa6AbYY1byW4guBwmhD9TrdYK4dYLFZSYBZ7vtniC6VwSjlwJWMfriA4nAbAGLOVTVSIRCKBqakpVDN2t5CCMAeuay1Myz1/NevhiqF8uILgcBrA0tJSVQFTXbin0+mi+xVz4RRSEJOTkxULU0mSmr7wjisK+2zeRuYcThOSSqVARE3RBdQcbM2lUutmZGSEC+BNRM0UBBF9FcAbAMwzxg5r274LQJ9G0g5glTF2GRENATgF4Iz23BOMsVtrtTYOp1HofZdaW1sbu5Aa0WwuqVqz2bOYamlBfA3APwL4ur6BMfYO/W8i+gyANdP+w4yxy2q4Hg5nw1EvgfiRu5/DFTvb8d+u2eXYOZ1ee7Xn049vNiXTzNQsBsEYexjAstVzpKrdtwP4dq2uz+Fw7BNJiPj5mcoD3tVSjtButIDf7FaDmUYFqa8HMMcYO2fatpuIniWiXxDR9YUOJKJbiOgYER2rJoODw+Hk86+Pjdb1eo0Q9o1WMBuJRimIdyHbepgBsJMxdjmAjwD4FhFZOmkZY3cyxo4yxo729PTUYakcztbhsfNLjV5CzahGMaRSKcvEgs1uTdRdQRCRG8BvAviuvo0xlmKMLWl/Pw1gGMD+eq+Nw2kWNrvgqYRGxhDGxsYwMjJS9+s2mkZYEK8GcJoxZnQJI6IeIhK0v/cA2Adg6/03OJwycUJYbjWXy1Z7vdVQMwVBRN8G8DiAA0Q0SUQf0J56J/KD0y8HcIKIngPwPQC3MsYsA9wcDsc+pYShz+dDqCXjzQ14hFovqWq4gK8fNUtzZYy9q8D237HY9n0A36/VWjicZqNaF5KTLihZE7guFyEtb6z+TOXA01zLh7fa4HC2OLpO8HtckBUGpUkFKBfs9YcrCA6nAVTar6gWQlJW1HMG3Kp7SZTqZ0VU8nqcKpjjlIYrCA7HJrIsO9aILpVKVXysKIolm/SVg6EgvKqCSG1iN5PTbPZsM64gOBybjI+PY3x8vKFrYIxhcXHR0ZGXsnZD7feUtiCcVJL1hlsO5cMVBIdjk2bowAo4P+hH0iwGv0cVB2m5sCC9cOFCUyjJRh6/leAKgsPZ4uj6QE9xTRexIKpxjRXDjtDmgr3+cAXB4WwgcoWkE0IzE4NQs94lhyyUZhP6PM21fLiC4HAchDGGRCLhyHnqhZQbpBYzCqIZhWkzrYkHqTkcjm3m5uYwMTGRlWWUSCQaJtTsXFe3IPxuVRyIPIupKJtdKZjhCoLDcRDdR28OJE9MTMCp1vROKxoiyigIPQZhQ0Fs5Dbd5c6ecMIi3KjwmdQcTh2oVXDXCXJdTPF04VnVzUCliqKS4+bn57G6ulrR9TYD3ILgcBykkBBKpVKQ5eoFL2PM8bt33dgJahbENx4fx2Ks8QrNqfcsl3Lev41a8+EUXEFwOHVAlmWMjtZ3WptdJKZqiPagB2G/6lQ4NRNx9BrJZLJsxTY2NobJycnSOzaQzR6P4AqCw6kTtbgbdgK97EFwufDJ3zgEAFBKyPJyUljT6TTGx8cxPz9f9nnMd/A8TbX+cAXB4dSAWt1Z1qRZnxaUFlyAT6umTknOKTM9YN9odw1XMOXDFQSH04TUtQ5Cu5TgIvgETUGIzqW6Ov1auICvH7WcKPdVIponopOmbZ8ioikiOq793Gx67uNEdJ6IzhDRa2u1Lg6nWppJQDmyFsEDAPB5fRBcBLeLkCrR8nsjC/1m+v81O7W0IL4G4CaL7Z9jjF2m/dwDAER0CdRRpIe0Y/5Jn1HN4WxGGGO49/lZzKyVdruYi+5qIdyY4MWk0oZwOAQA8HpcSDvoYqpoTbV4nTU4Jw9SVwhj7GEAdudKvwnAdxhjKcbYKIDzAF5aq7VxOI1mYjmB7z8ziW/+qnRn1NzsJ6cFnSQzMBAElyrsfG4Xkg4MDXK6F5NTMQRuQdinETGIPySiE5oLqkPbtgPABdM+k9q2PIjoFiI6RkTHnKpO5XDqzVxEtRwUJZPpY9XGu9rmfHb215vzmRVEsY6ulawD2Px325uReiuILwLYC+AyADMAPqNtt/rkWH4CGWN3MsaOMsaO9vT01GaVHE6VlGrPoPc70n+Pjo7iwoULefvVIzVW1Pp9e1yqOPC6BaTE5krJ5Xf9jaGuCoIxNscYkxljCoAvI+NGmgQwaNp1AMB0PdfG4djFjrAqNfFNF8rmYLBVzr/Tw4Gs0AcGCULGgigVpK41xd7jalttOKlsNrviqquCIKJ+08O3ANAznH4E4J1E5COi3QD2AXiynmvjcOqJbjmUK4hr0azv+ak1AIBAJhdTGR1dN7uQ3MrUrFkfEX0bwK8D6CaiSQCfBPDrRHQZVPfRGIAPAgBj7AUiuhvAiwAkALcxxprLxuVwHERXEHKpkuU68F8nZgAAbq0GwucRsLSeLnbIllEKVq9zq7x2oIYKgjH2LovNXymy/x0A7qjVejicZkJ3MUkVzF6olYAygtSCC2kHCuWqWWctXExOHV+rczUjvJKaw3EQuwJDtyBiKRmRpAjAXpZPLUaO5qLGIKwNeKczkezGBZx4nZtdmNcCriA4nDJxQtCYLYePfPc5AI1LA+1r9eNtVw4YjztCXsRSMmIpqeAxldQvNAvcgrAPVxAcTgMQLWIPlVgQTuwvKcyIPwDA7m61onpyOe7YNZxkswvlZoIrCA6nAYgNTiM1IykKPAIZCirkU0OT8bTcMNdOLVttcAVjH64gOJwKURQFoihWdKxVGqnZgqinEJNlBrcrIwqC2ujRYu02nG6jUc9zOUmzrsspuILgcCpkfHwcIyMjxmNRFG0XtllVKtfCxWQHUVHgFjLX9nv02dSFYxCNpNFCudHXrydcQXA4ZWKelGZmZGQEkmRPqKYkBS5XRiiLVRSmpVLVzY+WFQa3aS0BTUEk086UItUyZbUS9GumUqktJewrgSsIDqcA6XS6ZgJElBX4PZmvXyItV5TFlEqlMDY2VrL3UyEYYxDlbAUhuAgeNyFh4WKyWuNGac1tPqcoihgbG0M8bi8Qb+ecmxGuIDicHERRRDKZxOjoKJaX7XasL4+UpGCPli0EAHHRnoIoJJByrRm76MlU5iwmAAh43EgUsSCKCcaT0xH83l3HsKxVY+v7VpPGa76eE4VyuiuwWeeENwtcQXA4OYyMjGB8XJ3TUOmdeSlSooIWvwcfevU+AMB6kZqDWqK3+jDHIAAg6HEhWWFH14dOzwMATlxYrW5xNYTPlrBHzVptcDicwqRkBT63y8gYiue4mKwEz/NTa/jJaApX9XvR6c8W6JUKKlk7zuxiAlSL4snR5YrO2xZQR5iuxO3HRholaJslBbdZ4RYEh9MA0pICr9uFoFevOZCKumAeeHEOf//Tc/jSz4fxnycq74QfSYh46PS8qZ24ut2c5goAkyuq5aR3es2lmJDUFUSui6kcat3u2wkLoq2tbdMri4IWBBE9iwJDewCAMXZFTVbE4TQ5Trgl0pICn1swWRBKUQXx3IVVDHQEIIg+y+wiu2v61pMTODa2gm89OYG/eMulaGtRRYDbVChXLrnXDnhVZbNcoiOs1bH1wokeUC7X5r+/LuZieqv2+1YAAoBvaI/fAyBay0VxOPWGMQZJkuDxeGp+Lb1ILtvFVDwGIcoKWgMerAvukuNAzeQKODK2A0+NLWMtvQIg34K4dEcrnp+KlHUtHd0qqcaCyMXpILVT59LPsVnHqRZUgYyxYcbYMICXMcY+whh7Vvv5YwA31m+JHE5tYIwZldALCwtl1TFUgy50fW4XPIILHoFKprmKMoNHIPjcQlnDfHLpawsYf7cHvbjn5CyA/CD1my5XR8IncgLV+hqLCVY9rmHHgihErdNcK3Uxmfe/sJzA6OK6M4trUuzYSGEiukZ/QERXAwjXbkkcTn2YnZ3FyMgIFEUx8uHrkfaoT5HzutWvX8DrLhmDEBUFHsEFn4csFUQlAtVcnLejPZD1nE9bW7KCuRCKlhm1sl5ZG5J6UKmCMFfK/+5dT+GOH59ydF3Nhh0F8QEA/0JE54noHIB/AfB7pQ4ioq8S0TwRnTRt+xsiOk1EJ4jo34moXds+REQJIjqu/Xyp0hfE4VihKEqW8GeMIRKJGH+bt5eiEmH8L4+M4rHziwAyFoSuIEI+AeslLQgFHpcLfrdQVaM/s1JIa7UYVw114LqLurP287q1amqxfItKVxBL6ykoCttULiarVipTK9UV2zUzRRUEEQkAdjHGDgO4GsA1jLFLGWNP2Tj31wDclLPtAQCHGWMvAXAWwMdNzw0zxi7Tfm61/Qo4HBuMjIzg/PnzxuOlpaW6Xv+JkSX862NjAICnxlS/v36XHvQISKSKWy6ixOBxE3yC9TAfu4JOMlVNp2UFMgPaAt68/fS1pQpYEMWErKQ9lhSGCZstw+tNpRaEJEnGMXrX22cnVpxdXBNRVEEwdS707drfS4wx298qxtjDAJZztt3PGNNvSZ4AMJB3IIdTA3JdR9W2WCgHSc4WQv/53DTizIOu9lYAQMArYD3HxZQruAwXk5uQliq/gxYVBX6vABAgSQoUrdV3LrqCSBSYLFcM86iLE1NrFaWs1qvdd7nX0a1OANjVpVbCPzPRvAWB1WLHxfQTIrqdiPqJqFX/ceDavwvgXtPj3UT0LBH9goiuL3QQEd1CRMeI6NjCwoIDy+BwMuRl/TiQnVIoQ2nfNvVrFPK5ES/RGE+U9RiEALECoa0jyapC8AiEtMIgK8yYRW3G7SKAKmvYJ2sBdY9AeH4yIzydyvRx0mXlhIvpxWnrWpHNgJ1K6g9qvz9q2sYA7Kz0okT0PwBIAL6pbZoBsJMxtkREVwL4DyI6xBiL5B7LGLsTwJ0AcPTo0c1dpcKpGZXOcaiE9XR27AMA3nhku9H/KOgVsBpP5wkfSWaIpiS0B9xqFpNLdTFVGqRejYt47PwSWgMeeAUX0pIMSWHwaOswC3Ai9VqVBKllphYB7m0Pa4V2/WWfo5aYC+XKRZIk+Hw+AJmpgFMrtWnH0gyUtCAYY4MWP9Uoh/cBeAOA9zDtv8QYS+nuK8bY0wCGAeyv9BocTjEikUheOmst89jNFoSewdTdnkkEbPF7IMoM//Cz81nH3fX4GP7k355Tq5qZGjj2ul1Iy5UJt8eG1SB5JCHCI7ggykpeq28zPo8rz8VkleaaK2wVBgguFw72teDE5FpFge5calVQV0kMwvhbe2tmI4my2rVvJGyVAhLRQSL6TSJ6t/5TycWI6CYAfwrgjYyxuGl7jxYQBxHtAbAPwIj1WTic6shtwFdNLrwdzFlHCVHGGvOjo6PTdD719wMvzmUd98y4Gvz8x5+dR4vfjZfu6YRPcEGSFfzgmUlbazJv7wplgtEewYW0xCAxllcDoeN1u5DSrJ9yLC5FYRAIuHZvF+JpGU8Ml58QUMtWG+a/q1E8ombxMQbMriUrPk8zU1JBENEnoLp0vgTgdQD+Dpkq62LHfRvA4wAOENEkEX0AwD8CaAHwQE4668sBnCCi5wB8D8CtjLHa9FnmcIpQizvVlOnucno1CYm50BHMVGzrAeG8tWi/l9bTePUl29AV8hqpsfc8P1v2OvTg8Y2HtsGrWxA540bNeDUXUyQSwcjIiO3AvqQwuIhw+WA7Ql4BzzVZV9dqejGZj5Vlhlat79SFTZrqaicG8Q4AlwF4hjH2XiLqB/DPpQ5ijL3LYvNXCuz7fQDft7EWDmfDMBtJ4t7nZxBNZtwST46qd9OHd7SB0mqI7VUX9+J7T0/iFfu7Lc8DANta/ABUt0+lyNod76sv3oZzc1FVQRSxIHxuAQlJRjKp3h2bJ9cVE6wKUwPfbsGFsN9dcdvwelCdBcGwqyOA8wnga4+N4do9XZuu5YadT1tCS3eViKgFwCyAPbVdFoez8fnr+87gsfNLODGZyXJ57PwSBjuDGOjIVC57BBf62nwoFlroaVEDo2ZrQ1YKxwGs0A0ZgVThnZZ0C6KQglBnQpQrRNVxquqa/B4hy4LKxe65nSiUM1sNjriYZIaukBcfvXE/7n9xDrd96xnLGpWNjB0F8axW8fxVAMcAPAngmZquisOpI043b9NJFEhvvXywPe9O0yNkV0ib15GGYCgIczC0VIO/XBTtnIJA8LpdEGWmWRDWYsDndiFluvu3c3d838lZHJ9YhVtQK7H9bgHpCjKh6tXltToFocDtIrzv2iF87HUHcc/zs/jM/WcdXF3jKeliYozpaa5fIKKfAGhljHEFwdmy6J1fS+EiglXH/F3dwbxtHoGQzmkFwrRjo4rP6PoaTWSCxetpGS1+j7F/KSTNxeQigldwYU0WIStq+qwVHrcLiXXrO+JCSnU2oiYAvPuqQQCA3+NCWnauq6tT53GiWZ+kqPO7iQi3vmIvxhbX8eVHRvDBl+9BV9hX9RqbATtB6q8S0fuJ6CLG2HmuHDibETtdSnUWFhYwPDxccj9XjuD1uNXHrf781hZewZVvQWhLCfoy93GdpkyktXh2ZpEsy5aFXDr6U4JLdWulJAVMS0kF8i0En9uFWFLGajzTlbWUFSHJqjtsf1+Leg6PkGWFNIpchVaJgjgxuZqVaSbKCtyCyzjHKw/2gjFganXz1EXYcTF9B8BuAF/WGvZ9l4huq/G6OJy6Ue5dpFU2j9U5zLHf/nY/Wn3q3b4u8M3C1i1Q1uwFxhhELSjx+kszhWY3HerDO65SO9T8zU/O4ImRTArp+fPnjVnaVuhtuF1E8LrJaOVdOEjtwnI8jXfe+YT9Xk+KArPHyud2VTRTIvd6jaqDMD//+QfP4+uPjxkKJq25mHT62tREgplNlPJqp1DufgCfAvAnAL4M4FoAH67tsjicjY/Zgnjv1bsMC0KfuGbGk9OET9ZaYuzsDOLDr8nUjLpchJcOdRmPf6nVGOiCLJ0uPINB77IquAgelwsxLbvKqhcToBbmERhcYEaBXykkJZM2awSpJcV4nEslgr9aZeFEkHo9LYMxQtifse76WlUFMRfZPAqiZAxCizu0AXgKwCNQO7pWPhSXw9kimHscedwuQ3CGfflfO79bQErKuIzWEiJEmeG6i7rg0VpvA6pAC/syjwsPBc5HURhAqgXhMWVDvfZQn+X+PrcL7aQKu4SoFG0mqCPJSpZF4vdUN+CoVpSrILJjDwoiWiyo1e8xnusK++B20aYqmrNTB3EWwOVQq5vnAMwS0RJjLFX8MA5nY1CrLCaza8UjkCE4BQs/vt/rymrYpwes3YIrz+8f8JoUBooLuizBxphxba/mBxrqDhldSXMxp9QWa5ehX4OItOZ/meP8bhfSTRaDANT4wfeensR6WobyXMZleOlAG95z9a68483/m9X1FCJJCQxAq8mCEFyE3hYfZreSBcEY+yMAIKI2AL8NdTZ1L4BAseM4nK1OtoIQtLkLccP1ZBb8Aa+ARFqdNUBEkLRjrTqtukzHiWX0ZVJ7JKnH6hZEMYVotlwSNrq6EpHmYspsM1sQ5SjfWrTaMB9/ZjaC+07OIux3Y1lbcCwp4T+OT+HdL92Zp5RXtEC9AIbZhWVEk6oF0WKyIABgW5t/y7mYbgVwPYCroHZd/TpUVxOHs+mptDLWHGQGABcB779uCI+dX8Ql/fnd8oMeNxQt8OlzC0ZKqltLozSf18ya5uqwVSgnK4YFoccdiumXbAtCXU+xbC8igiSzLAvH76ksSJ1LKYUxvBBDUlTQFfaiv630vevEkmo1fOL1l+C6Ky8FAHz54RHccc8pRBIS2oLZ6cMrpoyxeFpCVLMgWgLZIrSv1Y8zc9GyXlszY8fF1AHgnwA8xRirfAo5h7MBsOOqsUMyRyh6BRfCPjdee6gPLoveR36PKlQXo2mkZQXdXWoevYuoqJJaTdj/SsqMGRlGXq2QzVyNnYtZQSRsuIl0F5OufBhj8LkFpKXyq7HL4ZfDS3jPv/wKgKqQjv/PG433UyfXjXhhOQ6PQOgKeQyrTc9Cmo0kDQWhY071jadkRBMiiNR4UpYF0erHw2c3z5waO1lMfwlABvBOACCiTiKquN03h9Noap1CyRjDH33rWQDAb10xgP/+2gPo7QgXPUbPbPqb+8/gjh+fwnpK9fl39GYHkPW1/v7Ld2OgIwBRYhBlxXarDV056U3/JFPdRK4i8uYoiEKKyhyDkBQlq/mf3+MCY+q1a1Uod2xsBUTA77xsCElRwXzEOjz64nQEd/z4FP7sByfw01Nz2NYWABEZ5zMriNxrmS2IF6bXsJoQ0RH0Zrn7AGBPTwjraRkXmnTUarnY7eb6SQCf0DYFAHyrlovicDYaZqG1GMvcbbYE3Njf14JgML962kxAu+PVU09HF2MAAI/bYymYr97dhVfs7wGgCm87wldhilGbod/lF6mrg89U0JA7Wa6Qiyl3Qp1+J58ucCEnLLbnp9awpzuEVxxQ34+FmLWCODG5ionlOFr8bhzoC+PGS7ZlXctIU7XIQlqNp6H/G356ah6PnFvEvt6WvP2u3aOmID92ftH2+psZO4VybwVwM4B1AGCMTQFwYuQoh9M0lKqkLkdgRZKZu02vxbS23GsCQMDrhnmPaa0a1y0UdjHpvv5S40p1ZDlTm6GvSyriYvKU6WICgLlIKuuu2udRaynue34GayXcYZUGpk9OreHSHW3o0dpbLBZQEOspGe1BDz72uovxqd84hJft7co6d2+rerzZglAUhlhKwsp6Gjs6spX8lbs68tZ2UW8YPS0+PFbBDIxmxI6CSDH1HWAAQETFb4U4nA2G0y4m85hOqywkK4EfyGnjHdFcGp6cRnrmIi9dQSTS9iwISVGMvkv6eeUiJoT5nLkxFSvGNbfKXCTTasJvml9x5y+cmQFmXtdCNIXZSBKHd7QZDQ0XovkKgjGG9bSEkFaDYvV++dwCOkPeLAXx2PAibv/OcTw/FUFX0IMrh1SlsMICeMlge945iAgv29uFx4eX6tZwsJbYURA/IKIvAGgjovcDuB/Av9Z2WRxOY7D7pY6lpILFVub5B3pBValsqEBOUFXPTsrNYjIT9KrCLm5bQWQ6t+rWQbEaNr0RIADEU6WbE85FVAvh1SbXjTlYHLNxDrskRRnX/uWD+PyD5wAAF/e3Gn2qClsQEsJed957lRtkNhe6jS+uG3+H/W5DwSqMEPZZN0o8OtSJxVhKHRW7wbETpP4rAP8F4EcAjgC4gzH2OTsn1xr9zRPRSdO2TiJ6gIjOab87tO1ERJ/X+j2dIKIrKntJHE5hKr2rMx83sRzH7d85jsdHrIcemt0xVneZVgQ82QmF0aQqbAu14gaAoCZ8P/fAWcyulRZGoqnK2WtYEIXfj54WHz75G5dge7t+50wPAAAgAElEQVQfkaRYMN1W/3t5XV3zoe1txnPdpq6mhVp6WFHK3fTY+UXMrCXxjSfU3lO7u0PwCOqkPisLAgDW0xKCWhV6oeLI7W1+/Oz0PPb/j3ux/xP34qenMs35fG6X8f4xAC1+6yTQy7X/+bNNNkmvEmyNp2KM3csY+zBj7HYA9xHRO2ye/2sAbsrZ9jEADzLG9gF4UHsMqONM92k/twD4os1rcDhVUUhYHBtbsczhf3FanQQ3vBCzPJ9uQfzdOy/LmgNdDH9Ofya9rbdQpA4iaGq5MWEja8bcJ8kIUpdQmIOdQXSFvEYWTzFLaCkuwiMQQqY6iL29ISO2YtUU0G4Ve+5zPz01b/ztc7uMAHNPi69oDCLkK25BfOjV+/D//PpefOD63cY5dbxuIStDK1xAQRzoa4Hf48KzEysFX89GoaCCIKIwEf0JEf0dEb1Su8O/FcAw1IrqkjDGHgaQe5v1JgB3aX/fBeDNpu1fZypPAGjXxptyOI5SrCW2zsRyHF/6xTA+fPfxvOdkJb8y2Py3XnVsriMwC1b9b/O23HTJtaR1DMJMllvKRnBdVjLT47zu0jEInfagF6s5rcWtWIql0R70Zr2uHpMFIUrO+eR/fiajIHZ1BY3ge3fYl5VFpqMoCtbTstEHq9D/7iUD7fjTmw7iT286iJsvzRY/Xrcrq/VVIReTR3DhJTvacXwTWBDFCuW+ATVz6XEAt0Ht5toC4O2MsWNVXHMbY2wGABhjM0TUq23fAeCCab9JbdtMFdfibGEURckrSkskEohGS1e66q6XiaU4okkRAXdG6OkB2/WUdWbPzFoSXSFvUeFuBZnEz3pKgh/5WUzpdNoQSGYFFLXh35dkBT5NQOprM3uYClkH7UEvIkkRaTETd1EUtfZClmWIoqo8ltZT6MgpMCPT0KRYKqNkGGMQRTFLWZv/X7lC1zygaT2VxsLaOjoDHkQSInZ3+Iwutt1BAccnojg1lZ1FFIunoCgMIZ8AxljWddPptGXx4tGBFvzY9JgpLGtdflNigSRJWefc2+XDz04vGOtaWU9jOe5s+7oWvw8DXcXra6qlmIK4iDF2KQAQ0ZcALALYxRiL1GgtVp/OvFsOIroFqgsKO3fyej2ONZFIBDMzMxgaGoLPl7mLTSTyffXmL72iKFiLi1kumxOTa7h6KBNL0AfgFJo/PB9Noq/db/mcXXRl4XFlN+tLpTJCxrzdThBZlPOD1NZfu2zagx6AAefGJ9ERVF1mS0tLWFrKFsJzMRkv6cm41PT39ZO/cQj/9MBJRBOZNa6srGBhIbvieHh4GIODg3k1I+l0GnNzaizA7XbjwmIE/a4orurvwFNjUQx6QxgdHQUAbKMIWHQet3zhPuvXEvJDkiRIkgRBECDLMmZmrO9Buyj7/5sQFUhab5JPvP5iQ6mkUilMT2c3uO6QV+FLLmJ4ZAQE4PbvHi94Q1EpV17Uj7/73Vc6es5ciikIQ90zxmQiGnVIOcwRUb9mPfQD0G3FSQCDpv0GAOS1FWeM3QngTgA4evToxs8j49SE9XU1+ySVSmUpCCs/tzl1dHFxER/9t+eyno8ls10WugVRKMCblhR0BrOzksrt6eSGAgbA5/XA5XJhaGgIADA2Npa132fefgQfvfs5WxlCapWz5mKyuGMuRHtAtQpW46KhIHJhjGEypuDXD+/E3r17sybuXb6zHS/f34N7Xpg32lrIsrWwTCQSeQpCtx46OzvR1taG0dg0FpUgXnpoL+4deRF7dg5g+/btAIDf6ejBJXuXLWMrXo+AVx7aAUVU/59+vx/pdLqoy/HzH2jHIyfO49+euoCkJBs1Lh2hzGdKX19XV5fxWeuZUbBwfA3+1i64XITxhA9vOrIdV+3uLHitctnead2F10mKKYgjRKTHDwhAi/aYADDGWKWv9EcA3gfg09rvH5q2/yERfQfA1QDWdFcUh1Mu5RS8TU1NFT3X/OQ4ZtszQiSl1TkUKjKTZFZwShtQWFkc3BbG8Jx6DyaAwePxoEMLcuuCx+PxGC4dAGgLeBDyCVhPFlcQK3ERs2spDGkuiXK8X3qqaiGLCQCiSQmizNDf2QK32w2PJ9vV5PcIkBV18JDfI1TU2TUcDsPr9WI5LSABL667eBDf3rkNh7a3Gi6zlhbgLb0lRJPflFmVs85cLm9pweLCAoAL2NMdxmPnVaunO5yvKIPBoKHc+rs7kIAX68wLgRES8OLlh3fmxTWanWIKwl76RRGI6NsAfh1ANxFNQm3Z8WkAdxPRBwBMAHibtvs9UCu2zwOIA3h/tdfnbC7MPX/skrtvJWmuksIMi0RWGJ4eV7NTpAKtUEVFyVMQpSqpAeANl27D32sK4qqhDvz+DRdn1SIUIuR1I5YuriD+5RH1jl5PwSUi9c7+koGS59cD2sWmys1rqaX6bInczCs9XhJPy3mN9Kwo9n+aXkuASK1ZGOysfd3unp4wPv1bl6I77MO+bSH87PQ8DvS15lXfm1+zXpW9EE0ZxZJ6Id9GoqCCYIxV7TBjjL2rwFOvstiXQQ2GcziWTE5OIh6P48CBA3W7ZpBEiKZqMnONg25B5AVUZZYXoLaj1PR9Du1oxQdfsRdtLfZGroR8QskYhL7Ec/OZAP1tN1yEcLh0kFMX7sXaduvFZXt6Cgwf8uhtQSR0hrwFFYDV+5QrgKdXE+gJ+7KaCdaa7rAPLpcL/W0BvOfqXVkV8lavRc/emo8mjQFK5oyujUL93mEOp0ricfsdMp3oqfSWy1W/tmgSjObBOVIB37UkM6PithL0JVq5P6wEaNDnLhmD6G9Xlc3vXDtU9nq8dhREJAmv4MJAh/Udvc/tAsF+3ygzuQpiZi1pvJ56UEi5F+vfpVsL85GUUbi3ES0IriA4m5pqXEzqBLjsqW26BeFxU5aLyXxetWK5sAVh10VmlXppuU6/B2vx4o3wFIWhPejBFVqDuXKwoyBm1pIY7ApY3lnnupjMz+Vix4KYWk1ge1t1WWLVUspl6PcIaAt4sBBTFUTIKxh9oDYSXEFwthTlKIiLtqnuF9FkKehtr1t8bssiM8YYJKVCF1POY6/XXhiwPeTBalzEN381jkdz2kzrr1dSmOUsbDtr1Nt+vzgTKVg9Hk9L6AwWvkP2ujMuJjsUKmRLSTImluIFXVm1pNQ8jFx6WnyYj6QwH01uSOsBKF5JvUJEyxY/K6bsJg6nKbHrYjo3H8OXHxkx7mz1eMNNh/vQ1+qH20VZMYiklskT9nsss5j0bcWymHRyBU6r1rpBD7yGQvaEYEfQB4UxPHR6AV97bMxyn9w5DeWgWxDPTqziL+85bbmPKCvwegrfb6ouJma46Cq1IIbn1yEpDAf7Gj9xoFiQGlDnS0yvJbAQTaG3pbEWT6UUs3m667YKDqdGmGsc9Mdm/upeVeBdWI7jf7/pMGJaMZN+x+cRXFktIqJaOmmb341Fi6ZwoqIXuJXOYsplsDOIj998EENdhRWD1Xk6gp6sKmwrZMYgWCgtu8Fzj0BZrrZc0pKCsFvIOkbHysVUDmYBfHpWzfI62Jc/rKdWlHqPCim7/dta8K0nx9HX6s9qYLiRKKjyGWOy+QdAG4Btph8Op+mZnp42qmyBwl/m6VU1C+f5SbV/TovmLw56BURNLSJm1pJwuQjbWv3WFoRmbVQag9jbEy77Tr9Q8ZoZ2YaLqRjuEvEQUWZZrT9yMRSE1ia9Ugvi9GwUXsGF3d31dzHlUkpxXNzfgqSoYGwpXpd03FpgZ+To64noLNRK519pv39W64VxONWQFTQWRcvtVnz3KbUdWKtWPdzf7sfZ2ShkhWF4IYb7Ts5CURi8HpelgtBHhoZ8pXP9iyEI9o9vD3pKNsyQZaViFxMAJIoUyQGqiym3vsH8Xutprivz0zh79mzJ6xWy+k7NRLBvW7hoG/RaUUjJF3IxXdyfcYO9fN/GdMjYeZfvAHAdgDOMsUEArwXw81ouisOpFcXaKvz0xTls07Jj9mpB0CMD7ViJizg9G8X5+UyA1ucWoCgMoqxk3REvaTMROkPZQUk7hXI6oVAIu3fvtv2awjayY2RmPd3ONiZd+PDZhbyn07JS0ILQXUwEIJFMGttsX9okgCdXEhhqAusBKD2m9qJe1RoMeQUcHXKuxUY9saMgJMbYAgAXERFj7AEAfJgPZ0OSTOYPpNf53jOTmFtL4pUHe4wv/9Xa3OKxpXVjyE7YJyCkTXP7X//5AoaHhzEyoo7TXNXSTXO7mpaDz+craEFYKRU7gl9WqrMgzHz98fE8oShKimElWKFOxitejV0I87Vm1hLob61vwLfSGITfI+AlA2141cXb6lrU5yR2EnPXiCgE4FEAXyeieQDl/5c5nArQm7vViqBXMAKnkswggRntIgB1alvAIyCaEME0l9L/euNhnNUqkmfXUjlpmOpXI9fdUk4dRCXtQHKPz72GrNjLrLJLQlKMiXZElBeDyL0+EcHnFow+VqWyzKxcTNGkjKSooK/BNRCFsPq/fuMDVxsNEjcidtTamwEkAdwO1bU0BeANNVwThwNA7ch69uzZvLt+OwLUjlJ5fnItL6umNeDB0ZxisqBPwHpKRiyl9hFqC3oMCyJ3PXrGU+54zVoquVySYv79WzVprlaYW3vIWgSklKfL63YhXaCLazF0hTcTUVu1N0pBWCk9oPjnMexz2+o91azYURAf1zKZRMbYVxhjnwXwkVovjMPRG+TlznA4e/YsVlaqH+f4r78cy9t23UVdea6SkM+N9bSEtCwbd8nmqlhz+qdeVFfusCAnsWq7ISsM7goL5awwzzZwCVrHWXe2Wy3XGmgNeIzJdOW0QtEVhN7vKXcUaK0pFj8ioqotvmbGzqc4d6Y0ALze6YVwOOUQiVQ3mmRmLYmINvf5/7zlMK7d24Urdnbg1y7qyds37FV7HaUlxfAld4YywjBpyvARtWyh3BGi5QSpi2HnGLOC0IWXrDBjLGclfPzmgzi8I5OVkzBVRIfaOzCvhBAMFhfcvWEfFrR50eUGqYkIcxFNQTSZi6mSLsMbhWKV1B8komcBHCCiZ0w/5wC8WL8lcrYCi4uLOHPmjG3BUe2X8btPTRh/97X68YFf240/uGEvtrXmt0ToCnsxH0kiJWUydcyZQymTS0eUFCOYXau1W3G9KY3yO09O5D0vs+qC1Ht7wrjKlInz7IVVPKfNXE5JClLwwO8u7krpafVhIZqy/B/HRRn//xPjiCYyFsY3fzWOUzMRI/NsZi0JIjSsKnkzKoBSFPsk3w11VsM92m/957oibbw5nIpYXVWFjS4MUqlUUTdSqS9rqefLaZw20BFALCVjIZY2XEdEhL42VZmYB+mICoPH7Yy1UA7/7Zpd+LObDwIAvvf0ZN7zsoKqg6XmTJwHT83jH352HgCQElWBb/a1W9UJ9IR9kGSG1YSYF0g/NR3Bz88s4JFzagptUlTw0OkFfOb+s1kWRFeovm2+c1+L1XNb0sXEGFthjJ1njL0NQADAa7SffBucw3GYYumogP1Op7mIsoLP3n8WJ6fWAACvOFD646wrk0hChM/Ub+jdL92lrjXHgvCUWFstspgEF2FPTxjX7+u2tBRkRanKxQTA0jKaWI7jTf/0GAAUraQGVEsMAJZiairwfDSFUzOqq3B2TY0zvTCt/l8WY5n/f0ZBpCwtvEazJV1MOkR0G1RrYqf2czcR/UGlFySiA0R03PQTIaLbiehTRDRl2n5zpdfgbC6c8t8DwNhSHC/ORLCekuF1u/Dea3aVPEYXjLGUZHQ2BWAoi6Qptz8tK5YWRCmcEi4egZAU8zOFJAcsCKvA+w+emTRmc5fK1gl51ecTaRmMMfzwuRl84aFhnJ6JYGJFVRC64l6MZtqX6woikhDRXkV9iZNslSC1HTv7gwBeyhiLAQAR/QWAXwL4p0ouyBg7A+Ay7VwC1LTZf4c6YvRzjLG/reS8nM3L3NxcXmfTSu/CY8lM242AhUALBoN5g4k82p2xojDjbyBzx5wSzUFqBrdFkZsdl5gTgsbrFpCSsoUroLrurHoxlaOYrHY9ORUBgxq89ud0c83NYgroCkJ7v+ajKSRFGX97f6b1xthSHGtxEUvrGQsiKUrwuAiRpIg9NibgOY1ZGdjZbzNhx04nAKLpsYj81vWV8ioAw4yxcYfOx9mkpNPFB+LYxZyemSvQAGBgIH9Gs9m14g61m45XBZ65OjielhAo0vYasC9wKsEjuJCS5DxlIyulK65LrSe3mWtuLMBXJEhNRPB71PvRhKiuT3c16fS3q8HnZy6sZFkQM6tJEBGiSQmtgY03dGcjUyyLSf9PfAPAE0T0CSL6BFTr4S6Hrv9OAN82Pf5DIjpBRF8lovJHX3G2DJUK13VTeqZ5zoP5vLnWilkQBvyZDBpdQegunbWEiHNzMbT67Y0KLfR8NZaEXqCnKy1jYBBTIFRZm9GeI5w/9Kp9uO6iLvzaRWoGlblS26qoTFecSVGGJCtYWhez9rmoJwzBBTwzvmKM6QSAO378IpbXRUQSIlos3ttGshmtBjPFPjFPAgBj7K8B3AIgDiAB4FYn3EBE5AXwRgD/pm36IoC9UN1PMwA+U+C4W4joGBEdW1jIbxrG2ZhYtVgoRqlpawVdTKYageUcAZWLz6cGRM0WhNnPrruY9P5Lf/xvzwEAwv7y73Kdi0FkhLAZRQGq7bQx0BHE//yNS4zHg11BvP+63fjyb1+Fr/7O0azupbmorTZcAKlB/eX1NMwrPDLQhpsO92NvTwjHxlbwwswadnapLbKjSRHHxpexnpYtlW+tsfu/2YzKopiCMF4tY+wpxthnGWOfYYw95dC1XwfgGcbYnHaNOa1iWwHwZQAvtTqIMXYnY+woY+xoTw9PqNqqLCwsIBazHn9ZDLOLqRR+zVowWxBmBaEHfR8+p4751HWSlRAzK6xqgu6l9vNqXVPNbi/GmNZqo/r00J2muQZ6LyafR8ArD+aPiMlV+kSEgFtAIi1jMZoEA2FvTwjX7+vGH71qH7a1+nB4exseH1nCubkYjgy04x/efTkIzJjX0VKB8nWKcvppbRaKvds9RFSwpYbWcqMa3gWTe4mI+hljM9rDtwA4WeX5ORsYO1/ASCSCsM2gpR4ENruYukL2Zj5nKQhvdq7//m1hTK8msiyToDffF19pWm655FoQjDHoYyuc6tX34dfsx9RKvPSOJvT/p9/rQlKSsBhLgTHg/b+2O6t1xsv2duFrz6gTjQ/2tSDgEdDqd2NqVc1y0ud0NAubXVEUUxACgDCcC0gbEFEQak3FB02b/5qILoPaeX4s5zkOpyz0u9a7fjmG6bUkPv46tYhsPZkR5P/9dQfhdrshSfm9i4DMl7/V71a/BUwX/hlr4GB/K87OxXD7d44b28zDbHp6euDz+eB2Z75qpSyIamIQqjuMZdVm6GmoTlgQAHBoeysObc+4k+wISUNBeAQkRAWLsRQAylPSlw22482XbccTL4xijzaToz3gweRKAoDXmNtdT7ayi6nYuz3DGPvftbgoYywOoCtn23trcS3OxsLpnPJHNPfPyekIDm9vRSwtozXgwVuvHChoQXg86l2qPpOBiNAb9mE+mtIydTIKpc3ijtZcBuFyufKC3lY4WQcBZKq7GWOQtPe0jCF1tim0bnParrlq2u924emxFYwsrGNHR8CytuJz77gM45PbkIqrLdXbAm6cX44D8DY0SF2qad9mxFYMgsNpVlZXV3HmzBnLSXFmZXNBK8RKpCQc3t6Kl+3tyttfp7e3Fzt27DBiEEDGzZRbO9ETzq/svebwXiPAXYxapbkSsqu7Fc2CyG0gWE/01zq2pLqmltfTuGKX9ZQ1IjJqJgDVgtDFUbMUym0ViimIV9VtFRxOhej9msxuovn5eSQSiaxArZ42GRflLOFjBREhHA5nCfCENjdie0cga98d7dmP33P1Tgz1tCAQULdX0nywKheTOz8GIcvOupiqQQ8yE4ArLRSEVTZba8ADBvW17e1pXKGc1bbNajnoFOvFtFzPhXA4gDMuJl1prCYyaazRpFqclRBlBL3l+7H1WdVDXdnuoragB+97WaZdR6VT25xMc1UtiEy2lu5iKtUBpNAairnIiq3bysX0sZsuxhsv245dXUEcGbRX6tSuufF2dQYbOrqzVNO+zUjjbyk4nBoRiWcURDKdRkJUAAYEvPY+9uYv/S0v34OPvGY/2oP5cYvr9/WgT2si58SgoGoL5cKUwtyFUSiKomYx6UHqCtdmdrWVi97CXX8ve1t9eOOR7fjzN1ySlRGmY2VBBH1uMBAGcqy3erFZhb8duILgbGh0QZJIJHDmzBmkUpkKXN2C6Ax5kBBlxLUU10osiLDPjUu2Fy4E0yN2pWZBFDzcwTqIFkpBlBXD7SYZWUwVLc2RO+dyaz/MCqLF7wYD8Ir9zVf3tNmVB29swtkURKNqxou50Z5e4dzXFsBCNIXj2oCbYIG5BblUUtBmnhdRDk5XUqdlhtHRUfj9fshaAF+g+t0P5jYftPteWllPh7e34m/fdgSvvHzI0TWWC3cxcTgNplL3ij6/2vxFXUgCPq8b3WF1ktl3nrwAAAj6nM/31LOGQhWeW0+tBap1MalfaVFLc00mk9BbTllZENUKtmLHmzPLyr1OrnK5dKC9KYVwM67JSbiC4Gwq5ubmjL9nI2m0tbcjnCO0Ax57hnOhL//Q0FDec1GtAK+vvXw/ORGhqyuTdmtWFuUS8LjQFnBjeHHd2KbodRAVzoOoVAhapR7bPW+ukmykIN7sSqAYXEFUAGMMZ86cwfIyT/RqZmYjSWxvC+Q1z6v0Lh/Qms75fHl3uB99zX7cdLgvy31l51wA8lJqq+kxRkS4cmcHTkyuIq2l+TpdSZ17vULIsmxrPzOFrKdmF9LNvr5K4QqiAvQ7o6WlpQavhFNIoDDGMB9Jor89kNc8z+486nICsPv7WvDWK/NnSdjFfK1q+zYd3tEGUWIYX1bjMbUMUldzjnIsiGZlsyoGHa4gqmCzfzgagVPzfVcTIiSZYXuOgnjrlQNZ1dCVXqeZO3vu1noYjSyo3W6VBlgQVvMg7GBVO1HO8bWg2f6/9YQriDqRTqdx5swZJBKJRi9lQ1LuHeVCRE133d6e7WK66XBf3r67d++2PIfdu95mEyCtfg+6wl6MaXEIw8VUYaGcE1RiQTSLgijGZq+o5gqiTujpl5FIpMEr2RroNRDdYV/JITOlhg9Z4aTwyhWGTtAZ8hrvgaSnuVZoQdRD+PX39+f1r2oWBbFZhb8duIKwYHh4GLOzs41exoZGURScOXMGa2trZR03Pj7uyPVTWquJkM+NcJGYQ6UZQ40SXnav1RrwIJpUFYQxD6IG33anCuVaW1sLdoDdCGyktZYDVxAWSJJUtmDjZKNnsCwuLpZ9LGOspEvpCw+dxw+PTxd8PqHVAQR9brgFwsX9Lfi96/NdSf39/QXPUanwq6RBn9O0+NyIJfVK6sZbEOXMjACa08Vk9X9thnXVEl5JXSc2SlaGE5w/fx7BoDqa0u7rNu9nTo+0el6SGZ6dWMVj4+t4zf4rLM+X0gvXPALiAD564wHL/YQyhyRY+ZzrJUDLocXvQSwlQ1aYUSjnrrAOohjVBKlLVVI3i4LY7EqgGA1TEEQ0BiAKQAYgMcaOElEngO8CGII6Ve7tjLGVRq2xEFtJ2FeCLMtG64tK3isrBWFmIZYq+jwAJCUFXjfB7a6u5qEQzf4Z0CevRZIivvroKADAVedCuWrOwRjLSvdtViHNg9S15QbG2GWMsaPa448BeJAxtg/Ag9rjTUG5H6D5+XkMDw/XaDX1oxJBmluBe/+Ls7jzF5n3Im6aK71YQFkkC8x92LVrl8XepdlofXj0zK2p1aSxze1ASm85z5V73mIxiGZ7j5ttPbWi0QoilzcBuEv7+y4Ab27gWhylXEG5srJScFbyRqISBZEbg7j7qUl85dFRJLTAc8I0Le3BU3N5xwNALCVZBqfLaV1tFgKVZDrlMjQ0hJ07d1Z9HjMDAwPYsWNH3nZ9MM+ySYEKJfJcmynNNff5aosHq8FqndW0Q9lINDIGwQDcT0QMwD8zxu4EsI0xNgMAjLEZIupt4Po4NWZ5eRkLCwvo6MgeHGNWDrFURknG0zICHgHJdMYFJcrWCmg1LqIz5HdM6A0MDCCZTBqCqlSnUivsjCEtl1AolNXiXEdXjiummRi1cDFVk8VktU8hC6KRCiIX8/9xs7uYGqkgrmOMTWtK4AEiOm3nICK6BcAtABy/G7NLJXfFm/UDVA0LCwsAMhPgrDgzGzX+TqQlIOTFyGIss03Mj1eMLq5jeD6Gqw+0OeYCcbvdCIetx11WG6uoRTxDn3mhtzwHAE8dYxB2g9RDQ0NZLsWNoCC2Eg171xlj09rveQD/DuClAOaIqB8AtN/zFsfdyRg7yhg7Wk1Ts3rT7EHNZiWiFXsRqS2111MS7n9BdSt1h32Ip7LdcLGUhDt+fAoA0Baw7wbYtWtXWTcczfT/tBK+IS3+sqIpiFtfsddy2l0lwj8YDBpWX7UWhM/nM+Z3ExGSySTGxsagKErTKAh9HVbT7ja7BdGQd52IQkTUov8N4EYAJwH8CMD7tN3eB+CHjVgfp3akUimcOXMG6XS64D7mL2DcsBDUedKxVMZi6A9lB6wB4O9/etb421XGl9bv9xuCSsdJ94mTx9nB63ZBcBFWE+r70xWuPIaSu06Px2OkMVdDodevu8yaRUFsZRr1rm8D8CgRPQfgSQA/ZozdB+DTAF5DROcAvEZ73PTMzs6WrADerHcY5aK3GonFYkX3M0aJmuINq/E0kprCuO2GvQj7BMS159OSgvtOzmJ0MTNRzi246vK+17M/UzlKK+gVDBeTp1QjpjKvWe7rq9ZNxRVEY2hIDIIxNgLgiMX2JQCvqv+KyiPXvZBbdX327Fn09PTkBV855ZPQ0lW73Wr1rpUAAB+iSURBVF48PxlBT4uahRTwCAh63JhMq80P731+Bv95Yibr2N+8vPL220DlsYV6u58KrTPkFTCrNS30llkQaOf8pZ4r97zFHjdroRx3MXHKQk/RnJ/PhE9kWc6adMYpjlnAJtIywj43dneHsBJPGz2W/F4BQZ9gWBhrSTHrHJftbEdLwN0wC6IZCHgz93+tgY3XNKFZ39etBFcQdSCZTJbeqQi6394qnbGZWF9fx7lz5wo+L0lS0Sl8jDFMrSSyFMTyehrtQQ+6Qj5EEqKRteT3CAh6VBdTPC1DMqW77ukJ4X3XFi6Is6obKBdzHnyjBVmh6+uzt4M+Af4yJt3ZPX+x5ypptWHnmGZjI6yxGriCqIByXQjVuhx0f73evsIuKysrRQW206ytrRWdQ6y3PC/E6OI6PvmjF/DFnw+DMQZRVjC2uI6+Vj+6wh6sJUSjSC7gFhDQBOD/++1nMbGSmbPxjqsG0eL3gIisM3xCoUpeXhY7d+505Dy1JKRZEF2h6uov6hljqfc17FBuo8HNBFcQZSBJUsmhP1bKoNTw9mIUUi5zc3OYmJgwHsdiMZw5cyarj9H8/HxV1y4Xt7s6N8ZiTA2o3vXLMazE0zg9E4WkMOzf1oKA1w1JYUYL66BPjUHoTC7HcWSwDb997S7s6bYW3E4GOs11EUSEwcHBigvhqr2BKCScOkNq5pLfU7uvuZMxCCfPXQuaKbW5XnAFUQa6YihW2GVFLT5Yq6urWYpKn49tlT66UT7Y5qrpLzx03nAnDXUHEdCE3Mp6Gh6B4BFcCOb0WtrZEcTL9/cUDG4ODQ05ul7z+xoMBmtSKW2HQorvqqFOAGqGVzXUq1kfdzE1HxsvctVAKq2KrYeAboYParVrODeXcaGlJRkpSX3ffG4BrpSaKfbIuUW0agVwub2WulsKTyQDMnEDp9+rRtdBFDrPrq4g3nZ0AIe2tzp6fnOV81awILayi4kriBxqIcxr4WLabMxFUnh0LAIfgIEOP9biElKi+vH0u4UsN0lIiz2YZ00DQE84/w6+ll9c/X9TT+FQ7rVeeyh/Bnc9sFrnwICadjw5OWnrmI0gdDfCGquBK4g6UE8LopAF0+wf5PlIEgyEI4PtkHxhzKzEkRTVNXvdLnhNbSL0PkO5FkRPa3an1kJB6nLYsWOHI51cNzJOfXZCoVBWh2K7FkSjP7tb2YLgMQiHcVoZbBULYnE9DcaA916zCyGvgHhaQlJS4HYR3AJlpWnqVcEtfjeu2NmBA31qsLg9J9ff5SpeSW0nqB4OhwsqiEZYEI2gWBZTNe1I7Ka5NlsV9Vb5TgLcgqgKu1+OZrAgakUkEsH8/Dz27t1b1XmWYim4BRfaAm6EfJqCEGXDtRQ0FX25kBFOf3BD4et2dnYWdO9t3769rNkQVuQqCH18abkKo5ECZyPcHTfL7AU+k5rTdHcHzbQexhiGh4exbds2tLS0AFDTbRVFKbrOxcVFxONxtLe3F9xnKZZCd9in9RByI5GWsZYQjYB0jykAzVD6PWltbS3qYtLX7yTd3d3weDwIh8NVF0duFKrpyVTqWLfbjb6+vobXm2wEJVormst2a3Jq1dt/bW3NsfGixa6fTqdx9uzZop1UiyFJEmRZzmojUuq6KUnG4uIiEolE0S/R0noavVoMIeQTICkMi7E02gOqe8ec0jrYYb+TaD2D1C6XCx0dHTXvAltvKimUy/082PleWLmY2traqq6vqSUb5X9YKVxB1IFSX47Z2VlIklSVtWCnsVw0GgVjLK+5oF30AT9W57Za++RKHLd981k8O7Fa9FxrCREjC+vo1ayEVp9qNVxYjqMtmHEvfOE9l+NjrzuINxzZbnvN9VAQVuh+840kQKrtuGqm0Htj3l+oooFgPdH/lxtlvU7CFUQO5g+2KIol9ynnuXKu7cT5zMiyeicPVC60SrX6yF3rvSdnAQDn52OWz+t87AcnAAC9reo8hsM7Mnn77SYF4XMLuKg3nFcgZ2cttaCY4Ojs7ERvb29Rt9pGpZr31uVyYdu2bdizZ0/ec82a5hoKhdDb24tiA8qaZa1OwxVEEdbX1+t6PScsiELnKNUHqVpyr7sQTeFXI2pjPrnE6xK1grgjO1VhOtiZcSH19vY5MpymFnR1daG3txetrfmFaERUlrvJKZrlesU+y+3t7bYCz80idPX/pVU2VbOssVZwBVEEp0xKp62KCxcuGH+Loohz584ZcYXp6emK/L/VknuNj//geePvR88tYjGWsmx5PqU12btiZwcu3aEqiJWlRbSRGuTd0dtZURaLeT07duxwvM0G0BglUOhaup++XmspNwZRi2twak/dFQQRDRLRQ0R0ioheIKIPads/RURTRHRc+7m53mubm5vDzExm6EzuB3R2drbkOex8MWRZxvnz5/Oa/tn9UpmtgWg0CkVRslqBM8bq2qRPv6ZVPOL6fd0QFYZP/PtJrMYzaxxfimMxlsKdD48AAA70hS0FQm+rv2pBEQ6HG9YnqV7s3LkTfX19TWtBVBKk3gg0SzFfrWhEeoAE4KOMsWdInUv9NBE9oD33OcbY3zZgTZiamsobg1nqQy1Jki3XTe554vE4ZFnG8vJy1mwCp2IQjLGsTCWz8AbUxn7d3d1ln7fUNXWeGss0MzzY34KOkBc/Oj6NieUE+loVfOz7z2cd2xny4IaDvZZfsm2tPqCCrKtmK64qhhN32x6PB21tbUacqVkgoopf32YVuhuJun+LGGMzjLFntL+jAE4BqH6CS5VYzUi288E2Z+MU2t/uF2RsbAyrq/kZPzqKothKh21E7YRZCZ2YVLOk9vSEcOlAO67fpwb3FqMpvDAdyTv2qt1dcOUIg33b1OronhZfSUHR29ubVajX29uL3t7eyl/MBqbZXEy1bObXDGx2C6Kht1lENATgcgC/0jb9IRGdIKKvElHDBzozxjA9PW15V/az0/NYiKayXDtra2sYGxvL2m9xcTErrTQtKwW/PIqiFB1NmkqlsnrZ6Gu0WrdVB04z5Q4fMiNJEmZnZzE6OmrpVppYXseRgTb82c0XI+gR0B5wwy0QFmMp3HtSdeH9/st3G/vLsuoOM6/5j165D39280H43EKee6izsxPbtm0zHnd0dGTFi3IfNyubIX2ylItpMwnOUokgm5GGKQgiCgP4PoDbGWMRAF8EsBfAZQBmAHymwHG3ENExIjpmlZdfCYX+4U+OLuHZ4Zk8BRFPS/jWrybw6XtPZx1rJdz1OQ2MMdx3chZv/IdH8fkH7U95K/VhtFJedvy/heIpjDGMjIyUzOBaW1vLcmOZR4kur4tZlc9EhO6wFzOrCSxG1WMuH8zo/8sG81NBg14Be3pUK6K1tTWrWMrn822K9FGfz4fe3l709/c7ds56CORyCkbLEaq5+2wm5bJRaUiJIhF5oCqHbzLGfgAAjLE50/NfBvBfVscyxu4EcCcAHD16tCpVzhjDhQsX0NGRbazMrCXw5//xgvH41lfsxcGDmefjaXWQzVrCuk7CionlBL739CQAL771qwm87ZJLbH0BzK6bano/2f3yiaIIURQxPz8PQRDQ2tpaUhiLsoK55TVs7+mEJDMkRTmv02pfWwDHtYI5wUXwul341Bsvgd8joFtr011oTUSEzs5Oo4K7Fm0yGoX5szc4OOjYeV0uV00TFey6VgRBgCRJjhbhNRPcxeQwpL6TXwFwijH2WdN2823UWwCcrPVaRkZGkEgkMD09nbX9G4+PZz2ei2b31VlPZtw8jDFEbCiKiDYqM0xp9LoTRe+ozGNDzfvZNW1zg9KVBr8TiQTm5uay1mPFVx4dxe3fOQ5JVrCWUC2EkF9NTdVrBI7uygjCj9+satuBjqChHEqhu2NCoVBFrR82AsFgsOqaD/19CIfDDW1yp3++9AIzfTyrnWM4zUMjXEzXAXgvgFfmpLT+NRE9T0QnANwA4MO1XARjLM+fr28fXVyH4CLceGgbAl4hTwHExYzA/P2vP42P3P0cTkyuYno1gYfPWru9EprVccPBHsgKw4On5hGJRAzlJCsZoW5u9FboLpAxVtCCKaYQZIXhM/efwdNjy3n7pNNpjI6O5p2jmLspJck4pmUtPTm6hD/VMpTCvuzOpvt6MwKiM2jdPruYkN+K/t9qqWcX4ULXFgQBnZ2dm8KCsPP53GzU3cXEGHsUgNW7eU+912JGkhnuOzmDvrYARJnh7VcN4MZL+nByag2LMfWuWBAELEYTeG5iKe/4zz943vj7kXOLiCRE3PJ6P/YGVCGq5/vv1BrNfeepC7h2bxeAKGIpCX9132mkJQWXDbbj9v7s1Ff9y2ZWFg+8OIe7j01ib08INx7qw2WD7RBc6tsaT0l51cv6OZ4cXcapmShenIniHa9+adYH2zxru5hwiadl/Pl/nMRvXrEja07DX9xzGkHtdPt7VTeQfv6usA9vv2oAT08n0d/dgfX1/Kwxuy43nb6+vqpbdm82zO9htbUHg4ODxjl27tyJtbU1I+HC7Xajt7e3oGVQaXq23bU1CxthjdXQvG0Sa4w5qAoA952cwX8cz7iaukOq62NnZxCnZ6NYW1vDfCSJP/2e2jfI53bhQ6/eD49AuOPHp7LONbqo3nH/5d2P4k9eewBed8ZQGzC1kViJp5GUFDx8dgEzq6rV8OCpeVx6YB6XtpNR8KZ/cRJpCZ+5/wwuH2zHLzRLZXhhHV/8+TDe8JJ+XHdRNz77wFlMRI5jV38P/uK1qqJhjBluoruPqVXYDIQzc1Ec7Mu0iTB/2M3KKPdL8KVfDGMtIeJfHxsztl2zpwsPDavC42/edsRosmc+9sZL+vDWa9SYRrkKQncxmYVIW1tbwf02K+V0iq3WgjC7uwKBAGRZzsrIy43dFVqHXbh12HxsWQWRm/3zwoyan390qAM72gM4omXW7O4O4YmRZfzwidPQe8QNdAbx3mt2Ym9PGGvxjJvnthv24gsPqXUKYb8bsaSEv/nJGfzWFeosXr9HwI72gLH/p370ouXanhxZxqVXtOLrj4/hQnwarzvUhV0thNNzKzg9E8XpmSj6WrN991OrCXzqRy8gJSkIEPDCdASMbcepmSj+8cnnsLS8gre9pBvRpISdXUGcXhRx898/grP/53Vwa+M8zRaEWUFIkmS4nk7PRvCiRS3D268awBMjqmXVYWqwZyUgCqV1mvdtb2/PCkbbcTH19fUhEAgUfH4rUK4FUSuqSXPt7FTbq2yEu/PNHqTesgpCR5IZvvHEGM7NxXDT4T689cqBrOcP9rUCBPzzL4bh0gbV/PGN+40snaAvI+wu39mBrpAXS+tpvO/aXZhYTuA/n5vGmTlVoP5/bz4Mr9uFz73jCD783eeyrtPT4sNf/ual+OwDZ3Fmbg1PjEp4+OwiVpkfpy7Mw0/Z8ZLZSAo3HOzBGy7djo9+7znMrCaQklShfmSwHQ+Oi3j43CK+8fg4VlgAHZTAnQ+rd3/vf9kQ/v3ZKTwwIWF0cR37thXPCopGo0in04ilJPztT84CAK4c6sDkchyirOCNR3ag1e/BP7z78rxRPlZfHPNdfjAYNKrRzfua6xzMzxUTelYWxVallAXR1tZWcdt3O1SiIPRjvN7/297ZB8dVXQf8d/bth3a1llbalVeSZSTLNjiOPzAYMMENDDQEuwSGYgY8pDAtmYQOaUimnQbapsRpOlMYJoRMO4zTNP1IGJMpzVDqZiAtoaXtBAMhHseBACYYfySOhY1tSZGltXT7x7tv9Xb3SVpJa0u7e34zO7vvvre797y7e8+95557TlTbcp5Q2/PxMti55xf83z535Puxi84rOb+oJc5DN69BbNfXkogUuHBG7Oj7ygva8s/JhjC9bUl+Y7kbzuLA8SFEoKnBfd+Chgh/e8fFfGxNB5+6spdoOMQtVjF1NDfwznuDvHrgJPGIw45PXIYj46P5Db3p/OtLl6RpTkS4aHELR065G/YuPC/F1SsWAoZv/fBdEjGHbde67pMDw6N5mTav7iAbGuBL/zbuzutxOjfK0VPDvN03wOe+s5uDfSd44c2+vFkrFBI+sXEJf3nTah7aspaNVs54xCERcchkMvm1gaANe579ure3tyDUiIjQ3t4eGFhPF6nLI6hD7urqKmtneSVHwTMx9Wnbzj/qfgZxxbIMS9uSrOhYQGMsHJhtrbUxysO3rOXbL+7nyvNLY8Jv/52LsevDbF7dwebVrsduzu4QPjWUozkeyS8ig/tnvHHdIuLxOJvWn8/JkyfJ5XIsbUsy/NNBXnrnGL3pOAsXRPMzl3uvWU5Xa4IXf36Mz37k/Lxn0NKFjbx6wDUPferDbpz9TMgdlWcaY6zrTtMUD3Nq6AxbL11MSISlbW4ax2NHDpPL5fKBA0fHDH+x8zV+dWp8h3hx7KRtN6zMK8Yg0ul03hOruKPwOgG//bqxsZHBwcF8BrEgVEGUR1AnH41GaWxsDMwEOJPPLkeRdHV1MTAwMKNscNVkrql1E1PdzyDaFsRY3dVMxAkV2LyLvWNaEhH+4OrlrOkq3TTmhIJzH/s70Ym6tVwuRzqdJhp1XT8v6Wnhoz1RHAzvHnOD+nU2u3VZ2dlESyLCN+5cz6rO8cVld8Yw/p0RJ0R32l1gzCRj5HI5LulpBeBDy9zRvohw+4Zu+n99mv/6yf58FNsX3uwrUA4AkfC4bOGQ0NE8tZ2/vb2dTCZTkk84qIP37t1ko07v3LmOUlut+H+Pleq8EokEmUymxPwXRCQSKWsR2085v4P5QD0NUuZ3S5wlihPKNzc3s3jx4oIf5kS7WqcbNvpLN34QgJUdpUllgLyJxT8SufsqdxawJJNgdHSUz193AfdtWlEwA/ETcUIlWdZuWd/FeekEd23sAWDrZd08fMta4j631Mt7W3FCwqP/+SbDuVEGhs/w+K4DAHz5plVct6qdr21dx2O3X8znN62gMebw+1ctpRhPucH4OoDjOKTT6ZIF6aAOPpvNkslkJl1gVgVRHmdzE6GIBLZppWhrawscVMxnanXm4FGXJia/ImhsbKS9vR0gH3ivubm5ZBSTTCYZGBigtbW1IGdEOp3Ox1sKojMV5wvXr6SjuXBG0tHRQTgcDrTVh0T48k2rWBALMzg4SFNDmKaGUn/zWCyWr/ODW9YwNjY+slnR3sSfX7+y4DP96TvB9aq67ZLFbH/xCM/sPUJbk1uXT1+9jPamhoIF++ULkzx627pAGcPhMGfOnGFsbKwkDEbxfQwaHYbDYdLpdEn5VO9TyqOcmVkx3n/iXOINKqqRWlUUdakg/HZR/0Yfb+pYbDfNZrN5T5twOEw2m80H5kulUrS0tHDixIkJY/F75h6Pzs7OKTvS9qaJN4B1dnYyNjaG4zgcPnwYID8zSKVSBSHDk0k3EY8np1+5gbuo/j/7+3lyz1FijJJujLK2q3wPkra2Npqamjh48CAjIyMlfxT/cSaTmbF3iioIl6k6oiCbeNB7vCgCQWsEsVhMvYgCqNUQL5NRlwrC39n4G3jBggXkcrmSP0cqlcpnJYvH4yQSibyCcBwnP/X2FIRfgQQRFGwu6IeWSqU4efJkic0zFosRjUYDbaHxeLxAQWSz2YJOwNs0Nzw8TFNTEydOnOCejV089MwbHBoQPr6h242+mskgIvT39xONRjl1qnTvA7g+6zB+T4PkaG9vJxaL6a7nClCpDsnbOFnNocaVs09dKghwg4gNDg4WJJyPRCITLsBNZAbx/2E7OztxHIdEIkE47HpE9fX14ThO/g/Z3d0d+PlBf1THcWhvby8Z9Qd9t3ecTCZpbW2lqakp0IukWPkNDQ2RTsb4q5tXk81m854uXvwcTwG0tLQwPDzM8ePHGRkZYdGiRQXrAZFIhNOnTweuEVRqNOopmnomnU4jIhOaNSdT1H5UQVSWWp1J1K2CaGlpmbaXhZ+urq6SztA/M0gmk4yNjeUzznV2dhIKhSYcRXsdsfc5/f39+U56ZGSEhoYGRkZGMMYULAr39PQwNDTE2NgYjY2NhEIh2tpcV9xyOtNkMsmxY8cQEVKpFNFolNHR0ZIffENDAw0NDSQSCfr7+0ti8GSzWaLR6KyjkU6Gmj1cBeDtMwnqlPwDgu7u7oKEVn5UQcyciSwQtUjdKohy8G/iKqYcT4tQKMTChQtJJBJTdtbeH9/Df/1k+aNjsdisRtVehxKNRhGRKeWKRCIFyszDcZyK57lWJmaiIHlehz86OppX6kF4g5uZ7FOoVxzHmTB0ea0qCv11TEI5MeynYjazlHNBOBwmk8nUVBKeesavIMpBF/+nR/HgqFYVg4cqCKVqXQuVUiZTENlsltHRUaLRaH7NSFEmQxWEotQQkymI4tSx6lVWOWp1JjHv5pcicp2IvCEi+0Tkvrmuj6JUE7NZU6j1uEJng1q/V/NqBiEiDvA3wEeAQ8DLIvK0MSY4cYKiKAXMZk0pGo2SyWQKXL+VyYlEIqTT6aoKDzId5pWCAC4F9hljfg4gIk8ANwKqIBSlTGazpqTrUdPD21Raq8w3E9Mi4KDv+JAtUxRFUc4x801BBBn0CuJJiMgnReQVEXnF24SmKIqiVJ75piAOAf44213AL/wXGGO+boxZb4xZ7+0YVhRFUSrPfFMQLwPLRWSJiESB24Cn57hOiqIodcm8WqQ2xpwRkU8DzwIO8E1jTGnSZEVRFOWsM68UBIAx5nvA9+a6HoqiKPXOfDMxKYqiKPMEVRCKoihKIBKUlaxaEJE+4N0Zvj0DBOcIrV1U5vpAZa4fZip3tzFmSjfQqlYQs0FEXjHGrJ/repxLVOb6QGWuH8623GpiUhRFUQJRBaEoiqIEUs8K4utzXYE5QGWuD1Tm+uGsyl23axCKoijK5NTzDEJRFEWZhLpUELWatU5EFovI8yLyuoj8VETuteWtIvIfIvKWfW6x5SIiX7P3YY+IXDS3EswMEXFE5McistMeLxGRXVbe79i4XohIzB7vs+d75rLes0FEUiLypIj8zLb35XXQzp+zv+u9IrJDRBpqra1F5JsiclRE9vrKpt2uInKnvf4tEblzpvWpOwXhy1q3CVgJbBWRlXNbq4pxBvhDY8wHgA3APVa2+4DnjDHLgefsMbj3YLl9fBJ47NxXuSLcC7zuO34QeMTK+z5wly2/C3jfGLMMeMReV608CjxjjFkBrMWVv2bbWUQWAZ8B1htjVuHGaruN2mvrfwCuKyqbVruKSCvwAHAZbhK2BzylMm2MMXX1AC4HnvUd3w/cP9f1Okuy/itu+tY3gA5b1gG8YV9vB7b6rs9fVy0P3JDwzwFXAztxc4q8B4SL2xs3COTl9nXYXidzLcMMZG4C3imue423s5dMrNW23U7go7XY1kAPsHem7QpsBbb7yguum86j7mYQ1EnWOjulXgfsArLGmF8C2OeF9rJauBdfBf4YGLPHaeCEMeaMPfbLlJfXnj9pr682eoE+4O+tae0bItJIDbezMeYw8DBwAPglbtv9iNpva5h+u1asvetRQUyZta7aEZEk8C/AZ40xpya7NKCsau6FiFwPHDXG/MhfHHCpKeNcNREGLgIeM8asAwYZNzsEUfVyWxPJjcASoBNoxDWxFFNrbT0ZE8lYMdnrUUFMmbWumhGRCK5yeNwY811b/CsR6bDnO4Cjtrza78UVwA0ish94AtfM9FUgJSJeKHu/THl57flm4Pi5rHCFOAQcMsbsssdP4iqMWm1ngN8E3jHG9BljcsB3gQ9R+20N02/XirV3PSqIms1aJyIC/B3wujHmK75TTwOeJ8OduGsTXvkd1htiA3DSm8pWA8aY+40xXcaYHtx2/IEx5nbgeWCLvaxYXu8+bLHXV92o0hhzBDgoIhfYomuA16jRdrYcADaISML+zj2Za7qtLdNt12eBa0Wkxc68rrVl02euF2TmaBFoM/Am8Dbwp3NdnwrKtRF3KrkH2G0fm3Ftr88Bb9nnVnu94Hp0vQ38BNdDZM7lmKHsVwE77ete4CVgH/DPQMyWN9jjffZ871zXexbyXgi8Ytv6KaCl1tsZ2Ab8DNgLfAuI1VpbAztw11hyuDOBu2bSrsDvWdn3Ab870/roTmpFURQlkHo0MSmKoihloApCURRFCUQVhKIoihKIKghFURQlEFUQiqIoSiCqIBTFh4iMishu32PSaL8icreI3FGB790vIpnZfo6iVBJ1c1UUHyIyYIxJzsH37sf1Y3/vXH+3okyEziAUpQzsCP9BEXnJPpbZ8i+KyB/Z158RkddsbP4nbFmriDxly14UkTW2PC0i37fB9rbji58jIh+337FbRLbbEPWKcs5RBaEohcSLTEy3+s6dMsZcCvw1bsynYu4D1hlj1gB327JtwI9t2Z8A/2TLHwD+17jB9p4GzgMQkQ8AtwJXGGMuBEaB2ysroqKUR3jqSxSlrhiyHXMQO3zPjwSc3wM8LiJP4Ya/ADf8yc0Axpgf2JlDM/Bh4Ldt+b+LyPv2+muAi4GX3ZBDxBkPzqYo5xRVEIpSPmaC1x6/hdvx3wB8QUQ+yOShl4M+Q4B/NMbcP5uKKkolUBOTopTPrb7nH/pPiEgIWGyMeR43gVEKSAIvYE1EInIV8J5xc3T4yzfhBtsDNxjbFhFZaM+1ikj3WZRJUSZEZxCKUkhcRHb7jp8xxniurjER2YU7sNpa9D4H+LY1HwlunuQTIvJF3Mxve4BfMx62eRuwQ0ReBf4bN5w1xpjXROTPgO9bpZMD7gHerbSgijIV6uaqKGWgbqhKPaImJkVRFCUQnUEoiqIogegMQlEURQlEFYSiKIoSiCoIRVEUJRBVEIqiKEogqiAURVGUQFRBKIqiKIH8P2mlauZjRdbnAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f93b1476080>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "eps, rews = np.array(rewards_list).T\n",
    "smoothed_rews = running_mean(rews, 10)\n",
    "plt.plot(eps[-len(smoothed_rews):], smoothed_rews)\n",
    "plt.plot(eps, rews, color='grey', alpha=0.3)\n",
    "plt.xlabel('Episode')\n",
    "plt.ylabel('Total Reward')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'Text' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-14-803cba1b0254>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mText\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0.5\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'Total Reward'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'Text' is not defined"
     ]
    }
   ],
   "source": [
    "Text(0,0.5,'Total Reward')\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![png](output_21_1.png)\n",
    "\n",
    "\n",
    "## Playing Atari Games\n",
    "\n",
    "So, Cart-Pole is a pretty simple game. However, the same model can be used to train an agent to play something much more complicated like Pong or Space Invaders. Instead of a state like we're using here though, you'd want to use convolutional layers to get the state from the screen images.\n",
    "\n",
    "![Deep Q-Learning Atari](assets/atari-network.png)\n",
    "\n",
    "I'll leave it as a challenge for you to use deep Q-learning to train an agent to play Atari games. Here's the original paper which will get you started: http://www.davidqiu.com:8888/research/nature14236.pdf."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}