My Solution to Assignments of CS234

This is my solution to three assignments of CS234.
CS234: Deep Reinforcement Learning is an interesting class, which teaches you what is the reinforcement learning: Learn to make good sequences of decisions. This class provides some basic knowledge and insights of cutting-edge research in reinforcement learning. More details are as follows:

• Define the key features of RL vs AI & other ML
• Define MDP, POMDP, bandit, batch offline RL, online RL
• Describe the exploration vs exploitation challenge and compare and contrast 2 or more approaches
• Given an application problem (e.g. from computer vision, robotics, etc) decide if it should be formulated as a RL problem, if yes how to formulate, what algorithm (from class) is best suited to address, and justify an answer
• Implement several RL algorithms incl. a deep RL approach
• Describe multiple criteria for analyzing RL algorithms and evaluate algorithms on these metrics: e.g. regret, sample complexity, computational complexity, convergence, etc.
• List at least two open challenges or hot topics in RL

---

Note: If you consult my source codes that you may want to incorporate into your algorithm or system, you should clearly cite references in your codes.

---

Table of Contents

• Assignment 1
  • Bellman Operator Properties
  • Value Iteration
  • Grid Policies
  • Frozen Lake MDP
  • Frozen Lake Reinforcement Learning
• Assignment 2
  • Q-learning
  • Linear Approximation
  • Deepmind's DQN
  • (Bonus) Double DQN
  • (Bonus) Dueling DQN
• Assignment 3
  • R-max algorithm
  • epsilon-greedy q-learning
  • Expected Regret Bounds

Dependencies

• Anaconda
• tensorflow>=0.12
• matplotlib
• scipy
• numpy
• sklearn
• six

Author

@zlpure