Skip to content

Latest commit

 

History

History
195 lines (122 loc) · 13.1 KB

syllabus.md

File metadata and controls

195 lines (122 loc) · 13.1 KB

Motivation

Design my own syllabus for self-study during the program.

themes: techniques in generative text; learning song representations; language/sequence modeling enhanced by reinforcement learning or GANs

[OpenAI Scholars intro blog post] [Designing a syllabus blog post]

Week 1: Setup

Get and understand data; set up environment; build an n-gram model as a ‘shallow’ baseline; read LSTM, seq2seq, PyTorch tutorials

[June 8 blog post]

Resources:

  1. Deep Learning book, chapter 5 (machine learning basics)
  2. Speech and Language Processing book, chapter 4 (language modeling with n-grams)
  3. "The unreasonable effectiveness of Character-level Language Models (and why RNNs are still cool)" by Yoav Goldberg [blog]
  4. Deep Learning with PyTorch: A 60 Minute Blitz [tutorial]

Optional:

  1. "The Unreasonable Effectiveness of Recurrent Neural Networks" by Andrej Karpathy [blog]
  2. Goodman, J. (2001). A Bit of Progress in Language Modeling. [paper]
  3. Practical PyTorch Series 1: RNNs for NLP [tutorial]
    • Good recommended reading section as well
  4. Speech and Language Processing book, chapter 8 (neural networks and neural language models)

Week 2: LSTMs, part 1

Define metrics for evaluating language model (LM) output; train an RNN (LSTM) on some sequential text data

[June 15 blog post]

Resources:

  1. Deep Learning book, chapters 10 (sequence modeling) and 11 (practical methodology)
  2. Graves, A. (2013). Generating sequences with recurrent neural networks. [paper]
  3. Merity, S., Keskar, N. S., Socher, R (2017). Regularizing and Optimizing LSTM Language Models. [paper] [code]
  4. PyTorch: Generating Names with a Character-Level RNN [tutorial]
  5. Deep Learning for NLP with PyTorch [tutorial]
  6. Natural Language Processing book (draft), chapter 6 (language models)

Optional:

  1. course.fast.ai lessons 6 (rnns) + 7 (grus, lstms)
  2. Karpathy, A. (2015). Visualizing and Understanding Recurrent Networks. [video] [paper]
  3. Colah, C. (2014). Deep Learning, NLP, and Representations. [blog]
  4. Bengio, Y (2003). A Neural Probabilistic Language Model. [blog] [paper]

Week 3: LSTMs, part 2

Modify the LSTM to take context as part of the input and see if text improves!

  • Genres from Genius.com
  • Audio features from Spotify

[June 22 blog post]

Resources:

  1. Learning song representations with deep learning for structured data

Fun extra blog post #1: Train an LSTM on song titles and something stodgy, like deep learning paper titles

Week 4: seq2seq

Train a seq2seq model with VAE loss; compare to LSTM results

[June 29 blog post]

Resources:

  1. Deep Learning book, chapter 14 (autoencoders)
  2. Sutskever, I., Vinyals, O., and Le, Q. V. (2014). Sequence to sequence learning with neural networks. [paper]
  3. Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A.M., Jozefowicz, R., Bengio, S (2016). Generating Sentences from a Continuous Space. [paper]
  4. Introduction to Variational Autoencoders [video]
  5. course.fast.ai lesson 11 (seq2seq, attention)

Optional:

  1. OpenAI: Generative Models [blog]
  2. Introducing Variational Autoencoders (in Prose and Code) [blog]
  3. Under the Hood of the Variational Autoencoder (in Prose and Code) [blog]
  4. Kingma, D.P., Welling, M (2014). Auto-Encoding Variational Bayes. [paper]
  5. Practical PyTorch: Translation with a Sequence to Sequence Network and Attention [tutorial]

Week 5: Classification and Attention

Use LSTM-LM transfer learning to do classification; learn all about attention

[July 6 blog post]

Resources:

  1. Bahdanau, D., Cho, K., Bengio, Y (2014). Neural Machine Translation by Jointly Learning to Align and Translate. [paper]
  2. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., Polosukhin, I (2017). Attention Is All You Need. [transformer paper] [blog]
  3. course.fast.ai lesson 11 (seq2seq, attention)
  4. Distill.pub: Attention and Augmented Recurrent Neural Networks [blog]

Optional:

  1. Howard, J., Ruder, S (2018). Universal Language Model Fine-tuning for Text Classification. [paper] [blog] [code]
  2. How to Visualize Your Recurrent Neural Network with Attention in Keras [blog]
  3. textgenrnn

Week 6: Model interpretability, part 1

Compare attention-based method of interpretability to other more explicit methods:

  • LIME/Anchor/SHAP
  • Input gradients
  • "Seeing what a neuron sees" à la Karpathy's char-rnn [blog] [paper]

[July 13 blog post]

Resources:

  1. Distill.pub: The Building Blocks of Interpretability [blog]
  2. Ribeiro, M.T., Singh, S., Guestrin, C. Why Should I Trust You?: Explaining the Predictions of Any Classifier. [LIME project] [paper]
  3. Ross, A. S., Hughes, M. C., Doshi-Velez, F. Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations. [input gradients paper]
  4. Ribeiro, M.T., Singh, S., Guestrin, C. Anchors: High-Precision Model-Agnostic Explanations. [Anchor project] [paper]

Optional:

  1. "Awesome Interpretable Machine Learning": an opinionated list of resources facilitating model interpretability (introspection, simplification, visualization, explanation) [repo]
  2. How neural networks learn - Part I: Feature Visualization [video]
  3. DeepMind: Understanding deep learning through neuron deletion [blog]

Week 7: Model interpretability, part 2

Examine latent space and forms of bias in my models

[July 21 blog post]

Resources:

  1. World Models: Can agents learn inside of their own dreams? [demo]
  2. "Many opportunities for discrimination in deploying machine learning systems" by Hal Daumé III [blog]

Optional:

  1. Hardt, M., Price, E., Srebro, N. Equality of Opportunity in Supervised Learning. [paper]

Week 8: GANs

Improve VAE generations with a latent constraints GAN (LC-GAN)

[July 28 blog post]

Resources:

  1. Engel, J., Hoffman, M., Roberts, A. (2017). Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models. [paper]
  2. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. [paper]
  3. Jaques, N., McCleary, J., Engel, J., Ha, D., Bertsch, F., Eck, D., Picard, R (2018). Learning via social awareness: Improving a deep generative sketching model with facial feedback. [paper]
  4. course.fast.ai lesson 12 (gans)

Optional:

  1. Deep Learning book, chapter 20 (deep generative models)
  2. Yoav Goldberg. "Adversarial training for discrete sequences (like RNN generators) is hard" [blog]

Note that the Week 8 project will lead directly into my final (4-week) project!

Final project ideas

[Project proposal]

[Project notes 1: genre and inspiration] [Project notes 2: topic modeling]

[Project summary]

Music blog review generation

  • Structured, topical, specific to attributes of the song (similar to ‘essay writing’ goals).
  • Conditioned on song representation, source blog (?), sentiment...
    • Could choose a limited set of source blogs ("voices"): e.g., Pitchfork, most consistent blogger on HypeM Time Machine, etc.
  • Datasets: blog scrape, Spotify API track audio analysis and audio features, Genius API’s song tags, description, lyrics

Lyric generation

What do I mean by song representation? I want to encode data from the Spotify/Genius knowledge graph for the song (e.g., title, genre, similarity to other songs, audio features, lyrics, description...) and use it to condition/seed the generated text.