Deep Learning for Text Analytics

The master module Deep Learning for Text Analytics (DELTA) is offered at the Humboldt-University of Berlin by the Chair of Information Systems.

DELTA introduces students to deep learning and natural language processing. We discuss algorithmic foundations, several deep learning methodologies, and their applications in business and society.

The module draws inspiration from several excellent resources including but not limited to:

• Natural Language Processing with Deep Learning by Stanford University
• Several awesome educational posts in Jay Alammar's blog
• The fantastic interactive e-book Dive into Deep Learning

We greatly appreciate the provision of this great content and highlight its contributions to the design of DELTA.

Teaching format

The module is offered every summer semester. Weekly sessions split into a two hour lecture session and a two hour tutorial session. The lecture introduces relevant concepts in the scope of the course. Lecture sessions are accompanied by Jupyter notebooks that demonstrate these concepts using the Python programming language and various Python libraries for deep learning and NLP. Providing working examples and executable codes, the Jupyter notebooks are meant for self-study. To deepen their understanding of the covered topics, students receive programming tasks as homework. The tutorial sessions provide a forum to discuss solutions to the homework task as well as general questions.

While the time and location of lecture and tutorial sessions may be subject to change, we aim at offering lecture/tutorial sessions as follows: Lecture: Thu, 10.15 - 11.45,
Q&A Session: Tue, 16:15 - 17:45,
Exercise 1: Thu, 12.15 - 13.45,
Exercise 2: Fri, 10.15 - 11.45

Outline

The outline of the module is as follows:

• Introduction
  • Course organization and logistics
  • (Supervised) Machine learning revisited [ Python demo, EDA demo, SKlearn demo] [exercise]
• Foundations of artificial neural networks
  • The perceptron model
  • Neural network training using backpropagation and gradient descent [demo] [exercise ]
  • Tweaking and tuning feedforward neural networks [demo] [exercise]
• Foundations of natural language processing (NLP)
  • NLP pre-processing pipeline, bag of words model, and extensions [demo] [exercise]
  • Word embeddings and the Word-to-Vec algorithm [W2V demo, W2V from scratch] [exercise]
• Recurrent neural networks (RNNs)
  • Processing sequential data using auto-regressive and state-space models
  • Recurrent cells and recurrent neural networks [demo ] [exercise]
  • RNN training using backpropagation through time
  • Modern RNN architectures [lstm demo, financial forecasting demo]
• Convolutional neural networks (CNNs)
  • Computer vision primer
  • Foundations of CNNs [demo]
  • Using CNNs for text data [external demo from MLMastery.com]
• Modern NLP
  • (NLP) Transfer learning [sentiment analysis demo]
  • Attention mechanism
  • The transformer architecture [BERT demo]
• Conclusions

Repository organization

The repository provides Jupyter notebooks that revisit concepts covered in the lecture and demonstrate their application using Python. The corresponding notebooks are available in the folder demos. The folder exercises provides another set of Jupyter notebooks, which task students to practice their Python and Deep Learning skills on programming exercises. The exercises relate to lecture chapters and the demo notebooks. The idea of the exercises is that students try to solve the programming tasks themselves; possibly together with peers in their study group.

More detailed information on the coures format, organization, and logistics is available on the DELTA Moodle page. That page also provides slides for lecture sessions and video recordings.