Multimodal Machine Learning- A Survey and Taxonomy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multimodal Machine Learning- A Survey and Taxonomy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Things that can be thought about when reading a paper\n",
    "- Why a certain experiment was done/ certain metric was chosen\n",
    "- Intuitions/inferences to validate\n",
    "- Ideas for real world application\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Abstract"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Modality** - It is the way in which something happens or is experienced.\n",
    "\n",
    "**Why MML?** - Aims to build models that can *process and relate\n",
    "information* from *multiple* modalities.\n",
    "\n",
    "**Challenges** - Broader\n",
    "challenges that are faced by multimodal machine learning, namely:\n",
    "  - Representation\n",
    "  - Translation\n",
    "  - Alignment\n",
    "  - Fusion\n",
    "  - Co-Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Focus in this paper on three modalities\n",
    "    - Natural Language - written/spoken\n",
    "    - Visual signals - images/videos\n",
    "    - Vocal  signals - encode sounds, prosody, vocal expressions\n",
    "- Aim - to build models that can process and relate information from multiple modalities. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# amogh comment - get the subpoints from the highlighted parts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " This paper identifies and explores five core technical challenges:\n",
    "- **Representation:**\n",
    "- **Translation:**\n",
    "- **Alignment:**\n",
    "- **Fusion:**\n",
    "- **Co-Learning:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Flow of the Paper sectionwise:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Section 2** - Applications and developments in five core challenges\n",
    "representation\n",
    "<br>\n",
    "**Section 3** - Representation\n",
    "<br>\n",
    "**Section 4** - Translation\n",
    "<br>\n",
    "**Section 5** - Alignment:\n",
    "<br>\n",
    "**Section 6** - Fusion\n",
    "<br>\n",
    "**Section 7** - Co-Learning:\n",
    "<br>\n",
    "**Section 8** - Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Applications: A Historical Perspective"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Audio Visual Speech Recognition (AVSR)**\n",
    " - Inspired by McGurk Effect\n",
    " - Vision of AVSR was to improve speech recognition performance (e.g., word error rate)\n",
    " - Experiments showed that the main advantage of visual information was when the speech signal was noisy (i.e., low signal-to-noise ratio) 4\n",
    " - ie Captured interactions between modalities supplementary rather than complementary\n",
    "\n",
    " <br>\n",
    "-  **Multimedia Content Indexing and Retrieval**\n",
    " -  Automatic shot-boundary detection - https://en.wikipedia.org/wiki/Shot_transition_detection\n",
    " - Video Summarization\n",
    "\n",
    " <br>\n",
    "- **Understanding human multimodal behaviors during social interactions**\n",
    " - Audio-visual emotion challenge (AVEC)\n",
    " - Healthcare applications -  depression and anxiety assessment\n",
    " \n",
    " <br>\n",
    "- **Media Description**\n",
    " - Image Captioning\n",
    " - Visual Question Answering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multimodal Representations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}