README_first.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "41c3e3bb",
   "metadata": {},
   "source": [
    "# Training HuggingFace 🤗 Models on an IPU using Paperspace Gradient\n",
    "\n",
    "Whether you are looking to generate images with Stable Diffusion, derive insights from text, or need to recognize audio samples, the examples in here have you covered. And by using Paperspace to access IPUs you can be up and running in minutes!\n",
    "\n",
    "## Getting Started with our interactive notebooks\n",
    "\n",
    "In this runtime, you will be able to run models on IPUs for training and inference on vision transformers (ViT) for images, BERT-Large for question answering, and text classification, GPT-J for text generation and OpenAI's Whisper for speech to text conversion, all from within a Jupyter notebook.\n",
    "\n",
    "You can find our notebooks in the folder directory on the left.\n",
    "\n",
    "For instance, familiarise yourself with IPU programming in our `Introduction to 🤗 Optimum Graphcore` notebook:\n",
    "\n",
    "<table align=\"left\">\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> <img align=\"left\" width=\"18\" src=\"./images/folder_logo.png\" /> natural-language-processing/ </td>\n",
    "    <td> </td\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> &nbsp&nbsp <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> introduction_to_optimum_graphcore.ipynb </td>\n",
    "    <td> Optimum-Graphcore BERT fine-tuning example.\n",
    "    </td>\n",
    "</td>\n",
    "</tr>\n",
    "\n",
    "</table>\n",
    "\n",
    "<br><br><br><br>\n",
    "\n",
    "For dedicated finetuning, inference, advanced ML and IPU related notebooks dive into the following notebooks:\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ee810189",
   "metadata": {},
   "source": [
    "<details><summary><big> Use 🤗 model checkpoints for inference</summary>\n",
    "\n",
    "Run inference on <b>Natural Language Processing</b> tasks using model checkpoints on the IPU in just a few lines of code by checking out these notebooks:\n",
    "\n",
    "<table align=\"left\">\n",
    "    \n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> <img align=\"left\" width=\"18\" src=\"./images/folder_logo.png\" /> natural-language-processing/ </td>\n",
    "    <td>  </td>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> &nbsp&nbsp&nbsp <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> sentiment_analysis.ipynb </td>\n",
    "    <td> Use the sentiment-analysis pipeline to quickly evaluate pre-trained models on the IPU.  </td>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> &nbsp&nbsp&nbsp <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> name-entity-extraction.ipynb </td>\n",
    "    <td> Use Gradio and HuggingFace pipelines to prototype a web application doing fast token classification.  </td>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> <img align=\"left\" width=\"18\" src=\"./images/folder_logo.png\" /> gptj-text-generation/ </td>\n",
    "    <td> </td>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> &nbsp&nbsp&nbsp&nbsp <img align=\"left\" width=\"16\" src=\"./images/jupyter_logo.png\" /> GPTJ-generative-inference.ipynb </td>\n",
    "    <td width = 70%> Use structured and few-shot prompting to tackle language understanding tasks with text generation powered by GPT-J. </td>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> <img align=\"left\" width=\"18\" src=\"./images/folder_logo.png\" /> dolly2-instruction-following/ </td>\n",
    "    <td> </td>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> &nbsp&nbsp&nbsp&nbsp <img align=\"left\" width=\"16\" src=\"./images/jupyter_logo.png\" /> Dolly2-an-OSS-instruction-LLM.ipynb.ipynb </td>\n",
    "    <td width = 70%> Use Dolly2, the first truly open source instruction following LLM. </td>\n",
    "</td>\n",
    "</tr>\n",
    "</table>\n",
    "\n",
    "<br><br><br><br><br><br><br><br>\n",
    "    \n",
    "Run inference on <b>audio segments</b> check out our wav2vec2 notebook:\n",
    "\n",
    "<table align=\"left\">\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> <img align=\"left\" width=\"18\" src=\"./images/folder_logo.png\" /> audio-processing/ </td>\n",
    "    <td>  </td>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> &nbsp&nbsp&nbsp <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> wav2vec2-inference-checkpoint.ipynb </td>\n",
    "    <td> Run inference on the wav2vec 2.0 model with PyTorch on the Graphcore IPU-POD16 system.  </td>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> <img align=\"left\" width=\"18\" src=\"./images/folder_logo.png\" /> whisper/ </td>\n",
    "    <td>  </td>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> &nbsp&nbsp&nbsp <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> whisper-example.ipynb </td>\n",
    "    <td> Run inference on OpenAI's whisper model for speech to text transcription.  </td>\n",
    "</td>\n",
    "</tr>\n",
    "</table>\n",
    "\n",
    "\n",
    "\n",
    "</details>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "dd57fcd8",
   "metadata": {},
   "source": [
    "<details><summary><big> Fine-tune our models on your datasets</summary><br/>\n",
    "\n",
    "Explore our <b>Natural Language Processing</b> notebooks to fine-tune models for a range of different tasks:\n",
    "\n",
    "<table align=\"left\">\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> <img align=\"left\" width=\"18\" src=\"./images/folder_logo.png\" /> natural-language-processing/ </td>\n",
    "    <td>  </td>\n",
    "</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> \n",
    "        &nbsp&nbsp&nbsp <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> translation.ipynb\n",
    "    </td>\n",
    "    <td> WMT in our translation notebook,\n",
    "    </td>\n",
    "</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> &nbsp&nbsp&nbsp \n",
    "        <img align=\"left\" width=\"18\" src=\"./images/folder_logo.png\" /> other-use-cases/\n",
    "    </td>\n",
    "    <td>  </td>\n",
    "</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> \n",
    "        &nbsp&nbsp&nbsp&nbsp&nbsp&nbsp \n",
    "        <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> multiple_choice.ipynb\n",
    "    </td>\n",
    "    <td> SWAG in our multiple choice notebook,\n",
    "    </td>\n",
    "</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> \n",
    "        &nbsp&nbsp&nbsp&nbsp&nbsp&nbsp \n",
    "        <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> question_answering.ipynb\n",
    "    </td>\n",
    "    <td> SQUAD in our question-answering notebook,\n",
    "    </td>\n",
    "</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> \n",
    "        &nbsp&nbsp&nbsp&nbsp&nbsp&nbsp \n",
    "        <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> text_classification.ipynb\n",
    "    </td>\n",
    "    <td> any GLUE task in our text_classification notebook,\n",
    "    </td>\n",
    "</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> \n",
    "        &nbsp&nbsp&nbsp&nbsp&nbsp&nbsp \n",
    "        <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> token_classification.ipynb\n",
    "    </td>\n",
    "    <td> the CONLL 2003 dataset, in our token classification notebook (NER, PoS).\n",
    "    </td>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> <img align=\"left\" width=\"18\" src=\"./images/folder_logo.png\" /> gptj-text-generation/ </td>\n",
    "    <td> </td>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> &nbsp&nbsp&nbsp&nbsp <img align=\"left\" width=\"16\" src=\"./images/jupyter_logo.png\" /> finetuning.ipynb </td>\n",
    "    <td width = 70%> This notebook demonstrates how to fine-tune a GPT-J model downloaded from the Hugging Face Hub on the MNLI dataset. </td>\n",
    "</td>\n",
    "</tr>\n",
    "</table>\n",
    "\n",
    "<br><br><br><br><br><br><br><br><br><br><br><br><br><br>\n",
    "\n",
    "Fine-tune your models for <b>audio processing</b> tasks here:\n",
    "\n",
    "<table align=\"left\">\n",
    "\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> <img align=\"left\" width=\"18\" src=\"./images/folder_logo.png\" /> audio-processing/ </td>\n",
    "    <td>  </td>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> &nbsp&nbsp&nbsp <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> wav2vec2-fine-tuning-checkpoint.ipynb </td>\n",
    "    <td> See how to fine-tune a pre-trained wav2vec 2.0 model with PyTorch on the Graphcore IPU-POD16 system.\n",
    "    </td>\n",
    "</td>\n",
    "</tr>\n",
    "</table>\n",
    "\n",
    "<br><br><br>\n",
    "\n",
    "Fine-tune your models for <b>image classification</b> here:\n",
    "<table align=\"left\">\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> <img align=\"left\" width=\"18\" src=\"./images/folder_logo.png\" /> image-classification/ </td>\n",
    "    <td>  </td>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> \n",
    "        &nbsp&nbsp&nbsp <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> image-classification.ipynb\n",
    "    </td>\n",
    "    <td> This notebook will show you how to preprocess the data and fine-tune a pretrained model on image classification.\n",
    "    </td>\n",
    "</td>\n",
    "</tr>\n",
    "\n",
    "</table>\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e42f0b63",
   "metadata": {},
   "source": [
    "<details><summary><big> Build your own models </summary>\n",
    "\n",
    "These notebooks will show you how to use your own models, along with the easy functionality of Optimum-Graphcore:\n",
    "    \n",
    "<table align=\"left\">\n",
    "    \n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> <img align=\"left\" width=\"18\" src=\"./images/folder_logo.png\" /> natural-language-processing/ <br><br>&nbsp&nbsp&nbsp <img align=\"left\" width=\"18\" src=\"./images/folder_logo.png\" />other-use-cases/ </td>\n",
    "    <td>  </td>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> &nbsp&nbsp&nbsp&nbsp&nbsp <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> text_to_image_sd2.ipynb </td>\n",
    "    <td> Train a model that is not yet supported by `optimum-graphcore`  </td\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> &nbsp&nbsp&nbsp&nbsp&nbsp <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> language_modelling_from_scratch.ipynb </td>\n",
    "    <td> Show how to train a model for causal or masked language modelling from scratch  </td\n",
    "</td>\n",
    "</tr>\n",
    "\n",
    "</table>\n",
    "    \n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b2cc02c",
   "metadata": {},
   "source": [
    "<details><summary><big>  Image generation with Stable Diffusion</summary>\n",
    "\n",
    "Explore our more advanced examples with our range of Stable Diffusion notebooks!\n",
    "    \n",
    "<table align=\"left\">\n",
    "\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> <img align=\"left\" width=\"18\" src=\"./images/folder_logo.png\" /> stable-diffusion </td>\n",
    "    <td>  </td>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> &nbsp&nbsp&nbsp <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> text_to_image_sd2.ipynb </td>\n",
    "    <td> Run text-to-image tasks on Stable Diffusion 2 (Conditional UNet) pipeline </td\n",
    "</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> &nbsp&nbsp&nbsp <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> image_to_image.ipynb </td>\n",
    "    <td> Run image-to-image tasks on Stable Diffusion (Conditional UNet) pipeline </td\n",
    "</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> &nbsp&nbsp&nbsp <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> inpainting.ipynb </td>\n",
    "    <td> Generate images using inpainting on the Stable Diffusion (Conditional UNet) pipeline </td\n",
    "</td>\n",
    "</tr>\n",
    "\n",
    "</table>\n",
    "    \n",
    "</details>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "460e27bd",
   "metadata": {},
   "source": [
    "<details><summary><big> Train efficiently with Packed sequences and BERT</summary>\n",
    "\n",
    "Use [packing](https://www.graphcore.ai/posts/introducing-packed-bert-for-2x-faster-training-in-natural-language-processing) to improve the throughput of BERT on a range of NLP tasks:\n",
    "    \n",
    "<table align=\"left\">\n",
    "\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> <img align=\"left\" width=\"18\" src=\"./images/folder_logo.png\" /> packed-bert </td>\n",
    "    <td>  </td>\n",
    "</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> &nbsp&nbsp&nbsp <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> packedBERT_multi_label_text_classification.ipynb </td>\n",
    "    <td> Fine tuning and fast inference for multi-label classification with packed-BERT.</td\n",
    "</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> &nbsp&nbsp&nbsp <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> packedBERT_question_answering.ipynb </td>\n",
    "    <td> Fine tuning and fast inference for question answering with packed-BERT. </td\n",
    "</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td align=\"left\">\n",
    "    <td> &nbsp&nbsp&nbsp <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> packedBERT_single_label_text_classification.ipynb </td>\n",
    "    <td> Fine tuning and fast inference for single-label classification with packed-BERT. </td\n",
    "</td>\n",
    "</tr>\n",
    "\n",
    "</table>\n",
    "    \n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d8b8d7a",
   "metadata": {},
   "source": [
    "\n",
    "### Useful tips\n",
    "\n",
    "Find some useful tips within our tutorial on Managing IPU resources.\n",
    "\n",
    "<table align=\"left\">\n",
    "\n",
    "<tr height=50px>\n",
    "<td align=\"left\">\n",
    "    <td> <img align=\"left\" width=\"18\" src=\"./images/folder_logo.png\" /> useful-tips/ </td>\n",
    "    <td> </td>\n",
    "</td>\n",
    "</tr>\n",
    "<tr height=50px>\n",
    "<td align=\"left\">\n",
    "    <td> &nbsp&nbsp&nbsp <img align=\"left\" width=\"18\" src=\"./images/jupyter_logo.png\" /> managing_ipu_resources.ipynb </td>\n",
    "    <td width = 70%> See how to monitor IPU usage, how to release IPUs when you are not using them, and how to re-attach your model to the IPU to use it again. \n",
    "    </td>\n",
    "</td>\n",
    "</tr>\n",
    "\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55286c85",
   "metadata": {},
   "source": [
    "#### Join our slack community and interact with our developers!\n",
    "\n",
    "\n",
    "[![Join our Slack Community](https://img.shields.io/badge/Slack-Join%20Graphcore's%20Community-blue?style=flat-square&logo=slack)](https://www.graphcore.ai/join-community)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "vscode": {
   "interpreter": {
    "hash": "9409b80169a82c0207afe9a460d7f88a38094a708839df55a31910312ecdb1ee"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}