finetuning_and_generation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "syn_gen_simple",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H7LoMj4GA4n_",
        "colab_type": "text"
      },
      "source": [
        "Adapted from [minimaxir's gpt-2 simple library](https://github.com/minimaxir/gpt-2-simple) and [demonstration notebook](https://colab.research.google.com/drive/1VLG8e7YSEwypxU-noRNhsv5dW4NfTGce), themselves built on [nshepperd's original GPT-2 package](https://github.com/nshepperd/gpt-2).\n",
        "\n",
        "Designed for implementation in Google Colab.\n"
      ]
    },
    {
      "source": [
        "## Loading GPT-2\n",
        "\n",
        "Here we'll load in the GPT-2 Simple package, download the 124M parameter \"small\" model, and mount Google Drive for data and model storage."
      ],
      "cell_type": "markdown",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KBkpRgBCBS2_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "a9b137ee-c370-4c7f-809b-f1611a54039f"
      },
      "source": [
        "# install GPT-2 Simple wrapper and enable drive mounting\n",
        "%tensorflow_version 1.x\n",
        "!pip install -q gpt-2-simple\n",
        "import gpt_2_simple as gpt2\n",
        "from datetime import datetime\n",
        "from google.colab import files"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TensorFlow 1.x selected.\n",
            "  Building wheel for gpt-2-simple (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "WARNING:tensorflow:\n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P8wSlgXoDPCR",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "outputId": "44d7c6ad-5f5e-4fce-bc22-e6e1d0366512"
      },
      "source": [
        "# download the 124M parameter GPT-2 small model\n",
        "gpt2.download_gpt2(model_name=\"124M\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Fetching checkpoint: 1.05Mit [00:00, 448Mit/s]                                                      \n",
            "Fetching encoder.json: 1.05Mit [00:00, 106Mit/s]                                                    \n",
            "Fetching hparams.json: 1.05Mit [00:00, 357Mit/s]                                                    \n",
            "Fetching model.ckpt.data-00000-of-00001: 498Mit [00:02, 187Mit/s]                                   \n",
            "Fetching model.ckpt.index: 1.05Mit [00:00, 356Mit/s]                                                \n",
            "Fetching model.ckpt.meta: 1.05Mit [00:00, 188Mit/s]                                                 \n",
            "Fetching vocab.bpe: 1.05Mit [00:00, 202Mit/s]                                                       \n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "puq4iC6vUAHc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "80a196fd-ca0b-41b6-e0db-93084f8b5eb2"
      },
      "source": [
        "# mount google drive for data and model storage\n",
        "gpt2.mount_gdrive()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6OFnPCLADfll",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# load dataset -- formatted as a single column CSV\n",
        "# assumes the file is in the default directory of a mounted Google Drive\n",
        "file_name = \"gpt_training_data.csv\"\n",
        "gpt2.copy_file_from_gdrive(file_name)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LdpZQXknFNY3",
        "colab_type": "text"
      },
      "source": [
        "## Model Finetuning\n",
        "\n",
        "Next we'll activate a persistent TensorFlow session for our loaded model and begin retraining on our synopses.\n",
        "\n",
        "The GPT-2 simple library provides us a customizable fine-tune function that has a lot to play with, but we'll focus on a few key parameters:\n",
        "\n",
        "* steps: the number of training steps to perform. 1000 steps overfit when using our GPT-2 Medium implementation, but since we've cut down the model size it's a bit more robust.\n",
        "* sample_every: we'll generate a randomly seeded sample every 200 steps to inspect how the model is learning\n",
        "* learning_rate: we'll stick to the default learning rate of 1e-4 based on the size of our data\n",
        "\n",
        "Finally we'll copy the trained model into Google Drive for generation or future training."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aeXshJM-Cuaf",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "c5ff7b30-6d7c-4b93-c25d-25d307e09a07"
      },
      "source": [
        "# persistent TensorFlow session\n",
        "sess = gpt2.start_tf_sess()\n",
        "\n",
        "# train on the specified dataset\n",
        "gpt2.finetune(sess,\n",
        "              dataset=file_name,\n",
        "              model_name='124M',\n",
        "              steps=1000,\n",
        "              restore_from='fresh',\n",
        "              run_name='run1',\n",
        "              print_every=10,\n",
        "              sample_every=200,\n",
        "              save_every=500\n",
        "              )"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/gpt_2_simple/src/sample.py:17: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
            "Loading checkpoint models/124M/model.ckpt\n",
            "INFO:tensorflow:Restoring parameters from models/124M/model.ckpt\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 1/1 [00:00<00:00,  9.56it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Loading dataset...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "dataset has 1408260 tokens\n",
            "Training...\n",
            "[10 | 17.34] loss=3.11 avg=3.11\n",
            "[20 | 29.80] loss=3.02 avg=3.06\n",
            "[30 | 42.27] loss=3.02 avg=3.05\n",
            "[40 | 54.75] loss=3.05 avg=3.05\n",
            "[50 | 67.23] loss=2.93 avg=3.02\n",
            "[60 | 79.70] loss=2.83 avg=2.99\n",
            "[70 | 92.17] loss=2.97 avg=2.99\n",
            "[80 | 104.63] loss=3.03 avg=2.99\n",
            "[90 | 117.13] loss=3.00 avg=2.99\n",
            "[100 | 129.59] loss=2.78 avg=2.97\n",
            "[110 | 142.08] loss=3.07 avg=2.98\n",
            "[120 | 154.56] loss=2.78 avg=2.96\n",
            "[130 | 167.03] loss=2.99 avg=2.97\n",
            "[140 | 179.50] loss=2.87 avg=2.96\n",
            "[150 | 191.98] loss=2.81 avg=2.95\n",
            "[160 | 204.45] loss=2.82 avg=2.94\n",
            "[170 | 216.92] loss=2.89 avg=2.94\n",
            "[180 | 229.40] loss=2.84 avg=2.93\n",
            "[190 | 241.89] loss=2.91 avg=2.93\n",
            "[200 | 254.36] loss=2.82 avg=2.92\n",
            "======== SAMPLE 1 ========\n",
            " and have gone through many trials to finally make it through their trials. With time, they can learn to be confident in both themselves and others, and begin to realize why they live their very lives. Will they live up to their expectations, or will they not?<|endoftext|>\n",
            "<|startoftext|>Special episodes included in the Blu-ray, Volume 1.<|endoftext|>\n",
            "<|startoftext|>The story centers on Ryō-san, a boy who gets dragged into a strange land where he meets her mother. At first he doesn't know this land is so important because of various things, but gradually they form a friendship and Ryō-san becomes closer to her mother. He thinks that Ryō-san is a girl-child, even though she is an orphan. But Ryō-san is far from her mother, and he doesn't know about her life, nor the life of the girl he is following.<|endoftext|>\n",
            "<|startoftext|>Music video for Kojuu Gakuen (Lovely Girl) by the Kōji-sensei.<|endoftext|>\n",
            "<|startoftext|>The story takes place on a deserted island in Korea. A boy, Ryu-san, who is a student, hears of a UFO with a strange light.<|endoftext|>\n",
            "<|startoftext|>A song by the band Nyan to celebrate Christmas. This is an animated movie by the band. All the songs are from the band Nyan to celebrate the holidays.<|endoftext|>\n",
            "<|startoftext|>Named after the classic Japanese animation film by Kansai Shoten (1956-1965).<|endoftext|>\n",
            "<|startoftext|>Yamada Yoshida, author of the popular Yamada Yoshida short story, also works as the director of the popular Yumada Yoshida short story.<|endoftext|>\n",
            "<|startoftext|>The story follows a young boy who goes on a fishing trip and loses his friends. But he starts to see a good thing and helps all the fishing boats. But what do he learn of the good and bad?<|endoftext|>\n",
            "<|startoftext|>The two biggest stars of the \"Baka ni Kotome\". As they compete for the title of \"World's Best Baka ni Kotome\", they meet a very unusual lady. As one who looks like the heroine, but has dark hair.... This may not be a good look for the boy, as one who is \"Komotsu ni Kotome\" thinks the way he looks. And she tries to kill him from an unexpected perspective... A boy's view of the girl might not be very good if he isn't close with her. Not to mention, in this one he is the hero, but sometimes there's something about his life...<|endoftext|>\n",
            "<|startoftext|>Yamada is a genius and a genius inventor, he works hard to meet his customers and make the world a better place. And he thinks he's the one who makes the world a better place! It sounds crazy, but he does his homework and the world gets better and better... The story revolves around a young inventor from a neighboring country, who is not a good inventor but a genius inventor and is looking for a job at the local restaurant.<|endoftext|>\n",
            "<|startoftext|>The story revolves around a boy from a neighboring country who is not a good inventor but is a genius inventor and is looking for a job at the local cafe.<|endoftext|>\n",
            "<|startoftext|>An educational educational film on Japan`s history of land grabs and deforestation. With historical examples of the land grab, Japan`s \"Land Laws\". It teaches that the Japanese are responsible for most of the deforestation in Europe, Africa, and Asia. It has historical examples of land confiscation, but there are historical lessons about the use of land. In the story, the Japanese are responsible for the land's sale and loss. It's a modern country learning about land values, and in the way to make it better and better.<|endoftext|>\n",
            "<|startoftext|>A story about a girl living an ordinary ordinary life in a small village with the only thing she had to do is walk around the village and eat. The only thing she had other than this was to keep herself from the ravines of her village. However, one day her friends come to find her. They want to see what kind of food the people eat. What kind of life do they live, what are the basic needs and pleasures...<|endoftext|>\n",
            "<|startoftext|>A story about a woman from a neighboring country. When she gets to the\n",
            "\n",
            "[210 | 276.78] loss=2.75 avg=2.91\n",
            "[220 | 289.25] loss=2.98 avg=2.92\n",
            "[230 | 301.75] loss=2.64 avg=2.90\n",
            "[240 | 314.22] loss=2.74 avg=2.90\n",
            "[250 | 326.68] loss=2.91 avg=2.90\n",
            "[260 | 339.14] loss=2.82 avg=2.89\n",
            "[270 | 351.60] loss=2.60 avg=2.88\n",
            "[280 | 364.09] loss=2.91 avg=2.88\n",
            "[290 | 376.55] loss=2.94 avg=2.88\n",
            "[300 | 389.07] loss=2.82 avg=2.88\n",
            "[310 | 401.55] loss=2.84 avg=2.88\n",
            "[320 | 414.03] loss=2.89 avg=2.88\n",
            "[330 | 426.52] loss=2.88 avg=2.88\n",
            "[340 | 439.00] loss=2.83 avg=2.88\n",
            "[350 | 451.49] loss=2.78 avg=2.88\n",
            "[360 | 463.96] loss=2.85 avg=2.87\n",
            "[370 | 476.44] loss=2.89 avg=2.88\n",
            "[380 | 488.91] loss=2.61 avg=2.87\n",
            "[390 | 501.39] loss=2.85 avg=2.87\n",
            "[400 | 513.86] loss=2.78 avg=2.86\n",
            "======== SAMPLE 1 ========\n",
            " E.G.T.I, it was a series of short films about the daily life of the city's people as well as to promote the city's energy and natural resources, such as gas, wind and solar power. In the series T.V., the city's official social media accounts have been filled with images of the stars and the moon.<|endoftext|>\n",
            "<|startoftext|>A spin-off of D.C. 039's Todo Movie series.<|endoftext|>\n",
            "<|startoftext|>On a distant alien world, a new species of extraterrestrial heroes is on the prowl after a mysterious alien invasion threatens humanity. It is up to Shii to stop this menace and prevent its destruction in the future, as he is a member of the newly created Cyber-Eskimo squad and pilot of the Cyber-E Skimo (the Cyber-E Skimo Team) composed of five Cyber-E Skimo heroes from different galactic empires. Shii is assigned his own special Cyber-E Skimo pilot, the Cyber-E Skimo Psi-E (or \"Midi\"), but his real job is to save her from the cyborg invaders who are threatening another world.<|endoftext|>\n",
            "<|startoftext|>In the distant future of humanity, a secret power-laser-wielding, bounty hunter named Gunma is out for vengeance after murdering the late King Meegan of Jawa, the king who had led him to the moon.<|endoftext|>\n",
            "<|startoftext|>When the Earth Federation's military intelligence agency is in danger and a new threat is discovered targeting the Earth Federation troops, the United Nations forces to protect the Earth from alien threats. With the United Nations as their last shot, an unprecedented, and dangerous, mission is set...<|endoftext|>\n",
            "<|startoftext|>A short special episode included with the Blu-rays and DVDs of the anime adaptation of Gatchaman ONA: Gatchaman.<|endoftext|>\n",
            "<|startoftext|>Naruha is determined to find her dream job in the entertainment industry. To do so, she and friends embark on an exciting adventure on a ship in the South Seas. On board the ship is a young woman named Rin. Along with the crew of the ship are two young penguins: Aoi and Haruka.<|endoftext|>\n",
            "<|startoftext|>Second season of Tazuo-kun.<|endoftext|>\n",
            "<|startoftext|>A promotional video for the Blu-ray vol. 3 volume of the manga. The DVD is bundled with the limited edition of the 12th volume of the manga in Japan.<|endoftext|>\n",
            "<|startoftext|>Second season of Tazuo-kun.<|endoftext|>\n",
            "<|startoftext|>A new series of spin-off episodes featuring the \"Asteroids Saga.\" Main characters are humanoids who appeared when the Space Galaxy had taken on a life of its own and came under the control of the Galactic Empire as a result of the War of Destruction which was fought between the Emperor and the Galactic Union. An alien, a super powered robot named \"The King,\" came on board the vessel, which is set to go on a mission to find the queen.<|endoftext|>\n",
            "<|startoftext|>Music video of Utena's \"Hidokai no Otoko\".<|endoftext|>\n",
            "<|startoftext|>Music video of Utena's \"Hidokai no Otoko\".<|endoftext|>\n",
            "<|startoftext|>Music video of Utena's \"Hidokai no Otoko\".<|endoftext|>\n",
            "<|startoftext|>A short OVA bundled with the limited edition of the DVD volume. The disc's story takes place in the future when artificial life has developed in the human realm to become something like a paradise. There are those that live in the human world and there are those who were born with a special ability that enables them to make themselves immortal. It's at this point that these people come to the end of the human era in the future where they become immortal. They became the first generation immortal who fight an impossible battle to eliminate the evil lord of the human realm, the evil lord who had defeated the king and turned a human to stone, the King whom the human realm loved forever in the future.<|endoftext|>\n",
            "<|startoftext|>Music video of Utena's \"Hidokai no Otoko\".<|endoftext|>\n",
            "<|startoftext|>A theatrical music video of Utena's \"Hidokai no Otoko\".<|endof\n",
            "\n",
            "[410 | 535.32] loss=2.92 avg=2.87\n",
            "[420 | 547.78] loss=2.64 avg=2.86\n",
            "[430 | 560.26] loss=2.70 avg=2.85\n",
            "[440 | 572.74] loss=2.59 avg=2.85\n",
            "[450 | 585.20] loss=2.89 avg=2.85\n",
            "[460 | 597.69] loss=2.72 avg=2.84\n",
            "[470 | 610.20] loss=2.67 avg=2.84\n",
            "[480 | 622.69] loss=2.54 avg=2.83\n",
            "[490 | 635.18] loss=2.60 avg=2.83\n",
            "[500 | 647.64] loss=2.49 avg=2.82\n",
            "Saving checkpoint/run1/model-500\n",
            "[510 | 662.80] loss=2.64 avg=2.81\n",
            "[520 | 675.27] loss=2.43 avg=2.80\n",
            "[530 | 687.75] loss=2.75 avg=2.80\n",
            "[540 | 700.24] loss=2.49 avg=2.79\n",
            "[550 | 712.73] loss=2.57 avg=2.79\n",
            "[560 | 725.20] loss=2.54 avg=2.78\n",
            "[570 | 737.68] loss=2.62 avg=2.78\n",
            "[580 | 750.15] loss=2.73 avg=2.78\n",
            "[590 | 762.63] loss=2.73 avg=2.78\n",
            "[600 | 775.12] loss=2.89 avg=2.78\n",
            "======== SAMPLE 1 ========\n",
            "esta, who is a human girl who grew up a magical girl named Eri Kazuma. Eri was also chosen to act as a test subject to the race known as the \"Dark Elf\" who had never been born. To make her own choices, Eri chose an ideal girl, called the \"Cure Queen.\" The girls chose three girls—Leona, the witch, and the boy, Pouka. One day, a magical girl named Pouka appears and summons a magical beast known as Lancer. Pouka is defeated and left in the snow. Pouka vows to help her in her quest to rid the world of Dark Elves.<|endoftext|>\n",
            "<|startoftext|>The \"Twin Sisters\" appear to be a series of battles between a female pair of sisters, and also a series of battles between a male and female \"Twin Sisters,\" among them is a group, and also two other twins \"in their natural state,\" one of which is a girl, and the other a boy. Though the girls in the group, called in a second chapter \"Twin Sisters,\" fight for their souls, the truth is that they are members of a group in which the twins seem to fight. And as these twin sisters fight, there is also a battle between them in the next chapter, \"The Crimson Cowl and the Wolf.\"<|endoftext|>\n",
            "<|startoftext|>In order to defend the city from alien attacks, the three warriors of Yomiga stop at least one alien. The enemy attacks them in their own race, which then proceeds to fight them on the moon. It is then that they discover that there exist two warriors: one is a woman whose father left her an arm after many years, two is a man who fell victim to a meteor. She now tries to fight against a man who wants to steal her arm. One way or the other, the two are allies and allies. Together, there is only victory, but also betrayal.<|endoftext|>\n",
            "<|startoftext|>In an alternate universe there exists three brothers: their mother, a girl, an eagle, and a crow. They are a little bird who is a guardian of the forest. One of the brothers is the keeper of the bird nest, the other two are a little dog and a hamster. An eagle boy and a hamster boy go to the nest to fight against the enemy.<|endoftext|>\n",
            "<|startoftext|>Second season of the anime series.<|endoftext|>\n",
            "<|startoftext|>Third season of the anime series.<|endoftext|>\n",
            "<|startoftext|>In the near future, humanity is trapped in the jungle. A child with superhuman strength is taken in by a young girl who is an orphan. As she grows, her face shows signs of growing up. In the future, a young man living in the mountains is saved by the beautiful and protective mother of his late mother. They try to free her by shooting poisonous flowers. But when they try to bring back the mysterious girl, they find that the seeds of the seeds are not as fresh as they seemed.<|endoftext|>\n",
            "<|startoftext|>The world where the world meets again, where the world is still living. In this era, the planet Metis was destroyed. To get rid of the planet, Aragorn has stolen the last of its energy: the soul of the princess who the Aragorn were supposed to be protecting. To protect this precious soul, Aragorn has been given the title \"the strongest knight,\" and one day his son, Eren, is chosen as the new king of the Metis Kingdom. But as his father prepares for the battle, the princess reveals her true feelings and asks him to bring back her soul. A new enemy comes, who plans to take the girl hostage in order to stop Aragorn's invasion of the Earth.<|endoftext|>\n",
            "<|startoftext|>Second season of the anime series.<|endoftext|>\n",
            "<|startoftext|>The story begins when a boy named Jubei wakes up in his room one day in a coma, a strange dream about him like a distant ghost. He awakens and finds himself in a strange world. There, his father has returned home and his mother who went to meet him, has returned to see him again! But this time, in a dark, twisted dimension, there are evil sorcerers who want to resurrect Jubei.<|endoftext|>\n",
            "<|startoftext|>Miyagawa Kouji is a school dropout who likes to make friends (with the goal of getting his way). One day, he arrives at an old building on the roof and notices a large building floating down in the middle. The building is covered with a dark yellow light\n",
            "\n",
            "[610 | 796.66] loss=2.45 avg=2.77\n",
            "[620 | 809.15] loss=2.55 avg=2.77\n",
            "[630 | 821.63] loss=2.47 avg=2.76\n",
            "[640 | 834.11] loss=2.56 avg=2.76\n",
            "[650 | 846.60] loss=2.34 avg=2.75\n",
            "[660 | 859.08] loss=2.43 avg=2.74\n",
            "[670 | 871.57] loss=2.52 avg=2.74\n",
            "[680 | 884.04] loss=2.55 avg=2.73\n",
            "[690 | 896.49] loss=2.87 avg=2.74\n",
            "[700 | 908.95] loss=2.66 avg=2.74\n",
            "[710 | 921.42] loss=2.62 avg=2.73\n",
            "[720 | 933.89] loss=2.57 avg=2.73\n",
            "[730 | 946.33] loss=2.29 avg=2.72\n",
            "[740 | 958.79] loss=2.44 avg=2.72\n",
            "[750 | 971.24] loss=2.39 avg=2.71\n",
            "[760 | 983.69] loss=2.72 avg=2.71\n",
            "[770 | 996.14] loss=2.28 avg=2.70\n",
            "[780 | 1008.59] loss=2.18 avg=2.69\n",
            "[790 | 1021.06] loss=2.52 avg=2.69\n",
            "[800 | 1033.51] loss=2.42 avg=2.68\n",
            "======== SAMPLE 1 ========\n",
            " have not been as effective? And thus begins their duel.<|endoftext|>\n",
            "<|startoftext|>Bundled with the fourth volume of the DAWN anime.<|endoftext|>\n",
            "<|startoftext|>It's winter break, and the residents of Kyoto are feeling the warmth of summer, but there's still snowfall. One by one, a group of young girls become involved in a heated sex game, and they're joined by a prince who's willing the couple to remain faithful. The game's main theme is \"don't count the chickens,\" a sentiment shared in many female classmates. A year has passed since those two years—the time when the three boys have become best friends, and the girls have begun to discover they could have more in common without having to share information about their romance. It seems that some have begun to get anxious, but it seems that things are slowly coming to a head. The prince, who'd become a popular boy by becoming an adult, has suddenly become a little less charming, and while the girls have learned to accept that fact, even the prince's current status is unclear. Will things get better, or only worse?<|endoftext|>\n",
            "<|startoftext|>An animated gag film about a girl named Kaiko who is forced to move from her home in a neighboring town to a small seaside town. Kaiko finds the small town deserted; a strange creature floats by her bedside, and suddenly sees a man-sized figure running through the streets. When she realizes there's really a mysterious figure running—and Kaiko is attacked mercilessly by the creature—She makes the decision not to try and stop him: she can't even call it that way. But maybe that can't be helped after she is attacked once more. But just as she realizes it her mind starts to track the man and the creature, and eventually runs into Kaiko.<|endoftext|>\n",
            "<|startoftext|>At the time when they fought for Edo, a young girl who lives in Kyoukawa village was murdered, with another girl whom she mistakenly thought was the girl who took her life. At that time, it happened at the corner of their school, they heard the girl in the bathroom saying \"Kyoukawa village\" and suddenly there were battles happening all around town. At that same time, a girl is found dead by the middle school boy in the middle of the battle...<|endoftext|>\n",
            "<|startoftext|>The series revolves around Nana and her friends, a race car driver and a high school student, who together with their friends, try to take a first lap of their race car.<|endoftext|>\n",
            "<|startoftext|>Nana gets a little too close to Hikari. When her friends, including Hikari herself, decide to go out with her to celebrate the upcoming season, she gets sick of them but manages to get used to it. But things are not so easy for her. The group, including herself, are in fact trying to turn Hikari away because they don't wanna attract attention from Yuu, the girl inside the family. While the others are having a hard time sleeping at night, she's determined to resist his advances, too.<|endoftext|>\n",
            "<|startoftext|>The year is 18XX, which is much earlier than humans existed. The inhabitants of a small island, known as \"Palia\", have lived alongside humans as long as humans exist. But their isolation has left them dependent for survival on the seas. One day, when a merchant ship is attacked by pirates, the Queen of Palia orders the survivors to leave. After an unsuccessful rescue, she returns them to their own island to live in peace. However, before the survivors can leave, the enemy attempts to assassinate them. At that point, a massive battle is called off and all that humans stood against the enemy was their last breath. One of the survivors, Hikari, decides that the only way to survive is to find and save their beloved island.<|endoftext|>\n",
            "<|startoftext|>After an important meeting between G-Dog's friend Muroyama and Shinpei Akihiya, the series will feature two more guests: former Toto-chan pilot Sayama Sato, and the new series' creator, Muroyama himself.<|endoftext|>\n",
            "<|startoftext|>A special 10 minute episodes aired on Souseito TV that are all made up of 7 stories. The episodes take place between episodes 10 and 11. There will be a song with the same name that will be played during the ending credits at the end of the first episode of each series.<|endoftext|>\n",
            "<|startoftext|>Based on a doujin strip convention with a lot of gay scenes. This is an\n",
            "\n",
            "[810 | 1054.75] loss=2.51 avg=2.68\n",
            "[820 | 1067.21] loss=2.51 avg=2.68\n",
            "[830 | 1079.66] loss=2.57 avg=2.68\n",
            "[840 | 1092.15] loss=2.30 avg=2.67\n",
            "[850 | 1104.62] loss=2.23 avg=2.66\n",
            "[860 | 1117.09] loss=2.29 avg=2.66\n",
            "[870 | 1129.57] loss=2.57 avg=2.65\n",
            "[880 | 1142.02] loss=2.34 avg=2.65\n",
            "[890 | 1154.46] loss=2.28 avg=2.64\n",
            "[900 | 1166.90] loss=2.42 avg=2.64\n",
            "[910 | 1179.35] loss=2.35 avg=2.63\n",
            "[920 | 1191.81] loss=2.40 avg=2.63\n",
            "[930 | 1204.25] loss=2.12 avg=2.62\n",
            "[940 | 1216.68] loss=2.48 avg=2.62\n",
            "[950 | 1229.13] loss=2.60 avg=2.62\n",
            "[960 | 1241.61] loss=2.34 avg=2.61\n",
            "[970 | 1254.07] loss=2.17 avg=2.61\n",
            "[980 | 1266.49] loss=2.69 avg=2.61\n",
            "[990 | 1278.93] loss=2.09 avg=2.60\n",
            "[1000 | 1291.37] loss=2.38 avg=2.60\n",
            "Saving checkpoint/run1/model-1000\n",
            "WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/training/saver.py:963: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to delete files with this prefix.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VHdTL8NDbAh3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# save trained model checkpoint to Google Drive\n",
        "gpt2.copy_checkpoint_to_gdrive(run_name='run1')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "source": [
        "## Text Generation\n",
        "\n",
        "Next we'll load the model from Google Drive and check out some generations. \n",
        "\n",
        "We'll use the built-in generate function and play with some of the settings to get a feel for the model."
      ],
      "cell_type": "markdown",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-fxL77nvAMAX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "b10caeec-f90b-4aba-898a-b1527d5d57bb"
      },
      "source": [
        "# restart TensorFlow session\n",
        "sess = gpt2.start_tf_sess()\n",
        "\n",
        "# identify the trained model checkpoint in Google Drive\n",
        "gpt2.copy_checkpoint_from_gdrive(run_name='run1')\n",
        "\n",
        "# load model from saved checkpoint\n",
        "gpt2.load_gpt2(sess, run_name='run1')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loading checkpoint checkpoint/run1/model-1000\n",
            "INFO:tensorflow:Restoring parameters from checkpoint/run1/model-1000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4RNY6RBI9LmL",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 207
        },
        "outputId": "1f6235f1-7950-43f4-e3db-0e89af08d807"
      },
      "source": [
        "# generate with default parameters\n",
        "gpt2.generate(sess, run_name='run1')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "By: Yuuta Takami and Ayumu Watanabe\n",
            "\n",
            "<|startoftext|>The story begins with the arrival of the last High School senior to enter the prestigious Genshiken High School. Honoka was quite shy when she first met her classmate, Morinomiya Morino, but after he accepted her to join her class, she began to learn and mature with him. As Morinomiya grew to like her, so too did his admiration of hers. Morinomiya was also made aware of the fact that Morinomiya was the only one who could change the world, and so he began to train with her to help him do so. Throughout the course of his training, Morinomiya learned to work alongside her classmates to improve their respective skills and abilities, and soon they also began to become friends. However, Morinomiya's love for Morinomiya grew and she began to question her own feelings. Morinomiya knew that there was no way that Morinomiya and her classmates could ever be the same person, but she also knew that she was not the only one who could change the world. In the end, Morinomiya learns that there are secrets behind the world. She must continue to search for those secrets, and she must find an answer to the many questions that are being asked of her. Will Morinomiya be able to bring Morinomiya closer to her true feelings or will she be unable to move at all?<|endoftext|>\n",
            "<|startoftext|>A special included on the fifth volume of the Blu-ray. In the anime, the students and staff of the club will fight against a mysterious organization that aims to destroy the world. A luscious photo, a short story, and a short music video are included in the \"Ai Luscious\" collection.<|endoftext|>\n",
            "<|startoftext|>Special included on the seventh volume of the Blu-ray. In the anime, the students and staff of the club will fight against a mysterious organization that aims to destroy the world. A man holds a young woman, who turns out to be a robotic girl...<|endoftext|>\n",
            "<|startoftext|>The story takes place in the year 2040, when humans have entered a state of sleep where the mind is much more powerful. When the atmosphere inside a house is turned on for the first time in 200 years, a phenomenon known as \"The Return of the Clock Tower\" occurs. When it is discovered that the house is not destroyed by humans, the residents of the house are extremely saddened and begin to cry. A group of girls, led by the mysterious, mysterious and mysterious-sounding title, are invited on a mission to investigate the phenomenon, but are stopped by the mysterious, mysterious-sounding title, \"The Clock Tower.\"<|endoftext|>\n",
            "<|startoftext|>A new OVA adaptation of The Wizard of Oz with new characters, scenes and music. The OVA is a spin-off of the popular children's television show Wizard of Oz and centers on three beautiful little buggers, Dorothy, Alistair, and Uncle Wizard. Dorothy, Alistair, and Uncle Wizard are super-deformed children who live in this time of total mystical deprivation. They spend their days living in the Beastie Boys' new home in London - the home of the wizard, the master of the house. But when the Beastie Boys (the main character) and his friends kidnap Dorothy and the other buggers, and do everything in their power to take away their powers, the Beastie Boys (the villains) decide to go rogue and kidnap them. The villains are actually the very same buggers who rescued the girls from the Beastie Boys! To defeat the Beastie Boys, the boys will have to become as powerless as the girls themselves. And that is only the beginning of their adventures.<|endoftext|>\n",
            "<|startoftext|>The movie is the first 3 episodes of the DVD/Blu-ray/DVD OVA series. The plot revolves around the Takasugi family and their adventures.<|endoftext|>\n",
            "<|startoftext|>A young boy and his pet cat live happily together in the small town of Reiji. The town's lively spirit and fairytale culture make this Japanese folktale so popular that it is said to have a meaning in the Japanese word \"bodhi\". In this town, the two worlds come into conflict with one another and begin to clash over the future of the town. The two worlds try to protect each other and push each other and the world around the town. But things are not always what it seems.<|endoftext|>\n",
            "<|startoftext|>Two years ago, a boy named Doraemon stumbled upon a parallel world with a young girl,\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "source": [
        "### Initial Observations\n",
        "\n",
        "Note that GPT-2 can't generate to specific lengths, instead it continues predicting tokens until it predicts an <|endoftext|>, so we end up with several independent generations in the batch.\n",
        "\n",
        "The default generations appear quite reasonable, the model has clearly honed in on Japan-oriented vocabulary and much of the specific vernacular of the dataset. We also noticed some of GPT-2's original training data leaking, especially when it produces named entities (Beastie Boys, Wizard of Oz, etc.). \n",
        "\n",
        "This level of fit may be ideal for the purpose of novel story generation, as the model seems to have captured the literary structure of synopses quite well while introducing new external information from the pre-training data. "
      ],
      "cell_type": "markdown",
      "metadata": {}
    },
    {
      "source": [
        "### Generator Tuning\n",
        " \n",
        "Let's play around with the generator settings a bit and see how our output changes.\n",
        "\n",
        "We'll use the nsamples and batch size settings here to generate multiple samples in parallel, and introduce the truncate parameter to exclude the end of text token. \n",
        "\n",
        "We also have access to temperature, which scales the logits of our final encoder layer to adjust the \"randomness\" of predicted tokens."
      ],
      "cell_type": "markdown",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8DKMc0fiej4N",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 207
        },
        "outputId": "ab79fbcc-3439-46c5-db02-436c4b880369"
      },
      "source": [
        "gpt2.generate(sess,\n",
        "              length=250,\n",
        "              temperature=0.7,\n",
        "              nsamples=5,\n",
        "              batch_size=5,\n",
        "              truncate='<|endoftext|>',\n",
        "              include_prefix=False\n",
        "              )"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Shirayuki Yagami is a young high schooler. One day, an evil organization is attacking the school. When Shirayuki is attacked, a girl named Naru is dragged into the fight. She is a classmate of the student council president and president of the \"Naru Clan,\" a clan that deals with various issues with the human world. She is also the first transfer student at Shirayuki's school. As the students come to join the fight, they learn more about the people facing the human world.\n",
            "====================\n",
            "Special thanks to the people who helped me with this project. Many thanks to the main character of the show, his name is Taku and he is one of the most popular characters in Kyoukai no Omoide (To You) and some other series. I hope that I have made some progress, but I don't know how to proceed. Thanks also go to the people who helped me with the music, animation and video. Thanks also go to the many people who helped me with the sound design, as well as the special effects. I hope that this will help others too. And finally, thanks to the people who helped me with the story, I hope that I have made a new \"Wake Up, Say Hello!\" song.\n",
            "====================\n",
            "He is a young man who lives in the mountains. His mother is a beautiful woman who also has a child named Aramis. They live in the mountains with the help of a human and a bear named Mousis. Aramis is a princess and they have a son named Farsis. Mousis is a beautiful boy and they have a daughter named Maris. They live in the mountains with the help of a wolf named Akumon.\n",
            "====================\n",
            "<|startoftext|>In a time where the word \"triple-triple\" has become a pejorative. To what end is it going? It is a time for the elimination of the word \"triple-triple\". The stage is now established, and the battle continues. But the ultimate goal is not yet decided. What is the ultimate goal? The answer lies within the hearts of those who wish to attain it. What is the ultimate goal? A group of people decide to have the battle!\n",
            "====================\n",
            "<|startoftext|>In what may be the most anticipated anime of the year, the new series \"Hikari Hatsume\" follows the adventures of each of the main heroine's friends and the resulting adventures of a mysterious new character.\n",
            "====================\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "source": [
        "### Custom Generation\n",
        "Let's try introducing a prefix here to serve as the 'seed' for our generation. We can use this parameter to force generations to begin with a particular sequence.\n",
        "\n",
        "This will be the basis of our deployed generator where users can generate to their own custom seed."
      ],
      "cell_type": "markdown",
      "metadata": {}
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o6PF53335jWz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 207
        },
        "outputId": "1817732f-59bc-4546-9148-f65cdcadd259"
      },
      "source": [
        "gpt2.generate(sess,\n",
        "              length=500,\n",
        "              temperature=0.7,\n",
        "              nsamples=5,\n",
        "              batch_size=5,\n",
        "              truncate='<|endoftext|>',\n",
        "              prefix='In the year 20XX'\n",
        "              )"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "In the year 20XX, a group of aliens attacks Earth with a \"Valkyrie Spirit\" attack. The war is a stalemate with no new incidents. The Guardian Angels inform humanity of the attack, but it is not long before the Guardian Angels appear again. The Guardian Angels are joined by Jean-Claude Van Damme, a young boy with superhuman powers. Jean-Claude and his friends have to try to track down the Valkyrie Spirit who attacked Earth. But they find out that the Valkyrie Spirit is still in the human world and they are being hunted by the army of the underworld. Jean-Claude and his friends must find a way to stop them before they become the next Guardian Angels.\n",
            "====================\n",
            "In the year 20XX, a new race called the \"Quilters\" exist on Earth. They are a race of beings that has no memories and is unable to remember the past. They are slow, clumsy and clumsy, but at least they have knowledge of what happened to their ancestors when they were young. In order to defeat them, a team of humans named Spriggan and Spriggan's brother were born. They are born with a special ability which allows them to see and smell. They are called the Spriggan's Children. The world is a dark place where the diseases that caused humanity's death are fought with the power of the Spriggan's Children. For decades, the Spriggan's Children have been waging a struggle against the evil that is spreading amongst the stars. They are called the Spriggan's Children and they fight against the Quilters, trying to exterminate their enemies using their power, but they seem to be losing the fight and sometimes being defeated. One day, they find a group of Spriggan's Children in a desert. They fight against the Quilters and find a way to save them.\n",
            "====================\n",
            "In the year 20XX, the aliens from another dimension have begun to attack Earth. However, a series of disasters begin to crop up around the world. In order to protect humanity and prevent disaster, a team of UN personnel known as the \"Top 10,\" consisting of four highly skilled hunters, are dispatched to the devastated south to investigate the causes of a natural disaster. In the first mission, they encounter a mysterious force known as the \"Black Hole,\" which happens to be the source of all the supernatural occurrences. But, the team is suddenly attacked by a group of Black Holes, comprised of monsters known as \"Rangers.\" Meanwhile, a mysterious alien artifact called a \"Spear of Doom\" is also upon the surface. Only the Top 10 have to stop the threat of Rangers from spreading throughout the world.\n",
            "====================\n",
            "In the year 20XX, God of war and savior is unleashed a new negative force of evil and mayhem. New alien threats are revealed like the disappearance of a trace of alien life, the disappearance of a trace of life-form, the disappearance of a trace of alien life and the disappearance of an alien threat to the world. But the secret is hidden, the secret is kept a secret, and the investigation is hidden...\n",
            "====================\n",
            "In the year 20XX, the evil Emperor Zor has begun an assault on the planet Elen, destroying everything in their path. Now, there are only three people left to save: a mysterious girl named Piña, a mysterious man who appears to be a young princess who had been kidnapped by Zor's evil forces, and an old pair of twins, Tej and Teja. Led by Kaoru, the twins' father, Teja's job is to take the place of Piña in a plot to take over the world. However, Teja is not the only one who appears to be the key to this puzzle. The twins are also facing trials when they are confronted with a mysterious woman, who is also facing trial.\n",
            "====================\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zjjEN2Tafhl2",
        "colab_type": "text"
      },
      "source": [
        "### Bulk Generation\n",
        "Since we have a few different options here, let's generate a bunch of samples of varying length and temperature and save them to a file to pass to our scorer for curation. We'll rerun this cell a few times with different parameters and download the results to our local machine."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0-LRex8lfv1g",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "outputId": "f95f107b-59dc-4d17-a004-7bd54ad4f121"
      },
      "source": [
        "# generate a .txt file path using the current datetime\n",
        "gen_file = 'gpt2_gentext_{:%Y%m%d_%H%M%S}.txt'.format(datetime.utcnow())\n",
        "\n",
        "# generate directly to the gpt object's file store\n",
        "gpt2.generate_to_file(sess,\n",
        "                      destination_path=gen_file,\n",
        "                      length=500,\n",
        "                      temperature=0.9,\n",
        "                      nsamples=100,\n",
        "                      batch_size=20\n",
        "                      )\n",
        "\n",
        "# download the current file to local machine\n",
        "files.download(gen_file)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "download(\"download_b97687d5-f6bc-410f-aedf-189ccfe99d0c\", \"gpt2_gentext_20200912_203040.txt\", 209266)"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    }
  ]
}