Tracking-without-bells-and-whistles.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "Tracking-without-bells-and-whistles",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.4"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hardik0/Multi-Object-Tracking-Google-Colab/blob/main/Tracking-without-bells-and-whistles.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RxKWwKoLR5Dj"
      },
      "source": [
        "# Tracking without bells and whistles\n",
        "[**Tracking without bells and whistles**](https://arxiv.org/abs/1903.05625) (Philipp Bergmann, [Tim Meinhardt](https://dvl.in.tum.de/team/meinhardt/), [Laura Leal-Taixe](https://dvl.in.tum.de/team/meinhardt/)). This [branch](https://github.com/phil-bergmann/tracking_wo_bnw) includes an updated version of Tracktor for PyTorch 1.3 with an improved object detector. The original results of the paper were produced with the [`iccv_19`](https://github.com/phil-bergmann/tracking_wo_bnw/tree/iccv_19) branch.\n",
        "\n",
        "**Original code :** [tracking_wo_bnw](https://github.com/phil-bergmann/tracking_wo_bnw) by [@phil-bergmann](https://github.com/phil-bergmann)\n",
        "\n",
        "<img src=\"https://github.com/phil-bergmann/tracking_wo_bnw/raw/master/data/method_vis_standalone.png\" width=\"600px\">\n",
        "\n",
        "### Citation\n",
        "```\n",
        "@InProceedings{tracktor_2019_ICCV,\n",
        "  author = {Bergmann, Philipp and Meinhardt, Tim and Leal{-}Taix{\\'{e}}, Laura},\n",
        "  title = {Tracking Without Bells and Whistles},\n",
        "  booktitle = {The IEEE International Conference on Computer Vision (ICCV)},\n",
        "  month = {October},\n",
        "  year = {2019}}\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HLG9PQdydnCP"
      },
      "source": [
        "! nvidia-smi\n",
        "! /usr/local/cuda/bin/nvcc --version"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yxXBbTJYqi5t",
        "outputId": "f0fd45de-868b-4627-c930-77e3e3cec878",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/gdrive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jfb-aowoY1dV"
      },
      "source": [
        "## Installation\n",
        "Clone this repository & Install packages"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nDuSey5aoFNx"
      },
      "source": [
        "%%shell\n",
        "\n",
        "git clone https://github.com/phil-bergmann/tracking_wo_bnw\n",
        "cd tracking_wo_bnw\n",
        "pip3 install -r requirements.txt\n",
        "pip3 install -e ."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cqta8FySZHOn"
      },
      "source": [
        "MOTChallenge data:\n",
        "\n",
        "* Download [MOT17Det](https://motchallenge.net/data/MOT17Det.zip), [MOT16Labels](https://motchallenge.net/data/MOT16Labels.zip), [2DMOT2015](https://motchallenge.net/data/2DMOT2015.zip),[ MOT16-det-dpm-raw](https://motchallenge.net/data/2DMOT2015.zip) and [MOT17Labels](https://motchallenge.net/data/MOT17Labels.zip) and place them in the data folder. As the images are the same for MOT17Det, MOT17 and MOT16 we only need one set of images for all three benchmarks.\n",
        "* Unzip all the data by executing:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DoY2y3G7ocLt"
      },
      "source": [
        "%%shell\n",
        "\n",
        "wget https://motchallenge.net/data/MOT17Det.zip\n",
        "wget https://motchallenge.net/data/MOT16-det-dpm-raw.zip\n",
        "wget https://motchallenge.net/data/MOT17Labels.zip\n",
        "cd tracking_wo_bnw/data\n",
        "pwd\n",
        "unzip -q -d MOT17Det ../../MOT17Det.zip\n",
        "unzip -q -d MOT16Labels ../../MOT16Labels.zip\n",
        "unzip -q -d 2DMOT2015 ../../2DMOT2015.zip\n",
        "unzip -q -d MOT16-det-dpm-raw ../../MOT16-det-dpm-raw.zip\n",
        "unzip -q -d MOT17Labels ../../MOT17Labels.zip\n",
        "#rm -r ../../*.zip"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-QZj7JplcM5F"
      },
      "source": [
        "Download object detector and re-identifiaction Siamese network weights and MOTChallenge result files:\n",
        "\n",
        "* Download zip file from [here](https://vision.in.tum.de/webshare/u/meinhard/tracking_wo_bnw-output_v2.zip).\n",
        "* Extract in output directory."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bW0VliKZZ8Zk"
      },
      "source": [
        "%%shell\n",
        "\n",
        "wget https://vision.in.tum.de/webshare/u/meinhard/tracking_wo_bnw-output_v2.zip\n",
        "cd tracking_wo_bnw\n",
        "pwd\n",
        "unzip -q ../tracking_wo_bnw-output_v2.zip  # output dir"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jWR53LxMvNtC",
        "outputId": "457a10a3-a607-4be0-cb3e-3a0239fe532e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "%cd tracking_wo_bnw/"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/tracking_wo_bnw\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DoYWtRCacuXD"
      },
      "source": [
        "## Evaluate Tracktor\n",
        "In order to configure, organize, log and reproduce our computational experiments we structured our code with the [Sacred](http://sacred.readthedocs.io/en/latest/index.html) framework. For a detailed explanation of the Sacred interface please read its documentation.\n",
        "\n",
        "1. Tracktor can be configured by changing the corresponding `experiments/cfgs/tracktor.yaml` config file. The default configuration runs Tracktor++ with the FPN object detector as described in the paper.\n",
        "\n",
        "2. The default configuration is `Tracktor++`. Run `Tracktor++` by executing:\n",
        "\n",
        "  `python experiments/scripts/test_tracktor.py`\n",
        "\n",
        "3. The results are logged in the corresponding `output` directory.\n",
        "\n",
        "For reproducability, we provide the new result metrics of this updated code base on the `MOT17` challenge. It should be noted, that these surpass the original Tracktor results. This is due to the newly trained object detector. This version of Tracktor does not differ conceptually from the original ICCV 2019 version (see branch `iccv_19`). The results on the offical MOTChallenge [webpage](https://motchallenge.net/results/MOT17/) are denoted as the `Tracktor++v2` tracker. The train and test results are:\n",
        "\n",
        "```\n",
        "********************* MOT17 TRAIN Results *********************\n",
        "IDF1  IDP  IDR| Rcll  Prcn   FAR|   GT  MT   PT   ML|    FP    FN   IDs    FM|  MOTA  MOTP MOTAL\n",
        "65.2 83.8 53.3| 63.1  99.2  0.11| 1638 550  714  374|  1732124291   903  1258|  62.3  89.6  62.6\n",
        "\n",
        "********************* MOT17 TEST Results *********************\n",
        "IDF1  IDP  IDR| Rcll  Prcn   FAR|   GT  MT   PT   ML|    FP    FN   IDs    FM|  MOTA  MOTP MOTAL\n",
        "55.1 73.6 44.1| 58.3  97.4  0.50| 2355 498 1026  831|  8866235449  1987  3763|  56.3  78.8  56.7\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ohsxjJ3evRbW",
        "outputId": "7da07759-8864-4aec-9011-169df5e664ac",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "! python experiments/scripts/test_tracktor.py"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING - test_tracktor - No observers have been added to this run\n",
            "INFO - test_tracktor - Running command 'main'\n",
            "INFO - test_tracktor - Started\n",
            "Configuration (\u001b[34mmodified\u001b[0m, \u001b[32madded\u001b[0m, \u001b[31mtypechanged\u001b[0m, \u001b[2mdoc\u001b[0m):\n",
            "  seed = 453695\n",
            "  reid:\n",
            "    db_train = 'train'\n",
            "    db_val = False\n",
            "    desription = 'all'\n",
            "    lr_scheduler = True\n",
            "    max_epochs = 69\n",
            "    module_name = 'reid'\n",
            "    name = 'res50-mot17-batch_hard'\n",
            "    seed = 12345\n",
            "    cnn:\n",
            "      output_dim = 128\n",
            "    dataloader:\n",
            "      K = 4\n",
            "      P = 18\n",
            "      crop_H = 256\n",
            "      crop_W = 128\n",
            "      max_per_person = 1000\n",
            "      normalize_mean = [0.485, 0.456, 0.406]\n",
            "      normalize_std = [0.229, 0.224, 0.225]\n",
            "      transform = 'random'\n",
            "      vis_threshold = 0.3\n",
            "    model_args:\n",
            "      loss = 'batch_hard'\n",
            "      margin = 0.2\n",
            "      prec_at_k = 3\n",
            "    solver:\n",
            "      optim = 'Adam'\n",
            "      optim_args:\n",
            "        lr = 0.0003\n",
            "        weight_decay = 0.0\n",
            "  tracktor:\n",
            "    dataset = 'mot17_train_FRCNN17'\n",
            "    desription = None\n",
            "    frame_split = [0.0, 1.0]\n",
            "    interpolate = False\n",
            "    module_name = 'MOT17'\n",
            "    name = 'Tracktor++'\n",
            "    network = 'fpn'\n",
            "    obj_detect_model = 'output/faster_rcnn_fpn_training_mot_17/model_epoch_27.model'\n",
            "    reid_config = 'output/tracktor/reid/res50-mot17-batch_hard/sacred_config.yaml'\n",
            "    reid_weights = 'output/tracktor/reid/res50-mot17-batch_hard/ResNet_iter_25245.pth'\n",
            "    seed = 12345\n",
            "    write_images = False\n",
            "    tracker:\n",
            "      detection_nms_thresh = 0.3\n",
            "      detection_person_thresh = 0.5\n",
            "      do_align = True\n",
            "      do_reid = True\n",
            "      inactive_patience = 10\n",
            "      max_features_num = 10\n",
            "      number_of_iterations = 100\n",
            "      public_detections = True\n",
            "      regression_nms_thresh = 0.6\n",
            "      regression_person_thresh = 0.5\n",
            "      reid_iou_threshold = 0.2\n",
            "      reid_sim_threshold = 2.0\n",
            "      termination_eps = 1e-05\n",
            "      warp_mode = 'cv2.MOTION_EUCLIDEAN'\n",
            "      motion_model:\n",
            "        center_only = True\n",
            "        enabled = False\n",
            "        n_steps = 1\n",
            "INFO - main - Initializing object detector.\n",
            "INFO - main - Tracking: MOT17-02-FRCNN\n",
            "100% 600/600 [05:39<00:00,  1.77it/s]\n",
            "INFO - main - Tracks found: 92\n",
            "INFO - main - Runtime for MOT17-02-FRCNN: 339.80 s.\n",
            "INFO - main - Writing predictions to: /content/tracking_wo_bnw/output/tracktor/MOT17/Tracktor++\n",
            "INFO - main - Tracking: MOT17-04-FRCNN\n",
            "100% 1050/1050 [11:34<00:00,  1.51it/s]\n",
            "INFO - main - Tracks found: 81\n",
            "INFO - main - Runtime for MOT17-04-FRCNN: 694.17 s.\n",
            "INFO - main - Writing predictions to: /content/tracking_wo_bnw/output/tracktor/MOT17/Tracktor++\n",
            "INFO - main - Tracking: MOT17-05-FRCNN\n",
            "100% 837/837 [03:26<00:00,  4.05it/s]\n",
            "INFO - main - Tracks found: 110\n",
            "INFO - main - Runtime for MOT17-05-FRCNN: 206.76 s.\n",
            "INFO - main - Writing predictions to: /content/tracking_wo_bnw/output/tracktor/MOT17/Tracktor++\n",
            "INFO - main - Tracking: MOT17-09-FRCNN\n",
            "100% 525/525 [05:34<00:00,  1.57it/s]\n",
            "INFO - main - Tracks found: 38\n",
            "INFO - main - Runtime for MOT17-09-FRCNN: 334.88 s.\n",
            "INFO - main - Writing predictions to: /content/tracking_wo_bnw/output/tracktor/MOT17/Tracktor++\n",
            "INFO - main - Tracking: MOT17-10-FRCNN\n",
            "100% 654/654 [09:32<00:00,  1.14it/s]\n",
            "INFO - main - Tracks found: 98\n",
            "INFO - main - Runtime for MOT17-10-FRCNN: 572.44 s.\n",
            "INFO - main - Writing predictions to: /content/tracking_wo_bnw/output/tracktor/MOT17/Tracktor++\n",
            "INFO - main - Tracking: MOT17-11-FRCNN\n",
            "100% 900/900 [13:00<00:00,  1.15it/s]\n",
            "INFO - main - Tracks found: 74\n",
            "INFO - main - Runtime for MOT17-11-FRCNN: 780.82 s.\n",
            "INFO - main - Writing predictions to: /content/tracking_wo_bnw/output/tracktor/MOT17/Tracktor++\n",
            "INFO - main - Tracking: MOT17-13-FRCNN\n",
            "100% 750/750 [14:40<00:00,  1.17s/it]\n",
            "INFO - main - Tracks found: 134\n",
            "INFO - main - Runtime for MOT17-13-FRCNN: 880.44 s.\n",
            "INFO - main - Writing predictions to: /content/tracking_wo_bnw/output/tracktor/MOT17/Tracktor++\n",
            "INFO - main - Tracking runtime for all sequences (without evaluation or image writing): 3809.31 s for 5316 frames (1.40 Hz)\n",
            "/usr/local/lib/python3.6/dist-packages/motmetrics/mot.py:243: FutureWarning: the 'labels' keyword is deprecated, use 'codes' instead\n",
            "  idx = pd.MultiIndex(levels=[[],[]], labels=[[],[]], names=['FrameId','Event'])\n",
            "                IDF1   IDP   IDR  Rcll  Prcn  GT  MT  PT  ML  FP    FN IDs   FM  MOTA  MOTP\n",
            "MOT17-02-FRCNN 45.9% 78.5% 32.4% 41.2% 99.8%  62   8  32  22  18 10926  57   65 40.8% 0.079\n",
            "MOT17-04-FRCNN 71.2% 90.5% 58.7% 64.7% 99.8%  83  32  29  22  67 16775  21   28 64.5% 0.096\n",
            "MOT17-05-FRCNN 63.9% 86.7% 50.6% 57.4% 98.3% 133  32  65  36  69  2948  38   60 55.8% 0.143\n",
            "MOT17-09-FRCNN 53.6% 68.2% 44.2% 64.2% 99.0%  26  11  13   2  33  1907  23   31 63.1% 0.087\n",
            "MOT17-10-FRCNN 65.3% 76.8% 56.8% 72.4% 97.9%  57  28  26   3 202  3546  67  119 70.3% 0.146\n",
            "MOT17-11-FRCNN 63.3% 77.0% 53.7% 69.0% 99.0%  75  24  33  18  67  2923  26   25 68.0% 0.082\n",
            "MOT17-13-FRCNN 72.7% 84.0% 64.1% 74.2% 97.3% 110  59  40  11 238  3006  60   84 71.6% 0.131\n",
            "OVERALL        65.0% 84.0% 53.1% 62.6% 99.0% 546 194 238 114 694 42031 292  412 61.7% 0.106\n",
            "INFO - test_tracktor - Completed after 1:10:29\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o1bG5umOfwKm"
      },
      "source": [
        "## Train and test object detector (Faster-RCNN with FPN) [<img src=\"https://colab.research.google.com/assets/colab-badge.svg\" align=\"center\">](https://colab.research.google.com/github/hardik0/Multi-Object-Tracking-Google-Colab/blob/main/utils/faster_rcnn_fpn_training_mot_17.ipynb)\n",
        "For the object detector we followed the new native torchvision implementations of Faster-RCNN with FPN which are pretrained on COCO. The provided object detection model was trained and tested with this Google Colab notebook. The `MOT17Det` train and test results are:\n",
        "\n",
        "```\n",
        "********************* MOT17Det TRAIN Results ***********\n",
        "Average Precision: 0.9090\n",
        "Rcll  Prcn|  FAR     GT     TP     FP     FN| MODA  MODP\n",
        "97.9  93.8| 0.81  66393  64989   4330   1404| 91.4  87.4\n",
        "\n",
        "********************* MOT17Det TEST Results ***********\n",
        "Average Precision: 0.8150\n",
        "Rcll  Prcn|  FAR     GT     TP     FP     FN| MODA  MODP\n",
        "86.5  88.3| 2.23 114564  99132  13184  15432| 75.0  78.3\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c7FA3DnxfR2r"
      },
      "source": [
        "## Training the reidentifaction model\n",
        "1. The training config file is located at `experiments/cfgs/reid.yaml`.\n",
        "2. Start training by executing:\n",
        "\n",
        "  `python experiments/scripts/train_reid.py`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NHTWNChNv8tc",
        "outputId": "5321b03f-2ad4-4b20-82b6-b863d8ca7ce2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "! python experiments/scripts/train_reid.py"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING - train_reid - No observers have been added to this run\n",
            "INFO - train_reid - Running command 'my_main'\n",
            "INFO - train_reid - Started\n",
            "{'reid': {'name': 'test', 'module_name': 'reid', 'desription': None, 'seed': 12345, 'db_train': 'mot_reid_small_train', 'db_val': False, 'model_args': {'loss': 'batch_hard', 'margin': 0.2, 'prec_at_k': 3}, 'solver': {'optim': 'Adam', 'optim_args': {'lr': 0.0003, 'weight_decay': 0.0}}, 'dataloader': {'vis_threshold': 0.3, 'P': 18, 'K': 4, 'max_per_person': 1000, 'crop_H': 256, 'crop_W': 128, 'transform': 'random', 'normalize_mean': [0.485, 0.456, 0.406], 'normalize_std': [0.229, 0.224, 0.225]}, 'cnn': {'output_dim': 128}}, 'seed': 97319404}\n",
            "[*] Initializing Dataloader\n",
            "[*] Loaded 50 persons from sequence MOT17-02.\n",
            "[*] Loaded 79 persons from sequence MOT17-04.\n",
            "[*] Loaded 112 persons from sequence MOT17-05.\n",
            "[*] Loaded 25 persons from sequence MOT17-09.\n",
            "[*] Loaded 57 persons from sequence MOT17-10.\n",
            "[*] Loaded 68 persons from sequence MOT17-11.\n",
            "[*] Loaded 104 persons from sequence MOT17-13.\n",
            "[*] Building CNN\n",
            "Downloading: \"https://download.pytorch.org/models/resnet50-19c8e357.pth\" to /root/.cache/torch/checkpoints/resnet50-19c8e357.pth\n",
            "100% 97.8M/97.8M [00:00<00:00, 214MB/s]\n",
            "[*] Solving ...\n",
            "START TRAIN.\n",
            "[Iteration 100/25137] 0.569 s/it\n",
            "prec_at_k: 0.000\n",
            "total_loss: 0.448\n",
            "[Iteration 200/25137] 0.593 s/it\n",
            "prec_at_k: 0.000\n",
            "total_loss: 0.225\n",
            "[Iteration 300/25137] 0.596 s/it\n",
            "prec_at_k: 0.000\n",
            "total_loss: 0.155\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_399.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 499/25137] 0.597 s/it\n",
            "prec_at_k: 0.010\n",
            "total_loss: 0.109\n",
            "[Iteration 599/25137] 0.597 s/it\n",
            "prec_at_k: 0.010\n",
            "total_loss: 0.098\n",
            "[Iteration 699/25137] 0.599 s/it\n",
            "prec_at_k: 0.020\n",
            "total_loss: 0.096\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_798.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 898/25137] 0.598 s/it\n",
            "prec_at_k: 0.000\n",
            "total_loss: 0.103\n",
            "[Iteration 998/25137] 0.598 s/it\n",
            "prec_at_k: 0.020\n",
            "total_loss: 0.083\n",
            "[Iteration 1098/25137] 0.598 s/it\n",
            "prec_at_k: 0.020\n",
            "total_loss: 0.072\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_1197.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 1297/25137] 0.598 s/it\n",
            "prec_at_k: 0.040\n",
            "total_loss: 0.062\n",
            "[Iteration 1397/25137] 0.598 s/it\n",
            "prec_at_k: 0.080\n",
            "total_loss: 0.044\n",
            "[Iteration 1497/25137] 0.599 s/it\n",
            "prec_at_k: 0.070\n",
            "total_loss: 0.054\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_1596.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 1696/25137] 0.599 s/it\n",
            "prec_at_k: 0.010\n",
            "total_loss: 0.082\n",
            "[Iteration 1796/25137] 0.598 s/it\n",
            "prec_at_k: 0.020\n",
            "total_loss: 0.077\n",
            "[Iteration 1896/25137] 0.597 s/it\n",
            "prec_at_k: 0.010\n",
            "total_loss: 0.071\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_1995.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 2095/25137] 0.598 s/it\n",
            "prec_at_k: 0.060\n",
            "total_loss: 0.054\n",
            "[Iteration 2195/25137] 0.598 s/it\n",
            "prec_at_k: 0.010\n",
            "total_loss: 0.069\n",
            "[Iteration 2295/25137] 0.598 s/it\n",
            "prec_at_k: 0.010\n",
            "total_loss: 0.081\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_2394.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 2494/25137] 0.599 s/it\n",
            "prec_at_k: 0.050\n",
            "total_loss: 0.048\n",
            "[Iteration 2594/25137] 0.596 s/it\n",
            "prec_at_k: 0.050\n",
            "total_loss: 0.052\n",
            "[Iteration 2694/25137] 0.597 s/it\n",
            "prec_at_k: 0.110\n",
            "total_loss: 0.053\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_2793.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 2893/25137] 0.597 s/it\n",
            "prec_at_k: 0.090\n",
            "total_loss: 0.045\n",
            "[Iteration 2993/25137] 0.596 s/it\n",
            "prec_at_k: 0.060\n",
            "total_loss: 0.049\n",
            "[Iteration 3093/25137] 0.596 s/it\n",
            "prec_at_k: 0.020\n",
            "total_loss: 0.057\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_3192.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 3292/25137] 0.596 s/it\n",
            "prec_at_k: 0.060\n",
            "total_loss: 0.060\n",
            "[Iteration 3392/25137] 0.596 s/it\n",
            "prec_at_k: 0.080\n",
            "total_loss: 0.063\n",
            "[Iteration 3492/25137] 0.598 s/it\n",
            "prec_at_k: 0.070\n",
            "total_loss: 0.048\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_3591.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 3691/25137] 0.599 s/it\n",
            "prec_at_k: 0.100\n",
            "total_loss: 0.036\n",
            "[Iteration 3791/25137] 0.598 s/it\n",
            "prec_at_k: 0.120\n",
            "total_loss: 0.040\n",
            "[Iteration 3891/25137] 0.598 s/it\n",
            "prec_at_k: 0.040\n",
            "total_loss: 0.038\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_3990.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 4090/25137] 0.601 s/it\n",
            "prec_at_k: 0.150\n",
            "total_loss: 0.039\n",
            "[Iteration 4190/25137] 0.601 s/it\n",
            "prec_at_k: 0.070\n",
            "total_loss: 0.054\n",
            "[Iteration 4290/25137] 0.601 s/it\n",
            "prec_at_k: 0.000\n",
            "total_loss: 0.095\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_4389.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 4489/25137] 0.601 s/it\n",
            "prec_at_k: 0.030\n",
            "total_loss: 0.053\n",
            "[Iteration 4589/25137] 0.598 s/it\n",
            "prec_at_k: 0.090\n",
            "total_loss: 0.047\n",
            "[Iteration 4689/25137] 0.599 s/it\n",
            "prec_at_k: 0.050\n",
            "total_loss: 0.058\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_4788.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 4888/25137] 0.597 s/it\n",
            "prec_at_k: 0.130\n",
            "total_loss: 0.040\n",
            "[Iteration 4988/25137] 0.599 s/it\n",
            "prec_at_k: 0.140\n",
            "total_loss: 0.040\n",
            "[Iteration 5088/25137] 0.597 s/it\n",
            "prec_at_k: 0.110\n",
            "total_loss: 0.041\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_5187.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 5287/25137] 0.597 s/it\n",
            "prec_at_k: 0.170\n",
            "total_loss: 0.030\n",
            "[Iteration 5387/25137] 0.596 s/it\n",
            "prec_at_k: 0.140\n",
            "total_loss: 0.044\n",
            "[Iteration 5487/25137] 0.593 s/it\n",
            "prec_at_k: 0.210\n",
            "total_loss: 0.029\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_5586.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 5686/25137] 0.596 s/it\n",
            "prec_at_k: 0.100\n",
            "total_loss: 0.043\n",
            "[Iteration 5786/25137] 0.595 s/it\n",
            "prec_at_k: 0.150\n",
            "total_loss: 0.030\n",
            "[Iteration 5886/25137] 0.594 s/it\n",
            "prec_at_k: 0.210\n",
            "total_loss: 0.030\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_5985.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 6085/25137] 0.596 s/it\n",
            "prec_at_k: 0.100\n",
            "total_loss: 0.046\n",
            "[Iteration 6185/25137] 0.596 s/it\n",
            "prec_at_k: 0.160\n",
            "total_loss: 0.043\n",
            "[Iteration 6285/25137] 0.596 s/it\n",
            "prec_at_k: 0.130\n",
            "total_loss: 0.035\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_6384.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 6484/25137] 0.595 s/it\n",
            "prec_at_k: 0.140\n",
            "total_loss: 0.039\n",
            "[Iteration 6584/25137] 0.595 s/it\n",
            "prec_at_k: 0.110\n",
            "total_loss: 0.035\n",
            "[Iteration 6684/25137] 0.596 s/it\n",
            "prec_at_k: 0.130\n",
            "total_loss: 0.033\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_6783.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 6883/25137] 0.597 s/it\n",
            "prec_at_k: 0.150\n",
            "total_loss: 0.032\n",
            "[Iteration 6983/25137] 0.594 s/it\n",
            "prec_at_k: 0.120\n",
            "total_loss: 0.040\n",
            "[Iteration 7083/25137] 0.596 s/it\n",
            "prec_at_k: 0.110\n",
            "total_loss: 0.048\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_7182.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 7282/25137] 0.595 s/it\n",
            "prec_at_k: 0.100\n",
            "total_loss: 0.043\n",
            "[Iteration 7382/25137] 0.596 s/it\n",
            "prec_at_k: 0.120\n",
            "total_loss: 0.040\n",
            "[Iteration 7482/25137] 0.595 s/it\n",
            "prec_at_k: 0.180\n",
            "total_loss: 0.030\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_7581.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 7681/25137] 0.595 s/it\n",
            "prec_at_k: 0.220\n",
            "total_loss: 0.028\n",
            "[Iteration 7781/25137] 0.594 s/it\n",
            "prec_at_k: 0.210\n",
            "total_loss: 0.025\n",
            "[Iteration 7881/25137] 0.593 s/it\n",
            "prec_at_k: 0.160\n",
            "total_loss: 0.035\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_7980.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 8080/25137] 0.593 s/it\n",
            "prec_at_k: 0.240\n",
            "total_loss: 0.020\n",
            "[Iteration 8180/25137] 0.595 s/it\n",
            "prec_at_k: 0.190\n",
            "total_loss: 0.025\n",
            "[Iteration 8280/25137] 0.592 s/it\n",
            "prec_at_k: 0.240\n",
            "total_loss: 0.024\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_8379.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 8479/25137] 0.594 s/it\n",
            "prec_at_k: 0.170\n",
            "total_loss: 0.027\n",
            "[Iteration 8579/25137] 0.593 s/it\n",
            "prec_at_k: 0.220\n",
            "total_loss: 0.030\n",
            "[Iteration 8679/25137] 0.593 s/it\n",
            "prec_at_k: 0.270\n",
            "total_loss: 0.029\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_8778.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 8878/25137] 0.595 s/it\n",
            "prec_at_k: 0.250\n",
            "total_loss: 0.027\n",
            "[Iteration 8978/25137] 0.595 s/it\n",
            "prec_at_k: 0.300\n",
            "total_loss: 0.021\n",
            "[Iteration 9078/25137] 0.594 s/it\n",
            "prec_at_k: 0.210\n",
            "total_loss: 0.034\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_9177.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 9277/25137] 0.596 s/it\n",
            "prec_at_k: 0.210\n",
            "total_loss: 0.027\n",
            "[Iteration 9377/25137] 0.594 s/it\n",
            "prec_at_k: 0.280\n",
            "total_loss: 0.026\n",
            "[Iteration 9477/25137] 0.594 s/it\n",
            "prec_at_k: 0.170\n",
            "total_loss: 0.039\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_9576.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 9676/25137] 0.592 s/it\n",
            "prec_at_k: 0.280\n",
            "total_loss: 0.023\n",
            "[Iteration 9776/25137] 0.593 s/it\n",
            "prec_at_k: 0.290\n",
            "total_loss: 0.019\n",
            "[Iteration 9876/25137] 0.595 s/it\n",
            "prec_at_k: 0.250\n",
            "total_loss: 0.017\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_9975.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 10075/25137] 0.593 s/it\n",
            "prec_at_k: 0.310\n",
            "total_loss: 0.014\n",
            "[Iteration 10175/25137] 0.595 s/it\n",
            "prec_at_k: 0.330\n",
            "total_loss: 0.017\n",
            "[Iteration 10275/25137] 0.592 s/it\n",
            "prec_at_k: 0.340\n",
            "total_loss: 0.017\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_10374.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 10474/25137] 0.595 s/it\n",
            "prec_at_k: 0.160\n",
            "total_loss: 0.044\n",
            "[Iteration 10574/25137] 0.593 s/it\n",
            "prec_at_k: 0.220\n",
            "total_loss: 0.021\n",
            "[Iteration 10674/25137] 0.595 s/it\n",
            "prec_at_k: 0.210\n",
            "total_loss: 0.023\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_10773.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 10873/25137] 0.591 s/it\n",
            "prec_at_k: 0.250\n",
            "total_loss: 0.023\n",
            "[Iteration 10973/25137] 0.593 s/it\n",
            "prec_at_k: 0.230\n",
            "total_loss: 0.026\n",
            "[Iteration 11073/25137] 0.593 s/it\n",
            "prec_at_k: 0.300\n",
            "total_loss: 0.021\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_11172.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 11272/25137] 0.593 s/it\n",
            "prec_at_k: 0.340\n",
            "total_loss: 0.017\n",
            "[Iteration 11372/25137] 0.594 s/it\n",
            "prec_at_k: 0.280\n",
            "total_loss: 0.024\n",
            "[Iteration 11472/25137] 0.594 s/it\n",
            "prec_at_k: 0.280\n",
            "total_loss: 0.019\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_11571.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 11671/25137] 0.590 s/it\n",
            "prec_at_k: 0.340\n",
            "total_loss: 0.016\n",
            "[Iteration 11771/25137] 0.592 s/it\n",
            "prec_at_k: 0.290\n",
            "total_loss: 0.019\n",
            "[Iteration 11871/25137] 0.592 s/it\n",
            "prec_at_k: 0.340\n",
            "total_loss: 0.016\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_11970.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 12070/25137] 0.590 s/it\n",
            "prec_at_k: 0.340\n",
            "total_loss: 0.018\n",
            "[Iteration 12170/25137] 0.593 s/it\n",
            "prec_at_k: 0.300\n",
            "total_loss: 0.019\n",
            "[Iteration 12270/25137] 0.590 s/it\n",
            "prec_at_k: 0.310\n",
            "total_loss: 0.020\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_12369.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 12469/25137] 0.592 s/it\n",
            "prec_at_k: 0.320\n",
            "total_loss: 0.019\n",
            "[Iteration 12569/25137] 0.587 s/it\n",
            "prec_at_k: 0.380\n",
            "total_loss: 0.014\n",
            "[Iteration 12669/25137] 0.595 s/it\n",
            "prec_at_k: 0.190\n",
            "total_loss: 0.033\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_12768.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 12868/25137] 0.595 s/it\n",
            "prec_at_k: 0.260\n",
            "total_loss: 0.025\n",
            "[Iteration 12968/25137] 0.588 s/it\n",
            "prec_at_k: 0.360\n",
            "total_loss: 0.017\n",
            "[Iteration 13068/25137] 0.588 s/it\n",
            "prec_at_k: 0.390\n",
            "total_loss: 0.018\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_13167.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 13267/25137] 0.590 s/it\n",
            "prec_at_k: 0.340\n",
            "total_loss: 0.019\n",
            "[Iteration 13367/25137] 0.593 s/it\n",
            "prec_at_k: 0.300\n",
            "total_loss: 0.018\n",
            "[Iteration 13467/25137] 0.586 s/it\n",
            "prec_at_k: 0.510\n",
            "total_loss: 0.013\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_13566.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 13666/25137] 0.583 s/it\n",
            "prec_at_k: 0.460\n",
            "total_loss: 0.013\n",
            "[Iteration 13766/25137] 0.589 s/it\n",
            "prec_at_k: 0.340\n",
            "total_loss: 0.017\n",
            "[Iteration 13866/25137] 0.587 s/it\n",
            "prec_at_k: 0.370\n",
            "total_loss: 0.013\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_13965.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 14065/25137] 0.585 s/it\n",
            "prec_at_k: 0.420\n",
            "total_loss: 0.015\n",
            "[Iteration 14165/25137] 0.591 s/it\n",
            "prec_at_k: 0.370\n",
            "total_loss: 0.016\n",
            "[Iteration 14265/25137] 0.589 s/it\n",
            "prec_at_k: 0.390\n",
            "total_loss: 0.016\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_14364.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 14464/25137] 0.585 s/it\n",
            "prec_at_k: 0.490\n",
            "total_loss: 0.010\n",
            "[Iteration 14564/25137] 0.588 s/it\n",
            "prec_at_k: 0.450\n",
            "total_loss: 0.015\n",
            "[Iteration 14664/25137] 0.590 s/it\n",
            "prec_at_k: 0.480\n",
            "total_loss: 0.014\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_14763.pth\n",
            "[*] New learning rate(s): [0.0003]\n",
            "[Iteration 14863/25137] 0.592 s/it\n",
            "prec_at_k: 0.350\n",
            "total_loss: 0.016\n",
            "[Iteration 14963/25137] 0.585 s/it\n",
            "prec_at_k: 0.450\n",
            "total_loss: 0.013\n",
            "[Iteration 15063/25137] 0.586 s/it\n",
            "prec_at_k: 0.450\n",
            "total_loss: 0.013\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_15162.pth\n",
            "[*] New learning rate(s): [0.00026823858825088617]\n",
            "[Iteration 15262/25137] 0.587 s/it\n",
            "prec_at_k: 0.390\n",
            "total_loss: 0.015\n",
            "[Iteration 15362/25137] 0.586 s/it\n",
            "prec_at_k: 0.550\n",
            "total_loss: 0.010\n",
            "[Iteration 15462/25137] 0.583 s/it\n",
            "prec_at_k: 0.550\n",
            "total_loss: 0.008\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_15561.pth\n",
            "[*] New learning rate(s): [0.00020362038521402613]\n",
            "[Iteration 15661/25137] 0.581 s/it\n",
            "prec_at_k: 0.520\n",
            "total_loss: 0.009\n",
            "[Iteration 15761/25137] 0.582 s/it\n",
            "prec_at_k: 0.580\n",
            "total_loss: 0.007\n",
            "[Iteration 15861/25137] 0.580 s/it\n",
            "prec_at_k: 0.580\n",
            "total_loss: 0.010\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_15960.pth\n",
            "[*] New learning rate(s): [0.00015456859337452692]\n",
            "[Iteration 16060/25137] 0.578 s/it\n",
            "prec_at_k: 0.670\n",
            "total_loss: 0.006\n",
            "[Iteration 16160/25137] 0.578 s/it\n",
            "prec_at_k: 0.660\n",
            "total_loss: 0.006\n",
            "[Iteration 16260/25137] 0.579 s/it\n",
            "prec_at_k: 0.600\n",
            "total_loss: 0.006\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_16359.pth\n",
            "[*] New learning rate(s): [0.00011733329171667887]\n",
            "[Iteration 16459/25137] 0.576 s/it\n",
            "prec_at_k: 0.690\n",
            "total_loss: 0.003\n",
            "[Iteration 16559/25137] 0.575 s/it\n",
            "prec_at_k: 0.720\n",
            "total_loss: 0.005\n",
            "[Iteration 16659/25137] 0.574 s/it\n",
            "prec_at_k: 0.700\n",
            "total_loss: 0.004\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_16758.pth\n",
            "[*] New learning rate(s): [8.906790858679119e-05]\n",
            "[Iteration 16858/25137] 0.574 s/it\n",
            "prec_at_k: 0.710\n",
            "total_loss: 0.006\n",
            "[Iteration 16958/25137] 0.573 s/it\n",
            "prec_at_k: 0.820\n",
            "total_loss: 0.003\n",
            "[Iteration 17058/25137] 0.569 s/it\n",
            "prec_at_k: 0.810\n",
            "total_loss: 0.002\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_17157.pth\n",
            "[*] New learning rate(s): [6.761160642437944e-05]\n",
            "[Iteration 17257/25137] 0.574 s/it\n",
            "prec_at_k: 0.780\n",
            "total_loss: 0.004\n",
            "[Iteration 17357/25137] 0.572 s/it\n",
            "prec_at_k: 0.750\n",
            "total_loss: 0.003\n",
            "[Iteration 17457/25137] 0.568 s/it\n",
            "prec_at_k: 0.870\n",
            "total_loss: 0.001\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_17556.pth\n",
            "[*] New learning rate(s): [5.1324089627979855e-05]\n",
            "[Iteration 17656/25137] 0.570 s/it\n",
            "prec_at_k: 0.870\n",
            "total_loss: 0.001\n",
            "[Iteration 17756/25137] 0.569 s/it\n",
            "prec_at_k: 0.850\n",
            "total_loss: 0.001\n",
            "[Iteration 17856/25137] 0.568 s/it\n",
            "prec_at_k: 0.830\n",
            "total_loss: 0.002\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_17955.pth\n",
            "[*] New learning rate(s): [3.896020691487491e-05]\n",
            "[Iteration 18055/25137] 0.572 s/it\n",
            "prec_at_k: 0.790\n",
            "total_loss: 0.002\n",
            "[Iteration 18155/25137] 0.575 s/it\n",
            "prec_at_k: 0.820\n",
            "total_loss: 0.003\n",
            "[Iteration 18255/25137] 0.568 s/it\n",
            "prec_at_k: 0.870\n",
            "total_loss: 0.002\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_18354.pth\n",
            "[*] New learning rate(s): [2.957476175129991e-05]\n",
            "[Iteration 18454/25137] 0.571 s/it\n",
            "prec_at_k: 0.830\n",
            "total_loss: 0.002\n",
            "[Iteration 18554/25137] 0.568 s/it\n",
            "prec_at_k: 0.880\n",
            "total_loss: 0.002\n",
            "[Iteration 18654/25137] 0.567 s/it\n",
            "prec_at_k: 0.870\n",
            "total_loss: 0.003\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_18753.pth\n",
            "[*] New learning rate(s): [2.2450253782204795e-05]\n",
            "[Iteration 18853/25137] 0.570 s/it\n",
            "prec_at_k: 0.820\n",
            "total_loss: 0.002\n",
            "[Iteration 18953/25137] 0.569 s/it\n",
            "prec_at_k: 0.860\n",
            "total_loss: 0.003\n",
            "[Iteration 19053/25137] 0.566 s/it\n",
            "prec_at_k: 0.930\n",
            "total_loss: 0.001\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_19152.pth\n",
            "[*] New learning rate(s): [1.7042027223203167e-05]\n",
            "[Iteration 19252/25137] 0.569 s/it\n",
            "prec_at_k: 0.840\n",
            "total_loss: 0.002\n",
            "[Iteration 19352/25137] 0.574 s/it\n",
            "prec_at_k: 0.810\n",
            "total_loss: 0.003\n",
            "[Iteration 19452/25137] 0.568 s/it\n",
            "prec_at_k: 0.910\n",
            "total_loss: 0.001\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_19551.pth\n",
            "[*] New learning rate(s): [1.2936632908203822e-05]\n",
            "[Iteration 19651/25137] 0.569 s/it\n",
            "prec_at_k: 0.880\n",
            "total_loss: 0.002\n",
            "[Iteration 19751/25137] 0.566 s/it\n",
            "prec_at_k: 0.880\n",
            "total_loss: 0.003\n",
            "[Iteration 19851/25137] 0.565 s/it\n",
            "prec_at_k: 0.890\n",
            "total_loss: 0.001\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_19950.pth\n",
            "[*] New learning rate(s): [9.820220846365146e-06]\n",
            "[Iteration 20050/25137] 0.568 s/it\n",
            "prec_at_k: 0.860\n",
            "total_loss: 0.002\n",
            "[Iteration 20150/25137] 0.570 s/it\n",
            "prec_at_k: 0.870\n",
            "total_loss: 0.002\n",
            "[Iteration 20250/25137] 0.565 s/it\n",
            "prec_at_k: 0.910\n",
            "total_loss: 0.001\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_20349.pth\n",
            "[*] New learning rate(s): [7.454546956359005e-06]\n",
            "[Iteration 20449/25137] 0.567 s/it\n",
            "prec_at_k: 0.880\n",
            "total_loss: 0.002\n",
            "[Iteration 20549/25137] 0.566 s/it\n",
            "prec_at_k: 0.910\n",
            "total_loss: 0.001\n",
            "[Iteration 20649/25137] 0.567 s/it\n",
            "prec_at_k: 0.910\n",
            "total_loss: 0.001\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_20748.pth\n",
            "[*] New learning rate(s): [5.6587597360531975e-06]\n",
            "[Iteration 20848/25137] 0.566 s/it\n",
            "prec_at_k: 0.910\n",
            "total_loss: 0.002\n",
            "[Iteration 20948/25137] 0.565 s/it\n",
            "prec_at_k: 0.900\n",
            "total_loss: 0.001\n",
            "[Iteration 21048/25137] 0.563 s/it\n",
            "prec_at_k: 0.930\n",
            "total_loss: 0.001\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_21147.pth\n",
            "[*] New learning rate(s): [4.295574491359432e-06]\n",
            "[Iteration 21247/25137] 0.568 s/it\n",
            "prec_at_k: 0.850\n",
            "total_loss: 0.002\n",
            "[Iteration 21347/25137] 0.567 s/it\n",
            "prec_at_k: 0.890\n",
            "total_loss: 0.001\n",
            "[Iteration 21447/25137] 0.571 s/it\n",
            "prec_at_k: 0.820\n",
            "total_loss: 0.002\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_21546.pth\n",
            "[*] New learning rate(s): [3.260778168978686e-06]\n",
            "[Iteration 21646/25137] 0.567 s/it\n",
            "prec_at_k: 0.880\n",
            "total_loss: 0.001\n",
            "[Iteration 21746/25137] 0.566 s/it\n",
            "prec_at_k: 0.860\n",
            "total_loss: 0.002\n",
            "[Iteration 21846/25137] 0.567 s/it\n",
            "prec_at_k: 0.840\n",
            "total_loss: 0.002\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_21945.pth\n",
            "[*] New learning rate(s): [2.4752624564364223e-06]\n",
            "[Iteration 22045/25137] 0.568 s/it\n",
            "prec_at_k: 0.890\n",
            "total_loss: 0.001\n",
            "[Iteration 22145/25137] 0.569 s/it\n",
            "prec_at_k: 0.860\n",
            "total_loss: 0.002\n",
            "[Iteration 22245/25137] 0.566 s/it\n",
            "prec_at_k: 0.890\n",
            "total_loss: 0.001\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_22344.pth\n",
            "[*] New learning rate(s): [1.878976094274667e-06]\n",
            "[Iteration 22444/25137] 0.566 s/it\n",
            "prec_at_k: 0.890\n",
            "total_loss: 0.002\n",
            "[Iteration 22544/25137] 0.568 s/it\n",
            "prec_at_k: 0.860\n",
            "total_loss: 0.001\n",
            "[Iteration 22644/25137] 0.567 s/it\n",
            "prec_at_k: 0.920\n",
            "total_loss: 0.001\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_22743.pth\n",
            "[*] New learning rate(s): [1.426334065575631e-06]\n",
            "[Iteration 22843/25137] 0.570 s/it\n",
            "prec_at_k: 0.880\n",
            "total_loss: 0.002\n",
            "[Iteration 22943/25137] 0.569 s/it\n",
            "prec_at_k: 0.940\n",
            "total_loss: 0.001\n",
            "[Iteration 23043/25137] 0.569 s/it\n",
            "prec_at_k: 0.920\n",
            "total_loss: 0.001\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_23142.pth\n",
            "[*] New learning rate(s): [1.082732703636045e-06]\n",
            "[Iteration 23242/25137] 0.572 s/it\n",
            "prec_at_k: 0.900\n",
            "total_loss: 0.002\n",
            "[Iteration 23342/25137] 0.574 s/it\n",
            "prec_at_k: 0.860\n",
            "total_loss: 0.002\n",
            "[Iteration 23442/25137] 0.572 s/it\n",
            "prec_at_k: 0.890\n",
            "total_loss: 0.001\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_23541.pth\n",
            "[*] New learning rate(s): [8.219043040593068e-07]\n",
            "[Iteration 23641/25137] 0.570 s/it\n",
            "prec_at_k: 0.940\n",
            "total_loss: 0.001\n",
            "[Iteration 23741/25137] 0.568 s/it\n",
            "prec_at_k: 0.890\n",
            "total_loss: 0.002\n",
            "[Iteration 23841/25137] 0.567 s/it\n",
            "prec_at_k: 0.910\n",
            "total_loss: 0.002\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_23940.pth\n",
            "[*] New learning rate(s): [6.239090061311086e-07]\n",
            "[Iteration 24040/25137] 0.565 s/it\n",
            "prec_at_k: 0.920\n",
            "total_loss: 0.001\n",
            "[Iteration 24140/25137] 0.563 s/it\n",
            "prec_at_k: 0.960\n",
            "total_loss: 0.001\n",
            "[Iteration 24240/25137] 0.568 s/it\n",
            "prec_at_k: 0.890\n",
            "total_loss: 0.002\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_24339.pth\n",
            "[*] New learning rate(s): [4.736104264316149e-07]\n",
            "[Iteration 24439/25137] 0.563 s/it\n",
            "prec_at_k: 0.960\n",
            "total_loss: 0.000\n",
            "[Iteration 24539/25137] 0.568 s/it\n",
            "prec_at_k: 0.860\n",
            "total_loss: 0.002\n",
            "[Iteration 24639/25137] 0.569 s/it\n",
            "prec_at_k: 0.900\n",
            "total_loss: 0.001\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_24738.pth\n",
            "[*] New learning rate(s): [3.5951850962318053e-07]\n",
            "[Iteration 24838/25137] 0.565 s/it\n",
            "prec_at_k: 0.950\n",
            "total_loss: 0.001\n",
            "[Iteration 24938/25137] 0.569 s/it\n",
            "prec_at_k: 0.930\n",
            "total_loss: 0.001\n",
            "[Iteration 25038/25137] 0.570 s/it\n",
            "prec_at_k: 0.860\n",
            "total_loss: 0.001\n",
            "Wrote snapshot to: /content/tracking_wo_bnw/output/tracktor/reid/test/ResNet_iter_25137.pth\n",
            "FINISH.\n",
            "INFO - train_reid - Completed after 4:23:49\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_6_jewdh_BLT",
        "outputId": "f8ab7aa1-92f5-4494-dbd1-f4232048e915",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 414
        }
      },
      "source": [
        "%ls output/tracktor/reid/test"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "ResNet_iter_10374.pth  ResNet_iter_18354.pth  ResNet_iter_3591.pth\n",
            "ResNet_iter_10773.pth  ResNet_iter_18753.pth  ResNet_iter_3990.pth\n",
            "ResNet_iter_11172.pth  ResNet_iter_19152.pth  ResNet_iter_399.pth\n",
            "ResNet_iter_11571.pth  ResNet_iter_19551.pth  ResNet_iter_4389.pth\n",
            "ResNet_iter_11970.pth  ResNet_iter_19950.pth  ResNet_iter_4788.pth\n",
            "ResNet_iter_1197.pth   ResNet_iter_1995.pth   ResNet_iter_5187.pth\n",
            "ResNet_iter_12369.pth  ResNet_iter_20349.pth  ResNet_iter_5586.pth\n",
            "ResNet_iter_12768.pth  ResNet_iter_20748.pth  ResNet_iter_5985.pth\n",
            "ResNet_iter_13167.pth  ResNet_iter_21147.pth  ResNet_iter_6384.pth\n",
            "ResNet_iter_13566.pth  ResNet_iter_21546.pth  ResNet_iter_6783.pth\n",
            "ResNet_iter_13965.pth  ResNet_iter_21945.pth  ResNet_iter_7182.pth\n",
            "ResNet_iter_14364.pth  ResNet_iter_22344.pth  ResNet_iter_7581.pth\n",
            "ResNet_iter_14763.pth  ResNet_iter_22743.pth  ResNet_iter_7980.pth\n",
            "ResNet_iter_15162.pth  ResNet_iter_23142.pth  ResNet_iter_798.pth\n",
            "ResNet_iter_15561.pth  ResNet_iter_23541.pth  ResNet_iter_8379.pth\n",
            "ResNet_iter_15960.pth  ResNet_iter_23940.pth  ResNet_iter_8778.pth\n",
            "ResNet_iter_1596.pth   ResNet_iter_2394.pth   ResNet_iter_9177.pth\n",
            "ResNet_iter_16359.pth  ResNet_iter_24339.pth  ResNet_iter_9576.pth\n",
            "ResNet_iter_16758.pth  ResNet_iter_24738.pth  ResNet_iter_9975.pth\n",
            "ResNet_iter_17157.pth  ResNet_iter_25137.pth  sacred_config.yaml\n",
            "ResNet_iter_17556.pth  ResNet_iter_2793.pth\n",
            "ResNet_iter_17955.pth  ResNet_iter_3192.pth\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IQSwkremsQyu"
      },
      "source": [
        "zip output directory (optional)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xKZRMgEh_QNT"
      },
      "source": [
        "! zip -r output.zip output"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tk01ckAC_1Tw",
        "outputId": "32b72c88-57cb-4047-e37c-bf803e404514",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "!du -h output.zip"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "5.9G\toutput.zip\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S7s__puUsu5T"
      },
      "source": [
        "copy zip file to google drive"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7LbmgL_QF_FI"
      },
      "source": [
        "! cp output.zip '../gdrive/My Drive/'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p4li421fBVb1",
        "outputId": "a0b40cac-e62f-4e12-e403-17217855415d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "! du -h '../gdrive/My Drive/output.zip'"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "5.9G\t../gdrive/My Drive/output.zip\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}