init

Author: Zielon

.gitignore

@@ -31,10 +31,8 @@
 *.out
 *.app
 
+imgui.ini
+
 data
-dependencies
-instant-ngp/dependencies
-cinema
-scripts/*.txt
-scripts/sandbox
-scripts/diagrams/output
+scripts/insta
+scripts/dependencies

.gitmodules

@@ -0,0 +1,24 @@
+[submodule "instant-ngp/dependencies/pybind11"]
+	path = instant-ngp/dependencies/pybind11
+	url = https://github.com/Tom94/pybind11.git
+[submodule "instant-ngp/dependencies/eigen"]
+	path = instant-ngp/dependencies/eigen
+	url = https://github.com/Tom94/eigen.git
+[submodule "instant-ngp/dependencies/dlss"]
+	path = instant-ngp/dependencies/dlss
+	url = https://github.com/NVIDIA/DLSS.git
+[submodule "instant-ngp/dependencies/glfw"]
+	path = instant-ngp/dependencies/glfw
+	url = https://github.com/Tom94/glfw.git
+[submodule "instant-ngp/dependencies/imgui"]
+	path = instant-ngp/dependencies/imgui
+	url = https://github.com/ocornut/imgui.git
+[submodule "instant-ngp/dependencies/tiny-cuda-nn"]
+	path = instant-ngp/dependencies/tiny-cuda-nn
+	url = https://github.com/NVlabs/tiny-cuda-nn.git
+[submodule "instant-ngp/dependencies/tinylogger"]
+	path = instant-ngp/dependencies/tinylogger
+	url = https://github.com/Tom94/tinylogger.git
+[submodule "instant-ngp/dependencies/args"]
+	path = instant-ngp/dependencies/args
+	url = https://github.com/Taywee/args.git

LICENSE

@@ -0,0 +1,88 @@
+License
+
+Software Copyright License for non-commercial scientific research purposes
+Please read carefully the following terms and conditions and any accompanying documentation before you download
+and/or use the INSTA model, data and software, (the "Model & Software"), including 3D meshes, software, and scripts.
+By downloading and/or using the Model & Software (including downloading, cloning, installing, and any other use
+of this github repository), you acknowledge that you have read these terms and conditions, understand them, and
+agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use
+the Model & Software. Any infringement of the terms of this agreement will automatically terminate your rights
+under this License
+
+Ownership / Licensees
+The Model & Software and the associated materials has been developed at the
+Max Planck Institute for Intelligent Systems (hereinafter "MPI").
+
+Any copyright or patent right is owned by and proprietary material of the
+Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (hereinafter “MPG”; MPI and MPG hereinafter
+collectively “Max-Planck”) hereinafter the “Licensor”.
+
+License Grant
+Licensor grants you (Licensee) personally a single-user, non-exclusive, non-transferable, free of charge right:
+
+	• To install the Model & Software on computers owned, leased or otherwise controlled by you and/or your organization.
+	• To use the Model & Software for the sole purpose of performing peaceful non-commercial scientific research, 
+    non-commercial education, or non-commercial artistic projects.
+
+Any other use, in particular any use for commercial, pornographic, military, or surveillance purposes is prohibited. 
+This includes, without limitation, incorporation in a commercial product, use in a commercial service, 
+or production of other artefacts for commercial purposes. 
+
+The Model & Software may not be used to create fake, libelous, misleading, or defamatory content of any kind, excluding 
+analyses in peer-reviewed scientific research. 
+
+The Model & Software may not be reproduced, modified and/or made available in any form to any third party 
+without Max-Planck’s prior written permission.
+
+The Model & Software may not be used for pornographic purposes or to generate pornographic material whether
+commercial or not. This license also prohibits the use of the Model & Software to train methods/algorithms/neural
+networks/etc. for commercial use of any kind. By downloading the Model & Software, you agree not to reverse engineer it.
+
+No Distribution
+The Model & Software and the license herein granted shall not be copied, shared, distributed, re-sold, offered
+for re-sale, transferred or sub-licensed in whole or in part except that you may make one copy for archive
+purposes only.
+
+Disclaimer of Representations and Warranties
+You expressly acknowledge and agree that the Model & Software results from basic research, is provided “AS IS”,
+may contain errors, and that any use of the Model & Software is at your sole risk.
+LICENSOR MAKES NO REPRESENTATIONS
+OR WARRANTIES OF ANY KIND CONCERNING THE MODEL & SOFTWARE, NEITHER EXPRESS NOR IMPLIED, AND THE ABSENCE OF ANY
+LEGAL OR ACTUAL DEFECTS, WHETHER DISCOVERABLE OR NOT. Specifically, and not to limit the foregoing, licensor
+makes no representations or warranties (i) regarding the merchantability or fitness for a particular purpose of
+the Model & Software, (ii) that the use of the Model & Software will not infringe any patents, copyrights or other
+intellectual property rights of a third party, and (iii) that the use of the Model & Software will not cause any
+damage of any kind to you or a third party.
+
+Limitation of Liability
+Because this Model & Software License Agreement qualifies as a donation, according to Section 521 of the German
+Civil Code (Bürgerliches Gesetzbuch – BGB) Licensor as a donor is liable for intent and gross negligence only.
+If the Licensor fraudulently conceals a legal or material defect, they are obliged to compensate the Licensee
+for the resulting damage.
+
+Licensor shall be liable for loss of data only up to the amount of typical recovery costs which would have
+arisen had proper and regular data backup measures been taken. For the avoidance of doubt Licensor shall be
+liable in accordance with the German Product Liability Act in the event of product liability. The foregoing
+applies also to Licensor’s legal representatives or assistants in performance. Any further liability shall 
+be excluded. Patent claims generated through the usage of the Model & Software cannot be directed towards the copyright holders.
+The Model & Software is provided in the state of development the licensor defines. If modified or extended by
+Licensee, the Licensor makes no claims about the fitness of the Model & Software and is not responsible
+for any problems such modifications cause.
+
+No Maintenance Services
+You understand and agree that Licensor is under no obligation to provide either maintenance services,
+update services, notices of latent defects, or corrections of defects with regard to the Model & Software.
+Licensor nevertheless reserves the right to update, modify, or discontinue the Model & Software at any time.
+
+Defects of the Model & Software must be notified in writing to the Licensor with a comprehensible description
+of the error symptoms. The notification of the defect should enable the reproduction of the error.
+The Licensee is encouraged to communicate any use, results, modification or publication.
+
+Publications using the Model & Software
+You acknowledge that the Model & Software is a valuable scientific resource and agree to appropriately reference
+the following paper in any publication making use of the Model & Software.
+
+Commercial licensing opportunities
+For commercial uses of the Model & Software, please send email to [email protected]
+
+This Agreement shall be governed by the laws of the Federal Republic of Germany except for the UN Sales Convention.

README.md

@@ -6,30 +6,111 @@
 
 <h4 align="center">
 <a href="https://youtu.be/HOgaeWTih7Q" target="_blank">Video&nbsp</a>
-<a href="https://arxiv.org/pdf/2211.12499.pdf" target="_blank">Paper&nbsp</a>
+<a href="https://arxiv.org/pdf/2211.12499v2.pdf" target="_blank">Paper&nbsp</a>
 <a href="https://zielon.github.io/insta/" target="_blank">Project Website&nbsp</a>
+<a href="https://keeper.mpdl.mpg.de/d/5ea4d2c300e9444a8b0b/" target="_blank"><b>Dataset&nbsp</b></a>
+<a href="https://github.com/Zielon/metrical-tracker" target="_blank">Face Tracker&nbsp</a>
+<a href="https://github.com/Zielon/INSTA-pytorch" target="_blank">INSTA Pytorch&nbsp</a>
+<a href="mailto:&#105;&#110;&#115;&#116;&#97;&#64;&#116;&#117;&#101;&#46;&#109;&#112;&#103;&#46;&#100;&#101;">Email</a>
 </h4>
 
 <div align="center"> 
-<img src="documents/teaser.jpg">
+<img src="documents/faces.gif">
+<br>
 <i style="font-size: 1.05em;">Official Repository for CVPR 2023 paper Instant Volumetric Head Avatars</i>
 </div>
+<br>
+
+This repository is based on [instant-ngp](https://github.com/NVlabs/instant-ngp), some of the features of the original code are not available in this work. Therefore, one should restrain the program options to the main menu only.
+
+<div align="center"> 
+&#x26A0 We also prepared a Pytorch demo version of the project <a href="https://github.com/Zielon/INSTA-pytorch" target="_blank">INSTA Pytorch&nbsp</a> &#x26A0
+</div>
 
 ### Installation
 
-The repository is based on instant-ngp
+The repository is based on `instant-ngp` [commit](https://github.com/NVlabs/instant-ngp/tree/e7631da9fca9d0f3467f826fccd7a5849b3f6309). The requirements for the installation are the same, therefore please follow the [guide](https://github.com/NVlabs/instant-ngp#building-instant-ngp-windows--linux).
+Remember to use the `--recursive` option during cloning.
+
+```shell
+git clone --recursive https://github.com/Zielon/INSTA.git
+cd INSTA
+cmake . -B build
+cmake --build build --config RelWithDebInfo -j
+```
+
+### Usage and Requirements
 
-### How To Use
+After building the project you can either start training an avatar from scratch or load a snapshot. For training, we recommend a graphics card higher or equal to `RTX3090 24GB`, (we have not tested any other GPU) and `64 GB` of RAM memory. Rendering from a snapshot does not require a high-end GPU
+and can be performed even on a laptop. We have tested it on `RTX 3080 8GB` laptop version.
+
+The viewer options are the same as in the case of [instant-ngp](https://github.com/NVlabs/instant-ngp#keyboard-shortcuts-and-recommended-controls), with some additional key `F` to raycast the FLAME mesh.
+
+Usage example
+
+```shell
+# Training
+./build/rta --config insta.json --scene data/obama --height 1024 --width 1024
+
+# Loading from a checkpoint
+./build/rta --config insta.json --scene data/obama/transforms_test.json --height 1024 --width 1024 --snapshot data/obama/snapshot.msgpack
+```
 
 ### Dataset and Training
 
+We are releasing part of our dataset together with publicly available preprocessed avatars from [NHA](https://github.com/philgras/neural-head-avatars), [NeRFace](https://github.com/gafniguy/4D-Facial-Avatars) and [IMAvatar](https://github.com/zhengyuf/IMavatar).
+The output of the training (**Record Video** in menu), including rendered frames, checkpoint, etc will be saved in the `./data/{actor}/experiments/{config}/debug`.
+After the specified number of steps, the program will automatically either render all videos with the `All` option or only the currently selected one in `Mode`.
+
+[Available avatars](https://keeper.mpdl.mpg.de/d/5ea4d2c300e9444a8b0b/). Click the selected avatar to download the training dataset and the checkpoint. The avatars have to be placed in the `data` folder.
+<div align="center" dis>
+    <table class="images" width="100%"  style="border:0px solid white; width:100%;">
+        <tr style="border: 0px;">
+            <td style="border: 0px;"><a href="https://keeper.mpdl.mpg.de/f/53e9988994914c93bb51/?dl=1"><img src="documents/gifs/justin.gif" height="128" width="128"></a></td>
+            <td style="border: 0px;"><a href="https://keeper.mpdl.mpg.de/f/1a58d09b3b7442988c3e/?dl=1"><img src="documents/gifs/nf_03.gif" height="128" width="128"></a></td>
+            <td style="border: 0px;"><a href="https://keeper.mpdl.mpg.de/f/c3632aaba19542d49f1d/?dl=1"><img src="documents/gifs/nf_01.gif" height="128" width="128"></a></td>
+            <td style="border: 0px;"><a href="https://keeper.mpdl.mpg.de/f/f273e0d5c6c14d8892a0/?dl=1"><img src="documents/gifs/marcel.gif" height="128" width="128"></a></td>
+            <td style="border: 0px;"><a href="https://keeper.mpdl.mpg.de/f/9acb4822310c4d5aa555/?dl=1"><img src="documents/gifs/biden.gif" height="128" width="128"></a></td>
+        </tr>
+    </table>
+    <table class="images" width="100%"  style="border:0px solid white; width:100%;">
+        <tr style="border: 0px;">
+            <td style="border: 0px;"><a href="https://keeper.mpdl.mpg.de/f/f1545b9e7ea74f9e802b/?dl=1"><img src="documents/gifs/obama.gif" height="128" width="128"></a></td>
+            <td style="border: 0px;"><a href="https://keeper.mpdl.mpg.de/f/ba379b9a5c384722939c/?dl=1"><img src="documents/gifs/wojtek_1.gif" height="128" width="128"></a></td>
+            <td style="border: 0px;"><a href="https://keeper.mpdl.mpg.de/f/0f81a65cbdee4e01bfae/?dl=1"><img src="documents/gifs/malte_1.gif" height="128" width="128"></a></td>
+            <td style="border: 0px;"><a href="https://keeper.mpdl.mpg.de/f/ae5a0b3ae4c84c25944c/?dl=1"><img src="documents/gifs/bala.gif" height="128" width="128"></a></td>
+            <td style="border: 0px;"><a href="https://keeper.mpdl.mpg.de/f/ece2fc07bbee424f80c6/?dl=1"><img src="documents/gifs/person_0004.gif" height="128" width="128"></a></td>
+        </tr>
+    </table>
+</div>
+
+### Dataset Generation
+
+For the input generation, a conda environment is needed, and a few other repositories. Simply run `install.sh` from [scripts](https://github.com/Zielon/INSTA/tree/master/scripts) folder to prepare the workbench.
+
+Next, you can use [Metrical Photometric Tracker](https://github.com/Zielon/metrical-tracker) for the tracking of a sequence. After the processing is done run the `generate.sh` script to prepare the sequence. As input please specify the absolute path of the tracker output.
+
+**For training we recommend at least 1000 frames.**
+
+```shell
+# 1) Run the tracker for a selected actor
+python tracker.py --cfg ./configs/actors/duda.yml
+
+# 2) Generate a dataset using the script. Importantly, use the absolute path to tracker input and desired output.
+./generate.sh /metrical-tracker/output/duda INSTA/data/duda 100
+
+# ./generate.sh {input} {output} {# of test frames from the end}
+```
+
 ### Citation
+
 If you use this project in your research please cite INSTA:
+
 ```bibtex
 @proceedings{INSTA:CVPR2023,
   author = {Zielonka, Wojciech and Bolkart, Timo and Thies, Justus},
   title = {Instant Volumetric Head Avatars},
   journal = {Conference on Computer Vision and Pattern Recognition},
   year = {2023}
 }
-```
+```

configs/nerf/insta.json

@@ -2,9 +2,14 @@
 	"loss": {
 		"otype": "Huber"
 	},
-	"max_cached_bvh": 4000,
 	"nerf_network": "concatenation",
 	"recorder_steps": 33000,
+	"beta_loss": 0,
+	"use_eyes": false,
+	"optimize_latent_code": false,
+	"use_ngp": false,
+	"penalize_density_alpha": false,
+	"geo_params": 0,
 	"exp_params": 16,
 	"seg_weight": 40,
 	"depth_weight": 1.25,
@@ -24,7 +29,7 @@
 		"otype": "HashGrid",
 		"n_levels": 16,
 		"n_features_per_level": 8,
-		"log2_hashmap_size": 17,
+		"log2_hashmap_size": 18,
 		"base_resolution": 16
 	},
 	"network": {

data/README

@@ -0,0 +1,3 @@
+The avatars are available under https://keeper.mpdl.mpg.de/d/5ea4d2c300e9444a8b0b/
+
+After downloading please place them in this folder.