Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for local datasets. #3

Open
wants to merge 18 commits into
base: main
Choose a base branch
from
Open

Add support for local datasets. #3

wants to merge 18 commits into from

Conversation

Santoi
Copy link
Collaborator

@Santoi Santoi commented Dec 17, 2024

Motivation

This PR provides the means to run SplaTAM on previously recorded datasets with the Spectacular AI tool using an iPhone Pro 14 or higher (basically, with a depth sensor).

Description

Original SplaTAM scripts have been adapted so the config files don't have to be manually edited.
The bash script entry point now takes the usual config file, and additionally the path to the dataset previously recorded with SpectacularAI.

bash bash_scripts/nerfcapture.bash configs/iphone/nerfcapture.py <spectacularAI_dataset_location>

Change log

  • nerfcapture.bash
    Changes support SpectacularAI input and output the dataset already processed with SplaTAM.
  • nerfcapture2dataset.py
    Changes make it possible to load a local NeRFCapture dataset and make it compatible with the SplaTAM expected input.
  • final_recon.py & splatam.py
    Override config file parameters with the command line arguments.
  • RESULTS.md
    Explored technologies and obtained results overview.
  • README.md
    Run instructions for other Ekumembers.

@Santoi Santoi marked this pull request as ready for review December 20, 2024 18:09
@Santoi Santoi requested a review from hidmic December 20, 2024 18:09
@Santoi Santoi requested a review from olmerg January 7, 2025 14:05
Copy link
Collaborator

@hidmic hidmic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First pass. Are there any datasets to play with this?

@Santoi
Copy link
Collaborator Author

Santoi commented Feb 14, 2025

Thanks for your review, @hidmic ! Ready for second pass.

Added a dataset in datasets/hq-pool-table-spec.

@hidmic hidmic self-requested a review February 18, 2025 14:40
@hidmic
Copy link
Collaborator

hidmic commented Feb 21, 2025

Added a dataset in datasets/hq-pool-table-spec

@Santoi this hangs the GitHub UI for PR reviews. Does the dataset need to live here? Can we put it elsewhere and instruct the user to request access?

On an unrelated note, I see references to Frodo in the repository. This is a public repository. Results are not particularly sensitive but I'd suggest removing company terminology.

@Santoi
Copy link
Collaborator Author

Santoi commented Feb 25, 2025

Can we put it elsewhere and instruct the user to request access?

@hidmic Sure. Does it make sense to add a README in the datasets directory with those instructions? Additionally I was thinking of adding a file with the URL to request access to.

@hidmic
Copy link
Collaborator

hidmic commented Feb 26, 2025

@Santoi a README.md under datasets sounds good. A GDrive URL, on the other hand, is a bit too much IMO. I'd just instruct folks to send an email to [email protected] for the datasets. We can vet them there.

@@ -0,0 +1,71 @@
# 3D Mapping Workflow (Frodo)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Santoi nit:

Suggested change
# 3D Mapping Workflow (Frodo)
# 3D Mapping Workflow

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it helps, use an LLM to adjust the writing (punctuation, tenses, overall cohesion).


# Recording devices

List of the recording devices used during the Frodo
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Santoi nit:

Suggested change
List of the recording devices used during the Frodo
Some of the recording devices used include:


## NeRF Capture

* Only available for IOS.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Santoi nit:

Suggested change
* Only available for IOS.
* Only available on iOS.

Same elsewhere.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, link to app store would be nice.


List of the recording devices used during the Frodo

* iPhone Pro
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Santoi nit: links to product briefs would be nice.


### Online mode

Requires the host to be connected into the recording device, so as to process the input while it is being captured.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Santoi nit:

Suggested change
Requires the host to be connected into the recording device, so as to process the input while it is being captured.
Requires the host to be connected to the recording device, so as to process the input while it is being captured.

### Offline mode

Saves a directory with the RBG captured frames and the depth frames \+ a transform.json file to feed into the processing tool.
Offline mode is currently bugged. There is an issue with the recording processing, which is not converting the depth buffer correctly into 16 bit numbers. See [this author comment](https://github.com/jc211/NeRFCapture/issues/10#issuecomment-1888164311) for more details.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Santoi nit:

Suggested change
Offline mode is currently bugged. There is an issue with the recording processing, which is not converting the depth buffer correctly into 16 bit numbers. See [this author comment](https://github.com/jc211/NeRFCapture/issues/10#issuecomment-1888164311) for more details.
Offline mode is currently bugged. There is an issue with the recording process, which is not converting the depth buffer correctly into 16 bit numbers. See [this author comment](https://github.com/jc211/NeRFCapture/issues/10#issuecomment-1888164311) for more details.

Comment on lines +68 to +70
## Instant NGP

\[TODO\]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Santoi nit: remove?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants