Build-only dependencies are listed as required runtime dependencies

[johnthagen]

Checklist

• I have searched for similar issues.
• For Python issues, I have tested with the latest development wheel.
• I have checked the release documentation and the latest documentation (for master branch).

Describe the issue

I am trying to add Open3D to a production application. In this application we are trying to minimize the resulting Docker image size and number of transitive dependencies. Open3D currently seems to publish several build dependencies as runtime dependencies.

Steps to reproduce the bug

For example, on macOS / Python 3.9:

When I create manual virtual environment and run:


(venv) $ pip install open3d==0.14.1

A small number of packages are installed:

Package         Version
--------------- -------
addict          2.4.0
cycler          0.11.0
fonttools       4.33.3
joblib          1.1.0
kiwisolver      1.4.2
matplotlib      3.5.2
numpy           1.22.4
open3d          0.14.1
packaging       21.3
pandas          1.4.2
Pillow          9.1.1
pip             21.3.1
pyparsing       3.0.9
python-dateutil 2.8.2
pytz            2022.1
PyYAML          6.0
scikit-learn    1.1.1
scipy           1.8.1
setuptools      60.5.0
six             1.16.0
threadpoolctl   3.1.0
tqdm            4.64.0
wheel           0.37.1

This matches what is expected from

• Open3D/python/requirements.txt

  Line 1 in 25da5cc

  numpy>=1.18.0

And

• https://github.com/isl-org/Open3D-ML/blob/master/requirements.txt

But the downloading a Windows wheel and inspecting METADATA:

Requires-Python: >=3.6
Description-Content-Type: text/x-rst
Requires-Dist: setuptools (>=40.8.0)
Requires-Dist: wheel (>=0.36.0)
Requires-Dist: numpy (>=1.18.0)
Requires-Dist: ipywidgets (>=7.6.0)
Requires-Dist: pygments (>=2.7.4)
Requires-Dist: jupyter-packaging (~=0.10)
Requires-Dist: jupyterlab (==3.*,>=3.0.0)

There appear to be build time requirements, as listed in

Open3D/python/pyproject.toml

Line 2 in 25da5cc

requires = ["ipywidgets>=7.6.0", "pygments>=2.7.4", "jupyter_packaging~=0.10", "jupyterlab>=3.0.0,==3.*", "setuptools>=40.8.0", "wheel"]

These seem to be leaked out into the packages overall dependencies and cause tools such as Poetry to lock these build dependencies and install them even outside Windows onto mac and Linux.

See: https://pypi.org/pypi/open3d/json

"requires_dist":["setuptools (>=40.8.0)","wheel (>=0.36.0)","numpy (>=1.18.0)","ipywidgets (>=7.6.0)","pygments (>=2.7.4)","jupyter-packaging (~=0.10)","jupyterlab (==3.*,>=3.0.0)","addict","pillow (>=8.2.0)","matplotlib (>=3)","numpy (>1.15)","pandas (>=1.0)","pyyaml (>=5.4.1)","scikit-learn (>=0.21)","tqdm","pyquaternion"]

• Poetry installs large numbers of build dependencies that aren't required python-poetry/poetry#5689

Expected Behavior

jupyterlab and other build time dependencies would not be published as runtime dependencies. If they are in fact needed for Windows operation, that they be marked as Windows-only so tools such as Poetry will not install the outside of Windows.

Open3D, Python and System information

- Operating system: macOS 11.6 
- Python version: Python 3.8 / output from `import sys; print(sys.version)`
- Open3D version: output from python: `3.9.13`
- System architecture: x86
- Is this a remote workstation?: no
- How did you install Open3D?: pip
- Compiler version (if built from source): N/A

[johnthagen]

I tried the latest Windows wheel for Python 3.9 at http://www.open3d.org/docs/latest/getting_started.html#development-version-pip and it's METADATA looks better:

Requires-Python: >=3.6
Description-Content-Type: text/x-rst
Requires-Dist: numpy (>=1.18.0)
Requires-Dist: ipywidgets (>=7.6.0)
Requires-Dist: pywinpty (==2.0.2) ; sys_platform == "win32" and python_version == "3.6"

But I'm not sure if that will translate to https://pypi.org/pypi/open3d/json once a new version is published?

[johnthagen]

Somewhat related to #4535

[johnthagen]

This should be fixed by

• Support for Python Optional Dependencies #5853

[johnthagen]

For discussion about why the JSON API cannot currently support wheels that have different dependencies depending on the platform, see:

• Why does Poetry select incorrect dependencies when pip does not? python-poetry/poetry#7128

[johnthagen]

It looks like the ideal here is for Open3d to use markers (e.g. platform_system == "Windows") to explicitly mark the dependencies they have that differ between platforms.

[ssheorey]

Here is a WIP PR to address this: #5027

[ssheorey]

Hi @johnthagen a workaround for your specific use case (minimum docker image size) is to just remove any Open3D dependencies that your code does not need after installing Open3D. This may need some trial and error, depending on which Open3D functions you use.

[johnthagen]

Edit: I remembered that this was about non-used build-only dependencies. You are correct that it could be possible to try to manually uninstall these dependencies through trial and error.

@ssheorey My understanding is that this will not work because import open3d imports these dependencies at its module-level scope, and if the are missing, will raise an ImportError when, for example import dash is hit in the Open3D library.