Create a requirements files with all the diffpy-cmi package dependencies so `conda install diffpy.cmi` installs everything (@Tieqiong)

[cadenmyers13]

Assigned to @Tieqiong

[sbillinge]

please see https://github.com/diffpy/conda-recipes/tree/main/diffpy-cmi for this.

We will NOT include diffpy-sasview in this release.
We will TRY and include pyobjcryst/libobjcryst

[cadenmyers13]

@sbillinge @Tieqiong whats the status on this? Can I do anything to help here?

[Tieqiong]

@sbillinge @cadenmyers13 diffpy.srreal and pyobjcryst are still not up to date on conda-forge yet, still waiting for them.

Also we should decide our philosophy on this. conda dependencies are installed recursively, so dependencies of a dependency would also be installed. We have diffpy.srreal depending on a lot of stuffs (libdiffpy, libobjcryst, pyobjcryst, diffpy.structure). Should we still add them into dependencies?

A popular view on this is to add everything that might be directly dependent on into the list too, such that downstream changes will not break current package. So from this point of view, I think we should include diffpy.srfit, diffpy.structure, pyobjcryst, diffpy.utils, diffpy.srreal (use cases from pdfttp_data). Do we want numpy and scipy?

This also relates to the problem of optional requirements.

[cadenmyers13]

@Tieqiong That makes sense to me. I think we should list them all then to be safe. I dont see why we wouldn't do the same for numpy and scipy too. Is there any harm in redundantly listing dependencies?

[Tieqiong]

It will make it huge, like taking minutes (including download time, dependency-resolution time, etc.) to install and use up gigabytes of disk space. It might be fine if the user is going to use everything, but very possibly not. I think we want diffpy.cmi to be extendable (as it's said in the summary: a modular framework)
I'm not particularly concerned about version resolution if we give loose or no version pins, but I think version conflicts are still possible...

[sbillinge]

This is the conversation about extendability coming!

The core of cmi is the modeling framework, which would be

• srfit, structure and utils
  then for pdf it would be
• srreal, pyobjcryst

Arguably structure could also be an extension, but it will be used widely, for example in a sas module or a raman or xafs or anything.

I don't know the answer to this. one is that we have a diffpy.cmi-core then extensions like diffpy.cmi-diffraction and so on. These are just lists of requirements. I'm not sure how we do that without making separate packages which I don't particularly want to do, but bundles of requirements is what I am after.

Some structure like this would probably be hte most rational. If we wanted to test things more broadly, we could include cmi-core in conda.txt but cmi-diffraction in tests.txt. The tutorial/example docs would have to say, install this and that.

[sbillinge]

that said, it may be more trouble than it is worth....lots to maintain there.....

[Tieqiong]

check this: One feedstock, many bundles!

We should be able to replicate how matplotlib does for having both matplotlib and matplotlib-base

[sbillinge]

check this: One feedstock, many bundles!

We should be able to replicate how matplotlib does for having both matplotlib and matplotlib-base

this will work at conda-forge. Would we bundle everything at PyPI and split it out at conda-forge? Then we recommend that people install from conda-forge but if they want they can get from pypi but at the cost of a huge blob?

Could we also just have a bunch of different installation files? so requirements/conda.txt (core) /diffraction.txt (diffraction) /sas.txt (sas) and so on. Then:

1. for users:, conda install diffpy.cmi gives them core. Then we need a way for them to do conda install --file diffraction.txt to get the rest. The question would be how do they get diffraction.txt. Is there a way of bundling this with the distribution and having an easy, non-annoying, way of accessing it?
2. for CI/testing: we just add everything in tests.txt

[Tieqiong]

@sbillinge The problem with PyPI is we will not get libobjcryst and libdiffpy from there, so essentially the installed stack will not work solely by pip install. And it's not guarantee to work even if the user installed them from conda-forge afterward, if packages and libraries are not maintained well, because of version mismatch.

We can have a bunch of different installation files, but they are only for advanced users and maintainers. We can do something like:
conda install --file <(curl -sSL https://raw.githubusercontent.com/diffpy/diffpy.cmi/refs/heads/main/requirements/conda.txt)
but the user will need to connect to the internet, and we have to make sure the requirements list is always up to date, and GitHub is running well. We also need to let the users know this, either in documentation or in help (diffpy.cmi -h).

If we bundle the requirement lists with the distribution, we can write a command script to install the dependencies for the user, e.g. diffpy.cmi --install-diffraction will run a shell script from a python module that install the dependencies. Still, it needs to be in the documentation.

I think there should be a way to bundle files inside installation from pyproject.toml, but I think this file has to be inside the package (e.g. under some __init__ or some defined package path). It would be easier to achieve using a setup.py.

[sbillinge]

Let's just make a big blob for now and install everything, but I really like these ideas. We can do the install scripts for the next release.

RE PyPI we will just tell people that they have to install from conda-forge. It's still a major upgrade to have it working on Py3.13 and windows!