Adding the draft status PEP for external dependency expression

[tleeuwenburg]

Hoping this will initiating a useful discussion around expressing external depencies within setup.py.

[rbtcollins]

I think this should say 'Python packages', as I don't think this pep is implying or intending to imply that installing e.g. Python3 itself would be a pip responsibility.

[tleeuwenburg]

done

[rbtcollins]

s/solved/solve/ I think

[tleeuwenburg]

done

[ncoghlan]

Another advantage to this approach it that any tool which treats setup_requires (et al) as a list of opaque strings should "just work", while a separate field causes problems.

Notably, this change should mostly be forwards compatible with the overall structure of PEP 426.

[tleeuwenburg]

added

[ncoghlan]

Revisiting this idea several years later for pypa/pipenv#2082, and I think if we ever pursue this, we're going to need to go with the "separate fields for external dependencies" approach. The problem is that there are now plenty of tools that expect requirements definitions to comply with the PEP 508 dependency specifier format, and "failed to do anything useful with the supplied external requirements definition" is a more graceful degradation to the status quo behaviour than "threw an exception complaining about a non-compliant dependency specifier".

[njsmith]

Some discussion of this PEP on distutils-sig: http://thread.gmane.org/gmane.comp.python.distutils.devel/23712

[leorochael]

Specifying install dependencies here on setup.py will be very useful, specially with regards to specifying external processes that will be needed by the package (e.g. bin!pdftk).

However, regarding dependencies into external shared libraries, it would be better if the build tool generated those automatically.

It's a lot easier for, say, setuptools or an extension to it to run ldd build/lxml_***/etree.so and figure out the library dependencies and put those in the wheel metadata, than forcing the poor package maintainer to do so to put it into setup.py.

In particular, different linux distros will have libraries that are source compatible at build time, but result in shared library dependencies with different versions on the python extension.

Forcing this information to be present in setup.py would force the package maintainer to build this package in a variety of platforms just to be able to add the proper 'lib!libsomenthing.so.X.Y:dist_name=="Z"' for every supported platform.

Whereas if the tooling automatically fed this information to the wheel metadata, not only would the package maintainer be relieved of this chore, but the package builder would be able to chose between different source compatible libraries, or even between static and dynamic linking, and the generated wheel would correctly carry the information needed to install it.

[ncoghlan]

We don't want to add the dependency automatically because the developer might want to find a way to remove it, statically link it, or figure out a suitable distro independent way of describing it, rather than including a distro specific dependency in the wheel metadata. It definitely makes sense for the build system to detect and report discrepancies between stated intent and actual build results when it comes to external dependencies, but they should still be made explicit rather than left implicit.

[tleeuwenburg]

Hi Leonardo,

I am coming to this discussion a bit late, because "life". I was involved
in authoring this PEP although rbt now has carriage of it.

We discussed the issues you mentioned in some depth. I am not myself a
packaging guru or compile expert, but I am a maintainer of small packages
and software engineer. I'm a 'user' of pip for the most part. You're not
wrong in identifying those concerns but I think there is a balance of
issues to consider. I will present my viewpoint based on my use cases:

(i) Firstly, my view is that the ability to store these things in
setup.py will more or less immediately address a very large proportion of
the really common issues and is well worth doing.

(ii) Engineering the relevant build tools is much harder than allowing
some simple manual configuration

(iii) Pulling the dependency gaps out of build tools might work for the
maintainer, but you would still need to store the dependency gap
information in some kind of file. In my use case, the install will often
fail 90 seconds into compiling some dependency, and the goal is to provide
the information back to users much faster than that. This isn't about
'fixing dependency resolution in across wheels and external packaging
systems (e.g. yum/conda), but rather about providing the pragmatic
workarounds required to fix the major pain points.

(iv) A previous accepted PEP did include some capability to include
metadata in one of the metadata files. To my knowledge it was never
implemented in tools. I think it was just too complicated. On the other
hand, I know of a lot of immediate issues a simple extension to setup.py
would allow

In general, the uses I know of are for 'the exception' rather than 'the
rule'. For example, packages like scipy are "mostly fine" to compile, but
fail in some environments.

I would also consider this might not be mutually exclusive with what you
are suggesting. It may be that the larger engineering work you are
suggesting is justified, I am not sure. I would certainly love it if there
were a two-way bridge between pip and the various external packaging
systems so that e.g. a pip install could also submit its dependencies to
yum for a request to install, however we just don't live in that world.

Thanks,
-Tennessee

On 14 August 2015 at 01:44, Leonardo Rochael Almeida <
[email protected]> wrote:

In pep-0497-externals.txt
#30 (comment)
:

+determine whether a named package is internal or external.
+
+::
+

• setup(

•    name='uses_scipy_with_atlas_extension'
  

•    install_requires='yaml, scipy[atlas]',
  

•    setup_requires='pyfortran',
  

•    external={
  

•      setup=[
  

•        'bin!fortran:dist_name=='ubuntu',
  

•        'include!kernel.h',
  

•        'include!liblas.h',
  

•      ],
  

•      install=[
  

•          'lib!liblas1.so'
  

Specifying install dependencies here on setup.py will be very useful,
specially with regards to specifying external processes that will be needed
by the package (e.g. bin!pdftk).

However, regarding dependencies into external shared libraries, it would
be better if the build tool generated those automatically.

It's a lot easier for, say, setuptools or an extension to it to run ldd
build/lxml_***/etree.so and figure out the library dependencies and put
those in the wheel metadata, than forcing the poor package maintainer to do
so to put it into setup.py.

In particular, different linux distros will have libraries that are source
compatible at build time, but result in shared library dependencies with
different versions on the python extension.

Forcing this information to be present in setup.py would force the
package maintainer to build this package in a variety of platforms just to
be able to add the proper 'lib!libsomenthing.so.X.Y:dist_name=="Z"' for
every supported platform.

Whereas if the tooling automatically fed this information to the wheel
metadata, not only would the package maintainer be relieved of this chore,
but the package builder would be able to chose between different source
compatible libraries, or even between static and dynamic linking, and the
generated wheel would correctly carry the information needed to install it.

—
Reply to this email directly or view it on GitHub
https://github.com/pypa/interoperability-peps/pull/30/files#r36989624.

---

Tennessee Leeuwenburg
http://myownhat.blogspot.com/
"Don't believe everything you think"

[qwcode]

just a minor point of order.. this should be in the "unpublished" directory. it's not really PEP497 yet, and there's actually something else now taking PEP497

[tleeuwenburg]

Yes, that's true. I was told to go ahead and use that number but history
has not passed us by. I expect someone else will do this update but I can
manage it if people would like me to.

On 14 September 2015 at 08:01, Marcus Smith [email protected]
wrote:

just a minor point of order.. this should be in the "unpublished"
directory. it's not really PEP497 yet, and there's actually something else
now taking PEP497

—
Reply to this email directly or view it on GitHub
#30 (comment)
.

---

Tennessee Leeuwenburg
http://myownhat.blogspot.com/
"Don't believe everything you think"

[tleeuwenburg]

I meant has "now" passed us by. Need more coffee :)

On 18 September 2015 at 09:11, Tennessee Leeuwenburg <[email protected]

wrote:

Yes, that's true. I was told to go ahead and use that number but history
has not passed us by. I expect someone else will do this update but I can
manage it if people would like me to.

On 14 September 2015 at 08:01, Marcus Smith [email protected]
wrote:

just a minor point of order.. this should be in the "unpublished"
directory. it's not really PEP497 yet, and there's actually something else
now taking PEP497

—
Reply to this email directly or view it on GitHub
#30 (comment)
.

---

Tennessee Leeuwenburg
http://myownhat.blogspot.com/
"Don't believe everything you think"

---

Tennessee Leeuwenburg
http://myownhat.blogspot.com/
"Don't believe everything you think"

[rbtcollins]

I've got it on my TODO.

[ppisar]

Besides dependencies on libraries, header files and programs, I'd like to see support for dependencies on pkg-config modules [http://www.freedesktop.org/wiki/Software/pkg-config/] including their versions.

[raiderrobert]

Bringing up this idea again, this may be more possible now than ever before. It was suggested over the pipenv issues list that this would be the best place to talk about a way to declare binary dependencies.