Skip to content

Commit 2943a27

Browse files
rwgkhenryiiiSkylion007virtualdiwanders
authored
squash-merge smart_holder branch into master (#5542)
* Pure `git merge --squash smart_holder` (no manual interventions). * Remove ubench/ directory. * Remove include/pybind11/smart_holder.h * [ci skip] smart_ptrs.rst updates [WIP/unfinished] * [ci skip] smart_ptrs.rst updates continued; also updating classes.rst, advanced/classes.rst * Remove README_smart_holder.rst * Restore original README.rst from master * [ci skip] Minimal change to README.rst, to leave a hint that this is pybind11v3 * [ci skip] Work in ChatGPT suggestions. * Change macro name to PYBIND11_RUN_TESTING_WITH_SMART_HOLDER_AS_DEFAULT_BUT_NEVER_USE_IN_PRODUCTION_PLEASE * Add a note pointing to the holder reinterpret_cast. * Incorporate suggestion by @virtuald: #5542 (comment) * Systematically change most py::class_ to py::classh under docs/ * Remove references to README_smart_holder.rst This should have been part of commit eb550d0. * [ci skip] Fix minor oversight (``class_`` -> ``py::class_``) noticed by chance. * [ci skip] Resolve suggestion by @virtuald #5542 (comment) * [ci skip] Apply suggestions by @timohl (thanks!) * #5542 (comment) * #5542 (comment) * #5542 (comment) * Replace `classh : class_` inhertance with `using`, as suggested by @henryiii #5542 (comment) * Revert "Systematically change most py::class_ to py::classh under docs/" This reverts commit ac9d31e. * docs: focus on py::smart_holder instead of py::classh Signed-off-by: Henry Schreiner <[email protected]> * Restore minor general fixes that got lost when ac9d31e was reverted. * Remove `- smart_holder` from list of branches in all .github/workflows * Extend classh note to explain whitespace noise motivation. * Suggest `py::smart_holder` for "most situations for safety" * Add back PYBIND11_HAS_INTERNALS_WITH_SMART_HOLDER_SUPPORT This define was * introduced with #5286 * removed with #5531 It is has been in use here: * https://github.com/pybind/pybind11_protobuf/blob/f02a2b7653bc50eb5119d125842a3870db95d251/pybind11_protobuf/native_proto_caster.h#L89-L101 Currently pybind11 unit tests for the two holder caster backwards compatibility traits * `copyable_holder_caster_shared_ptr_with_smart_holder_support_enabled` * `move_only_holder_caster_unique_ptr_with_smart_holder_support_enabled` are missing. * Add py::trampoline_self_life_support to all trampoline examples under docs/. Address suggestion by @timohl: * #5542 (comment) Add to the "please think twice" note: the overhead for safety is likely in the noise. Also fix a two-fold inconsistency introduced by revert-commit 1e646c9: 1. py::trampoline_self_life_support is mentioned in a note, but is missing in the example right before. 2. The section starting with To enable safely passing a ``std::unique_ptr`` to a trampoline object between is obsolete. * Fix whitespace accident (indentation) introduced with 1e646c9 Apparently the mis-indentation was introduced when resolving merge conflicts for what became 1e646c9 * WHITESPACE CHANGES ONLY in README.rst (list of people that made significant contributions) * Add Ethan Steinberg to list of people that made significant contributions (for completeness, unrelated to smart_holder work). * [ci skip] Add to list of people that made significant contributions: major and/or influential contributors to smart_holder branch * #2904 by @rhaschke was merged on Mar 16, 2021 * #3012 by @rhaschke was merged on May 28, 2021 * #3039 by @jakobandersen was merged on Jun 29, 2021 * #3048 by @Skylion007 was merged on Jun 18, 2021 * #3588 by @virtuald was merged on Jan 3, 2022 * #3633 by @wangxf123456 was merged on Jan 25, 2022 * #3635 by @virtuald was merged on Jan 26, 2022 * #3645 by @wangxf123456 was merged on Jan 25, 2022 * #3796 by @wangxf123456 was merged on Mar 10, 2022 * #3807 by @wangxf123456 was merged on Mar 18, 2022 * #3838 by @wangxf123456 was merged on Apr 15, 2022 * #3929 by @tomba was merged on May 7, 2022 * #4031 by @wangxf123456 was merged on Jun 27, 2022 * #4343 by @wangxf123456 was merged on Nov 18, 2022 * #4381 by @wangxf123456 was merged on Dec 5, 2022 * #4539 by @wangxf123456 was merged on Feb 28, 2023 * #4609 by @wangxf123456 was merged on Apr 6, 2023 * #4775 by @wangxf123456 was merged on Aug 3, 2023 * #4921 by @iwanders was merged on Nov 7, 2023 * #4924 by @iwanders was merged on Nov 6, 2023 * #5401 by @msimacek was merged on Oct 8, 2024 Co-authored-by: Aaron Gokaslan <[email protected]> Co-authored-by: Dustin Spicuzza <[email protected]> Co-authored-by: Ivor Wanders <[email protected]> Co-authored-by: Jakob Lykke Andersen <[email protected]> Co-authored-by: Michael Šimáček <[email protected]> Co-authored-by: Robert Haschke <[email protected]> Co-authored-by: Tomi Valkeinen <[email protected]> Co-authored-by: Xiaofei Wang <[email protected]> --------- Signed-off-by: Henry Schreiner <[email protected]> Co-authored-by: Henry Schreiner <[email protected]> Co-authored-by: Aaron Gokaslan <[email protected]> Co-authored-by: Dustin Spicuzza <[email protected]> Co-authored-by: Ivor Wanders <[email protected]> Co-authored-by: Jakob Lykke Andersen <[email protected]> Co-authored-by: Michael Šimáček <[email protected]> Co-authored-by: Robert Haschke <[email protected]> Co-authored-by: Tomi Valkeinen <[email protected]> Co-authored-by: Xiaofei Wang <[email protected]>
1 parent d8565ac commit 2943a27

File tree

66 files changed

+5522
-158
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

66 files changed

+5522
-158
lines changed

.codespell-ignore-lines

+11
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,17 @@ template <op_id id, op_type ot, typename L, typename R>
1212
template <detail::op_id id, detail::op_type ot, typename L, typename R, typename... Extra>
1313
class_ &def(const detail::op_<id, ot, L, R> &op, const Extra &...extra) {
1414
class_ &def_cast(const detail::op_<id, ot, L, R> &op, const Extra &...extra) {
15+
int valu;
16+
explicit movable_int(int v) : valu{v} {}
17+
movable_int(movable_int &&other) noexcept : valu(other.valu) { other.valu = 91; }
18+
explicit indestructible_int(int v) : valu{v} {}
19+
REQUIRE(hld.as_raw_ptr_unowned<zombie>()->valu == 19);
20+
REQUIRE(othr.valu == 19);
21+
REQUIRE(orig.valu == 91);
22+
(m.pass_valu, "Valu", "pass_valu:Valu(_MvCtor)*_CpCtor"),
23+
atyp_valu rtrn_valu() { atyp_valu obj{"Valu"}; return obj; }
24+
assert m.atyp_valu().get_mtxt() == "Valu"
25+
// valu(e), ref(erence), ptr or p (pointer), r = rvalue, m = mutable, c = const,
1526
@pytest.mark.parametrize("access", ["ro", "rw", "static_ro", "static_rw"])
1627
struct IntStruct {
1728
explicit IntStruct(int v) : value(v){};

.github/workflows/ci.yml

+14
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,20 @@ jobs:
8787
# Extra ubuntu latest job
8888
- runs-on: ubuntu-latest
8989
python: '3.11'
90+
# Run tests with py::smart_holder as the default holder
91+
# with recent (or ideally latest) released Python version.
92+
- runs-on: ubuntu-latest
93+
python: '3.12'
94+
args: >
95+
-DCMAKE_CXX_FLAGS="-DPYBIND11_RUN_TESTING_WITH_SMART_HOLDER_AS_DEFAULT_BUT_NEVER_USE_IN_PRODUCTION_PLEASE"
96+
- runs-on: macos-13
97+
python: '3.12'
98+
args: >
99+
-DCMAKE_CXX_FLAGS="-DPYBIND11_RUN_TESTING_WITH_SMART_HOLDER_AS_DEFAULT_BUT_NEVER_USE_IN_PRODUCTION_PLEASE"
100+
- runs-on: windows-2022
101+
python: '3.12'
102+
args: >
103+
-DCMAKE_CXX_FLAGS="/DPYBIND11_RUN_TESTING_WITH_SMART_HOLDER_AS_DEFAULT_BUT_NEVER_USE_IN_PRODUCTION_PLEASE /GR /EHsc"
90104
exclude:
91105
# The setup-python action currently doesn't have graalpy for windows
92106
- python: 'graalpy-24.1'

.pre-commit-config.yaml

+2
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,7 @@ repos:
7676
- id: mixed-line-ending
7777
- id: requirements-txt-fixer
7878
- id: trailing-whitespace
79+
exclude: \.patch?$
7980

8081
# Also code format the docs
8182
- repo: https://github.com/adamchainz/blacken-docs
@@ -90,6 +91,7 @@ repos:
9091
rev: "v1.5.5"
9192
hooks:
9293
- id: remove-tabs
94+
exclude: (^docs/.*|\.patch)?$
9395

9496
# Avoid directional quotes
9597
- repo: https://github.com/sirosen/texthooks

CMakeLists.txt

+4
Original file line numberDiff line numberDiff line change
@@ -131,10 +131,13 @@ set(PYBIND11_HEADERS
131131
include/pybind11/detail/common.h
132132
include/pybind11/detail/cpp_conduit.h
133133
include/pybind11/detail/descr.h
134+
include/pybind11/detail/dynamic_raw_ptr_cast_if_possible.h
134135
include/pybind11/detail/init.h
135136
include/pybind11/detail/internals.h
137+
include/pybind11/detail/struct_smart_holder.h
136138
include/pybind11/detail/type_caster_base.h
137139
include/pybind11/detail/typeid.h
140+
include/pybind11/detail/using_smart_holder.h
138141
include/pybind11/detail/value_and_holder.h
139142
include/pybind11/detail/exception_translation.h
140143
include/pybind11/attr.h
@@ -164,6 +167,7 @@ set(PYBIND11_HEADERS
164167
include/pybind11/stl.h
165168
include/pybind11/stl_bind.h
166169
include/pybind11/stl/filesystem.h
170+
include/pybind11/trampoline_self_life_support.h
167171
include/pybind11/type_caster_pyobject_ptr.h
168172
include/pybind11/typing.h
169173
include/pybind11/warnings.h)

README.rst

+29-6
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
.. figure:: https://github.com/pybind/pybind11/raw/master/docs/pybind11-logo.png
22
:alt: pybind11 logo
33

4-
**pybind11 — Seamless operability between C++11 and Python**
4+
**pybind11 (v3) — Seamless interoperability between C++ and Python**
55

66
|Latest Documentation Status| |Stable Documentation Status| |Gitter chat| |GitHub Discussions| |CI| |Build status|
77

@@ -134,11 +134,34 @@ About
134134

135135
This project was created by `Wenzel
136136
Jakob <http://rgl.epfl.ch/people/wjakob>`_. Significant features and/or
137-
improvements to the code were contributed by Jonas Adler, Lori A. Burns,
138-
Sylvain Corlay, Eric Cousineau, Aaron Gokaslan, Ralf Grosse-Kunstleve, Trent Houliston, Axel
139-
Huebl, @hulucc, Yannick Jadoul, Sergey Lyskov, Johan Mabille, Tomasz Miąsko,
140-
Dean Moldovan, Ben Pritchard, Jason Rhinelander, Boris Schäling, Pim
141-
Schellart, Henry Schreiner, Ivan Smirnov, Boris Staletic, and Patrick Stewart.
137+
improvements to the code were contributed by
138+
Jonas Adler,
139+
Lori A. Burns,
140+
Sylvain Corlay,
141+
Eric Cousineau,
142+
Aaron Gokaslan,
143+
Ralf Grosse-Kunstleve,
144+
Trent Houliston,
145+
Axel Huebl,
146+
@hulucc,
147+
Yannick Jadoul,
148+
Sergey Lyskov,
149+
Johan Mabille,
150+
Tomasz Miąsko,
151+
Dean Moldovan,
152+
Ben Pritchard,
153+
Jason Rhinelander,
154+
Boris Schäling,
155+
Pim Schellart,
156+
Henry Schreiner,
157+
Ivan Smirnov,
158+
Dustin Spicuzza,
159+
Boris Staletic,
160+
Ethan Steinberg,
161+
Patrick Stewart,
162+
Ivor Wanders,
163+
and
164+
Xiaofei Wang.
142165

143166
We thank Google for a generous financial contribution to the continuous
144167
integration infrastructure used by this project.

docs/advanced/cast/stl.rst

+2-2
Original file line numberDiff line numberDiff line change
@@ -169,8 +169,8 @@ macro must be specified at the top level (and outside of any namespaces), since
169169
it adds a template instantiation of ``type_caster``. If your binding code consists of
170170
multiple compilation units, it must be present in every file (typically via a
171171
common header) preceding any usage of ``std::vector<int>``. Opaque types must
172-
also have a corresponding ``class_`` declaration to associate them with a name
173-
in Python, and to define a set of available operations, e.g.:
172+
also have a corresponding ``py::class_`` declaration to associate them with a
173+
name in Python, and to define a set of available operations, e.g.:
174174

175175
.. code-block:: cpp
176176

docs/advanced/classes.rst

+44-29
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ helper class that is defined as follows:
6464

6565
.. code-block:: cpp
6666
67-
class PyAnimal : public Animal {
67+
class PyAnimal : public Animal, py::trampoline_self_life_support {
6868
public:
6969
/* Inherit the constructors */
7070
using Animal::Animal;
@@ -80,6 +80,18 @@ helper class that is defined as follows:
8080
}
8181
};
8282
83+
The ``py::trampoline_self_life_support`` base class is needed to ensure
84+
that a ``std::unique_ptr`` can safely be passed between Python and C++. To
85+
steer clear of notorious pitfalls (e.g. inheritance slicing), it is best
86+
practice to always use the base class, in combination with
87+
``py::smart_holder``.
88+
89+
.. note::
90+
For completeness, the base class has no effect if a holder other than
91+
``py::smart_holder`` used, including the default ``std::unique_ptr<T>``.
92+
Please think twice, though, the pitfalls are very real, and the overhead
93+
for using the safer ``py::smart_holder`` is very likely to be in the noise.
94+
8395
The macro :c:macro:`PYBIND11_OVERRIDE_PURE` should be used for pure virtual
8496
functions, and :c:macro:`PYBIND11_OVERRIDE` should be used for functions which have
8597
a default implementation. There are also two alternate macros
@@ -95,18 +107,18 @@ The binding code also needs a few minor adaptations (highlighted):
95107
:emphasize-lines: 2,3
96108
97109
PYBIND11_MODULE(example, m) {
98-
py::class_<Animal, PyAnimal /* <--- trampoline*/>(m, "Animal")
110+
py::class_<Animal, PyAnimal /* <--- trampoline */, py::smart_holder>(m, "Animal")
99111
.def(py::init<>())
100112
.def("go", &Animal::go);
101113
102-
py::class_<Dog, Animal>(m, "Dog")
114+
py::class_<Dog, Animal, py::smart_holder>(m, "Dog")
103115
.def(py::init<>());
104116
105117
m.def("call_go", &call_go);
106118
}
107119
108120
Importantly, pybind11 is made aware of the trampoline helper class by
109-
specifying it as an extra template argument to :class:`class_`. (This can also
121+
specifying it as an extra template argument to ``py::class_``. (This can also
110122
be combined with other template arguments such as a custom holder type; the
111123
order of template types does not matter). Following this, we are able to
112124
define a constructor as usual.
@@ -116,9 +128,9 @@ Bindings should be made against the actual class, not the trampoline helper clas
116128
.. code-block:: cpp
117129
:emphasize-lines: 3
118130
119-
py::class_<Animal, PyAnimal /* <--- trampoline*/>(m, "Animal");
131+
py::class_<Animal, PyAnimal /* <--- trampoline */, py::smart_holder>(m, "Animal");
120132
.def(py::init<>())
121-
.def("go", &PyAnimal::go); /* <--- THIS IS WRONG, use &Animal::go */
133+
.def("go", &Animal::go); /* <--- DO NOT USE &PyAnimal::go HERE */
122134
123135
Note, however, that the above is sufficient for allowing python classes to
124136
extend ``Animal``, but not ``Dog``: see :ref:`virtual_and_inheritance` for the
@@ -244,13 +256,13 @@ override the ``name()`` method):
244256

245257
.. code-block:: cpp
246258
247-
class PyAnimal : public Animal {
259+
class PyAnimal : public Animal, py::trampoline_self_life_support {
248260
public:
249261
using Animal::Animal; // Inherit constructors
250262
std::string go(int n_times) override { PYBIND11_OVERRIDE_PURE(std::string, Animal, go, n_times); }
251263
std::string name() override { PYBIND11_OVERRIDE(std::string, Animal, name, ); }
252264
};
253-
class PyDog : public Dog {
265+
class PyDog : public Dog, py::trampoline_self_life_support {
254266
public:
255267
using Dog::Dog; // Inherit constructors
256268
std::string go(int n_times) override { PYBIND11_OVERRIDE(std::string, Dog, go, n_times); }
@@ -272,7 +284,7 @@ declare or override any virtual methods itself:
272284
.. code-block:: cpp
273285
274286
class Husky : public Dog {};
275-
class PyHusky : public Husky {
287+
class PyHusky : public Husky, py::trampoline_self_life_support {
276288
public:
277289
using Husky::Husky; // Inherit constructors
278290
std::string go(int n_times) override { PYBIND11_OVERRIDE_PURE(std::string, Husky, go, n_times); }
@@ -287,13 +299,15 @@ follows:
287299

288300
.. code-block:: cpp
289301
290-
template <class AnimalBase = Animal> class PyAnimal : public AnimalBase {
302+
template <class AnimalBase = Animal>
303+
class PyAnimal : public AnimalBase, py::trampoline_self_life_support {
291304
public:
292305
using AnimalBase::AnimalBase; // Inherit constructors
293306
std::string go(int n_times) override { PYBIND11_OVERRIDE_PURE(std::string, AnimalBase, go, n_times); }
294307
std::string name() override { PYBIND11_OVERRIDE(std::string, AnimalBase, name, ); }
295308
};
296-
template <class DogBase = Dog> class PyDog : public PyAnimal<DogBase> {
309+
template <class DogBase = Dog>
310+
class PyDog : public PyAnimal<DogBase>, py::trampoline_self_life_support {
297311
public:
298312
using PyAnimal<DogBase>::PyAnimal; // Inherit constructors
299313
// Override PyAnimal's pure virtual go() with a non-pure one:
@@ -311,9 +325,9 @@ The classes are then registered with pybind11 using:
311325

312326
.. code-block:: cpp
313327
314-
py::class_<Animal, PyAnimal<>> animal(m, "Animal");
315-
py::class_<Dog, Animal, PyDog<>> dog(m, "Dog");
316-
py::class_<Husky, Dog, PyDog<Husky>> husky(m, "Husky");
328+
py::class_<Animal, PyAnimal<>, py::smart_holder> animal(m, "Animal");
329+
py::class_<Dog, Animal, PyDog<>, py::smart_holder> dog(m, "Dog");
330+
py::class_<Husky, Dog, PyDog<Husky>, py::smart_holder> husky(m, "Husky");
317331
// ... add animal, dog, husky definitions
318332
319333
Note that ``Husky`` did not require a dedicated trampoline template class at
@@ -499,12 +513,12 @@ an alias:
499513
// ...
500514
virtual ~Example() = default;
501515
};
502-
class PyExample : public Example {
516+
class PyExample : public Example, py::trampoline_self_life_support {
503517
public:
504518
using Example::Example;
505519
PyExample(Example &&base) : Example(std::move(base)) {}
506520
};
507-
py::class_<Example, PyExample>(m, "Example")
521+
py::class_<Example, PyExample, py::smart_holder>(m, "Example")
508522
// Returns an Example pointer. If a PyExample is needed, the Example
509523
// instance will be moved via the extra constructor in PyExample, above.
510524
.def(py::init([]() { return new Example(); }))
@@ -550,9 +564,10 @@ pybind11. The underlying issue is that the ``std::unique_ptr`` holder type that
550564
is responsible for managing the lifetime of instances will reference the
551565
destructor even if no deallocations ever take place. In order to expose classes
552566
with private or protected destructors, it is possible to override the holder
553-
type via a holder type argument to ``class_``. Pybind11 provides a helper class
554-
``py::nodelete`` that disables any destructor invocations. In this case, it is
555-
crucial that instances are deallocated on the C++ side to avoid memory leaks.
567+
type via a holder type argument to ``py::class_``. Pybind11 provides a helper
568+
class ``py::nodelete`` that disables any destructor invocations. In this case,
569+
it is crucial that instances are deallocated on the C++ side to avoid memory
570+
leaks.
556571

557572
.. code-block:: cpp
558573
@@ -871,7 +886,7 @@ Multiple Inheritance
871886

872887
pybind11 can create bindings for types that derive from multiple base types
873888
(aka. *multiple inheritance*). To do so, specify all bases in the template
874-
arguments of the ``class_`` declaration:
889+
arguments of the ``py::class_`` declaration:
875890

876891
.. code-block:: cpp
877892
@@ -946,23 +961,23 @@ because of conflicting definitions on the external type:
946961
// dogs.cpp
947962
948963
// Binding for external library class:
949-
py::class<pets::Pet>(m, "Pet")
964+
py::class_<pets::Pet>(m, "Pet")
950965
.def("name", &pets::Pet::name);
951966
952967
// Binding for local extension class:
953-
py::class<Dog, pets::Pet>(m, "Dog")
968+
py::class_<Dog, pets::Pet>(m, "Dog")
954969
.def(py::init<std::string>());
955970
956971
.. code-block:: cpp
957972
958973
// cats.cpp, in a completely separate project from the above dogs.cpp.
959974
960975
// Binding for external library class:
961-
py::class<pets::Pet>(m, "Pet")
976+
py::class_<pets::Pet>(m, "Pet")
962977
.def("get_name", &pets::Pet::name);
963978
964979
// Binding for local extending class:
965-
py::class<Cat, pets::Pet>(m, "Cat")
980+
py::class_<Cat, pets::Pet>(m, "Cat")
966981
.def(py::init<std::string>());
967982
968983
.. code-block:: pycon
@@ -980,13 +995,13 @@ the ``py::class_`` constructor:
980995
.. code-block:: cpp
981996
982997
// Pet binding in dogs.cpp:
983-
py::class<pets::Pet>(m, "Pet", py::module_local())
998+
py::class_<pets::Pet>(m, "Pet", py::module_local())
984999
.def("name", &pets::Pet::name);
9851000
9861001
.. code-block:: cpp
9871002
9881003
// Pet binding in cats.cpp:
989-
py::class<pets::Pet>(m, "Pet", py::module_local())
1004+
py::class_<pets::Pet>(m, "Pet", py::module_local())
9901005
.def("get_name", &pets::Pet::name);
9911006
9921007
This makes the Python-side ``dogs.Pet`` and ``cats.Pet`` into distinct classes,
@@ -1104,7 +1119,7 @@ described trampoline:
11041119
virtual int foo() const { return 42; }
11051120
};
11061121
1107-
class Trampoline : public A {
1122+
class Trampoline : public A, py::trampoline_self_life_support {
11081123
public:
11091124
int foo() const override { PYBIND11_OVERRIDE(int, A, foo, ); }
11101125
};
@@ -1114,7 +1129,7 @@ described trampoline:
11141129
using A::foo;
11151130
};
11161131
1117-
py::class_<A, Trampoline>(m, "A") // <-- `Trampoline` here
1132+
py::class_<A, Trampoline, py::smart_holder>(m, "A") // <-- `Trampoline` here
11181133
.def("foo", &Publicist::foo); // <-- `Publicist` here, not `Trampoline`!
11191134
11201135
Binding final classes
@@ -1195,7 +1210,7 @@ but once again each instantiation must be explicitly specified:
11951210
T fn(V v);
11961211
};
11971212
1198-
py::class<MyClass<int>>(m, "MyClassT")
1213+
py::class_<MyClass<int>>(m, "MyClassT")
11991214
.def("fn", &MyClass<int>::fn<std::string>);
12001215
12011216
Custom automatic downcasters

0 commit comments

Comments
 (0)