├── .gitignore ├── CMakeLists.txt ├── README.md ├── cover.jpg ├── cpp ├── CMakeLists.txt ├── include │ ├── automobile │ └── automobile_bits │ │ └── motorcycle.hpp └── src │ └── motorcycle.cpp ├── python ├── automobile.cpp └── motorcycle.cpp └── tests ├── enum.py ├── photograph.py └── shared_pointers.py /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | build/ 3 | -------------------------------------------------------------------------------- /CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 3.1) 2 | 3 | set(CMAKE_CXX_STANDARD 17) 4 | set(CMAKE_CXX_STANDARD_REQUIRED ON) 5 | set(CMAKE_CXX_EXTENSIONS OFF) 6 | 7 | if(NOT CMAKE_BUILD_TYPE) 8 | set(CMAKE_BUILD_TYPE Release) 9 | endif() 10 | 11 | set(CMAKE_CXX_FLAGS "-O3") 12 | set(CMAKE_CXX_FLAGS_RELEASE "-O3") 13 | 14 | project(automobile) 15 | 16 | include_directories("${CMAKE_SOURCE_DIR}/cpp/include/automobile_bits") 17 | include_directories("${CMAKE_SOURCE_DIR}/python") 18 | 19 | file (GLOB SOURCE_FILES "cpp/src/*.cpp") 20 | file (GLOB HEADER_FILES "cpp/include/automobile_bits/*.hpp") 21 | file (GLOB PYTHON_FILES "python/*.cpp" "python/*.hpp") 22 | 23 | # Set up such that XCode organizes the files 24 | source_group(TREE ${CMAKE_CURRENT_SOURCE_DIR} FILES ${SOURCE_FILES} ${HEADER_FILES} ${PYTHON_FILES} ) 25 | 26 | find_package(pybind11 REQUIRED) 27 | pybind11_add_module(automobile 28 | ${SOURCE_FILES} 29 | ${HEADER_FILES} 30 | ${PYTHON_FILES} 31 | ) 32 | 33 | target_link_libraries(automobile PUBLIC) 34 | 35 | install(TARGETS automobile 36 | COMPONENT python 37 | LIBRARY DESTINATION "${PYTHON_LIBRARY_DIR}" 38 | ) 39 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Advanced pybind11 features 2 | 3 | [You can find the corresponding Medium article here.](https://medium.com/practical-coding/three-advanced-pybind11-features-for-wrapping-c-code-into-python-6dbcac169b93) 4 | 5 | This repo details three advanced pybind11 features: 6 | * Shared pointers. 7 | * Enum. 8 | * Abstract base classes (ABC) and pure virtual methods. 9 | 10 | The starting point for this project is a previous project found [here](https://github.com/smrfeld/cmake_cpp_pybind11_tutorial). 11 | 12 | drawing 13 | 14 | [Image source](https://commons.wikimedia.org/wiki/File:Sr4002012.jpg). 15 | 16 | ## Requirements 17 | 18 | Obviously, you will need `pybind11`. On Mac: 19 | ``` 20 | brew install pybind11 21 | ``` 22 | will do it. 23 | 24 | ## Setup with CMake 25 | 26 | We will start with the `CMake` based setup from a [previous introduction found here](https://github.com/smrfeld/cmake_cpp_pybind11_tutorial). 27 | Are you gonna check it out? 28 | Of course not. 29 | No time for that! 30 | Here is the setup: 31 | 32 | The directory structure is as follows: 33 | ``` 34 | cpp/CMakeLists.txt 35 | cpp/include/automobile 36 | cpp/include/automobile_bits/motorcycle.hpp 37 | cpp/src/motorcycle.cpp 38 | python/automobile.cpp 39 | python/automobile.hpp 40 | CMakeLists.txt 41 | ``` 42 | The idea here is: 43 | 1. The inner `cpp` folder contains a `C++` project for a library. It can be built using the `CMakeLists.txt` as follows: 44 | ``` 45 | cd cpp 46 | mkdir build 47 | cd build 48 | cmake .. 49 | make 50 | make install 51 | ``` 52 | 2. The outer folder contains the wrapping code in the `python` library, and a second `CMakeLists.txt` for building the `python` library as follows: 53 | ``` 54 | mkdir build 55 | cd build 56 | cmake .. -DPYTHON_LIBRARY_DIR="/path/to/site-packages" -DPYTHON_EXECUTABLE="/path/to/executable/python3" 57 | make 58 | make install 59 | ``` 60 | My paths are: 61 | ``` 62 | DPYTHON_LIBRARY_DIR="/Users/USERNAME/opt/anaconda3/lib/python3.7/site-packages" 63 | DPYTHON_EXECUTABLE="/Users/USERNAME/opt/anaconda3/bin/python3" 64 | ``` 65 | 66 | I won't review all the files here - you can find them in this repo. 67 | 68 | ## Shared pointers 69 | 70 | `C++` standard 11 introduced shared and unique pointers which do not require manual memory cleanup. 71 | This is highly parallel to Python, where garbage collection is automatic. 72 | Wrapping shared pointers into Python is therefore only natural. 73 | 74 | We will add a static constructor method that returns a `shared_ptr` to a `Motorcycle`. Add to `cpp/include/automobile_bits/motorcycle.hpp` above the constructor: 75 | ``` 76 | /// Shared pointer constructor 77 | static std::shared_ptr create(std::string name); 78 | ``` 79 | and the implementation in `cpp/src/motorcycle.cpp`: 80 | ``` 81 | std::shared_ptr Motorcycle::create(std::string name) { 82 | return std::make_shared(name); 83 | } 84 | ``` 85 | 86 | There are two parts now: (1) we must allow a `shared_ptr` to be accessible by Python, and (2) we need to expose the `create` method. 87 | 88 | For the first part, we will modify the glue code in `python/motorcycle.cpp`: 89 | ``` 90 | // Old: 91 | // py::class_(m, "Motorcycle") 92 | // New: 93 | py::class_>(m, "Motorcycle") 94 | ``` 95 | For the second part, to wrap the static method, we will also add: 96 | ``` 97 | .def_static("create", 98 | py::overload_cast( &autos::Motorcycle::create), 99 | py::arg("name")) 100 | ``` 101 | Notice that we used `def_static` instead of `def` for a static method. 102 | 103 | Build and install the library as before. The test `python` code: 104 | ``` 105 | import automobile 106 | bike = automobile.Motorcycle.create("yamaha") 107 | bike.ride("mullholland") 108 | ``` 109 | works as expected with output: 110 | ``` 111 | Zoom Zoom on road: mullholland 112 | ``` 113 | 114 | ## Enum 115 | 116 | Enum are great for setting flags or options in a more verbose way than simply `true/false` or `1/2/3/4...`. They are supported in both `Python` and `C++`. 117 | 118 | Let's create an enum in `C++`. In the header `cpp/include/automobile_bits/motorcycle.hpp` add: 119 | ``` 120 | enum EngineType { 121 | TWO_STROKE = 0, 122 | FOUR_STROKE = 1 123 | }; 124 | ``` 125 | above the `Motorcycle` class, as well as the public method of the `Motorcycle` class: 126 | ``` 127 | /// Get engine type 128 | /// @return Engine type 129 | EngineType get_engine_type() const; 130 | ``` 131 | and it's implementation in `cpp/src/motorcycle.cpp`: 132 | ``` 133 | EngineType Motorcycle::get_engine_type() const { 134 | return EngineType::TWO_STROKE; 135 | } 136 | ``` 137 | 138 | Add the following glue code in `python/motorcycle.cpp` **below** the `Motorcycle` wrapper: 139 | ``` 140 | py::enum_(m, "EngineType") 141 | .value("TWO_STROKE", autos::EngineType::TWO_STROKE) 142 | .value("FOUR_STROKE", autos::EngineType::FOUR_STROKE) 143 | .export_values(); 144 | ``` 145 | and expose the method in the `Motorcycle` class: 146 | ``` 147 | .def("get_engine_type", py::overload_cast<>( &autos::Motorcycle::get_engine_type, py::const_ )) 148 | ``` 149 | 150 | Finally, the proof in Python is: 151 | ``` 152 | import automobile 153 | bike = automobile.Motorcycle("yamaha") 154 | bike.get_engine_type() 155 | ``` 156 | returns 157 | ``` 158 | EngineType.TWO_STROKE 159 | ``` 160 | 161 | ## Abstract base classes 162 | 163 | Python also has an `abc` module that is entirely underused. 164 | The basic principles of abstract base classes will translate nicely from `C++` to `Python`, but as we shall see some behavior is missing. 165 | 166 | Add to the header `cpp/include/automobile_bits/motorcycle.hpp`: 167 | ``` 168 | class Photograph { 169 | 170 | public: 171 | 172 | /// Constructor/destructor 173 | virtual ~Photograph() {}; 174 | 175 | /// Pure virtual method 176 | /// @param bike Bike 177 | /// @return true if beautiful 178 | virtual bool is_beautiful(std::shared_ptr bike) const = 0; 179 | }; 180 | ``` 181 | and of course, no implementation for `is_beautiful` (although you **could** have one!). 182 | 183 | What happens when we try to add the glue code in `python/motorcycle.cpp`? If we try: 184 | ``` 185 | py::class_(m, "Photograph") 186 | .def(py::init<>()) 187 | .def("is_beautiful", py::overload_cast>( &autos::Photograph::is_beautiful, py::const_ ), py::arg("bike")); 188 | ``` 189 | we'll get the error 190 | ``` 191 | Allocating an object of abstract class type 'autos::Photograph' 192 | ``` 193 | Uh oh! It looks like it is unhappy with the constructor. Of course, we could simply eliminate the constructor: 194 | ``` 195 | py::class_(m, "Photograph") 196 | .def("is_beautiful", py::overload_cast>( &autos::Photograph::is_beautiful, py::const_ ), py::arg("bike")); 197 | ``` 198 | This compiles - but now consider the following example in Python: 199 | ``` 200 | import automobile 201 | 202 | class YamahaPhoto(automobile.Photograph): 203 | 204 | def __init__(self): 205 | super().__init__() 206 | 207 | def is_beautiful(self, bike): 208 | return True 209 | 210 | bike = automobile.Motorcycle.create("yamaha") 211 | 212 | photo = YamahaPhoto() 213 | print(photo.is_beautiful(bike)) 214 | ``` 215 | This gives the error: 216 | ``` 217 | TypeError: YamahaPhoto: No constructor defined! 218 | ``` 219 | because of course, we deleted the constructor! So abstract base classes are no longer extensible. 220 | 221 | The solution is to define what `pybind11` refers to as a "trampoline" class. In `python/motorcycle.cpp`, define the trampoline at the top: 222 | ``` 223 | namespace autos { 224 | 225 | class PhotographTrampoline : public Photograph { 226 | 227 | public: 228 | 229 | using Photograph::Photograph; 230 | 231 | bool is_beautiful(std::shared_ptr bike) const override { 232 | PYBIND11_OVERLOAD_PURE( 233 | bool, /* Return type */ 234 | Photograph, /* Parent class */ 235 | is_beautiful, /* Name of function in C++ (must match Python name) */ 236 | bike /* args */ 237 | ); 238 | } 239 | }; 240 | 241 | } 242 | ``` 243 | and change the glue code to: 244 | ``` 245 | py::class_(m, "Photograph") 246 | .def(py::init<>()) 247 | .def("is_beautiful", py::overload_cast>( &autos::Photograph::is_beautiful, py::const_ ), py::arg("bike")); 248 | ``` 249 | Notice here the order in `py::class_` - first the parent class (the ABC), then the trampoline. 250 | Everywhere else, we use just the name of the ABC, i.e. `Photograph::is_beautiful`, not `PhotographTrampoline::is_beautiful`. 251 | Now we could also add the constructor without an error. 252 | 253 | The python example will now run and produce a resounding `True`. 254 | 255 | A limitation here is that the `Photograh` class in `Python` is no longer an abstract base class. That means, we can actually run the following: 256 | ``` 257 | import automobile 258 | 259 | class YamahaPhoto(automobile.Photograph): 260 | 261 | def __init__(self): 262 | super().__init__() 263 | 264 | bike = automobile.Motorcycle.create("yamaha") 265 | 266 | photo = YamahaPhoto() 267 | ``` 268 | which will construct a `YamahaPhoto` object, despite the fact that we did not implement the `is_beautiful` method. 269 | This is unfortunate, as it breaks some of the design principles enforced in `C++`. At the moment, it seems we just cannot have everything - but maybe one day! 270 | 271 | ## Final thoughts 272 | 273 | That's three advanced features of `pybind11` - some things are not so obvious, but it seems just about everything is possible! 274 | 275 | -------------------------------------------------------------------------------- /cover.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/smrfeld/advanced_pybind11_features/274167e03f91d21e2656e9fd1b516ce9751c6091/cover.jpg -------------------------------------------------------------------------------- /cpp/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 3.1) 2 | 3 | set(CMAKE_CXX_STANDARD 17) 4 | set(CMAKE_CXX_STANDARD_REQUIRED ON) 5 | set(CMAKE_CXX_EXTENSIONS OFF) 6 | 7 | project(automobile VERSION 0.1.0) 8 | 9 | # Include dir 10 | include_directories(/usr/local/include) 11 | 12 | # Src 13 | AUX_SOURCE_DIRECTORY(src SRC_FILES) 14 | 15 | # Headers 16 | set(PROJECT_SOURCE_DIR "src") 17 | set(PROJECT_INCLUDE_DIR "include/automobile_bits") 18 | 19 | # Source files 20 | set(SOURCE_FILES 21 | ${PROJECT_INCLUDE_DIR}/motorcycle.hpp 22 | ${PROJECT_SOURCE_DIR}/motorcycle.cpp 23 | ) 24 | 25 | # Set up such that XCode organizes the files correctly 26 | source_group(TREE ${CMAKE_CURRENT_SOURCE_DIR} FILES ${SOURCE_FILES}) 27 | 28 | # Add library 29 | add_library(automobile SHARED ${SOURCE_FILES}) 30 | 31 | # Include directories 32 | target_include_directories(automobile PRIVATE include/automobile_bits) 33 | 34 | # Install 35 | install(TARGETS automobile DESTINATION lib) 36 | 37 | # Install the headers 38 | install(FILES include/automobile DESTINATION include) 39 | 40 | # Create base directory 41 | install(DIRECTORY include/automobile_bits DESTINATION include) -------------------------------------------------------------------------------- /cpp/include/automobile: -------------------------------------------------------------------------------- 1 | #ifndef AUTOMOBILE_LIBRARY_H 2 | #define AUTOMOBILE_LIBRARY_H 3 | 4 | #include "automobile_bits/motorcycle.hpp" 5 | 6 | #endif -------------------------------------------------------------------------------- /cpp/include/automobile_bits/motorcycle.hpp: -------------------------------------------------------------------------------- 1 | #include 2 | 3 | #ifndef MOTORCYCLE_H 4 | #define MOTORCYCLE_H 5 | 6 | namespace autos { 7 | 8 | enum EngineType { 9 | TWO_STROKE = 0, 10 | FOUR_STROKE = 1 11 | }; 12 | 13 | class Motorcycle { 14 | 15 | private: 16 | 17 | /// Name 18 | std::string _name; 19 | 20 | public: 21 | 22 | /// Shared pointer constructor 23 | static std::shared_ptr create(std::string name); 24 | 25 | /// Constructor 26 | Motorcycle(std::string name); 27 | 28 | /// Get name 29 | /// @return Name 30 | std::string get_name() const; 31 | 32 | /// Ride the bike 33 | /// @param road Name of the road 34 | void ride(std::string road) const; 35 | 36 | /// Get engine type 37 | /// @return Engine type 38 | EngineType get_engine_type() const; 39 | }; 40 | 41 | class Photograph { 42 | 43 | public: 44 | 45 | /// Constructor/destructor 46 | virtual ~Photograph() {}; 47 | 48 | /// Pure virtual method 49 | /// @param bike Bike 50 | /// @return true if beautiful 51 | virtual bool is_beautiful(std::shared_ptr bike) const = 0; 52 | }; 53 | 54 | } 55 | 56 | #endif 57 | -------------------------------------------------------------------------------- /cpp/src/motorcycle.cpp: -------------------------------------------------------------------------------- 1 | #include "../include/automobile_bits/motorcycle.hpp" 2 | 3 | #include 4 | 5 | namespace autos { 6 | 7 | std::shared_ptr Motorcycle::create(std::string name) { 8 | return std::make_shared(name); 9 | } 10 | 11 | Motorcycle::Motorcycle(std::string name) { 12 | _name = name; 13 | } 14 | 15 | std::string Motorcycle::get_name() const { 16 | return _name; 17 | } 18 | 19 | void Motorcycle::ride(std::string road) const { 20 | std::cout << "Zoom Zoom on road: " << road << std::endl; 21 | } 22 | 23 | EngineType Motorcycle::get_engine_type() const { 24 | return EngineType::TWO_STROKE; 25 | } 26 | 27 | } 28 | -------------------------------------------------------------------------------- /python/automobile.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | 3 | namespace py = pybind11; 4 | 5 | void init_motorcycle(py::module &); 6 | 7 | namespace mcl { 8 | 9 | PYBIND11_MODULE(automobile, m) { 10 | // Optional docstring 11 | m.doc() = "Automobile library"; 12 | 13 | init_motorcycle(m); 14 | } 15 | } 16 | -------------------------------------------------------------------------------- /python/motorcycle.cpp: -------------------------------------------------------------------------------- 1 | #include "../cpp/include/automobile_bits/motorcycle.hpp" 2 | 3 | #include 4 | 5 | #include 6 | namespace py = pybind11; 7 | 8 | namespace autos { 9 | 10 | class PhotographTrampoline : public Photograph { 11 | 12 | public: 13 | 14 | using Photograph::Photograph; 15 | 16 | bool is_beautiful(std::shared_ptr bike) const override { 17 | PYBIND11_OVERLOAD_PURE( 18 | bool, /* Return type */ 19 | Photograph, /* Parent class */ 20 | is_beautiful, /* Name of function in C++ (must match Python name) */ 21 | bike /* args */ 22 | ); 23 | } 24 | }; 25 | 26 | } 27 | 28 | void init_motorcycle(py::module &m) { 29 | 30 | // Motorcycle 31 | py::class_>(m, "Motorcycle") 32 | .def(py::init(), py::arg("name")) 33 | .def_static("create", 34 | py::overload_cast( &autos::Motorcycle::create), 35 | py::arg("name")) 36 | .def("get_name", 37 | py::overload_cast<>( &autos::Motorcycle::get_name, py::const_)) 38 | .def("ride", 39 | py::overload_cast( &autos::Motorcycle::ride, py::const_), 40 | py::arg("road")) 41 | .def("get_engine_type", py::overload_cast<>( &autos::Motorcycle::get_engine_type, py::const_ )); 42 | 43 | // Photograph 44 | 45 | // Does not compile: 46 | /* 47 | py::class_(m, "Photograph") 48 | .def("is_beautiful", py::overload_cast>( &autos::Photograph::is_beautiful, py::const_ ), py::arg("bike")); 49 | */ 50 | // Compiles but error when extending Photograph in python: 51 | /* 52 | py::class_(m, "Photograph") 53 | .def(py::init<>()) 54 | .def("is_beautiful", py::overload_cast>( &autos::Photograph::is_beautiful, py::const_ ), py::arg("bike")); 55 | */ 56 | // Correct way: 57 | py::class_(m, "Photograph") 58 | .def(py::init<>()) 59 | .def("is_beautiful", py::overload_cast>( &autos::Photograph::is_beautiful, py::const_ ), py::arg("bike")); 60 | 61 | // Engine type 62 | py::enum_(m, "EngineType") 63 | .value("TWO_STROKE", autos::EngineType::TWO_STROKE) 64 | .value("FOUR_STROKE", autos::EngineType::FOUR_STROKE) 65 | .export_values(); 66 | } 67 | -------------------------------------------------------------------------------- /tests/enum.py: -------------------------------------------------------------------------------- 1 | import automobile 2 | 3 | bike = automobile.Motorcycle("yamaha") 4 | bike.get_engine_type() -------------------------------------------------------------------------------- /tests/photograph.py: -------------------------------------------------------------------------------- 1 | import automobile 2 | 3 | class YamahaPhoto(automobile.Photograph): 4 | 5 | def __init__(self): 6 | super().__init__() 7 | 8 | def is_beautiful(self, bike): 9 | return True 10 | 11 | bike = automobile.Motorcycle.create("yamaha") 12 | 13 | photo = YamahaPhoto() 14 | print(photo.is_beautiful(bike)) -------------------------------------------------------------------------------- /tests/shared_pointers.py: -------------------------------------------------------------------------------- 1 | import automobile 2 | 3 | bike = automobile.Motorcycle.create("yamaha") 4 | bike.ride("mullholland") --------------------------------------------------------------------------------