├── README.md └── guidelines.rst /README.md: -------------------------------------------------------------------------------- 1 | # api-evolution 2 | 3 | Discussions about issues related to the gradual evolution of the C API. 4 | Proposals here are expected to be relatively non-controversial, 5 | appeal to common sense, and not require big implementation or adoption efforts. 6 | 7 | When creating an issue that summarizes some discussion elsewhere, 8 | please link to the original discussion if you can (but *do* **summarize**). 9 | 10 | Try to indicate how controversial you expect a particular guideline to be. 11 | 12 | The goal of this repo is to collect the raw materials for a 13 | [process PEP](https://peps.python.org/pep-0001/#pep-types), 14 | to be co-authored by the C API Workgroup members 15 | (we don't know who that will be yet, but see #14). 16 | 17 | That process PEP will be a living document moderated by the C API Workgroup 18 | members (listed as co-authors of the PEP), similar to PEP 8. 19 | The dev guide will direct developers to the PEP for guidelines. 20 | Like [PEP 8](https://peps.python.org/pep-0008/), and unlike 21 | [standards track PEPs](https://peps.python.org/pep-0001/#pep-types), 22 | the PEP should focus on providing actionable guidelines, 23 | rather than on convincing the SC of the correctness of a "specification". 24 | 25 | (I originally thought that this would be an informational PEP, 26 | but after reading 27 | [that section](https://peps.python.org/pep-0001/#pep-types) 28 | of PEP 1 again, I think it should be a process PEP.) 29 | 30 | For more radical proposals (e.g. HPy), 31 | please open an issue in [api-revolution](https://github.com/capi-workgroup/api-revolution). 32 | -------------------------------------------------------------------------------- /guidelines.rst: -------------------------------------------------------------------------------- 1 | PEP: 9999 2 | Title: Guidelines for Python's C API 3 | Author: Petr Viktorin , 4 | Guido van Rossum , 5 | Victor Stinner , 6 | Steve Dower , 7 | Erlend Egeberg Aasland , 8 | Serhiy Storchaka, 9 | Michael Droettboom, 10 | Discussions-To: https://discuss.python.org/c/c-api/30 11 | Status: Draft 12 | Type: Process 13 | Requires: 731 14 | Created: 20-Nov-2023 15 | Post-History: XXX 16 | 17 | .. pep-banner:: 18 | 19 | This is a **incomplete draft**, published for initial discussion. 20 | It is likely to be amended substantially. 21 | 22 | 23 | Abstract 24 | ======== 25 | 26 | TODO: Write the abstract last :) 27 | 28 | 29 | About this document 30 | =================== 31 | 32 | These guidelines represent the consensus of the C API Working Group, 33 | established in :pep:`731` to oversee and coordinate 34 | the development and maintenance of the CPython C API. 35 | 36 | It is a living document. It can be changed with the approval of either 37 | all members of the C API Working Group, or the Steering Council. 38 | 39 | 40 | High-level goals 41 | ================ 42 | 43 | Our goal is that the C API is: 44 | 45 | * **Useful**: It should provide access to all relevant functionality. 46 | * **Ergonomic**: It should be easy to use. 47 | * **Fair**: It should serve the purposes of a variety of stakeholders. 48 | * **Stable**: It should avoid requiring users to update their code more than necessary. 49 | * **Evolvable**: It should give Python implementers enough freedom to change implementation details. 50 | * **Maintainable**: It should be easy to maintain and extend. 51 | 52 | .. note:: 53 | 54 | For background and discussions, see: 55 | 56 | - https://github.com/capi-workgroup/api-evolution/issues/26 57 | 58 | 59 | Meta-guidelines 60 | =============== 61 | 62 | Don't panic! 63 | ------------ 64 | 65 | We do not expect you, fellow CPython contributor, to read and remember 66 | all of the guidelines. 67 | If you have any doubts, or if you find that the guidelines are unclear, 68 | incomplete or contradictory, ask the C API Working Group for help 69 | and clarifications. 70 | 71 | 72 | Working group approvals 73 | ----------------------- 74 | 75 | In several cases, you should seek approval from the C API Working Group 76 | before changing public C API. 77 | Most notably: 78 | 79 | * when the guidelines tell you to do so, 80 | * when adding an exception to the rules, 81 | * when adding to the *limited* API, 82 | * when adding something that is likely to establish precedent, 83 | * when adding many APIs at once, or 84 | * when the guidelines are unclear or don't make sense. 85 | 86 | Approvals allow the working group to coordinate designs from multiple people, 87 | and to collect feedback needed to refine these guidelines. 88 | 89 | Request an approval by opening `an issue in the decisions repository`_. 90 | 91 | .. _an issue in the decisions repository: https://github.com/capi-workgroup/decisions/issues 92 | 93 | 94 | Exceptions to the guidelines 95 | ---------------------------- 96 | 97 | Where possible, exceptions to these guidelines should be added as 98 | *alternatives* to API that follows the guidelines. 99 | Such exceptions should be clearly identifiable from their names. 100 | 101 | If you want to add an exception that is not mentioned in the guidelines, 102 | please request approval from the C API Working Group. 103 | 104 | .. note:: 105 | 106 | For background and discussions, see: 107 | 108 | - https://github.com/capi-workgroup/api-evolution/issues/4 109 | 110 | 111 | Applicability 112 | ============= 113 | 114 | These guidelines only apply to API that is: 115 | 116 | * *Newly added*: existing API does not necessarily follow these guidelines; 117 | this PEP is not the place for plans to replace, deprecate or remove it. 118 | * *Public*: meant for users; that is, not in the *private* or *internal* 119 | tiers as defined in the next section. 120 | 121 | Note that we have a style guide, :pep:`7`, which applies to *all* C code 122 | in CPython, including the API. 123 | 124 | 125 | API tiers 126 | --------- 127 | 128 | CPython has several tiers of C API. 129 | The more stable the tier is, the stricter the rules for it are: 130 | 131 | * For **internal** API (only accessible with ``Py_BUILD_CORE``, or not declared 132 | in a public header at all), you are free to ignore these guidelines entirely. 133 | 134 | * **Private** API (not intended for users) should be *internal* if possible. 135 | Otherwise it must use the ``_Py`` prefix (see :ref:`naming`). 136 | The rest of these guidelines don't apply to it. 137 | 138 | * In **unstable** API (prefixed with ``PyUnstable_``), it is OK to ignore 139 | these guidelines if there is a reason to do so -- usually for performance. 140 | Please document the reason in a source comment. 141 | 142 | * In the **general** public API, these guidelines apply in full force. 143 | The C API working group can grant exceptions. 144 | 145 | * For the **limited** API, always seek explicit approval from the 146 | C API Working Group. 147 | 148 | 149 | .. note:: 150 | 151 | For background and discussions, see: 152 | 153 | - https://github.com/capi-workgroup/api-evolution/issues/42 (for limited API) 154 | 155 | 156 | One header 157 | ========== 158 | 159 | All public API should be available after including :file:`Python.h`. 160 | 161 | To allow selecting an alternate API, such as a subset, 162 | use feature flags/macros that users define before including the header 163 | (for example ``Py_LIMITED_API``). 164 | Adding such a feature macro needs approval from the C API Working Group. 165 | 166 | .. note:: 167 | 168 | For background and discussions, see: 169 | 170 | - https://github.com/capi-workgroup/api-evolution/issues/34 171 | 172 | 173 | .. XXX add a PEP number prefix to the anchor: 174 | 175 | .. _naming: 176 | 177 | Naming 178 | ====== 179 | 180 | [TODO: PEP 7 should be updated to link here once this PEP goes live.] 181 | 182 | All newly added public names must be prefixed with ``Py``. 183 | 184 | Names that users should not use directly, but need to be visible to the 185 | compiler/linker, should be prefixed with ``_Py``. 186 | (Such names are not considered public API, that is, they should not appear in 187 | third-party source code.) 188 | 189 | This applies to all names in a global namespace: functions, macros, variables, 190 | typedefs, structs, enums, etc.; not to parameters or struct fields. 191 | 192 | The ``Py_`` prefix is reserved for global service routines like 193 | ``Py_FatalError``; specific groups of APIs use a longer prefix, 194 | for example ``PyUnicode_`` for string functions. 195 | Use an existing prefix when applicable. If you want to add a new prefix, 196 | contact the C API Working Group. 197 | 198 | The ``Py`` prefix is in mixed case, even in macro names. 199 | For example: ``PyUnicode_AS_STRING``. 200 | (Several existing macros use the upper-case ``PY``; if you need this prefix 201 | for consistency, please get approval from the C API Working Group.) 202 | 203 | Unstable API is prefixed with ``PyUnstable_`` instead of ``Py``, 204 | for example ``PyUnstable_Long_IsCompact`` or (hypothetically) 205 | ``PyUnstable_String_GET_SIZE``. 206 | 207 | .. note:: 208 | 209 | For background and discussions, see: 210 | 211 | - https://github.com/capi-workgroup/api-evolution/issues/21 212 | 213 | 214 | Do not reuse names 215 | ------------------ 216 | 217 | If an API's interface or behavior changes in a backwards-incompatible way, 218 | add new API with a new name. 219 | You can deprecate and remove the old version following Python's 220 | :pep:`backwards compatibility policy <387>`. 221 | 222 | After API has been removed, do not reuse the old name, 223 | since existing documentation and tutorials will continue to refer to the 224 | old behavior. 225 | 226 | 227 | Language support 228 | ================ 229 | 230 | C standard and dialect 231 | ---------------------- 232 | 233 | [TODO: PEP 7 should be updated to link here once this PEP goes live.] 234 | 235 | Public C API must be compatible with: 236 | 237 | - C11, with optional features needed by CPython: 238 | 239 | - IEEE 754 floating point 240 | - Atomics (``!__STDC_NO_ATOMICS__``, or MSVC) 241 | 242 | - C99 243 | - C89 with several select C99 features: 244 | 245 | - ```` and ```` 246 | - ``static inline`` functions 247 | - designated initializers 248 | - intermingled declarations 249 | - line comments (``//``) 250 | 251 | - C++03 252 | 253 | It is OK to use other features -- compiler-specific ones, 254 | optional standard ones, or platform-specific ones -- if: 255 | 256 | - the behavior of correct user code is the same as with a standard compiler, 257 | - the feature is detected using appropriate preprocessor checks, and 258 | - their use does not produce warnings on any supported compiler, 259 | including earlier versions of the one it is specific to. 260 | 261 | For example, compiler-specific code is often used to improve performance 262 | or compiler diagnostics. 263 | 264 | It is also OK to use these other features for platform-specific API, 265 | which needs to be documented as such and have a feature test macro 266 | (for example, ``Py_HAVE_C_COMPLEX``). 267 | 268 | .. note:: 269 | 270 | The existing API uses a few C11 features which are 271 | commonly available as compiler extensions to C99. 272 | In particular, we do use an unnamed union. 273 | New API should not use these features. 274 | 275 | All function declarations and definitions must use full prototypes, 276 | that is, the types of all arguments must be specified. 277 | 278 | .. note:: 279 | 280 | For background and discussions, see: 281 | 282 | - https://github.com/capi-workgroup/api-evolution/issues/22 283 | 284 | 285 | Portable subset of C 286 | -------------------- 287 | 288 | While the C API is defined in terms of the C language, it supports building 289 | wrappers for languages other than C, such as Rust, Java, assembly, or 290 | Python with ``ctypes``. 291 | These wrappers cannot realisically use a full-featured C parser. 292 | To make the public API easier to describe and wrap, it should 293 | avoid some of C's features: 294 | 295 | * Avoid ``static inline`` functions and macros. 296 | All functions must be exported as actual library symbols. 297 | * Avoid variadic functions. 298 | * Avoid C-specific types like ``long long``, ``enum`` or bit fields 299 | (see :ref:`types`). 300 | 301 | Once you add API that conforms to this “portable subset”, 302 | you can add additional C/C++-specific API. Usually, the additional API 303 | will be either more performant, or easier to use from C. 304 | For example: 305 | 306 | * A function may be shadowed by a ``static inline`` function or macro with 307 | the same behavior. Typically, this allows better performance for C/C++. 308 | See `shadowing example`_. 309 | * A function that uses C-specific types, such as ``PyLong_AsLongLong``, 310 | is OK if equivalent functions are provided for :ref:`the preferred types `. 311 | * A variadic function is OK if there's a non-variadic function 312 | with equivalent functionality. 313 | Usually, the equivalent take an array or pre-constructed 314 | Python object: for example, `PyObject_Vectorcall` is the equivalent 315 | for `PyObject_CallFunction`; `PySys_AuditTuple` for `PySys_Audit`; 316 | `PyErr_SetObject` for `PyErr_Format`. 317 | 318 | Macros can be used in the following cases: 319 | 320 | * Feature flags (e.g. ``HAVE_FORK``, ``Py_LIMITED_API``) 321 | * Simple constants (e.g. ``Py_TPFLAGS_BASETYPE``, ``PY_VERSION_HEX``). 322 | * Shortcuts for functionality that can be accomplished trivially, 323 | but perhaps tediously, without macros (e.g. ``Py_VISIT``, 324 | ``Py_BEGIN_ALLOW_THREADS``, ``Py_RETURN_RICHCOMPARE``). 325 | In this case, the macro-less equivalent should be clear from documentation: 326 | consider adding the macro's expansion to the docs. 327 | Non-C wrappers are expected re-implement these macros. 328 | * Features that aren't needed in non-C languages (e.g. ``Py_MAX``, 329 | ``Py_STRINGIFY``). 330 | * Macros used to define the API (e.g. ``PyAPI_FUNC``, ``Py_ALWAYS_INLINE``, 331 | ``Py_OBJECT_H``). 332 | * As an implementation detail (for example, when shadowing a function). 333 | 334 | As always, new exceptions can be added with approval from the C API 335 | working group. 336 | 337 | .. note:: 338 | 339 | For background and discussions, see: 340 | 341 | - :pep:`670` 342 | - https://github.com/capi-workgroup/api-evolution/issues/11 (Treat the ABI as an API) 343 | - https://github.com/capi-workgroup/api-evolution/issues/18 (Avoid macros and static inline functions) 344 | - https://github.com/capi-workgroup/api-evolution/issues/12 (variadics) 345 | 346 | 347 | .. _types: 348 | 349 | Types 350 | ===== 351 | 352 | Arithmetic types 353 | ---------------- 354 | 355 | Avoid types with compiler-/platform-specific sizes, such as ``long`` 356 | or ``unsigned short``. 357 | 358 | Instead, use: 359 | 360 | * ``int32_t`` and other C99+ fixed width integer types 361 | * ``Py_ssize_t``, ``intptr_t``, ``ptrdiff_t`` for values of the appropriate 362 | platform-specific types 363 | * ``double`` (IEEE 754 ``binary64``) 364 | 365 | As an exception, use ``int`` for small ranges (typically, as a replacement 366 | for enum). 367 | If the 16-bit limit is relevant, and for all unsigned values, prefer 368 | explicit fixed width types over ``int``. 369 | 370 | As another exception, use ``char*`` for UTF-8/ASCII strings (and ``char`` 371 | for bytes of such text). For byte-strings, prefer ``uint8_t*`` or ``void*``. 372 | 373 | For memory sizes and byte counts, use the signed ``Py_ssize_t``, 374 | not the unsigned ``size_t``. 375 | 376 | 377 | Enums and bitfields 378 | ------------------- 379 | 380 | Avoid ``enum``, which have compiler-/platform-dependent size. 381 | (CPython cannot yet use C23's fixed underlying `enum` types.) 382 | Instead, use ``int`` with defined constants. 383 | 384 | Avoid bitfields, which have compiler-/platform-dependent memory layout. 385 | Instead, use fixed width integer types and bitmask constants. 386 | 387 | To be clear: it is fine to use ``enum`` and bitfields within CPython sources, 388 | provided they do not need to be used in public headers. 389 | 390 | 391 | Objects 392 | ------- 393 | 394 | Use ``PyObject*`` for all Python objects. 395 | Avoid using concrete types (e.g. ``PyDictObject*``). 396 | 397 | Public API should type-check all objects passed to it. 398 | When it gets an object of an unexpected type, public API should fail with 399 | ``TypeError`` rather than crash. 400 | 401 | As an exception, with approval from the C API Working Group you can use concrete types, 402 | such as ``PyTypeObject*``, ``PyCodeObject*`` & ``PyFrameObject*``, 403 | for consistency with existing API. 404 | These objects should be type-checked as if they were ``PyObject*``. 405 | 406 | 407 | .. note:: 408 | 409 | For background and discussions, see: 410 | 411 | - https://github.com/capi-workgroup/api-evolution/issues/29 412 | - https://github.com/capi-workgroup/decisions/issues/19 413 | 414 | 415 | Return values 416 | ============= 417 | 418 | The return value of a function must indicate whether an exception was set. 419 | It must not be necessary to use ``PyErr_Occurred`` to disambiguate. 420 | (Recall that these guidelines apply to *new* API; existing API does not 421 | necessarily follow this.) 422 | 423 | Generally, API functions can return one of: 424 | 425 | * An integral value, where ``-1`` is returned if and only if an exception was 426 | set, and other values signal an absence of exception. 427 | * A pointer, where ``NULL`` is returned if and only if an exception was set. 428 | * A few special cases: 429 | 430 | - Functions that never return, or always set an exception, should use 431 | the :c:expr:`void` return type. 432 | - Functions used when the runtime might not be initialized 433 | should either: 434 | 435 | * return ``PyStatus``, or 436 | * return ``-1``/``NULL`` to signal failure, but have an alternate way 437 | of reporting error details. 438 | 439 | In cases where ``-1`` or ``NULL`` is a valid result, use an 440 | :ref:`output argument ` to provide that result. 441 | 442 | See `return schemes`_ for concrete examples. 443 | 444 | .. note:: 445 | 446 | For background and discussions, see: 447 | 448 | - https://github.com/capi-workgroup/api-evolution/issues/13 449 | - https://github.com/capi-workgroup/decisions/issues/19 450 | 451 | 452 | Exceptions for infallible functions 453 | ----------------------------------- 454 | 455 | Some functions cannot fail, and callers cannot check for exceptions: 456 | 457 | * Deallocators and reference sinks like ``PyMem_Free`` and ``Py_DECREF``, 458 | which use ``void`` as the return type. 459 | 460 | Other functions cannot fail, and users may *optionally* skip error 461 | checking: 462 | 463 | * Refcounting operations, like ``Py_NewRef``. 464 | * Operations on native types that cannot have exceptional cases 465 | (e.g. overflow), like ``Py_HashPointer``. 466 | * Subtype-checking functions, like ``PyTuple_Check`` and ``PyTuple_CheckExact``, 467 | which return either ``0`` or ``1``. 468 | (This does not extend to *subclass* checking, like ``PyObject_IsSubclass``, 469 | which can call Python code.) 470 | 471 | Even in these cases, ``-1`` and ``NULL`` are reserved for errors; 472 | infallible functions must never return these values. 473 | (This allows auto-generated API wrappers to avoid unnecessary special cases.) 474 | 475 | As always, other exceptions can be added here with approval from the 476 | C API working group. 477 | 478 | Mind that infallibility is very often an implementation detail 479 | that should not be exposed in the API. That is, some functions' 480 | *current CPython implementations* cannot fail, 481 | but we may want the function to fail (or warn) in the future. 482 | Users should only skip error checking if the function's documentation 483 | explicitly allows it. 484 | 485 | 486 | .. _output argument: 487 | 488 | Output arguments 489 | ================ 490 | 491 | .. (There's nothing to say for *arguments* in general, it's all under 492 | types or reference conting. 493 | If that changes, "Output arguments" should be a subsection of "Arguments") 494 | 495 | Output arguments are pointers to memory that a function fills in. 496 | Use these when a result from a function cannot be returned as the return value, 497 | for example: 498 | 499 | * ``-1`` or ``NULL`` is a valid (non-exceptional) result: for example, 500 | in ``PyDict_GetItemRef``. 501 | * There are multiple results: for example, in ``PyUnicode_AsUTF8AndSize``. 502 | 503 | Guidelines for output arguments: 504 | 505 | * Functions must always fill in the output arguments. If an error 506 | occurs or the result is not available, the output should typically be set 507 | to ``NULL`` or zero. 508 | * When it might be useful for users to call a function but ignore an output, 509 | allow passing ``NULL`` as the output argument. 510 | * Ownership of a ``PyObject*`` result is transferred to the caller, 511 | as with return values. 512 | [TODO: Link to guidelines about ownership & borrowing, when those are added] 513 | 514 | Since existing API does not necessarily follow these guidelines, 515 | all of the above points should be explicitly mentioned in the 516 | documentation of each function they're relevant to. 517 | 518 | .. note:: 519 | 520 | For background and discussions, see: 521 | 522 | - https://github.com/capi-workgroup/api-evolution/issues/32 523 | 524 | 525 | .. _shadowing example: 526 | 527 | Appendix A. Shadowing example 528 | ============================= 529 | 530 | To provide a ``static inline`` equivalent to an exported function, 531 | write something like: 532 | 533 | Header: 534 | 535 | .. code-block:: c 536 | 537 | static inline returntype 538 | _Py_Foo_impl(ARGS) 539 | { 540 | ... 541 | } 542 | 543 | PyAPI_FUNC(returntype) Py_Foo (ARGS); 544 | 545 | #define Py_Foo _Py_Foo_impl 546 | 547 | Code: 548 | 549 | .. code-block:: c 550 | 551 | // at the end (after all calls to Py_Foo): 552 | #undef Py_Foo 553 | 554 | returntype 555 | Py_Foo(ARGS) 556 | { 557 | return _Py_Foo_impl(ARGS); 558 | } 559 | 560 | 561 | 562 | .. _return schemes: 563 | 564 | Appendix B. Return value schemes 565 | ================================ 566 | 567 | Here are common schemes of how to encode return values. 568 | 569 | * Success or failure: return ``int`` 570 | 571 | * ``0`` for success 572 | * ``-1``, with an exception set, for failure 573 | 574 | * Yes or no: return ``int`` 575 | 576 | * ``1`` for ``true`` 577 | * ``0`` for ``false`` 578 | * ``-1``, with an exception set, for failure 579 | 580 | * Lookup (“getattr”, “getitem” or “setdefault” style) functions: return 581 | ``int``; the lookup result is passed via an 582 | :ref:`output argument `): 583 | 584 | * ``1`` for “found” (*result* is set) 585 | * ``0`` for “not found” 586 | (*result* is set to ``NULL`` or other zero/empty value) 587 | * ``-1``, with an exception set, for failure 588 | (*result* is set to ``NULL`` or other zero/empty value) 589 | 590 | * Enumeration-style: return ``int`` 591 | 592 | * ``-1``, with an exception set, for failure 593 | * Zero and positive numbers for valid results 594 | 595 | * Hashes: return ``Py_hash_t`` 596 | 597 | * ``-1``, with an exception set, for failure 598 | * All other numbers for result. 599 | 600 | * Objects: return ``PyObject*`` 601 | 602 | * ``NULL``, with an exception set, for failure 603 | * Valid pointer for result 604 | --------------------------------------------------------------------------------