├── .github └── ISSUE_TEMPLATE │ └── new-issue-template.md ├── DesignPrinciples.md ├── DesignRules.md ├── Interop.md ├── LICENSE ├── README.md ├── api ├── PyAPI.h └── generate │ ├── __init__.py │ ├── __main__.py │ ├── abi.py │ └── api.py ├── examples.md └── implementation.md /.github/ISSUE_TEMPLATE/new-issue-template.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: New issue template 3 | about: Template for new issues 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | Please make new issues actionable. 11 | 12 | In other words, there needs to be something that can be done to close the issue. 13 | 14 | Issues don't need to be precise, but should do-able. 15 | 16 | For example, "have you looked at Floob" is not actionable, whereas, "please add references to Floob" is actionable. 17 | 18 | Better still, open a PR with the references to Floob. 19 | -------------------------------------------------------------------------------- /DesignPrinciples.md: -------------------------------------------------------------------------------- 1 | 2 | # Design principles 3 | 4 | The API will adhere, as much as possible, to the following design principles. 5 | The priniciples are listed in order of importance, with the most important ones first. 6 | 7 | ## No invalid states 8 | 9 | No safe input[1] to the an API can create an invalid state in the virtual machine. 10 | This principle must be applied to all API functions, without exception. 11 | Examples of this are: 12 | 13 | * If a function accepts pointers, it must accept `NULL`. 14 | * If a function takes a `signed` integer as an input for a length, 15 | it must handle negative numbers. 16 | 17 | [1] A safe input is one that does not break obvious invariants. 18 | For example, if an API takes an array and a length, those values must match. 19 | Likewise, inputs must be type-safe. 20 | 21 | ## Minimize the chance of the user supplying invalid input. 22 | 23 | This requires judgement, and depends on the use case. Nevertheless it is an important principle. 24 | For example, by making `PyRef` a struct, it cannot be simply cast to a `PyTupleRef`. 25 | 26 | ## Make it difficult for users to ignore error conditions 27 | 28 | Generally this means returning error codes in a way that is difficult to ignore. 29 | 30 | ## The API should be efficient as possible 31 | 32 | We assume that users of the API are using C, or other low-level language for performance. 33 | If users avoid the C-API, delving into CPython internals, it undermines the point of having 34 | the C-API 35 | 36 | ## Completeness 37 | 38 | The C-API should be complete. There should be no need to access VM internals. 39 | This means that the C-API needs to cover all Python features, and most of the VM functionality. 40 | 41 | ## No privileged users 42 | 43 | The standard library will use the same API as third-party code, without exception. 44 | This helps to ensure completeness and encourages implementers to make the API efficient. 45 | 46 | ## Consistency 47 | 48 | The API should be consistent in naming, and usage. It should be possible to infer 49 | what an API function does from its name and argument. 50 | 51 | ## Minimum of implicit state 52 | 53 | Some implicit state is necessary, but most, if not all, should be explicit. 54 | Passing state explicitly allows control over which extension defined functions 55 | have access to what state, improving robustness. 56 | 57 | ## API and ABI equivalence 58 | 59 | Provided that the extension is built in ABI compatible mode, then all code 60 | written to the API will continue to work on future versions of Python 61 | without recompilation. Recompilation using newer versions may be more efficient, 62 | but code compiled to older versions of the API will continue to work. 63 | 64 | ## API stability 65 | 66 | Once added to the API, a feature will be removed only if there is a very 67 | strong reason to do so. 68 | 69 | The semantics and interface of a function or struct will never change. 70 | It will either remain unchanged or be removed, and possibly replaced. 71 | 72 | ## The API should be portable across Python implementations and future proof 73 | 74 | The current design of CPython is constrained by the C-API. 75 | We want to provide a faster Python for all users, and the C-API 76 | should not prevent that. The HPy project shows that this can be done efficiently. 77 | 78 | ## The API should be pure C. 79 | 80 | That means no `#ifdef __cplusplus` or similar constructs. 81 | We do this, not because we don't want people to use C++, but because 82 | we want them to be able to use Rust, or whatever other low-level 83 | language they want. A C API offers a common basis for all languages. 84 | -------------------------------------------------------------------------------- /DesignRules.md: -------------------------------------------------------------------------------- 1 | 2 | # API design rules 3 | 4 | These rules should be applied when adding to the API. 5 | They exist to provide a consistent API that adheres to the 6 | [design principles](./DesignPrinciples.md). 7 | 8 | ## All functions to take an opaque context 9 | 10 | Passing a `context` parameter to API functions forces extension 11 | code to operate on the correct context, and prevents them from 12 | performing operations that are not safe from the given context. 13 | 14 | `PyContext` is the full context passed to most extension functions 15 | and that is expected by most API functions. 16 | 17 | There are also more limited contexts that get passed to some 18 | extension functions and allow only a more limited interaction 19 | with the virtual machine. 20 | 21 | For example `PyMemContext` is passed to destructor functions 22 | which prevents them from doing much more than freeing `PyRef`s 23 | and memory. 24 | 25 | The exception to this rule is the set of funtions that get references 26 | to per-process objects like `int` or `type`. 27 | 28 | ## Opaque, linear references 29 | 30 | The C-API will refer to Python objects through opaque references 31 | which must have exactly one owner. This design has been shown to 32 | be efficient, robust and portable by the HPy project, where the 33 | references are known as "handles". 34 | As each reference has exactly one owner, there will be no 35 | incrementing or decrementing of reference counts. References can 36 | be duplicated with 37 | ```C 38 | PyRef PyRef_Dup(PyContext ctx, PyRef ref); 39 | ``` 40 | and destroyed by 41 | ```C 42 | void PyRef_Close(PyContext ctx, PyRef ref); 43 | void PyRef_Free(PyMemContext mctx, PyRef ref); 44 | ``` 45 | 46 | Type specific variants will be provided for subtypes like `PyListRef`. 47 | 48 | ## No functions that mutate immutable objects 49 | 50 | The is a consequence of the "No invalid states" design principle. 51 | The legacy API allows inplace mutation of tuples, strings and other 52 | immutable object. These will not be allowed in the new API. 53 | 54 | ## All functions must clearly show if an error has occurred. 55 | 56 | Whether an API function has produced an error must be clear 57 | from the return value, and only from the return value. 58 | 59 | API functions must obey the following rules: 60 | 61 | * If no valid result can be returned, the the result value must be `PyRef_INVALID` or `-1`. 62 | * If the result is passed though a pointer, and the an error occurs, then the result value must be untouched. 63 | * If the result is valid, then an error must not have occured 64 | * If an error has occurred, then the result of immediately calling `PyApi_GetLatestException()` must be that error. 65 | 66 | Note that if an API function does not produce a result, the result of calling `PyApi_GetLatestException()` 67 | is undefined. It will be a legal, safe to use value; it will just be meaningless. 68 | 69 | 70 | ### Functions that return references 71 | 72 | Many functions return a result, but may also raise an exception. 73 | To handle this case, a special value is used to indicate an error, 74 | `PyRef_INVALID`. If an error occurs, `PyRef_INVALID` must be returned. 75 | Conversely, if `PyRef_INVALID` is returned an error must have occurred. 76 | 77 | ### Functions that return booleans. 78 | 79 | These functions will return `int`, with -1 indicating an error. 80 | 0 and 1 indicate `False` and `True`, respectively. 81 | 82 | ### Functions that return ints 83 | 84 | Functions that only return non-negative values can use -1 to indicate an error. 85 | Functions that can return negative values must return an error code, 86 | returning the real result via a pointer. 87 | 88 | ### Functions that may fail without an error 89 | 90 | Some functions can fail without an error. Those functions must return an `int` 91 | error code and pass the result through a pointer. 92 | 93 | Success should be indicated by zero, and non-error failures by positive values. 94 | 95 | For example, to get a value from a dictionary might have the following API: 96 | 97 | ```C 98 | int PyAPi_Dict_Get(PyContext ctx, PyDictRef dict, PyRef key, PyRef *result); 99 | ``` 100 | 101 | If the result is zero then the function has succeeded. 102 | Otherwise the error and failure cases can be distinguished by whether 103 | the result is negative. 104 | 105 | ### Functions without results 106 | 107 | Some functions, e.g. `PyApi_List_Append()` do not produce a result, but can raise. 108 | Those functions should return an `int`. 109 | ```C 110 | int PyApi_List_Append(PyContext ctx, PyListRef list, PyRef item); 111 | ``` 112 | Success is indicated by returning a negative value, usually -1. 113 | 114 | ### Cleanup after an error 115 | 116 | In order to allow the cleanup of references after an error, 117 | `PyRef_Dup`, `PyRef_Close` and `PyRef_Free` will not change the 118 | result of `PyApi_GetLatestException()`. 119 | 120 | 121 | ## Naming 122 | 123 | All API function and struct names should adhere to simple rules. 124 | For example, function names should take the form: 125 | Prefix_NameSpace_Operation[_REF_CONSUMPTION][_VERSION] 126 | E.g. 127 | ```C 128 | PyTupleRef PyApi_Tuple_FromArray(PyContext ctx, uintptr_t len, PyRef *array, PyExceptionRef *error); 129 | 130 | PyTupleRef PyApi_Tuple_FromNonEmptyArray_nC_v2(PyContext ctx, uintptr_tlen, PyRef *array); 131 | ``` 132 | 133 | ## Use C99 types 134 | 135 | ### Use C99 integers types, not custom ones 136 | 137 | In other words, use `intptr_t` not `Py_ssize_t`. 138 | This helps portability, and wrapping for other languages. 139 | 140 | ### Use C99 integers types, not legacy ones 141 | 142 | Use `int32_t`, `intptr_t`, etc. Never use `long` as it differs in size even on the same hardware. 143 | The use of `int` is sometimes acceptable, when it is used as an enumeration, or small range. 144 | If the return value can represent a value, then a `` type should be used. 145 | 146 | E.g. 147 | * `int PyApi_Byte_GetItem(PyRef *array, intptr_t index, PyExceptionRef *error)` is OK. 148 | * But `int PyApi_Tuple_GetSize(PyTupleRef *ref)` is not OK as it might overflow. 149 | * It should be `uintptr_t PyApi_Tuple_GetSize(PyTupleRef *ref)` 150 | 151 | ## No variable length argument lists. 152 | 153 | In other words no `...` at the end of the parameters. 154 | These functions tend to be cumbersome and unlikely to perform as well an 155 | array and length pair of arguments. 156 | Wrapping for other languages may not always be possible. 157 | 158 | ## Consumption of argument references 159 | 160 | For effficiency, there is a natural consumption of references in some API 161 | functions. For example, appending an item to a list naturally consumes the 162 | reference to the item, but not the list. 163 | We denote borrowed references by `B` and consumed references by `C`. 164 | 165 | Consequently we want the low-level API/ABI function to be: 166 | 167 | ```C 168 | PyExceptionRef PyApi_List_Append_BC(PyContext ctx, PyListRef list, PyRef item); 169 | ``` 170 | 171 | All ABI functions should get a higher level API function without a suffix. 172 | All non-suffix functions borrow the references to all their arguments. 173 | 174 | ```C 175 | PyExceptionRef PyApi_List_Append(PyContext ctx, PyListRef list, PyRef item); 176 | ``` 177 | is equivalent to `PyApi_List_Append_BB`. 178 | 179 | Functions taking arrays must consume all the references in the array, 180 | or borrow all references in the array. 181 | 182 | The reference behavior must be the safe regardless of the return value or 183 | result. 184 | 185 | Note that this doesn't impact the portability of the API as the borrow 186 | or consume forms can be mechanically create from the other. 187 | 188 | 189 | ## ABI functions should be efficient, API functions easy to use 190 | 191 | There is a tension between ease of use and performance. 192 | For example, it is the common case when creating a tuple that 193 | the length is known, yet the function needs to treat length zero 194 | differently, returning the empty tuple singleton. 195 | 196 | We handle this tension by providing an efficient, but difficult use 197 | ABI function: 198 | ```C 199 | PyTupleRef PyApi_Tuple_FromNonEmptyArray_nC(PyContext ctx, uintptr_tlen, PyRef *array); 200 | ``` 201 | and the easier to use API function 202 | ```C 203 | PyTupleRef PyApi_Tuple_FromArray(PyContext ctx, uintptr_t len, PyRef *array); 204 | ``` 205 | 206 | However, we can make this even easier to use by making a macro that 207 | takes an array directly. 208 | ```C 209 | PyTupleRef PyApi_Tuple_FromFixedArray(ctx, array, error); 210 | ``` 211 | See the [examples](./examples.md) for the implementation. 212 | 213 | ## Return and take specific types where possible 214 | 215 | To reduce the number of dynamic type errors, we should make the API 216 | as type-specific as we can reasonably do. 217 | For example instead of specifying `PyApi_Function_GetCode()` 218 | as 219 | ```C 220 | PyRef PyApi_Function_GetCode(PyContext ctx, PyRef f, PyExceptionRef *error) 221 | ``` 222 | we should specify it as 223 | ```C 224 | PyCodeRef PyApi_Function_GetCode(PyContext ctx, PyFunctionRef f); 225 | ``` 226 | which doesn't need an `error` parameter as it cannot fail. 227 | 228 | ## Provide a safe and easy to use set of casts 229 | 230 | If we want to use specific types, we need casts. 231 | 232 | ### Upcasts 233 | 234 | Upcasts are always safe, so don't need any error handling. 235 | Upcasts will generally take the form `PyApi_TypeName_Upcast` 236 | e.g. 237 | ```C 238 | PyRef PyApi_List_Upcast(PyListRef l); 239 | ``` 240 | 241 | ### Downcasts 242 | 243 | Downcasts are tricky, because we can't return a more type specific ``PyRef`` type. 244 | Either the ``PyRef`` is unsafe, due to potential errors, or it is useless as 245 | the result of the cast, being as general as its input. 246 | 247 | * `PyListRef PyApi_List_DownCast(PyRef obj)`: unsafe 248 | * `PyRef PyApi_List_DownCast(PyRef obj)`: useless 249 | 250 | Consequently the API contains macros to wrap the test then unsafe cast idiom. 251 | 252 | For example support we want to treat an reference as a reference to a `list`. 253 | 254 | We cannot just cast it: 255 | ``` 256 | PyListRef list = PyApi_List_UnsafeCast(obj); 257 | ``` 258 | As the name suggests, this is unsafe. 259 | 260 | But, the following is safe: 261 | ``` 262 | if (PyApi_IsList(obj)) { 263 | PyListRef list = PyApi_List_UnsafeCast(obj); 264 | } 265 | ``` 266 | 267 | To discourage the use of explicit, unsafe casts the API will include "check and downcast" macros: 268 | ``` 269 | PyApi_List_CheckAndDowncast(OBJ, LIST) 270 | ``` 271 | 272 | Which would be used as follows: 273 | ``` 274 | extern void do_something_with_list(PyContext ctx, PyListRef l); 275 | 276 | void do_something_with_maybe_list(PyContext ctx, PyRef ref) 277 | { 278 | PyListRef l; 279 | if (PyApi_List_CheckAndDowncast(ref, l)) { 280 | do_something_with_list(ctx, l); 281 | } 282 | } 283 | ``` 284 | 285 | ### Organizing the API to ensure ABI compatibility. 286 | 287 | * The API will be defined by `PyAPI.h`. 288 | * `PyABI.h` will define the ABI. 289 | * `PyAPI.h` will not contain any `extern` declarations. 290 | * `PyAPI.h` may have macros and inline functions. 291 | * All `extern` declarations must be in `PyABI.h`. 292 | 293 | ### No ABI mode 294 | 295 | There is a tension between performance and portability. The performance impact of using portable 296 | function calls for low-level operations may be unacceptable for some extensions and tools. 297 | 298 | To enable these extensions and tools to use the API, albeit at some cost in portability, there will be 299 | a "No ABI" mode. 300 | 301 | The "No ABI" mode would be turned on by compiling with the `PYAPI_NO_ABI` macro, used thus: 302 | ``` 303 | #define PYAPI_NO_ABI 1 304 | #include "PyAPI.h" 305 | ``` 306 | 307 | The "No ABI" mode would expose more efficient implementations of API functions, but would not be portable 308 | across different versions, or different implementations. 309 | 310 | For example, many of the class checks and casts can implemented in a few instructions as an inline function 311 | or macro, but such an implementation ties a build of code using it to a single Python implementation. 312 | 313 | Internally, the `PyABI.h` would conceptually be something like: 314 | ```C 315 | #ifdef PYAPI_NO_ABI 316 | #include "PyApi_unstable.h" 317 | #else 318 | #include "PyAbi_portable.h" 319 | #endif 320 | #include "PyAbi_common.h" 321 | ``` 322 | -------------------------------------------------------------------------------- /Interop.md: -------------------------------------------------------------------------------- 1 | # Interoperability API 2 | 3 | The interopability API consists of functions for converting 4 | `PyObject *` to `PyRef`, and vice versa. 5 | These functions are for internal use, so do not take a `PyContext`. 6 | 7 | ## Converting `PyObject *` to `PyRef` 8 | 9 | In general, converting a `PyObject *` to a `PyRef` is unsafe as it loses 10 | information about exceptions. Therefore conversion functions need to 11 | return a `kind` as well as a `PyRef`. 12 | 13 | ```C 14 | PyRef PyApi_Interop_FromObjectUnsafe_C(PyObject *obj); 15 | ``` 16 | 17 | The safe variant, which bundles the current exception and does consistency checks: 18 | ```C 19 | int PyApi_Interop_FromObject_C(PyObject *obj, PyRef *result); 20 | ``` 21 | 22 | ## Converting `PyRef` to `PyObject *` 23 | 24 | This function consumes the argument. 25 | 26 | ```C 27 | PyObject *PyApi_Interop_ToObject_C(PyRef ref); 28 | ``` 29 | 30 | The `PyApi_Interop_ToObject_C` never returns `NULL`, nor sets an exception. 31 | 32 | ## Implementation of the error handling forms 33 | 34 | Athough technically an implementation detail, it helps understanding 35 | and correct use if the implemenation of the error handling forms are shown. 36 | 37 | ```C 38 | 39 | PyRef PyApi_Interop_FromObjectUnsafe_C(PyObject *obj) 40 | { 41 | assert(obj != NULL); 42 | assert(!PyErr_Occurred()); 43 | return (PyRef) { obj }; 44 | } 45 | 46 | PyRef PyApi_Interop_FromObject_C(PyObject *obj, PyExceptionRef *exc) 47 | { 48 | if (PyErr_Occurred()) { 49 | PyObject *exception = get_normalized_exception(); 50 | if (object != NULL) { 51 | // Both object and exception 52 | PyObject *new_exception = _PyErr_CreateException( 53 | PyExc_SystemError, 54 | "value is not-NULL, but an exception is set" 55 | ); 56 | PyException_SetCause(new_exception, exception); 57 | Py_DECREF(exception); 58 | Py_DECREF(obj); 59 | exception = new_exception; 60 | } 61 | *exc = PyApi_Interop_FromException(exception); 62 | return PyRef_INVALID; 63 | } 64 | else { 65 | if (obj == NULL) { 66 | // Neither result nor exception 67 | PyObject *exception = _PyErr_CreateException( 68 | PyExc_SystemError, 69 | "value is NULL without setting an exception" 70 | ); 71 | *exc = PyApi_Interop_FromException(exception); 72 | return PyRef_INVALID; 73 | } 74 | } 75 | return PyApi_Interop_FromObjectUnsafe(obj); 76 | } 77 | ``` 78 | 79 | ```C 80 | PyObject *PyApi_Interop_ToObject_C(PyRef ref) 81 | { 82 | return ref.value; 83 | } 84 | ``` 85 | 86 | 87 | 88 | 89 | 90 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Creative Commons Legal Code 2 | 3 | CC0 1.0 Universal 4 | 5 | CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE 6 | LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN 7 | ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS 8 | INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES 9 | REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS 10 | PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM 11 | THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED 12 | HEREUNDER. 13 | 14 | Statement of Purpose 15 | 16 | The laws of most jurisdictions throughout the world automatically confer 17 | exclusive Copyright and Related Rights (defined below) upon the creator 18 | and subsequent owner(s) (each and all, an "owner") of an original work of 19 | authorship and/or a database (each, a "Work"). 20 | 21 | Certain owners wish to permanently relinquish those rights to a Work for 22 | the purpose of contributing to a commons of creative, cultural and 23 | scientific works ("Commons") that the public can reliably and without fear 24 | of later claims of infringement build upon, modify, incorporate in other 25 | works, reuse and redistribute as freely as possible in any form whatsoever 26 | and for any purposes, including without limitation commercial purposes. 27 | These owners may contribute to the Commons to promote the ideal of a free 28 | culture and the further production of creative, cultural and scientific 29 | works, or to gain reputation or greater distribution for their Work in 30 | part through the use and efforts of others. 31 | 32 | For these and/or other purposes and motivations, and without any 33 | expectation of additional consideration or compensation, the person 34 | associating CC0 with a Work (the "Affirmer"), to the extent that he or she 35 | is an owner of Copyright and Related Rights in the Work, voluntarily 36 | elects to apply CC0 to the Work and publicly distribute the Work under its 37 | terms, with knowledge of his or her Copyright and Related Rights in the 38 | Work and the meaning and intended legal effect of CC0 on those rights. 39 | 40 | 1. Copyright and Related Rights. A Work made available under CC0 may be 41 | protected by copyright and related or neighboring rights ("Copyright and 42 | Related Rights"). Copyright and Related Rights include, but are not 43 | limited to, the following: 44 | 45 | i. the right to reproduce, adapt, distribute, perform, display, 46 | communicate, and translate a Work; 47 | ii. moral rights retained by the original author(s) and/or performer(s); 48 | iii. publicity and privacy rights pertaining to a person's image or 49 | likeness depicted in a Work; 50 | iv. rights protecting against unfair competition in regards to a Work, 51 | subject to the limitations in paragraph 4(a), below; 52 | v. rights protecting the extraction, dissemination, use and reuse of data 53 | in a Work; 54 | vi. database rights (such as those arising under Directive 96/9/EC of the 55 | European Parliament and of the Council of 11 March 1996 on the legal 56 | protection of databases, and under any national implementation 57 | thereof, including any amended or successor version of such 58 | directive); and 59 | vii. other similar, equivalent or corresponding rights throughout the 60 | world based on applicable law or treaty, and any national 61 | implementations thereof. 62 | 63 | 2. Waiver. To the greatest extent permitted by, but not in contravention 64 | of, applicable law, Affirmer hereby overtly, fully, permanently, 65 | irrevocably and unconditionally waives, abandons, and surrenders all of 66 | Affirmer's Copyright and Related Rights and associated claims and causes 67 | of action, whether now known or unknown (including existing as well as 68 | future claims and causes of action), in the Work (i) in all territories 69 | worldwide, (ii) for the maximum duration provided by applicable law or 70 | treaty (including future time extensions), (iii) in any current or future 71 | medium and for any number of copies, and (iv) for any purpose whatsoever, 72 | including without limitation commercial, advertising or promotional 73 | purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each 74 | member of the public at large and to the detriment of Affirmer's heirs and 75 | successors, fully intending that such Waiver shall not be subject to 76 | revocation, rescission, cancellation, termination, or any other legal or 77 | equitable action to disrupt the quiet enjoyment of the Work by the public 78 | as contemplated by Affirmer's express Statement of Purpose. 79 | 80 | 3. Public License Fallback. Should any part of the Waiver for any reason 81 | be judged legally invalid or ineffective under applicable law, then the 82 | Waiver shall be preserved to the maximum extent permitted taking into 83 | account Affirmer's express Statement of Purpose. In addition, to the 84 | extent the Waiver is so judged Affirmer hereby grants to each affected 85 | person a royalty-free, non transferable, non sublicensable, non exclusive, 86 | irrevocable and unconditional license to exercise Affirmer's Copyright and 87 | Related Rights in the Work (i) in all territories worldwide, (ii) for the 88 | maximum duration provided by applicable law or treaty (including future 89 | time extensions), (iii) in any current or future medium and for any number 90 | of copies, and (iv) for any purpose whatsoever, including without 91 | limitation commercial, advertising or promotional purposes (the 92 | "License"). The License shall be deemed effective as of the date CC0 was 93 | applied by Affirmer to the Work. Should any part of the License for any 94 | reason be judged legally invalid or ineffective under applicable law, such 95 | partial invalidity or ineffectiveness shall not invalidate the remainder 96 | of the License, and in such case Affirmer hereby affirms that he or she 97 | will not (i) exercise any of his or her remaining Copyright and Related 98 | Rights in the Work or (ii) assert any associated claims and causes of 99 | action with respect to the Work, in either case contrary to Affirmer's 100 | express Statement of Purpose. 101 | 102 | 4. Limitations and Disclaimers. 103 | 104 | a. No trademark or patent rights held by Affirmer are waived, abandoned, 105 | surrendered, licensed or otherwise affected by this document. 106 | b. Affirmer offers the Work as-is and makes no representations or 107 | warranties of any kind concerning the Work, express, implied, 108 | statutory or otherwise, including without limitation warranties of 109 | title, merchantability, fitness for a particular purpose, non 110 | infringement, or the absence of latent or other defects, accuracy, or 111 | the present or absence of errors, whether or not discoverable, all to 112 | the greatest extent permissible under applicable law. 113 | c. Affirmer disclaims responsibility for clearing rights of other persons 114 | that may apply to the Work or any use thereof, including without 115 | limitation any person's Copyright and Related Rights in the Work. 116 | Further, Affirmer disclaims responsibility for obtaining any necessary 117 | consents, permissions or other rights required for any use of the 118 | Work. 119 | d. Affirmer understands and acknowledges that Creative Commons is not a 120 | party to this document and has no duty or obligation with respect to 121 | this CC0 or use of the Work. 122 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # New C-API for Python 2 | 3 | This repository is for developing the new C-API for Python. 4 | It will contain no implementation, but will serve as a repository 5 | of design principles, rules, examples and other such bits and pieces. 6 | it will also serve as a record of the discussions involved. 7 | 8 | ## Why do we need a new C API? 9 | 10 | The current C API has expanded over decades, in an unplanned way. 11 | Pieces get added without consideration of future maintainence, 12 | and in a way that exposes implementation details. 13 | 14 | This makes it hard to use, restrictive of how it can be implemented, 15 | and in many cases unsafe. 16 | 17 | We cannot incrementally change the current API, as each change 18 | will cause breakage and backwards incompatibility issues, leaving 19 | extension modules constantly broken. 20 | 21 | Rather than change the existing C API it is better to create an entirely new API, 22 | and deprecate the old one. This allows extension authors to switch to the new API 23 | at a time of their choosing, rather than constantly scrambling to keep up. 24 | 25 | ## Design principles 26 | 27 | The new C-API will follow a dozen or so [design principles](./DesignPrinciples.md). 28 | These are the high-level concepts that determine the nature of the API. 29 | 30 | ## Design rules 31 | 32 | It would be laborious and error prone to consider and apply the design principles 33 | each time a new function, struct or type were added to the API. 34 | To enable developers to follow the design principles when extending the API, 35 | there are a number of [design rules](./DesignRules.md) for them to follow. 36 | The intention is that by following the rules, the design principles should be observed. 37 | Some judgement will still be required, though. 38 | 39 | ## Extent of the API 40 | 41 | There about 1500 functions marked as `Py_API_FUNC` and about 200 objects 42 | marked in as `Py_ABI_DATA` in the current API. 43 | 44 | Many of these are artifacts of the implementation, but we would still expect 45 | the API to contain many hundreds of functions. 46 | 47 | ## Organization of the API 48 | 49 | The API should be organized in categories, reflected in the namespaces of the API. 50 | We want to break the API into namepsaces, so that it can be discoverable. 51 | 52 | We anticipate that many core objects will get their own namespace, for example: 53 | 54 | * Object 55 | * Tuple 56 | * List 57 | * Str 58 | * Bytes 59 | 60 | Various aspects of the runtime should also get their own namespace: 61 | 62 | * Interpreter 63 | * GC 64 | * FrameStack 65 | * Operations (numerical, logical, etc.) 66 | 67 | ## Interoperation with the legacy API 68 | 69 | We do not expect packages to be ported to the new API in one go. 70 | To help porting we will provide an [interopability API](./Interop.md) 71 | 72 | ## Implementation schedule and removal of the legacy C API 73 | 74 | This is very much provisional at this point. 75 | 76 | * 2024 (3.13): New C API specification complete. Deprecation of unsafe parts of legacy C API. 77 | * 2025 (3.14): New C API implementation in CPython complete. 78 | * 2026 (3.15): Removal of unsafe parts of legacy C API. Deprecation of legacy C API. 79 | * 2027 (3.16): Add warnings of parts of legacy C API with negative impact on performance. 80 | * 2028 (3.17): Add warnings on any use of legacy C API 81 | * 2029 (3.18): Remove parts of legacy C API with negative impact on performance. 82 | * 2030 (3.19) 83 | * 2031 (3.20) 84 | * 2032 (3.21/4.0): Removal of legacy C API 85 | 86 | The removal of the legacy C API will happen in three stages: 87 | 88 | * 2026 (3.15): Removal of the unsafe parts of the API: parts of the API that return borrowed references, or mutate immutable objects. 89 | * 2029 (3.18): Removal of performance limiting parts of the API. For example, parts of the API that prevent improvements to internal data structures. 90 | * 2032 (3.21/4.0): Removal of the rest of the legacy C API. 91 | 92 | 93 | ## Documentation 94 | 95 | ### API documentation 96 | 97 | Each function and struct will be fully documented. 98 | 99 | If it isn't documented it isn't part of the API. 100 | If it is part of the API it will be documented. 101 | 102 | ### Porting documentation 103 | 104 | For functions and structs in the legacy API, we will add documentation describing how code 105 | should be ported to the new API. 106 | 107 | ## Examples 108 | 109 | Here are some [example API functions with implementations](./examples.md) 110 | 111 | ## Implementation in CPython 112 | 113 | Porting CPython to the new C-API will be a lot of work, but there is a [plan](./implementation.md) 114 | -------------------------------------------------------------------------------- /api/PyAPI.h: -------------------------------------------------------------------------------- 1 | /* This file is generated from api.py by gen/__main__.py */ 2 | 3 | PyRef PyRef_INVALID; 4 | PyExceptionRef PyRef_NO_EXCEPTION; 5 | 6 | PyRef PyRef_Dup(PyRef ref); 7 | PyRef PyRef_Close(PyRef ref); 8 | 9 | /* Tuple */ 10 | PyTupleRef PyApi_Tuple_Empty(PyContext ctx); 11 | 12 | PyTupleRef PyApi_Tuple_FromNonEmptyArray(PyContext ctx, PyRef array[], uintptr_t length); 13 | 14 | PyTupleRef PyApi_Tuple_FromArray(PyContext ctx, PyRef array[], uintptr_t length); 15 | 16 | PyRef PyApi_Tuple_GetItem(PyContext ctx, PyTupleRef self, uintptr_t index); 17 | 18 | uintptr_t PyApi_Tuple_GetSize(PyContext ctx, PyTupleRef self); 19 | 20 | bool PyApi_IsATuple(PyRef ref); 21 | PyTupleRef PyApi_Tuple_UnsafeCast(PyRef ref); 22 | PyTupleRef PyApi_Tuple_DownCast(PyRef ref); 23 | PyRef PyApi_Tuple_UpCast(PyTupleRef ref); 24 | 25 | typedef PyRef (*PyApi_BinaryOperator_FuncPtr)(PyContext ctx, PyRef left, PyRef right); 26 | 27 | typedef PyRef (*PyApi_VectorCall_FuncPtr)(PyContext ctx, PyRef callable, PyRef args[], intptr_t nargsf, PyTupleRef kwnames); 28 | 29 | 30 | /* Str */ 31 | PyStrRef PyApi_Str_FromUtfString(PyContext ctx, const UtfString data, uintptr_t length); 32 | 33 | PyStrRef PyApi_Str_Join(PyContext ctx, PyStrRef self, PyStrRef array[]); 34 | 35 | PyRef PyApi_Str_GetItem(PyContext ctx, PyStrRef self, uintptr_t index); 36 | 37 | uintptr_t PyApi_Str_GetSize(PyContext ctx, PyStrRef self); 38 | 39 | bool PyApi_IsAStr(PyRef ref); 40 | PyStrRef PyApi_Str_UnsafeCast(PyRef ref); 41 | PyStrRef PyApi_Str_DownCast(PyRef ref); 42 | PyRef PyApi_Str_UpCast(PyStrRef ref); 43 | 44 | 45 | /* Class */ 46 | int PyApi_Class_AddBinaryOperator(PyContext ctx, uint8_t op, PyApi_BinaryOperator_FuncPtr func); 47 | 48 | int PyApi_Class_AddVectorCallMethod(PyContext ctx, PyStrRef name, PyApi_VectorCall_FuncPtr func); 49 | 50 | PyRef PyApi_Class_New(PyContext ctx, PyClassRef self); 51 | 52 | bool PyApi_IsAClass(PyRef ref); 53 | PyClassRef PyApi_Class_UnsafeCast(PyRef ref); 54 | PyClassRef PyApi_Class_DownCast(PyRef ref); 55 | PyRef PyApi_Class_UpCast(PyClassRef ref); 56 | 57 | bool PyApi_IsNone(PyContext ctx, PyRef obj); 58 | 59 | PyRef PyApi_None(); 60 | 61 | bool PyApi_IsTrue(PyContext ctx, PyRef obj); 62 | 63 | PyRef PyApi_True(); 64 | 65 | bool PyApi_IsFalse(PyContext ctx, PyRef obj); 66 | 67 | PyRef PyApi_False(); 68 | 69 | PyClassRef PyApi_TimeoutError(); 70 | 71 | PyClassRef PyApi_bool(); 72 | 73 | PyClassRef PyApi_memoryview(); 74 | 75 | PyClassRef PyApi_bytearray(); 76 | 77 | PyClassRef PyApi_bytes(); 78 | 79 | PyClassRef PyApi_classmethod(); 80 | 81 | PyClassRef PyApi_complex(); 82 | 83 | PyClassRef PyApi_dict(); 84 | 85 | PyClassRef PyApi_enumerate(); 86 | 87 | PyClassRef PyApi_filter(); 88 | 89 | PyClassRef PyApi_float(); 90 | 91 | PyClassRef PyApi_frozenset(); 92 | 93 | PyClassRef PyApi_property(); 94 | 95 | PyClassRef PyApi_int(); 96 | 97 | PyClassRef PyApi_list(); 98 | 99 | PyClassRef PyApi_map(); 100 | 101 | PyClassRef PyApi_object(); 102 | 103 | PyClassRef PyApi_range(); 104 | 105 | PyClassRef PyApi_reversed(); 106 | 107 | PyClassRef PyApi_set(); 108 | 109 | PyClassRef PyApi_slice(); 110 | 111 | PyClassRef PyApi_staticmethod(); 112 | 113 | PyClassRef PyApi_str(); 114 | 115 | PyClassRef PyApi_super(); 116 | 117 | PyClassRef PyApi_tuple(); 118 | 119 | PyClassRef PyApi_type(); 120 | 121 | PyClassRef PyApi_zip(); 122 | 123 | PyClassRef PyApi_BaseException(); 124 | 125 | PyClassRef PyApi_Exception(); 126 | 127 | PyClassRef PyApi_TypeError(); 128 | 129 | PyClassRef PyApi_StopAsyncIteration(); 130 | 131 | PyClassRef PyApi_StopIteration(); 132 | 133 | PyClassRef PyApi_GeneratorExit(); 134 | 135 | PyClassRef PyApi_SystemExit(); 136 | 137 | PyClassRef PyApi_KeyboardInterrupt(); 138 | 139 | PyClassRef PyApi_ImportError(); 140 | 141 | PyClassRef PyApi_ModuleNotFoundError(); 142 | 143 | PyClassRef PyApi_OSError(); 144 | 145 | PyClassRef PyApi_EnvironmentError(); 146 | 147 | PyClassRef PyApi_IOError(); 148 | 149 | PyClassRef PyApi_EOFError(); 150 | 151 | PyClassRef PyApi_RuntimeError(); 152 | 153 | PyClassRef PyApi_RecursionError(); 154 | 155 | PyClassRef PyApi_NotImplementedError(); 156 | 157 | PyClassRef PyApi_NameError(); 158 | 159 | PyClassRef PyApi_UnboundLocalError(); 160 | 161 | PyClassRef PyApi_AttributeError(); 162 | 163 | PyClassRef PyApi_SyntaxError(); 164 | 165 | PyClassRef PyApi_IndentationError(); 166 | 167 | PyClassRef PyApi_TabError(); 168 | 169 | PyClassRef PyApi_LookupError(); 170 | 171 | PyClassRef PyApi_IndexError(); 172 | 173 | PyClassRef PyApi_KeyError(); 174 | 175 | PyClassRef PyApi_ValueError(); 176 | 177 | PyClassRef PyApi_UnicodeError(); 178 | 179 | PyClassRef PyApi_UnicodeEncodeError(); 180 | 181 | PyClassRef PyApi_UnicodeDecodeError(); 182 | 183 | PyClassRef PyApi_UnicodeTranslateError(); 184 | 185 | PyClassRef PyApi_AssertionError(); 186 | 187 | PyClassRef PyApi_ArithmeticError(); 188 | 189 | PyClassRef PyApi_FloatingPointError(); 190 | 191 | PyClassRef PyApi_OverflowError(); 192 | 193 | PyClassRef PyApi_ZeroDivisionError(); 194 | 195 | PyClassRef PyApi_SystemError(); 196 | 197 | PyClassRef PyApi_ReferenceError(); 198 | 199 | PyClassRef PyApi_MemoryError(); 200 | 201 | PyClassRef PyApi_BufferError(); 202 | 203 | PyClassRef PyApi_Warning(); 204 | 205 | PyClassRef PyApi_UserWarning(); 206 | 207 | PyClassRef PyApi_EncodingWarning(); 208 | 209 | PyClassRef PyApi_DeprecationWarning(); 210 | 211 | PyClassRef PyApi_PendingDeprecationWarning(); 212 | 213 | PyClassRef PyApi_SyntaxWarning(); 214 | 215 | PyClassRef PyApi_RuntimeWarning(); 216 | 217 | PyClassRef PyApi_FutureWarning(); 218 | 219 | PyClassRef PyApi_ImportWarning(); 220 | 221 | PyClassRef PyApi_UnicodeWarning(); 222 | 223 | PyClassRef PyApi_BytesWarning(); 224 | 225 | PyClassRef PyApi_ResourceWarning(); 226 | 227 | PyClassRef PyApi_ConnectionError(); 228 | 229 | PyClassRef PyApi_BlockingIOError(); 230 | 231 | PyClassRef PyApi_BrokenPipeError(); 232 | 233 | PyClassRef PyApi_ChildProcessError(); 234 | 235 | PyClassRef PyApi_ConnectionAbortedError(); 236 | 237 | PyClassRef PyApi_ConnectionRefusedError(); 238 | 239 | PyClassRef PyApi_ConnectionResetError(); 240 | 241 | PyClassRef PyApi_FileExistsError(); 242 | 243 | PyClassRef PyApi_FileNotFoundError(); 244 | 245 | PyClassRef PyApi_IsADirectoryError(); 246 | 247 | PyClassRef PyApi_NotADirectoryError(); 248 | 249 | PyClassRef PyApi_InterruptedError(); 250 | 251 | PyClassRef PyApi_PermissionError(); 252 | 253 | PyClassRef PyApi_ProcessLookupError(); 254 | 255 | PyClassRef PyApi_TimeoutError(); 256 | 257 | 258 | /* Bytes */ 259 | uint8_t PyApi_Bytes_GetItem(PyContext ctx, PyBytesRef self, uintptr_t index); 260 | 261 | PyBytesRef PyApi_Bytes_FromArray(PyContext ctx, const char * data, uintptr_t length); 262 | 263 | uintptr_t PyApi_Bytes_GetSize(PyContext ctx, PyBytesRef self); 264 | 265 | bool PyApi_IsABytes(PyRef ref); 266 | PyBytesRef PyApi_Bytes_UnsafeCast(PyRef ref); 267 | PyBytesRef PyApi_Bytes_DownCast(PyRef ref); 268 | PyRef PyApi_Bytes_UpCast(PyBytesRef ref); 269 | 270 | 271 | /* StrBuilder */ 272 | PyStrBuilderRef PyApi_StrBuilder_New(PyContext ctx, uintptr_t capacity); 273 | 274 | int PyApi_StrBuilder_AppendStr(PyContext ctx, PyRef Self, PyStrRef s); 275 | 276 | int PyApi_StrBuilder_AppendUtf8String(PyContext ctx, PyRef Self, const UtfString s); 277 | 278 | PyStrRef PyApi_StrBuilder_ToStr(PyContext ctx, PyRef Self); 279 | 280 | bool PyApi_IsAStrBuilder(PyRef ref); 281 | PyStrBuilderRef PyApi_StrBuilder_UnsafeCast(PyRef ref); 282 | PyStrBuilderRef PyApi_StrBuilder_DownCast(PyRef ref); 283 | PyRef PyApi_StrBuilder_UpCast(PyStrBuilderRef ref); 284 | 285 | 286 | /* TupleBuilder */ 287 | PyTupleBuilderRef PyApi_TupleBuilder_New(PyContext ctx, uintptr_t capacity); 288 | 289 | int PyApi_TupleBuilder_Add(PyContext ctx, PyRef Self, PyStrRef s); 290 | 291 | PyStrRef PyApi_TupleBuilder_ToTuple(PyContext ctx, PyRef Self); 292 | 293 | bool PyApi_IsATupleBuilder(PyRef ref); 294 | PyTupleBuilderRef PyApi_TupleBuilder_UnsafeCast(PyRef ref); 295 | PyTupleBuilderRef PyApi_TupleBuilder_DownCast(PyRef ref); 296 | PyRef PyApi_TupleBuilder_UpCast(PyTupleBuilderRef ref); 297 | 298 | 299 | /* Object */ 300 | PyRef PyApi_Object_GetItem(PyContext ctx, PyRef obj, PyRef key); 301 | 302 | PyRef PyApi_Object_GetItem_i(PyContext ctx, PyRef obj, intptr_t key); 303 | 304 | PyRef PyApi_Object_GetItem_s(PyContext ctx, PyRef obj, const UtfString key); 305 | 306 | int PyApi_Object_SetItem(PyContext ctx, PyRef obj, PyRef key, PyRef value); 307 | 308 | int PyApi_Object_SetItem_i(PyContext ctx, PyRef obj, intptr_t key, PyRef value); 309 | 310 | int PyApi_Object_SetItem_s(PyContext ctx, PyRef obj, const UtfString key, PyRef value); 311 | 312 | PyRef PyApi_Object_GetAttr(PyContext ctx, PyRef obj, PyRef attr); 313 | 314 | PyRef PyApi_Object_GetAttr_s(PyContext ctx, PyRef obj, const UtfString attr); 315 | 316 | int PyApi_Object_HasAttr(PyContext ctx, PyRef obj, PyRef attr); 317 | 318 | int PyApi_Object_HasAttr_s(PyContext ctx, PyRef obj, const UtfString attr); 319 | 320 | int PyApi_Object_SetAttr(PyContext ctx, PyRef obj, PyRef attr, PyRef value); 321 | 322 | int PyApi_Object_SetAttr_s(PyContext ctx, PyRef obj, const UtfString attr, PyRef value); 323 | 324 | int PyApi_Object_Contains(PyContext ctx, PyRef container, PyRef key); 325 | 326 | PyClassRef PyApi_Object_Type(PyContext ctx, PyRef obj); 327 | 328 | bool PyApi_Object_TypeCheck(PyContext ctx, PyRef obj, PyClassRef cls); 329 | 330 | PyStrRef PyApi_Object_Repr(PyContext ctx, PyRef obj); 331 | 332 | PyStrRef PyApi_Object_Str(PyContext ctx, PyRef obj); 333 | 334 | intptr_t PyApi_Object_Hash(PyContext ctx, PyRef obj); 335 | 336 | PyRef PyApi_Object_CallMethod(PyContext ctx, PyRef attr, PyRef args[], intptr_t nargsf); 337 | 338 | int PyApi_Object_Compare(PyContext ctx, uint8_t op, PyRef left, PyRef right); 339 | 340 | int PyApi_Object_IsIter(PyContext ctx, PyRef obj); 341 | 342 | int PyApi_Object_IsAnIter(PyContext ctx, PyRef obj); 343 | 344 | 345 | /* Dict */ 346 | PyRef PyApi_Dict_Get(PyContext ctx, PyDictRef self, PyRef key, PyRef default); 347 | 348 | PyRef PyApi_Dict_GetItem(PyContext ctx, PyDictRef self, PyRef key); 349 | 350 | PyDictRef PyApi_Dict_New(PyContext ctx); 351 | 352 | bool PyApi_IsADict(PyRef ref); 353 | PyDictRef PyApi_Dict_UnsafeCast(PyRef ref); 354 | PyDictRef PyApi_Dict_DownCast(PyRef ref); 355 | PyRef PyApi_Dict_UpCast(PyDictRef ref); 356 | 357 | 358 | /* Int */ 359 | PyIntRef PyApi_Int_FromInt32(PyContext ctx, int32_t val); 360 | 361 | PyIntRef PyApi_Int_FromUInt32(PyContext ctx, int32_t val); 362 | 363 | PyIntRef PyApi_Int_FromInt64(PyContext ctx, int32_t val); 364 | 365 | PyIntRef PyApi_Int_FromUInt64(PyContext ctx, int32_t val); 366 | 367 | int32_t PyApi_Int_ToInt32(PyContext ctx, PyIntRef self); 368 | 369 | int64_t PyApi_Int_ToInt64(PyContext ctx, PyIntRef self); 370 | 371 | bool PyApi_IsAnInt(PyRef ref); 372 | PyIntRef PyApi_Int_UnsafeCast(PyRef ref); 373 | PyIntRef PyApi_Int_DownCast(PyRef ref); 374 | PyRef PyApi_Int_UpCast(PyIntRef ref); 375 | 376 | 377 | /* List */ 378 | PyListRef PyApi_List_New(PyContext ctx); 379 | 380 | int PyApi_List_Append(PyContext ctx, PyListRef self, PyRef item); 381 | 382 | PyRef PyApi_List_GetItem(PyContext ctx, PyListRef self, uintptr_t index); 383 | 384 | uintptr_t PyApi_List_GetSize(PyContext ctx, PyListRef self); 385 | 386 | PyRef PyApi_List_Pop(PyContext ctx, PyListRef self); 387 | 388 | bool PyApi_IsAList(PyRef ref); 389 | PyListRef PyApi_List_UnsafeCast(PyRef ref); 390 | PyListRef PyApi_List_DownCast(PyRef ref); 391 | PyRef PyApi_List_UpCast(PyListRef ref); 392 | 393 | 394 | /* Operators */ 395 | PyRef PyApi_Operators_UnaryOp(PyContext ctx, uint8_t op, PyRef argument); 396 | 397 | PyRef PyApi_Operators_BinaryOp(PyContext ctx, uint8_t op, PyRef left, PyRef right); 398 | 399 | PyRef PyApi_Operators_Compare(PyContext ctx, PyRef left, PyRef right, uint8_t op); 400 | 401 | int PyApi_Operators_CompareBool(PyContext ctx, PyRef left, PyRef right, uint8_t op); 402 | 403 | 404 | /* Exception */ 405 | PyExceptionRef PyApi_Exception_FromString(PyContext ctx, PyClassRef cls, const UtfString message); 406 | 407 | PyExceptionRef PyApi_Exception_FromValue(PyContext ctx, PyClassRef cls, PyRef message); 408 | 409 | void PyApi_Exception_Fatal(PyContext ctx, const UtfString message); 410 | 411 | PyExceptionRef PyApi_Exception_FromErrnoWithFilename(PyContext ctx, PyClassRef cls, const UtfString filename); 412 | 413 | /* Always fails. Barring another error, 414 | will set *error to the newly created exception. 415 | */ 416 | PyExceptionRef PyApi_Exception_RaiseFromString(PyContext ctx, PyClassRef cls, const UtfString message); 417 | 418 | /* Always fails. Barring another error, 419 | will set *error to the newly created exception. 420 | */ 421 | PyExceptionRef PyApi_Exception_RaiseFromValue(PyContext ctx, PyClassRef cls, PyRef message); 422 | 423 | bool PyApi_IsAnException(PyRef ref); 424 | PyExceptionRef PyApi_Exception_UnsafeCast(PyRef ref); 425 | PyExceptionRef PyApi_Exception_DownCast(PyRef ref); 426 | PyRef PyApi_Exception_UpCast(PyExceptionRef ref); 427 | 428 | /* If the immediately previous API call failed, 429 | then return the exception set by that exception. 430 | Otherwise, return either an arbitrary exception 431 | or PyRef_NO_EXCEPTION */ 432 | PyExceptionRef PyApi_GetLatestException(PyContext ctx); 433 | 434 | 435 | /* Call */ 436 | int PyApi_Call_IsCallable(PyContext ctx, PyRef obj); 437 | 438 | PyRef PyApi_Call_TupleDict(PyContext ctx, PyRef callable, PyTupleRef args, PyDictRef kwargs); 439 | 440 | PyRef PyApi_Call_Vector(PyContext ctx, PyRef callable, PyRef args[], intptr_t nargsf, PyTupleRef kwnames); 441 | 442 | 443 | /* Iter */ 444 | PyRef PyApi_Iter_Next(PyContext ctx, PyRef obj); 445 | 446 | /* If no more results are available then, instead of raising StopIteration, 447 | *error = PyRef_NO_EXCEPTION. 448 | This is more efficient than the plain Next function. */ 449 | PyRef PyApi_Iter_NextX(PyContext ctx, PyRef obj); 450 | 451 | PyRef PyApi_Iter_Send(PyContext ctx, PyRef obj); 452 | 453 | /* If no exception is raised, then returned is set to 0 for 454 | a yield, and 1 for a return. 455 | This is more efficient than the plain Send function. */ 456 | PyRef PyApi_Iter_SendX(PyContext ctx, PyRef obj, int * returned); 457 | 458 | 459 | /* Code */ 460 | bool PyApi_IsACode(PyRef ref); 461 | PyCodeRef PyApi_Code_UnsafeCast(PyRef ref); 462 | PyCodeRef PyApi_Code_DownCast(PyRef ref); 463 | PyRef PyApi_Code_UpCast(PyCodeRef ref); 464 | 465 | 466 | /* FrameStack */ 467 | PyRef PyApi_FrameStack_GetLocal(PyContext ctx, uintptr_t depth, uintptr_t index); 468 | 469 | PyRef PyApi_FrameStack_GetLocalByName(PyContext ctx, uintptr_t depth, PyStrRef name); 470 | 471 | PyRef PyApi_FrameStack_GetLocalByCName(PyContext ctx, uintptr_t depth, const UtfString name); 472 | 473 | PyCodeRef PyApi_FrameStack_GetCode(PyContext ctx, uintptr_t depth); 474 | 475 | -------------------------------------------------------------------------------- /api/generate/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/markshannon/New-C-API-for-Python/73b01ab9b96550a069813ae733c43606d3d64492/api/generate/__init__.py -------------------------------------------------------------------------------- /api/generate/__main__.py: -------------------------------------------------------------------------------- 1 | 2 | from __future__ import annotations 3 | from types import FunctionType 4 | from .abi import is_namespace, Int, Self, Void, Size, uint8_t, no_result, cannot_fail, is_function_pointer, is_shared 5 | 6 | from . import api 7 | 8 | CONTEXT = "PyContext ctx" 9 | 10 | def func_ptr_name(func): 11 | return f"PyApi_{func.__name__}_FuncPtr" 12 | 13 | def ctype_for_class(cls, namespace): 14 | if cls is object: 15 | return "PyRef" 16 | if isinstance(cls, FunctionType): 17 | return func_ptr_name(cls) 18 | if isinstance(cls, str): 19 | return cls 20 | if cls is Self: 21 | return f"Py{namespace}Ref" 22 | if cls is bool: 23 | return "bool" 24 | if cls is int: 25 | return "int" 26 | if cls is Void: 27 | return "void" 28 | if isinstance(cls, Int): 29 | return cls.name 30 | return f"Py{cls.__name__}Ref" 31 | 32 | def get_arg_decl(name, namespace, annotations): 33 | if name == "self": 34 | return [f"Py{namespace}Ref", "self"] 35 | cls = annotations.get(name, object) 36 | if isinstance(cls, list): 37 | return [f"{ctype_for_class(cls[0], namespace)}", f"{name}[]"] 38 | else: 39 | return [f"{ctype_for_class(cls, namespace)}", f"{name}"] 40 | 41 | def flatten_args(arg_list): 42 | return [ f"{type} {name}" for (type, name) in arg_list] 43 | 44 | def get_result_type(func): 45 | return func.__annotations__.get("return", object) 46 | 47 | def generate_function_pointer(func): 48 | func_name = func_ptr_name(func) 49 | arg_names = func.__code__.co_varnames[: func.__code__.co_argcount] 50 | annotations = func.__annotations__ 51 | arg_decls = [ get_arg_decl(name, None, annotations) for name in arg_names] 52 | assert not cannot_fail(func) 53 | result_type = get_result_type(func) 54 | assert result_type is object 55 | all_args = [ CONTEXT ] + flatten_args(arg_decls) 56 | print(f"typedef PyRef (*{func_name})({', '.join(all_args)});\n") 57 | 58 | 59 | def generate_api_func(namespace, func): 60 | if namespace: 61 | func_name = f"PyApi_{namespace}_{func.__name__}" 62 | else: 63 | func_name = f"PyApi_{func.__name__}" 64 | arg_names = func.__code__.co_varnames[: func.__code__.co_argcount] 65 | annotations = func.__annotations__ 66 | arg_decls = [ get_arg_decl(name, namespace, annotations) for name in arg_names] 67 | return_type = get_result_type(func) 68 | return_arg = None 69 | if not cannot_fail(func): 70 | if return_type is Void or return_type is bool: 71 | return_type = int 72 | if isinstance(cls, Int): 73 | return_arg = return_type 74 | return_type = int 75 | return_type = ctype_for_class(return_type, namespace) 76 | all_args = flatten_args(arg_decls) 77 | if not is_shared(func): 78 | all_args = [ CONTEXT ] + all_args 79 | if return_arg is not None: 80 | all_args.append(ctype_for_class(return_arg, namespace)+"*") 81 | if func.__doc__: 82 | print(f"/* {func.__doc__} */") 83 | print(f"{return_type} {func_name}({', '.join(all_args)});\n") 84 | 85 | 86 | print("/* This file is generated from api.py by gen/__main__.py */\n") 87 | 88 | #Helpers 89 | print("PyRef PyRef_INVALID;") 90 | print("PyExceptionRef PyRef_NO_EXCEPTION;\n") 91 | 92 | print("PyRef PyRef_Dup(PyRef ref);") 93 | print("PyRef PyRef_Close(PyRef ref);") 94 | 95 | for name, obj in api.__dict__.items(): 96 | if isinstance(obj, FunctionType): 97 | if is_function_pointer(obj): 98 | generate_function_pointer(obj) 99 | else: 100 | generate_api_func(None, obj) 101 | if not isinstance(obj, type): 102 | continue 103 | cls = obj 104 | print(f"\n/* {name} */") 105 | for fname, func in cls.__dict__.items(): 106 | if not fname.startswith("_"): 107 | generate_api_func(name, func) 108 | if not is_namespace(cls): 109 | #Checks and casts 110 | print(f"bool PyApi_IsA{'n' if name[0] in 'AEIOU' else ''}{name}(PyRef ref);") 111 | print(f"Py{name}Ref PyApi_{name}_UnsafeCast(PyRef ref);") 112 | print(f"Py{name}Ref PyApi_{name}_DownCast(PyRef ref);") 113 | print(f"PyRef PyApi_{name}_UpCast(Py{name}Ref ref);\n") 114 | -------------------------------------------------------------------------------- /api/generate/abi.py: -------------------------------------------------------------------------------- 1 | 2 | class Int: 3 | def __init__(self, size, signed = True): 4 | self.size = size 5 | self.signed = signed 6 | prefix = "" if signed else "u" 7 | self.name = f"{prefix}int{size}_t" 8 | 9 | intptr_t = Int("ptr") 10 | uintptr_t = Int("ptr", False) 11 | uint8_t = Int(8, False) 12 | int32_t = Int(32) 13 | uint32_t = Int(32, False) 14 | int64_t = Int(64) 15 | uint64_t = Int(64, False) 16 | 17 | def abi(abi_string): 18 | def deco(func): 19 | func.abi_string = abi_string 20 | return func 21 | return deco 22 | 23 | def namespace(cls): 24 | cls._is_namespace = True 25 | return cls 26 | 27 | def is_namespace(cls): 28 | return hasattr(cls, "_is_namespace") 29 | 30 | Self = object() 31 | Void = object() 32 | Size = object() 33 | 34 | def no_fail(func): 35 | func._no_fail = True 36 | return func 37 | 38 | def cannot_fail(func): 39 | return hasattr(func, "_no_fail") 40 | 41 | def function_pointer(func): 42 | func._function_pointer = True 43 | return func 44 | 45 | def is_function_pointer(func): 46 | return hasattr(func, "_function_pointer") 47 | 48 | def no_result(func): 49 | ret_type = func.__annotations__.get("return", None) 50 | return ret_type is Void 51 | 52 | def shared(func): 53 | "Marker for things that returned shared references." 54 | func._shared = True 55 | return func 56 | 57 | def is_shared(func): 58 | return hasattr(func, "_shared") 59 | 60 | exports = ( 61 | intptr_t, uintptr_t, uint8_t, int32_t, uint32_t, int64_t, uint64_t, 62 | abi, namespace, Self, Void, no_fail, function_pointer, shared 63 | ) 64 | 65 | 66 | __all__ = [name for name in globals() if globals()[name] in exports] -------------------------------------------------------------------------------- /api/generate/api.py: -------------------------------------------------------------------------------- 1 | import builtins 2 | from unicodedata import name 3 | from .abi import * 4 | 5 | utf_string = "const UtfString" 6 | 7 | class Tuple: 8 | 9 | @no_fail 10 | def Empty() -> Self: ... 11 | 12 | @abi("Cn") 13 | def FromNonEmptyArray(array: [object], length: uintptr_t) -> Self: ... 14 | 15 | def FromArray(array: [object], length: uintptr_t) -> Self: ... 16 | 17 | def GetItem(self, index: uintptr_t): ... 18 | 19 | @no_fail 20 | def GetSize(self) -> uintptr_t: ... 21 | 22 | 23 | @function_pointer 24 | def BinaryOperator(left, right): ... 25 | 26 | @function_pointer 27 | def VectorCall(callable, args: [object], nargsf: intptr_t, kwnames: Tuple): ... 28 | 29 | #Forward declaration 30 | class Str: 31 | pass 32 | 33 | class Class: 34 | 35 | def AddBinaryOperator(op: uint8_t, func: BinaryOperator) -> Void: ... 36 | 37 | def AddVectorCallMethod(name: Str, func: VectorCall) -> Void: ... 38 | 39 | def New(self): ... 40 | 41 | 42 | @no_fail 43 | def IsNone(obj) -> bool: ... 44 | 45 | @no_fail 46 | @shared 47 | def none(): ... 48 | none.__name__ = "None" 49 | 50 | @no_fail 51 | def IsTrue(obj) -> bool: ... 52 | 53 | @no_fail 54 | @shared 55 | def true(): ... 56 | true.__name__ = "True" 57 | 58 | @no_fail 59 | def IsFalse(obj) -> bool: ... 60 | 61 | @no_fail 62 | @shared 63 | def false(): ... 64 | false.__name__ = "False" 65 | 66 | builtin_base_exception = BaseException 67 | for name, cls in builtins.__dict__.items(): 68 | # Maybe we don't want every last exception class, 69 | # but that's the simplest approach for now. 70 | if isinstance(cls, type) and name[0] != "_": 71 | @no_fail 72 | @shared 73 | def f() -> Class: ... 74 | f.__name__ = name 75 | globals()["class_" + name] = f 76 | del name, cls, builtin_base_exception 77 | 78 | class Str: 79 | 80 | def FromUtfString(data: utf_string, length: uintptr_t) -> Self: ... 81 | 82 | def Join(self, array: [Self]) -> Self: ... 83 | 84 | def GetItem(self, index: uintptr_t): ... 85 | 86 | @no_fail 87 | def GetSize(self) -> uintptr_t: ... 88 | 89 | class Bytes: 90 | 91 | def FromCData(data: "const char*", length: uintptr_t) -> Self: ... 92 | 93 | def GetItem(self, index: uintptr_t): ... 94 | 95 | @no_fail 96 | def GetSize(self) -> uintptr_t: ... 97 | 98 | def CopyToBuffer(self, start: uintptr_t, end: uintptr_t, buffer: "const char*") -> Void: ... 99 | 100 | #Builders 101 | class StrBuilder: 102 | 103 | def New(capacity: uintptr_t) -> Self: ... 104 | 105 | def AppendStr(Self, s:Str) -> Void: ... 106 | 107 | def AppendUtf8String(Self, s:utf_string) -> Void: ... 108 | 109 | @abi("C") 110 | def ToStr(Self) -> Str: ... 111 | 112 | class TupleBuilder: 113 | 114 | def New(capacity: uintptr_t) -> Self: ... 115 | 116 | def Add(Self, s:Str) -> Void: ... 117 | 118 | @abi("C") 119 | def ToTuple(Self) -> Str: ... 120 | 121 | @namespace 122 | class Object: 123 | 124 | def GetItem(obj, key): ... 125 | 126 | def GetItem_i(obj, key: intptr_t): ... 127 | 128 | def GetItem_s(obj, key: utf_string): ... 129 | 130 | def SetItem(obj, key, value) -> Void: ... 131 | 132 | def SetItem_i(obj, key: intptr_t, value) -> Void: ... 133 | 134 | def SetItem_s(obj, key: utf_string, value) -> Void: ... 135 | 136 | def GetAttr(obj, attr): ... 137 | 138 | def GetAttr_s(obj, attr: utf_string): ... 139 | 140 | def HasAttr(obj, attr) -> bool: ... 141 | 142 | def HasAttr_s(obj, attr: utf_string) -> bool: ... 143 | 144 | def SetAttr(obj, attr, value) -> Void: ... 145 | 146 | def SetAttr_s(obj, attr: utf_string, value) -> Void: ... 147 | 148 | def Contains(container, key) -> bool: ... 149 | 150 | @no_fail 151 | def Type(obj) -> Class: ... 152 | 153 | @no_fail 154 | def TypeCheck(obj, cls: Class) -> bool: ... 155 | 156 | def Repr(obj) -> Str: ... 157 | 158 | def Str(obj) -> Str: ... 159 | 160 | def Hash(obj) -> intptr_t: ... 161 | 162 | def CallMethod(attr, args: [object], nargsf: intptr_t): ... 163 | 164 | def Compare(op: uint8_t, left, right) -> bool: ... 165 | 166 | def IsIter(obj) -> bool: ... 167 | 168 | def IsAnIter(obj) -> bool: ... 169 | 170 | class Dict: 171 | 172 | def Get(self, key, default): ... 173 | 174 | def GetItem(self, key): ... 175 | 176 | def New() -> Self: ... 177 | 178 | class Int: 179 | 180 | def FromInt32(val: int32_t) -> Self: ... 181 | 182 | def FromUInt32(val: int32_t) -> Self: ... 183 | 184 | def FromInt64(val: int32_t) -> Self: ... 185 | 186 | def FromUInt64(val: int32_t) -> Self: ... 187 | 188 | def ToInt32(self) -> int32_t: ... 189 | 190 | def ToInt64(self) -> int64_t: ... 191 | 192 | 193 | class List: 194 | 195 | def New() -> Self: ... 196 | 197 | @abi("BC") 198 | def Append(self, item) -> Void: ... 199 | 200 | def GetItem(self, index: uintptr_t): ... 201 | 202 | @no_fail 203 | def GetSize(self) -> uintptr_t: ... 204 | 205 | def Pop(self): ... 206 | 207 | 208 | @namespace 209 | class Operators: 210 | 211 | def UnaryOp(op: uint8_t, argument): ... 212 | 213 | def BinaryOp(op: uint8_t, left, right): ... 214 | 215 | def Compare(left, right, op: uint8_t): ... 216 | 217 | def CompareBool(left, right, op: uint8_t) -> bool: ... 218 | 219 | 220 | class Exception: 221 | 222 | def FromString(cls: Class, message: utf_string) -> Self: ... 223 | 224 | def FromValue(cls: Class, message) -> Self: ... 225 | 226 | @no_fail 227 | def Fatal(message: utf_string) -> Void: ... 228 | 229 | def FromErrnoWithFilename(cls: Class, filename: utf_string) -> Self: ... 230 | 231 | def RaiseFromString(cls: Class, message: utf_string) -> Self: 232 | """Always fails. Barring another error, 233 | will set *error to the newly created exception. 234 | """ 235 | 236 | def RaiseFromValue(cls: Class, message) -> Self: 237 | """Always fails. Barring another error, 238 | will set *error to the newly created exception. 239 | """ 240 | 241 | @no_fail 242 | def GetLatestException() -> Exception: 243 | """If the immediately previous API call failed, 244 | then return the exception set by that exception. 245 | Otherwise, return either an arbitrary exception 246 | or PyRef_NO_EXCEPTION""" 247 | 248 | @namespace 249 | class Call: 250 | 251 | def IsCallable(obj) -> bool: ... 252 | 253 | def TupleDict(callable, args: Tuple, kwargs: Dict): ... 254 | 255 | def Vector(callable, args: [object], nargsf: intptr_t, kwnames: Tuple): ... 256 | 257 | @namespace 258 | class Iter: 259 | 260 | def Next(obj): ... 261 | 262 | def NextX(obj): 263 | """If no more results are available then, instead of raising StopIteration, 264 | *error = PyRef_NO_EXCEPTION. 265 | This is more efficient than the plain Next function.""" 266 | 267 | def Send(obj): ... 268 | 269 | def SendX(obj, returned: "int *"): 270 | """If no exception is raised, then returned is set to 0 for 271 | a yield, and 1 for a return. 272 | This is more efficient than the plain Send function.""" 273 | 274 | class Bytes: 275 | 276 | def GetItem(self, index: uintptr_t)-> uint8_t: ... 277 | 278 | def FromArray(data: "const char *", length: uintptr_t) -> Self: ... 279 | 280 | def GetSize(self) -> uintptr_t: ... 281 | 282 | class Code: 283 | 284 | "To do" 285 | 286 | @namespace 287 | class FrameStack: 288 | 289 | def GetLocal(depth: uintptr_t, index: uintptr_t): ... 290 | 291 | def GetLocalByName(depth: uintptr_t, name: Str): ... 292 | 293 | def GetLocalByCName(depth: uintptr_t, name: utf_string): ... 294 | 295 | def GetCode(depth: uintptr_t) -> Code: ... 296 | 297 | del abi, namespace, no_fail, function_pointer, shared 298 | -------------------------------------------------------------------------------- /examples.md: -------------------------------------------------------------------------------- 1 | # Examples 2 | 3 | ## Example API functions 4 | 5 | ### A simple getter function 6 | 7 | This function gets the length of a tuple. 8 | Very simple getters that return a C scalar do not need a `PyContext` parameter. 9 | 10 | ```C 11 | uintptr_t PyApi_Tuple_GetLength(PyTupleRef t); 12 | ``` 13 | 14 | And here is an implementation for CPython (most versions up to 3.11) 15 | ```C 16 | typedef struct _py_tuple_ref { 17 | PyTupleObject *pointer; 18 | } PyTupleRef; 19 | 20 | uintptr_t 21 | PyApi_Tuple_GetLength(PyTupleRef t) 22 | { 23 | return (uintptr_t)Py_SIZE(t.pointer); 24 | } 25 | 26 | ``` 27 | 28 | ### A computed getter function 29 | 30 | This function gets the name of a class object 31 | ```C 32 | PyStrRef PyApi_Class_GetName(PyContext ctx, PyClassRef cls); 33 | ``` 34 | 35 | And here is an implementation for CPython (most versions up to 3.11) 36 | ```C 37 | typedef struct _py_class_ref { 38 | PyTypeObject *pointer; 39 | } PyClassRef; 40 | 41 | PyStrRef 42 | PyApi_Class_GetName(PyContext ctx, PyClassRef cls) 43 | { 44 | PyObject *name = PyUnicode_FromString(cls.pointer->tp_name); 45 | return PyApi_Interop_FromUnicodeObject_C(name); 46 | } 47 | ``` 48 | 49 | ### A function with failure, but no error 50 | 51 | This function unboxes an integer, with possible overflow. 52 | ```C 53 | 54 | intptr_t PyApi_Number_UnboxAsInt(PyIntRef i, int* overflow) 55 | { 56 | PyObject *obj = i.pointer; 57 | assert (PyLong_check(obj)); 58 | 59 | if (PyLong_WillOverflowSsize_t(obj)) { 60 | int sign = _PyLong_Sign(obj); 61 | *overflow = sign; 62 | return sign > 0 ? INTPTR_MAX : INTPTR_MIN; 63 | } 64 | *overflow = 0; 65 | return PyLong_AsSsize_t(obj); 66 | } 67 | ``` 68 | 69 | ### A function with both failure and error 70 | 71 | This function gets a value from a dictionary 72 | 73 | ```C 74 | 75 | PyRef PyApi_Dict_GetItem(PyContext ctx, PyDictRef d, PyRef key, PyExceptionRef *error) 76 | { 77 | PyObject *dp = d.pointer; 78 | PyObject *kp = key.pointer; 79 | PyObject *res = _PyDict_GetItemWithError(PyObject *dp, PyObject *kp); 80 | if (res != NULL) { 81 | assert(!Py_ErrOccurred()); 82 | return PyApi_Interop_FromObject_C(res); 83 | } 84 | if (Py_ErrOccurred()) { 85 | PyObject *exception = get_normalized_exception(); 86 | *error = PyApi_Interop_FromException(exception); 87 | return PyRef_INVALID; 88 | } 89 | /* Special case -- failure. No result, but no exception */ 90 | *error = PyRef_NO_EXCEPTION; 91 | return PyRef_INVALID; 92 | } 93 | ``` 94 | 95 | ## Constructing API functions from ABI functions 96 | 97 | ### Handling references 98 | 99 | For the ABI function 100 | ``` 101 | PyExceptionRef PyApi_Tuple_SetItem_BnC(PyContext ctx, PyTupleRef t, uintptr_t index, PyRef item); 102 | ``` 103 | 104 | We can construct the API function that borrows the reference simply: 105 | ``` 106 | inline PyExceptionRef 107 | PyApi_Tuple_SetItem(PyContext ctx, PyTupleRef t, uintptr_t index, PyRef item) 108 | { 109 | PyRef arg2 = PyRef_Dup(ctx, item); 110 | return PyApi_Tuple_SetItem_BnC(ctx, t, index, arg2) 111 | } 112 | ``` 113 | 114 | ### Handling edge cases 115 | 116 | Some ABI functions are designed primarily for efficiency, so we 117 | want to wrap them in a friendlier API that handles edge cases. 118 | 119 | For example, 120 | ``` 121 | PyTupleRef PyApi_Tuple_FromNonEmptyArray_nC(PyContext ctx, uintptr_t len, PyRef *values, PyExceptionRef *error); 122 | ``` 123 | can be wrapped in a macro to give the nicer API: 124 | ``` 125 | PyTupleRef PyApi_Tuple_FromFixedArray(ctx, array, error) 126 | ``` 127 | 128 | ``` 129 | #define PyApi_Tuple_FromFixedArray(ctx, array, error) \ 130 | ((sizeof(array) == 0) ? \ 131 | PyApi_Tuple_Empty(ctx) 132 | : \ 133 | PyApi_Tuple_FromNonEmptyArray(ctx, sizeof(array)/sizeof(PyRef), &array, error) 134 | ) 135 | ``` 136 | Allowing it be used like this: 137 | ``` 138 | PyRef args[4] = { 139 | PyNone, 140 | arg1, 141 | arg2, 142 | PyNone 143 | }; 144 | PyTupleRef new_tuple; 145 | int err = PyApi_Tuple_FromFixedArray(ctx, args, &new_tuple); 146 | ``` 147 | 148 | 149 | ### Downcast macros 150 | 151 | Here is a possible implementation of a downcast macro: 152 | ``` 153 | #define PyApi_List_CheckAndDowncast(OBJ, LIST) \ 154 | (PyRef_IsList(OBJ) ? (LIST = PyApi_List_UnsafeCast(OBJ), 1) : 0) 155 | ``` 156 | 157 | -------------------------------------------------------------------------------- /implementation.md: -------------------------------------------------------------------------------- 1 | # Implementing the new C API in CPython 2 | 3 | There are approximately 44 thousand occurences of `PyObject *` in the CPython codebase. 4 | That is a lot to change. 5 | 6 | However, we don't need to change all the code that uses `PyObject *` to `PyRef` overnight. 7 | In fact until we change the implementation of `PyRef` to tagged pointers, or similar, we 8 | don't need to change much at all. 9 | 10 | Starting with the simplest representation of `PyRef` as 11 | ``` 12 | typedef struct { 13 | PyObject *ptr; 14 | } PyRef; 15 | ``` 16 | 17 | Most uses of `PyObject *` are one of a few simple operations, such as `Py_INCREF()`, `Py_DECREF()` 18 | and `Py_TYPE()`. 19 | 20 | Changing `Py_TYPE()` is easy enough. 21 | `Py_TYPE(obj)` becomes `PyAPI_Object_GetClass(obj)`. As long as `PYAPI_NO_ABI` is defined, it remains efficient. 22 | 23 | As a stopgap, we can add `PyRef_INCREF()`, `PyRef_DECREF()`. 24 | Ultimately they will have to go, but we can leave them in for a few years. 25 | 26 | Most other uses of `PyObject *` involving passing it to another C API function, or casting to a more 27 | specific class. With the possible future exception of `int` and `float`, we can continue to cast. 28 | `(PyXXXObject *)obj` becomes `PyApi_XXX_Unsafecast(obj)`. 29 | Converting to safe casts can be done later. 30 | 31 | Adding two inline functions should help as well. 32 | ``` 33 | PyRef TO_REF(PyObject *); 34 | PyObject *TO_PTR(PyRef ref); 35 | ``` 36 | 37 | --------------------------------------------------------------------------------