├── README └── except_star.md /README: -------------------------------------------------------------------------------- 1 | This repo is used as a place to discuss adding a concept of exception 2 | groups (also known as multi-errors) to Python 3.10. Checkout Issues 3 | for more information. 4 | 5 | Experimental implementation : https://github.com/iritkatriel/cpython/tree/exceptionGroup-stage5 6 | PRs for discussion: 7 | ExceptionGroup and except*: https://github.com/iritkatriel/cpython/pull/7 8 | Plus raise inside except* block: https://github.com/iritkatriel/cpython/pull/8 9 | -------------------------------------------------------------------------------- /except_star.md: -------------------------------------------------------------------------------- 1 | 2 | # Exception Groups and except* 3 | 4 | ## Abstract 5 | 6 | This PEP proposes language extensions that allow programs to raise and handle 7 | multiple unrelated exceptions simultaneously: 8 | 9 | * A new standard exception type, the `ExceptionGroup`, which represents a 10 | group of unrelated exceptions being propagated together. 11 | 12 | * A new syntax `except*` for handling `ExceptionGroup`s. 13 | 14 | ## Motivation 15 | 16 | The interpreter is currently able to propagate at most one exception at a 17 | time. The chaining features introduced in 18 | [PEP 3134](https://www.python.org/dev/peps/pep-3134/) link together exceptions that are 19 | related to each other as the cause or context, but there are situations where 20 | multiple unrelated exceptions need to be propagated together as the stack 21 | unwinds. Several real world use cases are listed below. 22 | 23 | * **Concurrent errors**. Libraries for async concurrency provide APIs to invoke 24 | multiple tasks and return their results in aggregate. There isn't currently 25 | a good way for such libraries to handle situations where multiple tasks 26 | raise exceptions. The Python standard library's 27 | [`asyncio.gather()`](https://docs.python.org/3/library/asyncio-task.html#asyncio.gather) 28 | function provides two options: raise the first exception, or return the 29 | exceptions in the results list. The [Trio](https://trio.readthedocs.io/en/stable/) 30 | library has a `MultiError` exception type which it raises to report a 31 | collection of errors. Work on this PEP was initially motivated by the 32 | difficulties in handling `MultiError`s, which are detailed in a design 33 | document for an 34 | [improved version, `MultiError2`](https://github.com/python-trio/trio/issues/611). 35 | That document demonstrates how difficult it is to create an effective API 36 | for reporting and handling multiple errors without the language changes we 37 | are proposing. 38 | 39 | * **Multiple failures when retrying an operation.** The Python standard library's 40 | `socket.create_connection` may attempt to connect to different addresses, 41 | and if all attempts fail it needs to report that to the user. It is an open 42 | issue how to aggregate these errors, particularly when they are different 43 | [[Python issue 29980](https://bugs.python.org/issue29980)]. 44 | 45 | * **Multiple user callbacks fail.** Python's `atexit.register()` allows users 46 | to register functions that are called on system exit. If any of them raise 47 | exceptions, only the last one is reraised, but it would be better to reraised 48 | all of them together 49 | [[`atexit` documentation](https://docs.python.org/3/library/atexit.html#atexit.register)]. 50 | Similarly, the pytest library allows users to register finalizers which 51 | are executed at teardown. If more than one of these finalizers raises an 52 | exception, only the first is reported to the user. This can be improved with 53 | `ExceptionGroup`s, as explained in this issue by pytest developer Ran Benita 54 | [[Pytest issue 8217](https://github.com/pytest-dev/pytest/issues/8217)] 55 | 56 | * **Multiple errors in a complex calculation.** The Hypothesis library performs 57 | automatic bug reduction (simplifying code that demonstrates a bug). In the 58 | process it may find variations that generate different errors, and 59 | (optionally) reports all of them 60 | [[Hypothesis documentation](https://hypothesis.readthedocs.io/en/latest/settings.html#hypothesis.settings.report_multiple_bugs)]. 61 | An `ExceptionGroup` mechanism as we are proposing here can resolve some of 62 | the difficulties with debugging that are mentioned in the link above, and 63 | which are due to the loss of context/cause information (communicated 64 | by Hypothesis Core Developer Zac Hatfield-Dodds). 65 | 66 | * **Errors in wrapper code.** The Python standard library's 67 | `tempfile.TemporaryDirectory` context manager 68 | had an issue where an exception raised during cleanup in `__exit__` 69 | effectively masked an exception that the user's code raised inside the context 70 | manager scope. While the user's exception was chained as the context of the 71 | cleanup error, it was not caught by the user's except clause 72 | [[Python issue 40857](https://bugs.python.org/issue40857)]. 73 | The issue was resolved by making the cleanup code ignore errors, thus 74 | sidestepping the multiple exception problem. With the features we propose 75 | here, it would be possible for `__exit__` to raise an `ExceptionGroup` 76 | containing its own errors as well as the user's errors as unrelated errors, 77 | and this would allow the user to catch their own exceptions by their types. 78 | 79 | 80 | ## Rationale 81 | 82 | Grouping several exceptions together can be done without changes to the 83 | language, simply by creating a container exception type. 84 | [Trio](https://trio.readthedocs.io/en/stable/) is an example of a library that 85 | has made use of this technique in its 86 | [`MultiError` type](https://trio.readthedocs.io/en/stable/reference-core.html#trio.MultiError). 87 | However, such approaches require calling code to catch the container exception 88 | type, and then inspect it to determine the types of errors that had occurred, 89 | extract the ones it wants to handle and reraise the rest. 90 | 91 | Changes to the language are required in order to extend support for 92 | `ExceptionGroup`s in the style of existing exception handling mechanisms. At 93 | the very least we would like to be able to catch an `ExceptionGroup` only if 94 | it contains an exception type that we choose to handle. Exceptions of 95 | other types in the same `ExceptionGroup` need to be automatically reraised, 96 | otherwise it is too easy for user code to inadvertently swallow exceptions 97 | that it is not handling. 98 | 99 | The purpose of this PEP, then, is to add the `except*` syntax for handling 100 | `ExceptionGroup`s in the interpreter, which in turn requires that 101 | `ExceptionGroup` is added as a builtin type. The semantics of handling 102 | `ExceptionGroup`s are not backwards compatible with the current exception 103 | handling semantics, so we are not proposing to modify the behavior of the 104 | `except` keyword but rather to add the new `except*` syntax. 105 | 106 | 107 | ## Specification 108 | 109 | ### ExceptionGroup 110 | 111 | The new builtin exception type, `ExceptionGroup` is a subclass of 112 | `BaseException`, so it is assignable to `Exception.__cause__` and 113 | `Exception.__context__`, and can be raised and handled as any exception 114 | with `raise ExceptionGroup(...)` and `try: ... except ExceptionGroup: ...`. 115 | 116 | Its constructor takes two positional-only parameters: a message string and a 117 | sequence of the nested exceptions, for example: 118 | `ExceptionGroup('issues', [ValueError('bad value'), TypeError('bad type')])`. 119 | 120 | The `ExceptionGroup` class exposes these parameters in the fields `message` 121 | and `errors`. A nested exception can also be an `ExceptionGroup` so the class 122 | represents a tree of exceptions, where the leaves are plain exceptions and 123 | each internal node represent a time at which the program grouped some 124 | unrelated exceptions into a new `ExceptionGroup` and raised them together. 125 | The `ExceptionGroup` class is final, i.e., it cannot be subclassed. 126 | 127 | The `ExceptionGroup.subgroup(condition)` method gives us a way to obtain an 128 | `ExceptionGroup` that has the same metadata (cause, context, traceback) as 129 | the original group, and the same nested structure of `ExceptionGroup`s, but 130 | contains only those exceptions for which the condition is true: 131 | 132 | ```python 133 | >>> eg = ExceptionGroup("one", 134 | ... [TypeError(1), 135 | ... ExceptionGroup("two", 136 | ... [TypeError(2), ValueError(3)]), 137 | ... ExceptionGroup("three", 138 | ... [OSError(4)]) 139 | ... ]) 140 | >>> traceback.print_exception(eg) 141 | ExceptionGroup: one 142 | ------------------------------------------------------------ 143 | TypeError: 1 144 | ------------------------------------------------------------ 145 | ExceptionGroup: two 146 | ------------------------------------------------------------ 147 | TypeError: 2 148 | ------------------------------------------------------------ 149 | ValueError: 3 150 | ------------------------------------------------------------ 151 | ExceptionGroup: three 152 | ------------------------------------------------------------ 153 | OSError: 4 154 | >>> type_errors = eg.subgroup(lambda e: isinstance(e, TypeError)) 155 | >>> traceback.print_exception(type_errors) 156 | ExceptionGroup: one 157 | ------------------------------------------------------------ 158 | TypeError: 1 159 | ------------------------------------------------------------ 160 | ExceptionGroup: two 161 | ------------------------------------------------------------ 162 | TypeError: 2 163 | >>> 164 | ``` 165 | 166 | Empty nested `ExceptionGroup`s are omitted from the result, as in the 167 | case of `ExceptionGroup("three")` in the example above. The original `eg` 168 | is unchanged by `subgroup`, but the value returned is not necessarily a full 169 | new copy. Leaf exceptions are not copied, nor are `ExceptionGroup`s which are 170 | fully contained in the result. When it is necessary to partition an 171 | `ExceptionGroup` because the condition holds for some, but not all of its 172 | contained exceptions, a new `ExceptionGroup` is created but the `__cause__`, 173 | `__context__` and `__traceback__` fields are copied by reference, so are shared 174 | with the original `eg`. 175 | 176 | If both the subgroup and its complement are needed, the 177 | `ExceptionGroup.split(condition)` method can be used: 178 | 179 | ```Python 180 | >>> type_errors, other_errors = eg.split(lambda e: isinstance(e, TypeError)) 181 | >>> traceback.print_exception(type_errors) 182 | ExceptionGroup: one 183 | ------------------------------------------------------------ 184 | TypeError: 1 185 | ------------------------------------------------------------ 186 | ExceptionGroup: two 187 | ------------------------------------------------------------ 188 | TypeError: 2 189 | >>> traceback.print_exception(other_errors) 190 | ExceptionGroup: one 191 | ------------------------------------------------------------ 192 | ExceptionGroup: two 193 | ------------------------------------------------------------ 194 | ValueError: 3 195 | ------------------------------------------------------------ 196 | ExceptionGroup: three 197 | ------------------------------------------------------------ 198 | OSError: 4 199 | >>> 200 | ``` 201 | 202 | If a split is trivial (one side is empty), then None is returned for the 203 | other side: 204 | 205 | ```python 206 | >>> other_errors.split(lambda e: isinstance(e, SyntaxError)) 207 | (None, ExceptionGroup('one', [ExceptionGroup('two', [ValueError(3)]), ExceptionGroup('three', [OSError(4)])])) 208 | ``` 209 | 210 | Since splitting by exception type is a very common use case, `subgroup` and 211 | `split` can take an exception type or tuple of exception types and treat it 212 | as a shorthand for matching that type: `eg.split(T)` divides `eg` into the 213 | subgroup of leaf exceptions that match the type `T`, and the subgroup of those 214 | that do not (using the same check as `except` for a match). 215 | 216 | #### The Traceback of an `ExceptionGroup` 217 | 218 | For regular exceptions, the traceback represents a simple path of frames, 219 | from the frame in which the exception was raised to the frame in which it 220 | was caught or, if it hasn't been caught yet, the frame that the program's 221 | execution is currently in. The list is constructed by the interpreter, which 222 | appends any frame from which it exits to the traceback of the 'current 223 | exception' if one exists. To support efficient appends, the links in a 224 | traceback's list of frames are from the oldest to the newest frame. Appending 225 | a new frame is then simply a matter of inserting a new head to the linked 226 | list referenced from the exception's `__traceback__` field. Crucially, the 227 | traceback's frame list is immutable in the sense that frames only need to be 228 | added at the head, and never need to be removed. 229 | 230 | We do not need to make any changes to this data structure. The `__traceback__` 231 | field of the `ExceptionGroup` instance represents the path that the contained 232 | exceptions travelled through together after being joined into the 233 | `ExceptionGroup`, and the same field on each of the nested exceptions 234 | represents the path through which this exception arrived at the frame of the 235 | merge. 236 | 237 | What we do need to change is any code that interprets and displays tracebacks, 238 | because it now needs to continue into tracebacks of nested exceptions, as 239 | in the following example: 240 | 241 | ```python 242 | >>> def f(v): 243 | ... try: 244 | ... raise ValueError(v) 245 | ... except ValueError as e: 246 | ... return e 247 | ... 248 | >>> try: 249 | ... raise ExceptionGroup("one", [f(1)]) 250 | ... except ExceptionGroup as e: 251 | ... eg1 = e 252 | ... 253 | >>> try: 254 | ... raise ExceptionGroup("two", [f(2), eg1]) 255 | ... except ExceptionGroup as e: 256 | ... eg2 = e 257 | ... 258 | >>> import traceback 259 | >>> traceback.print_exception(eg2) 260 | Traceback (most recent call last): 261 | File "", line 2, in 262 | ExceptionGroup: two 263 | ------------------------------------------------------------ 264 | Traceback (most recent call last): 265 | File "", line 3, in f 266 | ValueError: 2 267 | ------------------------------------------------------------ 268 | Traceback (most recent call last): 269 | File "", line 2, in 270 | ExceptionGroup: one 271 | ------------------------------------------------------------ 272 | Traceback (most recent call last): 273 | File "", line 3, in f 274 | ValueError: 1 275 | >>> 276 | ``` 277 | 278 | ### except* 279 | 280 | We are proposing to introduce a new variant of the `try..except` syntax to 281 | simplify working with exception groups. The `*` symbol indicates that multiple 282 | exceptions can be handled by each `except*` clause: 283 | 284 | ```python 285 | try: 286 | ... 287 | except *SpamError: 288 | ... 289 | except *FooError as e: 290 | ... 291 | except *(BarError, BazError) as e: 292 | ... 293 | ``` 294 | 295 | In a traditional `try-except` statement there is only one exception to handle, 296 | so the body of at most one `except` clause executes; the first one that matches 297 | the exception. With the new syntax, an `except*` clause can match a subgroup 298 | of the `ExceptionGroup` that was raised, while the remaining part is matched 299 | by following `except*` clauses. In other words, a single `ExceptionGroup` can 300 | cause several `except*` clauses to execute, but each such clause executes at 301 | most once (for all matching exceptions from the group) and each exception is 302 | either handled by exactly one clause (the first one that matches its type) 303 | or is reraised at the end. 304 | 305 | For example, suppose that the body of the `try` block above raises 306 | `eg = ExceptionGroup('msg', [FooError(1), FooError(2), BazError()])`. 307 | The `except*` clauses are evaluated in order by calling `split` on the 308 | `unhandled` `ExceptionGroup`, which is initially equal to `eg` and then shrinks 309 | as exceptions are matched and extracted from it. In the first `except*` clause, 310 | `unhandled.split(SpamError)` returns `(None, unhandled)` so the body of this 311 | block is not executed and `unhandled` is unchanged. For the second block, 312 | `unhandled.split(FooError)` returns a non-trivial split `(match, rest)` with 313 | `match = ExceptionGroup('msg', [FooError(1), FooError(2)])` 314 | and `rest = ExceptionGroup('msg', [BazError()])`. The body of this `except*` 315 | block is executed, with the value of `e` and `sys.exc_info()` set to `match`. 316 | Then, `unhandled` is set to `rest`. 317 | Finally, the third block matches the remaining exception so it is executed 318 | with `e` and `sys.exc_info()` set to `ExceptionGroup('msg', [BazError()])`. 319 | 320 | 321 | Exceptions are matched using a subclass check. For example: 322 | 323 | ```python 324 | try: 325 | low_level_os_operation() 326 | except *OSerror as eg: 327 | for e in eg.errors: 328 | print(type(e).__name__) 329 | ``` 330 | 331 | could output: 332 | 333 | ``` 334 | BlockingIOError 335 | ConnectionRefusedError 336 | OSError 337 | InterruptedError 338 | BlockingIOError 339 | ``` 340 | 341 | The order of `except*` clauses is significant just like with the regular 342 | `try..except`: 343 | 344 | ```python 345 | >>> try: 346 | ... raise ExceptionGroup("problem", [BlockingIOError()]) 347 | ... except *OSError as e: # Would catch the error 348 | ... print(repr(e)) 349 | ... except *BlockingIOError: # Would never run 350 | ... print('never') 351 | ... 352 | ExceptionGroup('problem', [BlockingIOError()]) 353 | ``` 354 | 355 | ### Recursive Matching 356 | 357 | The matching of `except*` clauses against an `ExceptionGroup` is performed 358 | recursively, using the `ExceptionGroup.split()` method: 359 | 360 | ```python 361 | >>> try: 362 | ... raise ExceptionGroup( 363 | ... "eg", 364 | ... [ValueError('a'), 365 | ... TypeError('b'), 366 | ... ExceptionGroup("nested", [TypeError('c'), KeyError('d')]) 367 | ... ] 368 | ... ) 369 | ... except *TypeError as e1: 370 | ... print(f'e1 = {e1!r}') 371 | ... except *Exception as e2: 372 | ... print(f'e2 = {e2!r}') 373 | ... 374 | e1 = ExceptionGroup('eg', [TypeError('b'), ExceptionGroup('nested', [TypeError('c')])]) 375 | e2 = ExceptionGroup('eg', [ValueError('a'), ExceptionGroup('nested', [KeyError('d')])]) 376 | >>> 377 | ``` 378 | 379 | ### Unmatched Exceptions 380 | 381 | If not all exceptions in an `ExceptionGroup` were matched by the `except*` 382 | clauses, the remaining part of the `ExceptionGroup` is propagated on: 383 | 384 | ```python 385 | >>> try: 386 | ... try: 387 | ... raise ExceptionGroup( 388 | ... "msg", [ValueError('a'), TypeError('b'), TypeError('c'), KeyError('e')] 389 | ... ) 390 | ... except *ValueError as e: 391 | ... print(f'got some ValueErrors: {e!r}') 392 | ... except *TypeError as e: 393 | ... print(f'got some TypeErrors: {e!r}') 394 | ... except ExceptionGroup as e: 395 | ... print(f'propagated: {e!r}') 396 | ... 397 | got some ValueErrors: ExceptionGroup('msg', [ValueError('a')]) 398 | got some TypeErrors: ExceptionGroup('msg', [TypeError('b'), TypeError('c')]) 399 | propagated: ExceptionGroup('msg', [KeyError('e')]) 400 | >>> 401 | ``` 402 | 403 | ### Naked Exceptions 404 | 405 | If the exception raised inside the `try` body is not of type `ExceptionGroup`, 406 | we call it a `naked` exception. If its type matches one of the `except*` 407 | clauses, it is caught and wrapped by an `ExceptionGroup` with an empty message 408 | string. This is to make the type of `e` consistent and statically known: 409 | 410 | ```python 411 | >>> try: 412 | ... raise BlockingIOError 413 | ... except *OSError as e: 414 | ... print(repr(e)) 415 | ... 416 | ExceptionGroup('', [BlockingIOError()]) 417 | ``` 418 | 419 | However, if a naked exception is not caught, it propagates in its original 420 | naked form: 421 | 422 | ```python 423 | >>> try: 424 | ... try: 425 | ... raise ValueError(12) 426 | ... except *TypeError as e: 427 | ... print('never') 428 | ... except ValueError as e: 429 | ... print(f'caught ValueError: {e!r}') 430 | ... 431 | caught ValueError: ValueError(12) 432 | >>> 433 | ``` 434 | 435 | ### Raising exceptions in an `except*` block 436 | 437 | In a traditional `except` block, there are two ways to raise exceptions: 438 | `raise e` to explicitly raise an exception object `e`, or naked `raise` to 439 | reraise the 'current exception'. When `e` is the current exception, the two 440 | forms are not equivalent because a reraise does not add the current frame to 441 | the stack: 442 | 443 | ```python 444 | def foo(): | def foo(): 445 | try: | try: 446 | 1 / 0 | 1 / 0 447 | except ZeroDivisionError as e: | except ZeroDivisionError: 448 | raise e | raise 449 | | 450 | foo() | foo() 451 | | 452 | Traceback (most recent call last): | Traceback (most recent call last): 453 | File "/Users/guido/a.py", line 7 | File "/Users/guido/b.py", line 7 454 | foo() | foo() 455 | File "/Users/guido/a.py", line 5 | File "/Users/guido/b.py", line 3 456 | raise e | 1/0 457 | File "/Users/guido/a.py", line 3 | ZeroDivisionError: division by zero 458 | 1/0 | 459 | ZeroDivisionError: division by zero | 460 | ``` 461 | 462 | This holds for `ExceptionGroup`s as well, but the situation is now more complex 463 | because there can be exceptions raised and reraised from multiple `except*` 464 | clauses, as well as unhandled exceptions that need to propagate. 465 | The interpreter needs to combine all those exceptions into a result, and 466 | raise that. 467 | 468 | The reraised exceptions and the unhandled exceptions are subgroups of the 469 | original `ExceptionGroup`, and share its metadata (cause, context, traceback). 470 | On the other hand, each of the explicitly raised exceptions has its own 471 | metadata - the traceback contains the line from which it was raised, its 472 | cause is whatever it may have been explicitly chained to, and its context is the 473 | value of `sys.exc_info()` in the `except*` clause of the raise. 474 | 475 | In the aggregated `ExceptionGroup`, the reraised and unhandled exceptions have 476 | the same relative structure as in the original exception, as if they were split 477 | off together in one `subgroup` call. For example, in the snippet below the 478 | inner `try-except*` block raises an `ExceptionGroup` that contains all 479 | `ValueError`s and `TypeError`s merged back into the same shape they had in 480 | the original `ExceptionGroup`: 481 | 482 | ```python 483 | >>> try: 484 | ... try: 485 | ... raise ExceptionGroup("eg", 486 | ... [ValueError(1), 487 | ... TypeError(2), 488 | ... OSError(3), 489 | ... ExceptionGroup( 490 | ... "nested", 491 | ... [OSError(4), TypeError(5), ValueError(6)])]) 492 | ... except *ValueError as e: 493 | ... print(f'*ValueError: {e!r}') 494 | ... raise 495 | ... except *OSError as e: 496 | ... print(f'*OSError: {e!r}') 497 | ... except ExceptionGroup as e: 498 | ... print(repr(e)) 499 | ... 500 | *ValueError: ExceptionGroup('eg', [ValueError(1), ExceptionGroup('nested', [ValueError(6)])]) 501 | *OSError: ExceptionGroup('eg', [OSError(3), ExceptionGroup('nested', [OSError(4)])]) 502 | ExceptionGroup('eg', [ValueError(1), TypeError(2), ExceptionGroup('nested', [TypeError(5), ValueError(6)])]) 503 | >>> 504 | ``` 505 | 506 | When exceptions are raised explicitly, they are independent of the original 507 | exception group, and cannot be merged with it (they have their own cause, 508 | context and traceback). Instead, they are combined into a new `ExceptionGroup`, 509 | which also contains the reraised/unhandled subgroup described above. 510 | 511 | In the following example, the `ValueError`s were raised so they are in their 512 | own `ExceptionGroup`, while the `OSError`s were reraised so they were 513 | merged with the unhandled `TypeError`s. 514 | 515 | ```python 516 | >>> try: 517 | ... try: 518 | ... raise ExceptionGroup("eg", 519 | ... [ValueError(1), 520 | ... TypeError(2), 521 | ... OSError(3), 522 | ... ExceptionGroup( 523 | ... "nested", 524 | ... [OSError(4), TypeError(5), ValueError(6)])]) 525 | ... except *ValueError as e: 526 | ... print(f'*ValueError: {e!r}') 527 | ... raise e 528 | ... except *OSError as e: 529 | ... print(f'*OSError: {e!r}') 530 | ... raise 531 | ... except ExceptionGroup as e: 532 | ... traceback.print_exception(e) 533 | ... 534 | *ValueError: ExceptionGroup('eg', [ValueError(1), ExceptionGroup('nested', [ValueError(6)])]) 535 | *OSError: ExceptionGroup('eg', [OSError(3), ExceptionGroup('nested', [OSError(4)])]) 536 | Traceback (most recent call last): 537 | File "", line 3, in 538 | ExceptionGroup 539 | ------------------------------------------------------------ 540 | Traceback (most recent call last): 541 | File "", line 12, in 542 | File "", line 3, in 543 | ExceptionGroup: eg 544 | ------------------------------------------------------------ 545 | ValueError: 1 546 | ------------------------------------------------------------ 547 | ExceptionGroup: nested 548 | ------------------------------------------------------------ 549 | ValueError: 6 550 | ------------------------------------------------------------ 551 | Traceback (most recent call last): 552 | File "", line 3, in 553 | ExceptionGroup: eg 554 | ------------------------------------------------------------ 555 | TypeError: 2 556 | ------------------------------------------------------------ 557 | OSError: 3 558 | ------------------------------------------------------------ 559 | ExceptionGroup: nested 560 | ------------------------------------------------------------ 561 | OSError: 4 562 | ------------------------------------------------------------ 563 | TypeError: 5 564 | >>> 565 | ``` 566 | 567 | ### Chaining 568 | 569 | Explicitly raised `ExceptionGroup`s are chained as with any exceptions. The 570 | following example shows how part of `ExceptionGroup` "one" became the 571 | context for `ExceptionGroup` "two", while the other part was combined with 572 | it into the new `ExceptionGroup`. 573 | 574 | ```python 575 | >>> try: 576 | ... try: 577 | ... raise ExceptionGroup("one", [ValueError('a'), TypeError('b')]) 578 | ... except *ValueError: 579 | ... raise ExceptionGroup("two", [KeyError('x'), KeyError('y')]) 580 | ... except BaseException as e: 581 | ... traceback.print_exception(e) 582 | ... 583 | Traceback (most recent call last): 584 | File "", line 3, in 585 | ExceptionGroup 586 | ------------------------------------------------------------ 587 | Traceback (most recent call last): 588 | File "", line 3, in 589 | ExceptionGroup: one 590 | ------------------------------------------------------------ 591 | ValueError: a 592 | 593 | During handling of the above exception, another exception occurred: 594 | 595 | Traceback (most recent call last): 596 | File "", line 5, in 597 | ExceptionGroup: two 598 | ------------------------------------------------------------ 599 | KeyError: 'x' 600 | ------------------------------------------------------------ 601 | KeyError: 'y' 602 | 603 | ------------------------------------------------------------ 604 | Traceback (most recent call last): 605 | File "", line 3, in 606 | ExceptionGroup: one 607 | ------------------------------------------------------------ 608 | TypeError: b 609 | ``` 610 | 611 | ### Raising New Exceptions 612 | 613 | In the previous examples the explicit raises were of the exceptions that 614 | were caught, so for completion we show a new exception being raise, with 615 | chaining: 616 | 617 | ```python 618 | >>> try: 619 | ... try: 620 | ... raise TypeError('bad type') 621 | ... except *TypeError as e: 622 | ... raise ValueError('bad value') from e 623 | ... except ExceptionGroup as e: 624 | ... traceback.print_exception(e) 625 | ... 626 | Traceback (most recent call last): 627 | File "", line 3, in 628 | ExceptionGroup 629 | ------------------------------------------------------------ 630 | ExceptionGroup 631 | ------------------------------------------------------------ 632 | Traceback (most recent call last): 633 | File "", line 3, in 634 | TypeError: bad type 635 | 636 | The above exception was the direct cause of the following exception: 637 | 638 | Traceback (most recent call last): 639 | File "", line 5, in 640 | ValueError: bad value 641 | >>> 642 | ``` 643 | 644 | Note that exceptions raised in one `except*` clause are not eligible to match 645 | other clauses from the same `try` statement: 646 | 647 | ```python 648 | >>> try: 649 | ... try: 650 | ... raise TypeError(1) 651 | ... except *TypeError: 652 | ... raise ValueError(2) # <- not caught in the next clause 653 | ... except *ValueError: 654 | ... print('never') 655 | ... except ExceptionGroup as e: 656 | ... traceback.print_exception(e) 657 | ... 658 | Traceback (most recent call last): 659 | File "", line 3, in 660 | ExceptionGroup 661 | ------------------------------------------------------------ 662 | ExceptionGroup 663 | ------------------------------------------------------------ 664 | Traceback (most recent call last): 665 | File "", line 3, in 666 | TypeError: 1 667 | 668 | During handling of the above exception, another exception occurred: 669 | 670 | Traceback (most recent call last): 671 | File "", line 5, in 672 | ValueError: 2 673 | ``` 674 | 675 | 676 | Raising a new instance of a naked exception does not cause this exception to 677 | be wrapped by an `ExceptionGroup`. Rather, the exception is raised as is, and 678 | if it needs to be combined with other propagated exceptions, it becomes a 679 | direct child of the new `ExceptionGroup` created for that: 680 | 681 | 682 | ```python 683 | >>> try: 684 | ... try: 685 | ... raise ExceptionGroup("eg", [ValueError('a')]) 686 | ... except *ValueError: 687 | ... raise KeyError('x') 688 | ... except BaseException as e: 689 | ... traceback.print_exception(e) 690 | ... 691 | Traceback (most recent call last): 692 | File "", line 3, in 693 | ExceptionGroup 694 | ------------------------------------------------------------ 695 | Traceback (most recent call last): 696 | File "", line 3, in 697 | ExceptionGroup: eg 698 | ------------------------------------------------------------ 699 | ValueError: a 700 | 701 | During handling of the above exception, another exception occurred: 702 | 703 | Traceback (most recent call last): 704 | File "", line 5, in 705 | KeyError: 'x' 706 | >>> 707 | >>> try: 708 | ... try: 709 | ... raise ExceptionGroup("eg", [ValueError('a'), TypeError('b')]) 710 | ... except *ValueError: 711 | ... raise KeyError('x') 712 | ... except BaseException as e: 713 | ... traceback.print_exception(e) 714 | ... 715 | Traceback (most recent call last): 716 | File "", line 3, in 717 | ExceptionGroup 718 | ------------------------------------------------------------ 719 | Traceback (most recent call last): 720 | File "", line 3, in 721 | ExceptionGroup: eg 722 | ------------------------------------------------------------ 723 | ValueError: a 724 | 725 | During handling of the above exception, another exception occurred: 726 | 727 | Traceback (most recent call last): 728 | File "", line 5, in 729 | KeyError: 'x' 730 | 731 | ------------------------------------------------------------ 732 | Traceback (most recent call last): 733 | File "", line 3, in 734 | ExceptionGroup: eg 735 | ------------------------------------------------------------ 736 | TypeError: b 737 | >>> 738 | ``` 739 | 740 | Finally, as an example of how the proposed API can help us work effectively 741 | with `ExceptionGroup`s, the following code ignores all `EPIPE` OS errors, 742 | while letting all other exceptions propagate. 743 | 744 | ```python 745 | try: 746 | low_level_os_operation() 747 | except *OSerror as errors: 748 | raise errors.subgroup(lambda e: e.errno != errno.EPIPE) from None 749 | ``` 750 | 751 | ### Caught Exception Objects 752 | 753 | It is important to point out that the `ExceptionGroup` bound to `e` is an 754 | ephemeral object. Raising it via `raise` or `raise e` will not cause changes 755 | to the overall shape of the `ExceptionGroup`. Any modifications to it will 756 | likely be lost: 757 | 758 | ```python 759 | >>> eg = ExceptionGroup("eg", [TypeError(12)]) 760 | >>> eg.foo = 'foo' 761 | >>> try: 762 | ... raise eg 763 | ... except *TypeError as e: 764 | ... e.foo = 'bar' 765 | ... # ^----------- `e` is an ephemeral object that might get 766 | >>> # destroyed after the `except*` clause. 767 | >>> eg.foo 768 | 'foo' 769 | >>> 770 | ``` 771 | 772 | ### Forbidden Combinations 773 | 774 | * It is not possible to use both traditional `except` blocks and the new 775 | `except*` clauses in the same `try` statement. The following example is a 776 | `SyntaxErorr`: 777 | 778 | ```python 779 | try: 780 | ... 781 | except ValueError: 782 | pass 783 | except *CancelledError: # <- SyntaxError: 784 | pass # combining `except` and `except*` is prohibited 785 | ``` 786 | 787 | * It is possible to catch the `ExceptionGroup` type with `except`, but not 788 | with `except*` because the latter is ambiguous: 789 | 790 | ```python 791 | try: 792 | ... 793 | except ExceptionGroup: # <- This works 794 | pass 795 | 796 | try: 797 | ... 798 | except *ExceptionGroup: # <- Runtime error 799 | pass 800 | ``` 801 | 802 | * An empty "match anything" `except*` block is not supported as its meaning may 803 | be confusing: 804 | 805 | ```python 806 | try: 807 | ... 808 | except*: # <- SyntaxError 809 | pass 810 | ``` 811 | 812 | * `continue`, `break`, and `return` are disallowed in `except*` clauses, 813 | causing a `SyntaxError`. 814 | 815 | This is because the exceptions in an `ExceptionGroup` are assumed to be 816 | independent, and the presence or absence of one of them should not impact 817 | handling of the others, as could happen if we allow an `except*` clause to 818 | change the way control flows through other clauses. We believe that this is 819 | error prone and there are clearer ways to implement a check like this: 820 | 821 | ```python 822 | def foo(): 823 | try: 824 | raise ExceptionGroup("msg", A(), B()) 825 | except *A: 826 | return 1 # <- SyntaxError 827 | except *B as e: 828 | raise TypeError("Can't have B without A!") 829 | ``` 830 | 831 | ## Backwards Compatibility 832 | 833 | Backwards compatibility was a requirement of our design, and the changes we 834 | propose in this PEP will not break any existing code: 835 | 836 | * The addition of a new builtin exception type `ExceptionGroup` does not impact 837 | existing programs. The way that existing exceptions are handled and displayed 838 | does not change in any way. 839 | 840 | * The behaviour of `except` is unchanged so existing code will continue to work. 841 | Programs will only be impacted by the changes proposed in this PEP once they 842 | begin to use `ExceptionGroup`s and `except*`. 843 | 844 | 845 | Once programs begin to use these features, there will be migration issues to 846 | consider: 847 | 848 | * An `except Exception:` clause will not catch `ExceptionGroup`s because they 849 | are derived from `BaseException`. Any such clause will need to be replaced 850 | by `except (Exception, ExceptionGroup):` or `except *Exception:`. 851 | 852 | * Similarly, any `except T:` clause that wraps code which is now potentially 853 | raising `ExceptionGroup` needs to become `except *T:`, and its body may need 854 | to be updated. 855 | 856 | * Libraries that need to support older Python versions will not be able to use 857 | `except*` or raise `ExceptionGroup`s. 858 | 859 | 860 | ## How to Teach This 861 | 862 | `ExceptionGroup`s and `except*` will be documented as part of the language 863 | standard. Libraries that raise `ExceptionGroup`s such as `asyncio` will need to 864 | specify this in their documentation and clarify which API calls need to be 865 | wrapped with `try-except*` rather than `try-except`. 866 | 867 | ## Reference Implementation 868 | 869 | We developed these concepts (and the examples for this PEP) with 870 | the help of [a reference implementation](https://github.com/iritkatriel/cpython/tree/exceptionGroup-stage5). 871 | 872 | It has the builtin `ExceptionGroup` along with the changes to the traceback 873 | formatting code, in addition to the grammar, compiler and interpreter changes 874 | required to support `except*`. 875 | 876 | Two opcodes were added: one implements the exception type match check via 877 | `ExceptionGroup.split()`, and the other is used at the end of a `try-except` 878 | construct to merge all unhandled, raised and reraised exceptions (if any). 879 | The raised/reraised exceptions are collected in a list on the runtime stack. 880 | For this purpose, the body of each `except*` clause is wrapped in a traditional 881 | `try-except` which captures any exceptions raised. Both raised and reraised 882 | exceptions are collected in the same list. When the time comes to merge them 883 | into a result, the raised and reraised exceptions are distinguished by comparing 884 | their metadata fields (context, cause, traceback) with those of the originally 885 | raised exception. As mentioned above, the reraised exceptions have the same 886 | metadata as the original, while the raised ones do not. 887 | 888 | ## Rejected Ideas 889 | 890 | ### The ExceptionGroup API 891 | 892 | We considered making `ExceptionGroup`s iterable, so that `list(eg)` would 893 | produce a flattened list of the leaf exceptions contained in the group. 894 | We decided that this would not be not be a sound API, because the metadata 895 | (cause, context and traceback) of the individual exceptions in a group are 896 | incomplete and this could create problems. If use cases arise where this 897 | can be helpful, we can document (or even provide in the standard library) 898 | a sound recipe for accessing an individual exception: use the `split()` 899 | method to create an `ExceptionGroup` for a single exception and then 900 | extract the contained exception with the correct metadata. 901 | 902 | ### Traceback Representation 903 | 904 | We considered options for adapting the traceback data structure to represent 905 | trees, but it became apparent that a traceback tree is not meaningful once 906 | separated from the exceptions it refers to. While a simple-path traceback can 907 | be attached to any exception by a `with_traceback()` call, it is hard to 908 | imagine a case where it makes sense to assign a traceback tree to an exception 909 | group. Furthermore, a useful display of the traceback includes information 910 | about the nested exceptions. For this reason we decided it is best to leave 911 | the traceback mechanism as it is and modify the traceback display code. 912 | 913 | ### A full redesign of `except` 914 | 915 | We considered introducing a new keyword (such as `catch`) which can be used 916 | to handle both naked exceptions and `ExceptionGroup`s. Its semantics would 917 | be the same as those of `except*` when catching an `ExceptionGroup`, but 918 | it would not wrap a naked exception to create an `ExceptionGroup`. This 919 | would have been part of a long term plan to replace `except` by `catch`, 920 | but we decided that deprecating `except` in favour of an enhanced keyword 921 | would be too confusing for users at this time, so it is more appropriate 922 | to introduce the `except*` syntax for `ExceptionGroup`s while `except` 923 | continues to be used for simple exceptions. 924 | 925 | ### Applying an `except*` clause on one exception at a time 926 | 927 | We considered making `except*` clauses always execute on a single exception, 928 | possibly executing the same clause multiple times when it matches multiple 929 | exceptions. We decided instead to execute each `except*` clause at most once, 930 | giving it an `ExceptionGroup` that contains all matching exceptions. The reason 931 | for this decision was the observation that when a program needs to know the 932 | particular context of an exception it is handling, it handles it before 933 | grouping it with other exceptions and raising them together. 934 | 935 | For example, `KeyError` is an exception that typically relates to a certain 936 | operation. Any recovery code would be local to the place where the error 937 | occurred, and would use the traditional `except`: 938 | 939 | ```python 940 | try: 941 | dct[key] 942 | except KeyError: 943 | # handle the exception 944 | ``` 945 | 946 | It is unlikely that asyncio users would want to do something like this: 947 | 948 | ```python 949 | try: 950 | async with asyncio.TaskGroup() as g: 951 | g.create_task(task1); g.create_task(task2) 952 | except *KeyError: 953 | # handling KeyError here is meaningless, there's 954 | # no context to do anything with it but to log it. 955 | ``` 956 | 957 | When a program handles a collection of exceptions that were aggregated into 958 | an exception group, it would not typically attempt to recover from any 959 | particular failed operation, but will rather use the types of the errors to 960 | determine how they should impact the program's control flow or what logging 961 | or cleanup is required. This decision is likely to be the same whether the group 962 | contains a single or multiple instances of something like a `KeyboardInterrupt` 963 | or `asyncio.CancelledError`. Therefore, it is more convenient to handle all 964 | exceptions matching an `except*` at once. If it does turn out to be necessary, 965 | the handler can inpect the `ExceptionGroup` and process the individual 966 | exceptions in it. 967 | 968 | ### Not matching naked exceptions in `except*` 969 | 970 | We considered the option of making `except *T` match only `ExceptionGroup`s 971 | that contain `T`s, but not naked `T`s. To see why we thought this would not be a 972 | desirable feature, return to the distinction in the previous paragraph between 973 | operation errors and control flow exceptions. If we don't know whether 974 | we should expect naked exceptions or `ExceptionGroup`s from the body of a 975 | `try` block, then we're not in the position of handling operation errors. 976 | Rather, we are likely calling some callback and will be handling errors to make 977 | control flow decisions. We are likely to do the same thing whether we catch a 978 | naked exception of type `T` or an `ExceptionGroup` with one or more `T`s. 979 | Therefore, the burden of having to explicitly handle both is not likely to have 980 | semantic benefit. 981 | 982 | If it does turn out to be necessary to make the distinction, it is always 983 | possible to nest in the `try-except*` clause an additional `try-except` clause 984 | which intercepts and handles a naked exception before the `except*` clause 985 | has a change to wrap it in an `ExceptionGroup`. In this case the overhead 986 | of specifying both is not additional burden - we really do need to write a 987 | separate code block to handle each case: 988 | 989 | ```python 990 | try: 991 | try: 992 | ... 993 | except SomeError: 994 | # handle the naked exception 995 | except *SomeError: 996 | # handle the ExceptionGroup 997 | ``` 998 | 999 | ### Allow mixing `except:` and `except*:` in the same `try` 1000 | 1001 | This option was rejected because it adds complexity without adding useful 1002 | semantics. Presumably the intention would be that an `except T:` block handles 1003 | only naked exceptions of type `T`, while `except *T:` handles `T` in 1004 | `ExceptionGroup`s. We already discussed above why this is unlikely 1005 | to be useful in practice, and if it is needed then the nested `try-except` 1006 | block can be used instead to achieve the same result. 1007 | 1008 | ### `try*` instead of `except*` 1009 | 1010 | Since either all or none of the clauses of a `try` construct are `except*`, 1011 | we considered changing the syntax of the `try` instead of all the `except*` 1012 | clauses. We rejected this because it would be less obvious. The fact that we 1013 | are handling `ExceptionGroup`s of `T` rather than only naked `T`s should be 1014 | specified in the same place where we state `T`. 1015 | 1016 | ## See Also 1017 | 1018 | * An analysis of how exception groups will likely be used in asyncio 1019 | programs: 1020 | https://github.com/python/exceptiongroups/issues/3#issuecomment-716203284 1021 | 1022 | * The issue where the `except*` concept was first formalized: 1023 | https://github.com/python/exceptiongroups/issues/4 1024 | 1025 | ## References 1026 | 1027 | * Reference implementation: 1028 | 1029 | Branch: https://github.com/iritkatriel/cpython/tree/exceptionGroup-stage5 1030 | 1031 | PR: https://github.com/iritkatriel/cpython/pull/10 1032 | 1033 | * PEP 3134: Exception Chaining and Embedded Tracebacks 1034 | 1035 | https://www.python.org/dev/peps/pep-3134/ 1036 | 1037 | * The `asyncio` standard library 1038 | 1039 | https://docs.python.org/3/library/asyncio.html 1040 | 1041 | `asyncio.gather()`: 1042 | https://docs.python.org/3/library/asyncio-task.html#asyncio.gather 1043 | 1044 | * The Trio Library 1045 | 1046 | Trio: https://trio.readthedocs.io/en/stable/ 1047 | 1048 | `MultiError`: 1049 | https://trio.readthedocs.io/en/stable/reference-core.html#trio.MultiError 1050 | 1051 | `MultiError2` design document: https://github.com/python-trio/trio/issues/611. 1052 | 1053 | * Python issue 29980: OSError: multiple exceptions should preserve the 1054 | exception type if it is common 1055 | 1056 | https://bugs.python.org/issue29980 1057 | 1058 | * Python issue 40857: `tempfile.TemporaryDirectory()`` context manager can fail 1059 | to propagate exceptions generated within its context 1060 | 1061 | https://bugs.python.org/issue40857 1062 | 1063 | * `atexit` documentation: 1064 | 1065 | https://docs.python.org/3/library/atexit.html#atexit.register 1066 | 1067 | * PyTest issue 8217: Improve reporting when multiple teardowns raise an exception 1068 | 1069 | https://github.com/pytest-dev/pytest/issues/8217 1070 | 1071 | * The Hypothesis Library 1072 | 1073 | https://hypothesis.readthedocs.io/en/latest/index.html 1074 | 1075 | Reporting Multiple Errors: 1076 | https://hypothesis.readthedocs.io/en/latest/settings.html#hypothesis.settings.report_multiple_bugs 1077 | 1078 | ## Copyright 1079 | 1080 | This document is placed in the public domain or under the 1081 | CC0-1.0-Universal license, whichever is more permissive. 1082 | --------------------------------------------------------------------------------