├── README.md └── scripts ├── ast_analysis.py └── download_packages.py /README.md: -------------------------------------------------------------------------------- 1 | # `return` in `finally` considered harmful 2 | 3 | Irit Katriel 4 | Nov 9 2024 5 | 6 | [Discussion link](https://discuss.python.org/t/an-analysis-of-return-in-finally-in-the-wild/70633). 7 | 8 | ### TL;DR 9 | 10 | The semantics of `return`, `break` and `continue` in a `finally` block are 11 | surprising. This document describes an analysis of their use in real world 12 | code, which was conducted in order to assess the cost and benefit of blocking 13 | these features in future versions of Python. The results show that 14 | 15 | 1. they are not used often. 16 | 2. when used, they are usually used incorrectly. 17 | 3. code authors are typically receptive and quick to fix the code when 18 | the error is pointed out to them. 19 | 20 | My conclusion is that it is both desireable and feasible to make this 21 | a `SyntaxWarning`, and later a `SyntaxError`. 22 | 23 | ## Introduction 24 | 25 | The semantics of `return`, `break` and `continue` are surprising for many 26 | developers. The [documentation](https://docs.python.org/3/tutorial/errors.html#defining-clean-up-actions) 27 | mentions that 28 | 29 | - If the finally clause executes a break, continue or return statement, exceptions are not re-raised. 30 | - If a finally clause includes a return statement, the returned value will be the one 31 | from the finally clause’s return statement, not the value from the try clause’s return statement. 32 | 33 | Both of these behaviours cause confusion, but the first is particularly dangerous 34 | because a swallowed exception is more likely to slip through testing, than an incorrect 35 | return value. 36 | 37 | In 2019, [PEP 601](https://peps.python.org/pep-0601/) proposed to change Python to emit a 38 | `SyntaxWarning` for a few releases and then turn it into a `SyntaxError`. The PEP was 39 | rejected in favour of viewing this as a programming style issue, to be handled by linters 40 | and [PEP8](https://peps.python.org/pep-0008/). 41 | Indeed, PEP8 now recommends not to used control flow statements in a `finally` block, and 42 | linters such as [pylint](https://pylint.readthedocs.io/en/stable/), 43 | [ruff](https://docs.astral.sh/ruff/) and 44 | [flake8-bugbear](https://github.com/PyCQA/flake8-bugbear) flag them as a problem. 45 | 46 | It is not disputed that `return`, `break` and `continue` in a `finally` clause 47 | should be avoided, the question is whether changing Python to forbid it now is worth 48 | the churn. What was missing at the time that PEP 601 was rejected, was an understanding 49 | of how this issue manifests in practice. Are people using it? How often are they 50 | using it incorrectly? How much churn would the proposed change create? 51 | 52 | The purpose here is to bridge that gap by evaluating real code (the 8000 most popular 53 | packages on PyPI) to answer these questions. I believe that the results show that PEP 601 54 | should now be implemented. 55 | 56 | ## Method 57 | 58 | The analysis is based on the 8000 most popular PyPI packages, in terms of number 59 | of downloads in the last 30 days. They were downloaded on the 17th-18th of 60 | October, using 61 | [a script](https://github.com/faster-cpython/tools/blob/main/scripts/download_packages.py) 62 | written by Guido van Rossum, which in turn relies on Hugo van Kemenade's 63 | [tool](https://hugovk.github.io/top-pypi-packages/) 64 | that creates a list of the most popular packages. 65 | 66 | Once downloaded, a [second script](https://github.com/iritkatriel/finally/blob/main/scripts/ast_analysis.py) 67 | was used to construct an AST for each file, and traverse it to identify `break`, 68 | `continue` and `return` statements which are directly inside a `finally` block. 69 | By *directly*, I mean that the script looks for things like 70 | 71 | ```python 72 | finally: 73 | return 42 / break / continue 74 | ``` 75 | 76 | and not situations where a `return` exits a function defined in the `finally` block: 77 | 78 | ```python 79 | finally: 80 | def f(): 81 | return 42 82 | ... 83 | ``` 84 | 85 | or a `break` or `continue` relates to a loop which is nested in the `finally` block: 86 | 87 | ```python 88 | finally: 89 | for i in ...: 90 | break / continue 91 | ``` 92 | 93 | I then found the current source code for each occurrence, and categorized it. For 94 | cases where the code seems incorrect, I created an issue in the project's bug 95 | tracker. The responses to these issues are also part of the data collected in 96 | this investigation. 97 | 98 | ## Results 99 | 100 | I decided not to include a list of the incorrect usages, out of concern that 101 | it would make this report look like a shaming exercise. Instead I will describe 102 | the results in general terms, but will mention that some of the problems I found 103 | appear in very popular libraries, including a cloud security application. 104 | For those so inclined, it should not be hard to replicate my analysis, as I 105 | provided links to the scripts I used in the Method section. 106 | 107 | The projects examined contained a total of 120,964,221 lines of Python code, 108 | and among them the script found 203 instances of control flow instructions in a 109 | `finally` block. Most were `return`, a handful were `break`, and none were 110 | `continue`. Of these: 111 | 112 | - 46 are correct, and appear in tests that target this pattern as a feature (e.g., 113 | tests for linters that detect it). 114 | - 8 seem like they could be correct - either intentionally swallowing exceptions 115 | or appearing where an active exception cannot occur. Despite being correct, it is 116 | not hard to rewrite them to avoid the bad pattern, and it would make the code 117 | clearer: deliberately swallowing exceptions can be more explicitly done with 118 | `except BaseException:`, and `return` which doesn't swallow exceptions can be 119 | moved after the `finally` block. 120 | - 149 were clearly incorrect, and can lead to unintended swallowing of exceptions. 121 | These are analyzed in the next section. 122 | 123 | ### The Error Cases 124 | 125 | Many of the error cases followed this pattern: 126 | 127 | ```python 128 | try: 129 | ... 130 | except SomeSpecificError: 131 | ... 132 | except Exception: 133 | logger.log(...) 134 | finally: 135 | return some_value 136 | ``` 137 | 138 | Code like this is obviously incorrect because it deliberately logs and swallows 139 | `Exception` subclasses, while silently swallowing `BaseExceptions`. The intention 140 | here is either to allow `BaseExceptions` to propagate on, or (if the author is 141 | unaware of the `BaseException` issue), to log and swallow all exceptions. However, 142 | even if the `except Exception` was changed to `except BaseException`, this code 143 | would still have the problem that the `finally` block swallows all exceptions 144 | raised from within the `except` block, and this is probably not the intention 145 | (if it is, that can be made explicit with another `try`-`except BaseException`). 146 | 147 | Another variation on the issue found in real code looks like this: 148 | 149 | ```python 150 | try: 151 | ... 152 | except: 153 | return NotImplemented 154 | finally: 155 | return some_value 156 | ``` 157 | 158 | Here the intention seems to be to return `NotImplemented` when an exception is 159 | raised, but the `return` in the `finally` block would override the one in the 160 | `except` block. 161 | 162 | > [!NOTE] 163 | > Following [the discussion](https://discuss.python.org/t/an-analysis-of-return-in-finally-in-the-wild/70633/15), 164 | > I repeated the analysis on a random selection of PyPI packages (to 165 | > analyze code written by *average* programmers). The sample contained 166 | > in total 77,398,892 lines of code with 316 instances of `return`/`break`/`continue` 167 | > in `finally`. So about 4 instances per million lines of code. 168 | 169 | ### Author reactions 170 | 171 | Of the 149 incorrect instances of `return` or `break` in a `finally` clause, 27 172 | were out of date, in the sense that they do not appear in the main/master branch 173 | of the library, as the code has been deleted or fixed by now. The remaining 122 174 | are in 73 different packages, and I created an issue in each one to alert the 175 | authors to the problems. Within two weeks, 40 of the 73 issues received a reaction 176 | from the code maintainers: 177 | 178 | - 15 issues had a PR opened to fix the problem. 179 | - 20 received reactions acknowledging the problem as one worth looking into. 180 | - 3 replied that the code is no longer maintained so this won't be fixed. 181 | - 2 closed the issue as "works as intended", one said that they intend to 182 | swallow all exceptions, but the other seemed unaware of the distinction 183 | between `Exception` and `BaseException`. 184 | 185 | One issue was linked to a pre-existing open issue about non-responsiveness to Ctrl-C, 186 | conjecturing a connection. 187 | 188 | Two of the issue were labelled as "good first issue". 189 | 190 | ### The correct usages 191 | 192 | The 8 cases where the feature appears to be used correctly (in non-test code) also 193 | deserve attention. These represent the "churn" that would be caused by blocking 194 | the feature, because this is where working code will need to change. I did not 195 | contact the authors in these cases, so we will need to assess the difficulty of 196 | making these changes ourselves. 197 | 198 | - In [mosaicml](https://github.com/mosaicml/composer/blob/694e72159cf026b838ba00333ddf413185b4fb4f/composer/cli/launcher.py#L590) 199 | there is a return in a finally at the end of the `main` function, after an `except:` 200 | clause which swallows all exceptions. The return in the finally would swallow 201 | an exception raised from within the `except:` clause, but this seems to be the 202 | intention. A possible fix would be to assign the return value to a variable in 203 | the `finally` clause, dedent the return statement and wrap the body of the `except:` 204 | clause by another `try`-`except` that would swallow exceptions from it. 205 | 206 | - In [webtest](https://github.com/Pylons/webtest/blob/617a2b823c60e8d7c5f6e12a220affbc72e09d7d/webtest/http.py#L131) 207 | there is a `finally` block that contains only `return False`. It could be replaced 208 | by 209 | 210 | ```python 211 | except BaseException: 212 | pass 213 | return False 214 | ``` 215 | 216 | - In [kivy](https://github.com/kivy/kivy/blob/3b744c7ed274c1a99bd013fc55a5191ebd8a6a40/kivy/uix/codeinput.py#L204) 217 | there is a `finally` that contains only a `return` statement. Since there is also a 218 | bare `except` just before it, in this case the fix will be to just remove the `finally:` block 219 | and dedent the `return` statement. 220 | 221 | - In [logilab-common](https://forge.extranet.logilab.fr/open-source/logilab-common/-/blob/branch/default/test/data/module.py?ref_type=heads#L60) 222 | there is, once again, a `finally` clause that can be replace by an `except BaseException` with 223 | the `return` dedented one level. 224 | 225 | - In [pluggy](https://github.com/pytest-dev/pluggy/blob/c760a77e17d512c3572d54d368fe6c6f9a7ac810/src/pluggy/_callers.py#L141) 226 | there is a lengthy `finally` with two `return` statements (the second on line 182). Here the 227 | return value can be assigned to a variable, and the `return` itself can appear after we've 228 | exited the `finally` clause. 229 | 230 | - In [aws-sam-cli](https://github.com/aws/aws-sam-cli/blob/97e63dcc2738529eded8eecfef4b875abc3a476f/samcli/local/apigw/local_apigw_service.py#L721) there is a conditional return at the end of the block. 231 | From reading the code, it seems that the condition only holds when the exception has 232 | been handled. The conditional block can just move outside of the `finally` 233 | block and achieve the same effect. 234 | 235 | - In [scrappy](https://github.com/scrapy/scrapy/blob/52c072640aa61884de05214cb1bdda07c2a87bef/scrapy/utils/gz.py#L27) 236 | there is a `finally` that contains only a `break` instruction. Assuming that it was the intention 237 | to swallow all exceptions, it can be replaced by 238 | 239 | ```python 240 | except BaseException: 241 | pass 242 | break 243 | 244 | ``` 245 | 246 | ## Discussion 247 | 248 | The first thing to note is that `return`/`break`/`continue` in a `finally` 249 | block is not something we see often: 203 instance in over 120 million lines 250 | of code. This is, possibly, thanks to the linters that warn about this. 251 | 252 | The second observation is that most of the usages were incorrect: 73% in our 253 | sample (149 of 203). 254 | 255 | Finally, the author responses were overwhelmingly positive. Of the 40 responses 256 | received within two weeks, 35 acknowledged the issue, 15 of which also created 257 | a PR to fix it. Only two thought that the code is fine as it is, and three 258 | stated that the code is no longer maintained so they will not look into it. 259 | 260 | The 8 instances where the code seems to work as intended, are not hard to 261 | rewrite. 262 | 263 | ## Conclusion 264 | 265 | The results indicate that `return`, `break` and `continue` in a finally block 266 | 267 | - are rarely used. 268 | - when they are used, they are usually used incorrectly. 269 | - code authors are receptive to changing their code, and tend to find it easy to do. 270 | 271 | This indicates that it is both desireable and feasible to change Python to emit 272 | a `SyntaxWarning`, and in a few years a `SyntaxError` for these patterns. 273 | 274 | ## Acknowledgements 275 | 276 | I thank: 277 | 278 | - Alyssa Coghlan for 279 | [bringing this issue my attention](https://discuss.python.org/t/pep-760-no-more-bare-excepts/67182/97). 280 | 281 | - Guido van Rossum and Hugo van Kemenade for the 282 | [script](https://github.com/faster-cpython/tools/blob/main/scripts/download_packages.py) 283 | that downloads the most popular PyPI packages. 284 | 285 | - The many code authors I contacted for their responsiveness and grace. 286 | 287 | 288 | © 2024 Irit Katriel 289 | -------------------------------------------------------------------------------- /scripts/ast_analysis.py: -------------------------------------------------------------------------------- 1 | 2 | import argparse 3 | import ast 4 | import glob 5 | import os 6 | import sys 7 | import tarfile 8 | import warnings 9 | 10 | 11 | FINALLY = 'F' 12 | LOOP = 'L' 13 | DEF = 'D' 14 | 15 | class Visitor(ast.NodeVisitor): 16 | def __init__(self, source, filename, findings): 17 | self.source = source 18 | self.filename = filename 19 | self.findings = findings 20 | self.state = [] 21 | 22 | def do_Try(self, node): 23 | for n in node.body: 24 | self.visit(n) 25 | for n in node.handlers: 26 | self.visit(n) 27 | for n in node.orelse: 28 | self.visit(n) 29 | self.state.append(FINALLY) 30 | for n in node.finalbody: 31 | self.visit(n) 32 | self.state.pop() 33 | 34 | def do_Loop(self, node): 35 | self.state.append(LOOP) 36 | self.generic_visit(node) 37 | self.state.pop() 38 | 39 | def do_forbidden(self, node, good_state=None): 40 | assert good_state is not None 41 | if not self.state: 42 | return 43 | state = None 44 | for i in range(len(self.state), 0, -1): 45 | if self.state[i-1] in (FINALLY, good_state): 46 | state = self.state[i-1] 47 | break 48 | if state == FINALLY: 49 | self.findings.append([self.filename, self.source, node.lineno]) 50 | print(self.filename, node.lineno) 51 | 52 | def do_FunctionDef(self, node): 53 | self.state.append(DEF) 54 | self.generic_visit(node) 55 | self.state.pop() 56 | 57 | def visit_FunctionDef(self, node): 58 | self.do_FunctionDef(node) 59 | 60 | def visit_Try(self, node): 61 | self.do_Try(node) 62 | 63 | def visit_TryStar(self, node): 64 | self.do_Try(node) 65 | 66 | def visit_For(self, node): 67 | self.do_Loop(node) 68 | 69 | def visit_While(self, node): 70 | self.do_Loop(node) 71 | 72 | def visit_Break(self, node): 73 | self.do_forbidden(node, good_state=LOOP) 74 | 75 | def visit_Continue(self, node): 76 | self.do_forbidden(node, good_state=LOOP) 77 | 78 | def visit_Return(self, node): 79 | self.do_forbidden(node, good_state=DEF) 80 | 81 | 82 | class Reporter: 83 | 84 | def __init__(self): 85 | self.findings = [] 86 | self.lines = 0 87 | 88 | def report(self, source, filename, verbose): 89 | try: 90 | a = ast.parse(source) 91 | except (SyntaxError, RecursionError) as e: 92 | # print(f'>>> {type(e)} in ast.parse() for {filename}') 93 | return 94 | 95 | old = len(self.findings) 96 | Visitor(source, filename, self.findings).visit(a) 97 | self.lines += len(source.split(b'\n')) 98 | new = len(self.findings) 99 | if new > old: 100 | self.print_stats() 101 | 102 | def print_stats(self): 103 | lines = self.lines 104 | results = len(self.findings) 105 | in_site_packages = len([x for x in self.findings if 'site-packages' in x[0]]) 106 | print(f"{lines=} {results=} {in_site_packages=}") 107 | 108 | def file_report(self, filename, verbose): 109 | try: 110 | with open(filename, "rb") as f: 111 | source = f.read() 112 | self.report(source, filename, verbose) 113 | except Exception as err: 114 | if verbose > 0: 115 | print(filename + ":", err) 116 | 117 | def tarball_report(self, filename, verbose): 118 | if verbose > 1: 119 | print(f"\nExamining tarball {filename}") 120 | with tarfile.open(filename, "r") as tar: 121 | members = tar.getmembers() 122 | for m in members: 123 | info = m.get_info() 124 | name = info["name"] 125 | if name.endswith(".py"): 126 | try: 127 | source = tar.extractfile(m).read() 128 | except Exception as err: 129 | if verbose > 0: 130 | print(f"{name}: {err}") 131 | else: 132 | self.report(source, name, verbose-1) 133 | 134 | def expand_globs(filenames): 135 | for filename in filenames: 136 | if "*" in filename and sys.platform == "win32": 137 | for fn in glob.glob(filename): 138 | yield fn 139 | else: 140 | yield filename 141 | 142 | argparser = argparse.ArgumentParser() 143 | argparser.add_argument("-q", "--quiet", action="store_true", 144 | help="less verbose output") 145 | argparser.add_argument("-v", "--verbose", action="store_true", 146 | help="more verbose output") 147 | argparser.add_argument("filenames", nargs="*", metavar="FILE", 148 | help="files, directories or tarballs to count") 149 | 150 | 151 | def main(): 152 | args = argparser.parse_args() 153 | verbose = 1 + args.verbose - args.quiet 154 | filenames = args.filenames 155 | if not filenames: 156 | argparser.print_usage() 157 | sys.exit(0) 158 | 159 | if verbose < 2: 160 | warnings.filterwarnings("ignore", "", SyntaxWarning) 161 | 162 | if verbose >= 2: 163 | print("Looking for", ", ".join(OF_INTEREST_NAMES)) 164 | print("In", filenames) 165 | 166 | reporter = Reporter() 167 | 168 | for filename in expand_globs(filenames): 169 | if os.path.isfile(filename): 170 | if filename.endswith(".tar.gz"): 171 | try: 172 | reporter.tarball_report(filename, verbose) 173 | except tarfile.ReadError: 174 | pass 175 | else: 176 | reporter.file_report(filename, verbose) 177 | elif os.path.isdir(filename): 178 | for root, dirs, files in os.walk(filename): 179 | for file in files: 180 | if file.endswith(".py"): 181 | full = os.path.join(root, file) 182 | reporter.file_report(full, verbose) 183 | else: 184 | print(f"{filename}: Cannot open") 185 | 186 | print("-----------------------------------------\nIn Total:") 187 | reporter.print_stats() 188 | 189 | if __name__ == "__main__": 190 | main() 191 | -------------------------------------------------------------------------------- /scripts/download_packages.py: -------------------------------------------------------------------------------- 1 | 2 | # This script was copied from https://github.com/faster-cpython/tools/blob/main/scripts/download_packages.py 3 | 4 | # Download N most popular PyPI packages 5 | 6 | 7 | import json 8 | import os 9 | import argparse 10 | 11 | import requests 12 | 13 | TOP_PYPI_PACKAGES = ( 14 | "https://hugovk.github.io/top-pypi-packages/top-pypi-packages-30-days.json" 15 | ) 16 | 17 | PYPI_INFO = "https://pypi.python.org/pypi/{}/json" 18 | 19 | 20 | def dl_data(url): 21 | response = requests.get(url) 22 | response.raise_for_status() 23 | return response.content 24 | 25 | 26 | def dl_json(url): 27 | response = requests.get(url) 28 | response.raise_for_status() 29 | return response.json() 30 | 31 | 32 | def dl_package_info(package): 33 | return dl_json(PYPI_INFO.format(package)) 34 | 35 | 36 | parser = argparse.ArgumentParser() 37 | parser.add_argument( 38 | "-n", "--number", type=int, default=100, help="How many packages (default 100)" 39 | ) 40 | parser.add_argument( 41 | "-o", "--odir", default="packages", help="Where to download (default ./packages)" 42 | ) 43 | parser.add_argument( 44 | "-t", 45 | "--top-packages", 46 | default=TOP_PYPI_PACKAGES, 47 | help=f"URL for 'top PYPI packages (default {TOP_PYPI_PACKAGES})", 48 | ) 49 | 50 | 51 | def main(): 52 | args = parser.parse_args() 53 | os.makedirs(args.odir, exist_ok=True) 54 | packages = dl_json(args.top_packages) 55 | print("Last update:", packages["last_update"]) 56 | rows = packages["rows"] 57 | # Sort from high to low download count 58 | rows.sort(key=lambda row: -row["download_count"]) 59 | # Limit to top N packages 60 | rows = rows[: args.number] 61 | print(f"Downloading {len(rows)} packages...") 62 | index = 0 63 | count = 0 64 | skipped = 0 65 | missing = 0 66 | try: 67 | for row in rows: 68 | print( 69 | f"Project {row['project']}" 70 | f" was downloaded {row['download_count']:,d} times" 71 | ) 72 | index += 1 73 | info = dl_package_info(row["project"]) 74 | releases = info["releases"] 75 | # Assume the releases are listed in chronological order 76 | print(f'-- {index}') 77 | last_release = list(releases)[-1] 78 | print(f" Last release: {last_release}") 79 | files = releases[last_release] 80 | for file in files: 81 | filename = file["filename"] 82 | # Download the sdist, which is the .tar.gz filename 83 | if filename.endswith(".tar.gz"): 84 | print(f" File name: {filename}") 85 | dest = os.path.basename(filename) 86 | fulldest = os.path.join(args.odir, dest) 87 | if not os.path.exists(fulldest): 88 | url = file["url"] 89 | print(f" URL: {url}") 90 | data = dl_data(url) 91 | print(f" Writing {len(data)} bytes to {fulldest} ") 92 | with open(fulldest, "wb") as f: 93 | f.write(data) 94 | count += 1 95 | else: 96 | print(f" Skipping {fulldest} (already exists)") 97 | skipped += 1 98 | break 99 | else: 100 | missing += 1 101 | except KeyboardInterrupt: 102 | print(f"Interrupted at index {index}") 103 | finally: 104 | print( 105 | f"Out of {len(rows)} packages:" 106 | f" downloaded {count}, skipped {skipped}, missed {missing}" 107 | ) 108 | 109 | 110 | if __name__ == "__main__": 111 | main() 112 | --------------------------------------------------------------------------------