├── .gitignore ├── assets └── cyclomatic-complexity.png ├── todos.md └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | *env* 2 | .vscode -------------------------------------------------------------------------------- /assets/cyclomatic-complexity.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/phitoduck/python-software-development-course/HEAD/assets/cyclomatic-complexity.png -------------------------------------------------------------------------------- /todos.md: -------------------------------------------------------------------------------- 1 | Radon 2 | 3 | - Quiz questions to calculate code complexity for certain blocks 4 | 5 | ```bash 6 | radon cc cc.py --show-complexity 7 | 8 | cc.py 9 | F 1:0 report_driving_status_for_age - A (5) 10 | F 13:0 report_driving_and_drinking_status_for_age - A (4) 11 | ``` 12 | 13 | ```python 14 | def report_driving_status_for_age(age: int) -> None: 15 | if age < 0: 16 | print("You are not born yet.") 17 | elif age <= 15: 18 | print("You may be old enough to get a drivers' permit in some U.S. states.") 19 | elif age <= 17: 20 | print("You may be old enough to get a drivers' license in most U.S. states.") 21 | elif age <= 18: 22 | print("You may be old enough to get a drivers permit in all U.S. states.") 23 | else: 24 | print("You are old enough to get a drivers permit in all U.S. states.") 25 | ``` 26 | 27 | ```python 28 | def report_driving_and_drinking_status_for_age(age: int) -> None: 29 | if age >= 16 and age < 21: 30 | print("You are old enough to drive but not old enough to drink.") 31 | elif age >= 21: 32 | print("You are old enough to drive and old enough to drink.") 33 | else: 34 | print("You are not old enough to drive or drink.") 35 | ``` 36 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # Course Index - Taking Python to Production: A Professional Onboarding Guide 3 | 4 | > ⚠️ Disclaimer: This document is a work in progress. It is not yet complete, but will be before the course moved out of beta. ⚠️ 5 | 6 | > ⚠️ Code examples including config files and CLI tools may go out of date because tools 7 | > change often. Check the documentation for the tool you are using if you run into problems. 8 | > You should really get into the habit of referring to offical docs anyway! 9 | 10 | 11 | ## Table of Contents 12 | 13 | - [Unit 8 - Using automation to make continuous integration (the code review part) easier](#unit-8---using-automation-to-make-continuous-integration-the-code-review-part-easier) 14 | - [Pylint Part 3 - Configuring `pylint`](#pylint-part-3---configuring-pylint) 15 | - [Flake8](#flake8) 16 | - [Flake8 Plugin example: Darglint](#flake8-plugin-example-darglint) 17 | - [isort](#isort) 18 | - [Code Complexity: Radon, Xenon, McCabe](#code-complexity-radon-xenon-mccabe) 19 | - [Typing Part 1 - type hints, autocompletion, static vs dynamic type checking, and interpreted vs compiled](#typing-part-1---type-hints-autocompletion-static-vs-dynamic-type-checking-and-interpreted-vs-compiled) 20 | - [Typing Part 2 - mypy and basic typing](#typing-part-2---mypy-and-basic-typing) 21 | - [Typing Part 3 - Simple ("primitive" or "builtin") types and Complex ("generic") types](#typing-part-3---simple-primitive-or-builtin-types-and-complex-generic-types) 22 | - [Typing Part 4 - Union and Optional](#typing-part-4---union-and-optional) 23 | 24 | 25 | 26 | ## Unit 8 - Using automation to make continuous integration (the code review part) easier 27 | 28 | ### Pylint Part 3 - Configuring `pylint` 29 | 30 | 31 | 32 | #### Notes 33 | 34 | - List of all Pylint rules **link** 35 | - "A foolish consistency is the hobgoblin of little minds." -PEP8 Every good rule has an exception. Follow linting rules in general, and take time to understand what they are trying to achieve. "Pragmatism is the rule." The beauty of Python is that you *can* "break rules" when you have a valid reason to. 36 | - Ignore comments like `# pylint: disable=` statements, e.g. `pylint: disable=too-many-attributes` silences a single occurence of a Pylint error. 37 | - The fact that these go in your code makes it easy for code reviewers to see and therefore discuss why an exception was warranted. Additional comments *may* be desirable to preempt even those sorts of conversations. 38 | - Prefer using the human-readable "symbolic message" e.g. `too-many-attributes` over using the cryptic, error codes, e.g. `C0115` to save reviewers from having to look them up. 39 | - Most linters support ignore comments, though the exact syntax varies 40 | 41 | - You can disable rules for an entire project, too. Some pylint issues may not provide enough value to justify having developers spend time satisfying them. 42 | - Method 1: Pass arguments to `pylint` 43 | - `pylint --disable=too-many-attributes,missing-class-docstring ` disable particular errors 44 | - `pylint --disable=C ` disables all "convention violations" (error codes that start with `C`) 45 | - `pylint --enable=E ` only enable errors 46 | - Method 2: A config file (`.pylintrc`) 47 | - You can generate a starter file with `pylint --generate-rcfile > .pylintrc` 48 | - `pylint` will pick up this config file is in the current working directory 49 | - Explicitly point `pylint` at a config file with `pylint --rcfile path/to/.pylintrc` 50 | - Teams should have a standard config file for any linter they use, including `pylint` if they choose to use it. 51 | - Avoid disabling too many rules. Try to use the defaults. This avoids "Holy War" arguments and encourages developers to learn best practices. Let tools be opinionated for you. But be pragmatic. 52 | - There is no "magic" `pylint` ruleset. It's up for you or your team to decide. Start by enabling *every* `pylint` rule and taking the time to learn what they mean as they come up. You can incrementally remove rules if you or your team agree they are unreasonable. But you should always understand the *why* behind the rule before you do so. Books like *Clean Code* and *Refactoring* will go a long way toward that. 53 | 54 | 55 | #### Assignments 56 | 57 | 58 | ##### Generate a `.pylintrc` file 59 | 60 | 1. Generate the file 61 | 2. Validate that `pylint` respects your file by disabling/enabling a specific rule 62 | and running it on some code you know has that violation 63 | 64 | 65 | ##### Configure VS Code for `pylint` 66 | 67 | 1. Install the `pylint` extension by Microsoft if you have not already done so. 68 | 2. Configure your `.vscode/settings.json` 69 | 70 | ```jsonc 71 | { 72 | // ... 73 | "pylint.args": ["--rcfile=${workspaceFolder}/.pylintrc"], 74 | } 75 | ``` 76 | 77 | 78 | ##### Start creating your own `pylint` ruleset 79 | 80 | Apply `pylint` to some code and refactor *every* violation. 81 | 82 | This will be slow initially, but you will learn a lot. You will find that you remember the rules and in the future, you will naturally write code that does not have those violations. 83 | 84 | ### Flake8 85 | 86 | 87 | #### Notes 88 | 89 | - Pylint is not the only Python linter 90 | - Pylint and Flake8 catch many of the same things, but each catch some the other does not so many projects and teams choose to run both 91 | - Pylint supports plugins and there are *many*. Here is an open-source index of them **Link to awesome flake8 plugins** 92 | - `pip install flake8` 93 | 94 | Configuration options 95 | 96 | Use the Flake8 support within the Python extension: 97 | 98 | ```jsonc 99 | // .vscode/settings.json 100 | 101 | { 102 | "python.linting.flake8Enabled": true 103 | } 104 | ``` 105 | 106 | OR use the dedicated Flake8 extension. Installing the extension is enough to enable Flake8 hints in VS Code. 107 | 108 | - Disable comments: 109 | - `# noqa: E` ignore all errors 110 | - `# noqa: E305` ignore a specific error 111 | - Flake8 doesn't have a human-readable "symbolic message" like Pylint does! Very sad. 112 | - The standard Flake8 config file name is `.flake8` 113 | - `.flake8` uses the INI file format which has some major downsides over formats like TOML, YAML, JSON, etc. 114 | - There are plugins, e.g. `Flake8-pyproject` that enable TOML support for `flake8`. Pylint supports TOML natively. A future section will explore configuring all of our code quality tools with TOML. 115 | 116 | Example `.flake8` config file 117 | 118 | ```ini 119 | # .flake8 120 | 121 | [flake8] 122 | extend-ignore = 123 | E305 124 | ``` 125 | 126 | - `flake8 --config path/to/.flake8 ` directs Flake8 at this file 127 | 128 | 129 | #### Assignments 130 | 131 | 132 | ##### Configure VS Code for Flake8 133 | 134 | 1. Install the `Flake8` extension by Microsoft if you have not already done so. 135 | 2. Configure your `.vscode/settings.json` 136 | 137 | ```jsonc 138 | // .vscode/settings.json 139 | 140 | { 141 | // ... 142 | "flake8.args": ["--config=${workspaceFolder}/.flake8"] 143 | } 144 | ``` 145 | 146 | ### Flake8 Plugin example: Darglint 147 | 148 | 149 | #### Notes 150 | 151 | - You can find lots of Flake8 plugins here **Link to awesome list** 152 | - Installing plugins is easy. You just `pip install ...` the plugin and you are done! 153 | - Flake8 Plugins may: 154 | - Add rules that Flake8 looks for. Since Flake8 integrates with VS Code, installing standalone tools as Flake8 plugins allows you to surface the rules of that tool in the VS Code `Problems` tab and get squiggly lines from that tool. For example, 155 | - `darglint` adds rules that make sure function arguments are documented in docstrings. `darglint` error codes start with `DAR`. It's common for tools to have a prefix for their rules. 156 | - `flake8-docstrings` is a wrapper around `pydocstrings` that adds rules that ensure docstrings 157 | follow a consistent format. 158 | - `flake8-spellcheck` adds spelling violation rules 159 | - Add command line arguments to `flake8` that are not available otherwise. For example, `Flake8-Pyproject` adds a `--toml-config` argument which allows you to point Flake8 at a file such as `pyproject.toml` or `flake8.toml`. TOML is superior to the INI format. 160 | - Add config options for the `.flake8` config file that would not otherwise do anything. 161 | - Some tools, like one called `radon`, do not support `# noqa` statements, but since Flake8 does, 162 | if you install a tool as a Flake8 statement, you can take advantage of Flake8's `# noqa: ` 163 | comment engine to silence errors for that tool! 164 | 165 | ```ini 166 | # .flake8 167 | 168 | [flake8] 169 | # config option only available when darglint is installed 170 | docstring_style=google 171 | ``` 172 | 173 | 174 | #### Assignment 175 | 176 | This is completely optional. If this assignment does not seem interesting, skip ahead. 177 | 178 | 1. Install `darglint` with `pip install darglint`. This automatically registers `darglint` as a Flake8 plugin. 179 | 2. Run `darglint` on some of your code and see if it catches any docstring errors. 180 | 3. Add an ignore statement for a `DAR` error in the `.flake8` config file to see that Flake8 respects Darglint's rules. 181 | 182 | ### isort 183 | 184 | 185 | #### Notes 186 | 187 | - `isort` is a standalone CLI tool for organizing imports 188 | - the benefits of sorting imports include: 189 | - easier to find imports when there are many 190 | - easier to distinguish which types of imports are in a file (3rd party, 1st party, standard library, etc.) 191 | - reduces merge conflicts if imports are made to span multiple lines (`isort --fgw `) 192 | - find the `isort` [config file reference here](https://pycqa.github.io/isort/docs/configuration/options.html#force-grid-wrap) 193 | - `isort` is a code formatter, and so is `black`. `isort` might make some changes that `black` would undo. For example, if `isort` and `black` are configured with different line lengths, `isort` might put imports on multiple lines only to have `black` put them back. Manually specifying `--line-length` or `--profile=black` on `isort` to agree with black will fix this problem. 194 | 195 | Example `.isort.cfg` file: 196 | 197 | ```ini 198 | # .isort.cfg 199 | [settings] 200 | multi_line_output = VERTICAL_HANGING_INDENT 201 | force_grid_wrap = 2 202 | line_length = 99 203 | profile = black 204 | ``` 205 | 206 | 207 | #### Assignments 208 | 209 | 210 | ##### Configure VS Code for isort 211 | 212 | 1. Install the `isort` extension by Microsoft if you have not already done so. 213 | 2. Write an `.isort.cfg` file 214 | 3. Configure your `.vscode/settings.json` 215 | 216 | Example `.isort.cfg` file: 217 | 218 | ```jsonc 219 | // .vscode/settings.json 220 | 221 | { 222 | // ... 223 | "isort.args": ["--settings=${workspaceFolder}/.isort.cfg"], 224 | "[python]": { 225 | "editor.codeActionsOnSave": { 226 | "source.organizeImports": true 227 | }, 228 | // enable black formatting 229 | "editor.formatOnSave": true, 230 | "editor.defaultFormatter": "ms-python.black-formatter", 231 | }, 232 | } 233 | ``` 234 | 235 | 4. Validate that `isort` respects your file by running it on some code you know has that violation 236 | 237 | ### Code Complexity: Radon, Xenon, McCabe 238 | 239 | 240 | #### Notes 241 | 242 | - Clean code is secure, performant, and maintainable. Maintainable might be the most important. If code is easy to change, you can iterate to the others. *Readability* is key for code being easy to change. You should keep the complexity of your code low to help it be readable. 243 | - Radon and McCabe have code complexity metrics like cyclomatic complexity, maintainability index, etc. 244 | - [This page is a great reference](https://radon.readthedocs.io/en/latest/intro.html) for how Radon calculates these code complexity metrics. 245 | ![](./assets/cyclomatic-complexity.png) 246 | - These metrics can be used to alert you if code is getting too complex. 247 | - You can use these tools to set a complexity threshold. 248 | - Metrics can be "gamed". Just because you bring code beneath Radon's threshold does not mean it is readable. Metrics are just a tool. As always, pragmatism is the rule. Use your judgement. 249 | - Radon is a standalone CLI tool, but it is also a Flake8 plugin. To install it, just run `pip install radon` and it will auto-register with Flake8. Then you can use `# noqa: ` in exceptional cases to disable particular block of code that violates your complexity threshold. 250 | - Usage: 251 | - `pip install radon` 252 | - `radon cc --show-complexity ` calculates the cyclomatic complexity, specifically 253 | - `radon raw ` 254 | - You can add radon configuration settings in the `.flake8` file: 255 | 256 | ```ini 257 | # .flake8 258 | 259 | [settings] 260 | # vanilla flake8 settings 261 | extend-ignore = 262 | E305 263 | max-line-length = 99 264 | 265 | # radon settings 266 | radon-max-cc = 10 267 | ``` 268 | 269 | 270 | #### Assignments 271 | 272 | 273 | ##### Configure VS Code for Radon 274 | 275 | 1. Install radon with `pip install radon` 276 | 2. Write some code and measure its cyclomatic complexity 277 | - > Note: I have had the best luck with this when I write the logic **inside of functions** rather than writing the code in the global scope/namespace. 278 | 3. Install the Flake8 for VS Code if you have not already, and `pip install flake8 radon` in the same virtual environment, pointed at by VS Code. 279 | 4. Verify that you see red squiggly lines for functions that have high complexity 280 | 5. Write a `.flake8` file that configures Radon with a max complexity threshold 281 | 6. Validate your configuration file by writing code that gets a red squiggly line to appear when you go over your max CC threshold 282 | 283 | ### Typing Part 1 - type hints, autocompletion, static vs dynamic type checking, and interpreted vs compiled 284 | 285 | 286 | #### Notes 287 | 288 | - Python 3.5 introduced type hints to the language with syntax like 289 | - `num: float = 10.0` or 290 | - `def foo(a: int, b: str) -> bool:` 291 | - Type hints enable your IDE to give you autocompletion (putting a `.` after a variable and seeing a list of methods/properties that are available on that variable) 292 | - Popular libraries that were ported from Python 2 (or <3.5) struggled to apply type hints (Pandas, SQLAlchemy, Numpy, Flask, etc.). 293 | When Python 3.5 came out, many popular libraries were displaced by new libraries that used the statement "we have autocompletion all the way down" as a selling point. FastAPI and Typer are examples of this. 294 | - There is research indicating that adding type information to your code prevents a large number of bugs from ever occuring. 295 | - Lack of autocompletion in a codebase adds significant cognitive load for maintainers and customers. 296 | - Developers can work around this by "make round trips" to documentation sites as they reference the docs, but this slows down development. 297 | 298 | 299 | #### Assignments 300 | 301 | No assignments. 302 | 303 | ### Typing Part 2 - mypy and basic typing 304 | 305 | 306 | #### Notes 307 | 308 | - [PEP 484](https://peps.python.org/pep-0484/) is the design document that introduced type hints to Python 309 | - It assures Python developers that type annotations will never be mandatory, even by convention 310 | - Applying thorough type hints to your code usually requires you to 311 | structure your code in a way that is conducive to typing. This usually makes your code more readable. However, 312 | sometimes it forces you to complicate your code or prevents you from using "magic" that is not typeable. 313 | Developers should always have the option not to use type hints if there is a *compelling* reason not to. 314 | - `mypy` is a static type checker for Python. 315 | - It is a CLI tool that you can run against code to find type violations. E.g. `num: float = "I'm not a float!"` would be a type violation. 316 | - There is also a VS Code extension for `mypy` that adds red squiggly lines for typing violations. 317 | - `mypy` is considered Python's official "reference implementation" for a type checker: 318 | - Facebook wrote `wasabi` 319 | - Google wrote `pyre` 320 | - Microsoft wrote `pyright`, which is the type checker that powers VS Code's Python type checking in the Pylance (which is bundled with the official "Python" extension by Microsoft). 321 | 322 | 323 | #### Assignments 324 | 325 | 1. Install `mypy` with `pip install mypy`, ideally in a virtual environment 326 | 2. Run `mypy` against some code that has type violations. For example, 327 | 328 | ```python 329 | # file.py 330 | 331 | num: float = "I'm not a float!"` 332 | ``` 333 | 334 | 3. Install the `mypy` extension for VS Code if you have not already 335 | 4. Validate your installation of of the VS Code extension by viewing a red squiggly line in the file from step (2). 336 | 337 | ### Typing Part 3 - Simple ("primitive" or "builtin") types and Complex ("generic") types 338 | 339 | 340 | #### Notes 341 | 342 | - This video covers [the first typing section of the mypy docs](https://mypy.readthedocs.io/en/stable/builtin_types.html) 343 | - The video is commentary on this page. If that does not sound valuable to you, you could read the documentation and skip ahead. 344 | - We look at some basic types, i.e. `int`, `str`, `float`, `bytes`, `bool`, `Any` 345 | - And some complex types, e.g. `Dict[str, int]`, `Dict[str, Set[int]]` 346 | - "Broad" vs "Narrow" types, e.g. 347 | - `Iterable` vs `Set`, `List`, `Tuple`, etc. 348 | - `Sequence` vs `List`, `Tuple` 349 | - `Mapping` vs `Dict` 350 | - `Any` is the most broad type including all other types 351 | - Make your type annotations as narrow as possible to get the best autocompletion 352 | - Avoid using the `Any` type as much as you possible can 353 | - `Type` is used like this: 354 | 355 | ```python 356 | class Animal: 357 | ... 358 | 359 | # the entire class is type-hinted like this 360 | my_class: Type[Animal] = Animal 361 | 362 | # an instance of the class is type-hinted like this 363 | squirrel: Animal = Animal() 364 | ``` 365 | 366 | 367 | #### Assignments 368 | 369 | Read [this page](https://mypy.readthedocs.io/en/stable/builtin_types.html) of the mypy documentation. 370 | 371 | ### Typing Part 4 - Union and Optional 372 | 373 | 374 | #### Notes 375 | 376 | - "Type inference" is a process that static analyzers use to make guesses about which type a variable *should* be based on how it is used. Pylance does type inference. To see Pylance's guess at what type your variables are, hover over that variable with your cursor. 377 | - Type inference is not perfect. Hand-written type annotations are better. 378 | 379 | ```python 380 | floats = 1.0, 2.0, 3.0, 381 | ``` 382 | 383 | Pylance's type inference for this is 384 | 385 | ```python 386 | from typing import Literal, Tuple 387 | 388 | floats: Tuple[Literal[1.0], Literal[2.0], Literal[3.0]] 389 | ``` 390 | 391 | But this is not a useful type annotation. It would likely be better to annotate it like this 392 | 393 | ```python 394 | # assuming the tuple is of arbitrary length 395 | floats: Tuple[float, ...] 396 | 397 | # OR 398 | 399 | # assuming the typle is of length 3 400 | floats: Tuple[float, float, float] 401 | ``` 402 | 403 | - There is also a `Union` type. For example 404 | 405 | ```python 406 | # pre-Python 3.10 407 | from typing import List 408 | values: List[Union[int, float, str]] = [1, 1.0, "hi", 2.0, "hello",] 409 | 410 | # Python 3.10+ 411 | values: list[int | float | str] = [1, 1.0, "hi", 2.0, "hello",] 412 | ``` 413 | 414 | - I recommend avoiding the 3.10+ syntax since many Python environments (AWS Lambda, certain Docker images, certain Linux machines, etc.) do not support Python versions that high yet. 415 | - The newer language features make your code not backwards compatible with older versions of Python. 416 | - `Optional` is a type that is used to indicate that a variable can be `None`. For example 417 | 418 | ```python 419 | # pre-Python 3.10 420 | from typing import Optional 421 | maybe_num: Optional[int] = None 422 | maybe_num = 1 423 | 424 | # OR 425 | 426 | from typing import Union 427 | maybe_num: Union[int, None] = None 428 | maybe_num = 1 429 | 430 | # Python 3.10+ 431 | maybe_num: int | None = None 432 | maybe_num = 1 433 | ``` 434 | 435 | 436 | #### Assignments 437 | 438 | No assignments. 439 | --------------------------------------------------------------------------------