├── README.md ├── LICENSE ├── opsec.md ├── bug-bounty-program.md ├── versioning.md ├── organizing-code.md ├── go-to-libraries.md ├── ideology-and-subjectivity.md ├── mutability.md ├── antipatterns.md ├── template.md ├── naming-things.md ├── cutting-a-release.md ├── testing.md ├── documentation.md ├── git-and-github.md └── style-guide.md /README.md: -------------------------------------------------------------------------------- 1 | # Development Tactical Manual 2 | 3 | This document lays out guidelines and expectations for the Ethereum Foundation 4 | Python development team. 5 | 6 | 7 | * [Code Style Guide](./style-guide.md) 8 | * [Code Idioms](./idioms.md) 9 | * [Naming Things](./naming-things.md) 10 | * [Antipatterns](./antipatterns.md) 11 | * [Mutability](./mutability.md) 12 | * [Testing](./testing.md) 13 | * [Organizing Library Code](./organizing-code.md) 14 | * [Git and Github](./git-and-github.md) 15 | * [Documentation](./documentation.md) 16 | * [Operational Security](./opsec.md) 17 | * [Ideology and Subjectivity](./ideology-and-subjectivity.md) 18 | * [Gitcoin Bug Bounty Guide](./bug-bounty-program.md) 19 | * [Cutting a Release](./cutting-a-release.md) 20 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2017-2025 The Ethereum Foundation 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /opsec.md: -------------------------------------------------------------------------------- 1 | # Security 2 | 3 | The software we work on is used in financial applications. This makes us targets. 4 | The following minimum security practices should be used. 5 | 6 | - Use a password manager with strong passwords. 7 | - Don't use browser based password storage or autofilling of passwords. 8 | - Enable two factor authentication but **DO NOT** use SMS codes as a second 9 | factor. 10 | * Mobile carriers are notoriously bad at safe-guarding their customers' 11 | accounts against hijacking. In some cases, simply knowing someone's mobile 12 | number and appearing in person at one of their carrier's stores is enough 13 | to hijack their account. Accounts can also be socially hacked by knowing 14 | the mobile number in addition to a few publically available details of the 15 | account holder's life which are commonly used in account recovery processes 16 | (father's middle name, mother's maiden name, model of first car, etc.). 17 | - Setup auto-signing of your git commits 18 | - https://stackoverflow.com/questions/10161198/is-there-a-way-to-autosign-commits-in-git-with-a-gpg-key 19 | - No password sharing (even for family, close friends, etc) 20 | - Auto-lock machine after a short period (5 minutes) of inactivity 21 | - No shared machine usage (family computer, etc) 22 | - Use pre-boot encryption 23 | - Recommend using a VPN anytime you are on an untrusted network 24 | - ([ProtonVPN](https://protonvpn.com/) is a good VPN provider) 25 | - Never connect to untrusted devices (USB, etc). 26 | - This means any device that you cannot personally account for it's origins. 27 | - This means you cannot plug someone else's USB drive into your machine to transfer files. 28 | - This means you cannot plug someone else's phone into your computer to charge it. 29 | -------------------------------------------------------------------------------- /bug-bounty-program.md: -------------------------------------------------------------------------------- 1 | # Guidelines for bug bounty submissions 2 | 3 | - An issue will be considered un-claimed until a pull request has been made and 4 | the claim has been approved by a core contributor. Claiming an issue on gitcoin.co is not sufficient. 5 | 6 | - A good pull request might open with unit tests before any changes are written. Tests 7 | serve as good examples when discussing the scope of work to be performed. 8 | 9 | - Settle expectations early, either in discussion or with tests. This will save you from wasted effort. 10 | Make sure you have understood the parameters of the request before diving in. 11 | 12 | - Review the other sections of this tactical guide. Conforming to the team's coding style will help ensure your pull 13 | request will be integrated. 14 | 15 | - Some issues may prove to be bigger projects than estimated when the bounty was set. Feel free to make 16 | a case to either increase the bounty or reduce the scope of the request. It is up to a the core team whether an 17 | adjustment will be made. 18 | 19 | - The bounty value is set assuming a claimee who is familiar with the code base. In other words, a first time 20 | contributor is expected to take as long as they need to familiarize themselves with the project and the team's 21 | contribution guidelines. This added time is not a sufficient reason to increase the bounty. 22 | 23 | - Work will be considered complete when the following items have been met: 24 | 25 | - The submission fulfills all the parameters set forth in the initial discussion. 26 | - Tests have been written for any bugs described in the issue and/or all new code. 27 | - The project documentation has been updated to reflect changes to the api. 28 | - All review requests have been met or resolved. 29 | - The submission conforms to the team's coding style documented in this 30 | tactical guide. 31 | - The change has been approved and merged by a core contributor. 32 | -------------------------------------------------------------------------------- /versioning.md: -------------------------------------------------------------------------------- 1 | ## Dependency Versioning Philosophy 2 | 3 | Our goal is to keep dependency versions as open as possible to avoid limiting users of our libraries. 4 | 5 | - **No upper bounds** on versions unless there is a known compatibility issue with a newer major version. 6 | - **Lower bounds** should be set to the earliest version that supports the features we use, to maximize compatibility. 7 | - This approach helps ensure our packages work in a wider range of environments and don't unnecessarily constrain downstream users. 8 | 9 | > ⚠️ **Exception:** If a newer major version of a dependency introduces breaking changes or known issues with our code, it's okay to set an upper pin temporarily—just make sure to leave a comment explaining why. 10 | 11 | > 📚 **Further Reading:** 12 | > - [Bounded version constraints are bad](https://iscinumpy.dev/post/bound-version-constraints/) 13 | > - [The semantics of version numbers](https://bernat.tech/posts/version-numbers/) 14 | > These articles help explain the rationale behind avoiding upper pins and being cautious with lower bounds. 15 | 16 | ## Version Pinning Guidelines 17 | 18 | Follow these practices when setting or updating version pins: 19 | 20 | - **Fast-moving internal libraries** (`eth-utils`, `eth-typing`, `eth-account`, possibly `eth-abi`): 21 | - Set the lower version pin to the most recent major release during each breaking cycle. 22 | - This ensures we can adopt new features introduced throughout the year. 23 | 24 | - **Slower-moving internal libraries** (e.g., crypto libraries or other stable packages we maintain): 25 | - Only update the lower pin during a breaking cycle if necessary. 26 | - Stick to the current version unless there's a clear need to change. 27 | 28 | - **Third-party libraries** (`requests`, `websockets`, etc.): 29 | - Upgrade only when required (e.g., security patches, compatibility issues). 30 | - In practice, we often end up pinning to the latest major version to stay compatible. 31 | 32 | > 💡 **Reminder:** Always evaluate the impact of version changes on downstream packages and tests before updating pins. 33 | 34 | -------------------------------------------------------------------------------- /organizing-code.md: -------------------------------------------------------------------------------- 1 | # Organizing Library Code 2 | 3 | We use the following conventions for organizing our libraries. 4 | 5 | 6 | All of these examples assume that `./` is the *root* of the git repository. 7 | 8 | We use `./` to denote the root directory of the python module for a given library. 9 | 10 | 11 | ## Documentation 12 | 13 | Documentation should be under `./docs` 14 | 15 | 16 | ## Tests 17 | 18 | Tests should be under `./tests` 19 | 20 | 21 | ### `asyncio` & `trio` 22 | 23 | See: https://github.com/pytest-dev/pytest-asyncio/issues/124 24 | 25 | Currently `pytest-asyncio` doesn't play nicely with `trio`. If you need to 26 | have a test suite that uses both of these it is best to separate them at the 27 | top level using separate directorie `./tests-trio` and `./tests-asyncio`. 28 | 29 | ## Non-public utility functions 30 | 31 | For a simple singular python module containing utility functions. 32 | 33 | - `.//_utils.py` 34 | 35 | For a more complex set of shared utility functions. 36 | 37 | - `.//_utils/` 38 | - `.//_utils/__init__.py` 39 | - `.//_utils/filesystem.py` 40 | - `.//_utils/numbers.py` 41 | 42 | For utility functions that are specific to a sub-section of the library 43 | 44 | - `.//some_module/_utils.py` 45 | 46 | 47 | One *convention* that we use for *utils* modules is that they should **rarely** 48 | import things from elsewhere in the library to prevent circular imports. In 49 | some cases this cannot be avoided, in which case the import should be inlined 50 | into the function body with a comment indicating *why* the import isn't part of 51 | the top level module imports. 52 | 53 | 54 | ## Constants 55 | 56 | For constant values that don't belong in any specific module we use `.//constants.py` 57 | 58 | Or for constants that are specific to some sub-section of the library: `.//some_module/constants.py` 59 | 60 | 61 | ## Typing 62 | 63 | For type aliases or `typing.NewType` values that don't belong in any specific module we use `.//typing.py` 64 | 65 | Or for ones that are specific to some sub-section of the library: `.//some_module/typing.py` 66 | 67 | 68 | ## Non-core functionality 69 | 70 | - `.//tools/` 71 | 72 | This section encapsulates both functionality that isn't part of the *core* 73 | library but may be useful to end users, as well as functionality that is only 74 | used for things like the test suite. 75 | 76 | The convention is that you **may not** import things from `library.tools` 77 | anywhere in the rest of the library code. Although, it is 78 | acceptable and common to import `tools` in order to set up tests. 79 | -------------------------------------------------------------------------------- /go-to-libraries.md: -------------------------------------------------------------------------------- 1 | # Go-To Libraries 2 | 3 | This is the location for opinionated write-ups of different python libraries to 4 | accomplish certain tasks. 5 | 6 | 7 | ## `async` 8 | 9 | ### [trio](https://github.com/python-trio/trio) 10 | 11 | @piper recommends 12 | 13 | This is the best alternative to the built-in standard library `asyncio`. 14 | 15 | 16 | ### [asks](https://github.com/theelous3/asks) 17 | 18 | Like [`requests`](https://2.python-requests.org/en/master/) but `async` and built on `trio` 19 | 20 | 21 | 22 | ## Grammar Parsing 23 | 24 | ### [Parsimonious](https://github.com/erikrose/parsimonious) 25 | 26 | @piper recommends 27 | 28 | This is a good choice if you need to parse a simple grammar such as ABI type 29 | strings. For complex grammars it can be lacking in expressiveness. 30 | 31 | 32 | ### [PyParsing](https://pyparsing-docs.readthedocs.io/en/latest/) 33 | 34 | @piper recommends 35 | 36 | This is likely the most well established parsing library in the python 37 | ecosystem. It uses python classes for constructing the grammar which requires 38 | learning your way around the library some, but also provides extreme 39 | flexibility and expressiveness. 40 | 41 | Beyond the grammar parsing component, it also lends itself well to functional 42 | style programming as each grammar rule can be assigned a parse action which is 43 | just a function to process the parsed result. 44 | 45 | 46 | ### [Lark](https://github.com/lark-parser/lark) 47 | 48 | @piper **does not** recommend 49 | 50 | I can't currently recommend Lark. It's got a nice grammar and API but the 51 | library doesn't appear to be very well maintained or documented and there are 52 | places where the documentation doesn't match the library or the library doesn't 53 | behave in documented ways. Expect to end up reading the code some if you 54 | choose this for anything very complex. 55 | 56 | ### [`cached-property`](https://pypi.org/project/cached-property/) 57 | 58 | @piper recommends 59 | 60 | For `@property` methods that are safe to cache this provides a clean, well tested, 61 | and highly performant mechanism. Use of this however collides with the `__slots__` 62 | approach. 63 | 64 | > NOTE: https://github.com/pydanny/cached-property/issues/69 demonstrates how to combine this with `__slots__`. 65 | 66 | 67 | ## Sorted Containers 68 | 69 | You mostly don't need sorted sets or dicts or lists. Then someday you really do. 70 | 71 | ### `sortedcontainers` [docs](http://www.grantjenks.com/docs/sortedcontainers/index.html) and [code](https://github.com/grantjenks/python-sortedcontainers) 72 | 73 | @carver recommends 74 | 75 | This has the best docs of the bunch, top performance, a pure python implementation, and great APIs. 76 | -------------------------------------------------------------------------------- /ideology-and-subjectivity.md: -------------------------------------------------------------------------------- 1 | # Ideology and Subjectivity 2 | 3 | This section contains various ideological and subjective things about workflow, 4 | development style, etc. 5 | 6 | 7 | ## Breaking Changes and Deprecation Policies 8 | 9 | Libraries and APIs which are experimental should be clearly marked as such in 10 | the documentation. These APIs/libraries may be subject to arbitrary breaking 11 | changes. 12 | 13 | However, anything that is documented should be subject to a deprecation period 14 | before introducing **any** breaking API changes. Our users need to trust they 15 | can use our libraries as foundational components to their products and that 16 | they won't be surprised with sudden breaking changes. 17 | 18 | Exceptions to this would be things like security issues. 19 | 20 | 21 | ## Delivering Results 22 | 23 | As you develop features you should balance the timely delivery of shippable 24 | code with well designed architecture. 25 | 26 | New feature work can take multiple weeks to develop, but you should never go 27 | multiple weeks without merging code. There are always ways that large changes 28 | can be broken down into iterative smaller changes. 29 | 30 | It is ok to develop a large feature branch with broad sweeping API changes, but 31 | it will need to be broken down into smaller pieces that maintain backwards 32 | compatability in order to be merged. 33 | 34 | 35 | ## Community Support 36 | 37 | Our users interact with us primarily through Github issues and the Gitter chat 38 | channels. You should make an effort to respond when a user needs help. We are 39 | often the most knowledgeable people when it comes to our libraries and thus are 40 | often in the best position to provide help. 41 | 42 | 43 | ## The Zen of the Python Ethereum Team 44 | 45 | Here's a collection of things. 46 | 47 | * Write fewer classes. 48 | * Write more functions. 49 | * Write even more utility functions. 50 | * Hard to test code is probably wrong 51 | * Less statefulness 52 | * More pure functions 53 | * Avoid mutability at *almost* any cost. 54 | * The [toolz](http://toolz.readthedocs.io/) library is your friend. 55 | * One concept per line of code. 56 | * Don't be clever. 57 | 58 | 59 | ## Friendly Code Review 60 | 61 | ### Avoid negative attribution 62 | 63 | When reviewing people's code consider the following two comments. 64 | 65 | > I don't like the name of this function. 66 | 67 | vs. 68 | 69 | > What do you think about changing the name of this function to .... 70 | 71 | Your feedback will often be better received if you pose it in the form of a 72 | question. 73 | 74 | 75 | ### Keep it objective 76 | 77 | Similarly, consider whether your statement is objective or subjective. 78 | 79 | > This variable name is bad. 80 | 81 | vs 82 | 83 | > This variable name is imprecise. 84 | 85 | The later is closer to a statement of *fact* where as the former is a 86 | subjective opinion. The discussion sparked by the first will likely end up 87 | being an argument about two differing opinions, where-as we can have an 88 | objective discussion about the later since it captures a more 89 | objective/debatable issue. 90 | -------------------------------------------------------------------------------- /mutability.md: -------------------------------------------------------------------------------- 1 | # Mutability 2 | 3 | We have taken a strong stance against mutability and thus have established 4 | multiple conventions that should be followed to avoid the use of mutability. 5 | 6 | ## Why go to such lengths 7 | 8 | Here is a common example where use of a `list` is often the norm. 9 | 10 | ```python 11 | class User: 12 | def get_recent_events(self): 13 | events = [] 14 | for event in self.all_events: 15 | if event.is_recent(): 16 | events.append(event) 17 | return event 18 | ``` 19 | 20 | This example is quite benign and the mutation of the list object doesn't pose 21 | any real problems. There are no race conditions, or shared state. This raises 22 | the question *why* do we avoid this pattern if it is using mutability in a safe 23 | way? 24 | 25 | The answer has two parts. 26 | 27 | First, *internally* by being disciplined, we can avoid entire classes of bugs that *can* arrise inadvertantly due to the use of mutability. While the example above doesn't have any immediate issues related to mutability, it isn't unreasonable to imagine a refactor to the function above which ends up introducing a bug due to the use of mutability. 28 | 29 | Second, *externally* we want to set an easy to follow example. We have plenty 30 | of 3rd party contributors and they take cues on how to write new code based on 31 | existing code. Rather than try to define rules on what constitutes 32 | *acceptable* use of mutability, it is much simpler for the rule to be 33 | "immutable* by default unless there is an explicit reason to use mutability. 34 | 35 | 36 | ### Caching 37 | 38 | Another benefit of immutable objects is that they play nicely with caching as 39 | well as sharing them across processes. 40 | 41 | 42 | ## When can I use mutability 43 | 44 | Anytime you have an explicit reason to do so. Some good reasons are: 45 | 46 | - Performance critical code 47 | - Readability 48 | 49 | 50 | ## Types of multability and how to avoid it. 51 | 52 | 53 | ### Programatic generation of various collection data structures 54 | 55 | The use of lists 56 | 57 | ```python 58 | # list 59 | def get_things(all_items): 60 | things = [] 61 | for item in all_items: 62 | things.append(item) 63 | return things 64 | ``` 65 | 66 | We use `tuple` in place of lists and generators or comprehensions in place of 67 | `list.append` for programatic consprogramatic. 68 | 69 | 70 | ```python 71 | # using a comprehension 72 | def get_things(all_items): 73 | return tuple(item for item in all_items) 74 | 75 | # using a generator 76 | def get_things(all_items): 77 | for item in all_items: 78 | yield item 79 | ``` 80 | 81 | The `get_things` function above will produce a generator which can be a 82 | *foot-gun*. eg~ if you try to iterate the result twice, it will be empty on the 2nd round. 83 | You can either use the 84 | [`eth_utils.to_tuple`](https://eth-utils.readthedocs.io/en/latest/utilities.html#to-tuple-callable-callable-tuple) 85 | decorator or a thin wrapper pattern to force the generator to evaluate like this. 86 | 87 | 88 | ```python 89 | @to_tuple 90 | def get_things(all_items): 91 | for item in all_items: 92 | yield item 93 | 94 | 95 | # or a thin wrapper 96 | def get_things(all_items): 97 | return tuple(_get_things(all_items)) 98 | 99 | def _get_things(all_items): 100 | for item in all_items: 101 | yield item 102 | ``` 103 | 104 | 105 | > This approach for `list` objects also translates to `set` objects. 106 | 107 | 108 | The same applies to dictionaries. While python doesn't have an immutable 109 | dictionary, we tend to avoid mutating dictionaries. 110 | 111 | 112 | ```python 113 | # avoid this pattern 114 | def get_result(all_items): 115 | result = {} 116 | for key, value in all_items: 117 | result[key] = value 118 | ``` 119 | 120 | Avoiding mutation can be done either using a comprehension, the generator 121 | pattern, or using the [`toolz` library 122 | APIs](https://toolz.readthedocs.io/en/latest/api.html#dicttoolz) for doing 123 | dictionary mergers or key/value filtering/mapping. 124 | 125 | 126 | ```python 127 | # comprehension 128 | def get_result(all_items): 129 | return {key: value for key, value in all_items} 130 | 131 | # generator (using either thin wrapper) 132 | def get_result(all_items): 133 | return dict(_get_result(all_items)) 134 | 135 | def _get_result(all_items): 136 | for key, value in all_items: 137 | yield key, value 138 | 139 | # generator (using `eth_utils.to_dict`) 140 | @to_dict 141 | def get_result(all_items): 142 | for key, value in all_items: 143 | yield key, value 144 | ``` 145 | -------------------------------------------------------------------------------- /antipatterns.md: -------------------------------------------------------------------------------- 1 | # Antipatterns 2 | 3 | This is a collection of [antipatterns](https://en.wikipedia.org/wiki/Anti-pattern) which we actively avoid. 4 | 5 | 6 | ## Boolean checks on objects as functions. 7 | 8 | 9 | We try to avoid implementing boolean checks on objects as functions, instead preferring `@property` methods when appropriate. 10 | 11 | 12 | ```python 13 | class Thing: 14 | def is_active(self): 15 | ... 16 | 17 | 18 | def get_active_things(all_things): 19 | return tuple(thing for thing in all_things if thing.is_active) 20 | ``` 21 | 22 | Can you spot the bug in the code above? It should be: 23 | 24 | 25 | 26 | ```python 27 | def get_active_things(all_things): 28 | return tuple(thing for thing in all_things if thing.is_active()) 29 | ``` 30 | 31 | Note that we needed to call `thing.is_active()`. The nefarious part of this is 32 | that the first code will not throw any errors, but will just return the 33 | entirety of the original data set. This is due to the python semantics that 34 | function objects evalute truthy. 35 | 36 | For this reason, when implementing methods like this, we prefer to make them 37 | `@property` methods. 38 | 39 | 40 | ```python 41 | class Thing: 42 | @property 43 | def is_active(self): 44 | ... 45 | 46 | # correct 47 | thing.is_active 48 | 49 | # incorrect: throws a `TypeError` 50 | thing.is_active() 51 | ``` 52 | 53 | This pattern ensures that incorrect usage of the API throws an exception rather 54 | than silently doing the wrong thing. 55 | 56 | 57 | ## Return value checking 58 | 59 | 60 | We prefer raising exceptions over return value checking. Here is an example that requires return value checking. 61 | 62 | 63 | ```python 64 | def get_user(user_id): 65 | if orm.user_exists(user_id): 66 | return orm.get_user(user_id) 67 | else: 68 | return None 69 | 70 | 71 | # some "view" function in a web application 72 | def user_profile_view(request, user_id): 73 | user = get_user(user_id) 74 | return template.render({'user': user}) 75 | ``` 76 | 77 | The `get_user` function above can return either a user, or `None` if no user 78 | was found. This pattern requires all consumers of this API to check the return 79 | value, an action which is easy to forget to do. This often results in 80 | unexpected errors happening some distance away from the call to `get_user` when 81 | a `None` value is used for something that expects a user object. These types 82 | of bugs can be difficult and confusing to diagnose and can often remain hidden 83 | in an application until the right situation arises to trigger it. It is also 84 | easy for the *checking* of the return value to accidentally be removed since 85 | those checks are often only loosely coupled. 86 | 87 | Instead of returning `None` we should instead raise an exception. This changes 88 | the requirement for API consumers to include exception handling instead of 89 | return value checking. In the case that a call site forgets to add the proper 90 | exception handling, the error is raised at the call site. 91 | 92 | 93 | 94 | ```python 95 | def get_user(user_id): 96 | if orm.user_exists(user_id): 97 | return orm.get_user(user_id) 98 | else: 99 | raise Exception("User not found") 100 | ``` 101 | 102 | 103 | ## Validation functions should probably not have a return value 104 | 105 | This is an extension of the "no return value checking" antipattern. 106 | 107 | 108 | 109 | ```python 110 | # bad: requires return value checking 111 | def validate_age(user): 112 | if user.age >= 18: 113 | return True 114 | else: 115 | return False 116 | 117 | # good 118 | def validate_age(user): 119 | if user.age < 18: 120 | raise Exception("User must be 18 or older") 121 | ``` 122 | 123 | In the case where you need both *validation* and a check, they should be 124 | implemented separately, or in combination like follows. 125 | 126 | 127 | ```python 128 | def check_is_old_enough(user): 129 | return user.age >= 18 130 | 131 | def validate_age(user): 132 | if not check_is_old_enough(user): 133 | raise Exception("user must be 18 or older") 134 | ``` 135 | 136 | ## Assertions in runtime code. 137 | 138 | For Reference: https://stackoverflow.com/questions/1273211/disable-assertions-in-python 139 | 140 | 141 | ```python 142 | class User: 143 | def __init__(self, name, age): 144 | assert len(name) > 1 145 | assert age >= 18 146 | ... 147 | ``` 148 | 149 | A common pattern for doing validation is to use `assert` statements. We choose 150 | to use explicit exception raising due to how easy it is to run the python 151 | interpreter with assertions disabled. 152 | 153 | 154 | ```python 155 | class User: 156 | def __init__(self, name, age): 157 | if len(name) > 1: 158 | raise AssertionError(...) 159 | if age < 18: 160 | raise AssertionError(...) 161 | ... 162 | ``` 163 | 164 | Replacing `assert` statements with `raise AssertionError(...)` (or whatever 165 | exception class you prefer) ensures that these checks cannot be trivially 166 | disabled. 167 | -------------------------------------------------------------------------------- /template.md: -------------------------------------------------------------------------------- 1 | # Repository Template 2 | 3 | Many of our projects are based on a [repository template](https://github.com/ethereum/ethereum-python-project-template/). 4 | 5 | A key benefit of using the template is that we can easily propagate best practices to a bunch of repos. 6 | 7 | ## Using template to create a repo 8 | 9 | Basically, just clone the template repo and start any new repo with that clone. 10 | 11 | *NOTE: we do **not** squash the history when cloning. In order to easily pull in later updates from the repo, we need the full template history.* 12 | 13 | TODO: step-by-step instructions 14 | 15 | ## Upgrading a repo to use the latest templated features 16 | 17 | ### Confirm template is remote 18 | 19 | Check that you have the remote: 20 | 21 | ```sh 22 | git remote -v 23 | ``` 24 | 25 | #### What it looks like if you DO have the template remote 26 | 27 | ``` 28 | $ git remote -v 29 | origin git@github.com:carver/eth-typing.git (fetch) 30 | origin git@github.com:carver/eth-typing.git (push) 31 | template-host git@github.com:ethereum/ethereum-python-project-template.git (fetch) 32 | template-host git@github.com:ethereum/ethereum-python-project-template.git (push) 33 | upstream git@github.com:ethereum/eth-typing.git (fetch) 34 | upstream git@github.com:ethereum/eth-typing.git (push) 35 | ``` 36 | 37 | #### What it looks like if you do NOT have the template remote 38 | 39 | ``` 40 | $ git remote -v 41 | origin git@github.com:carver/eth-typing.git (fetch) 42 | origin git@github.com:carver/eth-typing.git (push) 43 | upstream git@github.com:ethereum/eth-typing.git (fetch) 44 | upstream git@github.com:ethereum/eth-typing.git (push) 45 | ``` 46 | 47 | In this case, add the template as a remote this way: 48 | 49 | ```sh 50 | git remote add template-host git@github.com:ethereum/ethereum-python-project-template.git 51 | ``` 52 | 53 | Then check out the main of the template repo as a local branch called `template`: 54 | 55 | ```sh 56 | git fetch template-host 57 | git checkout --track -b template template-host/main 58 | ``` 59 | 60 | ### Pull in the latest template changes 61 | 62 | #### Get the latest template changes into your template branch 63 | 64 | ```sh 65 | git checkout template 66 | git pull 67 | ``` 68 | 69 | #### Create a template upgrade feature branch 70 | 71 | ```sh 72 | git checkout main 73 | git pull 74 | git checkout -b upgrade-template 75 | ``` 76 | 77 | #### Merge the latest template changes into your feature branch 78 | 79 | ```sh 80 | git merge template 81 | ``` 82 | 83 | #### Resolve conflicts 84 | 85 | As usual. Feel free to **leave in** any template variables like `` or ``, 86 | because they will be filled in again, next. 87 | 88 | Go ahead and commit with the template variables unfilled. This makes it easy to undo any mistakes 89 | while filling in template variables. 90 | 91 | ### Fill any added template variables 92 | 93 | #### Check if you've run a template fill before 94 | 95 | If you have filled them in, then you'll see something like: 96 | ``` 97 | $ cat .project-template/template_vars.txt 98 | eth_typing 99 | eth-typing 100 | eth-typing 101 | eth-typing 102 | eth-typing 103 | Common type annotations for ethereum python packages 104 | ``` 105 | 106 | If you haven't filled them in, then you'll see something like: 107 | ``` 108 | $ cat .project-template/template_vars.txt 109 | 110 | 111 | 112 | 113 | 114 | 115 | ``` 116 | 117 | #### Fill variables 118 | 119 | If the variables are already fully filled, then run 120 | ```sh 121 | python .project-template/refill_template_vars.py 122 | ``` 123 | 124 | If not, run: 125 | ```sh 126 | python .project-template/fill_template_vars.py 127 | ``` 128 | 129 | Answer any questions that you're prompted with. 130 | 131 | #### Commit filled variables 132 | 133 | Be sure to double-check all of the fills, by using `git add -p`. 134 | 135 | ### Push up the changes for PR 136 | 137 | ```sh 138 | git push --set-upstream origin upgrade-template 139 | ``` 140 | 141 | Open the PR, resolve and issues, and merge. **Then you're done!** 142 | 143 | ## Configure an existing repo to use the template 144 | 145 | Sometimes we have existing repos that are close to the template, but were never actually derived from the same git history. These are the steps to merge the git histories. This provides the benefit that it's much easier to pull in all the upgrades in the template at a later time. 146 | 147 | ### Fetch the template repo 148 | 149 | ```sh 150 | git remote add tmpl git@github.com:ethereum/ethereum-python-project-template.git 151 | get fetch tmpl 152 | ``` 153 | 154 | Check out the template on a local branch: 155 | ```sh 156 | git checkout -b template tmpl/main 157 | ``` 158 | 159 | ### Merge in the template 160 | 161 | We want to force a merge, even though the repositories have no shared history. 162 | ```sh 163 | git checkout -b join-to-template main 164 | git merge --allow-unrelated-histories template 165 | ``` 166 | 167 | ### Resolve conflicts 168 | 169 | Yeah, no way around it. This will probably take a while. There's no handbook to follow here, you just have to figure it out patch by patch. 170 | 171 | After committing the merge, don't forget to re-fill the template variables. See the section above called "Fill any added template variables" 172 | -------------------------------------------------------------------------------- /naming-things.md: -------------------------------------------------------------------------------- 1 | # Naming Conventions 2 | 3 | Here are some go-to conventions for the names of things. 4 | 5 | 6 | ## Variables 7 | 8 | 9 | ### Looping over dictionaries 10 | 11 | You **should** use the names `key` and `value` 12 | 13 | ```python 14 | for key, value in thing.items(): 15 | ... 16 | ``` 17 | 18 | 19 | ### Looping over iterables 20 | 21 | 22 | When writing generic loops, you **should** use the name ``value``. 23 | 24 | ```python 25 | for value in things_to_process: 26 | ... 27 | ``` 28 | 29 | In cases where the term ``value`` is not available, you **should** use the term 30 | ``item`` 31 | 32 | ```python 33 | for key, value in thing.items(): 34 | for item in value: 35 | ... 36 | ``` 37 | 38 | 39 | ### Confusing Plurals 40 | 41 | For short names, the simple plural form *can* be ok but if often visually 42 | confusing, making it easy to inadvertantly swap one of the variables for 43 | another. 44 | 45 | ```python 46 | for user in users: 47 | ... 48 | ``` 49 | 50 | 51 | For longer names, avoid using plural names that only differ by one letter since 52 | they often much 53 | 54 | ```python 55 | for lockfile_path in lockfile_paths: 56 | ... 57 | ``` 58 | 59 | When using longer names, consider adding a prefix or suffix to help 60 | differentiate the variable names. 61 | 62 | ```python 63 | for lockfile_path in lockfile_paths_to_process: 64 | ... 65 | 66 | for user in all_users: 67 | ... 68 | ``` 69 | 70 | 71 | ### Booleans 72 | 73 | When naming boolean instance attributes, it is often a good idea to use the `is_` prefix. 74 | 75 | ```python 76 | class Plugin 77 | def __init__(self, enable_discovery: bool) -> None: 78 | self.is_discovery_enabled = enable_discovery 79 | ``` 80 | 81 | If the attribute were to be named `enable_discovery` it might be confused for a method and thus a call to `if self.enable_discovery` would always evaluate *truthy*. 82 | 83 | 84 | It is also best to avoid *negative* boolean properties since they end up producing double negatives in code. 85 | 86 | 87 | ```python 88 | # avoid this 89 | is_discovery_disabled 90 | 91 | if not is_discovery_disabled: 92 | ... 93 | ``` 94 | 95 | It is difficult to parse the `if` statement above since it contains a double 96 | negative. If you regularly need to check the negative case it is recommended 97 | to have both a positive and negative flag. 98 | 99 | ```python 100 | is_discover_enabled = ... 101 | is_discover_disabled = not is_discover_enabled 102 | 103 | if is_discover_enabled: 104 | ... 105 | elif is_discover_disabled: 106 | ... 107 | ``` 108 | 109 | 110 | 111 | ## Function and Method Names 112 | 113 | 114 | ### Prefixes 115 | 116 | The following guideline **should** be used when naming functions or methods. 117 | This provides logical consistency across methods to give future readers of the 118 | code a cue as to what this method likely does (or more importantly, does not 119 | do). 120 | 121 | - `get_`: 122 | - **should** be used for simple retrievals of data. Generally assumed to not have side effects. 123 | - `fetch_`: 124 | - **should** be used for retrievals that may have some sort of side effect such as lazy creation, making an HTTP request, or other IO. 125 | - `compute_`: 126 | - **should** be used for cases where something is being computed or derived from some other data set. Can be used to signal that this call might be computationally expensive. 127 | - `construct_` or `build_`: 128 | - Similar to `compute_` but focused more on creation of some other first class object that needs to be computed or derived from other data. 129 | - `from_thing`: 130 | - Typically used for `classmethod` implementations which act as convenience wrappers around instantiating an object from some other data type when we want to avoid coupling the objects `__init__` with that other data type. 131 | 132 | - `validate_`: 133 | - A method that either returns `None` or raise an exception if something is wrong. 134 | 135 | 136 | ## General comments on naming 137 | 138 | Strive to choose names that clearly communicate intent; you want to essentially use the "smallest" concept you can while still transmitting the relevant information to future readers (including yourself). 139 | 140 | Of particular importance is naming things based on what they provide, not what they do or details of how they do it. First an example, then further rationale: 141 | 142 | ```python 143 | # bad 144 | ## implies a package name of `enum` that doesn't tell 145 | ## the consumer of a codebase much about what is inside. 146 | from enum.signature_enum import SignatureEnum 147 | 148 | from datastructures.header_helpers import HeaderCounter 149 | 150 | signature_domain = SignatureEnum.some_application_domain() 151 | header_counter = HeaderCounter() 152 | # use signature_domain and header_counter... 153 | 154 | 155 | 156 | # better 157 | from signatures.domains import SignatureDomain 158 | from headers.counter import HeaderCounter 159 | 160 | signature_domain = SignatureDomain.for_some_application() 161 | header_counter = HeaderCounter() 162 | # use signature_domain and header_counter... 163 | ``` 164 | 165 | This guideline tends to lend higher semantic value to a given name so that it plugs into the surrounding conceptual framework of the codebase. In the second example, we stick to the domain of this application with an enumeration for "signature domains" and some utility to help count "headers". The relevant concepts are those at the level of the application. This contrasts with the first example that withholds this context and instead communicates lower-level implementation details (e.g. the fact that we have an ``enum`` implementation of the set of domains or that `HeaderCounter` belongs in package that bundles together what we may consider `data structures`.) 166 | 167 | 168 | ## CLI flags 169 | 170 | When writing CLI tools we use the following conventions: 171 | 172 | 173 | Flags should use `-` separators between words. 174 | 175 | 176 | ```bash 177 | # yes 178 | $ greet --full-name 179 | 180 | # no 181 | $ greet --fullname 182 | ``` 183 | 184 | Flags should have a double `--` prefix for the *full* non-abbreviated flag 185 | 186 | ```bash 187 | # yes 188 | $ greet --name piper 189 | 190 | # no 191 | $ greet -name piper 192 | ``` 193 | 194 | Flags should have a single `-` prefix for the short abbreviated flag 195 | 196 | ```bash 197 | # yes 198 | $ greet --name piper 199 | $ greet -n piper 200 | 201 | # no 202 | $ greet --n piper 203 | ``` 204 | -------------------------------------------------------------------------------- /cutting-a-release.md: -------------------------------------------------------------------------------- 1 | When to cut a new release 2 | ==== 3 | 4 | Releases should be considered for each change that is merged to the main branch. Not all merges require a new version so it is reasonable to bundle several smaller changes together in a single release. All libraries should follow Continuous Delivery and Semver best practices. We typically release on Wednesdays and at minimum once per month (but could be longer for less prominent libraries). 5 | 6 | > :warning: Make sure to cut the release when there is time to support it if something goes wrong. As a rule, don't do releases late in the afternoon or on Fridays. 7 | 8 | The team should review any outstanding PRs and recent merges at the weekly sync. If there are any newsfragments in the main branch there is a good chance a release is needed. 9 | 10 | See the Best Practices For Major Version Changes section below for more details on things to consider for a major version bump. 11 | 12 | ### Annually scheduled releases 13 | 14 | Python is now maintaining a regular schedule of releasing a new version and deprecating 15 | an old one every October. Adding support for a new version is considered a feature and 16 | can be included with any release. We consider removing support for a version breaking, 17 | thus should only be done in a major release. 18 | 19 | Updates from the [project template](https://github.com/ethereum/ethereum-python-project-template) 20 | can be saved up through the year and done in one update cycle over Q4 of each year. 21 | This should include adding the new python version and can be done as soon as CI has an 22 | image available. 23 | 24 | Removal of a supported python version should be coordinated with the major release 25 | of `web3.py`. In order to avoid breaking libraries downstream, remove it first in 26 | `web3.py`, then work backwards up the dependency tree. Utilizing beta releases can 27 | make this process less disruptive. 28 | 29 | How to cut a new release 30 | ==== 31 | 32 | A release of a library requires a version bump, compiling release notes, building the package, and publishing to PyPI. 33 | 34 | ### Releasing with Semver 35 | 36 | We use [Semantic Versioning](https://semver.org/) for all of our libraries. Decide which bump applies (major, minor, or patch) to the changes that will be released. Here's a quick cheat sheet: 37 | - Use a `major` version bump when you make incompatible API changes 38 | - Use a `minor` version bump when you add functionality in a backwards-compatible manner 39 | - Use a `patch` version bump when you make backwards-compatible bug fixes 40 | 41 | Follow the sections below to release with the desired version. 42 | 43 | ### Requirements 44 | 45 | 1. Changes are merged in the main or version branch 46 | 1. The latest build on the main or version branch was successful 47 | 1. Team buy-in: You'll need to get approval from at least two team members before a release. They'll want to know what lib you're releasing, what is in the release. 48 | 49 | ### Steps 50 | 51 | 1. From your local copy of the lib's repository, check out and pull the latest from the main or version branch. 52 | 1. Activate your virtual environment and run `python -m pip install -e ".[dev]"` to ensure the latest dev dependencies are installed. 53 | * It may be necessary to manually install dependencies listed under the `setup_requires` keyword. 54 | 1. Verify documentation and release notes 55 | * Run `make docs` to preview and verify the docs pages. 56 | * Run `towncrier build --draft` to verify the release notes. 57 | 1. If you are issuing a release for a major version, check out the [Best Practices for Major Version Changes](#major-release) below. 58 | 1. Generate release notes with `make notes bump=`. 59 | 1. The remaining steps are run via `make release bump=` command. This will typically build the docs, bump the version, push the tags to GitHub, and build and publish the package to PyPI. 60 | 61 | If any of the commands fail, make sure the release is performed to completion. Once you have resolved the issue, run the remaining commands manually and verify the package is available. 62 | 63 | ### Final Verification 64 | 65 | 1. Make sure to install the new version and confirm everything is working. 66 | * Create and activate a new virtualenv, then run `$ pip install -U ` 67 | 1. Check the latest documentation is published on ReadTheDocs. 68 | * Also check the ReadTheDocs build succeeded at `https://.readthedocs.io/en/stable/releases.html` 69 | 70 | ### New Version Announcement 71 | 72 | If the release was for a user-facing library, announce the latest version in the [web3py Discord channel](https://discord.com/channels/809793915578089484/817614427490615307). Typically this announcement includes a link to the release notes. :tada: 73 | 74 | It is also helpful to put a blurb into the ``#pending-announcements`` channel in the Snake Charmers internal Discord to help advertise what got changed. 75 | 76 | 77 | ## Release FAQ 78 | 79 | ### **Q: The `make release` command failed with "`The user '' isn't allowed to upload to project ''`"** 80 | 81 | **A:** This occurs when you do not have permissions set on your PyPI account. Reach out to the team to get access. 82 | 83 | ### **Q: What can I do if there is not a `make notes` command?** 84 | 85 | **A:** In this case, the release notes must be created manually. The following steps outline this process: 86 | 87 | 1. Modify the release notes in `docs/releases.rst` to add a section for the new version. 88 | 2. Create subsections to categorize each of the changes that were made. 89 | 90 | ### Subsections/Categories 91 | - Breaking Changes (only to be added in a major version bump) 92 | - Bugfixes 93 | - Deprecation 94 | - Docs 95 | - Features 96 | - Internal 97 | - Misc 98 | - Performance 99 | - Removal (only to be added in a major version bump) 100 | 101 | 3. Verify documentation and release notes 102 | * Run `make docs` to preview and verify the docs pages. 103 | 4. Commit the changes to the main branch and push to the upstream repo with `git push upstream`. 104 | 105 | ## Best Practices for Major Version Changes (or, Things Learned Through Experience) 🔗 106 | 107 | - Although it's not always possible, try to batch up breaking changes within a library. For example, if you're going to release a major version of a lib, try and get any other low-hanging fruit and/or prioritized breaking changes in at the same time. 108 | - Utilize beta versions. If there is a big breaking change going in, or multiple big breaking changes going in, consider releasing a beta version or two. 109 | * To perform a release from a pre-release version, use `make notes bump=stage` and `make release bump=stage` 110 | - If you know a change will be breaking in a downstream library, add an upper pin to the dependency requirement in said downstream library while the PR is open in the upstream library. 111 | - We prefer to have breaking changes that cascade across libraries queued up at the same time so that we can release affected libraries in concert. This cuts down on thrash and dependency hell for both ourselves and users. 112 | - Before moving from beta to stable, take one last look at the changes, and consider adding an upper pin of that dependency to any downstream libraries that may be affected. 113 | -------------------------------------------------------------------------------- /testing.md: -------------------------------------------------------------------------------- 1 | # Testing 2 | 3 | Testing is **essential** to the production of software with minimal flaws. The 4 | default should always be writing tests for the code you produce. 5 | 6 | ## Continuous Integration 7 | 8 | Tests **must** always be automated using a service like Travis-CI or Circle-CI. 9 | 10 | Testing also introduces overhead into our workflow. If a test suite takes a 11 | long time to run, it slows down our iteration cycle. This means finding a 12 | *pragmatic* balance between thorough testing, and the speed of our test suite, 13 | as well as always iterating on our testing infrastructure. 14 | 15 | 16 | ## Testing Framework 17 | 18 | [`pytest`](https://docs.pytest.org/en/latest/) is the default framework we use 19 | for testing. 20 | 21 | 22 | ## Test Suite Structure 23 | 24 | ### Test Module Naming/Organization 25 | 26 | Where possible, test suites should mirror the module structure of their 27 | corresponding package. For example, the 28 | [`eth-abi`](https://github.com/ethereum/eth-abi) package has a submodule called 29 | [`eth_abi.grammar`](https://github.com/ethereum/eth-abi/blob/main/eth_abi/grammar.py). 30 | Accordingly, it has a testing module called 31 | [`tests.grammar`](https://github.com/ethereum/eth-abi/blob/main/tests/grammar.py) 32 | that contains testing routines for resources provided by the `eth_abi.grammar` 33 | module. 34 | 35 | Test suites may also include a `tests.end_to_end` module that defines 36 | functional testing of a package's public API. This might include common tasks 37 | performed with a package that compose different individual components of that 38 | package. 39 | 40 | ### Test Case Naming 41 | 42 | Test case names should adhere to the following format where possible: 43 | 44 | ``` 45 | def test__: 46 | ... 47 | ``` 48 | 49 | For example, test cases for a function called `foo` might be named as follows: 50 | 51 | ```python 52 | def test_foo_does_kung_fu(): 53 | kung_fu = foo() 54 | 55 | def test_foo_raises_value_error(): 56 | with pytest.raises(ValueError): 57 | foo(...) 58 | 59 | ... 60 | ``` 61 | 62 | > **Note:** At the time of writing this section on test suite structure, many 63 | > ethereum python packages do not follow all of these guidelines. Therefore, 64 | > this section is meant to describe the desired test suite structure for future 65 | > packages and is also meant to inform refactoring efforts for existing 66 | > packages. 67 | 68 | 69 | ## Using Mocks 70 | 71 | The use of mock objects such as those provided by the 72 | [`pytest-mock`](https://pypi.python.org/pypi/pytest-mock) library should be 73 | considered an anti-pattern. 74 | 75 | ### Mocks are to be avoided 76 | 77 | Because: 78 | 79 | - Mock based tests tend to test the implementation and not the feature functionality 80 | - It is very easy to have mock-based tests pass, when the actual code fails 81 | - Mocking can be leaky if not done correctly: leaving objects monkey-patched after test cleanup, etc. 82 | 83 | ### Signal that your code is incorrectly structured 84 | 85 | In many cases, if you find yourself needing to mock objects in order to write 86 | your tests, this is a signal that your code may need to be refactored to make 87 | functionality more modular and easier to test in isolation. 88 | 89 | ### How to refactor an implementation to avoid mocking 90 | 91 | Consider the following steps when looking at your implementation if it seems the only easy way to test some function's logic is via mocking: 92 | 93 | 1. Prefer small, pure functions. They are easier to test if they just do one thing and only require the minimum amount of stuff required to do their job. Stateful inputs are harder to test as you have to recreate the context before you can see the effect of the routine on your parameters. 94 | 2. If (1) is not enough to expose just the functionality you want to test, you can introduce a function beyond what you may normally think is helpful to isolate the functionality you want to test. 95 | 96 | An example: 97 | 98 | ```python 99 | def foo(bar): 100 | baz = load_baz_from(bar) 101 | for i in range(ROUND_COUNT): 102 | some_value = compute_quux_given(i) 103 | quux = compute_quux_for(baz, some_value) 104 | return quux 105 | ``` 106 | 107 | We have a function `foo` that computes a value based on an input `bar` by executing an auxillary routine for some number of rounds `ROUND_COUNT`. To test the auxillary routine (here the body of the `for` loop), we can extract it from `foo`. 108 | 109 | ```python 110 | def _compute_quux(baz, i): 111 | some_value = compute_quux_given(i) 112 | return compute_quux_for(baz, some_value) 113 | 114 | def foo(bar): 115 | baz = load_baz_from(bar) 116 | for i in range(ROUND_COUNT): 117 | quux = _compute_quux(baz, i) 118 | return quux 119 | ``` 120 | 121 | We can now test `_compute_quux` independently of `foo` even if it didn't seem useful to pull out this inner function when writing the first implementation. Note we prefix the inner function with a `_` primarily to indicate that this function is "private" to the module it is defined in. 122 | 123 | 3. If (2) is not enough to get at the functionality you want to test, you can also refactor so that the explicit behavior you want to check is a parameter to the function. A default argument can be supplied so that callers of the function have no additional burden but the option to customize the function is still available in a testing context. 124 | 125 | ```python 126 | def foo(bar, quux_provider=_compute_quux): 127 | baz = load_baz_from(bar) 128 | for i in range(ROUND_COUNT): 129 | quux = quux_provider(baz, i) 130 | return quux 131 | 132 | # at the call site: 133 | the_thing = foo(bar) 134 | 135 | # during testing: 136 | constant_quux_provider = lambda _*: 2 137 | the_thing = foo(bar, quux_provider=constant_quux_provider) 138 | assert the_thing == the_expected_thing 139 | ``` 140 | 141 | ### When all else fails 142 | 143 | Mocks **can** be ok when all other options are exhausted, or it is the pragmatic solution, or some other reason to ignore the bad things. 144 | 145 | ### But maybe don't mock things 146 | 147 | A recurring pattern in our projects is a Backend Pattern, where you can swap in providers of functionality. Instead of mocking things that the code wasn't designed to do, try implementing the Backend Pattern, and swap in a dummy backend to stub out responses as fixtures. This provides most of the benefits of mocking, without sidestepping big swaths of code. You have more confidence that the majority of the feature was tested with the dummy backend. Then when you need to test a "live" backend, you can add use a few, targetted tests. 148 | 149 | ## Tips and Tricks 150 | 151 | ### Segfault during testing 152 | 153 | If pytest crashes with `Segmentation fault (core dumped)` it can be tricky to understand what happened, because you get no traceback printed. Two common causes are: infinite recursion that overflows the stack, and a compiled dependency bug. The first being more common. 154 | 155 | #### Identifying an infinite recursion 156 | 157 | One trick is to add this to your `tests/conftest.py`: 158 | ```py 159 | import sys 160 | 161 | def trace(frame, event, arg): 162 | print("%s, %s:%d" % (event, frame.f_code.co_filename, frame.f_lineno)) 163 | return trace 164 | 165 | sys.settrace(trace) 166 | ``` 167 | 168 | This will print out every line as it happens. You can then capture output to a file with `pytest ... --capture=no >trace.log` so that you don't assault your terminal screen. The final lines of the log should give you a hint where your infinite recursion is happening. 169 | 170 | [source](https://stackoverflow.com/a/2663863/8412986) 171 | 172 | #### Identifying a compiled dependency bug 173 | 174 | On linux you can use gdb to backtrace the crash: 175 | ```sh 176 | gdb python 177 | (gdb) -m pytest tests 178 | ## wait for segfault ## 179 | (gdb) backtrace 180 | ## stack trace of the c code 181 | ``` 182 | 183 | [source](https://stackoverflow.com/a/2664232/8412986) 184 | -------------------------------------------------------------------------------- /documentation.md: -------------------------------------------------------------------------------- 1 | # Documentation 2 | 3 | Good documentation will lead to quicker adoption and happier users. It should 4 | also lead to fewer requests for help and support. Documentation also serves to 5 | draw a clear line between public and private APIs. 6 | 7 | There is no one-size-fits-all approach to good documentation. Whether a documentation 8 | is considered good or not still remains a highly subjective thing after all. 9 | 10 | With that in mind, we still try to set up a rough guideline on how to create solid 11 | documentation for the software that we produce. 12 | 13 | ## Tooling 14 | 15 | We use [`Sphinx`](http://www.sphinx-doc.org/en/stable/) to create documentation for 16 | all Python projects. Sphinx is the de facto documentation generator in the Python 17 | eco-system because it provides lots of nice features specific to documenting Python 18 | code. 19 | 20 | We tend to host our documentation using the [ReadTheDocs](https://readthedocs.com/) 21 | service. 22 | 23 | The `README.md` file should always have a link to the hosted documentation. 24 | 25 | ### Robust examples 26 | 27 | Documentation that contains broken examples is **very** annoying for the reader. Therefore it is 28 | required to have a dedicated CI job that runs `doctest` to validate the correctness of the 29 | documentation. 30 | 31 | #### `doctest` code examples 32 | 33 | Code examples can be written using `.. doctest::` code blocks to ensure they are executed as 34 | regular code during the `doctest` CI run. 35 | 36 | Examples that use this feature are effectively structured as if they were written in an interactive 37 | Python session. 38 | 39 | Example: 40 | 41 | ``` 42 | .. doctest:: 43 | 44 | >>> current_vm = chain.get_vm() 45 | >>> account_db = current_vm.state.account_db 46 | >>> account_db.get_balance(SOME_ADDRESS) 47 | 10000000000000000000000 48 | ``` 49 | 50 | #### `literalinclude` code examples 51 | 52 | Sometimes, writing code examples using only the `.. doctest::` directive can feel a bit limiting 53 | and effectively clash with other desired properties of the example (e.g. order, brevity). There's 54 | another effective way to guarantee examples in the documentation continue to work as the 55 | underlying software evolves. 56 | 57 | We can write examples as standalone code examples that are tested through unit tests and then use 58 | the `.. literalinclude::` directive to take fine-grained control over the exposed parts that get 59 | exposed in the documentation. 60 | 61 | Example: 62 | 63 | ``` 64 | .. literalinclude:: ../../../trinity-external-plugins/examples/peer_count_reporter/peer_count_reporter_plugin/plugin.py 65 | :language: python 66 | :pyobject: PeerCountReporterPlugin 67 | :end-before: def configure_parser 68 | ``` 69 | 70 | ## General structure 71 | 72 | The following structure serves as a rough generalization on how to break down 73 | the documentation in different sections. It also achieves a side navigation 74 | that looks clean and makes it easy to navigate. 75 | 76 | **General** 77 | - Introduction 78 | - Quickstart 79 | - Release Notes 80 | 81 | **Fundamentals** 82 | - API 83 | - Guides 84 | - Cookbook (optional) 85 | 86 | **Community** 87 | - Contributing 88 | - Code of Conduct 89 | 90 | 91 | ### General 92 | 93 | #### Introduction 94 | 95 | This is the main landing page. It should contain: 96 | 97 | - Abstract (What is this?) 98 | - Goals (What do we want?) 99 | - Status (Where are we?) 100 | - Further reading (Links) 101 | - Quickstart 102 | - Source Code 103 | - Communication channels 104 | - Getting involved 105 | 106 | #### Quickstart 107 | 108 | The Quickstart is a very simple hands-on guide, sketching a **happy path** scenario for 109 | those who have just discovered the project. It demos how to install the software and 110 | how to use it in the most simplest way possible. 111 | 112 | It should also contain links to tell the reader what to explore next. Notice that the 113 | Quickstart is really just a guide (see Guides section further down) that is referenced 114 | in the side navigation as well as in the introduction to be exposed more prominently. 115 | 116 | #### Release Notes 117 | 118 | The release notes inform the reader about the progress of the project. They are continuously 119 | written as new releases are published. 120 | 121 | ### Fundamentals 122 | 123 | #### API 124 | 125 | API docs are the first level of documentation. They are the closest to actual source code and 126 | simply document how an API works. They do not necessarily give a good perspective about the bigger 127 | picture though. 128 | 129 | ##### Docstrings 130 | 131 | Public APIs are expected to be annotated with docstrings as seen in the following example. 132 | 133 | ```python 134 | 135 | def add_transaction(self, 136 | transaction: BaseTransaction, 137 | computation: BaseComputation, 138 | block: BaseBlock) -> Tuple[Block, Dict[bytes, bytes]]: 139 | """ 140 | Add a transaction to the given block and 141 | return `trie_data` to store the transaction data in chaindb in VM layer. 142 | 143 | Update the bloom_filter, transaction trie and receipt trie roots, bloom_filter, 144 | bloom, and used_gas of the block. 145 | 146 | :param transaction: the executed transaction 147 | :param computation: the Computation object with executed result 148 | :param block: the Block which the transaction is added in 149 | 150 | :return: the block and the trie_data 151 | """ 152 | ``` 153 | 154 | Docstrings are written in reStructuredText and allow certain type of directives. 155 | 156 | The `:param:` and `:return:` directives are being used to describe parameters as well as the return 157 | value. Usage of `:type:` and `:rtype:` directives is discouraged as sphinx automatically reads 158 | and displays the types from the source code type definitions making any further use of 159 | `:type:` and `:rtype:` obsolete and unnecessarily verbose. 160 | 161 | 162 | Whenever other APIs in the form of classes or methods are mentioned we should use the linking 163 | syntax as it will allow the reader to jump to the corresponding API docs. 164 | 165 | Example: `:class:`\`~eth_typing.misc.Address` 166 | 167 | ##### Grammar 168 | 169 | We use imperative, present tense to describe APIs: “return” not “returns” 170 | 171 | One way to test if we have it right is to complete the following sentence. 172 | 173 | If we call this API it will: __________________________ 174 | 175 | If that sounds right, it probably is. 176 | 177 | ##### Doc generation 178 | 179 | Sphinx can generated the API docs directly from the embedded docstrings in the source code. The 180 | `rst` files for API docs are mostly concerned about how the order of the described types or short 181 | intro paragraphs. 182 | 183 | ``` 184 | Chain 185 | ===== 186 | 187 | BaseChain 188 | --------- 189 | 190 | .. autoclass:: eth.chains.base.BaseChain 191 | :members: 192 | 193 | Chain 194 | ----- 195 | 196 | .. autoclass:: eth.chains.base.Chain 197 | :members: 198 | ``` 199 | 200 | #### Guides 201 | 202 | While solid API documentation is very important for those who know what they are looking for, 203 | it does not cover all the essential aspects of the entire documentation spectrum. 204 | 205 | Often, a narrative style documentation that walks the user through specific use cases is more user 206 | friendly. This is especially true for those who are not yet familiar with the software and 207 | lack a good feel for existing APIs. 208 | 209 | ##### Narrative Perspective 210 | 211 | Each narrative is written from a particular point of view and it is important to remain consistent 212 | in its use. 213 | 214 | There are two main styles of writing that are generally used for technical documentation and there 215 | really is no *right* or *wrong* when it comes to picking one over the other. 216 | 217 | Here are two prominent examples: 218 | 219 | *React uses second-person "you" form* 220 | 221 | ![image](https://user-images.githubusercontent.com/521109/48718244-47f04180-ec1b-11e8-9978-85173b4e78e3.png) 222 | 223 | *Rust uses first-person "we" form* 224 | 225 | ![image](https://user-images.githubusercontent.com/521109/48718325-70783b80-ec1b-11e8-8665-dd892edc1a51.png) 226 | 227 | The *second-person "you" form* is more casual as it directly speaks to the reader whereas the 228 | *first-person "we" form* is more formal and more popular in scientific writing. 229 | 230 | We use the *first-person "we" form* only. 231 | 232 | Examples: 233 | 234 | **Correct** 235 | > We can send the transaction after we sign it. 236 | 237 | **Wrong** 238 | 239 | > You can send the transaction after you sign it. 240 | 241 | **Wrong** 242 | 243 | > I can send the transaction after I sign it. 244 | 245 | ##### Writing problem driven 246 | 247 | It's tempting to write guides that drop the user right within a solution that highlights exciting 248 | features. While this may make a lot of sense for the writer, often readers will find themselves 249 | wondering: "Wait! **Why** are we doing this in the first place?" 250 | 251 | Therefore, it's a good idea for guides to be written in a *problem-driven* manner whenever possible. 252 | We want the reader to first understand **why** they want to read a section of documentation. What is 253 | the problem that we will learn how to fix or what are the possible use cases for the features that 254 | we are about to learn? 255 | 256 | 257 | #### Cookbook 258 | 259 | Between concise API docs and lengthy guides, there's often room for a third format of 260 | documentation. A cookbook is a collection of small recipes that demo the use of APIs through short 261 | examples. Things that do not need a full grown guide but also aren't entirely clear by just looking 262 | at the API docs are perfect candidates to go into a cookbook. 263 | 264 | Writing cookbooks takes less effort compared to writing guides and it's an effective tactic to 265 | start collecting short examples in a cookbook and later turn some of them into more in-depth 266 | guides. 267 | 268 | ### Community 269 | 270 | #### Contributing 271 | 272 | This section should describe how to get involved. 273 | 274 | - Setting up a work environment 275 | - Running the tests 276 | - Sending PRs 277 | - Developer communication channels 278 | 279 | #### Code of Conduct 280 | 281 | In the interest of fostering an open and welcoming environment, we as contributors and maintainers 282 | pledge to making participation in our project and our community a harassment-free experience for 283 | everyone. Therefore, every project documentation siteshould contain 284 | [this](https://www.contributor-covenant.org/version/1/4/code-of-conduct.html) code of conduct. 285 | -------------------------------------------------------------------------------- /git-and-github.md: -------------------------------------------------------------------------------- 1 | # Git 2 | 3 | We use `git` for version control and github for hosting of all of our code 4 | repositories. 5 | 6 | 7 | ## Commit Signing 8 | 9 | You **should** setup your local git to sign all of your commits. This can be 10 | done with `git config commit.gpgsign true` 11 | 12 | You will also need to inform github of your private key so that it can *verify* 13 | your commits. See github's documentation on that 14 | [here](https://help.github.com/articles/signing-commits-using-gpg/) 15 | 16 | 17 | ## Labels 18 | 19 | We tend to use the following standard labels in our repositories. 20 | 21 | - Ready to Merge 22 | - Needs Review 23 | - Work in Progress 24 | - Bug 25 | - Good for Bounty 26 | 27 | Many repositories may implement extra labels as well. You as the author of 28 | pull requests are responsible for applying the appropriate labels to pull 29 | requests, as well as removing labels when appropriate. 30 | 31 | 32 | ## Pull Requests 33 | 34 | We are a distributed team. The primary way we communicate about our code is 35 | through github via pull requests. 36 | 37 | * When you start work on something you should have a pull request opened that 38 | same day. 39 | * Mark unfinished pull requests with the "Work in Progress" label. 40 | * Pull requests **should** always be reviewed by another member of the team 41 | prior to being merged. 42 | * Obvious exceptions include very small pull requests. 43 | * Less obvious examples include things like time-sensitive fixes. 44 | * You should not expect feedback on a pull request which is not passing CI. 45 | * Obvious exceptions include soliciting high-level feedback on your approach. 46 | 47 | 48 | Large pull requests (above 200-400 lines of code changed) cannot be effectively 49 | reviewed. If your pull request exceeds this threshold you **should** make 50 | every effort to divide it into smaller pieces. 51 | 52 | You as the person opening the pull request should assign a reviewer. 53 | 54 | 55 | ## Commit Hygiene 56 | 57 | We do not have any stringent requirements on how you commit your work, however 58 | you should work towards the following with your git habits. 59 | 60 | ### Logical Commits 61 | 62 | This means that each commit contains one logical change to the code. For example: 63 | 64 | - commit `A` introduces new API 65 | - commit `B` deprecates or removes the old API being replaced. 66 | - commit `C` modifies the configuration for CI. 67 | 68 | This approach is sometimes easier to do *after* all of the code has been 69 | written. Once things are complete, you can `git reset main` to unstage all 70 | of the changes you've made, and then re-commit them in small chunks using `git 71 | add -p`. 72 | 73 | ### Rebasing 74 | 75 | You should be using `git rebase` when there are *upstream* changes that you 76 | need in your branch. You **should not** use `git merge` to pull in these 77 | changes. 78 | 79 | 80 | ### Commit Messages 81 | 82 | We don't care much about commit messages other than that they be sufficiently 83 | descriptive of what is being done in the commit. 84 | 85 | The *correct* phrasing of a commit message. 86 | 87 | - `fix bug #1234` (correct) 88 | - `fixes bug #1234` (wrong) 89 | - `fixing bug #1234` (wrong) 90 | 91 | One way to test if you have it right is to complete the following sentence. 92 | 93 | > If you apply this commit it will ________________. 94 | 95 | 96 | ## Code Review 97 | 98 | Every team member is responsible for reviewing code. The designations :speech_balloon:, :heavy_check_mark:, and :x: **should** be left by a reviewer as follows: 99 | 100 | - :speech_balloon: (Comment) should be used when there is not yet an opinion on overall validity of complete PR, for example: 101 | - comments from a partial review 102 | - comments from a complete review on a Work in Progress PR 103 | - questions or non-specific concerns, where the answer might trigger an expected change before merging 104 | - :heavy_check_mark: (Approve) should be used when the reviewer would consider it acceptable for the contributor to merge, *after addressing* all the comments. For example: 105 | - style nitpick comments 106 | - compliments or highlights of excellent patterns ("addressing" might be in the form of a reply that defines scenarios where the pattern could be used more in the code, or a simple :+1:) 107 | - a specific concern, where multiple reasonable solutions can adequately resolve the concern 108 | - a Work in Progress PR that is far enough along 109 | - :x: (Request changes) should be used when the reviewer considers it unacceptable to merge without another review of changes that address the comments. For example: 110 | - a specific concern, without a satisfactory solution in mind 111 | - a specific concern with a satisfactory solution provided, but *alternative* solutions **may** be unacceptable 112 | - any concern with significant subtleties 113 | 114 | Contributors **should** react to reviews as follows: 115 | - :x: if *any* review is marked as "Request changes": 116 | - make changes and/or request clarification 117 | - **should not** merge until reviewer has reviewed again and changed the status 118 | - (none) if there are no reviews, contributor should not merge. 119 | - :speech_balloon: if *all* reviews are comments, then address the comments. Otherwise, treat as if no one has reviewed the PR. 120 | - :heavy_check_mark: if *at least one* review is Approved, contributor **should** do these things before merging: 121 | - make requested changes 122 | - if any concern is unclear in any way, ask the reviewer for clarification before merging 123 | - solve a concern with suggested, or alternative, solution 124 | - if the reviewer's concern is clearly a misunderstanding, explain and merge. Contributor should be on the lookout for followup clarifications on the closed PR 125 | - if the contributor simply disagrees with the concern, it would be best to communicate with the reviewer before merging 126 | - if the PR is approved as a work-in-progress: consider reducing the scope of the PR to roughly the current state, and merging. (multiple smaller PRs is better than one big one) 127 | 128 | It is also recommended to use the emoji responses to signal agreement or that 129 | you've seen a comment and will address it rather than replying. This reduces 130 | github inbox spam. 131 | 132 | Everyone is free to review any pull request. 133 | 134 | Recommended Reading: 135 | 136 | - [How to Do Code Reviews Like a Human](https://mtlynch.io/human-code-reviews-1/) 137 | 138 | 139 | ## Merging 140 | 141 | Once your pull request has been *Approved* it may be merged at your discretion. In most cases responsibility for merging is left to the person who opened the pull request, however for simple pull requests it is fine for anyone to merge. 142 | 143 | When merging, please avoid the `Create a merge commit` option as this may have undesired outcomes by having each commit on the main branch. Ideally, the commits have been cleaned up, following best practices outlined in [Commit Hygiene](#commit-hygiene) above. 144 | 145 | > Use `Squash and merge` if you want to combine all your commits into a single commit on the main branch. Make sure you have rebased your branch prior to choosing this option. 146 | 147 | > Use `Rebase and merge` if you want to keep the commits and ensure it is applied starting from HEAD of the main branch. 148 | 149 | If substantive changes are made **after** the pull request has been marked *Approved* you should ask for an additional round of review. 150 | 151 | 152 | ## New Projects 153 | 154 | Many projects follow a similar pattern, and there are 155 | certain configurations and links that need to be repeated 156 | in each repository. One suggested way to do that is to 157 | clone [this template repo](https://github.com/carver/ethereum-python-project-template/) 158 | and then push to a new repository once you've named your new project. 159 | 160 | One benefit of working that way is that the template is 161 | maintained as new practices are adopted. Changes to the 162 | template are straightforward to merge in to projects that 163 | are based on it, so global maintenance can be 164 | propagated easily. 165 | 166 | 167 | ## Tips & Tricks 168 | 169 | ### Developing against locally modified dependencies 170 | 171 | Let's assume we have a project A depending on project B. We can often find us in a situation where 172 | we want to run project A with a modified version of project B. For instance, we might be working 173 | on upcoming changes for project B and want to test drive them in project A. Or maybe we just want 174 | to add some extra logging to project B in order to debug some related error that we are seeing in 175 | project A. 176 | 177 | Fortunately, `pip` allows us to add dependencies from any local directory without having actual 178 | wheel package for it. 179 | 180 | Let's assume we have project A and B sitting next to each other in the same directory. 181 | 182 | ``` 183 | ├── project-a 184 | └── project-b 185 | ``` 186 | 187 | We can install the local version of project B inside project A as follows: 188 | 189 | ``` 190 | cd project-a 191 | pip install -e ../project-b 192 | ``` 193 | 194 | Internally, this creates a link to `../project-b` so that any changes made to it are immediately 195 | reflected in project A. 196 | 197 | ### Pull Request with unreleased external dependencies 198 | 199 | Consider we have project A depending on project B. It's a common scenario that we 200 | are making changes to project B that will require project A to adapt to it (a "breaking change"). 201 | While the changes to project B may still be under review and therefore unreleased, it can be 202 | valuable to open a PR against project A to show the migration path for these upcoming changes. 203 | 204 | For us to be able to already use the new, unreleased version, we can set the dependency version of 205 | package B to a temporary tag, commit or branch on GitHub. This ensures the PR can do a proper CI 206 | run against the upcoming changes, allowing it to reveal potential problems that otherwise may 207 | go unnoticed. 208 | 209 | We can use the following syntax to specify a dependency directly from GitHub without creating an 210 | actual package for it. 211 | 212 | ``` 213 | "package-name@git+https://github.com/owner/repository.git@tag" 214 | ``` 215 | 216 | Let's assume package B is called `lahja` and the `setup.py` file normally specifies the version in 217 | the `install_requires` or `extras_requires` section as follows. 218 | 219 | ``` 220 | ... 221 | "web3==4.4.1", 222 | "lahja==0.11.2", 223 | "termcolor>=1.1.0,<2.0.0", 224 | ... 225 | ``` 226 | 227 | For the purpose of the PR, we can specify a temporary version such as. 228 | 229 | ``` 230 | ... 231 | "web3==4.4.1", 232 | "lahja@git+https://github.com/cburgdorf/lahja-1.git@christoph/perf/filter-predicate" 233 | "termcolor>=1.1.0,<2.0.0", 234 | ... 235 | ``` 236 | 237 | During the lifecycle of the PR the dependency version should be changed to the proper 238 | version of the package as soon as a released version exists. 239 | 240 | ### Fetch Pull Requests without manually adding remotes 241 | 242 | We often want or need to run code that someone proposes in a PR. Typically this involves adding the remote of the PR author locally and then fetching their branches. 243 | 244 | Example: 245 | 246 | ``` 247 | git remote add someone https://github.com/someone/reponame.git 248 | git fetch someone 249 | git checkout someone/branch-name 250 | ``` 251 | 252 | With an increasing number of different contributors this workflow becomes tedious. 253 | Luckily, there's a little trick that greatly improves the workflow as it lets us 254 | pull down any PR without adding another remote. 255 | 256 | To do this, we just have to add the following line in the `[remote "origin"]` 257 | section of the `.git/config` file in our local repository. 258 | 259 | ``` 260 | fetch = +refs/pull/*/head:refs/remotes/origin/pr/* 261 | ``` 262 | 263 | Then, checking out a PR locally becomes as easy as: 264 | 265 | ``` 266 | git fetch origin 267 | git checkout origin/pr/ 268 | ``` 269 | 270 | >Replace `origin` ☝ with the actual name (e.g. `upstream`) that we use for the 271 | remote that we want to fetch PRs from. 272 | 273 | Notice that fetching PRs this way is *read-only* which means that in case we do 274 | want to contribute back to the PR (and the author has this enabled), we would 275 | still need to add their remote explicitly. 276 | -------------------------------------------------------------------------------- /style-guide.md: -------------------------------------------------------------------------------- 1 | # Style Guide 2 | 3 | This document defines a style guide for python code. It includes both 4 | objective rules as well as some subjective preferences. 5 | 6 | 7 | # RFC2119 8 | 9 | The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", 10 | "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be 11 | interpreted as described in RFC 2119. 12 | 13 | 14 | ## What code does this apply to 15 | 16 | This applies to both the library code and the test modules. 17 | 18 | > The code within the test modules may be treated with slightly looser 19 | > standards than the main library code. 20 | 21 | 22 | ## Why would you want to use this 23 | 24 | ### Opinionated consistency 25 | 26 | When multiple people are working on the same body of code, it is important that 27 | they write code that conforms to a similar style. It often doesn't matter as 28 | much *which* style, but rather that they conform to *one* style. 29 | 30 | A good term for this is **opinionated consistency**. It's more important that 31 | the codebase be consistently styled and thus, there needs to be a definition of 32 | what that style is. 33 | 34 | ## Philosophy 35 | 36 | TODO 37 | 38 | 39 | # Linting 40 | 41 | ## PEP8 42 | 43 | All code **must** conform to [PEP8](https://www.python.org/dev/peps/pep-0008/) 44 | with the following exceptions. 45 | 46 | * Line length of 100 (instead of the default 80). 47 | 48 | 49 | ## Flake8 50 | 51 | All code **must** pass linting using the `flake8` command line tool with all ignore 52 | rules disabled. 53 | 54 | ```bash 55 | flake8 --ignore="" path-to-lint/ 56 | ``` 57 | 58 | 59 | ### Exceptions 60 | 61 | Code that do not pass ``flake8`` linting rules but which is either required for 62 | some reason or would be *worse* by conforming to the linter can be excluded 63 | using a `noqa` comment which excludes the specific rule. 64 | 65 | 66 | ```python 67 | # in __init__.py 68 | from something import ( # noqa: F401 69 | Something, 70 | ) 71 | ``` 72 | 73 | 74 | The `noqa` comment should not be used in the catch-all format: 75 | 76 | ``` 77 | # don't do this 78 | a = a+b # noqa 79 | ``` 80 | 81 | # Subjective Style Preferences 82 | 83 | The following style preferences cover areas where PEP8 does not establish a 84 | single way in which code should be formatted. 85 | 86 | 87 | ## Whitespace 88 | 89 | No empty lines at the top of modules. 90 | 91 | 92 | ## Import Statements 93 | 94 | ### Overall Ordering 95 | 96 | All imports should be ordered as follows: 97 | 98 | 1. Standard Library 99 | * Ordered alphabetically 100 | 2. Installed Libraries 101 | * Ordered by approximate popularity 102 | * The `requests` library is more popular than `web3.py` library 103 | 3. Local Library 104 | * Ordered hierarchically 105 | * The `my_module.core` should be above `my_module.utils.encoding` 106 | * Ordered alphabetically within hierarchy 107 | * The `my_module.utils.alphabet` should come before `my_module.utils.zebra` 108 | 109 | See example below with comment annotations. 110 | 111 | ```python 112 | # standard library 113 | import collections 114 | import functools 115 | import os 116 | import sys 117 | 118 | # installed libraries 119 | import requests 120 | 121 | from web3 import ( 122 | Web3, 123 | ) 124 | 125 | # local 126 | from my_module import ( 127 | something, 128 | ) 129 | from my_module.subsystem import ( 130 | others, 131 | ) 132 | 133 | from my_module.utils.encoding import ( 134 | encode_thing, 135 | ) 136 | ``` 137 | 138 | ### Individual Ordering 139 | 140 | Within a single import statement, the imported items should be sorted alphabetically 141 | 142 | ```python 143 | from something import ( 144 | aardvark, 145 | baboon, 146 | monkey, 147 | zebra, 148 | ) 149 | ``` 150 | 151 | ### Formatting 152 | 153 | Single imports **may** be done in a single line. 154 | 155 | ```python 156 | from something import Thing 157 | ``` 158 | 159 | Multiple imports **must** be split across multiple lines, one per line. 160 | 161 | ```python 162 | from something import ( 163 | ThingOne, 164 | ThingTwo, 165 | ) 166 | ``` 167 | 168 | ### Whitespace 169 | 170 | Standard library imports which import the top-level module do not need whitespace. 171 | 172 | ```python 173 | import collections 174 | import functools 175 | ``` 176 | 177 | Standard library imports which import things from the module **should** be separated with a single line of whitespace. 178 | 179 | ```python 180 | import collections 181 | 182 | from datetime import ( 183 | datetime, 184 | ) 185 | 186 | import functools 187 | ``` 188 | 189 | Installed library imports **must** be separated by a single line of whitespace. 190 | 191 | ```python 192 | import collections 193 | import functools 194 | 195 | import requests # 3rd party 196 | 197 | import pyethereum # 3rd party 198 | 199 | from local_module import ( 200 | ... 201 | ) 202 | ``` 203 | 204 | Local library imports **should** be separated by a single line of whitespace based on hierarchy. 205 | 206 | 207 | ```python 208 | # top level module import 209 | from my_module import ( 210 | MainThing, 211 | ) 212 | 213 | # 1st level module import 214 | from my_module.submodule import ( 215 | SubThing, 216 | ) 217 | 218 | # 2nd level module import 219 | from my_module.utils.encoding import ( 220 | encode_thing, 221 | ) 222 | from my_module.utils.formatting import ( 223 | format_thing, 224 | ) 225 | 226 | # local import 227 | from .helpers import ( 228 | do_thing, 229 | ) 230 | ``` 231 | 232 | 233 | ### Avoid `from x import y` for most standard library imports 234 | 235 | For most standard library usage, you should access the necessary 236 | functions/classes/etc from the module itself. 237 | 238 | ```python 239 | # good 240 | import functools 241 | 242 | do_specific_thing = functools.partial(do_thing, x=3) 243 | 244 | # less-good 245 | from functools import partial 246 | 247 | do_specific_thing = partial(do_thing, x=3) 248 | ``` 249 | 250 | This reduces cognitive overhead when reading code since a reader may already 251 | **know** what `functools.partial` does. In the case where `partial` is 252 | imported on its own, the reader must go find that import to verify that the 253 | `partial` function is in fact `functools.partial`. 254 | 255 | Exceptions to this rule: 256 | 257 | #### The `abc.abstractmethod` decorator 258 | 259 | This decorator should be imported directly as it is arguably easier to read 260 | than using it off of the `abc` namespace. 261 | 262 | ```python 263 | class MyClass: 264 | @abc.abstractmethod # harder to read 265 | def my_method(self): 266 | ... 267 | 268 | @abstractmethod # easier to read 269 | def my_other_method(self): 270 | ... 271 | ``` 272 | 273 | 274 | #### Don't use `from datetime import datetime` 275 | 276 | You **should not** import the `datetime` class from the `datetime` module. 277 | Both the class and module share the same name. By importing the `datetime` 278 | class, you introduce ambiguity for a reader as to whether the `datetime` 279 | variable they are looking at is the class or the module. 280 | 281 | ```python 282 | # good 283 | import datetime 284 | 285 | x = datetime.datetime(year=2017, month=10, day=1) 286 | 287 | # bad 288 | from datetime import datetime 289 | x = datetime(year=2017, month=10, day=1) 290 | ``` 291 | 292 | 293 | ### Relative and Absolute imports 294 | 295 | Relative imports **should** only be used for importing from the same module 296 | level. Relative imports are mentally difficult to traverse and are not 297 | portable if the module is moved. 298 | 299 | ```python 300 | # good 301 | from .helpers import ( 302 | do_helpful_thing, 303 | ) 304 | 305 | # less-good 306 | from ..helpers import ( # should instead use absolute import path. 307 | do_helpful_thing, 308 | ) 309 | ``` 310 | 311 | 312 | ## Raising Exceptions 313 | 314 | When using the ```raise NewException() from exc``` form, the new exception must have a message 315 | passed to its constructor, otherwise when it is stringified (e.g when it is logged), we get an 316 | empty string. In most cases we can reuse the message (and other arguments passed to the original 317 | exception) with the following: 318 | 319 | ``` 320 | try: 321 | ... 322 | except SomeException as exc: 323 | ... 324 | raise NewException(*exc.args) from exc 325 | ``` 326 | 327 | 328 | ## Multi-line statements 329 | 330 | We have traditionally avoided use of forward slash style multi-line statements. 331 | 332 | 333 | ```python 334 | result = something_long \ 335 | ``` 336 | 337 | ### Dangling Commas 338 | 339 | Multiline statements should use dangling commas. The reason for this is that 340 | it reduces the size of the diff when new items are added. 341 | 342 | ```python 343 | # good 344 | x = [ 345 | item_a, 346 | item_b, 347 | item_c, # <--- dangling comma 348 | ] 349 | 350 | # bad 351 | x = [ 352 | item_a, 353 | item_b, 354 | item_c # <--- doesn't have a dangling comma 355 | ] 356 | ``` 357 | 358 | 359 | ### Long Strings 360 | 361 | You **should** use the following pattern for long strings. 362 | 363 | ```python 364 | raise ValueError( 365 | "This exception message is exceptionally long and thus it " 366 | "requires that the message be broken up across multiple lines " 367 | "so that the linting gods do not smite us." 368 | ) 369 | ``` 370 | 371 | 372 | ### Formatting log messages 373 | 374 | You **should** use `%` style string formatting for log messages. This allows for logging aggregators like [Sentry](https://sentry.io/) to group logging messages. 375 | 376 | ```python 377 | logger.info("%s went %s wrong", 'Something', 'very') 378 | ``` 379 | 380 | 381 | ### Complex statements 382 | 383 | Statements with lots of operators can be broken up across multiple lines more 384 | easily if the entire statement is wrapped in parenthesis. This allows for line 385 | breaks that would otherwise not be allowed. 386 | 387 | ```python 388 | result = ( 389 | (variable_a + variable_b) * something_kind_of_long / 390 | some_divisor 391 | ) 392 | ``` 393 | 394 | 395 | ### Comprehensions 396 | 397 | When a list/dict/etc comprehension exceeds the 100 character line lenght it 398 | **should** be split across multiple lines with the `statement/for/in/if` 399 | each on their own lines. 400 | 401 | ```python 402 | result = [ 403 | item 404 | for item 405 | in value 406 | ] 407 | 408 | # or with an `if` statement 409 | result = [ 410 | item 411 | for item 412 | in value 413 | if item is not None 414 | ] 415 | ``` 416 | 417 | It is also acceptable to combine the `for` and `in` lines for simple comprehensions 418 | 419 | ```python 420 | result = [ 421 | item 422 | for item in value 423 | ] 424 | ``` 425 | 426 | 427 | ### One thing per line 428 | 429 | Any statement being split across multiple lines **should** include *one thing* 430 | per line. Multiline statements are inherently difficult to parse. By keeping 431 | each line down to *one thing* it reduces this difficulty. 432 | 433 | ```python 434 | result = get_things( 435 | argument_a, 436 | argument_b, 437 | argument_c, 438 | argument_d, 439 | ) 440 | ``` 441 | 442 | ### First thing on its own line 443 | 444 | Any statement being split across multiple lines **should** move the first *thing* 445 | to the next line, indented one deeper than the previous line. For example, changes 446 | to function names **should not** change indentation level of all arguments. Reading 447 | the diff is faster and less error prone when only important lines change. 448 | 449 | ```python 450 | # should 451 | result = get_things( 452 | argument_a, 453 | argument_b, 454 | argument_c, 455 | argument_d, 456 | ) 457 | 458 | # should not 459 | result = get_things(argument_a, 460 | argument_b, 461 | argument_c, 462 | argument_d, 463 | ) 464 | ``` 465 | 466 | ### Closing parenthesis/bracket/brace 467 | 468 | The closing parenthesis/bracket/brace for a multiline statement **should** be 469 | on it's own line, at the same level of indentation as the beginning of the 470 | statement. This provides an easy visual cue for where the multi-line statement 471 | begins and ends. 472 | 473 | 474 | ```python 475 | # good 476 | result = get_things( 477 | ... 478 | ) 479 | 480 | # less-good 481 | result = get_things( 482 | ...) 483 | ``` 484 | 485 | 486 | ### Function Signatures 487 | 488 | When a function signature exceeds the 100 character limit, it should be split 489 | across multiple lines with a single argument per line. The closing parenthesis 490 | for the function arguments should be on the same line as the final argument. 491 | 492 | ```python 493 | def long_signature( 494 | thing_a, 495 | thing_b, 496 | thing_c=None, 497 | thing_d=None): 498 | pass 499 | ``` 500 | 501 | ## Only use bare `else` block for validating unexpected values 502 | 503 | **Instead of** doing this: 504 | ```py 505 | if direction == 'north': 506 | print('N') 507 | elif direction == 'south': 508 | print('S') 509 | elif direction == 'west': 510 | print('W') 511 | else: 512 | print('E') 513 | ``` 514 | 515 | Prefer to use explicit `elif` blocks and a "validation" `else` block 516 | 517 | ```py 518 | if direction == 'north': 519 | print('N') 520 | elif direction == 'south': 521 | print('S') 522 | elif direction == 'west': 523 | print('W') 524 | elif direction == 'east': 525 | print('E') 526 | else: 527 | raise ValidationError(f"unrecognized direction {direction}") 528 | ``` 529 | 530 | One of the critical reasons to prefer this latter approach is to identify errors early. Even very simple-looking "switches" like the above can be buggy. Here's an example: 531 | https://github.com/ethereum/eth-account/pull/57/files#diff-56b414627669bfd488bd7d6313c1b7ceR62 532 | This above tests were passing, despite not testing 2/3 of the intended code paths. 533 | 534 | It's not so important to find the bug. What's more important is that we prevent this whole class of errors by using an else block for validation. If you must know about the specific bug above, click for explanation below: 535 |
536 | Click for bug spoiler 537 | When parameterizing pytest fixtures, the parameter is placed in `request.param` (not `request`). So the else block was *always* being triggered, and the first two paths were never running. 538 |
539 | 540 | # Typing 541 | 542 | Since all new code is required to come with type hints ([PEP 484](https://www.python.org/dev/peps/pep-0484/)), the following aims to provide answers to some common questions around working with types and `mypy`. 543 | 544 | ## Choosing the right type 545 | 546 | When working with types, we often have to choose between multiple different types that all may seem like the right candidate for the code in question. 547 | 548 | ### Function variance 549 | 550 | As a rule of thumb, the **parameters** of a function **should** be typed to accept the *most flexible* type that the function can work with. 551 | 552 | ```python 553 | # good 554 | def triple_len(seq: Sequence[T]) -> int: 555 | return len(seq) * 3 556 | 557 | res_1 = triple_len(["some", "irrelevant", "data"]) 558 | res_2 = triple_len(("some", "irrelevant", "data",)) 559 | 560 | # bad 561 | def triple_len(seq: List[T]) -> int: 562 | return len(seq) * 3 563 | 564 | res_1 = triple_len(["some", "irrelevant", "data"]) 565 | # Doesn't work if `triple_len` is limited to `List[T]` 566 | # Argument 1 to "triple_len" has incompatible type "Tuple[str, str, str]"; expected "List[]" 567 | 568 | # res_2 = triple_len(("some", "irrelevant", "data")) 569 | ``` 570 | 571 | In contrast, the **return type** of a function **should not** be more abstract than the type already known in the function body. 572 | 573 | ```python 574 | # good 575 | def create_things() -> Tuple[str, ...]: 576 | return ("some", "irrelevant", "data",) 577 | 578 | things = create_things() 579 | res = triple_len(things) 580 | 581 | # bad 582 | def create_things() -> Iterable[str]: 583 | return ("some", "irrelevant", "data",) 584 | 585 | things = create_things() 586 | # Doesn't work if `things` is falsly assumed to *only* qualify `Iterable[str]` 587 | # Type error: Argument 1 to "triple_len" has incompatible type "Iterable[str]"; expected "Sequence[]" 588 | 589 | # res = triple_len(things) 590 | ``` 591 | 592 | # Docstrings 593 | 594 | ## Preferred delimiters 595 | 596 | You **should** enclose docstrings with `"""` delimiters. That is, you **should 597 | not** enclose docstrings with `'''` delimiters. 598 | 599 | ## Layout 600 | 601 | You **should** lay out docstring content according to the conventions defined 602 | in [PEP 257](https://www.python.org/dev/peps/pep-0257/) and [PEP 603 | 287](https://www.python.org/dev/peps/pep-0287/). Where those two PEPs 604 | conflict, precedence is given to the recommendations of [PEP 605 | 287](https://www.python.org/dev/peps/pep-0287/). 606 | 607 | ## Exceptions to docstring style PEPs 608 | 609 | The aforementioned [PEP 257](https://www.python.org/dev/peps/pep-0257/) 610 | recommends that a summary of a method's functionality should appear on the 611 | first line of a docstring immediately following the opening `"""` delimiter. 612 | Departing from this recommendation, you **should** insert a newline after the 613 | opening `"""` delimiter and before the beginning of a docstring's content. For 614 | example, 615 | 616 | ```python 617 | # Do this 618 | def foo(): 619 | """ 620 | Summary of foo's functionality. 621 | 622 | Longer description of foo's functionality. 623 | """ 624 | pass 625 | 626 | # Don't do this 627 | def foo(): 628 | """Summary of foo's functionality. 629 | 630 | Longer description of foo's functionality. 631 | """ 632 | pass 633 | ``` 634 | --------------------------------------------------------------------------------