├── assets
├── logo.png
├── ai_basis.jpg
└── code_structure.jpg
├── chapter1_basic_code
├── images
│ ├── isort.jpg
│ ├── yapf.jpg
│ └── formatting.jpg
└── README.md
└── README.md
/assets/logo.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Mountchicken/CodeCookbook/HEAD/assets/logo.png
--------------------------------------------------------------------------------
/assets/ai_basis.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Mountchicken/CodeCookbook/HEAD/assets/ai_basis.jpg
--------------------------------------------------------------------------------
/assets/code_structure.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Mountchicken/CodeCookbook/HEAD/assets/code_structure.jpg
--------------------------------------------------------------------------------
/chapter1_basic_code/images/isort.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Mountchicken/CodeCookbook/HEAD/chapter1_basic_code/images/isort.jpg
--------------------------------------------------------------------------------
/chapter1_basic_code/images/yapf.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Mountchicken/CodeCookbook/HEAD/chapter1_basic_code/images/yapf.jpg
--------------------------------------------------------------------------------
/chapter1_basic_code/images/formatting.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Mountchicken/CodeCookbook/HEAD/chapter1_basic_code/images/formatting.jpg
--------------------------------------------------------------------------------
/chapter1_basic_code/README.md:
--------------------------------------------------------------------------------
1 |
2 |
3 | # Cookbook for Crafting Good Code
4 | Think of good code like a classic, well-fitting piece of clothing—it never goes out of style. Coding is both a science and an art, where neatness and logic come together. What makes some code stand out as great? Here are three key aspects:
5 | - **Readability**:
6 | - Good code should be easy to read and understand. This means using clear naming conventions, organizing code logically, and commenting where necessary to explain complex parts.
7 | - **Simplicity and Efficiency**:
8 | - Good code often follows the KISS principle ("Keep It Simple, Stupid"). It should accomplish its tasks in the simplest way possible, without unnecessary complexity.
9 | - Efficient code also performs its tasks quickly and resourcefully.
10 | - **Maintainability**:
11 | - Code should be easy to maintain and modify. This involves writing modular code, where different parts of the program are separated into distinct sections or functions that can be updated independently.
12 |
13 | In this guide, we'll dive into the essentials of crafting great code. We'll go through everything from how to name things clearly and highlight tools that make coding better and easier.
14 |
15 | ### Contents
16 | - [Cookbook for Crafting Good Code](#cookbook-for-crafting-good-code)
17 | - [Contents](#contents)
18 | - [1. Readability](#1-readability)
19 | - [Docstring](#docstring)
20 | - [Type Hinting](#type-hinting)
21 | - [Naming Conventions](#naming-conventions)
22 | - [Formatting](#formatting)
23 | - [2. Simplicity and Efficiency](#2-simplicity-and-efficiency)
24 | - [KISS Principle](#kiss-principle)
25 | - [Write Efficient Code](#write-efficient-code)
26 | - [3. Maintainability](#3-maintainability)
27 | - [Use Registry Mechanism to Manage Your Code](#use-registry-mechanism-to-manage-your-code)
28 | - [Organize Your Code by Functionality](#organize-your-code-by-functionality)
29 |
30 |
31 | ## 1. Readability
32 | Readability in code is akin to clear handwriting in a letter. It's not just about what you write, but how you present it. A well-written piece of code should speak to its reader, guiding them through its logic as effortlessly as a well-told story. Let's delve into some of the key practices that make code readable.
33 |
34 | ### Docstring
35 | A docstring, short for "documentation string," is a string literal that occurs as the first statement in a module, function, class, or method definition. Here are three most important definitions from the official Python documentation, [PEP257](https://peps.python.org/pep-0257/).
36 |
37 |
38 | Python PEP257
39 |
40 | - **1. What Should be Documented**:
41 | - PEP 257 suggests that ***all public modules, functions, classes, and methods should have docstrings***. Private methods (those starting with an underscore) are considered optional for documentation but are encouraged, especially for complex code.
42 |
43 | - **2. Docstring Format**:
44 | - Docstrings should be enclosed in triple double quotes (""").
45 | - The first line should be a short, concise summary of the object’s purpose.
46 |
47 | - **3. Multi-line Docstrings**:
48 | - For longer descriptions, the summary line should be followed by a blank line, then a more elaborate description. The detailed description may include usage, arguments, return values, and raised exceptions if applicable.
49 | - Multi-line docstrings should end with the closing triple quotes on a line by themselves.
50 |
51 |
52 |
53 | Here are two python templates for docstring of function and class that may give you a more concrete idea of how to write a docstring.
54 |
55 |
56 | Template for function
57 |
58 | ```python
59 | def function_name(param1, param2, ...):
60 | """A brief description of what the function does.
61 |
62 | A more detailed description of the function if necessary.
63 |
64 | Inputs:
65 | param1 (Type): Description of param1.
66 | param2 (Type): Description of param2.
67 |
68 | Returns:
69 | ReturnType: Description of the return value.
70 |
71 | Raises: (Optional)
72 | ExceptionType: Explanation of when and why the exception is raised.
73 |
74 | Notes: (Optional)
75 | Additional notes or examples of usage, if necessary.
76 |
77 | Examples: (Optional)
78 | >>> function_name(value1, value2)
79 | Expected return value
80 | """
81 | # Function implementation
82 | ...
83 | ```
84 |
85 |
86 |
87 |
88 | Template for class
89 |
90 | ```python
91 | class ClassName:
92 | """Brief description of the class's purpose and behavior.
93 |
94 | A more detailed description if necessary.
95 |
96 | Args:
97 | arg1 (Type): Description of arg1.
98 | arg2 (Type): Description of arg2.
99 | ...
100 |
101 | Attributes: (Optional)
102 | attribute1 (Type): Description of attribute1.
103 | attribute2 (Type): Description of attribute2.
104 | ...
105 |
106 | Methods: (Optional)
107 | method1: Brief description of method1.
108 | method2: Brief description of method2.
109 | ...
110 |
111 | Examples: (Optional)
112 | >>> instance = ClassName(arg1, arg2)
113 | >>> instance.method1()
114 |
115 | Notes:
116 | Additional information about the class, if necessary.
117 | """
118 | def __init__(self, arg1, arg2, ...):
119 | # Constructor implementation
120 | ...
121 | ```
122 |
123 |
124 |
125 | Here are some more detailed examples of docstrings that you can check out:
126 |
127 |
128 | Detailed examples for docstring
129 |
130 | ```python
131 | from typing import Union
132 |
133 | import torch
134 | import torch.nn as nn
135 | from torchvision.ops.boxes import box_area
136 |
137 | # simple functions
138 | def box_iou(boxes1, boxes2):
139 | """Compute the intersection over union (IoU) between two sets of bounding boxes.
140 |
141 | Inputs:
142 | boxes1 (Tensor): Bounding boxes in format (x1, y1, x2, y2). Shape (N, 4).
143 | boxes2 (Tensor): Bounding boxes in format (x1, y1, x2, y2). Shape (M, 4).
144 |
145 | Returns:
146 | Union[Tensor, Tensor]: A tuple containing two tensors:
147 | iou (Tensor): The IoU between the two sets of bounding boxes. Shape (N, M).
148 | union (Tensor): The area of the union between the two sets of bounding boxes.
149 | Shape (N, M).
150 | """
151 | area1 = box_area(boxes1)
152 | area2 = box_area(boxes2)
153 |
154 | # import ipdb; ipdb.set_trace()
155 | lt = torch.max(boxes1[:, None, :2], boxes2[:, :2]) # [N,M,2]
156 | rb = torch.min(boxes1[:, None, 2:], boxes2[:, 2:]) # [N,M,2]
157 |
158 | wh = (rb - lt).clamp(min=0) # [N,M,2]
159 | inter = wh[:, :, 0] * wh[:, :, 1] # [N,M]
160 |
161 | union = area1[:, None] + area2 - inter
162 |
163 | iou = inter / (union + 1e-6)
164 | return iou, union
165 |
166 | # simple function with dict as input
167 | def create_conv_layer(layer_config):
168 | """Create a convolutional layer for a neural network based on the provided configuration.
169 |
170 | Inputs:
171 | layer_config (dict): A dictionary with the following keys:
172 | 'in_channels' (int): The number of channels in the input.
173 | 'out_channels' (int): The number of channels produced by the convolution.
174 | 'kernel_size' (int or tuple): Size of the convolving kernel.
175 | 'stride' (int or tuple, optional): Stride of the convolution. Default: 1
176 | 'padding' (int or tuple, optional): Zero-padding added to both sides of the input.
177 | Default: 0
178 |
179 | Returns:
180 | nn.Module: A PyTorch convolutional layer configured according to layer_config.
181 |
182 | Example:
183 | >>> config = {'in_channels': 1, 'out_channels': 16, 'kernel_size': 3, 'stride': 1, 'padding': 0}
184 | >>> conv_layer = create_conv_layer(config)
185 | >>> isinstance(conv_layer, nn.Module)
186 | True
187 | """
188 | return nn.Conv2d(**layer_config)
189 |
190 | # simple class
191 | class SimpleConvNet(nn.Module):
192 | """A simple convolutional neural network wrapper class extending PyTorch's nn.Module.
193 | This class creates a neural network with a single convolutional layer.
194 |
195 | Args:
196 | in_channels (int): The number of channels in the input.
197 | out_channels (int): The number of channels produced by the convolution.
198 | kernel_size (int or tuple): Size of the convolving kernel.
199 |
200 | Attributes:
201 | conv_layer (nn.Module): A convolutional layer as defined in the __init__ method.
202 |
203 | Methods:
204 | forward(x): Defines the forward pass of the network.
205 |
206 | Example:
207 | >>> net = SimpleConvNet(1, 16, 3)
208 | >>> isinstance(net, nn.Module)
209 | True
210 | """
211 |
212 | def __init__(self, in_channels, out_channels, kernel_size):
213 | super(SimpleConvNet, self).__init__()
214 | self.conv_layer = nn.Conv2d(in_channels, out_channels, kernel_size)
215 |
216 | def forward(self, x):
217 | """Defines the forward pass of the neural network.
218 |
219 | Inputs:
220 | x (Tensor): The input tensor to the network.
221 |
222 | Returns:
223 | Tensor: The output tensor after passing through the convolutional layer.
224 | """
225 | return self.conv_layer(x)
226 | ```
227 |
228 |
229 |
230 | ### Type Hinting
231 | Type hinting is like attaching labels to your produce in the grocery store; you know exactly what you're getting. It enhance readability, facilitate debugging, and enable better tooling. Type hinting in Python is a formal solution to statically indicate the type of a variable. It was introduced in Python 3.5 and is supported by most IDEs and code editors. Let's look at an example:
232 |
233 | ```python
234 | def add_numbers(a: int, b: int) -> int:
235 | return a + b
236 | ```
237 |
238 | Anyone reading this function signature can quickly understand that the function expects two integers as inputs and will return an integer, and that's the beauty of type hinting. It makes code more readable and self-documenting. **It’s crucial to understand that type hints in Python do not change the dynamic nature of the language. They are simply hints and do not prevent runtime type errors.**
239 |
240 | Almost all built-in types are supported for type hinting. Let's start with some python in-built types.
241 | ```python
242 | int: Integer number.
243 | param: int = 5
244 |
245 | float: Floating point number.
246 | param: float = 3.14
247 |
248 | bool: Boolean value (True or False).
249 | param: bool = True
250 |
251 | str: String.
252 | param: str = "researcher"
253 | ```
254 |
255 | We can also use type hinting for more complex types by importing them from the `typing` module.
256 | ```python
257 | # Generic Types: List, Tuple, Dict, Set
258 | from typing import List, Tuple, Dict, Set
259 |
260 | param: List[int] = [1, 2, 3]
261 | param: Dict[str, int] = {"Time": 12, "Money": 13}
262 | param: Set[int] = {1, 2, 3}
263 | param: Tuple[float, float] = (1.0, 2.0)
264 |
265 | # Specialized Types: Any, Union, Optional
266 | # - Optional: For optional values.
267 | # - Union: To indicate that a value can be of multiple types.
268 | # - Any: For values of any type.
269 | from typing import Union, Optional, Any
270 |
271 | param: Optional[int] = None
272 | param: Union[int, str] = 5
273 | param: Any = "Hello"
274 |
275 | # Callable Types: For functions and methods.
276 | from typing import Callable
277 |
278 | param: Callable[[int], str] = lambda x: str(x)
279 | ```
280 |
281 | These are the most common types you'll encounter in Python. For a complete list of supported types, check out the [official documentation](https://docs.python.org/3/library/typing.html).
282 |
283 | Now let's look at some examples of combining type hinting and docstring in action.
284 |
285 |
286 | Type hinting Example
287 |
288 | ```python
289 | import torch
290 | import torch.nn as nn
291 | import torch.optim as optim
292 | from typing import Tuple, List, Optional
293 |
294 | def find_max(numbers: List[int]) -> Optional[int]:
295 | """Find the maximum number in a list. Returns None if the list is empty.
296 |
297 | Inputs:
298 | numbers (List[int]): A list of integers.
299 |
300 | Returns:
301 | Optional[int]: The maximum number in the list, or None if the list is empty.
302 | """
303 | return max(numbers) if numbers else None
304 |
305 | class SimpleNet(nn.Module):
306 | """A simple neural network with one fully connected layer.
307 |
308 | Args:
309 | input_size (int): The size of the input features.
310 | output_size (int): The size of the output features.
311 | """
312 |
313 | def __init__(self, input_size: int, output_size: int) -> None:
314 | super(SimpleNet, self).__init__()
315 | self.fc = nn.Linear(input_size, output_size)
316 |
317 | def forward(self, x: torch.Tensor) -> torch.Tensor:
318 | """Perform a forward pass of the network.
319 |
320 | Inputs:
321 | x (torch.Tensor): The input tensor.
322 |
323 | Returns:
324 | torch.Tensor: The output tensor after passing through the network.
325 | """
326 | return self.fc(x)
327 |
328 | def train_network(network: nn.Module, data: List[Tuple[torch.Tensor, torch.Tensor]],
329 | epochs: int, learning_rate: float) -> None:
330 | """Train a neural network.
331 |
332 | Inputs:
333 | network (nn.Module): The neural network to train.
334 | data (List[Tuple[torch.Tensor, torch.Tensor]]): Training data, a list of tuples with
335 | input and target tensors.
336 | epochs (int): The number of epochs to train for.
337 | learning_rate (float): The learning rate for the optimizer.
338 |
339 | Returns:
340 | None
341 | """
342 | criterion = nn.MSELoss()
343 | optimizer = optim.Adam(network.parameters(), lr=learning_rate)
344 |
345 | for epoch in range(epochs):
346 | for inputs, targets in data:
347 | optimizer.zero_grad()
348 | outputs = network(inputs)
349 | loss = criterion(outputs, targets)
350 | loss.backward()
351 | optimizer.step()
352 | ```
353 |
354 |
355 |
356 | Type hints in Python enhance code clarity, readability, and maintainability. Though Python remains dynamically typed, type hints offer the benefits of static typing, making them particularly useful in large codebases and complex applications like deep learning. Incorporating type hints is a straightforward way to make Python code more robust and easier to understand.
357 |
358 |
359 | ### Naming Conventions
360 |
361 | In programming, naming conventions are as crucial as the code itself. They are the first layer of documentation for anyone who reads your code. Good naming conventions in Python enhance readability, maintainability, and are essential for understanding the intent of the code. Let's look at some of the best practices for naming things in Python.
362 |
363 | - **General Python Naming Guidelines**:
364 | - Descriptive Names: Choose names that reflect the purpose of the variable, function, or class.
365 | - Short but Meaningful: While names should be descriptive, they should also be concise.
366 | - Avoid Ambiguity: Names should be clear enough to not be confused with language keywords or commonly used variables.
367 | - **Conventions in Python**:
368 | - Variable Names: Use lowercase letters and underscores to separate words.
369 | - Example: `num_epochs`, `learning_rate`, `train_data`, `learning_rate`
370 | - Function Names: Also use lowercase letters and underscores to separate words.
371 | - Example: `train_network`, `find_max`, `get_data`, `calculate_loss`, `update_weights`
372 | - Classes: Use CamelCase (capitalize the first letter of each word) for class names.
373 | - Example: `SimpleNet`, `ConvNet`, `ResidualBlock`, `DataLoader`, `Optimizer`
374 | - Constants: Use all uppercase letters and underscores to separate words.
375 | - Example: `NUM_EPOCHS`, `LEARNING_RATE`, `TRAIN_DATA`, `VALIDATION_DATA`
376 | - Private Variables: Use a leading underscore to indicate that a variable is private.
377 | - Example: `_weights`, `_optimizer`, `_loss`, `_init_params`
378 | - **Abbreviations**:
379 | - Variables and Functions: Lowercase with underscores. Examples: `img` for image, `calc_rmse`` for calculating root mean square error.
380 | - Classes: CamelCase for well-known abbreviations. Examples: `RNN` for Recurrent Neural Network, `GAN`for Generative Adversarial Network.
381 | - Constants: All uppercase for constants. Examples: `MAX_ITER` for maximum iterations, `DEFAULT_LR` for default learning rate.
382 | - Clarity: Use abbreviations that are clear and well-understood in the context; explain any that might be ambiguous.
383 |
384 | ### Formatting
385 | Code formatting is about more than just aesthetics; it's a crucial aspect of writing readable and maintainable code. In Python, adhering to a consistent formatting style helps developers understand and navigate the code more effectively. Before we dive into the details of code formatting, let's look at two code snippets and see which style you prefer.
386 |
387 | 
388 |
389 | The Python Enhancement Proposal 8 **(PEP8)** is the de facto code style guide for Python. It covers various aspects of code formatting like indentation, line length, whitespace usage, and more. The rules are complex and detailed, but here are two off-the-shelf tools that can help you format your code according to PEP8, **automatically**.
390 | - **[yapf](https://marketplace.visualstudio.com/items?itemName=eeyore.yapf)**: It's an open-source tool designed to automatically format Python code to conform to the PEP 8 style guide. It reformats your code in a way that it believes is the best formatting approach, not merely to adhere strictly to PEP 8. It is now available as a VS Code extension. Once installed and correctly configured, you can format your code automatically when saving the code.
391 |
392 | 
393 |
394 | - **[isort](https://marketplace.visualstudio.com/items?itemName=ms-python.isort)**: isort is a Python utility/library to sort imports alphabetically and automatically separate them into sections and by type.
395 |
396 |
397 | 
398 |
399 | Once you have installed the two extensions, yapf and isort, you can automatically format your Python code and organize your import statements with minimal effort.
400 |
401 |
402 | ## 2. Simplicity and Efficiency
403 |
404 | ### KISS Principle
405 | Keep It Simple, Stupid (KISS) is a design principle that emphasizes the importance of simplicity in software development. The core idea is that systems work best if they are kept simple rather than made complex. Simplicity here means avoiding unnecessary complexity, which can lead to code that is more reliable, easier to understand, maintain, and extend. Here are some of the key aspects of the KISS principle that you can apply to your code:
406 |
407 | - **Simple Function Definitions**:
408 | - Write functions that ***do one thing and do it well***. Each function should have a clear purpose and not be overloaded with multiple tasks. Here's an example of a function that violates the KISS principle:
409 | ```python
410 | >>>>>>>>>>>>>>>>>>>> Non-KISS <<<<<<<<<<<<<<<<<<<<<<
411 | def forward(input, target, optimizer, criterion):
412 | output = model(input)
413 | loss = criterion(output, target)
414 | optimizer.zero_grad()
415 | loss.backward()
416 | optimizer.step()
417 | return loss.item()
418 |
419 | >>>>>>>>>>>>>>>>>>>>>> KISS <<<<<<<<<<<<<<<<<<<<<<<<
420 | def forward(input, target):
421 | output = model(input)
422 | return output
423 |
424 | def train_step(loader, model, optimizer, criterion):
425 | for input, target in loader:
426 | output = model(input)
427 | loss = criterion(output, target)
428 | optimizer.zero_grad()
429 | loss.backward()
430 | optimizer.step()
431 | ```
432 | - **Use Build-in Libraies and Functions**:
433 | - Python, known for its comprehensive standard library, along with numerous third-party libraries, offers a wealth of pre-written modules and functions that can save time, reduce the likelihood of bugs, and improve efficiency. Here is an example of using a built-in function to calculate the Non Maximum Suppression (NMS) of bounding boxes:
434 | ```python
435 | >>>>>>>>>>>>>>>>>>>> Non-KISS <<<<<<<<<<<<<<<<<<<<<<
436 | import torch
437 | def nms(boxes, scores, threshold=0.5):
438 | """
439 | Apply non-maximum suppression to avoid detecting too many
440 | overlapping bounding boxes for the same object.
441 |
442 | Parameters:
443 | boxes (Tensor): The locations of the bounding boxes.
444 | scores (Tensor): The scores for each box.
445 | threshold (float): The overlap threshold for suppressing boxes.
446 |
447 | Returns:
448 | List[int]: The indices of the boxes that were kept.
449 | """
450 | x1 = boxes[:, 0]
451 | y1 = boxes[:, 1]
452 | x2 = boxes[:, 2]
453 | y2 = boxes[:, 3]
454 |
455 | areas = (x2 - x1) * (y2 - y1)
456 | _, order = scores.sort(0, descending=True)
457 |
458 | keep = []
459 | while order.numel() > 0:
460 | i = order[0]
461 | keep.append(i)
462 |
463 | if order.numel() == 1:
464 | break
465 |
466 | xx1 = torch.maximum(x1[i], x1[order[1:]])
467 | yy1 = torch.maximum(y1[i], y1[order[1:]])
468 | xx2 = torch.minimum(x2[i], x2[order[1:]])
469 | yy2 = torch.minimum(y2[i], y2[order[1:]])
470 |
471 | w = torch.maximum(torch.tensor(0.0), xx2 - xx1)
472 | h = torch.maximum(torch.tensor(0.0), yy2 - yy1)
473 | overlap = (w * h) / areas[order[1:]]
474 |
475 | ids = (overlap <= threshold).nonzero().squeeze()
476 | if ids.numel() == 0:
477 | break
478 | order = order[ids + 1]
479 | return keep
480 |
481 | >>>>>>>>>>>>>>>>>>>>>> KISS <<<<<<<<<<<<<<<<<<<<<<<<
482 | import torch
483 | import torchvision.ops as ops
484 | # Apply NMS from torchvision
485 | nms_indices = ops.nms(boxes, scores, threshold=0.5)
486 | ```
487 | - **Avoid Over-Engineering**:
488 | - Over-engineering refers to the practice of making a project or a system more complicated than necessary, often adding extra features or complexity that do not add significant value. This can manifest as overly complex code, over-abstracted architectures, or unnecessary features that complicate maintenance and understanding without providing proportional benefits. Here is an example of over-engineering, say
489 | we need a function to calculate then mean value of a list:
490 | ```python
491 | >>>>>>>>>>>>>>>>>>>> Non-KISS <<<<<<<<<<<<<<<<<<<<<<
492 | class Number:
493 | def __init__(self, value):
494 | self.value = value
495 |
496 | class NumberList:
497 | def __init__(self, numbers):
498 | self.numbers = [Number(num) for num in numbers]
499 |
500 | def sum(self):
501 | return sum(number.value for number in self.numbers)
502 |
503 | def length(self):
504 | return len(self.numbers)
505 |
506 | def average(self):
507 | return self.sum() / self.length() if self.length() > 0 else 0
508 |
509 | # Usage
510 | number_list = NumberList([10, 20, 30, 40, 50])
511 | avg = number_list.average()
512 | print("Average:", avg)
513 |
514 | >>>>>>>>>>>>>>>>>>>>>> KISS <<<<<<<<<<<<<<<<<<<<<<<<
515 | def average(numbers):
516 | return sum(numbers) / len(numbers) if numbers else 0
517 |
518 | # Usage
519 | avg = average([10, 20, 30, 40, 50])
520 | print("Average:", avg)
521 | ```
522 |
523 | ### Write Efficient Code
524 | Writing efficient code means optimizing both for speed and resource usage, ensuring that the application runs smoothly and responds quickly, even as the complexity of tasks or the size of data increases.
525 |
526 | - **Algorithmic Complexity:**
527 | - The efficiency of code often comes down to its algorithmic complexity, commonly referred to as Big O notation. It's crucial to choose the right algorithm and data structure for a given task. For example, in a sorting task, using QuickSort (average complexity $O(nlogn)$) is generally more efficient than Bubble Sort (average complexity $O(n^2)$).
528 | - **Efficient Data Handling**:
529 | - Efficient data handling is crucial, especially in data-heavy situations. For example, when working with large datasets, it's better to use generators instead of lists to avoid loading the entire dataset into memory. Similarly, when working with images, it's better to use a data loader that loads images on-demand rather than loading all images into memory at once. Here is an example of using `yield` to create a generator to load TSV (Tab-Separated Values) file:
530 | ```python
531 | >>>>>>>>>>>>>>>>>>>> Non-Efficient <<<<<<<<<<<<<<<<<<<<<<
532 | def load_data_into_list(file_path):
533 | data = []
534 | with open(file_path, 'r') as file:
535 | for line in file:
536 | columns = line.strip().split('\t')
537 | data.append(columns)
538 | return data
539 | # this will load all data into memory, which is not efficient
540 | data = load_data_into_list('data.tsv')
541 |
542 | >>>>>>>>>>>>>>>>>>>>>> Efficient <<<<<<<<<<<<<<<<<<<<<<<<
543 | def load_data_generator(file_path):
544 | with open(file_path, 'r') as file:
545 | for line in file:
546 | columns = line.strip().split('\t')
547 | yield columns
548 | # this will create a generator that loads data on-demand
549 | data_generator = load_data_generator('data.tsv')
550 | for data in data_generator:
551 | # Process each line of data here
552 | ```
553 | - **Vectorization and Parallelization**:
554 | - Vectorization and parallelization are two techniques that can significantly improve the performance of code. Vectorization refers to performing operations on entire arrays rather than individual elements. For example, in NumPy, we can add two arrays of the same shape using the `+` operator, which will add the corresponding elements of the two arrays. This is much more efficient than using a `for` loop to add the elements one by one. Similarly, parallelization refers to performing operations in parallel, using multiple processors or CPU cores. Here is an example of vectorization and parallelization in action:
555 | ```python
556 | >>>>>>>>>>>>>>>>>>>> Non-Efficient <<<<<<<<<<<<<<<<<<<<<<
557 | import numpy as np
558 | def add_arrays(a, b):
559 | result = []
560 | for i in range(len(a)):
561 | result.append(a[i] + b[i])
562 | return result
563 | a = np.random.rand(1000000)
564 | b = np.random.rand(1000000)
565 | # this will take a long time to execute
566 | c = add_arrays(a, b)
567 |
568 | >>>>>>>>>>>>>>>>>>>>>> Efficient <<<<<<<<<<<<<<<<<<<<<<<<
569 | import numpy as np
570 | def add_arrays(a, b):
571 | return a + b
572 | a = np.random.rand(1000000)
573 | b = np.random.rand(1000000)
574 | # this will execute much faster
575 | c = add_arrays(a, b)
576 | ```
577 |
578 | ## 3. Maintainability
579 | Maintainability refers to how easily software can be maintained over time. This includes fixing bugs, improving functionality, and update to meet new requirements or work with new technologies. High maintainability is crucial for the long-term success and adaptability of a deep learning project.
580 |
581 | ### Use Registry Mechanism to Manage Your Code
582 | Quoting from [MMEngine](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/registry.html#registry), "The registry can be considered as a union of a mapping table and a build function of modules. The mapping table maintains a ***mapping from strings to classes or functions***, allowing the user to find the corresponding class or function with its name/notation. For example, the mapping from the string "ResNet" to the ResNet class. The module build function defines how to find the corresponding class or function based on a string and how to instantiate the class or call the function. "
583 |
584 | Now let's look at an example of how registry mechanism works
585 | ```python
586 | @BACKBONES.register_module()
587 | class ClipViTWrapper(nn.Module):
588 | def __init__(self, return_pool_feature: bool = True, freeze: bool = False):
589 | super(ClipViTWrapper, self).__init__()
590 | self.model = CLIPVisionModel.from_pretrained(
591 | 'openai/clip-vit-base-patch32')
592 | self.return_pool_feature = return_pool_feature
593 | if freeze:
594 | for param in self.model.parameters():
595 | param.requires_grad = False
596 |
597 | def forward(self, x: torch.Tensor):
598 | ...
599 |
600 | >>>>>>>>>>>>>>>>>>>> W/O Registry <<<<<<<<<<<<<<<<<<<<<<
601 | from ...backbones import ClipViTWrapper
602 | model = ClipViTWrapper(return_pool_feature=True, freeze=False)
603 |
604 | >>>>>>>>>>>>>>>>>>>>>> W/ Registry <<<<<<<<<<<<<<<<<<<<<<<<
605 | model = BACKBONES.build(dict(type='ClipViTWrapper', return_pool_feature=True, freeze=False))
606 | ```
607 |
608 | - **Decoupling Code and Configuration**: The registry pattern allows for the creation and configuration of objects like `ClipViTWrapper` in a more dynamic and decoupled manner. Instead of directly importing and instantiating the class, you use a registry (`BACKBONES`) to build the object. This separation of object creation from usage enhances flexibility and maintainability.
609 |
610 | - **Enhanced Flexibility**: With the registry, you can easily switch between different backbone models (like `ClipViTWrapper`) by changing a configuration file or a dictionary.
611 |
612 | - **Scalability**: As the project grows, the registry pattern scales well, accommodating more components without increasing
613 |
614 | ### Organize Your Code by Functionality
615 | Organizing code by functionality is a crucial aspect of maintainability. It involves grouping related code into modules, packages, and libraries, and separating unrelated code into distinct sections. This makes it easier to find and update code, and also helps avoid conflicts between different parts of the codebase. Generally, an AI project is composed of three parts: model, dataset, and computation depicited in the following figure.
616 |
617 |
618 |
619 | 
620 |
621 |
622 | - `Model`: The model part is the core of an AI project, which is composed of the model architecture, loss function, other components. For model architecture, it can be divided into backbone, encoder, decoder, head and other components depending on the specific task.
623 | - `Dataset`: The dataset part is the data source of an AI project, which is composed of data loader, data augmentation, data preprocessing, and other components.
624 | - `Computation`: The computation part is where you train and test your model, which is composed of training, evaluation, and other components.
625 |
626 | Based on the above three parts, we can organize our code by functionality as follows:
627 |
628 |
629 |
630 | 
631 |
632 |
633 | - `configs`: The configs part is where you store all the configuration files, including the dataset configuration, model configuration, training configuration, and others.
634 | - `datasets`: The datasets part is where you store all the dataset-related code, including data loader, data augmentation, data preprocessing, and others.
635 | - `models`: The models part is where you store all the model-related code, including backbone, encoder, decoder, head, loss function, and others.
636 | - `engine`: The engine part is where you store all the computation-related code, including training, evaluation, and others.
637 | - `tools`: The tools part is where you store all the tools-related code, including visualization, logging, and others.
638 | - `tests`: The tests part is where you store all the test-related code, including unit test, integration test, and others.
639 | - `scripts`: The scripts part is where you store all the scripts-related code, which are used to launch the training, evaluation, and others, especially for slurm-based distributed training.
640 | - `README.md`: The `README.md` part is where you store the documentation of your project, including the installation, usage, and others.
641 | - `LICENSE`: The LICENSE part is where you store the license of your project, which is crucial for open-source projects.
642 | - `.gitignore`: The .gitignore part is where you store the files that you don't want to upload to the remote repository, which is crucial for open-source projects.
643 | - `requirements.txt`: The requirements.txt part is where you store the dependencies of your project, which is crucial for open-source projects.
644 | - `train.py`: The `train.py` part is a port to launch the training of your project.
645 | - `eval.py`: The `eval.py` part is a port to launch the evaluation of your project.
646 | - `inference.py`: The `inference.py` part is a port to launch the inference of your project.
647 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 |
2 |
3 | # Cookbook to Craft Good Code
4 | Think of good code like a classic, well-fitting piece of clothing—it never goes out of style. Coding is both a science and an art, where neatness and logic come together. What makes some code stand out as great? Here are three key aspects:
5 | - **Readability**:
6 | - Good code should be easy to read and understand. This means using clear naming conventions, organizing code logically, and commenting where necessary to explain complex parts.
7 | - **Simplicity and Efficiency**:
8 | - Good code often follows the KISS principle ("Keep It Simple, Stupid"). It should accomplish its tasks in the simplest way possible, without unnecessary complexity.
9 | - Efficient code also performs its tasks quickly and resourcefully.
10 | - **Maintainability**:
11 | - Code should be easy to maintain and modify. This involves writing modular code, where different parts of the program are separated into distinct sections or functions that can be updated independently.
12 |
13 | In this guide, we'll dive into the essentials of crafting great code. We'll go through everything from how to name things clearly and highlight tools that make coding better and easier.
14 |
15 | ### Contents
16 | - [Cookbook to Craft Good Code](#cookbook-to-craft-good-code)
17 | - [Contents](#contents)
18 | - [1. Readability](#1-readability)
19 | - [Docstring](#docstring)
20 | - [Type Hinting](#type-hinting)
21 | - [Naming Conventions](#naming-conventions)
22 | - [Formatting](#formatting)
23 | - [2. Simplicity and Efficiency](#2-simplicity-and-efficiency)
24 | - [KISS Principle](#kiss-principle)
25 | - [Write Efficient Code](#write-efficient-code)
26 | - [3. Maintainability](#3-maintainability)
27 | - [Use Registry Mechanism to Manage Your Code](#use-registry-mechanism-to-manage-your-code)
28 | - [Organize Your Code by Functionality](#organize-your-code-by-functionality)
29 |
30 |
31 | ## 1. Readability
32 | Readability in code is akin to clear handwriting in a letter. It's not just about what you write, but how you present it. A well-written piece of code should speak to its reader, guiding them through its logic as effortlessly as a well-told story. Let's delve into some of the key practices that make code readable.
33 |
34 | ### Docstring
35 | A docstring, short for "documentation string," is a string literal that occurs as the first statement in a module, function, class, or method definition. Here are three most important definitions from the official Python documentation, [PEP257](https://peps.python.org/pep-0257/).
36 |
37 |
38 | Python PEP257
39 |
40 | - **1. What Should be Documented**:
41 | - PEP 257 suggests that ***all public modules, functions, classes, and methods should have docstrings***. Private methods (those starting with an underscore) are considered optional for documentation but are encouraged, especially for complex code.
42 |
43 | - **2. Docstring Format**:
44 | - Docstrings should be enclosed in triple double quotes (""").
45 | - The first line should be a short, concise summary of the object’s purpose.
46 |
47 | - **3. Multi-line Docstrings**:
48 | - For longer descriptions, the summary line should be followed by a blank line, then a more elaborate description. The detailed description may include usage, arguments, return values, and raised exceptions if applicable.
49 | - Multi-line docstrings should end with the closing triple quotes on a line by themselves.
50 |
51 |
52 |
53 | Here are two python templates for docstring of function and class that may give you a more concrete idea of how to write a docstring.
54 |
55 |
56 | Template for function
57 |
58 | ```python
59 | def function_name(param1, param2, ...):
60 | """A brief description of what the function does.
61 |
62 | A more detailed description of the function if necessary.
63 |
64 | Inputs:
65 | param1 (Type): Description of param1.
66 | param2 (Type): Description of param2.
67 |
68 | Returns:
69 | ReturnType: Description of the return value.
70 |
71 | Raises: (Optional)
72 | ExceptionType: Explanation of when and why the exception is raised.
73 |
74 | Notes: (Optional)
75 | Additional notes or examples of usage, if necessary.
76 |
77 | Examples: (Optional)
78 | >>> function_name(value1, value2)
79 | Expected return value
80 | """
81 | # Function implementation
82 | ...
83 | ```
84 |
85 |
86 |
87 |
88 | Template for class
89 |
90 | ```python
91 | class ClassName:
92 | """Brief description of the class's purpose and behavior.
93 |
94 | A more detailed description if necessary.
95 |
96 | Args:
97 | arg1 (Type): Description of arg1.
98 | arg2 (Type): Description of arg2.
99 | ...
100 |
101 | Attributes: (Optional)
102 | attribute1 (Type): Description of attribute1.
103 | attribute2 (Type): Description of attribute2.
104 | ...
105 |
106 | Methods: (Optional)
107 | method1: Brief description of method1.
108 | method2: Brief description of method2.
109 | ...
110 |
111 | Examples: (Optional)
112 | >>> instance = ClassName(arg1, arg2)
113 | >>> instance.method1()
114 |
115 | Notes:
116 | Additional information about the class, if necessary.
117 | """
118 | def __init__(self, arg1, arg2, ...):
119 | # Constructor implementation
120 | ...
121 | ```
122 |
123 |
124 |
125 | Here are some more detailed examples of docstrings that you can check out:
126 |
127 |
128 | Detailed examples for docstring
129 |
130 | ```python
131 | from typing import Union
132 |
133 | import torch
134 | import torch.nn as nn
135 | from torchvision.ops.boxes import box_area
136 |
137 | # simple functions
138 | def box_iou(boxes1, boxes2):
139 | """Compute the intersection over union (IoU) between two sets of bounding boxes.
140 |
141 | Inputs:
142 | boxes1 (Tensor): Bounding boxes in format (x1, y1, x2, y2). Shape (N, 4).
143 | boxes2 (Tensor): Bounding boxes in format (x1, y1, x2, y2). Shape (M, 4).
144 |
145 | Returns:
146 | Union[Tensor, Tensor]: A tuple containing two tensors:
147 | iou (Tensor): The IoU between the two sets of bounding boxes. Shape (N, M).
148 | union (Tensor): The area of the union between the two sets of bounding boxes.
149 | Shape (N, M).
150 | """
151 | area1 = box_area(boxes1)
152 | area2 = box_area(boxes2)
153 |
154 | # import ipdb; ipdb.set_trace()
155 | lt = torch.max(boxes1[:, None, :2], boxes2[:, :2]) # [N,M,2]
156 | rb = torch.min(boxes1[:, None, 2:], boxes2[:, 2:]) # [N,M,2]
157 |
158 | wh = (rb - lt).clamp(min=0) # [N,M,2]
159 | inter = wh[:, :, 0] * wh[:, :, 1] # [N,M]
160 |
161 | union = area1[:, None] + area2 - inter
162 |
163 | iou = inter / (union + 1e-6)
164 | return iou, union
165 |
166 | # simple function with dict as input
167 | def create_conv_layer(layer_config):
168 | """Create a convolutional layer for a neural network based on the provided configuration.
169 |
170 | Inputs:
171 | layer_config (dict): A dictionary with the following keys:
172 | 'in_channels' (int): The number of channels in the input.
173 | 'out_channels' (int): The number of channels produced by the convolution.
174 | 'kernel_size' (int or tuple): Size of the convolving kernel.
175 | 'stride' (int or tuple, optional): Stride of the convolution. Default: 1
176 | 'padding' (int or tuple, optional): Zero-padding added to both sides of the input.
177 | Default: 0
178 |
179 | Returns:
180 | nn.Module: A PyTorch convolutional layer configured according to layer_config.
181 |
182 | Example:
183 | >>> config = {'in_channels': 1, 'out_channels': 16, 'kernel_size': 3, 'stride': 1, 'padding': 0}
184 | >>> conv_layer = create_conv_layer(config)
185 | >>> isinstance(conv_layer, nn.Module)
186 | True
187 | """
188 | return nn.Conv2d(**layer_config)
189 |
190 | # simple class
191 | class SimpleConvNet(nn.Module):
192 | """A simple convolutional neural network wrapper class extending PyTorch's nn.Module.
193 | This class creates a neural network with a single convolutional layer.
194 |
195 | Args:
196 | in_channels (int): The number of channels in the input.
197 | out_channels (int): The number of channels produced by the convolution.
198 | kernel_size (int or tuple): Size of the convolving kernel.
199 |
200 | Attributes:
201 | conv_layer (nn.Module): A convolutional layer as defined in the __init__ method.
202 |
203 | Methods:
204 | forward(x): Defines the forward pass of the network.
205 |
206 | Example:
207 | >>> net = SimpleConvNet(1, 16, 3)
208 | >>> isinstance(net, nn.Module)
209 | True
210 | """
211 |
212 | def __init__(self, in_channels, out_channels, kernel_size):
213 | super(SimpleConvNet, self).__init__()
214 | self.conv_layer = nn.Conv2d(in_channels, out_channels, kernel_size)
215 |
216 | def forward(self, x):
217 | """Defines the forward pass of the neural network.
218 |
219 | Inputs:
220 | x (Tensor): The input tensor to the network.
221 |
222 | Returns:
223 | Tensor: The output tensor after passing through the convolutional layer.
224 | """
225 | return self.conv_layer(x)
226 | ```
227 |
228 |
229 |
230 | ### Type Hinting
231 | Type hinting is like attaching labels to your produce in the grocery store; you know exactly what you're getting. It enhance readability, facilitate debugging, and enable better tooling. Type hinting in Python is a formal solution to statically indicate the type of a variable. It was introduced in Python 3.5 and is supported by most IDEs and code editors. Let's look at an example:
232 |
233 | ```python
234 | def add_numbers(a: int, b: int) -> int:
235 | return a + b
236 | ```
237 |
238 | Anyone reading this function signature can quickly understand that the function expects two integers as inputs and will return an integer, and that's the beauty of type hinting. It makes code more readable and self-documenting. **It’s crucial to understand that type hints in Python do not change the dynamic nature of the language. They are simply hints and do not prevent runtime type errors.**
239 |
240 | Almost all built-in types are supported for type hinting. Let's start with some python in-built types.
241 | ```python
242 | int: Integer number.
243 | param: int = 5
244 |
245 | float: Floating point number.
246 | param: float = 3.14
247 |
248 | bool: Boolean value (True or False).
249 | param: bool = True
250 |
251 | str: String.
252 | param: str = "researcher"
253 | ```
254 |
255 | We can also use type hinting for more complex types by importing them from the `typing` module.
256 | ```python
257 | # Generic Types: List, Tuple, Dict, Set
258 | from typing import List, Tuple, Dict, Set
259 |
260 | param: List[int] = [1, 2, 3]
261 | param: Dict[str, int] = {"Time": 12, "Money": 13}
262 | param: Set[int] = {1, 2, 3}
263 | param: Tuple[float, float] = (1.0, 2.0)
264 |
265 | # Specialized Types: Any, Union, Optional
266 | # - Optional: For optional values.
267 | # - Union: To indicate that a value can be of multiple types.
268 | # - Any: For values of any type.
269 | from typing import Union, Optional, Any
270 |
271 | param: Optional[int] = None
272 | param: Union[int, str] = 5
273 | param: Any = "Hello"
274 |
275 | # Callable Types: For functions and methods.
276 | from typing import Callable
277 |
278 | param: Callable[[int], str] = lambda x: str(x)
279 | ```
280 |
281 | These are the most common types you'll encounter in Python. For a complete list of supported types, check out the [official documentation](https://docs.python.org/3/library/typing.html).
282 |
283 | Now let's look at some examples of combining type hinting and docstring in action.
284 |
285 |
286 | Type hinting Example
287 |
288 | ```python
289 | import torch
290 | import torch.nn as nn
291 | import torch.optim as optim
292 | from typing import Tuple, List, Optional
293 |
294 | def find_max(numbers: List[int]) -> Optional[int]:
295 | """Find the maximum number in a list. Returns None if the list is empty.
296 |
297 | Inputs:
298 | numbers (List[int]): A list of integers.
299 |
300 | Returns:
301 | Optional[int]: The maximum number in the list, or None if the list is empty.
302 | """
303 | return max(numbers) if numbers else None
304 |
305 | class SimpleNet(nn.Module):
306 | """A simple neural network with one fully connected layer.
307 |
308 | Args:
309 | input_size (int): The size of the input features.
310 | output_size (int): The size of the output features.
311 | """
312 |
313 | def __init__(self, input_size: int, output_size: int) -> None:
314 | super(SimpleNet, self).__init__()
315 | self.fc = nn.Linear(input_size, output_size)
316 |
317 | def forward(self, x: torch.Tensor) -> torch.Tensor:
318 | """Perform a forward pass of the network.
319 |
320 | Inputs:
321 | x (torch.Tensor): The input tensor.
322 |
323 | Returns:
324 | torch.Tensor: The output tensor after passing through the network.
325 | """
326 | return self.fc(x)
327 |
328 | def train_network(network: nn.Module, data: List[Tuple[torch.Tensor, torch.Tensor]],
329 | epochs: int, learning_rate: float) -> None:
330 | """Train a neural network.
331 |
332 | Inputs:
333 | network (nn.Module): The neural network to train.
334 | data (List[Tuple[torch.Tensor, torch.Tensor]]): Training data, a list of tuples with
335 | input and target tensors.
336 | epochs (int): The number of epochs to train for.
337 | learning_rate (float): The learning rate for the optimizer.
338 |
339 | Returns:
340 | None
341 | """
342 | criterion = nn.MSELoss()
343 | optimizer = optim.Adam(network.parameters(), lr=learning_rate)
344 |
345 | for epoch in range(epochs):
346 | for inputs, targets in data:
347 | optimizer.zero_grad()
348 | outputs = network(inputs)
349 | loss = criterion(outputs, targets)
350 | loss.backward()
351 | optimizer.step()
352 | ```
353 |
354 |
355 |
356 | Type hints in Python enhance code clarity, readability, and maintainability. Though Python remains dynamically typed, type hints offer the benefits of static typing, making them particularly useful in large codebases and complex applications like deep learning. Incorporating type hints is a straightforward way to make Python code more robust and easier to understand.
357 |
358 |
359 | ### Naming Conventions
360 |
361 | In programming, naming conventions are as crucial as the code itself. They are the first layer of documentation for anyone who reads your code. Good naming conventions in Python enhance readability, maintainability, and are essential for understanding the intent of the code. Let's look at some of the best practices for naming things in Python.
362 |
363 | - **General Python Naming Guidelines**:
364 | - Descriptive Names: Choose names that reflect the purpose of the variable, function, or class.
365 | - Short but Meaningful: While names should be descriptive, they should also be concise.
366 | - Avoid Ambiguity: Names should be clear enough to not be confused with language keywords or commonly used variables.
367 | - **Conventions in Python**:
368 | - Variable Names: Use lowercase letters and underscores to separate words.
369 | - Example: `num_epochs`, `learning_rate`, `train_data`, `learning_rate`
370 | - Function Names: Also use lowercase letters and underscores to separate words.
371 | - Example: `train_network`, `find_max`, `get_data`, `calculate_loss`, `update_weights`
372 | - Classes: Use CamelCase (capitalize the first letter of each word) for class names.
373 | - Example: `SimpleNet`, `ConvNet`, `ResidualBlock`, `DataLoader`, `Optimizer`
374 | - Constants: Use all uppercase letters and underscores to separate words.
375 | - Example: `NUM_EPOCHS`, `LEARNING_RATE`, `TRAIN_DATA`, `VALIDATION_DATA`
376 | - Private Variables: Use a leading underscore to indicate that a variable is private.
377 | - Example: `_weights`, `_optimizer`, `_loss`, `_init_params`
378 | - **Abbreviations**:
379 | - Variables and Functions: Lowercase with underscores. Examples: `img` for image, `calc_rmse`` for calculating root mean square error.
380 | - Classes: CamelCase for well-known abbreviations. Examples: `RNN` for Recurrent Neural Network, `GAN`for Generative Adversarial Network.
381 | - Constants: All uppercase for constants. Examples: `MAX_ITER` for maximum iterations, `DEFAULT_LR` for default learning rate.
382 | - Clarity: Use abbreviations that are clear and well-understood in the context; explain any that might be ambiguous.
383 |
384 | ### Formatting
385 | Code formatting is about more than just aesthetics; it's a crucial aspect of writing readable and maintainable code. In Python, adhering to a consistent formatting style helps developers understand and navigate the code more effectively. Before we dive into the details of code formatting, let's look at two code snippets and see which style you prefer.
386 |
387 | 
388 |
389 | The Python Enhancement Proposal 8 **(PEP8)** is the de facto code style guide for Python. It covers various aspects of code formatting like indentation, line length, whitespace usage, and more. The rules are complex and detailed, but here are two off-the-shelf tools that can help you format your code according to PEP8, **automatically**.
390 | - **[yapf](https://marketplace.visualstudio.com/items?itemName=eeyore.yapf)**: It's an open-source tool designed to automatically format Python code to conform to the PEP 8 style guide. It reformats your code in a way that it believes is the best formatting approach, not merely to adhere strictly to PEP 8. It is now available as a VS Code extension. Once installed and correctly configured, you can format your code automatically when saving the code.
391 |
392 | 
393 |
394 | - **[isort](https://marketplace.visualstudio.com/items?itemName=ms-python.isort)**: isort is a Python utility/library to sort imports alphabetically and automatically separate them into sections and by type.
395 |
396 |
397 | 
398 |
399 | Once you have installed the two extensions, yapf and isort, you can automatically format your Python code and organize your import statements with minimal effort.
400 |
401 |
402 | ## 2. Simplicity and Efficiency
403 |
404 | ### KISS Principle
405 | Keep It Simple, Stupid (KISS) is a design principle that emphasizes the importance of simplicity in software development. The core idea is that systems work best if they are kept simple rather than made complex. Simplicity here means avoiding unnecessary complexity, which can lead to code that is more reliable, easier to understand, maintain, and extend. Here are some of the key aspects of the KISS principle that you can apply to your code:
406 |
407 | - **Simple Function Definitions**:
408 | - Write functions that ***do one thing and do it well***. Each function should have a clear purpose and not be overloaded with multiple tasks. Here's an example of a function that violates the KISS principle:
409 | ```python
410 | >>>>>>>>>>>>>>>>>>>> Non-KISS <<<<<<<<<<<<<<<<<<<<<<
411 | def forward(input, target, optimizer, criterion):
412 | output = model(input)
413 | loss = criterion(output, target)
414 | optimizer.zero_grad()
415 | loss.backward()
416 | optimizer.step()
417 | return loss.item()
418 |
419 | >>>>>>>>>>>>>>>>>>>>>> KISS <<<<<<<<<<<<<<<<<<<<<<<<
420 | def forward(input, target):
421 | output = model(input)
422 | return output
423 |
424 | def train_step(loader, model, optimizer, criterion):
425 | for input, target in loader:
426 | output = model(input)
427 | loss = criterion(output, target)
428 | optimizer.zero_grad()
429 | loss.backward()
430 | optimizer.step()
431 | ```
432 | - **Use Build-in Libraies and Functions**:
433 | - Python, known for its comprehensive standard library, along with numerous third-party libraries, offers a wealth of pre-written modules and functions that can save time, reduce the likelihood of bugs, and improve efficiency. Here is an example of using a built-in function to calculate the Non Maximum Suppression (NMS) of bounding boxes:
434 | ```python
435 | >>>>>>>>>>>>>>>>>>>> Non-KISS <<<<<<<<<<<<<<<<<<<<<<
436 | import torch
437 | def nms(boxes, scores, threshold=0.5):
438 | """
439 | Apply non-maximum suppression to avoid detecting too many
440 | overlapping bounding boxes for the same object.
441 |
442 | Parameters:
443 | boxes (Tensor): The locations of the bounding boxes.
444 | scores (Tensor): The scores for each box.
445 | threshold (float): The overlap threshold for suppressing boxes.
446 |
447 | Returns:
448 | List[int]: The indices of the boxes that were kept.
449 | """
450 | x1 = boxes[:, 0]
451 | y1 = boxes[:, 1]
452 | x2 = boxes[:, 2]
453 | y2 = boxes[:, 3]
454 |
455 | areas = (x2 - x1) * (y2 - y1)
456 | _, order = scores.sort(0, descending=True)
457 |
458 | keep = []
459 | while order.numel() > 0:
460 | i = order[0]
461 | keep.append(i)
462 |
463 | if order.numel() == 1:
464 | break
465 |
466 | xx1 = torch.maximum(x1[i], x1[order[1:]])
467 | yy1 = torch.maximum(y1[i], y1[order[1:]])
468 | xx2 = torch.minimum(x2[i], x2[order[1:]])
469 | yy2 = torch.minimum(y2[i], y2[order[1:]])
470 |
471 | w = torch.maximum(torch.tensor(0.0), xx2 - xx1)
472 | h = torch.maximum(torch.tensor(0.0), yy2 - yy1)
473 | overlap = (w * h) / areas[order[1:]]
474 |
475 | ids = (overlap <= threshold).nonzero().squeeze()
476 | if ids.numel() == 0:
477 | break
478 | order = order[ids + 1]
479 | return keep
480 |
481 | >>>>>>>>>>>>>>>>>>>>>> KISS <<<<<<<<<<<<<<<<<<<<<<<<
482 | import torch
483 | import torchvision.ops as ops
484 | # Apply NMS from torchvision
485 | nms_indices = ops.nms(boxes, scores, threshold=0.5)
486 | ```
487 | - **Avoid Over-Engineering**:
488 | - Over-engineering refers to the practice of making a project or a system more complicated than necessary, often adding extra features or complexity that do not add significant value. This can manifest as overly complex code, over-abstracted architectures, or unnecessary features that complicate maintenance and understanding without providing proportional benefits. Here is an example of over-engineering, say
489 | we need a function to calculate then mean value of a list:
490 | ```python
491 | >>>>>>>>>>>>>>>>>>>> Non-KISS <<<<<<<<<<<<<<<<<<<<<<
492 | class Number:
493 | def __init__(self, value):
494 | self.value = value
495 |
496 | class NumberList:
497 | def __init__(self, numbers):
498 | self.numbers = [Number(num) for num in numbers]
499 |
500 | def sum(self):
501 | return sum(number.value for number in self.numbers)
502 |
503 | def length(self):
504 | return len(self.numbers)
505 |
506 | def average(self):
507 | return self.sum() / self.length() if self.length() > 0 else 0
508 |
509 | # Usage
510 | number_list = NumberList([10, 20, 30, 40, 50])
511 | avg = number_list.average()
512 | print("Average:", avg)
513 |
514 | >>>>>>>>>>>>>>>>>>>>>> KISS <<<<<<<<<<<<<<<<<<<<<<<<
515 | def average(numbers):
516 | return sum(numbers) / len(numbers) if numbers else 0
517 |
518 | # Usage
519 | avg = average([10, 20, 30, 40, 50])
520 | print("Average:", avg)
521 | ```
522 |
523 | ### Write Efficient Code
524 | Writing efficient code means optimizing both for speed and resource usage, ensuring that the application runs smoothly and responds quickly, even as the complexity of tasks or the size of data increases.
525 |
526 | - **Algorithmic Complexity:**
527 | - The efficiency of code often comes down to its algorithmic complexity, commonly referred to as Big O notation. It's crucial to choose the right algorithm and data structure for a given task. For example, in a sorting task, using QuickSort (average complexity $O(nlogn)$) is generally more efficient than Bubble Sort (average complexity $O(n^2)$).
528 | - **Efficient Data Handling**:
529 | - Efficient data handling is crucial, especially in data-heavy situations. For example, when working with large datasets, it's better to use generators instead of lists to avoid loading the entire dataset into memory. Similarly, when working with images, it's better to use a data loader that loads images on-demand rather than loading all images into memory at once. Here is an example of using `yield` to create a generator to load TSV (Tab-Separated Values) file:
530 | ```python
531 | >>>>>>>>>>>>>>>>>>>> Non-Efficient <<<<<<<<<<<<<<<<<<<<<<
532 | def load_data_into_list(file_path):
533 | data = []
534 | with open(file_path, 'r') as file:
535 | for line in file:
536 | columns = line.strip().split('\t')
537 | data.append(columns)
538 | return data
539 | # this will load all data into memory, which is not efficient
540 | data = load_data_into_list('data.tsv')
541 |
542 | >>>>>>>>>>>>>>>>>>>>>> Efficient <<<<<<<<<<<<<<<<<<<<<<<<
543 | def load_data_generator(file_path):
544 | with open(file_path, 'r') as file:
545 | for line in file:
546 | columns = line.strip().split('\t')
547 | yield columns
548 | # this will create a generator that loads data on-demand
549 | data_generator = load_data_generator('data.tsv')
550 | for data in data_generator:
551 | # Process each line of data here
552 | ```
553 | - **Vectorization and Parallelization**:
554 | - Vectorization and parallelization are two techniques that can significantly improve the performance of code. Vectorization refers to performing operations on entire arrays rather than individual elements. For example, in NumPy, we can add two arrays of the same shape using the `+` operator, which will add the corresponding elements of the two arrays. This is much more efficient than using a `for` loop to add the elements one by one. Similarly, parallelization refers to performing operations in parallel, using multiple processors or CPU cores. Here is an example of vectorization and parallelization in action:
555 | ```python
556 | >>>>>>>>>>>>>>>>>>>> Non-Efficient <<<<<<<<<<<<<<<<<<<<<<
557 | import numpy as np
558 | def add_arrays(a, b):
559 | result = []
560 | for i in range(len(a)):
561 | result.append(a[i] + b[i])
562 | return result
563 | a = np.random.rand(1000000)
564 | b = np.random.rand(1000000)
565 | # this will take a long time to execute
566 | c = add_arrays(a, b)
567 |
568 | >>>>>>>>>>>>>>>>>>>>>> Efficient <<<<<<<<<<<<<<<<<<<<<<<<
569 | import numpy as np
570 | def add_arrays(a, b):
571 | return a + b
572 | a = np.random.rand(1000000)
573 | b = np.random.rand(1000000)
574 | # this will execute much faster
575 | c = add_arrays(a, b)
576 | ```
577 |
578 | ## 3. Maintainability
579 | Maintainability refers to how easily software can be maintained over time. This includes fixing bugs, improving functionality, and update to meet new requirements or work with new technologies. High maintainability is crucial for the long-term success and adaptability of a deep learning project.
580 |
581 | ### Use Registry Mechanism to Manage Your Code
582 | Quoting from [MMEngine](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/registry.html#registry), "The registry can be considered as a union of a mapping table and a build function of modules. The mapping table maintains a ***mapping from strings to classes or functions***, allowing the user to find the corresponding class or function with its name/notation. For example, the mapping from the string "ResNet" to the ResNet class. The module build function defines how to find the corresponding class or function based on a string and how to instantiate the class or call the function. "
583 |
584 | Now let's look at an example of how registry mechanism works
585 | ```python
586 | @BACKBONES.register_module()
587 | class ClipViTWrapper(nn.Module):
588 | def __init__(self, return_pool_feature: bool = True, freeze: bool = False):
589 | super(ClipViTWrapper, self).__init__()
590 | self.model = CLIPVisionModel.from_pretrained(
591 | 'openai/clip-vit-base-patch32')
592 | self.return_pool_feature = return_pool_feature
593 | if freeze:
594 | for param in self.model.parameters():
595 | param.requires_grad = False
596 |
597 | def forward(self, x: torch.Tensor):
598 | ...
599 |
600 | >>>>>>>>>>>>>>>>>>>> W/O Registry <<<<<<<<<<<<<<<<<<<<<<
601 | from ...backbones import ClipViTWrapper
602 | model = ClipViTWrapper(return_pool_feature=True, freeze=False)
603 |
604 | >>>>>>>>>>>>>>>>>>>>>> W/ Registry <<<<<<<<<<<<<<<<<<<<<<<<
605 | model = BACKBONES.build(dict(type='ClipViTWrapper', return_pool_feature=True, freeze=False))
606 | ```
607 |
608 | - **Decoupling Code and Configuration**: The registry pattern allows for the creation and configuration of objects like `ClipViTWrapper` in a more dynamic and decoupled manner. Instead of directly importing and instantiating the class, you use a registry (`BACKBONES`) to build the object. This separation of object creation from usage enhances flexibility and maintainability.
609 |
610 | - **Enhanced Flexibility**: With the registry, you can easily switch between different backbone models (like `ClipViTWrapper`) by changing a configuration file or a dictionary.
611 |
612 | - **Scalability**: As the project grows, the registry pattern scales well, accommodating more components without increasing
613 |
614 | ### Organize Your Code by Functionality
615 | Organizing code by functionality is a crucial aspect of maintainability. It involves grouping related code into modules, packages, and libraries, and separating unrelated code into distinct sections. This makes it easier to find and update code, and also helps avoid conflicts between different parts of the codebase. Generally, an AI project is composed of three parts: model, dataset, and computation depicited in the following figure.
616 |
617 |
618 |
619 | 
620 |
621 |
622 | - `Model`: The model part is the core of an AI project, which is composed of the model architecture, loss function, other components. For model architecture, it can be divided into backbone, encoder, decoder, head and other components depending on the specific task.
623 | - `Dataset`: The dataset part is the data source of an AI project, which is composed of data loader, data augmentation, data preprocessing, and other components.
624 | - `Computation`: The computation part is where you train and test your model, which is composed of training, evaluation, and other components.
625 |
626 | Based on the above three parts, we can organize our code by functionality as follows:
627 |
628 |
629 |
630 | 
631 |
632 |
633 | - `configs`: The configs part is where you store all the configuration files, including the dataset configuration, model configuration, training configuration, and others.
634 | - `datasets`: The datasets part is where you store all the dataset-related code, including data loader, data augmentation, data preprocessing, and others.
635 | - `models`: The models part is where you store all the model-related code, including backbone, encoder, decoder, head, loss function, and others.
636 | - `engine`: The engine part is where you store all the computation-related code, including training, evaluation, and others.
637 | - `tools`: The tools part is where you store all the tools-related code, including visualization, logging, and others.
638 | - `tests`: The tests part is where you store all the test-related code, including unit test, integration test, and others.
639 | - `scripts`: The scripts part is where you store all the scripts-related code, which are used to launch the training, evaluation, and others, especially for slurm-based distributed training.
640 | - `README.md`: The `README.md` part is where you store the documentation of your project, including the installation, usage, and others.
641 | - `LICENSE`: The LICENSE part is where you store the license of your project, which is crucial for open-source projects.
642 | - `.gitignore`: The .gitignore part is where you store the files that you don't want to upload to the remote repository, which is crucial for open-source projects.
643 | - `requirements.txt`: The requirements.txt part is where you store the dependencies of your project, which is crucial for open-source projects.
644 | - `train.py`: The `train.py` part is a port to launch the training of your project.
645 | - `eval.py`: The `eval.py` part is a port to launch the evaluation of your project.
646 | - `inference.py`: The `inference.py` part is a port to launch the inference of your project.
647 |
--------------------------------------------------------------------------------