├── databases └── README.md ├── .github └── FUNDING.yml ├── dump ├── 08.Retry-Mechanism.md ├── 14.Merkle-Trees.md ├── 03.Bloom-filter.md ├── 15.Why-use-Distributed-Databases?.md ├── 12.The-Antifragile-Organization.md ├── 13.Write-Caching.md ├── 05.Consistent-Hashing.md ├── 11.Failover.md ├── 16.Auth-and-ways.md ├── 10.Replication-Lag.md ├── 06.Consistency-Models.md ├── 07.Failure-Detection-Heart-beat-and-Gossip-Protocol.md ├── 01.Intverted-Index.md ├── 09.Message-Queues-Pub-Sub.md ├── 02.Database-Replications.md └── 04.CAP-Theorem.md ├── networking └── README.md └── README.md /databases/README.md: -------------------------------------------------------------------------------- 1 | ### ACID Transactions - Relational Database 2 | ### Primary Key and Secondary key 3 | ### B Tree and B+ Tree 4 | ### Database Indexing 5 | ### Leaking Postgres database connections 6 | ### Database Engines 7 | ### Fail-over and High-Availability 8 | ### Active-Active vs Active-Passive Cluster to achieve High Availability 9 | ### Connection Pooling in PostgresSQL 10 | ### Horizontal vs Vertical Database Partitioning 11 | ### Database Partitioning 12 | ### Database Sharding 13 | ### Can you get Eventual Consistency in Relational Database? 14 | ### Avoid Double Booking and Race Conditions -------------------------------------------------------------------------------- /.github/FUNDING.yml: -------------------------------------------------------------------------------- 1 | # These are supported funding model platforms 2 | 3 | github: dipjul # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2] 4 | patreon: # Replace with a single Patreon username 5 | open_collective: # Replace with a single Open Collective username 6 | ko_fi: # Replace with a single Ko-fi username 7 | tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel 8 | community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry 9 | liberapay: # Replace with a single Liberapay username 10 | issuehunt: # Replace with a single IssueHunt username 11 | lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry 12 | polar: # Replace with a single Polar username 13 | buy_me_a_coffee: # Replace with a single Buy Me a Coffee username 14 | thanks_dev: # Replace with a single thanks.dev username 15 | custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2'] 16 | -------------------------------------------------------------------------------- /dump/08.Retry-Mechanism.md: -------------------------------------------------------------------------------- 1 | ### **Retry Mechanisms with Back-off Strategy** 2 | 3 | #### **1. Core Concept: Why Retry?** 4 | 5 | In distributed systems, failures are not the exception; they are the norm. Network glitches, temporary service unavailability (HTTP 503), timeouts, and throttling (HTTP 429) are common. A retry mechanism allows an application to automatically re-attempt a failed operation, increasing the chance of eventual success without user intervention. 6 | 7 | **Key Principle:** Only retry operations that are **idempotent** (can be applied multiple times without changing the result beyond the initial application). Non-idempotent operations (e.g., `ProcessPayment()`) risk duplicate charges if retried naively. 8 | 9 | --- 10 | 11 | #### **2. The Problem with Simple Retries** 12 | 13 | A simple retry (e.g., "try 3 times with a 1-second wait") is often worse than no retry at all. 14 | * **Cascading Failures:** If a service is struggling under load, immediate retries from all clients create a "retry storm," overwhelming the service further and preventing recovery. This is a classic positive feedback loop that leads to an outage. 15 | * **Inefficiency:** Retrying too quickly is futile if the failure is due to a temporary load spike or a network partition that takes more than a few seconds to resolve. 16 | 17 | --- 18 | 19 | #### **3. The Solution: Back-off Strategies** 20 | 21 | A back-off strategy intelligently increases the wait time between subsequent retry attempts. This gives the failing system time to recover, drain queues, or scale out. 22 | 23 | **Core Components of a Retry Policy:** 24 | 1. **Retry Count:** Maximum number of attempts (`max_retries`). 25 | 2. **Back-off Strategy:** The algorithm to calculate the delay between attempts. 26 | 3. **Jitter:** A random value added to the delay to prevent client synchronization (thundering herd problem). 27 | 28 | --- 29 | 30 | #### **4. Common Back-off Strategies** 31 | 32 | | Strategy | Description | Formula (for attempt `n`) | Pros | Cons | 33 | | :--- | :--- | :--- | :--- | :--- | 34 | | **Constant** | Waits a fixed amount of time between each attempt. | `delay = constant_value` | Simple to implement. | Does not help reduce load on the overwhelmed system. | 35 | | **Linear** | Wait time increases by a fixed amount each attempt. | `delay = initial_delay + (n * increment)` | Slightly better than constant. | Still not aggressive enough for back-off; can contribute to load. | 36 | | **Exponential** | Wait time doubles (or scales by a factor) with each attempt. **This is the most common strategy.** | `delay = initial_delay * (base ^ n)`
e.g., base=2: 1s, 2s, 4s, 8s... | Very effective at reducing load on the server. Allows ample time for recovery. | Can lead to long waits for the client. Without jitter, clients can synchronize. | 37 | | **Fibonacci** | Wait time follows the Fibonacci sequence. | `delay = F(n) * initial_delay`
e.g., 1s, 1s, 2s, 3s, 5s... | A smoother alternative to exponential back-off. | Less common, slightly more complex to implement. | 38 | 39 | --- 40 | 41 | #### **5. Critical Enhancement: Jitter (Randomness)** 42 | 43 | Without jitter, all clients using the same exponential back-off policy will retry at the same times (e.g., at 1s, 2s, 4s, etc.). This synchronized retry storm defeats the purpose of back-off. 44 | 45 | **Adding jitter de-synchronizes the clients.** You can apply jitter in different ways: 46 | * **Full Jitter:** `delay = random_between(0, base ^ n)` 47 | * **Equal Jitter:** `delay = (base ^ n / 2) + random_between(0, base ^ n / 2)` 48 | * **Decorrelated Jitter:** `delay = random_between(initial_delay, previous_delay * 3)` 49 | 50 | **Recommendation:** Start with **Full Jitter** for its simplicity and effectiveness in spreading out load. 51 | 52 | --- 53 | 54 | #### **6. Circuit Breaker Pattern: When to Stop Retrying** 55 | 56 | A retry mechanism with back-off is powerful, but it must be combined with the **Circuit Breaker** pattern for production resilience. 57 | 58 | * **Retry:** Answers "Should I retry *this specific call* again?" 59 | * **Circuit Breaker:** Answers "Should I even *make any calls* to this failing service?" 60 | 61 | The circuit breaker trips after a threshold of failures is reached. All subsequent calls immediately fail ("fast fail") without even attempting the network request. After a timeout period, it allows a few test requests to pass to see if the service has recovered. 62 | 63 | **Libraries like `resilience4j` (Java) or `polly` (.NET)** seamlessly combine retry and circuit breaker logic. 64 | 65 | --- 66 | 67 | ### **External Materials for In-Depth Learning** 68 | 69 | #### **1. Foundational Articles & Papers** 70 | * **AWS Architecture Blog: Exponential Backoff and Jitter** 71 | * **Link:** [https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) 72 | * **Why:** The canonical resource on the topic. Clearly explains the different types of jitter with graphs showing their effectiveness. 73 | 74 | * **Google Cloud: Retry Strategies** 75 | * **Link:** [https://cloud.google.com/architecture/retry-strategies](https://cloud.google.com/architecture/retry-strategies) 76 | * **Why:** Excellent, practical guide from Google. Covers simple retries, exponential backoff, and how to handle non-idempotent operations. 77 | 78 | * **Martin Fowler: Circuit Breaker** 79 | * **Link:** [https://martinfowler.com/bliki/CircuitBreaker.html](https://martinfowler.com/bliki/CircuitBreaker.html) 80 | * **Why:** The definitive introduction to the Circuit Breaker pattern by the man who named it. Essential reading for understanding how to complement retries. 81 | 82 | #### **2. Implementation Libraries (See code for best practices)** 83 | * **Polly (.NET):** 84 | * **Link:** [https://github.com/App-vNext/Polly](https://github.com/App-vNext/Polly) 85 | * **Why:** A fantastic .NET library that provides fluent APIs for retry, circuit breaker, fallback, and more. Reading its documentation is a masterclass in resilience patterns. 86 | 87 | * **Resilience4j (Java):** 88 | * **Link:** [https://github.com/resilience4j/resilience4j](https://github.com/resilience4j/resilience4j) 89 | * **Why:** The modern successor to Netflix Hystrix, designed for Java 8 and functional programming. Excellent documentation and examples. 90 | 91 | * **Apache Commons Retry (Java):** 92 | * **Link:** [https://github.com/apache/commons-retry](https://github.com/apache/commons-retry) 93 | * **Why:** A simpler, dedicated library for retry operations in Java. 94 | 95 | #### **3. Books** 96 | * **"Designing Data-Intensive Applications" by Martin Kleppmann** 97 | * **Chapter 8: "The Trouble with Distributed Systems"** 98 | * **Why:** This book is the bible for understanding distributed systems fundamentals. This chapter specifically discusses faults, network problems, and the philosophy behind building reliable systems, providing the crucial context for why patterns like retry and back-off are necessary. 99 | 100 | * **"Release It!" by Michael T. Nygard** 101 | * **Chapter 4: "Stability Patterns" (Circuit Breaker, Retry, etc.)** 102 | * **Why:** A more pragmatic, operations-focused book. It's filled with war stories and directly addresses how to prevent cascading failures and build stable systems. The patterns discussed are the direct implementation of the concepts above. -------------------------------------------------------------------------------- /dump/14.Merkle-Trees.md: -------------------------------------------------------------------------------- 1 | ### **Merkle Trees** 2 | 3 | #### **1. Core Concept: The "Digital Fingerprint"** 4 | 5 | A Merkle Tree (or Hash Tree) is a data structure used in computer science to efficiently and securely verify the contents of a large set of data. It's a tree in which every leaf node is labelled with the cryptographic hash of a data block, and every non-leaf node is labelled with the cryptographic hash of the labels of its child nodes. 6 | 7 | **The simplest analogy:** Think of it as a tamper-evident seal for a dataset. If any single piece of the original data changes, the "root" of the tree changes completely, signaling that the data has been altered. 8 | 9 | #### **2. How It Works: Step-by-Step Construction** 10 | 11 | Let's say we have four transactions in a block: `TxA`, `TxB`, `TxC`, and `TxD`. 12 | 13 | 1. **Hash the Data:** First, each transaction is passed through a cryptographic hash function (like SHA-256). This gives us: 14 | * `Hash A = H(TxA)` 15 | * `Hash B = H(TxB)` 16 | * `Hash C = H(TxC)` 17 | * `Hash D = H(TxD)` 18 | 19 | 2. **Build the Tree:** These hashes are then paired and hashed together repeatedly. 20 | * Combine `Hash A` and `Hash B` and hash them to create `Hash AB = H(Hash A + Hash B)`. 21 | * Combine `Hash C` and `Hash D` and hash them to create `Hash CD = H(Hash C + Hash D)`. 22 | * Finally, combine `Hash AB` and `Hash CD` and hash them to create the **Merkle Root**: `Hash ABCD = H(Hash AB + Hash CD)`. 23 | 24 | This structure now looks like an inverted tree: 25 | ``` 26 | Merkle Root (Hash ABCD) 27 | / \ 28 | / \ 29 | Hash AB Hash CD 30 | / \ / \ 31 | Hash A Hash B Hash C Hash D 32 | | | | | 33 | Tx A Tx B Tx C Tx D 34 | ``` 35 | 36 | #### **3. Key Properties and Advantages** 37 | 38 | * **Efficient Verification (Merkle Proofs):** This is the killer feature. You don't need the entire dataset to verify if a specific transaction (`TxC`) is included in the block. You only need: 39 | * `TxC` itself 40 | * `Hash D` (its pair) 41 | * `Hash AB` (the hash of the opposite branch) 42 | With this small amount of data, you can recompute `Hash CD`, then recompute the Merkle Root and check if it matches the one you trust. This is called a **Merkle Proof** or **Authentication Path**. 43 | * **Tamper-Evident:** Any change in any transaction (e.g., `TxB` is altered) will change `Hash B`. This changes `Hash AB`, which ultimately changes the **Merkle Root**. A changed root is immediate proof of inconsistency. 44 | * **Scalability:** The Merkle Root is a single, fixed-size string (e.g., 32 bytes for SHA-256), regardless of whether it represents four transactions or four million. This allows a concise summary of vast amounts of data. 45 | 46 | #### **4. Common Applications** 47 | 48 | * **Blockchains (Bitcoin, Ethereum, etc.):** This is the most famous use case. Each block header contains the Merkle Root of all its transactions. This allows: 49 | * **Simplified Payment Verification (SPV):** Light clients (like mobile wallets) can download just the block headers and use Merkle proofs to verify that a transaction was included in a block without processing the entire blockchain. 50 | * Ensuring the immutability of the transaction history. 51 | * **Peer-to-Peer Networks (P2P):** Used in protocols like BitTorrent, Git, and IPFS to verify that data chunks downloaded from different peers are authentic and haven't been corrupted. 52 | * **Cryptographic File Systems:** Systems like ZFS and Btrfs use Merkle Trees to detect silent data corruption on disks. 53 | * **Database Systems:** Used for efficient data verification and synchronization (e.g., Apache Cassandra uses them for anti-entropy repair). 54 | 55 | #### **5. Important Considerations & Variations** 56 | 57 | * **Handling an Odd Number of Leaves:** If there's an odd number of items at any level, the common solution is to **duplicate the last hash**. (e.g., For three transactions: `[TxA, TxB, TxC]`, you would hash `TxC` with itself: `Hash CC = H(Hash C + Hash C)`). 58 | * **Binary vs. n-ary Trees:** The standard is a binary tree, but trees can be built with more than two children per node for different efficiency trade-offs. 59 | * **Merkle Patricia Trees/Tries:** Ethereum uses a more complex variant called a Merkle Patricia Tree for its state tree. This allows not only verification of transactions but also efficient verification of the entire state of accounts and smart contracts. 60 | 61 | --- 62 | 63 | ### **External Materials for In-Depth Learning** 64 | 65 | Here is a curated list of resources, from beginner-friendly to advanced. 66 | 67 | #### **1. Introductory & Visual Explanations** 68 | 69 | * **Everest Pipkin's "Merkle Trees" Illustration:** A beautiful, visual, and intuitive walkthrough. 70 | * **Link:** [https://everest-pipkin.com/teaching/merkle](https://everest-pipkin.com/teaching/merkle) 71 | * **Khan Academy (Bitcoin Course):** The "Bitcoin: Cryptographic hash functions" and "Bitcoin: Merkle trees" videos are excellent. 72 | * **Link:** [https://www.khanacademy.org/economics-finance-domain/core-finance/money-and-banking/bitcoin/v/bitcoin-merkle-trees](https://www.khanacademy.org/economics-finance-domain/core-finance/money-and-banking/bitcoin/v/bitcoin-merkle-trees) 73 | * **3Blue1Brown - "But how does bitcoin actually work?":** While about Bitcoin, it contains one of the best visual explanations of Merkle Trees and their purpose (~14:30 mark). 74 | * **Link:** [https://www.youtube.com/watch?v=bBC-nXj3Ng4](https://www.youtube.com/watch?v=bBC-nXj3Ng4) 75 | 76 | #### **2. Technical Deep Dives & Code** 77 | 78 | * **Bitcoin Wiki - Merkle Tree:** The canonical technical reference from the Bitcoin perspective. Very precise. 79 | * **Link:** [https://en.bitcoin.it/wiki/Merkle_tree](https://en.bitcoin.it/wiki/Merkle_tree) 80 | * **Wikipedia - Merkle Tree:** A great general overview with formal definitions and examples. 81 | * **Link:** [https://en.wikipedia.org/wiki/Merkle_tree](https://en.wikipedia.org/wiki/Merkle_tree) 82 | * **GeeksforGeeks - Merkle Tree:** Includes a Python implementation, which is great for understanding the algorithm. 83 | * **Link:** [https://www.geeksforgeeks.org/introduction-to-merkle-tree/](https://www.geeksforgeeks.org/introduction-to-merkle-tree/) 84 | 85 | #### **3. Advanced & Specialized Topics** 86 | 87 | * **Ethereum's Merkle Patricia Trees:** The official documentation explains the more complex tree structure used by Ethereum. 88 | * **Link:** [https://ethereum.org/en/developers/docs/data-structures-and-encoding/patricia-merkle-trie/](https://ethereum.org/en/developers/docs/data-structures-and-encoding/patricia-merkle-trie/) 89 | * **Vitalik Buterin's "Merkling in Ethereum":** A blog post by Ethereum's co-founder explaining the motivation and utility of Merkle trees in Ethereum. 90 | * **Link:** [https://blog.ethereum.org/2015/11/15/merkling-in-ethereum](https://blog.ethereum.org/2015/11/15/merkling-in-ethereum) 91 | * **RFC 6962 - Certificate Transparency:** The IETF standard that defines how Merkle Trees are used for Certificate Transparency in web security. A great real-world spec. 92 | * **Link:** [https://datatracker.ietf.org/doc/html/rfc6962](https://datatracker.ietf.org/doc/html/rfc6962) 93 | 94 | #### **4. For the Mathematically Inclined** 95 | 96 | * **Original Paper by Ralph Merkle:** For the historical and purely cryptographic perspective. 97 | * **Title:** "A Digital Signature Based on a Conventional Encryption Function" 98 | * **Link:** (Often paywalled, but searchable). It's a seminal paper in cryptography. -------------------------------------------------------------------------------- /dump/03.Bloom-filter.md: -------------------------------------------------------------------------------- 1 | ### **1. Core Concept: What is a Bloom Filter?** 2 | 3 | A Bloom filter is a **probabilistic, space-efficient** data structure used to test whether an element is a member of a set. It is designed to be incredibly fast and use minimal memory, but at a cost: it can have **false positives**. 4 | 5 | * **False Positive:** It might tell you "yes, this element is in the set" when in fact it is not. 6 | * **False Negative:** It will **never** tell you "no, this element is not in the set" if it actually is. This is a crucial guarantee. If a Bloom filter says an element is not present, it is definitely not present. 7 | 8 | **Analogy:** Think of it as a fuzzy, high-level membership test. "I'm *pretty sure* I've seen this before, but I'd have to check the actual list to be 100% certain." 9 | 10 | ### **2. How It Works: The Implementation** 11 | 12 | A Bloom filter consists of two parts: 13 | 1. A **bit array** of size `m` (all bits initially set to `0`). 14 | 2. `k` different **hash functions**, each of which maps an element to one of the `m` array positions. 15 | 16 | **Operations:** 17 | 18 | **a) Adding an Element:** 19 | For each element you want to add to the set: 20 | 1. Hash the element with all `k` hash functions. 21 | 2. For each hash output, calculate the position modulo `m` to get an index in the bit array. 22 | 3. Set the bit at each of these `k` indices to `1`. 23 | ``` 24 | Example: Add "hello" 25 | Hash1("hello") -> index 3 26 | Hash2("hello") -> index 8 27 | Hash3("hello") -> index 12 28 | 29 | Bit array: [0,0,0,1,0,0,0,0,1,0,0,0,1,0,0] 30 | ``` 31 | 32 | **b) Querying an Element (Checking Membership):** 33 | For an element you want to check: 34 | 1. Hash the element with all `k` hash functions. 35 | 2. For each hash output, calculate the position modulo `m`. 36 | 3. **If *any* of the bits at these `k` indices is `0`, the element is definitely NOT in the set.** 37 | 4. **If *all* of the bits are `1`, the element is PROBABLY in the set.** 38 | ``` 39 | Example: Query "world" 40 | Hash1("world") -> index 3 (bit is 1) 41 | Hash2("world") -> index 7 (bit is 0) -> STOP. Definitely not present. 42 | 43 | Example: Query "foo" 44 | Hash1("foo") -> index 3 (bit is 1) 45 | Hash2("foo") -> index 12 (bit is 1) 46 | Hash3("foo") -> index 8 (bit is 1) 47 | -> All bits are 1. "foo" is probably present (but we never added it! This is a false positive). 48 | ``` 49 | 50 | **Key Limitation:** You **cannot remove** an element from a standard Bloom filter. Setting a bit from `1` to `0` might break the membership test for other elements that also hash to that same bit. (Variants like *Counting Bloom Filters* exist to handle deletions). 51 | 52 | ### **3. How It's Set Up: Sizing and Parameters** 53 | 54 | The performance of a Bloom filter is determined by three parameters: 55 | 1. `n`: The number of elements you expect to add. 56 | 2. `m`: The size of the bit array (in bits). 57 | 3. `k`: The number of hash functions. 58 | 59 | The **false positive rate (FPR)** is approximated by the formula: 60 | $$(1 - e^{-kn/m})^k$$ 61 | 62 | You don't need to derive this, but you need to know its implications. To set up an efficient Bloom filter, you follow this process: 63 | 64 | 1. **Define your constraints:** Decide on your target false positive rate (`f`) and the expected number of elements (`n`). 65 | 2. **Calculate the required size `m`:** Use the formula or a common rule of thumb: 66 | > **Rule of Thumb:** For a 1% false positive rate (`f = 0.01`), you need approximately **9.6 bits per element**. So `m ≈ 9.6 * n`. 67 | > For example, for 1 million elements, you'd need ~9.6 million bits, which is only about **1.14 MB**. This incredible space efficiency is its superpower. 68 | 3. **Calculate the optimal number of hash functions `k`:** 69 | > **Rule of Thumb:** `k ≈ (m / n) * ln(2)`. For our 1% FPR example, this works out to `k ≈ 7`. 70 | 4. **Implement the hash functions:** You don't need `k` *different* hash algorithms. In practice, you can use two good hash functions (e.g., `xxHash`, `MurmurHash`) and simulate `k` functions using the formula: `h_i(x) = h1(x) + i * h2(x)` for `i` from `0` to `k-1`. 71 | 72 | ### **4. How It Scales: The False Positive Rate** 73 | 74 | The relationship between the filter's size, its capacity, and the false positive rate is its scaling property. 75 | 76 | * **As you add more elements (`n`) beyond what you designed for:** The false positive rate **increases non-linearly**. The filter becomes "saturated" with `1`s. The more `1`s in the bit array, the higher the probability that a query for a new element will hit all `1`s by chance. 77 | * **If you want to maintain the same FPR as `n` grows:** You must **increase the size of the bit array (`m`)** proportionally. Remember the rule: `m` must scale linearly with `n` to keep the same FPR (`m ≈ 9.6 * n` for 1%). 78 | * **The relationship is a trade-off:** It's a classic trade-off between **space (`m`)** and **accuracy (`f`)**. You can have a tiny filter, but it will have a high false positive rate. You can have a very low false positive rate, but it will require a larger filter. 79 | 80 | **Scaling in Practice: Resizing** 81 | A common strategy is to create a **scalable Bloom filter**. When the current filter becomes too full (i.e., the FPR exceeds a desired threshold), you create a new, larger Bloom filter and add all new elements to it. To check for membership, you check all filters in the chain. This prevents the FPR from growing uncontrollably. 82 | 83 | ### **5. Use Cases (Where to Use It)** 84 | 85 | Bloom filters are used in situations where the cost of a false positive is acceptable and space/ speed is critical. 86 | 87 | 1. **Web Crawling:** Check if a URL has already been crawled before adding it to the queue. 88 | 2. **Database Systems:** (e.g., Apache Cassandra, HBase) Avoid expensive disk lookups for non-existent rows. "Check the Bloom filter first; if it says no, skip the disk seek." 89 | 3. **Content Delivery Networks (CDNs):** Check if a piece of content is cached on an edge server. 90 | 4. **Malware & Password Checkers:** Check a file hash against a known set of bad hashes. A "no" from the filter means it's safe. A "yes" means it needs a more expensive, definitive check. 91 | 5. **Medium's "Seen Posts" / Facebook's "Seen Articles":** A lightweight way to track billions of user-article interactions without storing gigantic lists. 92 | 93 | --- 94 | 95 | ### **6. External Materials for Deeper Learning** 96 | 97 | **Articles & Tutorials:** 98 | * **Bloom Filters by Example (Jason Davies):** An excellent interactive website that lets you visualize adds and queries. **https://www.jasondavies.com/bloomfilter/** 99 | * **The Bloom Filter (LLHarbin on Medium):** A very clear, well-written explanation with graphics. **https://llharbin.medium.com/the-bloom-filter-3e5b690d9f1b** 100 | * **Wikipedia - Bloom Filter:** Surprisingly good for the mathematical details and variants. **https://en.wikipedia.org/wiki/Bloom_filter** 101 | 102 | **Videos:** 103 | * **Bloom Filters | The easy way! (Gaurav Sen):** A fantastic whiteboard explanation that builds intuition. **https://www.youtube.com/watch?v=V3pzxngeLqw** 104 | * **What are Bloom Filters? (Hussein Nasser):** A practical take with real-world examples and code. **https://www.youtube.com/watch?v=QRcV0H004Eg** 105 | 106 | **Books:** 107 | * **"Mining of Massive Datasets" by Jure Leskovec, Anand Rajaraman, Jeff Ullman:** Has an entire chapter dedicated to Bloom filters and other probabilistic data structures. Available free online: **http://www.mmds.org/** 108 | * **"Algorithm Design Manual" by Steven S. Skiena:** Discusses Bloom filters in the context of hashing and set membership. 109 | 110 | **Advanced Topics to Explore Later:** 111 | * **Counting Bloom Filters:** Allow for deletions. 112 | * **Scalable Bloom Filters:** Automatically grow to accommodate more data. 113 | * **Cuckoo Filters:** A modern alternative that can also support deletions and often has better performance. 114 | * **Quotient Filters:** Another space-efficient alternative. -------------------------------------------------------------------------------- /dump/15.Why-use-Distributed-Databases?.md: -------------------------------------------------------------------------------- 1 | ### **Why Use a Distributed Database?** 2 | 3 | #### **1. Core Concept: What is a Distributed Database?** 4 | A **distributed database** is a database in which data is stored across multiple physical locations, which may be spread across a network of interconnected computers (nodes). Crucially, to the end-user or application, it often appears as a single, logical database. The system manages the distribution, replication, and consistency of the data. 5 | 6 | #### **2. Primary Motivations and Advantages** 7 | 8 | **A. Scalability (The Biggest Reason)** 9 | * **Horizontal Scaling (Scale-Out):** Unlike a single monolithic server (which scales **vertically** by adding more CPU, RAM, etc.), distributed databases are designed to scale **horizontally**. You add more commodity machines (nodes) to the cluster to handle increased load. 10 | * **Handles Massive Data Volumes:** Ideal for Big Data applications where the dataset size exceeds the capacity of a single machine (e.g., petabytes of data). 11 | * **Handles High Throughput:** Distributes read and write operations across many nodes, allowing the system to serve a much larger number of simultaneous users and transactions. 12 | 13 | **B. High Availability and Fault Tolerance** 14 | * **No Single Point of Failure:** Data is **replicated** across multiple nodes. If one node (or even an entire data center) fails, the system can continue operating by routing requests to other nodes that have a copy of the data. 15 | * **Continuous Uptime:** Essential for mission-critical applications (e.g., financial systems, e-commerce, healthcare) where downtime translates directly to lost revenue or safety risks. 16 | * **Disaster Recovery:** Geographic distribution allows data to survive regional outages, natural disasters, or network partitions. 17 | 18 | **C. Improved Performance and Reduced Latency** 19 | * **Data Locality:** Data can be placed geographically close to its users. A user in Japan can read from a node in Tokyo, while a user in Brazil reads from a node in São Paulo. This significantly reduces read latency. 20 | * **Parallel Processing:** Queries can be broken down and executed in parallel across multiple nodes, dramatically speeding up complex analytical queries (a principle used extensively in data warehouses like BigQuery and Snowflake). 21 | 22 | **D. Organizational and Regulatory Alignment** 23 | * **Data Sovereignty:** Laws like GDPR (EU), CCPA (California), and others require that certain data must be stored and processed within specific geographic boundaries. A distributed database can easily ensure data resides in the correct region. 24 | * **Departmental Autonomy:** Different branches or departments of a large organization can have their own local node for autonomy and performance, while still being part of a larger, coherent database system. 25 | 26 | #### **3. Trade-offs and Challenges (The "Cost" of Distribution)** 27 | Distributed databases are not a silver bullet. Their advantages come with significant complexity: 28 | 29 | * **Complexity:** Design, deployment, and maintenance are vastly more complex than a single-node database. Requires specialized knowledge. 30 | * **Consistency Models:** Achieving **strong consistency** (where every read receives the most recent write) across distributed nodes is difficult and can impact performance and availability. Many distributed databases opt for **eventual consistency** or other relaxed models (see CAP Theorem) to achieve higher availability and partition tolerance. 31 | * **Increased Infrastructure & Operational Overhead:** Managing a cluster of machines requires more DevOps effort, monitoring, and tooling. 32 | * **Security:** A larger attack surface. Securing communication between nodes and ensuring consistent security policies across the cluster is critical. 33 | 34 | #### **4. When Should You Use One?** 35 | Consider a distributed database when: 36 | * Your data volume or transaction rate is outgrowing the capabilities of a single database server. 37 | * Your application demands extremely high uptime (e.g., "five-nines" or 99.999% availability). 38 | * Your user base is globally distributed, and you need to reduce latency for far-away users. 39 | * You have regulatory requirements to store data in specific locations. 40 | 41 | **When to Avoid:** For small applications, prototypes, or when strong consistency is an absolute requirement and you cannot tolerate the complexity of managing it in a distributed environment. 42 | 43 | --- 44 | 45 | ### **External Materials for In-Depth Learning** 46 | 47 | Here are categorized resources, from foundational theories to practical implementations. 48 | 49 | #### **1. Foundational Theories (Must-Understand Concepts)** 50 | * **CAP Theorem:** The fundamental theorem governing distributed systems. 51 | * **What it is:** It states that a distributed data store can only provide two of the following three guarantees: **C**onsistency, **A**vailability, and **P**artition Tolerance. 52 | * **Resource:** [**"Please stop calling databases CP or AP"** by Martin Kleppmann](https://martin.kleppmann.com/2015/05/11/please-stop-calling-databases-cp-or-ap.html) - A critical modern take on the often-misused theorem. 53 | * **PACELC Theorem:** An extension of CAP. 54 | * **Resource:** [**Wikipedia: PACELC theorem**](https://en.wikipedia.org/wiki/PACELC_theorem) - A good starting point. It refines CAP by considering the trade-offs during both network partitions (P) and normal operation (ELC). 55 | 56 | #### **2. Papers (Academic but Highly Influential)** 57 | * **Google's Bigtable:** The paper that inspired countless NoSQL databases (e.g., HBase, Cassandra). 58 | * **Link:** [**Bigtable: A Distributed Storage System for Structured Data**](https://static.googleusercontent.com/media/research.google.com/en//archive/bigtable-osdi06.pdf) 59 | * **Amazon's Dynamo:** The paper that inspired eventually consistent, highly available key-value stores (e.g., Cassandra, DynamoDB, Riak). 60 | * **Link:** [**Dynamo: Amazon’s Highly Available Key-value Store**](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf) 61 | * **Spanner:** Google's globally-distributed, strongly consistent database. 62 | * **Link:** [**Spanner: Google’s Globally-Distributed Database**](https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf) 63 | 64 | #### **3. Books** 65 | * **_Designing Data-Intensive Applications_ by Martin Kleppmann** - **This is arguably the single most recommended resource on the topic.** It thoroughly covers the concepts behind distributed databases, including replication, partitioning, consistency, and batch/stream processing. 66 | * **_Database Internals_ by Alex Petrov** - A deep dive into how databases work under the hood, with significant focus on distributed systems and storage engines. 67 | 68 | #### **4. Online Courses & Blogs** 69 | * **MIT Course: Distributed Systems (6.824)** 70 | * **Link:** [**MIT 6.824: Distributed Systems**](https://pdos.csail.mit.edu/6.824/) - A legendary course. The lectures and labs are challenging but incredibly educational. Lecture videos are often available on YouTube. 71 | * **Jepsen Distributed Systems Safety Research** 72 | * **Link:** [**Jepsen.io**](https://jepsen.io/) - Kyle Kingsbury's analyses of various distributed databases are famous for putting their claims of consistency and availability to the test. Reading the analyses for databases like Cassandra, MongoDB, or etcd is highly instructive. 73 | * **AWS re:Invent Talks** 74 | * **Search on YouTube** for talks on Amazon DynamoDB, Aurora, and other distributed database services. They often provide great high-level overviews of the architecture and trade-offs. 75 | 76 | #### **5. Hands-On Exploration** 77 | The best way to learn is to try. Set up a simple cluster on your local machine using Docker: 78 | * **Apache Cassandra:** Excellent for understanding masterless architecture, ring topology, and tunable consistency. 79 | * **CockroachDB** or **YugabyteDB:** NewSQL databases that offer a PostgreSQL-like API with horizontal scaling and strong consistency. Great for understanding distributed SQL. 80 | * **Elasticsearch:** Fantastic for understanding distributed search and index sharding. 81 | -------------------------------------------------------------------------------- /dump/12.The-Antifragile-Organization.md: -------------------------------------------------------------------------------- 1 | ### **The Antifragile Organization** 2 | 3 | #### **1. The Core Concept: Beyond Resilience and Robustness** 4 | 5 | * **Fragile**: Breaks under stress and volatility. Seeks calm and predictability. (e.g., A wine glass) 6 | * **Robust**: Withstands stress and remains unchanged. (e.g., A rock) 7 | * **Resilient**: Withstands stress and recovers to its original state. (e.g., A forest that regrows after a fire) 8 | * **Antifragile (Nassim Nicholas Taleb's concept)**: **Gains from stress, volatility, and disorder.** It *requires* shocks to thrive and grow stronger. (e.g., The human immune system, which gets stronger after exposure to germs). 9 | 10 | An **Antifragile Organization** doesn't just survive chaos; it uses market shifts, unexpected competition, technological disruptions, and even failures as a source of information and energy to become better, more innovative, and more dominant. 11 | 12 | #### **2. Key Principles of an Antifragile Organization** 13 | 14 | **a. Optionality & The Barbell Strategy** 15 | * **Concept**: Avoid putting all your resources into a single, high-stakes bet. Instead, structure your activities like a barbell: 16 | * **One end (85-90%):** Extremely safe, conservative, and robust core business. This is your cash cow, protected from downturns. 17 | * **The other end (10-15%):** Highly aggressive, high-risk, high-optionality experiments. These are small bets that could pay off enormously. 18 | * **Organizational Implication:** Protect the core while actively encouraging and funding small, experimental "skunkworks" projects. The failure of any one experiment is cheap and provides valuable data. 19 | 20 | **b. Skin in the Game & Decentralization** 21 | * **Concept**: Decision-makers must share in the risks and rewards of their decisions. This eliminates the problem of people taking massive risks with other people's money (OPM) and no downside. 22 | * **Organizational Implication:** Push decision-making authority to the edges of the organization—to the teams closest to the customer, the market, and the problem. This creates a distributed sensing-and-responding network that is faster and more adaptive than a centralized hierarchy. 23 | 24 | **c. The Redundancy & Over-Engineering Trap** 25 | * **Concept**: Redundancy (backups, spare capacity) is often seen as inefficient. For antifragile systems, it's essential. It's a form of insurance that allows the system to absorb shocks without collapsing. 26 | * **Organizational Implication:** Have slack in the system. Don't run at 100% efficiency (which is fragile). Allow for spare capacity, free time for employees to experiment (e.g., Google's "20% time"), and cross-train employees so they can cover multiple roles. 27 | 28 | **d. The Via Negativa (Less is More)** 29 | * **Concept**: It is often easier to know what is wrong than what is right. Strength and antifragility come more from *removing* the harmful (bureaucracy, single points of failure, toxic customers) than from adding new things. 30 | * **Organizational Implication:** Regularly conduct "stop-doing" reviews. What processes can we eliminate? What rules are obsolete? What products or services are making us fragile? Focus on subtraction. 31 | 32 | **e. Hormesis: Stressors Make You Stronger** 33 | * **Concept**: Small, acute doses of stress followed by recovery lead to growth. The body builds muscle by tearing it and allowing it to repair. 34 | * **Organizational Implication:** Deliberately introduce small, controlled stresses to test your systems. This is the concept behind "Chaos Engineering" (e.g., Netflix's Chaos Monkey, which randomly shuts down servers in production to ensure the system can handle it). Run fire drills. Stress-test your financials. 35 | 36 | #### **3. Actionable Strategies to Build Antifragility** 37 | 38 | * **Embrace (and Design for) Failure**: Create a culture where small, fast, and cheap failures are celebrated for the learning they provide. Conduct blameless post-mortems. 39 | * **Modular & Loose Coupling**: Design your organization and technology in discrete, independent modules (e.g., small autonomous teams, microservices architecture). This ensures a failure in one part doesn't cascade and bring down the whole system. 40 | * **Avoid Narrative Fallacy & Focus on Data**: Don't fall in love with your own forecasts and stories. Base decisions on empirical data and real-world results from your experiments. 41 | * **Hire for Learning Agility, Not Just Skills**: Prioritize candidates who are curious, adaptable, and can learn quickly over those with a specific but rigid skillset. 42 | * **Focus on Cash Flow & Strong Balance Sheets**: Financial fragility is a primary killer. Having a strong cash position gives you the optionality to pivot and take advantage of opportunities during a crisis (when others are struggling). 43 | 44 | #### **4. What to Avoid (Fragilizers)** 45 | 46 | * **Predicting the Future**: Relying on detailed 5-year plans. Instead, build a system that can react to any future. 47 | * **Debt & Excessive Leverage**: Debt forces you to be predictably right and removes your optionality. It is a major source of fragility. 48 | * **Centralized Decision-Making**: Creates bottlenecks and single points of failure. 49 | * **Efficiency Over Everything**: Maximizing efficiency (e.g., Just-In-Time inventory) removes slack, making the system brittle when shocks occur. 50 | * **Suppressing Volatility**: Trying to smooth out all minor fluctuations (e.g., avoiding all small customer complaints) can lead to a massive, catastrophic blow-up later. 51 | 52 | --- 53 | 54 | ### **External Materials for Deeper Learning** 55 | 56 | #### **1. Foundational Books (Must-Read)** 57 | 58 | * **`Antifragile: Things That Gain from Disorder` by Nassim Nicholas Taleb** 59 | * **Why:** The primary source. It's a dense, philosophical, and brilliant read that introduces the entire concept. Be prepared for Taleb's abrasive and opinionated style—focus on the core ideas. 60 | 61 | * **`The Black Swan: The Impact of the Highly Improbable` by Nassim Nicholas Taleb** 62 | * **Why:** The prequel to *Antifragile*. It explains why we are so bad at predicting rare, high-impact events and why building systems to withstand them is crucial. 63 | 64 | * **`Adapt: Why Success Always Starts with Failure` by Tim Harford** 65 | * **Why:** A more accessible and practical application of similar ideas. Harford uses excellent case studies to show how experimenting and adapting to failure is key to success in a complex world. 66 | 67 | #### **2. Complementary Books & Concepts** 68 | 69 | * **`Thinking, Fast and Slow` by Daniel Kahneman** 70 | * **Why:** Understanding cognitive biases (like narrative fallacy) is key to understanding why we build fragile systems and make fragile decisions in the first place. 71 | 72 | * **`The Lean Startup` by Eric Ries** 73 | * **Why:** This is essentially an antifragile operating manual for new ventures. The Build-Measure-Learn loop is a process for gaining from volatility and uncertainty through rapid experimentation. 74 | 75 | * **`Team of Teams: New Rules of Engagement for a Complex World` by Gen. Stanley McChrystal** 76 | * **Why:** A masterclass in decentralizing command and control to create a agile, adaptive organization—a key tenet of antifragility. 77 | 78 | #### **3. Articles, Essays & Online Resources** 79 | 80 | * **Farnam Street Blog (fs.blog)**: Search for "Antifragile," "Taleb," "Optionality." This blog is dedicated to mental models and has excellent summaries and applications of these concepts. 81 | * **`Chaos Engineering`** (Principles and resources from Netflix, Amazon, etc.) 82 | * **Why:** This is the purest technical implementation of antifragility. Reading the principles on **principlesofchaos.org** shows you how to deliberately stress a system to make it stronger. 83 | * **`The Barbell Strategy for Career and Business` (Various essays online)**: Look for articles that apply the barbell model to investing, product portfolios, and personal development. 84 | 85 | #### **4. Video & Audio Content** 86 | 87 | * **Nassim Taleb on YouTube**: Search for his interviews and lectures. Hearing him explain the concepts can bring clarity. 88 | * **The Knowledge Project Podcast (Shane Parrish)**: Episodes with guests like Taleb, Daniel Kahneman, and Adam Grant often delve into related themes of decision-making, risk, and complexity. -------------------------------------------------------------------------------- /dump/13.Write-Caching.md: -------------------------------------------------------------------------------- 1 | ### Core Concept: Why Caching Writes? 2 | A cache is a temporary, high-speed data storage layer. The primary challenge is maintaining consistency between the cache and the underlying primary database (or "backing store"). Write strategies define *how* and *when* data is written from the cache to this primary database, each offering a different trade-off between performance, consistency, and durability. 3 | 4 | --- 5 | 6 | ### 1. Write-Through Cache 7 | 8 | **How it works:** The application writes data to the cache. The cache then immediately and synchronously writes the same data to the underlying database. Only after both writes are confirmed is the write operation considered successful. 9 | 10 | * **Sequence:** `Application -> Cache -> Database (synchronously) -> Confirm to Application` 11 | * **Analogy:** Buying something and immediately updating your budget spreadsheet. The action isn't "done" until both the physical item is in your hand and the spreadsheet is updated. 12 | 13 | **Pros:** 14 | * **Strong Consistency:** The cache and database are always in sync. This is the biggest advantage. 15 | * **Data Durability:** Data is persisted to the database immediately, reducing the risk of data loss on a cache failure. 16 | * **Read Resilience:** If the cache fails, the database has the latest data, so subsequent reads (even if slower) will be correct. 17 | 18 | **Cons:** 19 | * **Higher Write Latency:** Every write operation incurs the penalty of a database write, which is often the slowest part of the system. This can become a bottleneck. 20 | * **Database Load:** The database handles every write operation, which might be unnecessary for transient data. 21 | 22 | **Best for:** 23 | * Applications where data consistency is critical (e.g., financial transactions, user account balances). 24 | * Read-heavy workloads where the same data is written once and read frequently. 25 | 26 | --- 27 | 28 | ### 2. Write-Behind (Write-Back) Cache 29 | 30 | **How it works:** The application writes data only to the cache. The write is confirmed to the application immediately. The cache then asynchronously batches these writes and flushes them to the database after a delay or under specific conditions (e.g., every 10 seconds, when the cache is full). 31 | 32 | * **Sequence:** `Application -> Cache (confirm immediately) -> ...later... Cache -> Database (asynchronously)` 33 | * **Analogy:** Taking notes on a sticky note during a meeting. You jot things down quickly to keep up and then, after the meeting, you transfer the important points neatly into your official notebook. 34 | 35 | **Pros:** 36 | * **Very Low Write Latency & High Throughput:** The application is not waiting for a slow database write. This is the primary benefit. 37 | * **Reduced Database Load:** Writes are batched and combined, significantly reducing the number of I/O operations on the database. 38 | 39 | **Cons:** 40 | * **Weak Consistency:** There is a lag between the cache write and the database persistence ("eventual consistency"). 41 | * **Risk of Data Loss:** If the cache fails (e.g., power loss) before the data is flushed to the database, the recent writes are lost forever. 42 | * **Complexity:** Requires more sophisticated logic to track and batch dirty data. 43 | 44 | **Best for:** 45 | * Write-heavy workloads with transient or non-critical data (e.g., user activity logging, clickstream analytics, session data). 46 | * Situations where performance and write throughput are more important than immediate durability (e.g., a product's "like" counter). 47 | 48 | --- 49 | 50 | ### 3. Write-Around Cache 51 | 52 | **How it works:** Writes go directly to the database, *bypassing the cache entirely*. On a subsequent read request, if the data is needed, it will be loaded from the database into the cache ("cache miss") and then returned. 53 | 54 | * **Sequence (Write):** `Application -> Database -> Confirm` 55 | * **Sequence (Read):** `Application -> Cache (MISS) -> Database -> Load into Cache -> Return data` 56 | * **Analogy:** Filing a document directly into a large filing cabinet (database). You only pull it out and put it on your desk (cache) when you actually need to work with it. 57 | 58 | **Pros:** 59 | * **Prevents Cache Pollution:** Avoids filling the cache with data that is written once and never read again, which is common in many systems. 60 | * **Fresh Data on Read:** Ensures that subsequent reads will always get the latest data from the database on a cache miss. 61 | 62 | **Cons:** 63 | * **Read Penalty for New Data:** The first read after a write will always be a cache miss and suffer the latency of a database read. 64 | 65 | **Best for:** 66 | * Workloads with data that is written frequently but read infrequently (e.g., archival data, bulk uploads). 67 | 68 | --- 69 | 70 | ### 4. Write-Through vs. Write-Behind: A Quick Comparison 71 | 72 | | Feature | Write-Through | Write-Behind | 73 | | :--- | :--- | :--- | 74 | | **Consistency** | **Strong.** Cache & DB are always in sync. | **Eventual.** Cache & DB are temporarily out of sync. | 75 | | **Performance** | Higher write latency. | **Very low write latency, high throughput.** | 76 | | **Durability** | **High.** Data is safe in DB immediately. | **Lower.** Risk of data loss on cache failure. | 77 | | **Database Load** | High (every write hits DB). | **Low (writes are batched).** | 78 | | **Use Case** | Critical data, read-heavy. | Non-critical, write-heavy data. | 79 | 80 | --- 81 | 82 | ### 5. Cache-Aside (Lazy Loading) - The Most Common Pattern 83 | 84 | While not strictly a *write strategy*, Cache-Aside is the overarching pattern that often employs the above techniques for writes. It's client-side logic. 85 | 86 | * **Read:** The application first checks the cache. On a hit, it uses the data. On a miss, it loads the data from the database, stores it in the cache, and then returns it. 87 | * **Write:** The application updates the database directly and then **invalidates** the corresponding cache entry. This is a form of write-around for the cache. 88 | 89 | **Why Invalidate?** Instead of updating the cache on write (which is more complex), simply deleting the stale data is simpler. The next read will lazy-load the fresh data from the database. 90 | 91 | --- 92 | 93 | ### Summary Table of Techniques 94 | 95 | | Technique | Write Path | Read Path (on miss) | Key Characteristic | 96 | | :--- | :--- | :--- | :--- | 97 | | **Write-Through** | Cache -> DB (sync) | DB -> Cache | Strong consistency, higher latency | 98 | | **Write-Behind** | Cache -> (later) DB (async) | DB -> Cache | High performance, risk of data loss | 99 | | **Write-Around** | DB (bypass cache) | DB -> Cache | Prevents cache pollution | 100 | | **Cache-Aside** | DB -> Invalidate Cache | DB -> Cache | Most common; client-managed | 101 | 102 | --- 103 | 104 | ### External Materials for In-Depth Learning 105 | 106 | #### Articles & Blogs 107 | 1. **AWS Database Caching Strategies**: A fantastic, practical guide from AWS that discusses these patterns in the context of their services. 108 | * **Link:** [https://aws.amazon.com/caching/database-caching/](https://aws.amazon.com/caching/database-caching/) 109 | 2. **Martin Fowler's Caching Patterns**: A more academic and pattern-oriented take from a renowned software thought leader. 110 | * **Link:** [https://martinfowler.com/bliki/CachingPattern.html](https://martinfowler.com/bliki/CachingPattern.html) 111 | 3. **Redis Caching Patterns**: Redis, the most popular caching database, has excellent documentation on how to implement these patterns with their technology. 112 | * **Link:** [https://redis.io/docs/develop/connect/caching/patterns/](https://redis.io/docs/develop/connect/caching/patterns/) (Look for "Write-through", "Write-behind", etc.) 113 | 114 | #### Videos 115 | 1. **System Design Interview Basics: Caching Concepts**: A great visual explanation of these patterns, perfect for auditory learners. 116 | * **Channel:** Tech Dummies Narendra L 117 | * **Likely Link:** [https://www.youtube.com/watch?v=mhUQe4BKZXs](https://www.youtube.com/watch?v=mhUQe4BKZXs) (Search for the title if the link changes) 118 | 119 | #### Books 120 | 1. **"Designing Data-Intensive Applications" by Martin Kleppmann**: This is the bible for understanding data systems. Chapter 3 "Storage and Retrieval" and parts of Chapter 5 "Replication" provide the foundational knowledge that makes caching strategies make sense. It's a must-read for any serious engineer. 121 | 2. **"Site Reliability Engineering" (Google SRE Book)**: While not exclusively about caching, the chapters on latency and efficient data storage are invaluable for understanding the *why* behind these techniques at scale. The "Distributed Caching" chapter in the sequel ("The Workbook") is also excellent. -------------------------------------------------------------------------------- /dump/05.Consistent-Hashing.md: -------------------------------------------------------------------------------- 1 | ### **Consistent Hashing** 2 | 3 | #### **1. The Problem: Naive Hashing & Rehashing** 4 | 5 | * **Traditional Approach:** In a distributed system (like a cache cluster with nodes A, B, C), a common way to assign a key (e.g., `user_123_profile`) to a node is to use the modulo operator: `hash(key) % number_of_nodes`. 6 | * **The Rehashing Catastrophe:** This works fine until a node is added or removed (e.g., node D joins or node B fails). The `number_of_nodes` changes, meaning `hash(key) % N` becomes `hash(key) % (N-1)` or `hash(key) % (N+1)`. This change causes **almost every key to be remapped** to a different node. 7 | * **Impact:** This massive remapping (rehashing) is disastrous for caches (cause of cache misses and a stampede on the database) and stateful systems (data becomes assigned to the wrong node). 8 | 9 | #### **2. The Solution: Consistent Hashing - Core Idea** 10 | 11 | Consistent Hashing is a special kind of hashing that minimizes the number of keys that need to be remapped when a hash table (or a node in a distributed system) is resized. 12 | 13 | The core idea is to **map both keys and nodes onto the same abstract circle (a hash ring)**. 14 | 15 | #### **3. How It Works: The Hash Ring** 16 | 17 | 1. **Imagine a Circle:** Picture a circle that represents the entire possible output range of a hash function (e.g., from 0 to 2^128 - 1). This circle is called the **hash ring**. 18 | 2. **Map Nodes onto the Ring:** 19 | * Each node (e.g., its IP address `A`, `B`, `C`) is hashed to determine its position on the ring. 20 | * This results in points for `A`, `B`, and `C` placed at random locations around the circle. 21 | 3. **Map Keys onto the Ring:** 22 | * Each data key (e.g., `user_123_profile`) is also hashed and placed on the same ring. 23 | 4. **Locate the Responsible Node:** 24 | * To find which node a key belongs to, start at the key's position on the ring and **walk clockwise** until you encounter the first node. 25 | * That node is responsible for that key. 26 | 27 | #### **4. Key Advantage: Minimal Reassignment on Node Changes** 28 | 29 | * **Adding a Node (e.g., `D`):** When a new node `D` is added, it gets placed at a random point on the ring. Only the keys that fall between `D` and the previous node (counter-clockwise) are moved from the next node (clockwise) to `D`. **The vast majority of keys are unaffected.** 30 | * **Removing a Node (e.g., `B` fails):** When node `B` is removed, only the keys that were assigned to `B` need to be reassigned. They get reassigned to the next node clockwise from `B`'s old position (in this case, `C`). **All other keys remain on their original nodes.** 31 | 32 | This property is called **consistency**, hence the name. 33 | 34 | #### **5. The Real-World Challenge: Non-Uniform Distribution & Virtual Nodes** 35 | 36 | The basic approach has a flaw: 37 | * **Imbalanced Load:** If nodes are assigned to only a few points on the ring, they can end up responsible for very different-sized arcs of the circle. One node might get a huge chunk of keys, while another gets very few. 38 | * **Solution: Virtual Nodes (Vnodes):** 39 | * Instead of mapping one physical node to one point on the ring, we map each physical node to **multiple points** on the ring. 40 | * For example, instead of `A` being one point, it becomes 1000 "virtual nodes" (`A-1`, `A-2`, ..., `A-1000`) scattered across the ring. The same is done for `B` and `C`. 41 | * **Benefits:** 42 | 1. **Load Balancing:** A physical node that is more powerful can be assigned more virtual nodes, giving it a larger share of the keys. 43 | 2. **Smoothness:** When a node is added/removed, its load (number of keys) is transferred evenly to/from many other nodes, not just one. This prevents hotspots. 44 | * Virtually all production systems (Dynamo, Cassandra, etc.) use virtual nodes. 45 | 46 | #### **6. Properties of Consistent Hashing** 47 | 48 | * **Minimal Reorganization:** Changes in the set of nodes cause minimal movement of keys. This is its primary goal. 49 | * **Load Balancing:** With virtual nodes, keys are distributed approximately evenly among physical nodes. 50 | * **Scalability:** The system can easily grow or shrink by adding/removing nodes with minimal impact. 51 | * **Decentralization:** No central coordinator is needed to decide key placement; any node can determine the correct location for a key using the same algorithm. 52 | 53 | #### **7. Common Use Cases** 54 | 55 | * **Distributed Caching Systems:** Memcached, Redis Cluster. 56 | * **Distributed Data Stores:** Amazon DynamoDB, Apache Cassandra, Riak. 57 | * **Load Balancers:** To direct client requests to a specific application server in a stateful manner (sticky sessions). 58 | * **Content Delivery Networks (CDNs):** To route requests to the nearest edge server. 59 | 60 | --- 61 | 62 | ### **Analogy: The Dinner Table** 63 | 64 | Imagine a circular dinner table (the hash ring) with 12 seats (hash positions). Dishes of food are the keys. 65 | 66 | * **Initial Setup:** Three chefs (nodes) sit at seats 12, 4, and 8. Each chef is responsible for all dishes placed on the table clockwise from their position until the next chef. 67 | * **Serving a Dish:** You want to place a pizza (a key). You calculate its "hash" – it belongs at seat 2. You walk clockwise and find the first chef at seat 4. Chef at seat 4 gets the pizza. 68 | * **A Chef Leaves:** The chef at seat 4 leaves. Only the dishes between seat 12 and seat 4 (which belonged to the leaving chef) need a new home. They are now assigned to the next chef clockwise, the one at seat 8. The dishes assigned to chefs at 8 and 12 remain untouched. 69 | * **A Chef Joins:** A new chef sits at seat 10. The dishes between seat 8 and seat 10, which previously belonged to the chef at seat 12, are now assigned to the new chef at seat 10. The rest are unchanged. 70 | 71 | --- 72 | 73 | ### **External Materials for In-Depth Learning** 74 | 75 | #### **1. Foundational Paper (A Must-Read)** 76 | * **Title:** "Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web" 77 | * **Authors:** David Karger, Eric Lehman, Tom Leighton, et al. (1997) 78 | * **Why read it?** This is the original academic paper that introduced the concept. It provides the rigorous mathematical foundation and the original motivations. 79 | * **Link:** [ACM Digital Library](https://dl.acm.org/doi/10.1145/258533.258660) (may be behind a paywall). A free PDF is often findable via a search for the title. 80 | 81 | #### **2. Classic Blog Posts (Excellent Explanations)** 82 | * **Title:** "Consistent Hashing" by Tom White 83 | * **Source:** Originally on the "JavaWorld" blog, now famously archived on his personal site. 84 | * **Why read it?** This is perhaps the most widely cited blog post on the topic. It provides a incredibly clear, step-by-step explanation with helpful diagrams. It's a canonical reference. 85 | * **Link:** [https://www.tom-e-white.com/2007/11/consistent-hashing.html](https://www.tom-e-white.com/2007/11/consistent-hashing.html) 86 | 87 | * **Title:** "Consistent Hashing: Algorithmic Tradeoffs" 88 | * **Source:** by Damien Gryski on his blog. 89 | * **Why read it?** This post goes beyond the basics and dives into different implementations, trade-offs, and optimizations (like bounded-load consistent hashing). It's great for understanding the practical nuances. 90 | * **Link:** [https://dgryski.medium.com/consistent-hashing-algorithmic-tradeoffs-ef6b8e2fcae8](https://dgryski.medium.com/consistent-hashing-algorithmic-tradeoffs-ef6b8e2fcae8) 91 | 92 | #### **3. Video Explanations (Visual Learning)** 93 | * **Channel:** System Design Interview 94 | * **Title:** "What is Consistent Hashing and Where is it used?" 95 | * **Why watch it?** This video uses clean animations to visually walk through the problem and the solution, making the concept very intuitive. 96 | * **Link:** [https://www.youtube.com/watch?v=UF9Iqmg94tk](https://www.youtube.com/watch?v=UF9Iqmg94tk) 97 | 98 | #### **4. Implementations & Code (For Hands-On Learning)** 99 | * **Language:** Go 100 | * **Package:** `stathat/c` 101 | * **Link:** [https://pkg.go.dev/github.com/stathat/c](https://pkg.go.dev/github.com/stathat/c) - A widely used, simple implementation. 102 | * **Language:** Java 103 | * **Library:** Apache Cassandra's implementation. You can browse the source code to see how a real-world database does it. 104 | * **Language:** Python 105 | * **Search for:** `python consistent-hash-ring` on PyPI. Several simple implementations exist. 106 | 107 | Reading the original paper and Tom White's blog post will give you a rock-solid understanding of both the theory and the common practical implementation. -------------------------------------------------------------------------------- /dump/11.Failover.md: -------------------------------------------------------------------------------- 1 | ### **Failover** 2 | 3 | #### **1. Core Concept** 4 | 5 | **Failover** is an automatic process in a high-availability (HA) system where a standby component (server, network, storage, service) takes over the operations of a failed primary component. The primary goal is to minimize downtime and maintain service continuity without manual intervention. 6 | 7 | * **Analogy:** A spare tire in a car. When the primary tire fails (goes flat), you switch to the spare to continue your journey, albeit potentially with some limitations (e.g., speed, distance). 8 | 9 | #### **2. Key Objectives** 10 | 11 | * **High Availability (HA):** To ensure applications and services are operational and accessible with minimal disruption, often measured as a percentage of uptime (e.g., 99.999% or "five nines"). 12 | * **Disaster Recovery (DR):** To recover systems and data after a catastrophic event, often involving a secondary geographic location. Failover is a key action within a DR plan. 13 | * **Business Continuity:** To maintain critical business functions during and after a failure. 14 | 15 | #### **3. Key Components & Terminology** 16 | 17 | * **Primary/Active Node:** The system currently handling the production workload. 18 | * **Secondary/Standby/Passive Node:** The system that remains on standby, ready to take over if the primary fails. 19 | * **Hot Standby:** Fully powered on, synchronized, and ready to take over immediately (fastest failover). 20 | * **Warm Standby:** Powered on and partially synchronized, may take a short time to become current. 21 | * **Cold Standby:** Powered off and requires manual intervention, configuration, and data restoration (slowest failover). 22 | * **Failback:** The process of returning operations to the original (now repaired) primary system after a failover has occurred. This can be automated or manual. 23 | * **Heartbeat (or Health Check):** A continuous signal sent between nodes to indicate they are alive and healthy. 24 | * **Quorum:** A voting mechanism to prevent "split-brain" scenarios (see below). It ensures only one cluster has a majority of votes and can become active. 25 | * **Virtual IP (VIP):** A floating IP address that is assigned to the active node. Clients connect to this VIP, which automatically moves to the standby node during a failover, making the transition transparent to the end-user. 26 | 27 | #### **4. Types of Failover** 28 | 29 | | Type | Description | Use Case | RTO* / RPO** | 30 | | :--- | :--- | :--- | :--- | 31 | | **Automatic Failover** | System detects failure and switches without human intervention. | Mission-critical applications requiring maximum uptime. | Very Low RTO | 32 | | **Manual Failover** | An administrator triggers the switchover. | Planned maintenance, testing, or less critical systems. | Higher RTO | 33 | | **Instant Failover** | Near-zero downtime; state is continuously synchronized. | Financial trading systems, real-time databases. | Near-Zero RTO/RPO | 34 | | **Graceful Failover** | The primary node gracefully hands off connections and state to the standby before shutting down. | Planned maintenance, upgrades. | Low RTO | 35 | | **Disaster Recovery Failover** | Failover to a geographically distant site (geo-redundancy). | Recovery from natural disasters, regional outages. | Varies (minutes to hours) | 36 | 37 | * **RTO (Recovery Time Objective):** The maximum acceptable time an application can be down. 38 | * **RPO (Recovery Point Objective):** The maximum acceptable amount of data loss, measured in time. 39 | 40 | #### **5. Common Challenges & Pitfalls** 41 | 42 | * **Split-Brain:** A catastrophic scenario where both the primary and secondary nodes believe they are active, often due to a faulty heartbeat connection. This can lead to data corruption. **Mitigation:** Use a quorum, reliable heartbeats, and fencing. 43 | * **Fencing (or STONITH - Shoot The Other Node In The Head):** The process of isolating a faulty node to prevent it from causing damage (like split-brain). This could mean cutting its power or network access. 44 | * **Data Synchronization Lag (Replication Lag):** The standby node might not have the very latest data from the primary at the moment of failure, leading to data loss (affecting RPO). 45 | * **Testing:** Failover systems are complex and must be tested regularly. A failover system that has never been tested is **guaranteed to fail** when needed. 46 | * **Performance Impact:** The standby node may have lower specifications, leading to degraded performance after a failover. 47 | * **DNS/Network Propagation:** Clients might cache the old server's IP address, delaying the redirection to the new active node. 48 | 49 | #### **6. Best Practices** 50 | 51 | 1. **Design for Failure:** Assume components will fail; build redundancy at every layer (application, database, network, power). 52 | 2. **Automate Everything:** Prefer automatic failover for critical systems to avoid human delay and error. 53 | 3. **Monitor and Test Relentlessly:** Continuously monitor heartbeats and replication status. Conduct regular, scheduled failover drills. 54 | 4. **Implement Proper Fencing:** Always have a reliable fencing mechanism to avoid split-brain. 55 | 5. **Clear Documentation:** Have a runbook that details the failover process, roles, and responsibilities for both automated and manual scenarios. 56 | 6. **Understand Your RTO and RPO:** Your failover architecture should be designed to meet these business-defined objectives. 57 | 58 | --- 59 | 60 | ### **External Materials for In-Depth Learning** 61 | 62 | Here are categorized resources to deepen your understanding. 63 | 64 | #### **A. General Concepts & Theory** 65 | 66 | 1. **Wikipedia - Failover:** 67 | * **Link:** [https://en.wikipedia.org/wiki/Failover](https://en.wikipedia.org/wiki/Failover) 68 | * **Why:** A great starting point for a general overview, terminology, and basic concepts. 69 | 70 | 2. **VMware Availability Zone - Failover Fundamentals:** 71 | * **Link:** [https://core.vmware.com/availability-fundamentals](https://core.vmware.com/availability-fundamentals) 72 | * **Why:** VMware provides excellent, vendor-agnostic explanations of high availability, failover, and related concepts in a clear, structured way. 73 | 74 | #### **B. Specific Technology Implementations** 75 | 76 | 3. **Amazon Web Services (AWS) - Disaster Recovery:** 77 | * **Link:** [https://aws.amazon.com/blogs/aws/new-aws-failover-to-azure-with-azure-arc-protection/](https://aws.amazon.com/blogs/aws/new-aws-failover-to-azure-with-azure-arc-protection/) (and the broader AWS DR whitepapers) 78 | * **Why:** The industry standard for cloud-based failover and disaster recovery strategies. Learn about pilot light, warm standby, and multi-site active/active setups. 79 | 80 | 4. **Microsoft SQL Server - Always On Availability Groups:** 81 | * **Link:** [https://docs.microsoft.com/en-us/sql/database-engine/availability-groups/windows/overview-of-always-on-availability-groups](https://docs.microsoft.com/en-us/sql/database-engine/availability-groups/windows/overview-of-always-on-availability-groups) 82 | * **Why:** A deep dive into a mature and robust database-level failover technology, covering synchronization modes, read-only replicas, and failover mechanics. 83 | 84 | 5. **Pacemaker/Corosync Cluster Stack:** 85 | * **Link:** [https://clusterlabs.org/](https://clusterlabs.org/) 86 | * **Why:** The open-source standard for building high-availability clusters on Linux. Essential for understanding how heartbeats, quorum, and resource management work under the hood. 87 | 88 | 6. **Kubernetes - Pod Disruption Budgets & Liveness Probes:** 89 | * **Link:** [https://kubernetes.io/docs/tasks/run-application/configure-pdb/](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) 90 | * **Why:** In a modern containerized world, failover is often managed by the orchestrator. Kubernetes handles node/pod failures automatically, and this explains how to control that process. 91 | 92 | #### **C. In-Depth Articles & Blogs** 93 | 94 | 7. **Shopify Engineering - "Deconstructing the Database: The Concept of Database Failover":** 95 | * **Link:** (Search for this title, as URLs change) 96 | * **Why:** A fantastic real-world case study that explains the complexities, challenges (like split-brain), and solutions Shopify implemented for their massive database failover needs. 97 | 98 | 8. **NetApp Blog - "The 4 Types of Disaster Recovery Failover":** 99 | * **Link:** (Search for this title) 100 | * **Why:** A clear and concise breakdown of different DR failover strategies with excellent visuals and practical advice. 101 | 102 | #### **D. Books** 103 | 104 | 9. **"Designing Data-Intensive Applications" by Martin Kleppmann:** 105 | * **Why:** While not exclusively about failover, this book is the definitive guide to building robust systems. Chapter 5 (Replication) and Chapter 8 (The Trouble with Distributed Systems) are **mandatory reading** for anyone who wants to truly understand the challenges and solutions behind replication and failover. -------------------------------------------------------------------------------- /dump/16.Auth-and-ways.md: -------------------------------------------------------------------------------- 1 | ### 1. Core Concept: Authentication vs. Authorization 2 | 3 | The difference is fundamental to security and is often summarized as: 4 | 5 | * **Authentication (AuthN):** **Who are you?** The process of verifying the identity of a user or system. It's about proving you are who you say you are. 6 | * **Analogy:** Showing your passport or driver's license at the airport check-in counter. The agent is verifying that you are the person whose name is on the ticket. 7 | 8 | * **Authorization (AuthZ):** **What are you allowed to do?** The process of determining what permissions, access rights, or resources an authenticated user has. 9 | * **Analogy:** Your boarding pass after check-in. It specifies *what* you are allowed to do: which plane you can board (the resource), and whether you can access the first-class cabin (a privilege). 10 | 11 | You **must always authenticate first** to establish an identity, and then **authorize** to determine what that identity can access. 12 | 13 | | Aspect | Authentication (AuthN) | Authorization (AuthZ) | 14 | | :--- | :--- | :--- | 15 | | **Purpose** | Verifies identity | Grants permissions | 16 | | **Question** | "Who are you?" | "What are you allowed to do?" | 17 | | **Process** | Credentials (password, biometrics) | Policies, Roles, Access Control Lists (ACLs) | 18 | | **Example** | Logging into your email | Reading emails vs. changing account settings | 19 | | **Data** | Manages Identity (e.g., User ID) | Manages Access (e.g., Roles, Permissions) | 20 | | **Order** | Comes first | Comes after successful authentication | 21 | 22 | --- 23 | 24 | ### 2. Common Ways to Authenticate (AuthN Methods) 25 | 26 | Authentication typically relies on one or more **factors**, which are categories of credentials. 27 | 28 | #### The Three Factors of Authentication 29 | 30 | 1. **Something You Know:** Knowledge factor (e.g., Password, PIN, Security Question). 31 | 2. **Something You Have:** Possession factor (e.g., Smartphone (for an app), Security Key (YubiKey), Bank Card, Hardware Token). 32 | 3. **Something You Are:** Inherence factor (e.g., Fingerprint, Facial recognition, Iris scan, Voice pattern). 33 | 34 | Using more than one factor is called **Multi-Factor Authentication (MFA)**, which is significantly more secure than single-factor (like just a password). 35 | 36 | #### Specific Authentication Methods & Protocols 37 | 38 | * **Passwords:** The most common single-factor method. Vulnerable to phishing, brute-force attacks, and poor user habits (reuse, simple passwords). 39 | 40 | * **Multi-Factor Authentication (MFA / 2FA):** 41 | * **Time-based One-Time Password (TOTP):** Apps like Google Authenticator or Authy generate a temporary code. Combines something you know (password) with something you have (your phone). 42 | * **SMS/Email Codes:** A code sent via SMS or email. **Note:** SMS-based 2FA is considered less secure due to SIM-swapping attacks but is better than nothing. 43 | * **Push Notifications:** A service like Duo sends a push notification to an app on your phone for approval. 44 | * **Security Keys:** Physical devices (e.g., YubiKey) that use protocols like FIDO2/WebAuthn for phishing-resistant authentication. 45 | 46 | * **Single Sign-On (SSO):** 47 | * Allows a user to authenticate with one set of credentials (e.g., their company login) to access multiple independent applications. 48 | * It **delegates** authentication to a trusted third party (called an Identity Provider or IdP). 49 | * **Protocols:** **SAML** (common in enterprise), **OAuth 2.0** (the foundation for most social logins and modern API access), and **OpenID Connect (OIDC)** (an identity layer built *on top of* OAuth 2.0 that provides authentication). 50 | 51 | * **Passwordless Authentication:** 52 | * Aims to replace the password entirely. Often uses a possession factor (a security key or authenticator app) or a magic link sent via email. 53 | * **WebAuthn** is a core W3C standard enabling passwordless logins using biometrics or security keys. 54 | 55 | * **Biometric Authentication:** 56 | * Uses the inherence factor (fingerprint, face ID). It's convenient but raises privacy concerns. The stored data is usually a mathematical representation (a hash) of your biometric data, not the raw data itself. 57 | 58 | * **Certificate-Based Authentication:** 59 | * Uses digital certificates (like SSL/TLS certificates for websites) to identify users or machines. Common in corporate and government environments. 60 | 61 | * **API Keys:** 62 | * A unique identifier used to authenticate a project or application calling an API. Not for human users, but for service-to-service communication. They are secrets that must be protected. 63 | 64 | --- 65 | 66 | ### 3. Common Ways to Authorize (AuthZ Models) 67 | 68 | Once a user is authenticated, these models control their access: 69 | 70 | * **Access Control Lists (ACLs):** A list of permissions attached to an object (e.g., a file). The list defines which users or system processes can access the object and what operations (read, write, execute) they can perform. 71 | * **Role-Based Access Control (RBAC):** Access rights are assigned to *roles* (e.g., "Admin," "Editor," "Viewer"), and users are assigned to these roles. This is very common and scalable for organizations. 72 | * **Attribute-Based Access Control (ABAC):** A more dynamic model. Access decisions are based on attributes (characteristics) of the user, resource, action, and environment. 73 | * *Example:* "A `User` with the attribute `department=HR` can `read` a `Document` with the attribute `classification=Internal` only if the `environment` attribute `current-time` is between `9 AM and 5 PM`." 74 | 75 | --- 76 | 77 | ### 4. External Materials for In-Depth Learning 78 | 79 | #### Articles & Guides (Beginner to Intermediate) 80 | 1. **Auth0 Blog - "Authentication vs. Authorization"** 81 | * **Link:** [https://auth0.com/docs/get-started/identity-fundamentals/authentication-and-authorization](https://auth0.com/docs/get-started/identity-fundamentals/authentication-and-authorization) 82 | * **Why:** Excellent, clear explanation from a leading identity management company. 83 | 84 | 2. **Cloudflare Learning Center - "What is multi-factor authentication (MFA)?"** 85 | * **Link:** [https://www.cloudflare.com/learning/access-management/what-is-multi-factor-authentication/](https://www.cloudflare.com/learning/access-management/what-is-multi-factor-authentication/) 86 | * **Why:** Great primer on MFA and why it's important. 87 | 88 | 3. **Okta Developer Resources - Identity 101** 89 | * **Link:** [https://developer.okta.com/books/api-security/authn-authz/](https://developer.okta.com/books/api-security/authn-authz/) (Their entire developer site is a fantastic resource) 90 | * **Why:** Okta is another identity giant. Their content is practical and well-structured for developers. 91 | 92 | #### Protocols Deep Dives (Intermediate to Advanced) 93 | 4. **OAuth 2.0 and OpenID Connect (OIDC) - "OAuth 2 Simplified" by Aaron Parecki** 94 | * **Link:** [https://aaronparecki.com/oauth-2-simplified/](https://aaronparecki.com/oauth-2-simplified/) 95 | * **Why:** This is arguably the best and most famous explanation of OAuth 2.0. It breaks down the complex flows into understandable concepts. 96 | 97 | 5. **MDN Web Docs - HTTP Authentication** 98 | * **Link:** [https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication) 99 | * **Why:** The definitive resource for understanding how authentication works at the HTTP protocol level (Basic Auth, Bearer Tokens, etc.). 100 | 101 | 6. **WebAuthn Guide - by Duo** 102 | * **Link:** [https://duo.com/blog/the-webauthn-api-and-the-future-of-passwordless-authentication](https://duo.com/blog/the-webauthn-api-and-the-future-of-passwordless-authentication) 103 | * **Why:** A good starting point for understanding the passwordless future. 104 | 105 | #### Video Explanations 106 | 7. **F5 DevCentral - "SAML vs. OAuth vs. OIDC"** 107 | * **Link:** [https://www.youtube.com/watch?v=0p5rU_AE4zA](https://www.youtube.com/watch?v=0p5rU_AE4zA) 108 | * **Why:** A clear and concise whiteboard video explaining the differences between these key protocols. 109 | 110 | 8. **Computerphile - "OAuth and it's Alternatives"** 111 | * **Link:** [https://www.youtube.com/watch?v=ktiJ_ofO5xQ](https://www.youtube.com/watch?v=ktiJ_ofO5xQ) 112 | * **Why:** Computerphile is excellent at breaking down complex technical topics into digestible videos. 113 | 114 | #### Books (Structured Learning) 115 | 9. **"API Security in Action" by Neil Madden** 116 | * **Why:** This book provides fantastic, practical coverage of not just OAuth 2.0 and OpenID Connect, but also JWTs, API keys, and other critical security concepts for modern developers. 117 | 118 | 10. **"OAuth 2 in Action" by Justin Richer and Antonio Sanso** 119 | * **Why:** The definitive deep dive into the OAuth 2.0 protocol. It's technical, detailed, and perfect for implementers. -------------------------------------------------------------------------------- /dump/10.Replication-Lag.md: -------------------------------------------------------------------------------- 1 | ### **Replication Lag: A Problem in Eventual Consistency** 2 | 3 | #### **1. Core Concepts: The "What" and "Why"** 4 | 5 | * **Eventual Consistency:** A consistency model used in distributed systems where, if no new updates are made to a given data item, eventually all accesses to that item will return the last updated value. It's a trade-off that sacrifices strong consistency (immediate uniformity) for higher availability and lower latency. 6 | * **Asynchronous Replication:** A method where the primary node (or leader) acknowledges a write operation to the client *before* it has been fully replicated and applied to all secondary (replica) nodes. This is the mechanism that makes systems highly available and performant but introduces the possibility of lag. 7 | * **Replication Lag:** The delay between the time a write operation is committed on the primary node and the time it is applied to a given replica node. This lag can be milliseconds, seconds, or even minutes under heavy load or network issues. 8 | 9 | **Why is it used?** The CAP theorem tells us that a distributed network cannot simultaneously provide Consistency, Availability, and Partition Tolerance. In many modern, large-scale applications (e.g., social media, e-commerce), designers choose **Availability and Partition Tolerance over strong Consistency (AP)**. Asynchronous replication is the technical choice that enables this AP design. 10 | 11 | #### **2. The Problem: Symptoms and Consequences (The "Why It's Bad")** 12 | 13 | Replication lag itself is not a bug; it's an inherent characteristic of the system design. The **problem** arises when application logic or user experience implicitly assumes strong consistency but the underlying system only provides eventual consistency. 14 | 15 | This mismatch leads to several well-documented anomalies: 16 | 17 | 1. **Stale Reads (The Most Common Issue):** 18 | * **Description:** A client reads data from a lagging replica and receives an old value, even though a newer value has been written to the primary. 19 | * **Example:** You update your profile picture. You refresh the page immediately, but the old picture still shows because your read request was served by a lagging replica. 20 | 21 | 2. **Read-After-Write Inconsistency:** 22 | * **Description:** A user writes data and then immediately reads it, but the read returns stale data because it goes to a replica that hasn't received the write yet. 23 | * **Example:** You post a new comment on a blog post. The page reloads to show your comment, but it doesn't appear. This breaks user expectations and is incredibly frustrating. 24 | 25 | 3. **Monotonic Reads:** 26 | * **Description:** A user makes two reads in sequence. The second read returns data that is *older* than the first read. This can happen if the first read goes to a more up-to-date replica and the second read goes to a more lagged one. 27 | * **Example:** You see a new email notification (read #1 from an updated replica). You click into your inbox, but the email isn't there (read #2 from a lagging replica). 28 | 29 | 4. **Causal (or Time-Travel) Reads:** 30 | * **Description:** This breaks causality. A user sees the effect before seeing the cause. 31 | * **Example:** On a forum, you see a reply to a post (effect) that you haven't seen yet because the reply replica is ahead of the post replica (cause). 32 | 33 | #### **3. Mitigation Strategies: How to Cope (The "How to Fix It")** 34 | 35 | You cannot eliminate lag, but you can design your system and application to manage its effects. 36 | 37 | 1. **Application-Level Awareness:** 38 | * **Read-Your-Writes Consistency:** Ensure that a user always reads their own writes. This can be done by: 39 | * **Sticky Sessions:** Routing all reads for a specific user session to the primary node (or the same replica) for a period of time. 40 | * **Version Checks:** Having the client send the timestamp/version of its last write. If a replica's data is older, the read can be rerouted to a more current node. 41 | * **Monotonic Reads:** Implement logic to route all reads for a user session to the same replica, preventing time-travel backwards. 42 | 43 | 2. **Database/System-Level Solutions:** 44 | * **Synchronous Replication for Critical Operations:** For certain critical operations (e.g., finalizing a payment), use a synchronous write. The trade-off is higher latency for that specific operation. 45 | * **Leader-Based Reads:** Simply send all read requests to the primary node. This solves all consistency issues but completely negates the scalability benefits of having read replicas. It's often used for a small subset of operations. 46 | * **Wait-Based Mechanisms:** Some systems (like Amazon DynamoDB) offer a `ConsistentRead` option. Others (like PostgreSQL) allow you to query a replica and ask it to wait until it has replayed all transactions up to a specific Log Sequence Number (LSN) before executing your read. 47 | * **Monitoring and Alerting:** Actively monitor replica lag (e.g., `SHOW REPLICA STATUS` in MySQL, `pg_stat_replication` in PostgreSQL). Set up alerts if the lag exceeds a certain threshold that is acceptable for your application. 48 | 49 | 3. **Architectural Patterns:** 50 | * **Conflict-Free Replicated Data Types (CRDTs):** Use data structures designed to be merged correctly even if they receive updates in different orders. This avoids the "last write wins" conflict resolution that can cause data loss. Excellent for counters, sets, and registers. 51 | * **Operational Transformation (OT):** The algorithm behind collaborative editing tools like Google Docs. It resolves conflicts by transforming operations so they can be applied in any order while preserving intent. 52 | 53 | --- 54 | 55 | ### **External Materials for In-Depth Learning** 56 | 57 | Here is a curated list of resources, from foundational texts to practical articles. 58 | 59 | #### **Books (The Deep Dive)** 60 | 61 | 1. ***Designing Data-Intensive Applications*** **by Martin Kleppmann** 62 | * **Why:** This is the bible for this topic. **Chapter 5 (Replication)** and **Chapter 7 (Transactions)** provide the most lucid, comprehensive explanation of replication lag, consistency models, and the trade-offs involved. It is a must-read. 63 | * **Link:** [Official Book Website](https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/) 64 | 65 | 2. ***Database Internals*** **by Alex Petrov** 66 | * **Why:** If you want to understand the low-level mechanics of how replication (both synchronous and asynchronous) is implemented in storage engines and database systems, this book is exceptional. 67 | * **Link:** [Official Book Website](https://www.oreilly.com/library/view/database-internals/9781492040330/) 68 | 69 | #### **Online Articles & Papers (The Classics & Practical Guides)** 70 | 71 | 1. **Amazon DynamoDB: Consistency Models** 72 | * **Why:** A practical, real-world explanation from a major cloud provider. It clearly defines the difference between "strongly consistent" and "eventually consistent" reads in their system and when to use each. 73 | * **Link:** [AWS Documentation on Consistency](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html) 74 | 75 | 2. **Google Cloud Spanner: TrueTime & External Consistency** 76 | * **Why:** Spanner is a fascinating counterpoint. It's a globally distributed database that offers strong consistency. Understanding its use of atomic clocks and GPS (**TrueTime**) to minimize commit wait times shows the extreme engineering required to "beat" the CAP theorem. 77 | * **Link:** [Spanner: Google's Globally-Distributed Database (Research Paper)](https://research.google/pubs/pub39966/) 78 | 79 | 3. **The CAP FAQ** by Henry Robinson 80 | * **Why:** A fantastic, clear FAQ that dispels common myths about the CAP theorem, which is the fundamental theory behind the choice of eventual consistency. 81 | * **Link:** [https://www.the-paper-trail.org/page/cap-faq/](https://www.the-paper-trail.org/page/cap-faq/) 82 | 83 | 4. **Jepsen Distributed Systems Safety Research** 84 | * **Why:** Kyle Kingsbury's Jepsen analyses are famous for putting distributed databases (MongoDB, Redis, PostgreSQL, etc.) through rigorous tests to find consistency anomalies under network partitions and replication lag. Reading these reports shows the practical, real-world manifestations of these problems. 85 | * **Link:** [https://jepsen.io/analyses](https://jepsen.io/analyses) 86 | 87 | #### **Video Lectures (The Visual Learner's Path)** 88 | 89 | 1. **MIT 6.824: Distributed Systems (Course)** 90 | * **Why:** A legendary university course on distributed systems. The lectures on fault tolerance, replication (state machine replication), and the Raft consensus algorithm provide the deep theoretical background. 91 | * **Link:** [YouTube Playlist (2020 version)](https://www.youtube.com/playlist?list=PLrw6a1wE39_tb2fErI4-WkMbsvGQk9_UB) 92 | 93 | 2. **Martin Kleppmann: Transactions: Myths, Surprises, and Opportunities** 94 | * **Why:** A great talk by the author of DDIA that covers many of the same topics in an engaging video format. 95 | * **Link:** [YouTube Video](https://www.youtube.com/watch?v=5ZjhNTM8XU8) -------------------------------------------------------------------------------- /dump/06.Consistency-Models.md: -------------------------------------------------------------------------------- 1 | ### **Consistency Models** 2 | 3 | #### **1. Core Concept: What is a Consistency Model?** 4 | 5 | A **consistency model** is a formal contract between a distributed data-store system and its processes (clients/applications). It defines the rules about how and when a write operation (an update) by one process becomes visible to other processes reading the data. 6 | 7 | In simpler terms, it's the answer to the question: **"If I write a new value to a data item, what guarantees do I have about what value another client will see when they read that same item, and when will they see it?"** 8 | 9 | Stronger models provide simple, intuitive guarantees but are slower and less available. Weaker models are faster and more fault-tolerant but require the application to handle complex, stale, or out-of-order data. 10 | 11 | #### **2. The Fundamental Trade-off: The CAP Theorem** 12 | 13 | You cannot understand consistency models without the CAP theorem. It states that a distributed system can only provide two of the following three guarantees simultaneously: 14 | 15 | * **C**onsistency: Every read receives the most recent write or an error. (Equivalent to "Strong Consistency"). 16 | * **A**vailability: Every request receives a (non-error) response, without the guarantee that it contains the most recent write. 17 | * **P**artition Tolerance: The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes. 18 | 19 | In practice, network partitions (*P*) are a fact of life, so the real trade-off is between **Consistency (C)** and **Availability (A)**. Consistency models are the spectrum of choices we make on this C-A axis. 20 | 21 | #### **3. Spectrum of Consistency Models (Strong to Weak)** 22 | 23 | Here are the most common models, ordered from strongest (most strict) to weakest (most relaxed). 24 | 25 | ##### **A. Strong Consistency Models** 26 | 27 | These models provide a system that behaves like a single, central copy of the data. 28 | 29 | 1. **Linearizability (a.k.a. Atomic Consistency)** 30 | * **Guarantee:** The most intuitive model. Every operation appears to take effect atomically at some instant *between its start and end time*. All subsequent operations (after that instant) must see the effect of that write. 31 | * **Analogy:** A single, global ledger where every transaction is recorded instantly and in a strict, universally agreed-upon order. If you read after a write completes, you *must* see the value you wrote. 32 | 33 | 2. **Sequential Consistency** 34 | * **Guarantee:** Slightly weaker than Linearizability. All operations appear to take effect in a single, sequential order that is consistent with the program order of each individual process. However, the order in the global sequence doesn't have to correspond to real-time. 35 | * **Key Difference from Linearizability:** If Process A writes `x=1` and then `y=2`, and Process B sees `y=2`, it *must* later see `x=1`. But it might not see the writes immediately after they complete in real-time. 36 | 37 | ##### **B. Eventual Consistency Model** 38 | 39 | * **Guarantee:** If no new updates are made to a data item, eventually all reads to that item will return the last updated value. **It makes no guarantees about when this will happen.** 40 | * **Characteristics:** This is a very weak model. Reads might get old data for an unspecified period. It is highly available and partition-tolerant. 41 | * **Common Use Case:** DNS, social media follower counts, likes. 42 | 43 | ##### **C. Weak Consistency Models** 44 | 45 | These models provide explicit mechanisms for managing consistency, often requiring the programmer to trigger synchronization. 46 | 47 | 1. **Causal Consistency** 48 | * **Guarantee:** Preserves "happens-before" relationships. Writes that are causally related (e.g., a comment on a post) must be seen by all processes in the same order. Concurrent writes (writes that are not causally related) may be seen in different orders on different replicas. 49 | * **Analogy:** You will always see a reply to a email *after* seeing the original email. But you and a colleague might see two unrelated emails in different orders. 50 | 51 | 2. **Read-Your-Writes Consistency** 52 | * **Guarantee:** A process that updates a data item will always see its own update. It does not guarantee that other processes will see that update immediately. 53 | * **Use Case:** User profile updates. After you change your password, you must be able to log in with it immediately, even if others see the old one for a few more seconds. 54 | 55 | 3. **Session Consistency** 56 | * **Guarantee:** Provides Read-Your-Writes and Monotonic Reads consistency guarantees within a single session. If a user interacts with the system in a session, their view of the data will be self-consistent. 57 | * **Use Case:** Web shopping carts. Items you add to your cart during a session will remain there and be visible to you until you check out. 58 | 59 | #### **4. Mechanisms to Achieve Consistency** 60 | 61 | How do systems actually implement these models? 62 | 63 | * **Quorums:** A common technique for strong consistency. A write operation must be acknowledged by a majority (a *write quorum*, `W`) of replicas before it's considered successful. A read operation must query a majority (a *read quorum*, `R`) of replicas and return the value with the latest timestamp. If `W + R > N` (where `N` is the total number of replicas), the read and write quorums are guaranteed to overlap, ensuring the read gets the latest value. 64 | * **Conflict-Free Replicated Data Types (CRDTs):** Data structures designed for eventual consistency that can be updated concurrently and will always converge to the same state mathematically, without needing complex conflict resolution. (e.g., counters, sets, registers). 65 | * **Version Vectors & Vector Clocks:** Mechanisms to track the history of updates across different replicas to detect causal relationships and potential conflicts between concurrent writes. 66 | * **Paxos & Raft:** Consensus algorithms used to achieve strong consistency (linearizability) by ensuring a majority of nodes agree on every operation before it's committed. These are the foundation for systems like etcd and ZooKeeper. 67 | 68 | --- 69 | 70 | ### **External Materials for In-Depth Learning** 71 | 72 | Here is a curated list of materials, from foundational papers to practical articles. 73 | 74 | #### **1. Foundational Papers (The Classics)** 75 | 76 | * **"Linearizability: A Correctness Condition for Concurrent Objects"** by Herlihy and Wing (1990) 77 | * **Why:** The paper that formally defined Linearizability. It's the canonical source. 78 | * **Difficulty:** Academic. 79 | 80 | * **"Time, Clocks, and the Ordering of Events in a Distributed System"** by Leslie Lamport (1978) 81 | * **Why:** One of the most influential papers in distributed systems. Introduces logical clocks and the "happens-before" relationship, which is fundamental to understanding causal consistency. 82 | * **Difficulty:** Academic but essential reading. 83 | 84 | #### **2. Practical Articles & Blog Posts (Highly Recommended)** 85 | 86 | * **Jepsen.io "Consistency Models" Page:** 87 | * **Link:** [https://jepsen.io/consistency](https://jepsen.io/consistency) 88 | * **Why:** Kyle Kingsbury's work is the industry standard for analyzing distributed databases. This page provides brilliant, clear explanations of various models with helpful diagrams. Start here. 89 | 90 | * **AWS re:Post Article: "Introduction to Distributed System Concepts: Consistency and Consensus":** 91 | * **Link:** [https://repost.aws/articles/AR5TZMLrSfdF3jf4vqSA4hFw/introduction-to-distributed-system-concepts-consistency-and-consensus](https://repost.aws/articles/AR5TZMLrSfdF3jf4vqSA4hFw/introduction-to-distributed-system-concepts-consistency-and-consensus) 92 | * **Why:** A very well-written, practical overview that connects theory to real-world cloud systems. 93 | 94 | * **Martin Kleppmann's Blog: "Please stop calling databases CP or AP":** 95 | * **Link:** [https://martin.kleppmann.com/2015/05/11/please-stop-calling-databases-cp-or-ap.html](https://martin.kleppmann.com/2015/05/11/please-stop-calling-databases-cp-or-ap.html) 96 | * **Why:** A critical take on the oversimplification of the CAP theorem, which is crucial for developing a nuanced understanding. 97 | 98 | #### **3. Books** 99 | 100 | * **"Designing Data-Intensive Applications" by Martin Kleppmann (2017)** 101 | * **Why:** This is arguably the best single resource on the topic. **Chapter 9 ("Consistency and Consensus") is a masterpiece.** It covers linearizability, ordering guarantees, quorums, consensus algorithms (Paxos, Raft), and much more with incredible clarity. It is a must-read. 102 | * **Difficulty:** Accessible to software engineers; no heavy academic jargon. 103 | 104 | * **"Distributed Systems for Fun and Profit" by Mikito Takada** 105 | * **Link:** [http://book.mixu.net/distsys/](http://book.mixu.net/distsys/) 106 | * **Why:** A concise and excellent free online book. It covers the core ideas, including the CAP theorem and consistency models, very effectively. 107 | 108 | #### **4. Interactive Explorers** 109 | 110 | * **"Visualizing Consistency Models" by Chris Meiklejohn:** 111 | * **Link:** [http://christophermeiklejohn.com/consistency/2016/10/12/visual-consistency-models.html](http://christophermeiklejohn.com/consistency/2016/10/12/visual-consistency-models.html) 112 | * **Why:** Interactive diagrams that let you play with client operations to see how different models (e.g., Linearizable, Sequential, Causal) constrain the possible outcomes. Excellent for building intuition. -------------------------------------------------------------------------------- /dump/07.Failure-Detection-Heart-beat-and-Gossip-Protocol.md: -------------------------------------------------------------------------------- 1 | ### **Core Concept: Failure Detection in Distributed Systems** 2 | 3 | A fundamental challenge in distributed systems is knowing whether a node (a server, process, or service) is alive and functioning. We can't rely on a single "are you alive?" check because a network delay might make a live node appear dead (a **false positive**), or we might not notice a node has crashed for a long time (high **detection time**). 4 | 5 | Heartbeat and Gossip are two complementary mechanisms used to solve this problem. 6 | 7 | --- 8 | 9 | ## **1. Heartbeat Protocol** 10 | 11 | The heartbeat protocol is a simple, direct, and push-based method for failure detection. 12 | 13 | ### **How It Works:** 14 | 1. **Regular Pings:** A designated node (or a set of nodes) periodically sends a small "I am alive" message (a *heartbeat*) to one or more other nodes. 15 | 2. **Expectation:** The receiving nodes expect these messages at a regular interval (e.g., every `T` milliseconds). 16 | 3. **Timeout & Detection:** If a receiver does not get a heartbeat within a predefined timeout period (e.g., `k * T`, where `k` is usually 2 or 3), it suspects that the sender has failed. 17 | 18 | ### **Key Characteristics:** 19 | * **Centralized or Hierarchical:** Often used in master-slave or leader-follower architectures. The slaves heartbeats to the master, or a central monitoring service collects heartbeats from all. 20 | * **Simple & Low Overhead:** The messages are very small, containing minimal data (often just a node ID and a timestamp). 21 | * **Scalability Issues:** In a system with `N` nodes, if every node heartbeats to every other node, the number of messages scales as `O(N^2)`, which becomes a bottleneck. This is usually avoided by having nodes report to a central coordinator instead. 22 | * **Tunable:** The detection time and network load can be tuned by adjusting the interval `T` and the timeout multiplier `k`. A smaller `T` means faster detection but higher network load. 23 | 24 | ### **Pros:** 25 | * Simple to implement and understand. 26 | * Provides strong, predictable consistency in failure detection (within the timeout window). 27 | * Low per-message overhead. 28 | 29 | ### **Cons:** 30 | * The central coordinator is a **single point of failure (SPOF)**. If it crashes, the entire failure detection system fails. 31 | * Doesn't scale well to massive, decentralized systems (like thousands of nodes) due to the central coordinator bottleneck. 32 | 33 | --- 34 | 35 | ## **2. Gossip Protocol (Epidemic Protocol)** 36 | 37 | The gossip protocol is a decentralized, probabilistic, and peer-to-peer method for disseminating information (including failure detection data) across a cluster. 38 | 39 | ### **How It Works (for Membership/Failure Detection):** 40 | 1. **Local Membership List:** Each node maintains a local list of cluster members and their suspected state (e.g., `ALIVE`, `SUSPECT`, `DEAD`). 41 | 2. **Gossip Rounds:** Periodically (e.g., every `T` ms), each node randomly selects a few (usually 1-3) other nodes from its list and sends them its entire membership list (or a digest of it). 42 | 3. **Merging Information:** When a node receives a gossip message, it merges the incoming information with its own. For example: 43 | * If it sees a higher heartbeat counter for a node, it updates its own. 44 | * If it sees a node it thought was alive is marked `SUSPECT` by others, it may update its state. 45 | 4. **Failure Detection:** If a node `A` doesn't hear about node `B` (either directly or indirectly through gossip) for a prolonged period, it will eventually mark `B` as `SUSPECT`. As other nodes also stop hearing about `B`, they too will mark it as `SUSPECT` and then `DEAD`, creating a consensus about its failure. 46 | 47 | ### **Key Characteristics:** 48 | * **Decentralized:** There is no single point of failure or bottleneck. 49 | * **Probabilistic:** It doesn't provide *absolute* consistency at every moment. There's a small chance that different nodes have a slightly different view of the cluster, but these views **eventually converge** (this is called **Eventual Consistency**). 50 | * **Scalable:** The number of messages per node is constant (`O(1)`), so the total network traffic scales linearly (`O(N)`) with the number of nodes. This makes it excellent for very large systems. 51 | * **Robust:** Highly fault-tolerant. The failure of any number of nodes doesn't prevent the remaining ones from communicating. 52 | * **High Overhead (per message):** The messages are larger as they carry state information about multiple nodes. 53 | 54 | ### **Pros:** 55 | * Extremely scalable and fault-tolerant. 56 | * No single point of failure. 57 | * Naturally load-balances itself as communication is random. 58 | 59 | ### **Cons:** 60 | * More complex to implement correctly. 61 | * Failure detection is not instantaneous; it takes some time for the knowledge of a failure to propagate (though this time is logarithmic with cluster size). 62 | * Provides only eventual consistency, which can be harder to reason about. 63 | 64 | --- 65 | 66 | ## **Comparison Table: Heartbeat vs. Gossip** 67 | 68 | | Feature | Heartbeat Protocol | Gossip Protocol | 69 | | :--- | :--- | :--- | 70 | | **Architecture** | Centralized / Hierarchical | Decentralized / Peer-to-Peer | 71 | | **Scalability** | Poor (`O(N^2)` or SPOF) | Excellent (`O(N)`) | 72 | | **Fault Tolerance** | Low (SPOF on coordinator) | Very High (no SPOF) | 73 | | **Failure Detection** | Strong, within timeout | Probabilistic, eventual | 74 | | **Detection Speed** | Predictable (based on timeout) | Less predictable (logarithmic spread) | 75 | | **Message Size** | Very Small | Larger (contains state info) | 76 | | **Complexity** | Simple | Complex | 77 | | **Use Case** | Smaller clusters, master-slave setups | Large, dynamic, decentralized clusters | 78 | 79 | --- 80 | 81 | ## **How They Work Together** 82 | 83 | In modern systems, these protocols are often used together in a hybrid approach to leverage the strengths of both. 84 | 85 | A common pattern is: 86 | 1. **Intra-Rack/Zone Heartbeat:** Use heartbeats within a smaller, reliable group (e.g., a rack or availability zone) for fast, predictable failure detection. 87 | 2. **Inter-Rack/Zone Gossip:** Use gossip protocol *between* these groups or coordinators to disseminate the cluster-wide state in a scalable and fault-tolerant way. 88 | 89 | **Example: Apache Cassandra** 90 | * Each node uses a **Gossip Protocol** to discover and share the state of other nodes in the cluster every second. 91 | * It also uses a lighter-weight **accrual failure detector** (a sophisticated form of heartbeat analysis) internally to decide if a specific node is alive based on the statistical history of heartbeat arrival times. 92 | 93 | --- 94 | 95 | ## **External Materials for In-Depth Learning** 96 | 97 | ### **1. Research Papers (The Classics)** 98 | * **"A Gossip-Style Failure Detection Service" (1998)** 99 | * **Link:** [PDFs are easily found via search](https://www.cs.cornell.edu/home/rvr/papers/GossipFD.pdf) 100 | * **Why:** This is one of the foundational papers that formalized the use of gossip for failure detection. It's very readable. 101 | * **"The Phi Accrual Failure Detector" (2004)** 102 | * **Link:** [https://issues.apache.org/jira/secure/attachment/12376610/HD.pdf](https://issues.apache.org/jira/secure/attachment/12376610/HD.pdf) 103 | * **Why:** Describes a sophisticated failure detector (used in Cassandra, Akka) that uses heartbeats but provides a probabilistic, adaptive output (a value called *Phi*) instead of a simple binary "up/down". It's a great deep dive into improving heartbeat mechanisms. 104 | 105 | ### **2. Online Articles & Blogs** 106 | * **Martin Kleppmann's "Gossip" Chapter** 107 | * **Link:** [https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/ch08.html](https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/ch08.html) 108 | * **Why:** His famous book, "Designing Data-Intensive Applications," has an entire section on gossip and membership protocols. It's an incredibly clear and practical explanation. 109 | * **CockroachDB Blog: "How Gossip Works"** 110 | * **Link:** [https://www.cockroachlabs.com/blog/how-gossip-works/](https://www.cockroachlabs.com/blog/how-gossip-works/) 111 | * **Why:** A fantastic, modern explanation from a major database that uses gossip internally. It breaks down the concept with clear examples. 112 | 113 | ### **3. Video Lectures** 114 | * **MIT 6.824: Distributed Systems (Lecture 8: eventual consistency, Bitcoin)** 115 | * **Link:** [https://www.youtube.com/watch?v=Q2d2BziqZ-M](https://www.youtube.com/watch?v=Q2d2BziqZ-M) (Check the course schedule for the exact lecture on gossip). 116 | * **Why:** One of the best university courses on distributed systems. The lectures are deep and assume a good level of technical proficiency. 117 | * **CSE 138 (UC Santa Cruz): Lecture 6 - Gossip** 118 | * **Link:** [https://www.youtube.com/watch?v=Q2d2BziqZ-M](https://www.youtube.com/watch?v=Q2d2BziqZ-M) (Search for "CSE 138 Gossip") 119 | * **Why:** A more accessible video lecture that focuses specifically on explaining the gossip protocol. 120 | 121 | ### **4. Practical Implementations (Read the Code/Specs)** 122 | * **Apache Cassandra: Gossip Documentation** 123 | * **Link:** [https://cassandra.apache.org/doc/latest/architecture/gossip.html](https://cassandra.apache.org/doc/latest/architecture/gossip.html) 124 | * **Why:** See how a real-world, production-grade system implements and uses gossip. 125 | * **Hashicorp Serf (Library)** 126 | * **Link:** [https://www.serf.io/](https://www.serf.io/) 127 | * **Why:** Serf is a decentralized solution for cluster membership, failure detection, and orchestration built on gossip. Its documentation is a great practical guide. Tools like Consul and Nomad build on top of it. -------------------------------------------------------------------------------- /dump/01.Intverted-Index.md: -------------------------------------------------------------------------------- 1 | ### **Inverted Index** 2 | 3 | #### **1. Core Concept: What is it?** 4 | 5 | An **Inverted Index** is a data structure that maps content (like words or numbers) to its locations in a set of documents. It is the opposite of a "forward index," which maps documents to their content. 6 | 7 | * **Analogy:** Think of the index at the back of a textbook. You look up a **keyword** (e.g., "photosynthesis") and it gives you a **list of page numbers** where that term appears. The inverted index does this digitally and at a massive scale. 8 | * **Primary Goal:** To allow fast full-text searches. Without it, searching for a word would require scanning every document in a collection—a process far too slow for modern data volumes. 9 | 10 | #### **2. Key Terminology** 11 | 12 | * **Document:** A unit of information you are indexing (e.g., a web page, a book chapter, a tweet, a product description). 13 | * **Corpus / Collection:** The entire set of documents being indexed. 14 | * **Term / Token:** A normalized word or element from the document text (e.g., "search", "engine"). The process of converting text into terms is called **tokenization** and **normalization** (lowercasing, stemming, removing stopwords like "the", "a"). 15 | * **Posting List:** The heart of the index. For a given term, it is the sorted list of all documents that contain that term. Each entry in the list is called a **posting**. 16 | * **Posting:** An entry in a posting list. A simple posting is just a **Document ID (docID)**. An enriched posting can also include: 17 | * **Term Frequency (tf):** How many times the term appears in the document. 18 | * **Positions:** The exact word offsets where the term appears (crucial for phrase queries like "black sea"). 19 | * **Other metadata:** e.g., if the term was in the title, a heading, etc. 20 | 21 | #### **3. Basic Structure** 22 | 23 | An inverted index consists of two main parts: 24 | 25 | 1. **Dictionary (or Vocabulary):** A sorted list of all unique terms (tokens) found in the corpus. 26 | 2. **Postings:** For each term in the dictionary, a pointer to its corresponding posting list. 27 | 28 | **Visual Representation:** 29 | 30 | ``` 31 | Dictionary (Terms) | Postings Lists (docID:positions; ...) 32 | --------------------------------------------------------------------- 33 | ... | 34 | "cat" ----------->| [ (doc17: tf=2, pos=[4, 12]), (doc84: tf=1, pos=[7]) ] 35 | "dog" ----------->| [ (doc17: tf=1, pos=[8]), (doc23: tf=3, pos=[1, 5, 9]) ] 36 | "mouse" ----------->| [ (doc84: tf=1, pos=[3]) ] 37 | ... | 38 | ``` 39 | 40 | #### **4. How it Works: A Step-by-Step Process** 41 | 42 | **A. Index Construction (Building the Index)** 43 | 1. **Document Acquisition:** Gather the documents to be indexed. 44 | 2. **Tokenization:** Break each document's text into a stream of tokens (words, punctuation, etc.). 45 | 3. **Linguistic Preprocessing:** 46 | * **Normalization:** Convert text to a standard form (e.g., lowercasing "The" to "the"). 47 | * **Stopword Removal:** Filter out extremely common words that carry little meaning (e.g., "the", "is", "at"). This significantly reduces index size. 48 | * **Stemming/Lemmatization:** Reduce words to their root form (e.g., "fishing", "fished", "fisher" -> "fish"). 49 | 4. **Building Postings:** For each term, record the docID and any other required metadata (position, frequency). 50 | 5. **Sorting:** Sort the dictionary alphabetically and the postings lists by docID (this is critical for efficient query processing). 51 | 52 | **B. Query Processing (Using the Index)** 53 | For a simple query for a single term (e.g., "dog"): 54 | 1. **Lookup:** Find the term "dog" in the dictionary. 55 | 2. **Retrieve:** Fetch its entire posting list. 56 | 3. **Return Results:** The results are the documents in the posting list (e.g., doc17 and doc23). 57 | 58 | For a complex query (e.g., "cat AND dog"): 59 | 1. **Lookup:** Find the posting lists for "cat" and "dog". 60 | 2. **Algorithm:** Perform an **intersection** of the two sorted lists. This is done efficiently using a two-pointer walk since the lists are sorted by docID. 61 | * Start at the beginning of both lists. 62 | * Compare the current docIDs. If they match, add that docID to the result list and advance both pointers. 63 | * If one ID is smaller, advance that pointer. 64 | * Repeat until the end of one list is reached. 65 | 3. **Ranking:** The resulting list of documents is often then scored and ranked by relevance (using metrics like TF-IDF, BM25, etc.) before being returned to the user. 66 | 67 | #### **5. Optimizations & Advanced Concepts** 68 | 69 | * **Skip Pointers:** To speed up the intersection of long posting lists, "skip lists" are used. These are pointers that allow the algorithm to jump over large blocks of non-matching docIDs. 70 | * **Compression:** Posting lists can be *massively long*. Compression techniques (like Variable Byte Encoding or Frame-of-Reference) are essential to reduce their memory and disk footprint. The goal is to store the *gaps between docIDs* (d-gaps) rather than the raw IDs, as the gaps are smaller numbers and compress better. 71 | * **Distributed Indexing:** For web-scale corpora (billions of documents), the index is too large for one machine. It is partitioned. 72 | * **Term Partitioning:** Split the dictionary across machines (e.g., Machine A handles terms A-F). Rarely used as it makes querying for multiple terms slow. 73 | * **Document Partitioning (Sharding):** Split the documents across machines. Each machine builds an index for its subset of documents. A query is sent to all shards and the results are merged. This is the most common approach (used by Google, etc.). 74 | * **Dynamic Indexing:** How to handle new, updated, or deleted documents. Simple solutions involve periodically rebuilding the entire index. More advanced systems use an auxiliary "smaller" index for new documents and merge it with the main index later. 75 | 76 | #### **6. Pros and Cons** 77 | 78 | | Pros | Cons | 79 | | -------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | 80 | | **Extremely fast** for Boolean and keyword queries. | **High storage overhead.** The index can be larger than the original corpus. | 81 | | The foundation for **ranked retrieval** (using TF, IDF, etc.). | **Complex to build and maintain** for large, dynamic collections. | 82 | | **Efficient** for complex query operations (AND, OR, NOT, phrases). | **Slow for updates.** Adding a single document requires updating many posting lists. | 83 | | Naturally supports **compression** and **distribution**. | Not ideal for "fuzzy" searches or wildcard queries at the beginning of a term (e.g., `*search`). | 84 | 85 | --- 86 | 87 | ### **External Materials for In-Depth Learning** 88 | 89 | #### **1. Foundational Textbooks (The "Bible")** 90 | 91 | * **Introduction to Information Retrieval** by Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 92 | * **Why:** This is the canonical academic textbook on the subject. It dedicates entire chapters to the inverted index, its construction, compression, and query processing. 93 | * **Link:** The full book is available **free online** from the authors: [https://nlp.stanford.edu/IR-book/information-retrieval-book.html](https://nlp.stanford.edu/IR-book/information-retrieval-book.html) 94 | * **Key Chapters:** 1, 2, and 4 are essential reading. 95 | 96 | #### **2. Online Courses & Lectures** 97 | 98 | * **Stanford CS276: Information Retrieval and Web Search (Lecture Videos)** 99 | * **Why:** Taught by the authors of the textbook above. The lectures provide an excellent visual and narrative explanation of the concepts. 100 | * **Link:** Search for "CS276" on YouTube. Specific lectures on indexing are among the first few. 101 | 102 | * **Coursera: Text Retrieval and Search Engines (University of Illinois)** 103 | * **Why:** A very well-structured course that covers the inverted index in the context of building a full search engine. 104 | * **Link:** [https://www.coursera.org/learn/text-retrieval](https://www.coursera.org/learn/text-retrieval) (You can often audit for free). 105 | 106 | #### **3. Practical Implementations & Blogs** 107 | 108 | * **Elasticsearch / Apache Lucene Documentation:** 109 | * **Why:** Lucene is the most widely used, open-source search library (it powers Elasticsearch, Solr, etc.). Its documentation and code are the *real-world implementation* of all these concepts. 110 | * **Link:** Read about Lucene's **"Inverted Index"** and **"Postings"** format. The blog posts from Elastic are also excellent. 111 | * **Key Search Terms:** "Apache Lucene inverted index", "Elasticsearch inverted index", "Lucene postings format". 112 | 113 | * **Blogs:** 114 | * **"How Elasticsearch works: An overview of the architecture"** (by Opster): A great high-level overview. 115 | * **"Building a search engine"** posts on sites like Medium often walk through building a simple inverted index in Python, which is fantastic for understanding. 116 | 117 | #### **4. Academic Papers (For Depth)** 118 | 119 | * **"Indexing and Searching"** chapter in **Managing Gigabytes** by Ian H. Witten, Alistair Moffat, and Timothy C. Bell. 120 | * **Why:** This book is a classic focused specifically on compression and efficient storage of text, with the inverted index being a central theme. It goes into incredible detail on compression techniques. 121 | 122 | * **Google's Original Paper:** While not solely about the index, it describes the system that proved web-scale search was possible. 123 | * **Title:** "The Anatomy of a Large-Scale Hypertextual Web Search Engine" (Sergey Brin and Lawrence Page). 124 | * **Why:** For historical context and to see how these concepts are applied at an unprecedented scale. -------------------------------------------------------------------------------- /dump/09.Message-Queues-Pub-Sub.md: -------------------------------------------------------------------------------- 1 | ### Core Concepts: The Problem They Solve 2 | 3 | Modern applications are rarely monolithic. They are distributed systems composed of many decoupled, specialized services (microservices, serverless functions, etc.). These services need to communicate. Doing this directly via HTTP calls (synchronous communication) creates tight coupling and several problems: 4 | 5 | * **Tight Coupling:** If Service A calls Service B directly, if B is down or slow, A also fails or becomes slow. 6 | * **No Buffering:** A sudden spike in traffic can overwhelm a service. 7 | * **Complexity:** Point-to-point connections between services become a tangled web that is hard to manage and scale. 8 | 9 | **Message-Driven Architectures** solve this by introducing an intermediary—a *message broker*—that facilitates **asynchronous communication**. 10 | 11 | --- 12 | 13 | ### 1. Messaging Queues (Point-to-Point) 14 | 15 | #### Core Idea 16 | A **producer** sends a message to a **queue**. **Exactly one consumer** (a service that is listening to that queue) receives and processes that message. Once processed, the message is removed from the queue. 17 | 18 | #### Key Characteristics 19 | * **Consumption Model:** Competing Consumers Pattern. Multiple consumer instances can listen to the same queue for load balancing, but each message is processed by only **one** of them. 20 | * **Message Lifecycle:** The message is deleted from the queue after successful processing (acknowledgement). 21 | * **Ordering:** Messages are typically consumed in a FIFO (First-In-First-Out) order, though this isn't always guaranteed in all systems without configuration. 22 | * **Use Case:** **Task Distribution**, **Decoupling**, **Load Leveling**. Ideal for triggering asynchronous jobs or processing commands. 23 | 24 | #### Example 25 | An e-commerce website places an order. It sends a "ProcessPayment" message to a queue. One of many available "Payment Processor" services picks it up and handles it. 26 | * **Producer:** Web Application 27 | * **Queue:** `payment-queue` 28 | * **Consumer:** Payment Processing Service 29 | 30 | #### Key Terminology 31 | * **Producer/Sender:** The service that sends the message. 32 | * **Consumer/Receiver:** The service that receives and processes the message. 33 | * **Queue:** The buffer that stores messages. 34 | * **Message:** The data packet being sent, often JSON or Protocol Buffers. 35 | * **Acknowledgement (Ack):** A signal from the consumer to the broker that the message was processed successfully and can be deleted. 36 | * **Negative-Acknowledgement (Nack):** A signal that processing failed. The broker can then redeliver the message or move it to a Dead-Letter Queue (DLQ). 37 | 38 | --- 39 | 40 | ### 2. Publish/Subscribe (Pub/Sub) 41 | 42 | #### Core Idea 43 | A **publisher** sends a message to a **topic**. The message is then delivered to **all subscribers** who are currently subscribed to that topic. The topic doesn't know or care what the subscribers do with the message. 44 | 45 | #### Key Characteristics 46 | * **Consumption Model:** Fan-out. A single message is broadcast to **multiple, independent consumers**. 47 | * **Message Lifecycle:** The message is delivered to all subscribers and then typically discarded (unless retention is configured). It's not "consumed" in the same way as a queue. 48 | * **Coupling:** Publishers are completely decoupled from subscribers. A publisher doesn't know who or how many subscribers exist. 49 | * **Use Case:** **Event Notification**, **Event Sourcing**, **Real-time Feeds**. Ideal for broadcasting state changes or events that multiple parts of a system need to react to. 50 | 51 | #### Example 52 | The same e-commerce website completes a payment. The "Payment Service" publishes a "PaymentCompleted" event to a topic. Multiple independent services are subscribed: 53 | * The "Order Service" listens to update the order status to 'confirmed'. 54 | * The "Email Service" listens to send a confirmation email to the customer. 55 | * The "Analytics Service" listens to update the sales dashboard. 56 | * **Publisher:** Payment Service 57 | * **Topic:** `payment-completed-topic` 58 | * **Subscribers:** Order Service, Email Service, Analytics Service 59 | 60 | #### Key Terminology 61 | * **Publisher:** The service that sends the message to a topic. 62 | * **Subscriber:** The service that receives messages from a topic it is interested in. 63 | * **Topic:** The channel to which messages are sent. (In AWS SNS, it's called a "Topic"; in Kafka, it's a "Topic" with partitions; in RabbitMQ, it's an "Exchange" of type "topic"). 64 | * **Subscription:** The link between a topic and a subscriber. A subscriber must have a subscription to a topic to receive messages. 65 | 66 | --- 67 | 68 | ### Comparison Table: Queues vs. Pub/Sub 69 | 70 | | Feature | Messaging Queue (Point-to-Point) | Publish/Subscribe (Pub/Sub) | 71 | | :--- | :--- | :--- | 72 | | **Message Consumption** | **One consumer** processes each message. | **All active subscribers** receive each message. | 73 | | **Consumer Pattern** | Competing Consumers (load balancing) | Fan-out (broadcast) | 74 | | **Coupling** | Producer knows a specific task needs to be done. | Producer only knows an event occurred; doesn't care who reacts. | 75 | | **Ideal Use Case** | Distributing work/tasks, decoupling processing. | Broadcasting events/notifications, system-wide state changes. | 76 | | **Analogy** | A line of customers at a bank teller. | A radio station broadcasting a signal; any radio can tune in. | 77 | 78 | --- 79 | 80 | ### Important Patterns & Concepts 81 | 82 | 1. **Message Broker:** The middleware that implements messaging patterns (e.g., RabbitMQ, Apache Kafka, Amazon SQS/SNS, Google Pub/Sub). 83 | 2. **Persistence:** Messages can be durable (survive broker restart) or transient. 84 | 3. **Delivery Guarantees:** 85 | * **At-most-once:** Message may be lost (fire and forget). 86 | * **At-least-once:** Message is never lost but may be delivered multiple times (requires idempotent consumers). 87 | * **Exactly-once:** Very hard to achieve; usually implemented as at-least-once plus deduplication on the consumer side. 88 | 4. **Dead-Letter Queue (DLQ):** A special queue where messages are sent after repeated failed delivery attempts. Crucial for debugging and handling poison pills. 89 | 5. **Idempotency:** A property of an operation where applying it multiple times has the same effect as applying it once. **Consumers must be designed to be idempotent** because messages can be redelivered (at-least-once semantics). 90 | 91 | --- 92 | 93 | ### Popular Technologies & When to Use Them 94 | 95 | * **RabbitMQ:** A traditional, powerful **message broker**. Excellent for complex routing, point-to-point queues, and simple pub/sub using exchanges. Great for task queues and RPC. 96 | * **Apache Kafka:** A **distributed event streaming platform**. It uses a pub/sub model but persists all messages for a set time. Think of it as a giant, append-only log. Ideal for high-throughput event streaming, event sourcing, and building real-time data pipelines. 97 | * **Amazon SQS & SNS:** AWS's managed services. 98 | * **SQS** is a simple queue service (point-to-point). 99 | * **SNS** is a simple notification service (pub/sub). They are often used together: SNS fans out a message to multiple SQS queues, allowing different services to process messages at their own pace. 100 | * **Google Pub/Sub:** A globally distributed, managed pub/sub service on GCP. Designed for high throughput and low latency, similar to Kafka but fully managed. 101 | * **Redis Pub/Sub & Streams:** Redis can be used for simple, fast, but **transient** (non-persistent) pub/sub. Redis Streams provides more persistence and complex consumer group features. 102 | 103 | --- 104 | 105 | ### External Materials for In-Depth Learning 106 | 107 | #### Articles & Tutorials 108 | 1. **IBM Cloud Docs: Messaging Patterns** 109 | * *Link:* [https://www.ibm.com/cloud/learn/messaging-patterns](https://www.ibm.com/cloud/learn/messaging-patterns) 110 | * *Why:* A very clear and concise explanation of the fundamental patterns with good diagrams. 111 | 112 | 2. **AWS: SQS vs. SNS – Simple Comparison** 113 | * *Link:* [https://aws.amazon.com/compare/the-difference-between-sqs-and-sns/](https://aws.amazon.com/compare/the-difference-between-sqs-and-sns/) 114 | * *Why:* A practical comparison from a major cloud provider, explaining how their services implement these patterns and how they can be used together. 115 | 116 | 3. **The Java Space: Messaging Patterns (Udi Dahan)** 117 | * *Link:* [https://www.udidahan.com/2009/12/09/clarified-command-query-responsibility-segregation/](https://www.udidahan.com/2009/12/09/clarified-command-query-responsibility-segregation/) (Read his older articles on messaging) 118 | * *Why:* Udi Dahan is a renowned expert on messaging and service-oriented architecture. His writings provide deep architectural insights. 119 | 120 | #### Books 121 | 1. **"Designing Data-Intensive Applications" by Martin Kleppmann** 122 | * *Why:* **The definitive book** on building robust systems. Chapter 11 ("Stream Processing") is an absolute masterpiece, explaining messaging, log-based systems like Kafka, and the trade-offs in incredible depth. A must-read for any serious engineer. 123 | 124 | 2. **"Enterprise Integration Patterns" by Gregor Hohpe & Bobby Woolf** 125 | * *Why:* This is the classic pattern catalog for messaging. It's a bit older but defines the vocabulary and patterns (like Message Router, Dead Letter Channel, Competing Consumers) that all modern systems are built upon. The website [https://www.enterpriseintegrationpatterns.com/](https://www.enterpriseintegrationpatterns.com/) has all the patterns with diagrams. 126 | 127 | #### Videos & Courses 128 | 1. **IBM Technology: What is a Message Queue?** 129 | * *Link:* [https://www.youtube.com/watch?v=xErwDaOc-Gs](https://www.youtube.com/watch?v=xErwDaOc-Gs) 130 | * *Why:* A short, excellent visual explanation of the core concept. 131 | 132 | 2. **Kafka Explained (Confluent)** 133 | * *Link:* [https://www.confluent.io/what-is-apache-kafka/](https://www.confluent.io/what-is-apache-kafka/) 134 | * *Why:* Confluent (founded by the creators of Kafka) has fantastic explanations, blogs, and tutorials that dive deep into event streaming. 135 | -------------------------------------------------------------------------------- /dump/02.Database-Replications.md: -------------------------------------------------------------------------------- 1 | ### **Database Replication** 2 | 3 | #### **1. What is Database Replication?** 4 | 5 | Database replication is the process of copying and maintaining database objects (like tables, rows, etc.) in multiple databases that make up a distributed database system. The goal is to ensure that data is consistently and reliably available across different locations, servers, or data centers. 6 | 7 | **Key Purposes:** 8 | * **High Availability:** If the primary database fails, a replica can take over with minimal downtime. 9 | * **Disaster Recovery:** Replicas in geographically distant locations protect against site-wide failures (natural disasters, power outages). 10 | * **Improved Performance:** Read operations can be distributed across multiple replica servers, reducing the load on the primary (source) database. This is often called **read scaling**. 11 | * **Reduced Latency:** Data can be replicated to a location geographically closer to the end-users, speeding up query response times. 12 | * **Analytics and Reporting:** Run heavy analytical queries on a replica without affecting the performance of the primary transactional database. 13 | 14 | #### **2. Core Concepts & Terminology** 15 | 16 | * **Primary (Master/Source):** The main database where all write operations (INSERT, UPDATE, DELETE) are first applied. 17 | * **Replica (Slave/Target):** A copy of the primary database that receives and applies data changes from the primary. 18 | * **Replication Lag:** The delay between a write operation on the primary and that operation being applied to a replica. This is a critical metric to monitor. 19 | * **Conflict Resolution:** Rules to determine which data change "wins" when the same data is modified in two different locations simultaneously (crucial in multi-primary setups). 20 | 21 | --- 22 | 23 | ### **3. Types of Database Replication** 24 | 25 | Replication strategies can be categorized along several axes. The most common are based on **topology** and **synchronization**. 26 | 27 | #### **A. Based on Synchronization (How data is copied)** 28 | 29 | **1. Synchronous Replication** 30 | * **How it works:** A write transaction on the primary is only considered **complete** once it has been successfully written to both the primary *and* at least one replica. 31 | * **Pros:** Guarantees **zero data loss** if the primary fails. The replica is always an exact, up-to-date copy. 32 | * **Cons:** High latency for write operations, as the transaction waits for network round-trips to the replica(s). Performance degrades with distance or network issues. If a replica goes down, writes to the primary may also be blocked. 33 | * **Use Case:** Critical financial systems where data integrity is more important than speed. 34 | 35 | **2. Asynchronous Replication** 36 | * **How it works:** A write transaction on the primary is considered **complete** as soon as it is written to the primary's local storage. The changes are then queued and sent to the replicas at a later time. 37 | * **Pros:** Very low latency for write operations on the primary. The primary is not affected by the performance or availability of the replicas. 38 | * **Cons:** Risk of **data loss**. If the primary fails before the queued changes are sent to the replicas, those recent writes are lost. Replicas are often slightly behind (replication lag). 39 | * **Use Case:** Most common setup. Ideal for read-scaling and disaster recovery where some minor data loss is acceptable. 40 | 41 | **3. Semi-Synchronous Replication** 42 | * **How it works:** A hybrid approach. The primary waits for **acknowledgement** that at least one replica has *received* the data (not necessarily applied it) before committing the transaction. It's a balance between safety and performance. 43 | * **Pros:** Reduces the risk of data loss compared to async, but with lower latency impact than full sync. 44 | * **Cons:** More complex to configure. Still potential for minimal data loss if the primary fails right after sending the acknowledgement but before the replica applies the change. 45 | * **Use Case:** Environments that need stronger guarantees than async but cannot tolerate the performance hit of full sync. 46 | 47 | #### **B. Based on Topology (The flow of data)** 48 | 49 | **1. Single-Primary (Master-Slave) Replication** 50 | * **How it works:** There is only one primary database that accepts write operations. All replicas are read-only and receive a stream of changes from this single primary. 51 | * **Pros:** Simple to implement. Avoids write conflicts because all writes go to one place. 52 | * **Cons:** The primary is a single point of failure for writes. Write throughput is limited by the capacity of a single server. 53 | * **Use Case:** The standard and most widely used model for read-scaling. 54 | 55 | **2. Multi-Primary (Master-Master) Replication** 56 | * **How it works:** Multiple nodes (primaries) can accept write operations. Each primary coordinates with the others to propagate its changes. 57 | * **Pros:** No single point of failure for writes. Enables write operations in different geographic regions with low latency. 58 | * **Cons:** Extremely complex. Requires sophisticated **conflict resolution** mechanisms (e.g., "last write wins," custom logic) to handle cases where the same data is written in two different locations. Increased risk of data inconsistencies. 59 | * **Use Case:** Collaborative applications (like Google Docs) or global applications where users need to write to their local region. 60 | 61 | **3. Logical vs. Physical Replication** 62 | 63 | * **Physical Replication:** Copies the exact disk blocks and byte-by-byte changes from the primary to the replica. The replica is a **bit-for-bit identical copy** of the primary. (e.g., PostgreSQL Physical Replication, Oracle Data Guard). 64 | * *Pros:* Very efficient, low overhead. The replica is always in a consistent state and can be used for failover. 65 | * *Cons:* The primary and replica must be the same database version and often the same OS/architecture. The replica is an exact copy, so you can't replicate a subset of tables. 66 | 67 | * **Logical Replication:** Replicates changes based on the database's transaction log (Write-Ahead Log - WAL) but translates them into logical SQL statements (INSERT, UPDATE) that are executed on the replica. 68 | * *Pros:* Flexible. You can replicate a subset of tables, filter data, or even replicate between different database versions. Useful for zero-downtime upgrades and data warehousing. 69 | * *Cons:* Higher overhead than physical replication. More potential for conflicts and replication lag. 70 | 71 | --- 72 | 73 | ### **4. Summary Table of Replication Types** 74 | 75 | | Type | Description | Pros | Cons | Best For | 76 | | :--- | :--- | :--- | :--- | :--- | 77 | | **Synchronous** | Waits for replica confirmation | Zero data loss | High write latency | Financial systems, critical data | 78 | | **Asynchronous** | Doesn't wait for replica | Low write latency | Risk of data loss | Read-scaling, general use | 79 | | **Single-Primary** | One write source, many read replicas | Simple, no write conflicts | Primary is a SPOF, write bottleneck | Most common use cases | 80 | | **Multi-Primary** | Multiple write sources | Write scalability, no SPOF | Complex conflict resolution | Global write availability | 81 | | **Physical** | Copies disk blocks | Efficient, consistent | Inflexible (must be identical) | High Availability, Failover | 82 | | **Logical** | Replays SQL statements | Flexible (subset, filtering) | Higher overhead | Data aggregation, upgrades | 83 | 84 | --- 85 | 86 | ### **5. External Materials for In-Depth Learning** 87 | 88 | #### **Books** 89 | * **"Designing Data-Intensive Applications" by Martin Kleppmann:** This is the definitive modern book on this topic. **Chapter 5 (Replication)** and **Chapter 7 (Transactions)** are directly relevant and provide an exceptional, deep dive into the theory, trade-offs, and implementation details of replication and consistency models. It's a must-read. 90 | * **"Database Internals" by Alex Petrov:** This book goes deep into how databases work under the hood, including storage engines and replication algorithms. Excellent for understanding the implementation perspective. 91 | 92 | #### **Documentation (Best for specific technologies)** 93 | * **PostgreSQL Replication:** The official docs are superb. 94 | * [Physical Replication (Streaming Replication)](https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION) 95 | * [Logical Replication](https://www.postgresql.org/docs/current/logical-replication.html) 96 | * **MySQL Replication:** The standard for master-slave setups. 97 | * [MySQL Replication Documentation](https://dev.mysql.com/doc/refman/8.0/en/replication.html) 98 | * **Amazon Aurora:** A commercial database that has a very advanced, cloud-native replication implementation (quorum-based storage). 99 | * [Amazon Aurora Global Database](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database.html) (Multi-Region replication) 100 | * **MongoDB Replication:** Based on a replica set (a form of asynchronous single-primary replication with automatic failover). 101 | * [MongoDB Replication](https://www.mongodb.com/docs/manual/replication/) 102 | 103 | #### **Articles & Blogs** 104 | * **AWS Database Blog:** Search for articles on replication for RDS, Aurora, and DynamoDB Global Tables. They provide practical insights into cloud-based replication. 105 | * **Vitess (Sharding/Replication for MySQL):** [Vitess Documentation](https://vitess.io/docs/)) - Learn how large-scale systems like YouTube manage replication and sharding together. 106 | * **CockroachDB (Multi-Primary DB):** [CockroachDB Multi-Region Overview](https://www.cockroachlabs.com/docs/stable/multiregion-overview.html) - Excellent resource for understanding how a modern distributed SQL database handles multi-primary replication and consistency. 107 | 108 | #### **Videos & Courses** 109 | * **CMU Database Group YouTube Channel:** Lectures from one of the top database research groups in the world. Search for topics on "replication" and "distributed systems." 110 | * **Coursera / edX:** Courses on **Distributed Systems** or **Cloud Computing** often cover replication in depth. For example: 111 | * **Cloud Computing Concepts** (University of Illinois on Coursera) 112 | * **Distributed Systems** (MIT OpenCourseWare) -------------------------------------------------------------------------------- /dump/04.CAP-Theorem.md: -------------------------------------------------------------------------------- 1 | ### **CAP Theorem** 2 | 3 | #### **1. The Core Principle: What is the CAP Theorem?** 4 | 5 | The **CAP Theorem**, also known as **Brewer's Theorem** after computer scientist Eric Brewer who proposed it in 2000, is a fundamental principle in the field of distributed computing systems. It states that it is **impossible** for a distributed data store to simultaneously provide more than two out of the following three guarantees: 6 | 7 | * **C**onsistency 8 | * **A**vailability 9 | * **P**artition Tolerance 10 | 11 | **Important Note:** The theorem was formally proven by Seth Gilbert and Nancy Lynch of MIT in 2002, giving it a solid mathematical foundation. It's often misunderstood as a strict "choose two" rule, but its practical implications are more nuanced. 12 | 13 | --- 14 | 15 | #### **2. Detailed Breakdown of the Three Guarantees** 16 | 17 | * **C - Consistency (Linearizability)** 18 | * **Meaning:** Every read receives the *most recent write* or an error. After a write is completed, any client reading from any node in the system will see the same, updated value. 19 | * **Analogy:** It's like having a single, global state of truth. The system behaves as if there is only one copy of the data, even though there are many replicas. 20 | * **Technical Term:** This is often referred to as **strong consistency**. 21 | 22 | * **A - Availability** 23 | * **Meaning:** Every request (read or write) received by a non-failing node in the system must result in a *response* (it cannot just hang or return an error). The system remains operational 100% of the time. 24 | * **Key Point:** The response does *not* have to be the most recent write. It just has to be a valid response. This is a guarantee about the system's *liveness*. 25 | 26 | * **P - Partition Tolerance** 27 | * **Meaning:** The system continues to operate *despite an arbitrary number of messages being dropped (or delayed) by the network* between nodes. A "network partition" is a break in communication, splitting the network into isolated groups of nodes that can't talk to each other. 28 | * **Crucial Insight:** In a distributed system, networks are inherently unreliable. Partitions *will* happen. Therefore, **Partition Tolerance is not a choice—it is a necessity.** You cannot avoid building for `P`. 29 | 30 | --- 31 | 32 | #### **3. The Three Impossible Combinations (and the "2 of 3" Myth)** 33 | 34 | Since `P` is mandatory in any modern distributed system (you can't have a system that fails every time a network glitch occurs), the real-world choice boils down to **CP vs. AP**. 35 | 36 | | Combination | Description | What you sacrifice | Example Systems | 37 | | :---------- | :--------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------- | 38 | | **CA** | Consistency + Availability. | **Partition Tolerance.** This is only possible in a non-distributed (single-node) system. In reality, **CA does not exist** for distributed systems. | A single database server. | 39 | | **CP** | Consistency + Partition Tolerance. | **Availability.** During a network partition, the system will become unavailable (return errors or time out) to prevent inconsistent data. | MongoDB (with specific config), Redis, HBase, Zookeeper | 40 | | **AP** | Availability + Partition Tolerance. | **Consistency.** During a partition, the system remains available but may return stale or conflicting data. Consistency is achieved eventually when the partition heals. | Cassandra, DynamoDB, Riak, Voldemort, CouchDB | 41 | 42 | **Visualizing the Trade-off:** 43 | When a network partition occurs, you are forced to make a decision: 44 | * **Cancel the operation?** (Choose **C**, sacrifice **A**): You ensure data is consistent across partitions by refusing writes/reads that can't be verified, making parts of the system unavailable. 45 | * **Proceed with the operation?** (Choose **A**, sacrifice **C**): You allow writes to continue on both sides of the partition to maintain availability, but this creates data inconsistency that must be resolved later. 46 | 47 | --- 48 | 49 | #### **4. Modern Interpretations and Nuances** 50 | 51 | The classic CAP theorem is a useful model but is often considered too simplistic for modern systems. 52 | 53 | * **It's not all-or-nothing:** Systems can be fine-tuned to be *mostly* consistent or *mostly* available. The choice is often on a spectrum and can be configured per operation or per data type. 54 | * **The "P" in CAP is not the only failure:** CAP only considers network partitions. Systems also have to handle node failures, slow nodes, and other issues, which are often managed with techniques like retries and load balancing. 55 | * **PACELC Theorem:** An extension of CAP that provides a more complete model. 56 | * **If there is a Partition (P), how does the system trade-off between Availability and Consistency (A & C)?** 57 | * **Else (E), when the system is running normally in the absence of partitions, how does the system trade-off between Latency (L) and Consistency (C)?** 58 | * This explains why an AP system like Cassandra might still be configured for strong consistency during normal operation, at the cost of higher latency. 59 | 60 | --- 61 | 62 | #### **5. Practical Implications for System Design** 63 | 64 | 1. **Know Your Requirements:** The choice between CP and AP is a business and product decision, not just a technical one. 65 | * Does your application **require** strong consistency? (e.g., financial transaction systems, booking systems). 66 | * Can your application tolerate eventual consistency for higher availability and performance? (e.g., social media likes, comments, product catalogs). 67 | 68 | 2. **It's not forever:** The CP/AP choice is a behavior **during a partition**. Once the partition heals, both types of systems work to resolve any inconsistencies and return to a fully consistent state. 69 | 70 | 3. **Hybrid Approaches:** Many large systems use a mix of databases. They might use a CP database for core, critical data and an AP database for less critical, high-volume data. 71 | 72 | --- 73 | 74 | ### **External Materials for In-Depth Learning** 75 | 76 | Here is a curated list of resources, from foundational papers to accessible articles. 77 | 78 | #### **Foundational Papers & Academic Sources** 79 | 80 | 1. **Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services** (2002) 81 | * **Authors:** Seth Gilbert and Nancy Lynch. 82 | * **Why read it?** This is the paper that formally *proved* Brewer's conjecture, turning it into a theorem. It's the canonical source. 83 | * **Link:** [https://users.ece.cmu.edu/~adrian/731-sp04/readings/GL-cap.pdf](https://users.ece.cmu.edu/~adrian/731-sp04/readings/GL-cap.pdf) 84 | 85 | 2. **Harvest, Yield, and Scalable Tolerant Systems** (1999) & **CAP Twelve Years Later: How the "Rules" Have Changed** (2012) 86 | * **Author:** Eric Brewer. 87 | * **Why read them?** These are essential readings from the original creator of the theorem. The first introduces the "harvest/yield" metaphor for thinking about CAP. The second is a retrospective where Brewer clarifies common misconceptions and discusses how the theory has evolved with modern practices. 88 | * **Links:** 89 | * Harvest, Yield: [https://arxiv.org/pdf/2002.08364](https://arxiv.org/pdf/2002.08364) (Note: This is a later reprint) 90 | * CAP Twelve Years Later: [https://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed/](https://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed/) 91 | 92 | #### **High-Quality Articles & Blog Posts** 93 | 94 | 3. **Martin Kleppmann's "Please stop calling databases CP or AP"** 95 | * **Why read it?** A fantastic critique of the oversimplified application of the CAP theorem. Kleppmann, author of "Designing Data-Intensive Applications," argues that the labels are misleading and explains the nuances brilliantly. 96 | * **Link:** [https://martin.kleppmann.com/2015/05/11/please-stop-calling-databases-cp-or-ap.html](https://martin.kleppmann.com/2015/05/11/please-stop-calling-databases-cp-or-ap.html) 97 | 98 | 4. **AWS Documentation: "The CAP Theorem in Amazon DynamoDB"** 99 | * **Why read it?** A perfect example of how a major cloud provider explains the trade-offs in the context of their own managed AP database service. It's practical and directly applicable. 100 | * **Link:** [https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/BestPractices.html#BP.CAP](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/BestPractices.html#BP.CAP) 101 | 102 | 5. **Ivy's Tech Blog: "Visual Guide to NoSQL Systems"** 103 | * **Why read it?** A simple, brilliant visual chart that maps popular databases onto the CAP triangle. It's an excellent, at-a-glance resource to see where different technologies generally fall. 104 | * **Link:** [http://blog.iyi.ca/archives/2010/12/visual-guide-to-nosql-systems.html](http://blog.iyi.ca/archives/2010/12/visual-guide-to-nosql-systems.html) (Note: The original is old but the concept is still useful. Newer versions exist online). 105 | 106 | #### **Books** 107 | 108 | 6. **"Designing Data-Intensive Applications" by Martin Kleppmann** 109 | * **Why read it?** This is arguably the best single resource on the topic. **Chapter 9 ("Consistency and Consensus") and Chapter 5 ("Replication")** provide a deep, nuanced, and practical exploration of the CAP theorem, its limitations, and the real-world techniques (like consensus algorithms) that systems use to navigate these trade-offs. It is a must-read. 110 | 111 | 7. **"Database Internals" by Alex Petrov** 112 | * **Why read it?** For those who want to understand how databases are actually built, this book dives deep into storage engines and distributed algorithms. It provides the low-level implementation details behind the high-level CAP guarantees. -------------------------------------------------------------------------------- /networking/README.md: -------------------------------------------------------------------------------- 1 | ### OSI Model 2 | Open System Interconnection by International Organization for Standardization in 1984. It provides a framework for creating and implementing networking standards, devices, and internetworking schemes. 3 | | Group | Layer No. | Layer Name | Description | 4 | | ------ | --------- | ----------- | ----------- | 5 | | Top Layers | 7 | Application | Provide a user interface for sending and receiving data | 6 | | - | 6 | Presentation | Encrypt, format, and compress data for transmission | 7 | | - | 5 | Session | Initiate and terminate a session with the remote system | 8 | | Bottom Layers | 4 | Transport | Break the data stream into smaller segments and provide reliable and unreliable data delivery | 9 | | - | 3 | Network | Provide logical addressing | 10 | | - | 2 | Data Link | Prepare data for transmission | 11 | | - | 1 | Physical | Move data between devices | 12 | #### Application Layer 13 | The Top layer of the OSI model is the application layer. It provides the protocols and services that are required by the network-aware applications to connect to the network. FTP, TFTP, POP3, SMTP, and HTTP are examples of standards and protocols used in this layer. 14 | #### Presentation Layer 15 | Conversion, compression, and encryption are the main functions that the Presentation layer performs on the sending computer while on the receiving computer these functions are reconversion, decompression, and decryption. ASCII, BMP, GIF, JPEG, WAV, AVI, and MPEG are examples of standards and protocols that work in this layer. 16 | #### Session Layer 17 | The session layer is responsible for establishing, managing, and terminating communications between two computers. RPCs and NFS are examples of the session layer. 18 | #### Transport Layer 19 | The main functionalities of the Transport layer are segmentation, data transportation, and connection multiplexing. For data transportation, it uses TCP and UDP protocols. TCP is a connection-oriented protocol. It provides reliable data delivery. 20 | #### Network Layer 21 | Defining logical addresses and finding the best path to reach the destination address are the main functions of this layer. Routers work in this layer. Routing also takes place in this layer. IP, IPX, and AppleTalk are examples of this layer. 22 | #### Data Link Layer 23 | Defining physical addresses, finding hosts in the local network, specifying standards and methods to access the media are the primary functions of this layer. Switching takes place in this layer. Switches and Bridges work in this layer. HDLC, PPP, and Frame Relay are examples of this layer. 24 | 25 | This layer has two sub-layers: MAC(Media Access Control) and LLC(Logical Link Control) 26 | #### Physical Layer 27 | The Physical Layer mainly defines standards for media and devices that are used to move data across the network. 10BaseT, 10Base100, CSU/DSU, DCE, and DTE are examples of the standards used in this layer. 28 | 29 |
30 | 31 | ### TCP/IP 32 | TCP/IP stands for Transmission Control Protocol/ Internet Protocol. TCP/IP Stack is specifically designed as a model to offer highly reliable and end-to-end byte stream over an unreliable internetwork 33 | ![image](https://user-images.githubusercontent.com/20329508/194799658-2d5f3b36-1222-4266-b97b-78a1df08d832.png) 34 | 35 | ![image](https://user-images.githubusercontent.com/20329508/194799832-a3bf5a6e-8dde-4963-aedc-20811c4aecd1.png) 36 | 37 | 38 | TCP/IP model is developed by ARPANET (Advanced Research Project Agency Network). 39 | Some widely used most common TCP/IP protocol are: 40 | 41 | ![image](https://user-images.githubusercontent.com/20329508/194826313-8bb5c9da-f9ca-4641-a3b6-8897793ce443.png) 42 | 43 | #### TCP: 44 | Transmission Control Protocol is an internet protocol suite which breaks up the message into TCP Segments and reassembling them at the receiving side. 45 | 46 | #### IP: 47 | An Internet Protocol address that is also known as an IP address is a numerical label. It is assigned to each device that is connected to a computer network which uses the IP for communication. Its routing function allows internetworking and essentially establishes the Internet. Combination of IP with a TCP allows developing a virtual connection between a destination and a source. 48 | 49 | #### HTTP: 50 | The Hypertext Transfer Protocol is a foundation of the World Wide Web. It is used for transferring webpages and other such resources from the HTTP server or web server to the web client or the HTTP client. Whenever you use a web browser like Google Chrome or Firefox, you are using a web client. It helps HTTP to transfer web pages that you request from the remote servers. 51 | 52 | #### SMTP: 53 | SMTP stands for Simple mail transfer protocol. This protocol supports the e-mail is known as a simple mail transfer protocol. This protocol helps you to send the data to another e-mail address. 54 | 55 | #### SNMP: 56 | SNMP stands for Simple Network Management Protocol. It is a framework which is used for managing the devices on the internet by using the TCP/IP protocol. 57 | 58 | #### DNS: 59 | DNS stands for Domain Name System. An IP address that is used to identify the connection of a host to the internet uniquely. However, users prefer to use names instead of addresses for that DNS. 60 | 61 | #### TELNET: 62 | TELNET stands for Terminal Network. It establishes the connection between the local and remote computer. It established connection in such a manner that you can simulate your local system at the remote system. 63 | 64 | #### FTP: 65 | FTP stands for File Transfer Protocol. It is a mostly used standard protocol for transmitting the files from one machine to another. 66 |
67 | 68 | #### [How OSI and TCP/IP work?](https://youtu.be/3kfO61Mensg) 69 | 70 |
71 | 72 | ### IP Address 73 | An Internet Protocol address (IP address) serves two main functions: network interface identification and location addressing. 74 | IPv4 defines an IP address as a 32-bit number. However, because of the growth of the Internet and the depletion of available IPv4 addresses, a new version of IP i.e., IPv6, using 128 bits for the IP address, was standardized in 1998. 75 | 76 |
77 | 78 | ### NAT 79 | Network address translation (NAT) is a method of mapping an IP address space into another by modifying network address information in the IP header of packets while they are in transit across a traffic routing device. The technique was originally used to bypass the need to assign a new address to every host when a network was moved, or when the upstream Internet service provider was replaced, but could not route the network's address space. It has become a popular and essential tool in conserving global address space in the face of IPv4 address exhaustion. One Internet-routable IP address of a NAT gateway can be used for an entire private network. 80 | ![image](https://user-images.githubusercontent.com/20329508/194806584-3040cdf4-1551-4bad-a15f-a0fce34bd7f9.png) 81 | 82 | #### Routing 83 | Network address translation can be used to mitigate IP address overlap. Address overlap occurs when hosts in different networks with the same IP address space try to reach the same destination host. This is most often a misconfiguration and may result from the merger of two networks or subnets, especially when using RFC 1918 private network addressing. The destination host experiences traffic apparently arriving from the same network, and intermediate routers have no way to determine where reply traffic should be sent to. The solution is either renumbering to eliminate overlap or network address translation. 84 | 85 | #### Load balancing 86 | In client–server applications, load balancers forward client requests to a set of server computers to manage the workload of each server. Network address translation may be used to map a representative IP address of the server cluster to specific hosts that service the request. 87 | 88 |
89 | 90 | ### Virtual IP Addressing 91 | A virtual IP address (VIP or VIPA) is an IP address that does not correspond to a physical network interface. 92 | ![image](https://user-images.githubusercontent.com/20329508/194818785-855f2c07-9b17-4f8b-bb6b-dbddfa25fdb6.png) 93 | 94 | ![image](https://user-images.githubusercontent.com/20329508/194818964-2a08f081-ff42-40be-871d-df3374d1722b.png) 95 | 96 | Uses for VIPs include network address translation (especially, one-to-many NAT), fault-tolerance, and mobility. It advertises virtual links connected via itself to all of its actual network interfaces. 97 | 98 |
99 | 100 | ### iptables 101 | iptables allows a system administrator to configure the IP packet filter rules of the Linux kernel firewall. The filters are organized in different tables, which contain chains of rules for how to treat network traffic packets. Different kernel modules and programs are currently used for different protocols. On most Linux systems, iptables is installed as /usr/sbin/iptables and documented in its man pages. 102 | 103 | ![iptables](https://user-images.githubusercontent.com/20329508/194821305-30539c80-9d8a-4524-a211-b5f512bbfaf6.png) 104 | 105 | ![iptables1](https://user-images.githubusercontent.com/20329508/194821868-66ab88d5-1328-448d-8141-a3a4d8127a23.png) 106 | 107 | ![iptables3](https://user-images.githubusercontent.com/20329508/194823527-71fa7248-6102-42df-8269-d1c54f99b199.png) 108 | 109 |
110 | 111 | #### [iptables - I: Advance](https://youtu.be/NAdJojxENEU) 112 | #### [iptables - II: Advance](https://youtu.be/-CraNvj48J0) 113 |
114 | 115 | ### TCP & UDP 116 | TCP, which stands for Transmission Control Protocol, and UDP, or User Datagram Protocol, are part of the internet protocol suite, layer 4 (Transport). TCP and UDP are different methods to send information across the internet. 117 | 118 | #### TCP Pros 119 | - acknowledgement 120 | - guranteed delivery 121 | - connection based 122 | - congestion control 123 | - ordered packets 124 | 125 | #### TCP Cons 126 | - larger packets 127 | - more bandwidth 128 | - slower than UDP 129 | - stateful 130 | - server memory (DoS) 131 | 132 | #### UDP Pros 133 | - smaller package 134 | - less bandwidth 135 | - faster than TCP 136 | - stateless 137 | 138 | #### UDP Cons 139 | - no acknowledgement 140 | - no guranteed delivery 141 | - connectionless 142 | - no congestion control 143 | - no ordered packets 144 | - no security 145 | 146 | #### [TCP vs UDP](https://youtu.be/qqRYkcta6IE) 147 |
148 | 149 | ### TCP 3-way handshake 150 | #### Connection Establishment Process 151 | ![image](https://user-images.githubusercontent.com/20329508/194834935-a688a253-1bba-4a64-b0c6-e3dae31978b8.png) 152 | 153 | #### Closing Connection Process 154 | ![image](https://user-images.githubusercontent.com/20329508/194835123-402a45b0-182b-4c18-a5d8-aebc6e6569a5.png) 155 |
156 | 157 | ### When to use UDP vs TCP? 158 | When data integrity is your top priority, then TCP will always be the best choice. The protocol guarantees complete delivery and accurate reconstruction of the original data. When streaming video, however, accuracy is less important than continuity. This is why real-time applications like audio and video streaming will often use UDP. 159 |
160 | 161 | ### Stateless vs Stateful applications 162 | In a traditional stateful web application, the server is doing all of the work associated with recreating a web page. A stateless application doesn't save any client session (state) data on the server where the application lives. By using a Representational State Transfer (REST) API, developers can augment HTTP to produce stateless apps. 163 | 164 | When an application is stateful, it relies on saved client session data to process new transactions. A stateful app still uses a database for back-end storage, but it also uses the server where the application runs to store data from previous interactions. This allows the app to process subsequent transactions in the context of preceding ones. Stateful systems use databases like any application, but they also maintain "state data" (related to client authentication and past requests) on the server itself. Stateful apps are fast because they don't need to process as much data in each client request. This makes stateful apps fast and it allows clients to interact with the application within the historical context of previous interactions. 165 | 166 | Stateful apps bind clients and users to the same server so it can process subsequent requests in the context of previous ones. Stateful apps work best under predictable workloads that the system can manage. If traffic grows, you can't simply replicate a stateful app and redirect new client requests because users will need to start from scratch. This makes stateful apps more prone to system unavailability when client traffic increases. 167 | 168 | A stateless application or stateless process doesn't store any data related to past transactions on its server. It accepts each transaction or user interaction like a blank slate without knowledge of previous interactions. Similar to a Coke machine, the stateless app receives a single short-term request and delivers a single response. This is why many of the apps (clients) on your phone or computer have a cache. A stateless system acts like a nameless agent that clients use to interact with the databases it connects with. 169 | 170 | Since the application is stateless, a load balancer can infinitely replicate new instances of it and balance client requests across those instances. This allows the system to scale and manage any level of traffic. Similar to the security guard, DreamFactory waits for a client to submit an API request. 171 | 172 | The decision to use stateful versus stateless apps boils down to your scalability requirements and what you need the app to do. If your app needs to store session data to process transactions in-context and if the server can handle the expected processing load, a stateful system is probably best. On the other hand, if you are building an app that needs to process REST API transactions, provide information in response to client requests – and traffic levels can grow exponentially – a stateless app is what you’ll be working with. 173 |
174 | 175 | ### HTTP 176 | It is a aapplication layer protocol, functions as a request–response protocol in the client–server model. 177 | An example HTTP request: 178 | ![image](https://user-images.githubusercontent.com/20329508/195249102-e78ebeda-d220-4cfb-b067-b3398f2fe14e.png) 179 | 180 | An example response: 181 | ![image](https://user-images.githubusercontent.com/20329508/195249164-e5435ad7-f7b9-42f2-9fc0-9ae476edc19a.png) 182 | 183 | #### HTTP 1.0 over TCP 184 | - new TCP connection for each request 185 | - slow 186 | - buffering 187 | 188 | #### HTTP 1.1 over TCP 189 | ![image](https://user-images.githubusercontent.com/20329508/195250864-a6da3bdd-dc2c-4653-afe8-f75e5f73f0ec.png) 190 | 191 | - persisted TCP connection 192 | - low latency 193 | - streaming with chunked transfer 194 | - pipelining (disabled by default) 195 | 196 | ![image](https://user-images.githubusercontent.com/20329508/195251181-e52d6b39-a015-4c86-a72c-28816580db9f.png) 197 | 198 | #### HTTP 2 over TCP 199 | - compression 200 | - multiplexing 201 | - server push 202 | - SPDY (initial version of HTTP2 by google) 203 | - secure by default 204 | - protocol negotiation during TLS (NPN/ALPN) 205 | 206 | ![image](https://user-images.githubusercontent.com/20329508/195253974-3ee453d4-ec61-4b6e-b7cd-9d62422de732.png) 207 | 208 | 209 | ### Journey of an HTTP request 210 | 1. HTTP request (format as shown earlier) is created 211 | 2. Transport Layer 212 | - TCP handshake 213 | - If a message is too large to be sent in one go, it is broken down into several segments that each have their own sequence number (situated in a header field, as well). This way, when they arrive at their destination, they will be processed in the correct order. 214 | - The HTTP request is encapsulated in a TCP segment, which has a header and data payload. The Transport Layer is responsible for sending the segment from the requesting application to the correct process in the server. It uses the use of ports: integers in the range of 0–65535 that act as identifiers for a specific application or process in a host. 215 | ![image](https://user-images.githubusercontent.com/20329508/195340082-5b0574c4-4ded-42f0-9075-acce1eacdd74.png) 216 | 217 | 3. Internet Layer 218 | - Onwards, downwards, to the Internet Layer. This is where TCP segments are turned into IP packets by the Internet Protocol (IP) The Internet Protocol is the one that enables a packet to travel from your home to a server on the other side of the world. An IP address might look something like this: 207.126.144.100, for IPv4, or this: 2a02:687:1211:7500:c8cb:bcf9:7acb:a966,. for IPv6. 219 | ![image](https://user-images.githubusercontent.com/20329508/195340164-93fb2b43-8aba-4a03-a584-61b5914623e6.png) 220 | 221 | 4. Data Link Layer 222 | - Ethernet wraps the IP packet, containing the TCP segment, into a frame. The frame takes its final form: it becomes a stream of bits travelling as electrical signals (or light signals or radio waves) towards the destination MAC address’— the router that will act as a gateway to the outside world. It's much like how real packages travel across the world: each courier processes the package and dispatches it to the next courier until it reaches its final destination. 223 | ![image](https://user-images.githubusercontent.com/20329508/195340239-35deb5ad-df9d-4d65-a065-01d96823d12d.png) 224 | 225 | 226 | ### HTTP GET / through Switches, Routers, Gateways, and Proxies 227 | ![http](https://user-images.githubusercontent.com/20329508/195497332-21156ead-d5fa-4b7b-a87f-c3ac2ff8adee.png) 228 | 229 | ![httpExternal](https://user-images.githubusercontent.com/20329508/195497607-1191a764-f582-4e25-b225-55d4dcba4e53.png) 230 | 231 | ![httpProxy](https://user-images.githubusercontent.com/20329508/195497940-2f3c6175-4efc-4082-85ff-80e54fda20cd.png) 232 | 233 | ### cURL Verbose mode 234 | ``` 235 | curl -v https://www.stackoverflow.com 236 | * Trying 151.101.1.69:443... 237 | * TCP_NODELAY set 238 | * Connected to www.stackoverflow.com (151.101.1.69) port 443 (#0) 239 | * ALPN, offering h2 240 | * ALPN, offering http/1.1 241 | * successfully set certificate verify locations: 242 | * CAfile: /etc/ssl/certs/ca-certificates.crt 243 | CApath: /etc/ssl/certs 244 | * TLSv1.3 (OUT), TLS handshake, Client hello (1): 245 | * TLSv1.3 (IN), TLS handshake, Server hello (2): 246 | * TLSv1.2 (IN), TLS handshake, Certificate (11): 247 | * TLSv1.2 (IN), TLS handshake, Server key exchange (12): 248 | * TLSv1.2 (IN), TLS handshake, Server finished (14): 249 | * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): 250 | * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): 251 | * TLSv1.2 (OUT), TLS handshake, Finished (20): 252 | * TLSv1.2 (IN), TLS handshake, Finished (20): 253 | * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 254 | * ALPN, server accepted to use h2 255 | * Server certificate: 256 | * subject: CN=*.stackexchange.com 257 | * start date: Sep 4 13:06:50 2022 GMT 258 | * expire date: Dec 3 13:06:49 2022 GMT 259 | * subjectAltName: host "www.stackoverflow.com" matched cert's "*.stackoverflow.com" 260 | * issuer: C=US; O=Let's Encrypt; CN=R3 261 | * SSL certificate verify ok. 262 | * Using HTTP2, server supports multi-use 263 | * Connection state changed (HTTP/2 confirmed) 264 | * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 265 | * Using Stream ID: 1 (easy handle 0x5604558be8c0) 266 | > GET / HTTP/2 267 | > Host: www.stackoverflow.com 268 | > user-agent: curl/7.68.0 269 | > accept: */* 270 | > 271 | * Connection state changed (MAX_CONCURRENT_STREAMS == 100)! 272 | < HTTP/2 301 273 | < cache-control: private 274 | < content-type: text/html; charset=utf-8 275 | ``` 276 | ### HTTP Code 502 Bad Gateway 277 | The server was acting as a gateway or proxy and received an invalid response from the upstream server. 278 | 279 | The 502 error code is a by-product of the source or origin server being out of order. A range of connectivity issues, a server that is powered down, or spikes in traffic can all lead to this message. Communication issues between online servers or DNS issues such as incorrectly cached IP addresses play a big role in the appearance of this error. 280 | 281 | If your website is generating a 502 status code error, then try disabling the firewall or the CDN if you are behind one. Website themes and browser plugins can also cause the error as well. If disabling the plugins does not help then try updating your website theme. Most hosting providers have customer support teams that can triage the issue with you. 282 | 283 | Error codes use by cloudflare for more meaningfull error: 284 | 285 | ![image](https://user-images.githubusercontent.com/20329508/195806075-a44554cf-818f-4ce3-a823-4de79b7da2c7.png) 286 | 287 | ### HTTP CONNECT method 288 | #### HTTP Proxy 289 | - Client wants to connect to a server beyond its reach 290 | - Client doesn't know how to communicate target-server protocol 291 | - Proxy has access to the server and knows how to talk to it 292 | - Client connects the Proxy and Proxy makes the request on behalf of the client to the target server 293 | 294 | ![image](https://user-images.githubusercontent.com/20329508/196020889-6c103f21-4123-4dd9-a675-24a5f0705de4.png) 295 | 296 | #### HTTPS Proxy 297 | - In HTTPS Proxy, the proxy decrypts and serves its certificate to the client instead of that of the target server 298 | - Proxy sees everything(BAD, but good for debugging web traffic) 299 | - We need a way to establish End to End encryption between the client and the target server 300 | 301 | ![image](https://user-images.githubusercontent.com/20329508/196021143-d7ff3f3d-5915-4d1b-bae8-d22faf85b17a.png) 302 | 303 | #### HTTP CONNECT 304 | - Creates a tunnel between the client and the target server 305 | - Client sends HTTP CONNECT to Proxy containing target server 306 | - Proxy creates a TCP connection to target server 307 | - Once successful, proxy returns success to client 308 | - Any packet the client sends goes as-is to the target server 309 | - This includes TLS handshake, establishing e2e 310 | 311 | ![image](https://user-images.githubusercontent.com/20329508/196021388-83e0a293-8994-4df1-b6f0-a172385e4f58.png) 312 | 313 | ![image](https://user-images.githubusercontent.com/20329508/196021535-5e5f43e0-c837-471c-9ec6-a29d4cdba050.png) 314 | 315 | #### HTTP CONNECT Chaining 316 | ![image](https://user-images.githubusercontent.com/20329508/196021674-e7d66347-be0c-4361-abbf-8be64dfd4959.png) 317 | 318 | **Pros:** 319 | - Connect to secure servers 320 | - Support protocols that normally not supported through proxies (WebSockets, WebRTC) 321 | - Proxy can't read encrypted traffic 322 | - Chained Proxies CONNECT 323 | 324 | **Cons** 325 | - Only TCP, Can't proxy UDP traffic (won't work with QUIC/DNS) 326 | - Each CONNECT opens new TCP, expensive, (no multiplexing) 327 | - Bad implementation would allow tunneling to port (eg 25 SMTP can load to spam email) 328 | 329 | **MASQUE: Multiplexed Application Substrate over QUIC Encryption** 330 | ***Why MASQUE?*** 331 | - Allows UDP tunneling, Can't proxy UDP traffic with CONNECT (won't work with QUIC/DNS) 332 | - Allows multiplexing, each CONNECT opens new TCP to same host, expensive 333 | - Secure, bad CONNECT implementation would allow tunneling to port (eg 25 SMTP can load to spam email) 334 | 335 | ### HTTP/2 336 | #### HTTP 1.1 337 | - We can't utilize the same connection for parallel requests 338 | - Browsers use 6 connections, all those connections can be use for sending more request (to overcome the above issue) 339 | 340 | #### HTTP 2 341 | - Multiplexed streams, as we have identifier for streams, we can compress the header too 342 | 343 | ![image](https://user-images.githubusercontent.com/20329508/196023812-d739f08b-af66-460e-b7ca-7a9cc995f158.png) 344 | 345 | - Server push 346 | 347 | ![image](https://user-images.githubusercontent.com/20329508/196023824-b0f67378-231b-4da8-8ad3-9f88e20b1941.png) 348 | 349 | **Pros:** 350 | - Multiplexing over Single Connection 351 | - Compression (Header & Data) 352 | - Server Push 353 | - Secure by default 354 | - Protocol Negotiation during TLS (ALPN) 355 | 356 | **Cons:** 357 | - Server push can be abushed when configured incorrectly 358 | - Can be slower when in mixed mode (Backend is http/2 but load balancer is http 1 or vice versa) 359 | 360 | 361 | ### WebSockets 362 | ![image](https://user-images.githubusercontent.com/20329508/196024821-56760d3f-ef78-4fa2-b151-f6af84b6273a.png) 363 | 364 | ![image](https://user-images.githubusercontent.com/20329508/196024861-4655f16a-db9c-4bbf-b633-65b8517dc023.png) 365 | 366 | #### WebSockets usecases: 367 | - Chatting 368 | - Live Feed 369 | - Multiplayer gaming 370 | - Showing client progress/logging 371 | 372 | **Pros:** 373 | - Full-duplex (no polling) 374 | - HTTP compatible 375 | - Firewall friendly (standard) 376 | 377 | **Cons:** 378 | - Proxying is tricky 379 | - L7 Load Balance challenging (timeouts) 380 | - Stateful, difficult to horizontally scale 381 | 382 | **Do you have to use WebSockets?** 383 | - NO! Rule of thumb - do you absolutely need bidirectional communication? 384 | - Long polling 385 | - EventSource 386 | 387 | ### HTTP/2 limitation that leads to QUIC or HTTP/3 388 | HTTP/2 over TCP suffers from a slight inefficiency caused by TCP. Consider the following example: Suppose you have 3 streams A, B and C. Denote packets (frames) of each stream by lower case letters (a, b, c) and a sequence number. Let's have a look at what happens with HTTP/2 over TCP when the following sequence is sent: 389 | 390 | server ---> a2, c2, b2, *c1, b1, a1 ---> client 391 | 392 | Where the *c1 means that this frame was lost. The receiving end (client) must wait for a re-transmission of the lost *c1 frame before it can pass later frames to the application layer (namely b2,c2,a2), because the communication is over TCP and TCP guarantees in-order delivery! 393 | 394 | That is in contrast to HTTP/3 & QUIC, where over UDP these are just independent packets, thus the loss of *c1 would not delay the delivery of b2, c2 and a2 to the application layer! 395 | 396 | ### QUIC or HTTP/3 397 | 398 | 399 | ### gRPC over HTTP/3 400 | ### Should RabbitMQ implement QUIC Protocol for their Channels Message Queue? 401 | ### Can QUIC Protocol improve Database Performance in Web Applications? 402 | ### Facebook moves their backend and frontend to QUIC 403 | ### Symmetrical and Asymmetrical Encryption 404 | ### TLS 405 | ### TLS Handshake 406 | ### Web Servers 407 | ### Proxy and Reverse Proxy Server 408 | ### Anatomy of a Proxy Server 409 | ### Layer 4 vs Layer 7 proxying 410 | ### NginX 411 | ### NGINX Internal Architectur - Workers 412 | ### HAProxy 413 | ### Envoy Proxy 414 | ### Slack migrating millions of WebSockets from HAProxy to Envoy 415 | ### Load balancing Server-Sent Events 416 | ### Load balancing in Layer 4 vs Layer 7 417 | ### Load Balance multiple RTMP Servers to Horizontally Scale Streaming 418 | ### Can you Max-out the connections between load balancer and backend servers? 419 | ### Caching is hard 420 | ### Long Polling, how it deffers from push, pull and SSE 421 | ### First port your computer hits -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # The Backend Engineering Fundamentals 2 | 3 | ### OSI Model 4 | Open System Interconnection by International Organization for Standardization in 1984. It provides a framework for creating and implementing networking standards, devices, and internetworking schemes. 5 | | Group | Layer No. | Layer Name | Description | 6 | | ------ | --------- | ----------- | ----------- | 7 | | Top Layers | 7 | Application | Provide a user interface for sending and receiving data | 8 | | - | 6 | Presentation | Encrypt, format, and compress data for transmission | 9 | | - | 5 | Session | Initiate and terminate a session with the remote system | 10 | | Bottom Layers | 4 | Transport | Break the data stream into smaller segments and provide reliable and unreliable data delivery | 11 | | - | 3 | Network | Provide logical addressing | 12 | | - | 2 | Data Link | Prepare data for transmission | 13 | | - | 1 | Physical | Move data between devices | 14 | #### Application Layer 15 | The Top layer of the OSI model is the application layer. It provides the protocols and services that are required by the network-aware applications to connect to the network. FTP, TFTP, POP3, SMTP, and HTTP are examples of standards and protocols used in this layer. 16 | #### Presentation Layer 17 | Conversion, compression, and encryption are the main functions that the Presentation layer performs on the sending computer while on the receiving computer these functions are reconversion, decompression, and decryption. ASCII, BMP, GIF, JPEG, WAV, AVI, and MPEG are examples of standards and protocols that work in this layer. 18 | #### Session Layer 19 | The session layer is responsible for establishing, managing, and terminating communications between two computers. RPCs and NFS are examples of the session layer. 20 | #### Transport Layer 21 | The main functionalities of the Transport layer are segmentation, data transportation, and connection multiplexing. For data transportation, it uses TCP and UDP protocols. TCP is a connection-oriented protocol. It provides reliable data delivery. 22 | #### Network Layer 23 | Defining logical addresses and finding the best path to reach the destination address are the main functions of this layer. Routers work in this layer. Routing also takes place in this layer. IP, IPX, and AppleTalk are examples of this layer. 24 | #### Data Link Layer 25 | Defining physical addresses, finding hosts in the local network, specifying standards and methods to access the media are the primary functions of this layer. Switching takes place in this layer. Switches and Bridges work in this layer. HDLC, PPP, and Frame Relay are examples of this layer. 26 | 27 | This layer has two sub-layers: MAC(Media Access Control) and LLC(Logical Link Control) 28 | #### Physical Layer 29 | The Physical Layer mainly defines standards for media and devices that are used to move data across the network. 10BaseT, 10Base100, CSU/DSU, DCE, and DTE are examples of the standards used in this layer. 30 | 31 |
32 | 33 | ### TCP/IP 34 | TCP/IP stands for Transmission Control Protocol/ Internet Protocol. TCP/IP Stack is specifically designed as a model to offer highly reliable and end-to-end byte stream over an unreliable internetwork 35 | ![image](https://user-images.githubusercontent.com/20329508/194799658-2d5f3b36-1222-4266-b97b-78a1df08d832.png) 36 | 37 | ![image](https://user-images.githubusercontent.com/20329508/194799832-a3bf5a6e-8dde-4963-aedc-20811c4aecd1.png) 38 | 39 | 40 | TCP/IP model is developed by ARPANET (Advanced Research Project Agency Network). 41 | Some widely used most common TCP/IP protocol are: 42 | 43 | ![image](https://user-images.githubusercontent.com/20329508/194826313-8bb5c9da-f9ca-4641-a3b6-8897793ce443.png) 44 | 45 | #### TCP: 46 | Transmission Control Protocol is an internet protocol suite which breaks up the message into TCP Segments and reassembling them at the receiving side. 47 | 48 | #### IP: 49 | An Internet Protocol address that is also known as an IP address is a numerical label. It is assigned to each device that is connected to a computer network which uses the IP for communication. Its routing function allows internetworking and essentially establishes the Internet. Combination of IP with a TCP allows developing a virtual connection between a destination and a source. 50 | 51 | #### HTTP: 52 | The Hypertext Transfer Protocol is a foundation of the World Wide Web. It is used for transferring webpages and other such resources from the HTTP server or web server to the web client or the HTTP client. Whenever you use a web browser like Google Chrome or Firefox, you are using a web client. It helps HTTP to transfer web pages that you request from the remote servers. 53 | 54 | #### SMTP: 55 | SMTP stands for Simple mail transfer protocol. This protocol supports the e-mail is known as a simple mail transfer protocol. This protocol helps you to send the data to another e-mail address. 56 | 57 | #### SNMP: 58 | SNMP stands for Simple Network Management Protocol. It is a framework which is used for managing the devices on the internet by using the TCP/IP protocol. 59 | 60 | #### DNS: 61 | DNS stands for Domain Name System. An IP address that is used to identify the connection of a host to the internet uniquely. However, users prefer to use names instead of addresses for that DNS. 62 | 63 | #### TELNET: 64 | TELNET stands for Terminal Network. It establishes the connection between the local and remote computer. It established connection in such a manner that you can simulate your local system at the remote system. 65 | 66 | #### FTP: 67 | FTP stands for File Transfer Protocol. It is a mostly used standard protocol for transmitting the files from one machine to another. 68 |
69 | 70 | #### [How OSI and TCP/IP work?](https://youtu.be/3kfO61Mensg) 71 | 72 |
73 | 74 | ### IP Address 75 | An Internet Protocol address (IP address) serves two main functions: network interface identification and location addressing. 76 | IPv4 defines an IP address as a 32-bit number. However, because of the growth of the Internet and the depletion of available IPv4 addresses, a new version of IP i.e., IPv6, using 128 bits for the IP address, was standardized in 1998. 77 | 78 |
79 | 80 | ### NAT 81 | Network address translation (NAT) is a method of mapping an IP address space into another by modifying network address information in the IP header of packets while they are in transit across a traffic routing device. The technique was originally used to bypass the need to assign a new address to every host when a network was moved, or when the upstream Internet service provider was replaced, but could not route the network's address space. It has become a popular and essential tool in conserving global address space in the face of IPv4 address exhaustion. One Internet-routable IP address of a NAT gateway can be used for an entire private network. 82 | ![image](https://user-images.githubusercontent.com/20329508/194806584-3040cdf4-1551-4bad-a15f-a0fce34bd7f9.png) 83 | 84 | #### Routing 85 | Network address translation can be used to mitigate IP address overlap. Address overlap occurs when hosts in different networks with the same IP address space try to reach the same destination host. This is most often a misconfiguration and may result from the merger of two networks or subnets, especially when using RFC 1918 private network addressing. The destination host experiences traffic apparently arriving from the same network, and intermediate routers have no way to determine where reply traffic should be sent to. The solution is either renumbering to eliminate overlap or network address translation. 86 | 87 | #### Load balancing 88 | In client–server applications, load balancers forward client requests to a set of server computers to manage the workload of each server. Network address translation may be used to map a representative IP address of the server cluster to specific hosts that service the request. 89 | 90 |
91 | 92 | ### Virtual IP Addressing 93 | A virtual IP address (VIP or VIPA) is an IP address that does not correspond to a physical network interface. 94 | ![image](https://user-images.githubusercontent.com/20329508/194818785-855f2c07-9b17-4f8b-bb6b-dbddfa25fdb6.png) 95 | 96 | ![image](https://user-images.githubusercontent.com/20329508/194818964-2a08f081-ff42-40be-871d-df3374d1722b.png) 97 | 98 | Uses for VIPs include network address translation (especially, one-to-many NAT), fault-tolerance, and mobility. It advertises virtual links connected via itself to all of its actual network interfaces. 99 | 100 |
101 | 102 | ### iptables 103 | iptables allows a system administrator to configure the IP packet filter rules of the Linux kernel firewall. The filters are organized in different tables, which contain chains of rules for how to treat network traffic packets. Different kernel modules and programs are currently used for different protocols. On most Linux systems, iptables is installed as /usr/sbin/iptables and documented in its man pages. 104 | 105 | ![iptables](https://user-images.githubusercontent.com/20329508/194821305-30539c80-9d8a-4524-a211-b5f512bbfaf6.png) 106 | 107 | ![iptables1](https://user-images.githubusercontent.com/20329508/194821868-66ab88d5-1328-448d-8141-a3a4d8127a23.png) 108 | 109 | ![iptables3](https://user-images.githubusercontent.com/20329508/194823527-71fa7248-6102-42df-8269-d1c54f99b199.png) 110 | 111 |
112 | 113 | #### [iptables - I: Advance](https://youtu.be/NAdJojxENEU) 114 | #### [iptables - II: Advance](https://youtu.be/-CraNvj48J0) 115 |
116 | 117 | ### TCP & UDP 118 | TCP, which stands for Transmission Control Protocol, and UDP, or User Datagram Protocol, are part of the internet protocol suite, layer 4 (Transport). TCP and UDP are different methods to send information across the internet. 119 | 120 | #### TCP Pros 121 | - acknowledgement 122 | - guranteed delivery 123 | - connection based 124 | - congestion control 125 | - ordered packets 126 | 127 | #### TCP Cons 128 | - larger packets 129 | - more bandwidth 130 | - slower than UDP 131 | - stateful 132 | - server memory (DoS) 133 | 134 | #### UDP Pros 135 | - smaller package 136 | - less bandwidth 137 | - faster than TCP 138 | - stateless 139 | 140 | #### UDP Cons 141 | - no acknowledgement 142 | - no guranteed delivery 143 | - connectionless 144 | - no congestion control 145 | - no ordered packets 146 | - no security 147 | 148 | #### [TCP vs UDP](https://youtu.be/qqRYkcta6IE) 149 |
150 | 151 | ### TCP 3-way handshake 152 | #### Connection Establishment Process 153 | ![image](https://user-images.githubusercontent.com/20329508/194834935-a688a253-1bba-4a64-b0c6-e3dae31978b8.png) 154 | 155 | #### Closing Connection Process 156 | ![image](https://user-images.githubusercontent.com/20329508/194835123-402a45b0-182b-4c18-a5d8-aebc6e6569a5.png) 157 |
158 | 159 | ### When to use UDP vs TCP? 160 | When data integrity is your top priority, then TCP will always be the best choice. The protocol guarantees complete delivery and accurate reconstruction of the original data. When streaming video, however, accuracy is less important than continuity. This is why real-time applications like audio and video streaming will often use UDP. 161 |
162 | 163 | ### Stateless vs Stateful applications 164 | In a traditional stateful web application, the server is doing all of the work associated with recreating a web page. A stateless application doesn't save any client session (state) data on the server where the application lives. By using a Representational State Transfer (REST) API, developers can augment HTTP to produce stateless apps. 165 | 166 | When an application is stateful, it relies on saved client session data to process new transactions. A stateful app still uses a database for back-end storage, but it also uses the server where the application runs to store data from previous interactions. This allows the app to process subsequent transactions in the context of preceding ones. Stateful systems use databases like any application, but they also maintain "state data" (related to client authentication and past requests) on the server itself. Stateful apps are fast because they don't need to process as much data in each client request. This makes stateful apps fast and it allows clients to interact with the application within the historical context of previous interactions. 167 | 168 | Stateful apps bind clients and users to the same server so it can process subsequent requests in the context of previous ones. Stateful apps work best under predictable workloads that the system can manage. If traffic grows, you can't simply replicate a stateful app and redirect new client requests because users will need to start from scratch. This makes stateful apps more prone to system unavailability when client traffic increases. 169 | 170 | A stateless application or stateless process doesn't store any data related to past transactions on its server. It accepts each transaction or user interaction like a blank slate without knowledge of previous interactions. Similar to a Coke machine, the stateless app receives a single short-term request and delivers a single response. This is why many of the apps (clients) on your phone or computer have a cache. A stateless system acts like a nameless agent that clients use to interact with the databases it connects with. 171 | 172 | Since the application is stateless, a load balancer can infinitely replicate new instances of it and balance client requests across those instances. This allows the system to scale and manage any level of traffic. Similar to the security guard, DreamFactory waits for a client to submit an API request. 173 | 174 | The decision to use stateful versus stateless apps boils down to your scalability requirements and what you need the app to do. If your app needs to store session data to process transactions in-context and if the server can handle the expected processing load, a stateful system is probably best. On the other hand, if you are building an app that needs to process REST API transactions, provide information in response to client requests – and traffic levels can grow exponentially – a stateless app is what you’ll be working with. 175 |
176 | 177 | ### HTTP 178 | It is a aapplication layer protocol, functions as a request–response protocol in the client–server model. 179 | An example HTTP request: 180 | ![image](https://user-images.githubusercontent.com/20329508/195249102-e78ebeda-d220-4cfb-b067-b3398f2fe14e.png) 181 | 182 | An example response: 183 | ![image](https://user-images.githubusercontent.com/20329508/195249164-e5435ad7-f7b9-42f2-9fc0-9ae476edc19a.png) 184 | 185 | #### HTTP 1.0 over TCP 186 | - new TCP connection for each request 187 | - slow 188 | - buffering 189 | 190 | #### HTTP 1.1 over TCP 191 | ![image](https://user-images.githubusercontent.com/20329508/195250864-a6da3bdd-dc2c-4653-afe8-f75e5f73f0ec.png) 192 | 193 | - persisted TCP connection 194 | - low latency 195 | - streaming with chunked transfer 196 | - pipelining (disabled by default) 197 | 198 | ![image](https://user-images.githubusercontent.com/20329508/195251181-e52d6b39-a015-4c86-a72c-28816580db9f.png) 199 | 200 | #### HTTP 2 over TCP 201 | - compression 202 | - multiplexing 203 | - server push 204 | - SPDY (initial version of HTTP2 by google) 205 | - secure by default 206 | - protocol negotiation during TLS (NPN/ALPN) 207 | 208 | ![image](https://user-images.githubusercontent.com/20329508/195253974-3ee453d4-ec61-4b6e-b7cd-9d62422de732.png) 209 | 210 | 211 | ### Journey of an HTTP request 212 | 1. HTTP request (format as shown earlier) is created 213 | 2. Transport Layer 214 | - TCP handshake 215 | - If a message is too large to be sent in one go, it is broken down into several segments that each have their own sequence number (situated in a header field, as well). This way, when they arrive at their destination, they will be processed in the correct order. 216 | - The HTTP request is encapsulated in a TCP segment, which has a header and data payload. The Transport Layer is responsible for sending the segment from the requesting application to the correct process in the server. It uses the use of ports: integers in the range of 0–65535 that act as identifiers for a specific application or process in a host. 217 | ![image](https://user-images.githubusercontent.com/20329508/195340082-5b0574c4-4ded-42f0-9075-acce1eacdd74.png) 218 | 219 | 3. Internet Layer 220 | - Onwards, downwards, to the Internet Layer. This is where TCP segments are turned into IP packets by the Internet Protocol (IP) The Internet Protocol is the one that enables a packet to travel from your home to a server on the other side of the world. An IP address might look something like this: 207.126.144.100, for IPv4, or this: 2a02:687:1211:7500:c8cb:bcf9:7acb:a966,. for IPv6. 221 | ![image](https://user-images.githubusercontent.com/20329508/195340164-93fb2b43-8aba-4a03-a584-61b5914623e6.png) 222 | 223 | 4. Data Link Layer 224 | - Ethernet wraps the IP packet, containing the TCP segment, into a frame. The frame takes its final form: it becomes a stream of bits travelling as electrical signals (or light signals or radio waves) towards the destination MAC address’— the router that will act as a gateway to the outside world. It's much like how real packages travel across the world: each courier processes the package and dispatches it to the next courier until it reaches its final destination. 225 | ![image](https://user-images.githubusercontent.com/20329508/195340239-35deb5ad-df9d-4d65-a065-01d96823d12d.png) 226 | 227 | 228 | ### HTTP GET / through Switches, Routers, Gateways, and Proxies 229 | ![http](https://user-images.githubusercontent.com/20329508/195497332-21156ead-d5fa-4b7b-a87f-c3ac2ff8adee.png) 230 | 231 | ![httpExternal](https://user-images.githubusercontent.com/20329508/195497607-1191a764-f582-4e25-b225-55d4dcba4e53.png) 232 | 233 | ![httpProxy](https://user-images.githubusercontent.com/20329508/195497940-2f3c6175-4efc-4082-85ff-80e54fda20cd.png) 234 | 235 | ### cURL Verbose mode 236 | ``` 237 | curl -v https://www.stackoverflow.com 238 | * Trying 151.101.1.69:443... 239 | * TCP_NODELAY set 240 | * Connected to www.stackoverflow.com (151.101.1.69) port 443 (#0) 241 | * ALPN, offering h2 242 | * ALPN, offering http/1.1 243 | * successfully set certificate verify locations: 244 | * CAfile: /etc/ssl/certs/ca-certificates.crt 245 | CApath: /etc/ssl/certs 246 | * TLSv1.3 (OUT), TLS handshake, Client hello (1): 247 | * TLSv1.3 (IN), TLS handshake, Server hello (2): 248 | * TLSv1.2 (IN), TLS handshake, Certificate (11): 249 | * TLSv1.2 (IN), TLS handshake, Server key exchange (12): 250 | * TLSv1.2 (IN), TLS handshake, Server finished (14): 251 | * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): 252 | * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): 253 | * TLSv1.2 (OUT), TLS handshake, Finished (20): 254 | * TLSv1.2 (IN), TLS handshake, Finished (20): 255 | * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 256 | * ALPN, server accepted to use h2 257 | * Server certificate: 258 | * subject: CN=*.stackexchange.com 259 | * start date: Sep 4 13:06:50 2022 GMT 260 | * expire date: Dec 3 13:06:49 2022 GMT 261 | * subjectAltName: host "www.stackoverflow.com" matched cert's "*.stackoverflow.com" 262 | * issuer: C=US; O=Let's Encrypt; CN=R3 263 | * SSL certificate verify ok. 264 | * Using HTTP2, server supports multi-use 265 | * Connection state changed (HTTP/2 confirmed) 266 | * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 267 | * Using Stream ID: 1 (easy handle 0x5604558be8c0) 268 | > GET / HTTP/2 269 | > Host: www.stackoverflow.com 270 | > user-agent: curl/7.68.0 271 | > accept: */* 272 | > 273 | * Connection state changed (MAX_CONCURRENT_STREAMS == 100)! 274 | < HTTP/2 301 275 | < cache-control: private 276 | < content-type: text/html; charset=utf-8 277 | ``` 278 | ### HTTP Code 502 Bad Gateway 279 | The server was acting as a gateway or proxy and received an invalid response from the upstream server. 280 | 281 | The 502 error code is a by-product of the source or origin server being out of order. A range of connectivity issues, a server that is powered down, or spikes in traffic can all lead to this message. Communication issues between online servers or DNS issues such as incorrectly cached IP addresses play a big role in the appearance of this error. 282 | 283 | If your website is generating a 502 status code error, then try disabling the firewall or the CDN if you are behind one. Website themes and browser plugins can also cause the error as well. If disabling the plugins does not help then try updating your website theme. Most hosting providers have customer support teams that can triage the issue with you. 284 | 285 | Error codes use by cloudflare for more meaningfull error: 286 | 287 | ![image](https://user-images.githubusercontent.com/20329508/195806075-a44554cf-818f-4ce3-a823-4de79b7da2c7.png) 288 | 289 | ### HTTP CONNECT method 290 | #### HTTP Proxy 291 | - Client wants to connect to a server beyond its reach 292 | - Client doesn't know how to communicate target-server protocol 293 | - Proxy has access to the server and knows how to talk to it 294 | - Client connects the Proxy and Proxy makes the request on behalf of the client to the target server 295 | 296 | ![image](https://user-images.githubusercontent.com/20329508/196020889-6c103f21-4123-4dd9-a675-24a5f0705de4.png) 297 | 298 | #### HTTPS Proxy 299 | - In HTTPS Proxy, the proxy decrypts and serves its certificate to the client instead of that of the target server 300 | - Proxy sees everything(BAD, but good for debugging web traffic) 301 | - We need a way to establish End to End encryption between the client and the target server 302 | 303 | ![image](https://user-images.githubusercontent.com/20329508/196021143-d7ff3f3d-5915-4d1b-bae8-d22faf85b17a.png) 304 | 305 | #### HTTP CONNECT 306 | - Creates a tunnel between the client and the target server 307 | - Client sends HTTP CONNECT to Proxy containing target server 308 | - Proxy creates a TCP connection to target server 309 | - Once successful, proxy returns success to client 310 | - Any packet the client sends goes as-is to the target server 311 | - This includes TLS handshake, establishing e2e 312 | 313 | ![image](https://user-images.githubusercontent.com/20329508/196021388-83e0a293-8994-4df1-b6f0-a172385e4f58.png) 314 | 315 | ![image](https://user-images.githubusercontent.com/20329508/196021535-5e5f43e0-c837-471c-9ec6-a29d4cdba050.png) 316 | 317 | #### HTTP CONNECT Chaining 318 | ![image](https://user-images.githubusercontent.com/20329508/196021674-e7d66347-be0c-4361-abbf-8be64dfd4959.png) 319 | 320 | **Pros:** 321 | - Connect to secure servers 322 | - Support protocols that normally not supported through proxies (WebSockets, WebRTC) 323 | - Proxy can't read encrypted traffic 324 | - Chained Proxies CONNECT 325 | 326 | **Cons** 327 | - Only TCP, Can't proxy UDP traffic (won't work with QUIC/DNS) 328 | - Each CONNECT opens new TCP, expensive, (no multiplexing) 329 | - Bad implementation would allow tunneling to port (eg 25 SMTP can load to spam email) 330 | 331 | **MASQUE: Multiplexed Application Substrate over QUIC Encryption** 332 | ***Why MASQUE?*** 333 | - Allows UDP tunneling, Can't proxy UDP traffic with CONNECT (won't work with QUIC/DNS) 334 | - Allows multiplexing, each CONNECT opens new TCP to same host, expensive 335 | - Secure, bad CONNECT implementation would allow tunneling to port (eg 25 SMTP can load to spam email) 336 | 337 | ### HTTP/2 338 | #### HTTP 1.1 339 | - We can't utilize the same connection for parallel requests 340 | - Browsers use 6 connections, all those connections can be use for sending more request (to overcome the above issue) 341 | 342 | #### HTTP 2 343 | - Multiplexed streams, as we have identifier for streams, we can compress the header too 344 | 345 | ![image](https://user-images.githubusercontent.com/20329508/196023812-d739f08b-af66-460e-b7ca-7a9cc995f158.png) 346 | 347 | - Server push 348 | 349 | ![image](https://user-images.githubusercontent.com/20329508/196023824-b0f67378-231b-4da8-8ad3-9f88e20b1941.png) 350 | 351 | **Pros:** 352 | - Multiplexing over Single Connection 353 | - Compression (Header & Data) 354 | - Server Push 355 | - Secure by default 356 | - Protocol Negotiation during TLS (ALPN) 357 | 358 | **Cons:** 359 | - Server push can be abushed when configured incorrectly 360 | - Can be slower when in mixed mode (Backend is http/2 but load balancer is http 1 or vice versa) 361 | 362 | 363 | ### WebSockets 364 | ![image](https://user-images.githubusercontent.com/20329508/196024821-56760d3f-ef78-4fa2-b151-f6af84b6273a.png) 365 | 366 | ![image](https://user-images.githubusercontent.com/20329508/196024861-4655f16a-db9c-4bbf-b633-65b8517dc023.png) 367 | 368 | #### WebSockets usecases: 369 | - Chatting 370 | - Live Feed 371 | - Multiplayer gaming 372 | - Showing client progress/logging 373 | 374 | **Pros:** 375 | - Full-duplex (no polling) 376 | - HTTP compatible 377 | - Firewall friendly (standard) 378 | 379 | **Cons:** 380 | - Proxying is tricky 381 | - L7 Load Balance challenging (timeouts) 382 | - Stateful, difficult to horizontally scale 383 | 384 | **Do you have to use WebSockets?** 385 | - NO! Rule of thumb - do you absolutely need bidirectional communication? 386 | - Long polling 387 | - EventSource 388 | 389 | ### HTTP/2 limitation that leads to QUIC or HTTP/3 390 | HTTP/2 over TCP suffers from a slight inefficiency caused by TCP. Consider the following example: Suppose you have 3 streams A, B and C. Denote packets (frames) of each stream by lower case letters (a, b, c) and a sequence number. Let's have a look at what happens with HTTP/2 over TCP when the following sequence is sent: 391 | 392 | server ---> a2, c2, b2, *c1, b1, a1 ---> client 393 | 394 | Where the *c1 means that this frame was lost. The receiving end (client) must wait for a re-transmission of the lost *c1 frame before it can pass later frames to the application layer (namely b2,c2,a2), because the communication is over TCP and TCP guarantees in-order delivery! 395 | 396 | That is in contrast to HTTP/3 & QUIC, where over UDP these are just independent packets, thus the loss of *c1 would not delay the delivery of b2, c2 and a2 to the application layer! 397 | 398 | ### QUIC or HTTP/3 399 | 400 | 401 | ### gRPC over HTTP/3 402 | ### Should RabbitMQ implement QUIC Protocol for their Channels Message Queue? 403 | ### Can QUIC Protocol improve Database Performance in Web Applications? 404 | ### Facebook moves their backend and frontend to QUIC 405 | ### Symmetrical and Asymmetrical Encryption 406 | ### TLS 407 | ### TLS Handshake 408 | ### Web Servers 409 | ### Proxy and Reverse Proxy Server 410 | ### Anatomy of a Proxy Server 411 | ### Layer 4 vs Layer 7 proxying 412 | ### NginX 413 | ### NGINX Internal Architectur - Workers 414 | ### HAProxy 415 | ### Envoy Proxy 416 | ### Slack migrating millions of WebSockets from HAProxy to Envoy 417 | ### Load balancing Server-Sent Events 418 | ### Load balancing in Layer 4 vs Layer 7 419 | ### Load Balance multiple RTMP Servers to Horizontally Scale Streaming 420 | ### Can you Max-out the connections between load balancer and backend servers? 421 | ### Caching is hard 422 | ### Long Polling, how it deffers from push, pull and SSE 423 | ### First port your computer hits 424 | ### ACID Transactions - Relational Database 425 | ### Primary Key and Secondary key 426 | ### B Tree and B+ Tree 427 | ### Database Indexing 428 | ### Leaking Postgres database connections 429 | ### Database Engines 430 | ### Fail-over and High-Availability 431 | ### Active-Active vs Active-Passive Cluster to achieve High Availability 432 | ### Connection Pooling in PostgresSQL 433 | ### Horizontal vs Vertical Database Partitioning 434 | ### Database Partitioning 435 | ### Database Sharding 436 | ### Can you get Eventual Consistency in Relational Database? 437 | ### Avoid Double Booking and Race Conditions 438 | ### 439 | 440 | --------------------------------------------------------------------------------