├── CN Module ├── Hands-on Project: _Building _a _Simple _Network.md ├── Introduction _to _Computer_ Networks.md ├── Network _Monitoring _& _Automation.md ├── Network _Security _and _Troubleshooting.md ├── Network_ Devices _and _Hardware.md └── Network_ Protocols _and _Communication.md ├── DBMS Module ├── Advanced _Indexing _and _Transactions.md ├── CRUD _Operations _in _SQL - Part 1.md ├── CRUD_ Operations _in _SQL - Part 2.md ├── High_ Availability _and _Replication.md ├── Introduction _to _DBMS _and SQL _Keys.md ├── Performance_ and _Database _Security.md ├── SQL _Administration.md ├── SQL_ Optimization _and _Indexing.md ├── Scaling,_ Networking, _and _Automation.md ├── Understanding _Joins _in _SQL.md └── check.md ├── DevOps Tools 1 Module ├── Deployment in Kubernetes - Kustomize.md ├── Docker Basics (Continued) & Docker Swarm.md ├── Docker Basics.md ├── Docker Container NeDocker Container Networking And Security Conttworking And Security Cont.md ├── Docker Container Networking and Security (Cont).md ├── Docker Container Networking and Security.md ├── Docker Orchestration & Container Storage.md ├── Docker and Docker Swarm.md ├── Docker swarm (Cont).md ├── Git Advance and Yaml Introduction.md ├── Introduction to Devops & Git Foundation.md ├── Kubernetes - Helm.md ├── Kubernetes API Versioning, K8s Extension , Certification Tip.md ├── Kubernetes Basics.md ├── Kubernetes Configuration and Advance Pod Concepts.md ├── Kubernetes Core Concepts.md ├── Kubernetes Jobs & Networking.md ├── Kubernetes Observability and Pod Design cont.md ├── Kubernetes Observability and Pod Design.md ├── Kubernetes Security.md └── Kubernetes State Persistence.md ├── DevOps Tools 2 Module ├── Advance Jenkins Concepts.md ├── Advanced ArgoCD Usage and Configuration.md ├── Advanced Features and Integration.md ├── Ansible Fundamentals.md ├── Ansible Playbooks & Modules Notes.md ├── Ansible Roles & Advanced Topics.md ├── Ansible Vault & Best Practices.md ├── Argocd-jenkins Integration.md ├── Efficient Workflows and Security Practices.md ├── Github Actions - Advanced Techniques and Best Practices ├── Introduction to CI CD & Jenkins.md ├── Introduction to Chef.md ├── Introduction to GitHub Actions Notes ├── Introduction to GitOps and ArgoCD Setup.md ├── Introduction to Puppet.md ├── Jenkins Miscellaneous ├── Jenkins, Docker, and Kubernetes Integration.md ├── Mastering Jenkins Pipelines and Administration.md ├── Secrets, Events, and Workflow Optimization.md └── Templates, Variables & Facts.md └── Linux Module ├── "Text _processing _in _Linux _Scripting .md ├── Advanced _Process _Management.md ├── Advanced _Shell _Scripting_ Techniques.md ├── Concurrency_and_Beyond.md ├── Design_Shell_ Scripts.md ├── Introduction _to _Operating _Systems _(Continued).md ├── Introduction _to_ Shell_ Scripting.md ├── Introduction _to_Concurrency.md ├── Introduction_to_Operating_Systems.md ├── Memory _Storage _Management ├── Real_World _Shell_ Scripting _Projects.md ├── System Access Management.md └── System _Resource _Monitoring.md /CN Module/Hands-on Project: _Building _a _Simple _Network.md: -------------------------------------------------------------------------------- 1 | ## Hands-on Project: Building a Simple Network 2 | 3 | 4 | ### Agenda of the Lecture 5 | 6 | 1. **Load Balancing** 7 | 2. **Blocking Client Manually** 8 | 3. **Rate Limiting** 9 | 10 | 11 | ### Load Balancing 12 | 13 | #### Overview 14 | 15 | - **server1**: Load balancer (Nginx) 16 | - **server2**, **server3**: Web servers 17 | - **server4**: Client or local machine 18 | 19 | **Objective**: Set up **Nginx** as a load balancer on **server1** to distribute traffic evenly between **server2** and **server3**. 20 | 21 | #### Part 1: Load Balancer Setup 22 | 23 | 1. The client (server4) sends requests to **server1**. 24 | 2. **Server1** (Nginx load balancer) distributes traffic to **server2** and **server3**. 25 | 3. Nginx's default method is round-robin, sending requests alternately between servers. 26 | 27 | ##### Server1: Nginx Load Balancer Configuration 28 | 29 | 1. **Update the system**: 30 | ```bash 31 | sudo apt update -y 32 | ``` 33 | 2. **Install Nginx**: 34 | ```bash 35 | sudo apt install nginx -y 36 | ``` 37 | 3. **Start Nginx**: 38 | ```bash 39 | sudo systemctl start nginx 40 | ``` 41 | 4. **Check Nginx Status**: 42 | ```bash 43 | sudo systemctl status nginx 44 | ``` 45 | 5. **Configure Nginx for Load Balancing**: 46 | - Open the Nginx configuration: 47 | ```bash 48 | sudo vi /etc/nginx/conf.d/load_balancing.conf 49 | ``` 50 | - Add the following configuration: 51 | ```nginx 52 | upstream backend { 53 | server 192.168.1.2; # Server 2 54 | server 192.168.1.3; # Server 3 55 | } 56 | 57 | server { 58 | listen 80; 59 | location / { 60 | proxy_pass http://backend; 61 | } 62 | } 63 | ``` 64 | - Replace IPs with actual server2 and server3 IPs. 65 | 66 | 6. **Reload Nginx** to apply the changes: 67 | ```bash 68 | sudo systemctl reload nginx 69 | ``` 70 | 7. **Test connection** by sending a request: 71 | ```bash 72 | curl http:// 73 | ``` 74 | 75 | ##### Server2: Web Server Setup (Nginx) 76 | 77 | 1. **Install Nginx**: 78 | ```bash 79 | sudo apt install nginx -y 80 | ``` 81 | 2. **Create an HTML response**: 82 | ```bash 83 | echo "Welcome to Server 2" | sudo tee /var/www/html/index.html 84 | ``` 85 | 3. **Restart Nginx**: 86 | ```bash 87 | sudo systemctl restart nginx 88 | ``` 89 | 90 | ##### Server3: Web Server Setup (Apache) 91 | 92 | 1. **Install Apache**: 93 | ```bash 94 | sudo apt install apache2 -y 95 | ``` 96 | 2. **Enable Apache to start on boot**: 97 | ```bash 98 | sudo systemctl enable apache2 99 | ``` 100 | 3. **Create an HTML response**: 101 | ```bash 102 | echo "

Hello from Server 3 $(hostname)

" | sudo tee /var/www/html/index.html 103 | ``` 104 | 105 | 106 | ### Blocking Client Manually 107 | 108 | #### Part 2: Blocking a Client 109 | 110 | 1. **Check access logs** on **server2**: 111 | ```bash 112 | vi /var/log/nginx/access.log 113 | ``` 114 | 2. **Identify suspicious requests**: 115 | - Open the **load balancer logs** on **server1** to find the origin IP. 116 | 3. **Manually block an IP**: 117 | ```bash 118 | sudo iptables -A INPUT -s 122.162.148.247 -p tcp --dport 80 -j DROP 119 | ``` 120 | - This command blocks incoming requests from the specified IP on port 80. 121 | 4. **View blocked IPs**: 122 | ```bash 123 | sudo iptables -L 124 | ``` 125 | 126 | ##### Automating Client Blocking 127 | 128 | 1. **Extract unique IP addresses**: 129 | ```bash 130 | awk '{print $1}' /var/log/nginx/access.log | uniq 131 | ``` 132 | 2. **Block each IP using iptables**: 133 | ```bash 134 | awk '{print $1}' /var/log/nginx/access.log | uniq | xargs -I {} sudo iptables -A INPUT -s {} -p tcp --dport 80 -j DROP 135 | ``` 136 | 137 | 3. **Unblock an IP**: 138 | ```bash 139 | sudo iptables -D INPUT -s 122.162.148.247 -p tcp --dport 80 -j DROP 140 | ``` 141 | 4. **Flush all IP tables** (use cautiously): 142 | ```bash 143 | sudo iptables -F 144 | ``` 145 | 146 | 147 | ### Rate Limiting 148 | 149 | #### Part 3: Implementing Rate Limiting 150 | 151 | 1. **Use Nginx for rate limiting**. 152 | 2. **Open the Nginx configuration file**: 153 | ```bash 154 | sudo vi /etc/nginx/nginx.conf 155 | ``` 156 | 157 | ##### Nginx Rate Limiting Configuration 158 | 159 | 1. **Add rate limiting rules** inside the HTTP block: 160 | ```nginx 161 | limit_req_zone $binary_remote_addr zone=mylimit:10m rate=5r/m; 162 | ``` 163 | - This limits 5 requests per minute from the same IP. 164 | 165 | 2. **Apply the rate limiting rule** inside the server block: 166 | ```nginx 167 | limit_req zone=mylimit burst=5 nodelay; 168 | ``` 169 | 170 | 3. **Reload Nginx** to apply the changes: 171 | ```bash 172 | sudo systemctl reload nginx 173 | ``` 174 | 175 | 4. **Test the rate limiting** by sending multiple requests: 176 | ```bash 177 | curl http:// 178 | ``` 179 | 180 | 181 | 182 | ### Why Rate Limiting? 183 | 184 | - **Automates traffic control** to prevent overload. 185 | - Tools like **Nginx** simplify implementation, making it scalable across environments. 186 | 187 | -------------------------------------------------------------------------------- /DBMS Module/Advanced _Indexing _and _Transactions.md: -------------------------------------------------------------------------------- 1 | # Advanced Indexing and Transactions 2 | 3 | 4 | ## Agenda of the Lecture 5 | 6 | - **Topics to Cover**: 7 | 1. Covering Index 8 | 2. Composite Index 9 | 3. Transactions 10 | 4. Isolation Levels 11 | 5. Live Demo 12 | 13 | ## Covering Index 14 | 15 | ### Understanding Covering Index 16 | 17 | - **Definition**: A covering index in MySQL includes all columns needed for a query, allowing data retrieval directly from the index without accessing the main table. 18 | - **Purpose**: Enhances performance by reducing table traversals and I/O operations. 19 | 20 | ### Benefits of Using a Covering Index 21 | 22 | - **Reduced Table Access**: Queries retrieve results using only the index, lessening the load on the main table. 23 | - **Faster Query Performance**: Optimized steps lead to quicker execution times. 24 | - **Improved Cache Utilization**: Streamlined data access results in efficient caching. 25 | 26 | #### Example of Covering Index 27 | 28 | **SQL Syntax**: 29 | ```sql 30 | CREATE INDEX idx_film_covering ON film (rating, rental_rate, title, release_year); 31 | ``` 32 | 33 | **Query Using the Index**: 34 | ```sql 35 | SELECT title, release_year 36 | FROM film 37 | WHERE rating = 'PG-13' AND rental_rate = 2.99; 38 | ``` 39 | 40 | - **Effectiveness**: 41 | - The query fetches `title` and `release_year` without directly accessing the `film` table. 42 | - Filters on `rating` and `rental_rate` ensure minimal rows are traversed. 43 | 44 | ### Key Takeaways 45 | 46 | - **Best Practice**: Use covering indexes for queries accessing frequently used column groups. 47 | - **Execution**: MySQL prioritizes filtering (`WHERE`), then sorting (`ORDER BY`), leveraging the indexed columns. 48 | - **Efficiency**: Creates a balance between performance improvement and storage costs. 49 | 50 | 51 | 52 | ## Composite Index 53 | 54 | ### What is a Composite Index? 55 | 56 | - **Definition**: An index containing multiple columns, ideal for queries filtering on several row-based conditions. 57 | - **Order Matters**: MySQL processes columns left-to-right; place frequently queried columns first. 58 | 59 | #### Example of Composite Index 60 | 61 | **SQL Syntax**: 62 | ```sql 63 | CREATE INDEX idx_rental_customer_date ON rental (customer_id, rental_date); 64 | ``` 65 | 66 | **Query Example**: 67 | ```sql 68 | SELECT rental_id, rental_date 69 | FROM rental 70 | WHERE customer_id = 123 71 | AND rental_date BETWEEN '2024-01-01' AND '2024-12-31'; 72 | ``` 73 | 74 | - **Functionality**: 75 | - Filters on `customer_id` first, then narrows down using `rental_date`. 76 | - Significantly fewer rows are checked compared to non-indexed tables. 77 | 78 | ### Composite Index vs. Covering Index 79 | 80 | | **Feature** | **Composite Index** | **Covering Index** | 81 | |----------------------|---------------------------------------------|--------------------------------------------------| 82 | | **Purpose** | Filters multiple columns (`WHERE`, `JOIN`) | Satisfies queries entirely using the index | 83 | | **Storage** | Lower storage requirement | Higher storage requirement | 84 | | **Usage** | Efficient for partial column matches | Requires all queried columns in the index | 85 | 86 | **Key Notes**: 87 | - Use composite indexes for filtering across multiple columns. 88 | - Opt for covering indexes when queries rely heavily on `SELECT` columns. 89 | 90 | ## Transactions 91 | 92 | ### Overview of Transactions 93 | 94 | - **Definition**: A sequence of operations where either all succeed or none apply, ensuring consistency. 95 | - **Steps**: 96 | 1. Begin with `START TRANSACTION;`. 97 | 2. Execute the required operations. 98 | 3. Use `COMMIT` to finalize or `ROLLBACK` to revert. 99 | 100 | #### Example 101 | 102 | ```sql 103 | START TRANSACTION; 104 | INSERT INTO rental (C1, C2, C3, C4) VALUES (A, B, C, D); 105 | COMMIT; -- Finalizes changes 106 | ROLLBACK; -- Reverts if an issue arises 107 | ``` 108 | 109 | ### SAVEPOINTs 110 | 111 | - **Purpose**: Checkpoints within transactions to selectively roll back changes. 112 | - **Example**: 113 | ```sql 114 | START TRANSACTION; 115 | INSERT INTO rental (C1, C2, C3, C4) VALUES (A, B, C, D); 116 | SAVEPOINT AFTER_INSERT; 117 | ``` 118 | - Roll back to the savepoint with: 119 | ```sql 120 | ROLLBACK TO SAVEPOINT AFTER_INSERT; 121 | ``` 122 | 123 | ### Key Notes 124 | 125 | - SAVEPOINTs allow partial rollbacks. 126 | - Releasing unused savepoints optimizes system resources: 127 | ```sql 128 | RELEASE SAVEPOINT AFTER_INSERT; 129 | ``` 130 | 131 | 132 | ## Isolation Levels 133 | 134 | ### Importance of Isolation Levels 135 | 136 | - **Objective**: Ensure that transactions do not interfere with one another, preserving data integrity. 137 | - **Levels**: 138 | 139 | 1. **READ UNCOMMITTED**: 140 | - Uncommitted changes are visible to others (**Dirty Reads**). 141 | - Example: A change not finalized by User 1 is readable by User 2. 142 | 143 | 2. **READ COMMITTED**: 144 | - Only committed changes are visible. 145 | - Ensures users cannot see ongoing, uncommitted transactions. 146 | 147 | 3. **REPEATABLE READ**: 148 | - Guarantees consistent data during a transaction. 149 | - Example: A user’s view of the data remains unchanged, even if another user modifies it. 150 | 151 | 4. **SERIALIZABLE**: 152 | - Highest isolation level, ensuring transactions execute in a strict sequence. 153 | - Example: Transactions wait for preceding ones to finish before execution. 154 | 155 | ### Practical Usage 156 | 157 | - **READ COMMITTED**: Suitable for most applications needing reliable data reads. 158 | - **SERIALIZABLE**: Used in systems requiring strict accuracy, e.g., banking. 159 | 160 | 161 | ## Live Demo 162 | 163 | ### Objectives 164 | 165 | - Demonstrate behavior across different isolation levels. 166 | 167 | ### Steps 168 | 169 | 1. **Setup**: 170 | - Open two terminals and log into the same database. 171 | 172 | 2. **READ COMMITTED Example**: 173 | - **Terminal 1**: 174 | ```sql 175 | START TRANSACTION; 176 | ``` 177 | Update an entry without committing. 178 | - **Terminal 2**: 179 | Run the same query. Changes from **Terminal 1** are not visible until committed. 180 | 181 | 3. **SERIALIZABLE Example**: 182 | - **Terminal 1**: 183 | Start a transaction and make updates. 184 | - **Terminal 2**: 185 | Attempt to update the same data; transaction blocks until **Terminal 1** completes. 186 | 187 | ### Key Insights 188 | 189 | - **READ COMMITTED**: Prevents dirty reads by hiding uncommitted changes. 190 | - **SERIALIZABLE**: Ensures complete isolation but may delay transaction completion. 191 | 192 | -------------------------------------------------------------------------------- /DBMS Module/CRUD _Operations _in _SQL - Part 1.md: -------------------------------------------------------------------------------- 1 | ## Topics to be discussed 2 | 3 | - **CREATE**: Adding new data to a table. 4 | - **READ**: Retrieving and viewing data from a table. 5 | - **UPDATE**: Modifying existing data in a table (covered in Part 2). 6 | - **DELETE**: Removing data from a table (covered in Part 2). 7 | 8 | --- 9 | 10 | ## CREATE Operation 11 | 12 | The **CREATE** operation allows you to add new entries (rows) to a table in the database. The **INSERT** statement is used to perform this operation. 13 | 14 | ### Syntax of INSERT Command: 15 | ```sql 16 | INSERT INTO table_name (column1, column2, column3, ...) 17 | VALUES 18 | (value1, value2, value3, ...), 19 | (value1, value2, value3, ...), 20 | (value1, value2, value3, ...); 21 | ``` 22 | 23 | ### Key Points: 24 | 1. The values must correspond to the column order specified in the `INSERT` statement. 25 | Example: 26 | ```sql 27 | INSERT INTO employees (first_name, last_name, email) 28 | VALUES ('John', 'Doe', 'john.doe@example.com'); 29 | ``` 30 | 31 | 2. If you do not specify the column names, values will be inserted into the table columns in the order they were defined during table creation. 32 | Example: 33 | ```sql 34 | INSERT INTO employees 35 | VALUES (1, 'John', 'Doe', 'john.doe@example.com', 'HR'); 36 | ``` 37 | 38 | 3. **NULL and DEFAULT Values**: 39 | - If a column value is not provided or explicitly set as **NULL**, the value of that column will be `NULL`. 40 | - If the column has a **DEFAULT** value, using the `DEFAULT` keyword will assign that value to the column. 41 | Example: 42 | ```sql 43 | INSERT INTO employees (first_name, last_name, department) 44 | VALUES ('Jane', 'Smith', DEFAULT); 45 | ``` 46 | 47 | --- 48 | 49 | ## READ Operation 50 | 51 | 52 | The **READ** operation retrieves data from a table, typically using the **SELECT** statement. This operation allows you to extract specific columns, filter rows, or apply conditions to obtain the required data. 53 | 54 | ### Basic Syntax of SELECT Command: 55 | 1. **Select All Columns**: 56 | ```sql 57 | SELECT * FROM employees; 58 | ``` 59 | 60 | 2. **Select Specific Columns**: 61 | ```sql 62 | SELECT first_name, last_name FROM employees; 63 | ``` 64 | 65 | 3. **Using Column Aliases**: 66 | You can rename columns in the result set using aliases for better readability. 67 | ```sql 68 | SELECT first_name AS fname, last_name AS lname FROM employees; 69 | ``` 70 | Note: Aliases are temporary and do not alter the actual column names in the table. 71 | 72 | ### Selecting Distinct Values 73 | To eliminate duplicate rows, use the **DISTINCT** keyword. 74 | Example: 75 | ```sql 76 | SELECT DISTINCT job_title FROM employees; 77 | ``` 78 | 79 | ### Selecting Constant Values 80 | You can include constant values in your result set along with table columns. 81 | Example: 82 | ```sql 83 | SELECT first_name, 'Employee' AS role FROM employees; 84 | ``` 85 | This query returns each employee's first name along with the constant value `'Employee'` under the column "role". 86 | 87 | ### Inserting Data from Another Table 88 | You can copy data from one table to another using the **INSERT INTO ... SELECT** statement. 89 | Example: 90 | ```sql 91 | INSERT INTO archived_employees (first_name, last_name, email) 92 | SELECT first_name, last_name, email 93 | FROM employees; 94 | ``` 95 | 96 | --- 97 | 98 | ## WHERE Clause 99 | 100 | The **WHERE** clause is used to filter rows based on specific conditions, allowing you to retrieve only the relevant data from a table. 101 | 102 | ### Syntax of WHERE Clause: 103 | ```sql 104 | SELECT column1, column2, ... 105 | FROM table_name 106 | WHERE condition; 107 | ``` 108 | 109 | ### Examples of WHERE Clause: 110 | 1. **Simple Condition**: 111 | Retrieve employees with a salary greater than 50,000. 112 | ```sql 113 | SELECT first_name, last_name, salary 114 | FROM employees 115 | WHERE salary > 50000; 116 | ``` 117 | 118 | 2. **Multiple Conditions (AND/OR)**: 119 | Combine conditions using **AND** or **OR** operators. 120 | ```sql 121 | SELECT first_name, last_name, salary 122 | FROM employees 123 | WHERE salary > 50000 AND department = 'Sales'; 124 | ``` 125 | This query retrieves employees with a salary greater than 50,000 who work in the Sales department. 126 | 127 | 3. **Using NOT**: 128 | Exclude specific rows using the **NOT** operator. 129 | ```sql 130 | SELECT first_name, last_name, department 131 | FROM employees 132 | WHERE NOT department = 'HR'; 133 | ``` 134 | This query retrieves employees who are not part of the HR department. 135 | 136 | 4. **Using IN Clause**: 137 | The **IN** operator allows you to specify multiple possible values for a column. 138 | ```sql 139 | SELECT first_name, last_name, department 140 | FROM employees 141 | WHERE department IN ('HR', 'Sales', 'Finance'); 142 | ``` 143 | This query retrieves employees working in HR, Sales, or Finance departments. 144 | 145 | ### Key Points about WHERE Clause: 146 | - Conditions can include comparisons (`>`, `<`, `=`, etc.), ranges (`BETWEEN`), and patterns (`LIKE`). 147 | - Combine multiple conditions for more complex queries. 148 | 149 | **[Message for Instructor]**: Demonstrate queries using different WHERE clauses, showing how conditions refine the result set. 150 | 151 | --- 152 | 153 | ## Summary of Part 1 154 | 155 | 1. **CREATE** operation: Adds new rows to a table using the **INSERT** statement. 156 | 2. **READ** operation: Retrieves data from a table using the **SELECT** statement, with options for filtering and formatting the results using aliases, DISTINCT, and WHERE clauses. 157 | 3. Demonstrated the syntax and practical examples for: 158 | - Adding data to tables. 159 | - Retrieving data with and without filters. 160 | - Using the WHERE clause to refine data selection. 161 | 162 | In the next session, we will cover: 163 | - **UPDATE** operation: Modifying existing rows. 164 | - **DELETE** operation: Removing data from tables. 165 | -------------------------------------------------------------------------------- /DBMS Module/High_ Availability _and _Replication.md: -------------------------------------------------------------------------------- 1 | # High Availability and Replication 2 | 3 | ## Agenda of the Lecture 4 | 5 | - **Topics to Cover**: 6 | 1. High Availability 7 | 2. Types of Replication in MySQL 8 | 3. Live Demo 9 | 4. Factors to Monitor in Replication 10 | 11 | 12 | ## High Availability 13 | 14 | ### Understanding High Availability 15 | 16 | - **Definition**: High availability ensures a system remains operational and accessible despite failures, minimizing downtime and impact on services. 17 | - **Example**: A bank with multiple branches can continue functioning if one branch is closed, as transactions shift to other locations. While there may be delays, complete downtime is avoided. 18 | 19 | ### Importance of High Availability 20 | 21 | 1. **Minimizing Downtime**: 22 | - Reduces service unavailability. 23 | - Example: For global services like Google or Instagram, even seconds of downtime can be unacceptable. In contrast, banking systems may tolerate short delays. 24 | - Acceptable downtime must be defined based on the system's criticality. 25 | 26 | 2. **Protection Against Data Loss**: 27 | - **Data Redundancy**: Multiple copies of data in different locations protect against complete data loss. 28 | - Example: A library with backups in multiple locations minimizes losses in case of fire or disaster. 29 | 30 | 3. **Ensuring Continuous Operations**: 31 | - Systems remain online and accessible during hardware failures or maintenance tasks. 32 | 33 | 4. **Enhancing User Experience**: 34 | - Reliability reduces disruptions and ensures user trust. 35 | 36 | ### Key Components of High Availability Architecture in MySQL 37 | 38 | 1. **Replication**: 39 | - Data is copied across multiple servers. 40 | - Ensures availability during server failures by redirecting traffic to replicas. 41 | 42 | 2. **Failover**: 43 | - Automatic redirection from the primary server to a secondary server if the primary fails. 44 | - Balances traffic during outages. 45 | 46 | 3. **Load Balancing**: 47 | - Distributes incoming requests across servers to prevent overload. 48 | - Improves system stability and performance. 49 | 50 | 51 | ## Types of Replication in MySQL 52 | 53 | ### Overview 54 | 55 | Replication in MySQL involves duplicating data between servers to ensure consistency, high availability, and disaster recovery. 56 | 57 | ### Types of Replication 58 | 59 | 1. **Source-Replica (Master-Slave) / Asynchronous Replication**: 60 | - **How It Works**: 61 | - The master handles read/write operations. 62 | - Replicas handle only read operations, ensuring consistency. 63 | - **Key Characteristics**: 64 | - A **locking mechanism** prevents "dirty reads" during write operations. 65 | - Asynchronous: The master does not wait for replicas to sync before continuing. 66 | - **Failover**: A replica can be promoted to master if the original master fails. 67 | - **Diagram**: 68 | ![image](https://hackmd.io/_uploads/H169uP5Zkx.png) 69 | 70 | 2. **Semi-Synchronous Replication**: 71 | - **How It Works**: 72 | - The primary server waits for at least one replica to acknowledge data receipt before proceeding. 73 | - **Pros**: 74 | - Ensures data reliability by confirming at least one replica has up-to-date information. 75 | - **Cons**: 76 | - Slower performance due to the acknowledgment process. 77 | - Affected by network delays. 78 | - **Diagram**: 79 | ![image](https://hackmd.io/_uploads/SygZ9wcZ1g.png) 80 | 81 | 3. **Group Replication**: 82 | - **How It Works**: 83 | - A cluster of MySQL servers (multi-master setup) handles both read and write operations. 84 | - Data is replicated across all nodes. 85 | - **Benefits**: 86 | - Fault tolerance: Surviving nodes continue handling requests if one fails. 87 | - Ideal for systems requiring seamless failover and load balancing. 88 | 89 | ## Live Demo 90 | 91 | ### Objective 92 | 93 | - Demonstrate how to set up replication between two MySQL servers. 94 | 95 | ### Steps 96 | 97 | 1. **Create Two Servers**: 98 | - Open two terminals and set up `server1` and `server2` with MySQL installed. 99 | 100 | 2. **Enable Binary Logging on `server1`**: 101 | - Edit the MySQL configuration file to enable binary logging and set the server ID. 102 | - **Command**: 103 | ```bash 104 | sudo systemctl restart mysql 105 | ``` 106 | 107 | 3. **Create a Replication User**: 108 | - On `server1`: 109 | ```sql 110 | CREATE USER 'replicator'@'%' IDENTIFIED BY 'Devl@1234'; 111 | GRANT REPLICATION SLAVE ON *.* TO 'replicator'@'%'; 112 | ``` 113 | 114 | 4. **Lock Tables on `server1`**: 115 | - Use: 116 | ```sql 117 | FLUSH TABLES WITH READ LOCK; 118 | ``` 119 | - Prevents writes while setting up replication. 120 | 121 | 5. **Configure `server2`**: 122 | - Edit `my.cnf` to point `server2` to `server1`: 123 | ```bash 124 | sudo vi /etc/mysql/my.cnf 125 | ``` 126 | - Add: 127 | ``` 128 | server-id = 2 129 | replicate-do-db = demo 130 | ``` 131 | 132 | 6. **Start Replication**: 133 | - Run on `server2`: 134 | ```sql 135 | CHANGE MASTER TO 136 | MASTER_HOST='54.206.122.120', 137 | MASTER_USER='replicator', 138 | MASTER_PASSWORD='Devl@1234', 139 | MASTER_LOG_FILE='mysql-bin.000001', 140 | MASTER_LOG_POS=839; 141 | START SLAVE; 142 | ``` 143 | 144 | 7. **Verify Replication**: 145 | - On `server2`: 146 | ```sql 147 | SHOW SLAVE STATUS\G; 148 | ``` 149 | - Ensure replication is active and no errors are reported. 150 | 151 | 8. **Replication Test**: 152 | - Create a database and table on `server1`: 153 | ```sql 154 | CREATE DATABASE demo; 155 | USE demo; 156 | CREATE TABLE students (id INT, name VARCHAR(50)); 157 | INSERT INTO students VALUES (1, 'John Doe'); 158 | ``` 159 | - Verify the data is replicated on `server2`. 160 | 161 | 9. **Network Configuration**: 162 | - Update MySQL configuration to allow external connections: 163 | ```bash 164 | sudo vi /etc/mysql/mysql.conf.d/mysqld.cnf 165 | ``` 166 | - Set `bind-address` to `0.0.0.0`. 167 | 168 | 10. **Verify Connectivity**: 169 | - Test with: 170 | ```bash 171 | telnet 54.206.122.120 3306 172 | ``` 173 | 174 | 175 | ## Factors to Monitor in Replication 176 | 177 | ### Key Metrics 178 | 179 | 1. **Replication Lag**: 180 | - Monitored using: 181 | ```sql 182 | SHOW REPLICA STATUS\G; 183 | ``` 184 | - Indicates delays in data synchronization. 185 | - High lag can cause inconsistencies in read operations. 186 | 187 | 2. **Duplicate Key Errors**: 188 | - Occurs when duplicate entries are created due to replication delays. 189 | - Error logs can be reviewed: 190 | ```bash 191 | vi /var/log/mysql/error.log 192 | ``` 193 | 194 | 3. **Network Issues**: 195 | - Network disruptions can halt replication. 196 | - Logs record errors related to connectivity issues. 197 | 198 | ### Best Practices 199 | 200 | - Monitor logs regularly for replication errors or lag. 201 | - Use alert systems to notify of issues in real-time. 202 | - Test failover scenarios periodically to ensure readiness. 203 | -------------------------------------------------------------------------------- /DBMS Module/Performance_ and _Database _Security.md: -------------------------------------------------------------------------------- 1 | # Performance and Database Security 2 | 3 | 4 | ## Agenda of the Lecture 5 | 6 | - **Topics to Cover**: 7 | 1. Introduction 8 | 2. Creation of Certificates 9 | 3. Audit Logging 10 | 11 | 12 | ## Introduction 13 | 14 | ### SQL Monitoring: Then and Now 15 | 16 | **[Ask Learners] How did monitoring work 4–5 years ago?** 17 | In earlier times, monitoring relied on simple scripts and Linux utilities to track critical metrics such as CPU usage, memory consumption, and database size. Modern tools like Grafana have since streamlined this process by visualizing data more effectively. 18 | 19 | ### Key Metrics for Monitoring 20 | 21 | 1. **CPU Usage**: 22 | - Tracks how much processing power the database consumes. 23 | 24 | 2. **Memory Usage**: 25 | - Command to monitor memory: 26 | ```bash 27 | free -h 28 | ``` 29 | 30 | 3. **Database Size**: 31 | - Command to calculate database size: 32 | ```bash 33 | sudo mysql -u root -e "SELECT table_schema AS 'Database', SUM(data_length + index_length)/1024/1024 AS 'Size (MB)' FROM information_schema.TABLES GROUP BY table_schema;" 34 | ``` 35 | 36 | ### Creating a Basic Monitoring Script 37 | 38 | Example script to log CPU, memory, and database size: 39 | ```bash 40 | #!/bin/bash 41 | echo "Logging system metrics..." 42 | echo "CPU Usage:" >> sql_monitor.log 43 | top -b -n1 | grep "Cpu(s)" >> sql_monitor.log 44 | echo "Memory Usage:" >> sql_monitor.log 45 | free -h >> sql_monitor.log 46 | echo "Database Sizes:" >> sql_monitor.log 47 | sudo mysql -u root -e "SELECT table_schema AS 'Database', SUM(data_length + index_length)/1024/1024 AS 'Size (MB)' FROM information_schema.TABLES GROUP BY table_schema;" >> sql_monitor.log 48 | ``` 49 | - Save the script as `logs_monitor_script_sql.sh` and execute: 50 | ```bash 51 | chmod +x logs_monitor_script_sql.sh 52 | ./logs_monitor_script_sql.sh 53 | ``` 54 | 55 | ### Securing the Database Installation 56 | 57 | 1. Use the `mysql_secure_installation` utility to enhance security: 58 | ```bash 59 | sudo mysql_secure_installation 60 | ``` 61 | - Prompts include: 62 | - Remove anonymous users. 63 | - Disallow remote root login. 64 | - Remove test database. 65 | - Strengthen password requirements. 66 | 67 | 2. Example configuration: 68 | ![image](https://hackmd.io/_uploads/S1OPTIvMkg.png) 69 | 70 | ### Modern Monitoring Tools 71 | 72 | - **Grafana**: 73 | - Collects and visualizes data from databases. 74 | - Enhances monitoring by providing intuitive dashboards. 75 | 76 | 77 | 78 | ## Creation of Certificates 79 | 80 | ### What Are Certificates? 81 | 82 | Certificates authenticate servers and secure communications between clients and servers. MySQL supports SSL certificates to ensure secure database connections. 83 | 84 | ### Creating SSL Certificates 85 | 86 | 1. **Prepare Directory**: 87 | - Create a directory for certificates: 88 | ```bash 89 | sudo mkdir /etc/mysql/certs 90 | sudo chmod 700 /etc/mysql/certs 91 | sudo chown mysql:mysql /etc/mysql/certs 92 | ``` 93 | 94 | 2. **Generate CA Key**: 95 | ```bash 96 | openssl genrsa 2048 > /etc/mysql/certs/ca-key.pem 97 | ``` 98 | 99 | 3. **Create a CA Certificate**: 100 | ```bash 101 | openssl req -new -x509 -nodes -days 365 -key /etc/mysql/certs/ca-key.pem -out /etc/mysql/certs/ca-cert.pem -subj "/CN=MySQL_CA" 102 | ``` 103 | 104 | 4. **Create a Server Key**: 105 | - Generate a server key for MySQL: 106 | ```bash 107 | openssl genrsa 2048 > /etc/mysql/certs/server-key.pem 108 | openssl req -new -key /etc/mysql/certs/server-key.pem -out /etc/mysql/certs/server-req.pem -subj "/CN=MySQL_Server" 109 | ``` 110 | 111 | 5. **Sign the Server Certificate**: 112 | ```bash 113 | openssl x509 -req -new -in /etc/mysql/certs/server-req.pem -days 365 -CA /etc/mysql/certs/ca-cert.pem -CAkey /etc/mysql/certs/ca-key.pem -set_serial 01 -out /etc/mysql/certs/server-cert.pem 114 | ``` 115 | 116 | 6. **Secure Certificates**: 117 | ```bash 118 | sudo chmod 600 /etc/mysql/certs/*.pem 119 | sudo chown mysql:mysql /etc/mysql/certs/*.pem 120 | ``` 121 | 122 | ### Configuring MySQL to Use SSL 123 | 124 | 1. Edit the MySQL configuration file: 125 | ```bash 126 | sudo vi /etc/mysql/mysql.conf.d/mysqld.cnf 127 | ``` 128 | - Add certificate paths: 129 | ``` 130 | ssl-ca=/etc/mysql/certs/ca-cert.pem 131 | ssl-cert=/etc/mysql/certs/server-cert.pem 132 | ssl-key=/etc/mysql/certs/server-key.pem 133 | ``` 134 | 135 | 2. Restart MySQL: 136 | ```bash 137 | sudo systemctl daemon-reload 138 | sudo systemctl restart mysql 139 | ``` 140 | 141 | 3. Verify SSL: 142 | ```sql 143 | SHOW VARIABLES LIKE 'have_ssl'; 144 | SHOW VARIABLES LIKE 'ssl_%'; 145 | ``` 146 | 147 | 148 | ## Audit Logging 149 | 150 | ### What is Audit Logging? 151 | 152 | Audit logging tracks system and database activities to maintain security and operational oversight. It is crucial for identifying unauthorized access and understanding system changes. 153 | 154 | ### Key Benefits 155 | 156 | 1. **User Login Attempts**: 157 | - Logs successful and failed login attempts, identifying potential unauthorized access. 158 | 159 | 2. **Query Execution**: 160 | - Tracks all executed queries, helping in change audits. 161 | 162 | 3. **Administrative Actions**: 163 | - Records user creation, deletion, and permission changes for accountability. 164 | 165 | ### Implementation 166 | 167 | 1. **Use Audit Plugins**: 168 | - Plugins like `audit_log` in MySQL provide detailed logs of user actions. 169 | 170 | 2. **Enable Audit Logging**: 171 | - Add to the MySQL configuration file: 172 | ``` 173 | plugin-load-add=audit_log.so 174 | audit_log_policy=ALL 175 | audit_log_file=/var/log/mysql_audit.log 176 | ``` 177 | 178 | 3. **Restart MySQL**: 179 | ```bash 180 | sudo systemctl restart mysql 181 | ``` 182 | 183 | 4. **Review Logs**: 184 | - Check audit logs for user activities: 185 | ```bash 186 | cat /var/log/mysql_audit.log 187 | ``` 188 | 189 | ## Comparing Server and CA Certificates 190 | 191 | | **Feature** | **Server Certificate** | **CA Certificate** | 192 | |--------------------------|--------------------------------------------------------------------------|--------------------------------------------------------------| 193 | | **Purpose** | Authenticates a specific server for secure communication. | Establishes trust for server certificates. | 194 | | **Role in Security** | Used for client-server data exchange security. | Validates and signs server certificates. | 195 | | **Usage** | Applied in HTTPS, SSL, and TLS connections. | Used in creating and verifying server and client certificates. | 196 | 197 | -------------------------------------------------------------------------------- /DBMS Module/SQL _Administration.md: -------------------------------------------------------------------------------- 1 | # SQL Administration 2 | 3 | ## Agenda of the Lecture 4 | 5 | - **Topics to Cover**: 6 | 1. User and Permission Management 7 | 2. Maintenance Tasks in MySQL 8 | 3. Monitoring the Database 9 | 4. Managing Logs and Cleanup 10 | 5. Optimizing Performance and Automation 11 | 12 | 13 | ## User and Permission Management 14 | 15 | ### Responsibilities of a DevOps Engineer 16 | 17 | - Manage access to production environments. 18 | - Ensure proper permission assignment, data security, and backups. 19 | - Perform performance tuning and monitor database usage. 20 | 21 | ### Authentication vs. Authorization 22 | 23 | - **Authentication**: Verifies the user's identity (e.g., using a username and password). 24 | - Example: Checking an ID to confirm entry to a building. 25 | - **Authorization**: Determines what an authenticated user can do or access. 26 | - Example: Restricting access to certain floors in a building even after verifying the ID. 27 | 28 | **Key Considerations**: 29 | - Protect sensitive data from unauthorized access. 30 | - Mitigate risks of internal threats by clearly defining and managing permissions. 31 | 32 | ### Creating Users and Assigning Permissions 33 | 34 | **Create a New User**: 35 | ```sql 36 | CREATE USER 'username'@'host' IDENTIFIED BY 'password'; 37 | ``` 38 | - Example: 39 | ```sql 40 | CREATE USER 'John'@'%' IDENTIFIED BY 'Pass'; 41 | ``` 42 | - `@'host'`: Restricts the user to a specific IP (e.g., `@192.168.1.100`). 43 | - `@'%'`: Allows access from any IP. 44 | 45 | **Granting Permissions**: 46 | ```sql 47 | GRANT SELECT, INSERT ON database_name.* TO 'username'@'host'; 48 | ``` 49 | - **Revoke Permissions**: 50 | ```sql 51 | REVOKE ALL PRIVILEGES ON database_name.* FROM 'username'@'host'; 52 | ``` 53 | 54 | **Managing Roles**: 55 | - **Create and Assign Roles**: 56 | ```sql 57 | CREATE ROLE 'developer_role'; 58 | GRANT SELECT, UPDATE ON database_name.* TO 'developer_role'; 59 | GRANT 'developer_role' TO 'username'@'host'; 60 | ``` 61 | 62 | **Best Practice**: 63 | - Use roles to simplify and standardize permission management for groups of users. 64 | 65 | 66 | 67 | ## Maintenance Tasks in MySQL 68 | 69 | ### Importance of Database Maintenance 70 | 71 | Regular maintenance ensures: 72 | - Data availability and security. 73 | - Fast recovery in case of system failure. 74 | - Efficient database performance. 75 | 76 | ### Types of Backups 77 | 78 | 1. **Logical Backup**: 79 | - **Definition**: Exports database content (schema and data) in a human-readable format, such as SQL. 80 | - **Command**: 81 | ```bash 82 | mysqldump -u user -p database_name > backup.sql 83 | ``` 84 | - **Use Case**: Short-term backup needs or for data transfer between servers. 85 | 86 | 2. **Physical Backup**: 87 | - **Definition**: Creates a binary copy of database files (e.g., `.ibd` files). 88 | - **Tool**: `rsync` for syncing data with minimal resource usage. 89 | - **Command**: 90 | ```bash 91 | rsync -av --progress /var/lib/mysql /backup_location/ 92 | ``` 93 | - **Use Case**: Long-term backup, faster for large databases. 94 | 95 | **Choosing Between Logical and Physical Backups**: 96 | - Logical backups are better for smaller databases and when a readable format is required. 97 | - Physical backups are more efficient for large databases. 98 | 99 | 100 | 101 | ## Monitoring the Database 102 | 103 | ### Key Areas to Monitor 104 | 105 | 1. **Query Performance**: 106 | - Identify and optimize slow queries. 107 | - **Enable Slow Query Log**: 108 | ```sql 109 | SET GLOBAL slow_query_log = 'ON'; 110 | SET GLOBAL long_query_time = 2; 111 | ``` 112 | - **Analyze Logs**: Use `mysqldumpslow` to review slow queries. 113 | 114 | 2. **Replication Status**: 115 | - Monitor data synchronization between primary and replica servers. 116 | - **Command**: 117 | ```sql 118 | SHOW SLAVE STATUS\G; 119 | ``` 120 | - **Metric**: `Seconds_Behind_Master` indicates replication lag. 121 | 122 | 3. **Resource Utilization**: 123 | - Monitor server resources like CPU, memory, and disk I/O. 124 | - **Commands**: 125 | ```sql 126 | SHOW GLOBAL STATUS LIKE 'Threads_connected'; 127 | SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_reads'; 128 | ``` 129 | 130 | 4. **Optimizing Table Performance**: 131 | - **Analyze Tables**: Updates table statistics for better query optimization: 132 | ```sql 133 | ANALYZE TABLE table_name; 134 | ``` 135 | - **Optimize Tables**: Reclaims unused space and improves efficiency: 136 | ```sql 137 | OPTIMIZE TABLE table_name; 138 | ``` 139 | 140 | 141 | 142 | ## Managing Logs and Cleanup 143 | 144 | ### Types of Logs 145 | 146 | 1. **General Query Logs**: 147 | - Enable and specify the log file location: 148 | ```sql 149 | SET GLOBAL general_log = 'ON'; 150 | SET GLOBAL general_log_file = '/var/log/mysqllog'; 151 | ``` 152 | 153 | 2. **Binary Logs**: 154 | - Used for replication and recovery. 155 | - **Purge Old Logs**: 156 | ```sql 157 | PURGE BINARY LOGS TO 'mysql_bin.00010'; 158 | ``` 159 | 160 | ### Importance of Log Cleanup 161 | 162 | - Prevent logs from consuming excessive disk space. 163 | - Use cron jobs to automate periodic log archiving and cleanup. 164 | 165 | 166 | 167 | ## Optimizing Performance and Automation 168 | 169 | ### SQL Performance Optimization 170 | 171 | 1. **Optimize MySQL Configurations**: 172 | - Example settings: 173 | ```ini 174 | max_connections = 200 175 | query_cache_size = 0 176 | ``` 177 | 178 | 2. **Query Optimization**: 179 | - Use `EXPLAIN` to analyze queries: 180 | ```sql 181 | EXPLAIN SELECT column_name FROM table_name WHERE condition; 182 | ``` 183 | - Avoid `SELECT *` to minimize data transfer. 184 | 185 | 3. **Indexing**: 186 | - Index frequently queried columns to speed up data retrieval. 187 | 188 | 4. **Enable Slow Query Logs**: 189 | - Identify and optimize slow queries: 190 | ```sql 191 | SET GLOBAL slow_query_log = 'ON'; 192 | ``` 193 | 194 | ### Automating Database Backups 195 | 196 | **Script for Automating Backups**: 197 | ```bash 198 | #!/bin/bash 199 | USER="username" 200 | PASSWORD="password" 201 | DATABASE="database_name" 202 | BACKUP_PATH="/path/to/backup/" 203 | FILE_NAME="${DATABASE}_$(date +%F_%H-%M-%S).sql" 204 | 205 | mysqldump -u $USER -p$PASSWORD $DATABASE > $BACKUP_PATH$FILE_NAME 206 | ``` 207 | - **Explanation**: 208 | - The script uses `mysqldump` to back up the database with a timestamped filename. 209 | - Backups are organized and saved to the specified path. 210 | 211 | **Automating with Cron Jobs**: 212 | - Schedule the script to run at regular intervals using cron: 213 | ```bash 214 | crontab -e 215 | ``` 216 | Example cron job for daily backups at midnight: 217 | ``` 218 | 0 0 * * * /path/to/backup_script.sh 219 | ``` 220 | 221 | **Result**: 222 | - Regular, automated backups ensure data availability and minimize manual effort. 223 | 224 | ![Backup Script Example](https://hackmd.io/_uploads/BJWVVYMzJg.png) 225 | ![Backup File Example](https://hackmd.io/_uploads/SyUCEtGG1l.png) 226 | 227 | This setup allows for consistent, automated database backups, with file organization to aid in quick retrieval. You can schedule this script using cron jobs to automate the process, ensuring timely and reliable backups. 228 | -------------------------------------------------------------------------------- /DBMS Module/SQL_ Optimization _and _Indexing.md: -------------------------------------------------------------------------------- 1 | # Topics to be discussed 2 | 3 | - Indexing and its purpose in databases. 4 | - Types of indexes and their specific use cases. 5 | - Concepts of B-Tree and Hash Indexes. 6 | - Commands for analyzing and optimizing tables. 7 | 8 | --- 9 | 10 | ## Indexing 11 | Indexing is a mechanism that improves the speed of data retrieval operations in a database. It acts as a reference, similar to an index in a book, allowing users to locate information quickly without scanning the entire dataset. 12 | 13 | ### Key Features of Indexing 14 | 15 | #### **Advantages (Pros):** 16 | 1. **Fast Performance**: 17 | - Speeds up query execution, especially for large datasets. 18 | 2. **Efficient Sorting**: 19 | - Indexes help in sorting data efficiently. 20 | 21 | #### **Disadvantages (Cons):** 22 | 1. **Increased Storage**: 23 | - Indexes consume additional disk space. 24 | 2. **Maintenance Overhead**: 25 | - Updating, inserting, or deleting data requires updating the index, which slows down write operations. 26 | 27 | --- 28 | 29 | ### Creating an Index 30 | 31 | Indexes are created using the `CREATE INDEX` statement. 32 | 33 | **Syntax**: 34 | ```sql 35 | CREATE INDEX index_name ON table_name (column_name); 36 | ``` 37 | 38 | **Example**: 39 | ```sql 40 | CREATE INDEX idx_order_date ON orders (order_date); 41 | ``` 42 | **Best Practice**: Follow naming conventions by prefixing index names with `idx_`. 43 | 44 | --- 45 | 46 | ### Listing Existing Indexes 47 | 48 | To view all indexes on a table, use the `SHOW INDEX` command: 49 | ```sql 50 | SHOW INDEX FROM table_name; 51 | ``` 52 | 53 | --- 54 | 55 | ### Dropping an Index 56 | 57 | Indexes can be removed using the `DROP INDEX` command: 58 | ```sql 59 | DROP INDEX idx_order_date ON orders; 60 | ``` 61 | 62 | **Important Note**: 63 | Excessive indexing can slow down write operations and increase storage utilization. Avoid creating unnecessary indexes. 64 | 65 | --- 66 | 67 | ## Types of Index 68 | 69 | **Duration**: 25 minutes 70 | 71 | Different types of indexes are used based on the specific requirements of the query. 72 | 73 | ### 1. **Primary Key Index (Clustered Index)** 74 | - A **clustered index** is automatically created on the primary key column of a table. 75 | - Rows in the table are physically ordered based on the primary key. 76 | 77 | **Key Characteristics**: 78 | - Each table can have only one clustered index. 79 | - Significantly speeds up queries that search by the primary key. 80 | 81 | **Example**: Query without indexing vs. with indexing. 82 | 83 | - **Without Indexing**: SQL scans all rows sequentially to find the required record. 84 | - **With Indexing**: SQL directly accesses the required record using the sorted structure. 85 | 86 | **Diagram**: 87 | ![Clustered Index Example](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/096/933/original/one.png?1732041730) 88 | 89 | --- 90 | 91 | ### 2. **Unique Index** 92 | - Ensures that values in the indexed column are unique. 93 | - Prevents duplicate entries. 94 | 95 | **Syntax**: 96 | ```sql 97 | CREATE UNIQUE INDEX idx_email ON instructor (email); 98 | ``` 99 | 100 | --- 101 | 102 | ### 3. **Single Column Index (Non-Unique)** 103 | - Speeds up searches on a specific column. 104 | - Allows duplicate values. 105 | 106 | --- 107 | 108 | ### 4. **Full-Text Index** 109 | - Used for advanced text searches within large text fields. 110 | - Useful for searching specific words or phrases in large text columns. 111 | 112 | **Example**: Searching for the word "epic" in the `description` column of a table: 113 | ```sql 114 | SELECT * 115 | FROM film 116 | WHERE MATCH(description) AGAINST ('epic' IN NATURAL LANGUAGE MODE); 117 | ``` 118 | 119 | --- 120 | 121 | ## B-Tree 122 | 123 | A **B-Tree (Balanced Tree)** is a self-balancing tree data structure that maintains sorted data. It is commonly used as the underlying structure for most SQL indexes. 124 | 125 | ### Key Features: 126 | 1. **Efficient Query Types**: 127 | - **Range Queries**: Queries using operators like `<`, `>`, and `BETWEEN`. 128 | - **Equality Queries**: Queries using `=` or specific values. 129 | - **Ordered Queries**: Queries with `ORDER BY`. 130 | 131 | 2. **Fast Data Access**: 132 | - Enables logarithmic search time for indexed queries. 133 | 134 | **Diagram**: 135 | ![B-Tree Example](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/096/934/original/two.png?1732041770) 136 | 137 | --- 138 | 139 | ## Hash Indexes 140 | 141 | **Duration**: 10 minutes 142 | 143 | A **Hash Index** uses a hash function to convert a search key into a specific memory location in a hash table. 144 | 145 | ### Use Cases: 146 | 1. **Exact Matches**: 147 | - Ideal for queries using `=` or `IN`. 148 | 149 | 2. **Limitations**: 150 | - Not suitable for range queries (e.g., `<`, `>`, `BETWEEN`). 151 | 152 | **Example**: 153 | For exact matches, a hash index provides faster access compared to a B-Tree index. 154 | 155 | --- 156 | 157 | ## Key Optimization Commands 158 | 159 | **Duration**: 10 minutes 160 | 161 | ### 1. **ANALYZE** 162 | 163 | The `ANALYZE TABLE` command collects statistics about a table's content, helping the query optimizer make better decisions. 164 | 165 | **Syntax**: 166 | ```sql 167 | ANALYZE TABLE table_name; 168 | ``` 169 | 170 | --- 171 | 172 | ### 2. **OPTIMIZE** 173 | 174 | The `OPTIMIZE TABLE` command reorganizes the physical storage of data, reducing fragmentation and improving query performance. 175 | 176 | **Syntax**: 177 | ```sql 178 | OPTIMIZE TABLE table_name; 179 | ``` 180 | 181 | **Diagram**: 182 | ![OPTIMIZE TABLE Example](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/096/935/original/three.png?1732041854) 183 | 184 | --- 185 | 186 | ## Practical Examples and Best Practices 187 | 188 | **Message for Instructor**: 189 | - Demonstrate creating an index and querying with and without it to show performance improvements. 190 | - Highlight the impact of excessive indexing on write operations. 191 | 192 | --- 193 | 194 | ## Common SQL Optimization Interview Questions 195 | 196 | 1. **How do you identify slow queries in a database?** 197 | - Use `EXPLAIN` to analyze the query execution plan. 198 | - Check for full table scans and optimize accordingly. 199 | 200 | 2. **How to check if indexes are present for a query?** 201 | - Use `SHOW INDEX FROM table_name` or `EXPLAIN` query. 202 | 203 | 3. **How to avoid redundant data retrieval?** 204 | - Use proper joins, filter unnecessary columns, and avoid `SELECT *`. 205 | 206 | 4. **How to use aliases effectively in SQL queries?** 207 | - Simplify table or column names using aliases for better readability and reduced ambiguity. 208 | 209 | 5. **How to maintain optimized queries over time?** 210 | - Regularly analyze tables with `ANALYZE TABLE`. 211 | - Reorganize data storage with `OPTIMIZE TABLE`. 212 | - Update indexes as data evolves. 213 | 214 | --- 215 | 216 | ## Summary 217 | 218 | 1. **Indexing**: A powerful mechanism to speed up data retrieval, but with trade-offs in storage and maintenance. 219 | 2. **Types of Indexes**: 220 | - Primary Key (Clustered), Unique, Single Column, and Full-Text Indexes. 221 | 3. **B-Tree**: Supports range, equality, and ordered queries efficiently. 222 | 4. **Hash Indexes**: Optimized for exact matches but unsuitable for range queries. 223 | 5. **Optimization Commands**: 224 | - `ANALYZE TABLE` for better query planning. 225 | - `OPTIMIZE TABLE` for storage reorganization and defragmentation. 226 | 6. **Best Practices**: 227 | - Avoid excessive indexing to maintain a balance between query speed and write performance. 228 | - Regularly monitor and refine queries for long-term performance. 229 | ``` 230 | -------------------------------------------------------------------------------- /DBMS Module/Scaling,_ Networking, _and _Automation.md: -------------------------------------------------------------------------------- 1 | 2 | # Scaling, Networking, and Automation Notes 3 | 4 | ## **1. Horizontal vs Vertical Scaling** 5 | 6 | ### **Understanding Scaling Through a Real-World Analogy** 7 | - **Scenario**: Imagine a restaurant during a festive season. 8 | - The restaurant adds tables (e.g., increasing from 4 to 6) to accommodate more guests. 9 | - **Drawback**: Despite additional seating, delays in service occur due to overloading of the kitchen and staff. 10 | - **Solution**: 11 | - Expand physical space by adding new floors or outlets. 12 | - Hire more staff and upgrade the kitchen. 13 | - **Relation to DevOps**: 14 | - This reflects the two primary scaling strategies: 15 | 1. Adding capacity by expanding resources (Vertical Scaling). 16 | 2. Adding more units to share the load (Horizontal Scaling). 17 | 18 | 19 | ### **Concept of Scaling in DevOps** 20 | Scaling ensures that applications can handle increased traffic and workload efficiently. 21 | 22 | 1. **Horizontal Scaling** (Scaling Out): 23 | - Adds multiple servers to distribute traffic. 24 | - A load balancer is used to direct incoming requests to these servers. 25 | - Example: A web application serving thousands of users by distributing requests across multiple instances. 26 | 27 | 2. **Vertical Scaling** (Scaling Up): 28 | - Upgrades the existing server’s capacity (e.g., more RAM, CPU). 29 | - Ideal for monolithic applications that are not designed for distributed environments. 30 | 31 | 32 | ### **Comparison Between Horizontal and Vertical Scaling** 33 | 34 | | **Aspect** | **Horizontal Scaling** | **Vertical Scaling** | 35 | |-------------------------|--------------------------------------------------|--------------------------------------------------| 36 | | **Scaling Method** | Adds multiple servers to share the load | Increases resources (CPU, RAM) on one server | 37 | | **Limitation** | Complexity in management and synchronization | Physical hardware limits | 38 | | **Cost** | Higher initial setup cost, scales well | Expensive with diminishing returns | 39 | | **Use Case** | Distributed apps, high-read volume databases | Smaller apps with balanced read/write needs | 40 | 41 | 42 | ### **Traditional vs. Modern Scaling** 43 | - **Traditional Approach**: 44 | - Companies managed physical data centers with limited scalability options. 45 | - Scaling required manual intervention, including migrations to larger hardware. 46 | - **Modern Approach**: 47 | - Cloud providers (AWS, GCP) simplify scaling with: 48 | - Auto-scaling groups for Horizontal Scaling. 49 | - On-demand instance upgrades for Vertical Scaling. 50 | - Tools like Docker and Kubernetes enable easy Horizontal Scaling by running applications in containers. 51 | 52 | 53 | - Why are modern databases more suited to Horizontal Scaling? 54 | - **Answer**: Containers (via Docker and Kubernetes) allow applications to be easily replicated and deployed in distributed environments, ensuring fault tolerance and scalability. 55 | 56 | ## **2. Load Balancing: Nginx vs. HAProxy** 57 | 58 | ### **Load Balancer Functionality Overview** 59 | Load balancers distribute incoming traffic across servers to ensure: 60 | - High availability. 61 | - Fault tolerance. 62 | - Optimized performance. 63 | 64 | --- 65 | 66 | ### **Nginx** 67 | - **Analogy**: Think of Nginx as a multitasking station manager who directs trains to platforms while handling various administrative duties. 68 | - **Features**: 69 | - Optimized for HTTP/HTTPS (Layer 7) traffic. 70 | - Performs basic TCP load balancing but lacks advanced routing capabilities. 71 | - Efficient for handling low-to-moderate traffic. 72 | - **Limitations**: 73 | - Less robust health checks compared to specialized tools. 74 | - Struggles with large-scale traffic management. 75 | 76 | --- 77 | 78 | ### **HAProxy** 79 | - **Analogy**: HAProxy is like a dedicated railway traffic controller, specializing in efficiently routing trains (traffic) during peak times. 80 | - **Features**: 81 | - Optimized for TCP (Layer 4) load balancing with advanced algorithms. 82 | - Supports robust health checks for backend servers. 83 | - Handles high-traffic scenarios efficiently. 84 | - **Advantages Over Nginx**: 85 | - Prioritizes reliability in large-scale setups. 86 | - Better suited for applications requiring fine-grained traffic control. 87 | 88 | --- 89 | 90 | ### **Key Insight** 91 | Both tools have unique strengths: 92 | - **Nginx** excels at Layer 7 traffic and web server integration. 93 | - **HAProxy** is better for Layer 4 traffic and high-demand environments. 94 | 95 | --- 96 | 97 | ## **3. Practical Demonstration: Setting Up HAProxy for MySQL Load Balancing** 98 | 99 | ### **Overview** 100 | - **Objective**: Set up HAProxy to distribute MySQL traffic across backend servers. 101 | - **Setup**: 102 | - Frontend Server: Runs HAProxy (no MySQL installed). 103 | - Backend Server: Hosts MySQL database(s). 104 | 105 | --- 106 | 107 | ### **Step-by-Step Guide** 108 | 109 | #### 1. **Server Setup** 110 | - Log into two servers: 111 | - **Frontend Server**: Dedicated for HAProxy installation. 112 | - **Backend Server**: Running MySQL for database operations. 113 | - Verify MySQL installation on the backend: 114 | ```bash 115 | sudo mysql 116 | ``` 117 | 118 | --- 119 | 120 | #### 2. **Installing HAProxy** 121 | - Update the frontend server: 122 | ```bash 123 | sudo apt update 124 | sudo apt install haproxy -y 125 | ``` 126 | 127 | --- 128 | 129 | #### 3. **Configuring HAProxy** 130 | - Open HAProxy configuration file: 131 | ```bash 132 | sudo nano /etc/haproxy/haproxy.cfg 133 | ``` 134 | - Add the following configuration: 135 | ```haproxy 136 | frontend mysql_front 137 | bind *:3306 138 | default_backend mysql_servers 139 | 140 | backend mysql_servers 141 | mode tcp 142 | option tcplog 143 | balance roundrobin 144 | server mysql_backend :3306 check 145 | ``` 146 | - **Explanation**: 147 | - `frontend`: Defines the entry point for MySQL traffic (port 3306). 148 | - `backend`: Lists MySQL servers with IP addresses for traffic distribution. 149 | 150 | --- 151 | 152 | #### 4. **Creating a MySQL User for HAProxy** 153 | - Create a user for HAProxy to perform health checks: 154 | ```sql 155 | CREATE USER 'haproxy_check'@'%' IDENTIFIED BY 'password'; 156 | GRANT REPLICATION CLIENT ON *.* TO 'haproxy_check'@'%'; 157 | FLUSH PRIVILEGES; 158 | ``` 159 | 160 | --- 161 | 162 | #### 5. **Testing Connectivity** 163 | - Test connectivity between HAProxy and the backend server: 164 | ```bash 165 | ping 166 | telnet 3306 167 | ``` 168 | 169 | --- 170 | 171 | #### 6. **Installing MySQL Client on HAProxy** 172 | - Install the MySQL client utility: 173 | ```bash 174 | sudo apt install mysql-server 175 | sudo systemctl status mysql 176 | ``` 177 | 178 | --- 179 | 180 | #### 7. **Testing the Setup** 181 | - Connect to the MySQL backend through HAProxy: 182 | ```bash 183 | mysql -h localhost -u haproxy_check -p 184 | ``` 185 | 186 | --- 187 | 188 | #### 8. **Adding More Backend Servers** 189 | - Update the HAProxy configuration to include additional backend servers: 190 | ```haproxy 191 | server mysql_backend_2 :3306 check 192 | ``` 193 | - Create the `haproxy_check` user on each new server. 194 | 195 | --- 196 | 197 | ### **Advantages of HAProxy Load Balancing** 198 | - Distributes load effectively, reducing the risk of server overloading. 199 | - Simplifies scaling by allowing new servers to be added with minimal changes. 200 | - Provides robust health checks to monitor backend server availability. 201 | -------------------------------------------------------------------------------- /DBMS Module/check.md: -------------------------------------------------------------------------------- 1 | Content here! 2 | -------------------------------------------------------------------------------- /DevOps Tools 1 Module/Deployment in Kubernetes - Kustomize.md: -------------------------------------------------------------------------------- 1 | 2 | # Deployment in Kubernetes - Kustomize 3 | 4 | ### Topics Covered: 5 | - **Kustomize**: Overview and introduction to its purpose in Kubernetes. 6 | - **Practical Demonstration**: Hands-on demo showcasing how to use Kustomize effectively. 7 | - **Real-life Scenarios**: Practical examples of how Kustomize is used in production settings. 8 | - **Interview Questions**: Key questions that might arise in interviews related to Kustomize. 9 | 10 | --- 11 | 12 | ## Kustomize 13 | 14 | ### What is Kustomize? 15 | Kustomize is a **Kubernetes-native configuration management tool** that allows you to customize Kubernetes resource YAML files without modifying the original files. It dynamically applies configurations at runtime, making it easier to maintain multiple environment configurations, such as development, staging, and production. 16 | 17 | --- 18 | 19 | ### Working with Kustomize 20 | 21 | #### Pre-existing YAML Files: 22 | - You typically start with Kubernetes resource files: 23 | - `deployment.yaml` 24 | - `service.yaml` 25 | - `configmap.yaml` 26 | 27 | #### Adding Kustomize: 28 | 1. Create a new file named **`kustomization.yaml`**. 29 | 2. Add your existing resource files to this file. For example: 30 | ```yaml 31 | resources: 32 | - deployment.yaml 33 | - service.yaml 34 | ``` 35 | 3. Kustomize dynamically applies the configurations defined in the `kustomization.yaml` file to your original resource files **at runtime**. 36 | 37 | #### Running Kustomize: 38 | Use the following commands to execute Kustomize: 39 | - **View transformed YAML**: 40 | `kubectl kustomize .` 41 | - **Apply configurations directly to your cluster**: 42 | `kubectl apply -k .` 43 | - **Chain commands for flexibility**: 44 | `kubectl kustomize . | kubectl apply -f` 45 | 46 | --- 47 | 48 | ### Demonstration (Instructor Focus) 49 | 50 | **Steps to demonstrate Kustomize:** 51 | 1. Create a `kustomization.yaml` file in the same directory as your resource files. 52 | 2. Show how parameters in the `kustomization.yaml` file are applied dynamically to existing files. 53 | 3. Explain the command outputs and how they affect the Kubernetes cluster. 54 | 55 | Example `kustomization.yaml` file: 56 | ```yaml 57 | resources: 58 | - deployment.yaml 59 | - service.yaml 60 | ``` 61 | 62 | --- 63 | 64 | ### Organizing Kustomize with Directories 65 | 66 | To manage environment-specific configurations, use the following directory structure: 67 | 68 | #### Directory Structure: 69 | - **`base` directory**: 70 | - Contains common YAML files, such as: 71 | - `deployment.yaml` 72 | - `service.yaml` 73 | - `configmap.yaml` 74 | - `kustomization.yaml` 75 | - **`overlays` directory**: 76 | - Contains environment-specific configurations: 77 | - `dev/` 78 | - `staging/` 79 | - `prod/` 80 | 81 | #### Example: 82 | 1. Move all base resources (`deployment.yaml`, `service.yaml`, etc.) into the `base` directory. 83 | 2. In each environment folder under `overlays`, create a `kustomization.yaml` file that references the base directory: 84 | ```yaml 85 | resources: 86 | - ../../base 87 | ``` 88 | 3. Apply patches for specific environments using **`patchesStrategicMerge`**: 89 | - Add a new file `deployment-patch.yaml` inside the specific environment folder. 90 | - Reference this file in the `kustomization.yaml` for the environment: 91 | ```yaml 92 | patchesStrategicMerge: 93 | - deployment-patch.yaml 94 | ``` 95 | 96 | #### Applying the Configuration: 97 | - To apply the dev configuration: 98 | ```bash 99 | kubectl apply -k overlays/dev/ 100 | ``` 101 | 102 | ![Kustomization Example](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/104/590/original/kustomization.png?1737643403) 103 | 104 | For a more detailed example, refer to the following structure: 105 | 106 | ![Dev Environment Example](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/104/598/original/devkustomize.png?1737643872) 107 | 108 | --- 109 | 110 | ## Understanding Kustomize Terminologies 111 | 112 | ### Directory Structure: 113 | 1. **Base Directory**: 114 | - Contains shared resources for all environments. 115 | - Files: 116 | - `base-deployment.yaml` 117 | - `kustomization.yaml` 118 | 2. **Overlays Directory**: 119 | - Contains environment-specific configurations. 120 | - Files: 121 | - `kustomization.yaml` 122 | - `patch.yaml` 123 | 124 | ### Key Features of `kustomization.yaml`: 125 | - **ConfigMap Generator**: 126 | Dynamically generate ConfigMaps: 127 | ```yaml 128 | configMapGenerator: 129 | - name: my-configmap 130 | literals: 131 | - key=value 132 | ``` 133 | - **Secret Generator**: 134 | Similar to ConfigMap generator but for secrets: 135 | ```yaml 136 | secretGenerator: 137 | ``` 138 | - **Common Labels and Annotations**: 139 | Add labels/annotations globally: 140 | ```yaml 141 | commonLabels: 142 | app: my-app 143 | ``` 144 | - **Cross-cutting Fields**: 145 | Fields like common labels, annotations, or namespaces defined in the `kustomization.yaml` file will apply across all resources. 146 | 147 | ### Patching Options: 148 | - **Strategic Merge Patches**: 149 | Use the `patchesStrategicMerge` field to define YAML patches that selectively modify base resources. 150 | - **JSON Patches**: 151 | Use the `patchJson6902` field for JSON-style patching. 152 | 153 | ### Variables: 154 | Kustomize supports variables using the `$` syntax. 155 | 156 | --- 157 | 158 | ## Kustomize Examples 159 | 160 | ### Scenario 1: Multi-Datacenter Deployment 161 | Suppose a company has four datacenters running an Nginx application. The environments are: 162 | - Preview 163 | - Sales 164 | - Production 165 | 166 | #### Configuration: 167 | 1. The **base directory** contains common Kubernetes resources: 168 | - Deployments 169 | - Services 170 | - ConfigMaps 171 | 2. Each datacenter has environment-specific configurations, including: 172 | - Environment variables 173 | - Secrets 174 | - Persistent Volumes (PV) and Persistent Volume Claims (PVC) 175 | - Role-based Access Control (RBAC) roles 176 | 177 | Kustomize enables you to define the shared resources in the **base** directory and override environment-specific values in the respective **overlay** directories. 178 | 179 | --- 180 | 181 | ### Real-Life Example: 182 | Consider a scenario with multiple clusters spread across different AWS regions. The challenges include: 183 | - Managing **compliance issues**. 184 | - Supporting **multi-tenancy** for different customers. 185 | - Enabling **region-specific features**. 186 | 187 | Kustomize simplifies this by: 188 | - Providing a common base configuration for all clusters. 189 | - Allowing region-specific customizations using overlays. 190 | 191 | --- 192 | 193 | ## Helm vs Kustomize 194 | 195 | ### Comparison Table: 196 | 197 | | Feature | **Helm** | **Kustomize** | 198 | |------------------|---------------------------------------|----------------------------------------| 199 | | **Type** | Package Manager for Kubernetes | Kubernetes Native Configuration Tool | 200 | | **Templating** | Yes (with Go templates) | No | 201 | | **Overlays** | Uses values files for customization | Uses patch-based overlays | 202 | | **Releases** | Manages lifecycle (install/upgrade) | Doesn’t manage state, just config | 203 | | **Dependencies** | Supports chart dependencies | No built-in support | 204 | | **Complexity** | Can be complex for simple cases | Simpler, focuses on YAML transformation| 205 | | **Community** | Large ecosystem (charts) | Part of Kubernetes (kubectl integration)| 206 | | **Versioning** | Yes | No, relies on Git for version control | 207 | 208 | ### Key Takeaways: 209 | - **Helm**: Ideal for managing complex applications with lifecycle management. 210 | - **Kustomize**: Best for lightweight YAML transformations without requiring templating. 211 | 212 | --- 213 | ``` 214 | -------------------------------------------------------------------------------- /DevOps Tools 1 Module/Docker Container Networking and Security.md: -------------------------------------------------------------------------------- 1 | # **Docker Container Networking and Security** 2 | 3 | This document provides a detailed discussion on **Docker Storage**, **Storage Drivers**, **NFS (Network File System)**, and **Docker Networking**. It explains how data is stored and managed in Docker containers and how Docker networking enables interaction between isolated containers. 4 | 5 | --- 6 | 7 | ## **Topics to be discussed:** 8 | 9 | 1. Docker Storage 10 | 2. Storage Drivers 11 | 3. NFS (Network File System) 12 | 4. Docker Networking 13 | 14 | --- 15 | 16 | # **Docker Storage** 17 | 18 | Docker Storage refers to the methods used to **persist data** in Docker containers and manage **container file systems**. 19 | 20 | --- 21 | 22 | ### **Why Do We Need Storage in Docker?** 23 | 1. **Data Persistence**: Ensures that data remains available even if the container is deleted. 24 | 2. **Scalability and Flexibility**: Allows storage to be adjusted as needed for various applications. 25 | 3. **Efficient Resource Management**: Prevents duplication of data across containers. 26 | 4. **High Availability and Reliability**: Ensures data remains accessible during container failures. 27 | 28 | --- 29 | 30 | ### **Types of Docker Storage** 31 | 32 | 1. **Volumes** 33 | - Volumes are **directories** or **files** outside the container’s file system, managed by Docker. 34 | - They are **independent of the container’s lifecycle**. 35 | - Even if the container is deleted, the volume persists. 36 | 37 | **Command to Create a Volume**: 38 | ```bash 39 | docker volume create my-volume 40 | ``` 41 | 42 | **Command to Use a Volume**: 43 | ```bash 44 | docker run -d -v my-volume:/data 45 | ``` 46 | 47 | **Example Illustration**: 48 | ![Docker Storage](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/100/523/original/DockerStorage.png?1734448525) 49 | 50 | --- 51 | 52 | 2. **Bind Mounts** 53 | - **Bind mounts** map a **host directory** to a **container directory**. 54 | - Changes in either directory will reflect in the other. 55 | 56 | **Command to Use Bind Mounts**: 57 | ```bash 58 | docker run -d -v /host/data:/container/data my_image 59 | ``` 60 | 61 | --- 62 | 63 | 3. **Tmpfs Mounts** 64 | - **Temporary file systems** that store data in **RAM** instead of disk storage. 65 | - Data is **lost** when the container stops. 66 | - Useful for **caching** and **temporary data**. 67 | 68 | **Command to Use Tmpfs Mounts**: 69 | ```bash 70 | docker run -d --tmpfs /container/tmp my_image 71 | ``` 72 | 73 | --- 74 | 75 | # **Storage Drivers** 76 | 77 | Docker uses **Storage Drivers** to manage how data is **stored and accessed** in the container’s filesystem. 78 | 79 | --- 80 | 81 | ### **Types of Storage Drivers** 82 | 83 | 1. **Overlay2 (Default Driver)** 84 | - Uses **Copy-on-Write** mechanism. 85 | - A **base image** is shared across multiple containers, with each container having its own **read-write layer**. 86 | 87 | **Example:** 88 | - Containers **C1**, **C2**, and **C3** use the same base image layer, but each has a unique **read-write layer** for modifications. 89 | 90 | --- 91 | 92 | 2. **AUFS (Advanced Multilayer Universal File System)** 93 | - Supports **Copy-on-Write**. 94 | - Combines multiple layers of files into a **unified view**. 95 | 96 | --- 97 | 98 | 3. **Device Mapper** 99 | - Uses **block-level operations** to allocate storage. 100 | - Ideal for systems that need **block-level management** and **high performance**. 101 | 102 | --- 103 | 104 | ### **Important Command: Cleaning Unused Volumes** 105 | ```bash 106 | docker volume prune 107 | ``` 108 | - Deletes **unused volumes** to free up space. 109 | 110 | --- 111 | 112 | # **NFS (Network File System)** 113 | 114 | ### **What is NFS?** 115 | NFS is a **shared storage solution** that allows multiple machines to **access the same storage** over a network. 116 | 117 | --- 118 | 119 | ### **When to Use NFS?** 120 | - When you have **multiple machines** and want them to use **shared storage**. 121 | - **Example:** 122 | - You have **3 servers** that need to access the **same database files**. 123 | 124 | --- 125 | 126 | ### **Setup Steps for NFS** 127 | 1. **Install NFS Kernel on the Server** 128 | ```bash 129 | apt-get install nfs-kernel-server 130 | ``` 131 | 2. **Install NFS on Client Machines** 132 | ```bash 133 | apt-get install nfs-common 134 | ``` 135 | 136 | --- 137 | 138 | # **Docker Networking** 139 | 140 | Docker Networking allows **containers** to communicate with: 141 | - Each other. 142 | - The **host machine**. 143 | - **External networks**. 144 | 145 | --- 146 | 147 | ### **Why is Docker Networking Important?** 148 | - Docker containers are **isolated** by default. 149 | - Networking provides a **logical layer** for containers to **interact** securely. 150 | 151 | --- 152 | 153 | ### **Types of Docker Networks** 154 | 155 | 1. **Bridge Network** 156 | - The **default network** for Docker containers. 157 | - Containers can communicate **within the same host**. 158 | - Ideal for **local development**. 159 | 160 | 2. **Host Network** 161 | - Removes the **network isolation** between the container and the host. 162 | - The container shares the **host’s network stack**. 163 | 164 | 3. **None Network** 165 | - Completely disables networking for the container. 166 | - Ideal for **security-focused** applications. 167 | 168 | 4. **Overlay Network** 169 | - Used in **Docker Swarm** to enable communication between containers across **multiple hosts**. 170 | 171 | 5. **Macvlan Network** 172 | - Assigns a **unique MAC address** to each container. 173 | - Containers appear as **separate devices** on the network. 174 | 175 | --- 176 | 177 | # **Summary of Key Commands** 178 | 179 | | **Command** | **Description** | 180 | |----------------------------------------|-----------------------------------------------------| 181 | | `docker volume create ` | Creates a Docker volume. | 182 | | `docker run -d -v :/path` | Mounts a volume inside the container. | 183 | | `docker volume prune` | Removes unused volumes. | 184 | | `docker network create ` | Creates a new Docker network. | 185 | | `docker network ls` | Lists all Docker networks. | 186 | | `docker network connect ` | Connects a container to a network. | 187 | | `docker run --network=` | Runs a container in a specific network. | 188 | 189 | --- 190 | 191 | # **Example Docker Networking Commands** 192 | 193 | 1. **Creating a Bridge Network** 194 | ```bash 195 | docker network create my_bridge 196 | ``` 197 | 198 | 2. **Running a Container in a Custom Network** 199 | ```bash 200 | docker run -d --name my_container --network=my_bridge nginx 201 | ``` 202 | 203 | 3. **Inspecting a Network** 204 | ```bash 205 | docker network inspect my_bridge 206 | ``` 207 | 208 | 4. **Connecting an Existing Container to a Network** 209 | ```bash 210 | docker network connect my_bridge my_container 211 | ``` 212 | 213 | --- 214 | 215 | # **Conclusion** 216 | 217 | In this session, we explored: 218 | - **Docker Storage** methods for persisting data in containers. 219 | - **Storage Drivers** used to manage container filesystems. 220 | - **NFS (Network File System)** for shared storage across machines. 221 | - **Docker Networking** to enable communication between containers and external systems. 222 | -------------------------------------------------------------------------------- /DevOps Tools 1 Module/Docker and Docker Swarm.md: -------------------------------------------------------------------------------- 1 | # Typed Notes of Docker and Docker Swarm 2 | 3 | ### 📘 **How Dockerfile Works?** 4 | 5 | A Dockerfile is a script with a set of instructions used to automate the creation of Docker images. Here’s a conceptual breakdown: 6 | 7 | - In a typical Linux environment, developers often run **shell scripts** or **Python applications** directly on the machine. 8 | - However, Docker introduces **containers**, which serve as isolated environments to run applications. 9 | - With a **Dockerfile**, we can: 10 | 1. Copy an application (e.g., a Python script) into a container. 11 | 2. Run the application inside the container. 12 | 13 | This means the container acts as a **lightweight virtual machine**. 14 | To run a container, you need: 15 | 1. **Application Code** 16 | 2. **Dockerfile** – Docker will consume the application code and create an image based on the Dockerfile. 17 | 18 | --- 19 | 20 | ### 🧩 **Components of a Dockerfile** 21 | 22 | #### 1️⃣ **EXPOSE** 23 | - The `EXPOSE` instruction informs Docker about the ports on which the container will listen at runtime. 24 | 25 | **Difference between EXPOSE and `-p` flag:** 26 | | EXPOSE | -p (publish) | 27 | |----------------|----------------------------| 28 | | Informs Docker about the port | Maps the container port to the host machine | 29 | | Works internally within Docker | Exposes the port externally to users | 30 | 31 | #### 2️⃣ **HEALTH CHECK** 32 | - Used to ensure the container is working as expected. 33 | Example: 34 | ```bash 35 | HEALTH CHECK --interval=30s --timeout=10s --retries=3 CMD curl http://localhost 36 | ``` 37 | This runs a health check every **30 seconds**, times out in **10 seconds**, and retries up to **3 times**. 38 | 39 | --- 40 | 41 | #### 3️⃣ **ARG** (Build-time Variables) 42 | - Defines variables that can be passed during the **build process** using the `docker build` command. 43 | 44 | Example: 45 | ```dockerfile 46 | ARG build_version 47 | RUN echo "version $build_version" 48 | ``` 49 | 50 | Usage during build: 51 | ```bash 52 | docker build --build-arg build_version=1.0 . 53 | ``` 54 | 55 | --- 56 | 57 | #### 4️⃣ **USER** 58 | - The `USER` instruction specifies which user will execute subsequent Dockerfile commands. 59 | This is useful for running commands with **different permissions**. 60 | 61 | 📷 *Diagram: USER Instruction* 62 | ![Docker USER Command](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/102/962/original/dockeruser.png?1736599792) 63 | 64 | --- 65 | 66 | #### 5️⃣ **Multi-Stage Builds** 67 | - **Multi-stage builds** help create smaller and more efficient Docker images by using multiple `FROM` statements in the same Dockerfile. 68 | - The **artifacts** generated in one stage can be copied into the final image, reducing image size. 69 | 70 | 📷 *Diagram: Multi-Stage Builds* 71 | ![Multi-Stage Builds](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/102/963/original/dockermultistagebuilds.png?1736600048) 72 | 73 | --- 74 | 75 | #### 6️⃣ **INSPECT** 76 | - The `docker inspect` command provides detailed information about a container or image, including its configuration, state, and network settings. 77 | 78 | Common usage examples: 79 | ```bash 80 | docker inspect container_id 81 | docker inspect --format='{{.NetworkSettings.IPAddress}}' container_id 82 | ``` 83 | The second command returns the **IP address** of the container. 84 | 85 | --- 86 | 87 | # 🔧 **Building Efficient Images** 88 | 89 | To optimize Docker images, follow these best practices: 90 | 91 | ### ✅ **Best Practices for Building Efficient Docker Images** 92 | 93 | 1. **Use Multi-Stage Builds** 94 | - Reduce image size by building artifacts in one stage and copying them to the final stage. 95 | 96 | 2. **Minimize the Number of Layers** 97 | - Instead of creating a new layer for each instruction, combine related instructions into a single `RUN` command. 98 | 99 | **Example:** 100 | ❌ Incorrect approach: 101 | ```dockerfile 102 | RUN apt-get update 103 | RUN apt-get install curl 104 | RUN apt-get install vim 105 | ``` 106 | 107 | ✅ Correct approach: 108 | ```dockerfile 109 | RUN apt-get update && apt-get install -y curl vim 110 | ``` 111 | 112 | 3. **Use a `.dockerignore` File** 113 | - Exclude unnecessary files and directories from the image to reduce its size. 114 | 115 | 4. **Avoid Installing Unnecessary Packages** 116 | - Only install packages that are essential for your application. 117 | 118 | 5. **Clean Up After Installations** 119 | - Remove cached files or unnecessary dependencies after installations. 120 | 121 | 6. **Use `ARG` for Build-Time Variables** 122 | - Leverage `ARG` to pass variables during the build process. 123 | 124 | --- 125 | 126 | # 📏 **Flattening a Docker Image** 127 | 128 | 129 | ### 🛠️ **What is Flattening a Docker Image?** 130 | 131 | Flattening a Docker image involves combining all the **layers** of an image into a **single layer**. This can help simplify images and optimize them for deployment. 132 | 133 | ### 📋 **Why Flatten an Image?** 134 | 1. **Reduce Complexity** 135 | - Fewer layers mean a simpler image structure. 136 | 2. **Reduce Size** 137 | - Flattening can help remove unnecessary layers and reduce image size. 138 | 3. **Increase Security** 139 | - Reduces the risk of leftover sensitive files in layers. 140 | 4. **Optimize for Deployment** 141 | - Faster deployment due to reduced size and complexity. 142 | 143 | --- 144 | 145 | ### 📘 **Steps to Flatten an Image** 146 | 147 | 1. **Run the Image and Create a Container** 148 | ```bash 149 | docker run --name my_container my_image 150 | ``` 151 | 152 | 2. **Export the Container Filesystem** 153 | ```bash 154 | docker export my_container -o flat_image.tar 155 | ``` 156 | 157 | 3. **Import the Flattened Image as a New Image** 158 | ```bash 159 | docker import flat_image.tar my_flat_image 160 | ``` 161 | 162 | --- 163 | 164 | # 📌 **Summary of Key Commands** 165 | | **Command** | **Description** | 166 | |--------------------------|----------------------------------------------| 167 | | `EXPOSE 80` | Informs Docker that the container listens on port 80 | 168 | | `HEALTH CHECK` | Adds a health check to verify container health | 169 | | `ARG build_version` | Defines a build-time variable | 170 | | `USER myuser` | Specifies the user to run subsequent commands | 171 | | `docker inspect` | Inspects a container or image configuration | 172 | | `docker run` | Runs a Docker container | 173 | | `docker export` | Exports a container filesystem as a tar file | 174 | | `docker import` | Imports a tar file as a new Docker image | 175 | -------------------------------------------------------------------------------- /DevOps Tools 1 Module/Kubernetes - Helm.md: -------------------------------------------------------------------------------- 1 | # Kubernetes - Helm Notes 2 | 3 | 4 | ## **What is Helm?** 5 | Helm is a package manager for Kubernetes that simplifies the deployment, management, and scaling of applications. It uses preconfigured templates, called Helm Charts, to automate the process of creating Kubernetes resources. 6 | 7 | ### **Why Use Helm?** 8 | - **Simplifies Kubernetes deployments**: Automates the creation and management of Kubernetes resources. 9 | - **Provides version control**: Manage different versions of your application. 10 | - **Reusable templates**: Create consistent and repeatable deployments. 11 | 12 | --- 13 | 14 | ## **Key Concepts in Helm** 15 | 16 | ### **1. Helm Charts** 17 | Helm Charts are collections of templates that define Kubernetes resources. These charts contain all the necessary files to deploy an application on Kubernetes. 18 | 19 | #### **Structure of a Helm Chart** 20 | - **Chart.yaml**: Contains metadata about the chart (name, version, description). 21 | - **Values.yaml**: Contains default configuration values for the chart. 22 | - **Templates/**: Directory containing YAML templates for Kubernetes resources. 23 | 24 | #### **Example of Chart.yaml:** 25 | ```yaml 26 | apiVersion: v2 27 | name: my-application 28 | version: 1.0.0 29 | description: A Helm chart for my application 30 | ``` 31 | 32 | ### **2. Helm Repositories** 33 | Helm Repositories are storage locations for Helm Charts. These repositories can be public or private and allow you to store and retrieve charts for your deployments. 34 | 35 | #### **Popular Helm Repositories:** 36 | - Bitnami 37 | - ArtifactHub 38 | - Helm Stable Repository 39 | 40 | ### **3. Helm Releases** 41 | A Helm Release is a specific deployment of a Helm Chart. Every time a chart is deployed, it is tracked as a release. 42 | 43 | --- 44 | 45 | ## **Key Features of Helm** 46 | 47 | | **Feature** | **Description** | 48 | |-----------------------|-------------------------------------------------| 49 | | Simplified Deployment | Automates Kubernetes resource creation | 50 | | Version Control | Tracks application versions and rollbacks | 51 | | Reusable Templates | Creates consistent configurations using charts | 52 | | Scalability | Easily manage application scaling | 53 | 54 | --- 55 | 56 | ## **Helm Workflow** 57 | The typical workflow for using Helm involves creating charts, installing releases, upgrading them, and rolling back if necessary. 58 | 59 | ### **Step 1: Create a Chart** 60 | ```bash 61 | helm create my-chart 62 | ``` 63 | This command generates a basic chart structure with the necessary files. 64 | 65 | ### **Step 2: Install a Release** 66 | ```bash 67 | helm install 68 | ``` 69 | This command installs a release from the specified chart. 70 | 71 | #### **Example:** 72 | ```bash 73 | helm install my-app ./my-chart 74 | ``` 75 | 76 | ### **Step 3: Upgrade a Release** 77 | ```bash 78 | helm upgrade 79 | ``` 80 | This command upgrades an existing release with new chart configurations. 81 | 82 | #### **Example:** 83 | ```bash 84 | helm upgrade my-app ./my-chart 85 | ``` 86 | 87 | ### **Step 4: Rollback a Release** 88 | ```bash 89 | helm rollback 90 | ``` 91 | This command rolls back a release to a previous revision. 92 | 93 | #### **Example:** 94 | ```bash 95 | helm rollback my-app 1 96 | ``` 97 | 98 | --- 99 | 100 | ## **File Descriptions in Helm** 101 | 102 | ### **1. Chart.yaml** 103 | This file contains metadata about the Helm Chart. 104 | 105 | #### **Example:** 106 | ```yaml 107 | apiVersion: v2 108 | name: my-chart 109 | version: 1.0.0 110 | description: A sample Helm chart 111 | ``` 112 | 113 | ### **2. Values.yaml** 114 | Contains the default configuration values for the chart. 115 | 116 | #### **Example:** 117 | ```yaml 118 | replicaCount: 2 119 | image: 120 | repository: nginx 121 | tag: latest 122 | pullPolicy: IfNotPresent 123 | service: 124 | type: ClusterIP 125 | port: 80 126 | ``` 127 | 128 | ### **3. Templates/** 129 | Contains Kubernetes resource templates. 130 | 131 | #### **Example Template:** ConfigMap 132 | ```yaml 133 | apiVersion: v1 134 | kind: ConfigMap 135 | metadata: 136 | name: {{ .Release.Name }}-configmap 137 | data: 138 | myValue: "Example" 139 | ``` 140 | 141 | --- 142 | 143 | ## **Pre-existing Helm Charts** 144 | There are several pre-existing Helm Charts available in repositories: 145 | 146 | | **Chart** | **Description** | 147 | |-----------------|--------------------------------------| 148 | | Nginx | Web server | 149 | | Prometheus | Monitoring system | 150 | | Grafana | Dashboard and visualization tool | 151 | | Jenkins | Continuous Integration server | 152 | | MySQL | Database server | 153 | 154 | --- 155 | 156 | ## **Helm Commands** 157 | 158 | | **Command** | **Description** | 159 | |--------------------------------------|-----------------------------------------------| 160 | | `helm install ` | Install a Helm chart | 161 | | `helm upgrade ` | Upgrade an existing release | 162 | | `helm rollback ` | Rollback a release to a previous revision | 163 | | `helm package ` | Package a chart into a `.tgz` archive | 164 | | `helm push ` | Push a chart to a repository | 165 | 166 | --- 167 | 168 | ## **Helm Templates** 169 | Helm templates use Go template syntax to create dynamic configurations. 170 | 171 | ### **1. Helper Templates** 172 | Helper templates are reusable components defined in a `_helpers.tpl` file. 173 | 174 | #### **Example:** 175 | ```yaml 176 | {{- define "my-chart.fullname" -}} 177 | {{ .Release.Name }}-{{ .Chart.Name }} 178 | {{- end -}} 179 | ``` 180 | 181 | ### **2. Dynamic Names in Templates** 182 | Dynamic names use variables to customize resource names. 183 | 184 | #### **Examples:** 185 | ```yaml 186 | metadata: 187 | name: {{ .Release.Name }}-configmap 188 | ``` 189 | ```yaml 190 | metadata: 191 | name: {{ .Release.Name | lower }}-deployment 192 | ``` 193 | 194 | ### **3. Conditional Rendering** 195 | Conditional rendering allows templates to include or exclude configurations based on values. 196 | 197 | #### **Example:** 198 | ```yaml 199 | selector: 200 | app: my-app 201 | {{- if .Values.service.external }} 202 | externalIPs: 203 | - {{ .Values.service.external }} 204 | {{- end }} 205 | ``` 206 | 207 | --- 208 | 209 | ## **Deployment Example: Nginx** 210 | 211 | 1. **Install Nginx using Helm:** 212 | ```bash 213 | helm install my-nginx bitnami/nginx 214 | ``` 215 | 216 | 2. **Customize using values.yaml:** 217 | ```yaml 218 | replicaCount: 2 219 | service: 220 | type: LoadBalancer 221 | ``` 222 | 223 | --- 224 | 225 | ## **Deployment Example: Prometheus and Grafana** 226 | Use existing Helm charts to set up monitoring tools. 227 | 228 | 1. **Install Prometheus:** 229 | ```bash 230 | helm install prometheus prometheus-community/prometheus 231 | ``` 232 | 233 | 2. **Install Grafana:** 234 | ```bash 235 | helm install grafana grafana/grafana 236 | ``` 237 | 238 | --- 239 | 240 | ## **Summary** 241 | Helm simplifies Kubernetes application deployments by providing reusable charts, version control, and dynamic templates. It reduces the complexity of managing Kubernetes resources and ensures consistent configurations across environments. 242 | -------------------------------------------------------------------------------- /DevOps Tools 1 Module/Kubernetes API Versioning, K8s Extension , Certification Tip.md: -------------------------------------------------------------------------------- 1 | ## Kubernetes API Versioning, K8s Extension , Certification Tip: Lecture Notes 2 | 3 | 4 | 5 | ## Admission Controllers 6 | 7 | 8 | ### **Purpose:** 9 | Admission controllers are plugins that enforce policies on objects being created or updated in a Kubernetes cluster. These policies are essential to maintain security, resource limits, and operational efficiency. 10 | 11 | ### **Types of Admission Controllers:** 12 | 1. **Mutating Admission Controller:** 13 | - Modifies requests to enforce certain policies before they are accepted into the cluster. 14 | - Example: Automatically adding labels to pods. 15 | 16 | 2. **Validating Admission Controller:** 17 | - Ensures requests meet certain criteria and validates configurations. 18 | - Example: Rejecting deployments with insecure configurations. 19 | 20 | ![image](https://hackmd.io/_uploads/SkOaSVgU1g.png) 21 | 22 | ### **Commonly Used Admission Controllers:** 23 | | Controller | Purpose | 24 | |--------------------|------------------------------------------------------| 25 | | Namespace Lifecycle | Prevents operations on resources in non-existent namespaces | 26 | | Limit Ranger | Enforces default resource limits | 27 | | Resource Quota | Enforces quotas on the number of resources used | 28 | 29 | --- 30 | 31 | ## Kubernetes API Versions 32 | 33 | ### **Stages of API Versions:** 34 | 1. **Alpha:** 35 | - Experimental features. 36 | - May change or be removed in future releases. 37 | - **Not recommended** for production use. 38 | 39 | 2. **Beta:** 40 | - Features have broader testing. 41 | - Guaranteed not to be removed. 42 | - May have minor changes. 43 | 44 | 3. **Stable:** 45 | - Fully tested and ready for production. 46 | - No further breaking changes. 47 | - Example: `v1`. 48 | 49 | ### **Examples of Kubernetes API Versions:** 50 | 1. **v1:** Core stable API version including resources like: 51 | - Pods 52 | - Services 53 | - ConfigMaps 54 | 55 | 2. **Other API Groups:** 56 | - `apps/v1` for Deployments 57 | - `batch/v1` for Jobs and CronJobs 58 | 59 | --- 60 | 61 | ## Kubernetes Resources and Autoscaling 62 | 63 | 64 | ### **Core Resources:** 65 | 1. **Pods:** 66 | - Smallest deployable units in Kubernetes. 67 | - Represents a group of containers with shared storage, network, and specifications. 68 | 69 | 2. **Services:** 70 | - Abstracts and exposes pods as network services. 71 | - Ensures stable communication between pods and users. 72 | 73 | 3. **ConfigMaps:** 74 | - Stores non-sensitive configuration data in a key-value format. 75 | 76 | 4. **Secrets:** 77 | - Stores sensitive data like passwords and tokens securely. 78 | 79 | 5. **Namespaces:** 80 | - Provides a mechanism to isolate groups of resources within a cluster. 81 | 82 | 6. **Persistent Volumes (PVs):** 83 | - Provides storage resources for pods, independent of their lifecycle. 84 | 85 | ### **Workload Resources (apps/v1, batch/v1):** 86 | 1. **Deployment:** 87 | - Manages replicated applications. 88 | - Supports rolling updates. 89 | 90 | 2. **ReplicaSet:** 91 | - Ensures a specified number of pod replicas are running at any time. 92 | 93 | 3. **StatefulSet:** 94 | - Manages stateful applications with stable network identities and persistent storage. 95 | 96 | 4. **DaemonSet:** 97 | - Ensures a copy of a pod runs on all or selected nodes in the cluster. 98 | 99 | 5. **Job:** 100 | - Manages batch jobs. 101 | - Ensures specified tasks are completed successfully. 102 | 103 | 6. **CronJob:** 104 | - Schedules jobs to run at specific times or intervals. 105 | 106 | ### **Networking Resources (networking.k8s.io):** 107 | 1. **Ingress:** 108 | - Manages external access to services within the cluster. 109 | - Acts as an HTTP/HTTPS proxy. 110 | 111 | 2. **Network Policy:** 112 | - Controls traffic flow between pods. 113 | - Example: Allowing only specific pods to communicate. 114 | 115 | 3. **Service:** 116 | - Exposes a set of pods and ensures stable networking. 117 | 118 | ### **Storage Resources:** 119 | 1. **Storage Class:** 120 | - Defines different types of storage and policies for dynamic provisioning. 121 | 122 | ### **RBAC & Authorization:** 123 | Kubernetes provides **Role-Based Access Control (RBAC)** to manage permissions. 124 | 125 | **API Group:** `rbac.authorization.k8s.io` 126 | 127 | **Resources:** 128 | 1. **Roles:** 129 | - Assign permissions within a namespace. 130 | 131 | 2. **Role Bindings:** 132 | - Binds roles to users or groups within a namespace. 133 | 134 | 3. **Cluster Roles:** 135 | - Assign permissions cluster-wide. 136 | 137 | 4. **Cluster Role Bindings:** 138 | - Binds cluster roles to users or groups cluster-wide. 139 | 140 | ### **Custom Resources:** 141 | 1. **Custom Resource Definition (CRD):** 142 | - Extends the Kubernetes API. 143 | - Allows users to define and use their own resources. 144 | 145 | --- 146 | 147 | ## Autoscaling 148 | Kubernetes provides tools to automatically adjust resources based on usage. 149 | 150 | 1. **Horizontal Pod Autoscaler (HPA):** 151 | - Scales pods horizontally based on CPU or custom metrics. 152 | 153 | 2. **Vertical Pod Autoscaler (VPA):** 154 | - Adjusts resource requests and limits for containers based on actual usage. 155 | 156 | --- 157 | 158 | ## Key Commands (kubectl) 159 | 160 | 161 | ### **Basic Pod Commands:** 162 | 1. `kubectl get pods -o wide` 163 | - Displays detailed pod information in a wide format. 164 | 165 | 2. `kubectl get pods -o json` 166 | - Outputs pod details in JSON format. 167 | 168 | 3. `kubectl get pods --field-selector=status.phase=Running` 169 | - Filters pods based on their running status. 170 | 171 | 4. `kubectl get pods --watch` 172 | - Watches for changes in pod status in real-time. 173 | 174 | ### **Aliases:** 175 | 1. `alias kgp='kubectl get pods'` 176 | - Shortens the command to list pods. 177 | 178 | ### **Other Commands:** 179 | 1. `kubectl exec` 180 | - Executes commands directly within a container. 181 | 182 | --- 183 | 184 | ## **Examples for Better Understanding:** 185 | 1. **Mutating Admission Controller Example:** 186 | - Automatically adding a label to pods using `MutatingWebhookConfiguration`. 187 | 188 | 2. **Validating Admission Controller Example:** 189 | - Ensuring deployments do not use `latest` tag for images. 190 | 191 | 3. **API Version Usage Example:** 192 | - Using `apps/v1` for deploying a sample application. 193 | 194 | --- 195 | -------------------------------------------------------------------------------- /DevOps Tools 1 Module/Kubernetes Core Concepts.md: -------------------------------------------------------------------------------- 1 | # Kubernetes Core Concepts 2 | 3 | 4 | ## Agenda of the Lecture 5 | 6 | ### Today's Discussion Topics 7 | - **Why Kubernetes?** 8 | - **Kubernetes (K8s) Architecture** 9 | - **Different Types of K8s** 10 | - **Demo** 11 | - **Kubectl** 12 | 13 | 14 | ## Why Kubernetes? 15 | 16 | ### Key Points: 17 | - Kubernetes is abbreviated as **K8s**, with 8 letters between K and s. 18 | - Evolution: 19 | 1. Physical Machines: One application per machine. 20 | 2. Virtual Machines (VMs): Multiple isolated applications per physical machine. 21 | 3. Containers: Running multiple independent applications within VMs using Dockerization. 22 | - Challenges with Scaling: 23 | - Increased number of containers leads to increased complexity. 24 | - Orchestration became necessary to manage containers. 25 | - Kubernetes (K8s): 26 | - Acts as a **container orchestration tool**. 27 | - Dynamically assigns resources (containers) based on demand, e.g., in an e-commerce platform. 28 | - Benefits: 29 | - **Scalability** 30 | - **Reliability** 31 | - **Resource Efficiency** 32 | 33 | 34 | ## Kubernetes (K8s) Architecture 35 | 36 | ### Overview: 37 | - Divided into: 38 | - **Control Plane** 39 | - **Worker Nodes** 40 | 41 | ### Control Plane: 42 | - Manages the entire Kubernetes cluster and takes high-level decisions. 43 | - Components: 44 | - **API Server**: Central communication hub for users and nodes. 45 | - **ETCD**: A distributed key-value store; acts as the cluster's database. 46 | - **Scheduler**: Decides the placement of pods across nodes. 47 | - **Control Manager**: Ensures the desired state of the cluster using various controllers. 48 | 49 | ### Worker Nodes: 50 | - Responsible for executing the tasks. 51 | - Components: 52 | - **Kubelet**: Ensures containers in the node are running as defined. 53 | - **Pods**: 54 | - Smallest unit in Kubernetes where containers run. 55 | - Pods have IPs, not containers. 56 | - **CRI (Container Runtime Interface)**: Software responsible for running containers (e.g., Docker). 57 | - **Kube Proxy**: Manages networking within the cluster. 58 | 59 | ### Architectural Analogy: 60 | - **Control Plane**: Brain (decision-making). 61 | - **Worker Nodes**: Body parts (execution). 62 | 63 | 64 | ## Types of Kubernetes 65 | 66 | ### Categories: 67 | 1. **Vanilla Kubernetes**: 68 | - Open-source and highly customizable. 69 | 2. **Kubernetes for Developers**: 70 | - Tools like Kind, Minikube, and Micro K8s for local environments. 71 | 3. **Managed Kubernetes Services**: 72 | - Cloud-based solutions like GKE, EKS, AKS, IBM Kubernetes Service, and Oracle Kubernetes Engine. 73 | 74 | 75 | ## Installation and Practical Demo 76 | 77 | ### Installing Kubernetes: 78 | 1. Install Kind: 79 | ```bash 80 | curl -Lo ./kind https://kind.sigs.k8s.io/d1/v0.14.0/kind-linux-amd64 81 | chmod +x ./kind 82 | sudo mv ./kind /usr/local/bin/kind 83 | kind --version 84 | kind create cluster --name my-cluster 85 | ``` 86 | 2. Install Kubectl: 87 | ```bash 88 | sudo apt-get update 89 | sudo apt-get install -y apt-transport-https ca-certificates curl gnupg 90 | sudo mkdir -p -m 755 /etc/apt/keyrings 91 | sudo apt-get install -y kubectl 92 | ``` 93 | 94 | ### Kubectl Commands: 95 | - Check cluster info: 96 | ```bash 97 | kubectl cluster-info dump 98 | ``` 99 | - Node details: 100 | ```bash 101 | kubectl describe nodes kind-control-plane 102 | ``` 103 | - Other Commands: 104 | - `kubectl get nodes`: Lists all nodes. 105 | - `kubectl get pods -A`: Lists all pods across namespaces. 106 | -------------------------------------------------------------------------------- /DevOps Tools 1 Module/Kubernetes Jobs & Networking.md: -------------------------------------------------------------------------------- 1 | # Kubernetes Jobs & Networking 2 | 3 | 4 | 5 | ## Jobs in Kubernetes 6 | 7 | 8 | 9 | ### **Definition:** 10 | A Kubernetes Job ensures that a specified number of pods run to successful completion. It is used to manage batch or one-off tasks efficiently within a Kubernetes cluster. 11 | 12 | ### **Parameters:** 13 | 1. **Parallelism:** Specifies how many pods can run simultaneously. 14 | 2. **Completions:** Specifies the total number of successful pod runs required to consider the job complete. 15 | 16 | ### **Examples:** 17 | - **Example 1:** 18 | - Parallelism = 3, Completions = 3 19 | - Three pods will run simultaneously, each completing once. After three successful completions, the job terminates. 20 | 21 | - **Example 2:** 22 | - Parallelism = 3, Completions = 5 23 | - Three pods run simultaneously. Once a pod completes its task, another will start, continuing until five total completions are achieved. 24 | 25 | ### **Use Cases:** 26 | - Running batch processing tasks like database migrations. 27 | - Automating tasks that need a fixed number of successful runs. 28 | - Distributed computing where tasks can be split into parallelized workloads. 29 | 30 | --- 31 | 32 | ## Services in Kubernetes 33 | 34 | 35 | 36 | ### **Definition:** 37 | Services in Kubernetes provide a stable network endpoint for accessing pods. They abstract the complexity of networking and ensure seamless communication within and outside the cluster. 38 | 39 | ### **Types of Services:** 40 | 1. **ClusterIP (Default):** 41 | - Exposes the service on an internal IP address. 42 | - Accessible only within the cluster. 43 | - **Use Case:** Internal communication between application components. 44 | 45 | ```yaml 46 | apiVersion: v1 47 | kind: Service 48 | metadata: 49 | name: my-clusterip-service 50 | spec: 51 | type: ClusterIP 52 | ports: 53 | - port: 80 54 | selector: 55 | app: my-app 56 | ``` 57 | ![image](https://hackmd.io/_uploads/ByPUKvdHJl.png) 58 | 59 | 2. **NodePort:** 60 | - Exposes the service on a static port on each node's IP address. 61 | - Accessible externally via `:`. 62 | - **Use Case:** Simple external access during development. 63 | 64 | ```yaml 65 | apiVersion: v1 66 | kind: Service 67 | metadata: 68 | name: my-nodeport-service 69 | spec: 70 | type: NodePort 71 | ports: 72 | - port: 80 73 | targetPort: 8080 74 | nodePort: 30001 75 | selector: 76 | app: my-app 77 | ``` 78 | ![image](https://hackmd.io/_uploads/SknpFvOBke.png) 79 | 80 | 3. **LoadBalancer:** 81 | - Exposes the service externally via a cloud provider's load balancer. 82 | - **Use Case:** Managing external traffic in production environments. 83 | 84 | ```yaml 85 | apiVersion: v1 86 | kind: Service 87 | metadata: 88 | name: my-loadbalancer-service 89 | spec: 90 | type: LoadBalancer 91 | ports: 92 | - port: 80 93 | selector: 94 | app: my-app 95 | ``` 96 | 97 | 4. **ExternalName:** 98 | - Acts as a DNS pointer to an external service. 99 | - **Use Case:** Redirecting Kubernetes traffic to external DNS names. 100 | 101 | ```yaml 102 | apiVersion: v1 103 | kind: Service 104 | metadata: 105 | name: my-externalname-service 106 | spec: 107 | type: ExternalName 108 | externalName: external.example.com 109 | ``` 110 | 111 | --- 112 | 113 | ## Pod Management 114 | 115 | 116 | ### **Overview:** 117 | Pods are the smallest deployable units in Kubernetes. They can be managed as individual units or as a group. Pod management is crucial for ensuring efficient execution of tasks and workload distribution. 118 | 119 | ### **Key Features:** 120 | 1. **Grouped Management:** 121 | - Pods are often managed in groups to ensure scalability and fault tolerance. 122 | 123 | 2. **Log Aggregation:** 124 | - Logs from pods can be centralized for better debugging and monitoring. 125 | 126 | 3. **Task Distribution:** 127 | - Tasks can be distributed across multiple pods for efficiency. 128 | 129 | ### **Examples:** 130 | - Running pods with specific parallelism and completion settings (as discussed under Jobs). 131 | - Scaling pods horizontally to handle increased traffic. 132 | 133 | --- 134 | 135 | 136 | 137 | ### **Additional Notes:** 138 | 1. **Service Ports:** 139 | - Example: Service1 runs on port `30001`, Service2 runs on port `30002`. 140 | 141 | 2. **Pod Details:** 142 | - Each pod contains a container that executes tasks assigned by jobs. 143 | 144 | 3. **External Services:** 145 | - `ExternalName` services help connect Kubernetes traffic to external DNS names for seamless integration. 146 | 147 | ### **FAQ Examples:** 148 | - **Question:** Can a Job run indefinitely? 149 | - **Answer:** No, Jobs are designed for finite tasks. For indefinite execution, consider using a Deployment. 150 | 151 | - **Question:** What is the difference between NodePort and LoadBalancer? 152 | - **Answer:** NodePort exposes a service on each node’s IP and port, while LoadBalancer uses a cloud provider's external load balancer for broader access. 153 | 154 | --- 155 | 156 | -------------------------------------------------------------------------------- /DevOps Tools 1 Module/Kubernetes Observability and Pod Design cont.md: -------------------------------------------------------------------------------- 1 | # Kubernetes Observability and Pod Design 2 | 3 | 4 | 5 | ## Pods 6 | ### Pod Structure and Usage 7 | - **Definition:** Pods are the smallest deployable units in Kubernetes. They represent a single instance of a running process in a cluster. 8 | - **Components of a Pod:** 9 | - **Containers:** Typically one container per pod, but can include multiple containers for closely related processes. Containers within a pod share the same: 10 | - **Network Namespace:** Containers can communicate using `localhost`. 11 | - **Storage:** Shared volumes that persist data. 12 | - **Init Containers:** Special containers that run before the main application container starts. Useful for initialization tasks. 13 | 14 | - **Networking:** 15 | - Each pod gets its own IP address, enabling communication between containers inside and outside the pod. 16 | 17 | - **Usage:** 18 | - Ideal for running closely coupled processes. 19 | - Commonly used for applications like microservices, batch jobs, and single-instance databases. 20 | 21 | ### Significance of Label Selectors and Annotations in Pods 22 | #### Labels 23 | - **Definition:** Key-value pairs assigned to Kubernetes objects. 24 | - **Purpose:** Used to categorize and organize resources efficiently. 25 | - **Examples:** 26 | ```yaml 27 | labels: 28 | app: frontend 29 | env: prod 30 | ``` 31 | - **Benefits:** 32 | - Simplifies grouping of objects for queries and operations. 33 | - Enables targeted updates and deployments. 34 | 35 | #### Selectors 36 | - **Definition:** Filters that group objects based on their labels. 37 | - **Types:** 38 | - **Equality-Based Selectors:** Match objects with specific key-value pairs. 39 | - **Set-Based Selectors:** Match objects based on a set of criteria. 40 | 41 | - **Examples:** 42 | ```yaml 43 | matchLabels: 44 | app: frontend 45 | env: prod 46 | ``` 47 | 48 | #### Annotations 49 | - **Definition:** Non-identifying metadata for Kubernetes objects. 50 | - **Purpose:** Provides additional information, often used by external tools and systems. 51 | - **Examples:** 52 | ```yaml 53 | annotations: 54 | prometheus.io/scrape: "true" 55 | description: "This pod runs the frontend service." 56 | ``` 57 | - **Common Use Cases:** 58 | - Monitoring (e.g., Prometheus scraping configurations). 59 | - Documentation. 60 | 61 | --- 62 | 63 | ## Probes 64 | ### Introduction 65 | Probes are used to determine the health and operational status of containers. Kubernetes uses probes to decide whether a container is ready to accept traffic or needs to be restarted. 66 | 67 | ### Types of Probes 68 | #### 1. Startup Probe 69 | - **Purpose:** Ensures containers with long initialization times are not prematurely marked as failed. 70 | - **When to Use:** Applications that perform significant setup tasks before becoming operational. 71 | - **Behavior:** Delays the execution of readiness and liveness probes. 72 | - **Example:** 73 | ```yaml 74 | startupProbe: 75 | httpGet: 76 | path: /healthz 77 | port: 8080 78 | failureThreshold: 30 79 | periodSeconds: 10 80 | ``` 81 | 82 | #### 2. Readiness Probe 83 | - **Purpose:** Determines if a container is ready to accept traffic. 84 | - **Behavior:** Traffic is routed only to pods that pass the readiness probe. 85 | - **Use Cases:** 86 | - Applications that load configuration files or establish database connections before serving traffic. 87 | - **Example:** 88 | ```yaml 89 | readinessProbe: 90 | httpGet: 91 | path: /ready 92 | port: 8080 93 | initialDelaySeconds: 5 94 | periodSeconds: 10 95 | ``` 96 | 97 | #### 3. Liveness Probe 98 | - **Purpose:** Ensures a container is functioning as expected. If a container fails this probe, it is restarted. 99 | - **Behavior:** Used to recover from application crashes or deadlocks. 100 | - **Example:** 101 | ```yaml 102 | livenessProbe: 103 | httpGet: 104 | path: /healthz 105 | port: 8080 106 | initialDelaySeconds: 3 107 | periodSeconds: 5 108 | ``` 109 | 110 | ![image](https://hackmd.io/_uploads/S1cjyZvSkl.png) 111 | 112 | --- 113 | 114 | ## Label Selectors and Annotations 115 | ### Labels 116 | - **Primary Use:** Categorization and identification of objects. 117 | - **Advanced Example:** 118 | ```yaml 119 | metadata: 120 | labels: 121 | app: ecommerce 122 | tier: backend 123 | release: stable 124 | ``` 125 | 126 | ### Selectors 127 | - **Advanced Usage:** 128 | ```yaml 129 | matchExpressions: 130 | - key: tier 131 | operator: In 132 | values: 133 | - frontend 134 | - backend 135 | ``` 136 | - **Purpose:** Enables complex filtering logic for grouping resources. 137 | 138 | ### Annotations 139 | - **Use in Observability:** Helps integrate tools like Prometheus or Grafana. 140 | - **Example with Multiple Annotations:** 141 | ```yaml 142 | annotations: 143 | prometheus.io/scrape: "true" 144 | prometheus.io/path: "/metrics" 145 | team: "devops" 146 | ``` 147 | 148 | --- 149 | 150 | ## Deployment Strategies 151 | ### Update Strategies 152 | - **Maximum Unavailable:** 153 | - Limits the number of unavailable pods during updates. 154 | - Ensures a minimum number of pods remain operational. 155 | - **Maximum Surge:** 156 | - Defines how many additional pods can be temporarily created during updates. 157 | - Allows updates without reducing available capacity. 158 | 159 | ### Deployment Types 160 | #### Blue-Green Deployment 161 | - **Workflow:** 162 | 1. Deploy Blue (current version). 163 | 2. Deploy Green (new version) alongside Blue. 164 | 3. Switch traffic to Green after successful testing. 165 | - **Benefits:** 166 | - Zero downtime. 167 | - Immediate rollback capability. 168 | - **Challenges:** 169 | - Requires additional resources for parallel deployments. 170 | 171 | #### Canary Deployment 172 | - **Workflow:** 173 | 1. Incrementally roll out changes to a small subset of users. 174 | 2. Monitor the impact and error rate. 175 | 3. Gradually scale Canary or revert to Base if needed. 176 | - **Benefits:** 177 | - Fine-grained control over deployment. 178 | - Early detection of issues. 179 | 180 | --- 181 | 182 | ## Practical Demonstrations of Deployment Strategies 183 | ### Key Steps for Deployment 184 | 1. **Create Base Deployment:** Defines the stable version of the application. 185 | 2. **Create Canary Deployment:** Defines the new version of the application with reduced initial traffic. 186 | 3. **Configure Services:** Use labels and selectors to route traffic appropriately. 187 | 4. **Monitor Performance:** 188 | - Check logs for errors. 189 | - Validate system performance. 190 | 5. **Handle Failures:** Scale back Canary to zero if errors occur. 191 | 6. **Finalize Rollout:** Scale Canary incrementally and balance traffic distribution. 192 | 193 | ### Example Configuration 194 | #### Base Deployment 195 | ```yaml 196 | apiVersion: apps/v1 197 | kind: Deployment 198 | metadata: 199 | name: base-deployment 200 | spec: 201 | replicas: 3 202 | selector: 203 | matchLabels: 204 | app: web 205 | version: base 206 | template: 207 | metadata: 208 | labels: 209 | app: web 210 | version: base 211 | spec: 212 | containers: 213 | - name: nginx 214 | image: nginx:1.14 215 | ``` 216 | 217 | #### Canary Deployment 218 | ```yaml 219 | apiVersion: apps/v1 220 | kind: Deployment 221 | metadata: 222 | name: canary-deployment 223 | spec: 224 | replicas: 1 225 | selector: 226 | matchLabels: 227 | app: web 228 | version: canary 229 | template: 230 | metadata: 231 | labels: 232 | app: web 233 | version: canary 234 | spec: 235 | containers: 236 | - name: nginx 237 | image: nginx:1.19 238 | ``` 239 | 240 | --- 241 | 242 | ## Tools and Example Configurations 243 | - **Nginx Example:** 244 | - Base Deployment: Version 1.14. 245 | - Canary Deployment: Version 1.19. 246 | - **Command-Line Tools:** kubectl, Helm. 247 | - **Monitoring Tools:** Prometheus, Grafana. 248 | 249 | --- 250 | -------------------------------------------------------------------------------- /DevOps Tools 1 Module/Kubernetes Observability and Pod Design.md: -------------------------------------------------------------------------------- 1 | # Kubernetes Observability and Pod Design 2 | 3 | This document provides comprehensive lecture notes covering key concepts such as service accounts, pod scheduling, init containers, and effective pod design. It incorporates examples, use cases, and detailed explanations for instructors to teach effectively. 4 | 5 | --- 6 | 7 | ## Service Accounts 8 | 9 | ### Introduction to Service Accounts 10 | - **Purpose**: Service accounts provide an identity to processes running within a pod, granting them specific permissions to interact with Kubernetes resources. 11 | - **Example**: A pod running a web application that needs to read data from a ConfigMap would require a service account with the appropriate read permissions. 12 | 13 | ### Key Concepts 14 | 1. **Permissions and Namespace**: 15 | - Service accounts are tied to a specific namespace. 16 | - Resources like secrets and ConfigMaps are also namespace-scoped. If no namespace is specified, they are assigned to the `default` namespace. 17 | - Service accounts must be explicitly assigned to the same namespace as the resources they interact with. 18 | 19 | 2. **Creating a Service Account**: 20 | - Via CLI: 21 | ``` 22 | kubectl create serviceaccount 23 | ``` 24 | - Via YAML: 25 | ```yaml 26 | apiVersion: v1 27 | kind: ServiceAccount 28 | metadata: 29 | name: 30 | namespace: 31 | ``` 32 | 33 | 3. **Use Case**: 34 | - Assign permissions for pods to perform specific actions: 35 | - Pod A can read resources. 36 | - Pod B can read, write, and delete resources. 37 | 38 | --- 39 | 40 | ## Pod Scheduling to Nodes 41 | 42 | ### Introduction to Pod Scheduling 43 | - Pod scheduling involves determining which node will host a pod. This ensures optimal resource usage and prevents conflicts or failures. 44 | - **Example Scenario**: 45 | - Node1 has 20GB of memory. 46 | - Node2 has 100GB of memory. 47 | - A pod requiring 40GB of memory would fail to schedule on Node1 but succeed on Node2. 48 | 49 | ### Factors Influencing Pod Scheduling 50 | 1. **Resource Requests and Limits**: 51 | - Define how much CPU and memory a pod requests and caps its usage. 52 | 53 | 2. **Taints and Tolerations**: 54 | - **Taints**: Applied to nodes to restrict pods from scheduling unless they tolerate the taint. 55 | - **Tolerations**: Applied to pods to enable them to schedule on tainted nodes. 56 | - Effects of taints: 57 | - `NoSchedule`: Pods without toleration are not scheduled. 58 | - `PreferNoSchedule`: Avoid scheduling but not strictly enforced. 59 | - `NoExecute`: Evicts existing pods if they don’t tolerate the taint. 60 | - **Command Example**: 61 | ``` 62 | kubectl taint nodes =:NoSchedule 63 | ``` 64 | - **Use Cases**: 65 | - Dedicated workloads. 66 | - Node maintenance. 67 | 68 | 3. **Node Selectors and Labels**: 69 | - Nodes and pods can be labeled with key-value pairs to match workloads to nodes. 70 | - **Commands**: 71 | - Labeling a node: 72 | ``` 73 | kubectl label nodes instanceType=spot 74 | ``` 75 | - Assigning a pod to a labeled node: 76 | ```yaml 77 | nodeSelector: 78 | instanceType: spot 79 | ``` 80 | 81 | 4. **Node Affinity**: 82 | - A flexible approach to scheduling pods on specific nodes based on rules. 83 | - **Key Differences**: 84 | - **Taints and Tolerations**: Block or isolate workloads. 85 | - **Node Selectors**: Require exact label matches. 86 | - **Node Affinity**: Supports both mandatory (`requiredDuringSchedulingIgnoredDuringExecution`) and preferred scheduling rules (`preferredDuringSchedulingIgnoredDuringExecution`). 87 | 88 | | **Aspect** | **Taints & Tolerations** | **Node Selectors** | **Node Affinity** | 89 | |--------------------------|----------------------------------------|----------------------------------------|----------------------------------------| 90 | | **Purpose** | Block pods unless tolerated | Match labels strictly | Control placement with flexible rules | 91 | | **Flexibility** | Strict | Strict | Flexible | 92 | | **Configuration** | `tolerations` in pod spec | `nodeSelector` in pod spec | `affinity` in pod spec | 93 | | **Effect when unmatched**| Won't schedule | Won't schedule | May block or prefer scheduling | 94 | 95 | 96 | --- 97 | 98 | ## Init Containers 99 | 100 | ### Overview of Init Containers 101 | - **Definition**: Init containers run before the main application containers and are designed to complete specific initialization tasks. 102 | - **Characteristics**: 103 | - They run sequentially and must complete successfully before the main container starts. 104 | - Multiple init containers run one after the other. 105 | 106 | ### Use Cases 107 | 1. Setting up configurations or preparing environment variables. 108 | 2. Verifying dependencies or required services. 109 | 3. Performing security checks or downloading initialization data. 110 | 111 | 112 | --- 113 | 114 | ## Pod Design and Observability 115 | 116 | ### Designing Pods with Dependencies 117 | - Applications with dependencies must be designed such that: 118 | - Application1 only starts after Application2 completes. 119 | - Kubernetes ensures proper sequencing through tools like init containers and probes. 120 | 121 | ### Key Elements of Pod Design 122 | 1. **Probes**: 123 | - Liveness Probe: Ensures the container is running as expected. 124 | - Readiness Probe: Verifies the application is ready to serve requests. 125 | - Startup Probe: Confirms the application has started successfully. 126 | 127 | 2. **Labels, Selectors, and Annotations**: 128 | - Labels help organize and select specific pods or nodes. 129 | - Annotations provide metadata that doesn’t affect scheduling. 130 | 131 | 3. **Deployments and Services**: 132 | - Use Deployments to manage replicas and updates. 133 | - Services expose pods for network communication within or outside the cluster. 134 | 135 | 4. **Deployment Strategies**: 136 | - Rolling Updates: Incrementally update pods without downtime. 137 | - Blue-Green Deployments: Use separate environments for new and old versions. 138 | --- 139 | -------------------------------------------------------------------------------- /DevOps Tools 1 Module/Kubernetes Security.md: -------------------------------------------------------------------------------- 1 | # Kubernetes Security Lecture Notes 2 | 3 | --- 4 | 5 | 6 | 7 | ## **1. StatefulSets and Deployment** 8 | 9 | 10 | ### **1.1 What are StatefulSets?** 11 | StatefulSets manage stateful applications where each pod requires a unique identity and dedicated storage. Unlike Deployments, StatefulSets maintain stable network identities and persistent storage for pods. 12 | 13 | ### **1.2 Key Features of StatefulSets:** 14 | - **Unique Pod Identifiers:** Each pod created by a StatefulSet gets a unique identifier (e.g., pod-0, pod-1, pod-2). 15 | - **Persistent Storage:** StatefulSets use Persistent Volumes (PVs) to ensure each pod retains its data, even after restarts. 16 | - **Ordered Deployment and Scaling:** Pods are started, scaled, and terminated in a defined order. 17 | 18 | ### **1.3 What are Deployments?** 19 | Deployments manage stateless applications where all pods are identical and interchangeable. They ensure high availability and easy scaling by treating pods as identical replicas. 20 | 21 | ### **1.4 Differences Between StatefulSets and Deployments:** 22 | | Feature | StatefulSets | Deployments | 23 | |-----------------------|--------------------------------|-------------------------| 24 | | **Pod Identity** | Unique for each pod | Identical for all pods | 25 | | **Storage** | Dedicated Persistent Volumes | Shared storage (if any) | 26 | | **Startup Sequence** | Sequential | Parallel | 27 | | **Use Case** | Databases, Stateful Apps | Web Servers, APIs | 28 | 29 | ### **1.5 Use Cases:** 30 | - **StatefulSets:** 31 | - Suitable for databases, distributed systems, or applications that require stable storage and unique pod identities. 32 | - Example: A Cassandra database where each node has a specific role. 33 | - **Deployments:** 34 | - Ideal for stateless applications like web servers or APIs where any pod can handle requests. 35 | - Example: A frontend web application with multiple identical instances. 36 | 37 | ### **1.6 Example YAML Configuration:** 38 | **StatefulSet Example:** 39 | ```yaml 40 | apiVersion: apps/v1 41 | kind: StatefulSet 42 | metadata: 43 | name: web 44 | spec: 45 | serviceName: "web" 46 | replicas: 3 47 | selector: 48 | matchLabels: 49 | app: nginx 50 | template: 51 | metadata: 52 | labels: 53 | app: nginx 54 | spec: 55 | containers: 56 | - name: nginx 57 | image: nginx:1.17 58 | ports: 59 | - containerPort: 80 60 | ``` 61 | 62 | --- 63 | 64 | ## **2. Headless Services** 65 | 66 | 67 | ### **2.1 What are Headless Services?** 68 | Headless Services allow direct DNS-based access to individual pods without a cluster IP. Unlike traditional services that abstract pod IPs behind a single cluster IP, Headless Services create DNS records for each pod, enabling direct communication. 69 | 70 | ### **2.2 Configuration:** 71 | To configure a Headless Service, set `ClusterIP: None` in the service definition. 72 | 73 | Example YAML: 74 | ```yaml 75 | apiVersion: v1 76 | kind: Service 77 | metadata: 78 | name: my-headless-service 79 | spec: 80 | clusterIP: None 81 | selector: 82 | app: my-app 83 | ports: 84 | - protocol: TCP 85 | port: 80 86 | ``` 87 | ![image](https://hackmd.io/_uploads/BkB5_y181x.png) 88 | 89 | 90 | ### **2.3 DNS Format:** 91 | Pods in a Headless Service can be accessed using the following DNS format: 92 | ``` 93 | ...svc.cluster.local 94 | ``` 95 | For example: 96 | ``` 97 | pod-0.my-headless-service.default.svc.cluster.local 98 | ``` 99 | 100 | ### **2.4 Use Case:** 101 | Headless Services are ideal for stateful applications requiring stable DNS names for each pod, such as: 102 | - Databases (e.g., Cassandra, MongoDB) 103 | - Applications requiring direct pod-to-pod communication 104 | 105 | ![image](https://hackmd.io/_uploads/B1ubtk1Uyg.png) 106 | 107 | --- 108 | 109 | ## **3. Kubernetes Security and RBAC** 110 | 111 | 112 | ### **3.1 Authentication:** 113 | Authentication verifies the identity of users and components interacting with the Kubernetes API server. 114 | 115 | **Methods:** 116 | - Client Certificates 117 | - Service Account Tokens 118 | - External Identity Providers (e.g., AWS, GCP, Azure) 119 | - OpenID Connect (OIDC) 120 | 121 | ### **3.2 Authorization:** 122 | Authorization determines what actions an authenticated user can perform. 123 | 124 | **Mechanisms:** 125 | - Role-Based Access Control (RBAC) 126 | - Namespace-Level: Roles and RoleBindings 127 | - Cluster-Level: ClusterRoles and ClusterRoleBindings 128 | 129 | ### **3.3 RBAC Key Concepts:** 130 | | Concept | Description | 131 | |------------------|---------------------------------------------------------| 132 | | **Role** | Grants access to resources within a namespace | 133 | | **ClusterRole** | Grants access to resources cluster-wide | 134 | | **RoleBinding** | Binds a Role to users, groups, or service accounts | 135 | | **ClusterRoleBinding** | Binds a ClusterRole to users, groups, or service accounts cluster-wide | 136 | 137 | ### **3.4 RBAC Syntax Examples:** 138 | **Role Example:** 139 | ```yaml 140 | apiVersion: rbac.authorization.k8s.io/v1 141 | kind: Role 142 | metadata: 143 | namespace: default 144 | name: pod-reader 145 | rules: 146 | - apiGroups: [""] 147 | resources: ["pods"] 148 | verbs: ["get", "list"] 149 | ``` 150 | 151 | **RoleBinding Example:** 152 | ```yaml 153 | apiVersion: rbac.authorization.k8s.io/v1 154 | kind: RoleBinding 155 | metadata: 156 | name: read-pods 157 | namespace: default 158 | subjects: 159 | - kind: User 160 | name: jane-doe 161 | apiGroup: "" 162 | roleRef: 163 | kind: Role 164 | name: pod-reader 165 | apiGroup: rbac.authorization.k8s.io 166 | ``` 167 | 168 | ### **3.5 Best Practices:** 169 | - Follow the principle of least privilege to minimize risks. 170 | - Regularly review and audit RBAC policies. 171 | - Use namespace isolation to restrict access boundaries. 172 | 173 | --- 174 | 175 | ## **4. Summary Commands** 176 | 177 | 178 | ### **4.1 Authentication and Authorization Commands:** 179 | - `kubectl get pods` - List all pods in a namespace. 180 | - `kubectl delete pods ` - Delete a specific pod. 181 | - `kubectl describe roles` - Describe defined RBAC roles within a namespace. 182 | 183 | --- 184 | 185 | -------------------------------------------------------------------------------- /DevOps Tools 1 Module/Kubernetes State Persistence.md: -------------------------------------------------------------------------------- 1 | # Kubernetes State Persistence 2 | 3 | --- 4 | 5 | 6 | 7 | 8 | 9 | ## Network Policies 10 | 11 | 12 | 13 | ### **Overview:** 14 | Network policies define rules for how pods communicate with: 15 | - Each other. 16 | - External endpoints. 17 | 18 | Policies can manage both **ingress** (incoming traffic) and **egress** (outgoing traffic). 19 | 20 | ### **Key Concepts:** 21 | 1. **Selectors:** 22 | - Used to identify the pods to which the policy applies. 23 | - Example: A policy might target pods labeled `app=frontend`. 24 | - **Example:** Consider a microservices architecture where you only want the `frontend` pod to communicate with the `backend` pod but restrict it from accessing the `database` pod. Using selectors, you can define this scope effectively. 25 | 26 | 2. **Ingress Rules:** 27 | - Define the allowed incoming traffic to pods. 28 | - Example: Allow HTTP traffic on port 80 from a specific pod group. 29 | - **Example:** If a `shopping-cart` pod accepts traffic from a `user-session` pod, the ingress rule can specify that only requests on port 8080 (HTTP) are allowed while blocking others. 30 | 31 | 3. **Egress Rules:** 32 | - Define the allowed outgoing traffic from pods. 33 | - Example: Allow outgoing requests to a database pod. 34 | - **Example:** A `logging` pod might need to send data to an external monitoring service. The egress rule can restrict traffic to only the specific IP address of that service. 35 | 36 | 4. **Policy Types:** 37 | - **Ingress:** Rules for incoming traffic. 38 | - **Egress:** Rules for outgoing traffic. 39 | - Both can be used simultaneously to create comprehensive traffic policies. 40 | 41 | ![image](https://hackmd.io/_uploads/HkazMxsrJg.png) 42 | 43 | ### **Lab Workflow:** 44 | 1. **Deploy an Application:** 45 | - Example: Deploy a web application that communicates with a database pod. 46 | - **Example:** Deploy a `frontend` service that interacts with both a `backend` service and a `cache` service while limiting communication to specific ports and protocols. 47 | 48 | 2. **Deploy a Second Pod with Specific Policies:** 49 | - Create a pod with policies to allow only certain traffic. 50 | - Example: Deploy a monitoring pod that only listens to logs from the application pod. 51 | 52 | 3. **Create an Allow Policy:** 53 | - Example: Permit HTTP traffic from pod `frontend` to pod `backend`. 54 | - **Example:** Create a policy to allow TCP connections from a `worker` pod to a `queue` pod only on port 5672 for message queueing. 55 | 56 | 4. **Create a Deny Policy:** 57 | - Example: Block all other traffic except necessary requests. 58 | - **Example:** Deny all outbound traffic from a `debug` pod except for SSH connections to a specific admin server. 59 | 60 | --- 61 | 62 | ## Volumes in Kubernetes 63 | 64 | 65 | 66 | ### **Types of Volumes:** 67 | 1. **ConfigMap as Volume:** 68 | - Mount configuration data as files or environment variables. 69 | - Example: Store database connection strings in a `ConfigMap` and use it in pods. 70 | - **Detailed Example:** A `ConfigMap` containing multiple configurations (e.g., environment-specific variables for dev, staging, and production) can be mounted as a volume, ensuring consistent deployment across environments. 71 | 72 | 2. **Secret as Volume:** 73 | - Store sensitive data like passwords or tokens securely. 74 | - Example: Mount an API token secret as a volume. 75 | - **Example:** Use a `Secret` to store TLS certificates for a secure connection between pods. 76 | 77 | 3. **HostPath:** 78 | - Maps a directory or file on the host node’s filesystem to a pod. 79 | - Example: Use host paths to share logs between the host and pod. 80 | - **Example:** Map a specific directory, such as `/var/logs/app`, from the host node to a pod for real-time log monitoring. 81 | 82 | 4. **EmptyDir:** 83 | - A temporary directory created when a pod is assigned to a node. 84 | - Data is cleared when the pod stops. 85 | - Example: Cache data processing results temporarily. 86 | - **Example:** Use `EmptyDir` for temporary storage in a data analytics pipeline where intermediate results need to be processed within the same pod lifecycle. 87 | 88 | ![image](https://hackmd.io/_uploads/rkyK7gorye.png) 89 | 90 | --- 91 | 92 | ## Persistent Volumes (PV), Claims (PVC), and Storage Classes 93 | 94 | 95 | 96 | ### **Persistent Volumes (PV):** 97 | - **Independent of Pod Lifecycle:** Persistent storage remains even after pod deletion. 98 | - **Access Modes:** 99 | - **ReadWriteOnce:** Mounted by a single node for read/write. 100 | - **ReadOnlyMany:** Mounted by multiple nodes as read-only. 101 | - **ReadWriteMany:** Mounted by multiple nodes for read/write. 102 | 103 | #### **Reclaim Policies:** 104 | - **Retain:** Keeps the data for manual reclamation. 105 | - **Recycle:** Performs a basic cleanup (e.g., deleting files). 106 | - **Delete:** Deletes the data when the volume is released. 107 | 108 | ### **Persistent Volume Claims (PVC):** 109 | - A request for storage by a pod. 110 | - **Dynamic Provisioning:** Automatically provisions a PV when a PVC is created. 111 | 112 | ### **Storage Class:** 113 | 1. **Provisioner:** The backend storage responsible for provisioning. 114 | 2. **Parameters:** Define properties like storage size or type. 115 | 3. **Reclaim Policy:** Configured at the storage class level. 116 | 117 | #### **Example Workflow:** 118 | 1. Define a **Storage Class** with specific parameters. 119 | - Example: Create a storage class for SSD-backed storage with high IOPS for database workloads. 120 | 2. Create a **PVC** to request storage from the defined class. 121 | - Example: Request a 10GB volume with `ReadWriteMany` access mode for shared access. 122 | 3. Automatically provision a **PV** based on the request. 123 | 4. Mount the PV to the pod for data persistence. 124 | - Example: Mount a dynamically provisioned PV to a MySQL pod for data storage. 125 | 126 | --- 127 | 128 | ## Quality of Service (QoS) 129 | 130 | 131 | 132 | ### **Goals:** 133 | - Efficient resource management. 134 | - Classify workloads based on resource requests and limits. 135 | 136 | ### **QoS Classes:** 137 | 1. **Guaranteed:** 138 | - Resources are fully reserved for the pod. 139 | - Example: A pod with exact resource requests and limits specified. 140 | - **Example:** Deploy a critical financial transaction processing pod with CPU and memory requests of 2 cores and 4GB respectively, ensuring consistent performance. 141 | 142 | 2. **Burstable:** 143 | - Some resource limits are set, but the pod can exceed requests. 144 | - Example: A pod with higher limits than requests. 145 | - **Example:** A `logging` pod with 500m CPU and 1GB memory requests but limits of 1 CPU and 2GB memory can burst when additional resources are available. 146 | 147 | 3. **BestEffort:** 148 | - No resource limits or requests specified. 149 | - Example: A pod that consumes resources only if available. 150 | - **Example:** Deploy an auxiliary data analysis pod without resource reservations, utilizing idle cluster resources. 151 | 152 | --- 153 | -------------------------------------------------------------------------------- /DevOps Tools 2 Module/Advance Jenkins Concepts.md: -------------------------------------------------------------------------------- 1 | ## Typed Notes - Class-2 Advance Jenkins Concepts 2 | 3 | --- 4 | 5 | ### 1. **Pipelines** 6 | 7 | #### **Overview** 8 | - **Definition**: Pipelines represent a sequence of automated processes for building, testing, and deploying applications in continuous integration and deployment (CI/CD) systems. They ensure the smooth progression of code changes from development to production. 9 | - **Importance**: 10 | - Facilitates automation to reduce manual effort. 11 | - Enhances efficiency and consistency in software delivery. 12 | - Streamlines error detection and resolution early in the development cycle. 13 | 14 | --- 15 | 16 | #### **Types of Pipelines** 17 | 18 | ##### **1. Scripted Pipelines** 19 | - **Definition**: A flexible pipeline style written in Groovy with a programmatic approach. 20 | - **Key Characteristics**: 21 | - Offers granular control over the pipeline logic. 22 | - Tasks and execution flow are explicitly defined by the developer. 23 | - Provides access to advanced Groovy functionalities for custom behavior. 24 | - **Advantages**: 25 | - Highly customizable for unique and complex workflows. 26 | - Can accommodate non-standard CI/CD requirements. 27 | - **Challenges**: 28 | - Steep learning curve for those unfamiliar with Groovy or programming. 29 | - Higher potential for errors due to manual coding. 30 | - Requires careful debugging and maintenance. 31 | 32 | ##### **2. Declarative Pipelines** 33 | - **Definition**: A pipeline style using structured, predefined syntax for simplicity and ease of use. 34 | - **Key Characteristics**: 35 | - Simplifies pipeline creation with a predefined structure. 36 | - Automatically validates the pipeline syntax before execution. 37 | - Encourages standardization across teams. 38 | - **Advantages**: 39 | - Easier to understand, even for beginners. 40 | - Faster to implement due to its straightforward syntax. 41 | - Reduces human error with automated validation. 42 | - **Limitations**: 43 | - Less flexibility compared to scripted pipelines. 44 | - May not be suitable for highly customized or complex workflows. 45 | 46 | --- 47 | 48 | #### **Pipeline Features** 49 | 50 | 1. **Parallel Execution**: 51 | - Allows multiple pipeline stages to run concurrently. 52 | - Benefits: 53 | - Optimizes resource utilization. 54 | - Reduces overall pipeline execution time by processing independent tasks simultaneously. 55 | 56 | 2. **Stages in a Pipeline**: 57 | - Logical sections where specific tasks or processes are executed. 58 | - Common Stages: 59 | - **Checkout**: Retrieves source code from version control systems (e.g., Git). 60 | - **Build**: Compiles the source code into deployable artifacts like binaries. 61 | - **Unit Test**: Validates the correctness of individual software modules. 62 | - **Regression Test**: Ensures new code changes do not disrupt existing functionality. 63 | - **Preprod Deployment**: Deploys the application to a pre-production environment for final validation. 64 | 65 | --- 66 | 67 | #### **Pipeline Architecture** 68 | 69 | 1. **Master-Slave Architecture**: 70 | - Jenkins employs a distributed architecture for scalability and performance. 71 | - Components: 72 | - **Master**: 73 | - Central server that orchestrates pipeline execution. 74 | - Delegates tasks to slave nodes based on pipeline requirements. 75 | - **Slave**: 76 | - Remote nodes responsible for executing tasks such as building, testing, and deployment. 77 | - Must have **Java** installed to communicate with the master. 78 | 79 | 2. **Labels in Pipelines**: 80 | - **Agent Labels**: 81 | - Tags that categorize agents (slaves) based on their roles or capabilities (e.g., "Linux agent," "Windows agent"). 82 | - **Node Labels**: 83 | - Identifiers used to specify where particular tasks should be executed. 84 | 85 | --- 86 | 87 | #### **Pipeline Examples** 88 | 89 | 1. **Single Server Pipeline**: 90 | - All tasks (checkout, build, test, deploy) are executed on a single server. 91 | - **Use Case**: 92 | - Ideal for small-scale projects with limited resource requirements. 93 | - **Advantages**: 94 | - Simplicity in configuration and management. 95 | - **Limitations**: 96 | - Limited scalability. 97 | - Can become a bottleneck for resource-intensive workflows. 98 | 99 | 2. **Multi-Node Pipeline**: 100 | - Tasks are distributed across multiple nodes for better scalability and efficiency. 101 | - **Example Workflow**: 102 | - **Node 1**: Performs initial stages like code checkout and build. 103 | - **Node 2**: Handles testing tasks, including unit and regression testing. 104 | - **Node 3**: Executes deployment-related activities. 105 | - **Use Case**: 106 | - Suitable for large-scale projects with complex requirements. 107 | - **Advantages**: 108 | - Optimized resource utilization. 109 | - Reduced pipeline execution time. 110 | - **Challenges**: 111 | - Requires careful node configuration and management. 112 | 113 | --- 114 | 115 | ### 2. **Agents and Credentials** 116 | 117 | --- 118 | 119 | #### **Agent Management** 120 | 121 | 1. **Components**: 122 | - **Master**: 123 | - The central Jenkins server that manages pipeline orchestration and task delegation. 124 | - **Slaves**: 125 | - Distributed nodes that execute the tasks assigned by the master. 126 | - Requirements: 127 | - **Java installation** is mandatory to establish communication with the master. 128 | 129 | 2. **Key Points**: 130 | - Use **Agent Labels** to assign tasks to specific agents based on their roles or capabilities. 131 | - Utilize **Node Labels** to specify the nodes on which certain pipeline stages should execute. 132 | - Pipelines typically execute tasks on slave nodes for efficient resource management. 133 | 134 | --- 135 | 136 | #### **Credentials Management** 137 | 138 | 1. **Purpose**: 139 | - To securely store and manage sensitive information needed for pipeline execution, such as: 140 | - **SSH Keys**: Used for secure communication between systems. 141 | - **API Tokens**: Enable authentication and access to external services or APIs. 142 | - **Environment Variables**: Store reusable, sensitive data for consistent use across pipelines. 143 | 144 | 2. **Usage**: 145 | - Define credentials within Jenkins' credential management system. 146 | - Access credentials as secure variables in pipeline scripts. 147 | - Benefits: 148 | - Prevents unauthorized access to sensitive information. 149 | - Enhances security by avoiding hardcoding of secrets in pipeline scripts. 150 | - Ensures compliance with security best practices. 151 | 152 | --- 153 | 154 | -------------------------------------------------------------------------------- /DevOps Tools 2 Module/Ansible Fundamentals.md: -------------------------------------------------------------------------------- 1 | # Ansible Fundamentals 2 | In this session, we will be covering the following topics related to Ansible fundamentals: 3 | 4 | - **Configuration as Code** 5 | Discuss how infrastructure is managed with machine-readable definition files to achieve consistency, scalability, and automation. 6 | - **Ansible Overview** 7 | Introduction to Ansible as a tool for automation, configuration management, and how it compares to Chef and Puppet. 8 | - **Installation** 9 | Guide to setting up Ansible and configuring control/managed nodes. 10 | - **Server Setup** 11 | Steps to configure servers using Ansible for infrastructure automation. 12 | - **Adhoc Commands** 13 | Explanation of quick and temporary commands for one-off tasks in Ansible. 14 | 15 | 16 | ### Configuration as Code 17 | 18 | **Definition:** 19 | Configuration as Code is the practice of managing and provisioning infrastructure through machine-readable definition files, rather than through physical hardware configuration or interactive configuration tools. 20 | 21 | **Key Benefits:** 22 | 23 | - **Consistency** 24 | Ensures identical environments across development, testing, and production. 25 | - **Version Control** 26 | Infrastructure changes are tracked, just like source code. 27 | - **Scalability** 28 | Easy to replicate infrastructure for scaling up or down. 29 | - **Automation** 30 | Allows for automated setup and configuration without manual intervention. 31 | 32 | **Popular Tools:** 33 | - Chef 34 | - Puppet 35 | - Ansible 36 | 37 | ### Comparison of Chef, Puppet, and Ansible 38 | 39 | | Feature | Chef | Puppet | Ansible | 40 | |----------------------|---------------------|---------------------|-------------------| 41 | | **Language** | Ruby (DSL) | Puppet DSL | YAML (Playbooks) | 42 | | **Model** | Pull | Pull | Push | 43 | | **Agent Required** | Yes | Yes | No | 44 | | **Ease of Use** | Medium | Medium | Easy | 45 | | **Control Node** | Chef Server | Puppet Master | Control Machine | 46 | | **Managed Nodes** | Chef Clients (Agent)| Puppet Agents (Agent)| Agentless (SSH) | 47 | | **Configuration** | Cookbooks/Recipes | Manifests/Modules | Playbooks/Roles | 48 | | **Scalability** | High | High | Moderate | 49 | 50 | 51 | ### Ansible Overview 52 | 53 | **Definition:** 54 | Ansible is an open-source automation tool that simplifies IT orchestration by offering an agentless architecture. It manages configurations, deployment, and infrastructure tasks. 55 | 56 | #### Key Concepts: 57 | 58 | - **Playbooks** 59 | YAML files that describe the desired state of your infrastructure. For example, installing Nginx on a server can be defined in a playbook, and Ansible will match the server’s configuration to this desired state. 60 | 61 | - **Modules** 62 | Reusable standalone scripts that perform specific tasks. For example, the `apt` module installs packages on Debian systems. 63 | 64 | Types of modules: 65 | - Core module (e.g., apt) 66 | - Custom module (user-created) 67 | 68 | - **Inventory** 69 | A list of managed nodes (servers) that Ansible will configure. Inventory is often a simple text file. 70 | 71 | - **Templates** 72 | Use variables and logic to create dynamic configurations that adapt based on server or environment specifics. 73 | 74 | - **Roles** 75 | A method for organizing playbooks and tasks into reusable components. These help to structure your Ansible code for better readability and maintenance. 76 | 77 | ### Ansible Architecture 78 | 79 | **Overview:** 80 | Ansible uses a control node and managed nodes to perform its tasks. The control node sends instructions (playbooks) via SSH to managed nodes without needing any agents. 81 | 82 | **Flow of Operation:** 83 | 84 | 1. Write the playbook defining the desired state. 85 | 2. Define the inventory (managed nodes) and group the servers as needed. 86 | 3. Run the playbook from the control node. 87 | 4. Ansible connects to the managed nodes via SSH and executes the required modules. 88 | 5. The output is displayed after completion. 89 | 90 | ![Ansible Architecture](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/110/156/original/ansiblearch.png?1739985222) 91 | 92 | **Use Cases of Ansible:** 93 | 94 | - Configuration Management 95 | - Application Deployment 96 | - Continuous Delivery 97 | 98 | 99 | ### Instructor Notes for Demo: 100 | 101 | - **Setup:** 102 | Ensure you have access to the control node from a local machine via SSH. The control node must be able to connect to the managed nodes. 103 | 104 | - **Key Steps for Setup:** 105 | 1. Copy the public key from the control node to the managed nodes' authorized keys. 106 | 2. Options for setting up SSH: 107 | - Manually copy the public key from the control node to each managed node. 108 | - Use a PEM file to log in to managed nodes from the control node and then copy the public key. 109 | 110 | -------------------------------------------------------------------------------- /DevOps Tools 2 Module/Ansible Roles & Advanced Topics.md: -------------------------------------------------------------------------------- 1 | # Ansible Roles & Advanced Topics 2 | 3 | ## **Jinja2 Templating** 4 | - Jinja2 is a powerful templating engine for Python, used by Ansible to manage configuration files dynamically. 5 | - It allows generating dynamic content and manipulating data efficiently. 6 | - Useful for creating configuration files for different environments. 7 | 8 | ### **Example Use Case** 9 | If managing an Apache server where each server needs different virtual host configurations, Jinja2 allows using a single template. At runtime, Ansible fills in details specific to each server. 10 | 11 | ### **Jinja2 Syntax** 12 | - Variables: `{{ variable_name }}` 13 | - Logical Statements: `{% logic %}` 14 | 15 | #### **Example Jinja2 Template for Apache VirtualHost** 16 | ```apache 17 | 18 | ServerName {{ server_name }} 19 | {% for alias in server_aliases %} 20 | ServerAlias {{ alias }} 21 | {% endfor %} 22 | DocumentRoot {{ document_root }} 23 | ErrorLog {{ error_log }} 24 | CustomLog {{ custom_log }} combined 25 | 26 | ``` 27 | This template dynamically generates an Apache VirtualHost configuration. 28 | 29 | ### **Jinja2 in Ansible Playbooks** 30 | ```yaml 31 | - name: Deploy Apache VirtualHost configuration 32 | hosts: webservers 33 | become: yes 34 | vars: 35 | server_name: "example.com" 36 | server_aliases: 37 | - "www.example.com" 38 | - "example.org" 39 | document_root: "/var/www/html" 40 | error_log: "/var/log/apache2/error.log" 41 | custom_log: "/var/log/apache2/access.log" 42 | 43 | tasks: 44 | - name: Deploy VirtualHost configuration file 45 | template: 46 | src: apache_virtualhost.j2 47 | dest: /etc/apache2/sites-available/000-default.conf 48 | mode: '0644' 49 | ``` 50 | 51 | --- 52 | 53 | ## **Prompting for Variables** 54 | - Instead of hardcoding variables, Ansible allows prompting users for input at runtime using `vars_prompt`. 55 | 56 | #### **Example: Prompting for User Input** 57 | ```yaml 58 | - name: Prompt for variables 59 | hosts: localhost 60 | gather_facts: no 61 | vars_prompt: 62 | - name: username 63 | prompt: "Enter the username" 64 | private: no 65 | 66 | tasks: 67 | - name: Display entered username 68 | debug: 69 | msg: "The username is {{ username }}" 70 | ``` 71 | 72 | --- 73 | 74 | ## **Roles in Ansible** 75 | - Roles provide a structured way to organize Ansible playbooks into reusable components. 76 | - This makes automation code modular, scalable, and easier to maintain. 77 | 78 | ### **Example: A Web Server Role** 79 | A role for setting up a web server might include: 80 | - Installing the web server 81 | - Configuring it 82 | - Ensuring it runs 83 | 84 | ### **Ansible Role Directory Structure** 85 | ``` 86 | roles/ 87 | └── my-role/ 88 | ├── tasks/ 89 | │ └── main.yml 90 | ├── vars/ 91 | │ └── main.yml 92 | ├── handlers/ 93 | │ └── main.yml 94 | ├── templates/ 95 | │ └── main.yml 96 | ├── defaults/ 97 | ``` 98 | - Instead of keeping everything in a single playbook, roles help modularize automation tasks. 99 | 100 | --- 101 | 102 | ## **Ansible Galaxy** 103 | - **Ansible Galaxy** is a platform for sharing and downloading Ansible roles. 104 | - Helps reuse existing roles instead of creating everything from scratch. 105 | 106 | ### **Common Ansible Galaxy Commands** 107 | - **Creating a new role:** 108 | ```bash 109 | ansible-galaxy init [role_name] 110 | ``` 111 | - **Installing predefined roles:** 112 | ```bash 113 | ansible-galaxy install [role_name] 114 | ``` 115 | - **Listing installed roles:** 116 | ```bash 117 | ansible-galaxy list 118 | ``` 119 | 120 | --- 121 | 122 | ## **Conditions & Loops in Ansible** 123 | 124 | ### **Conditions in Ansible** 125 | #### **Using `when` for Conditional Execution** 126 | ```yaml 127 | - name: Ensure httpd is installed 128 | yum: 129 | name: httpd 130 | state: present 131 | when: ansible_facts["os_family"] == "RedHat" 132 | ``` 133 | 134 | #### **AND Condition** 135 | ```yaml 136 | when: 137 | - ansible_facts["os_family"] == "RedHat" 138 | - ansible_facts["open-in"] >= 0 139 | ``` 140 | 141 | #### **OR Condition** 142 | ```yaml 143 | when: 144 | - ansible_facts["os_family"] == "RedHat" or ansible_facts["os_family"] == "Debian" 145 | ``` 146 | 147 | ### **Loops in Ansible** 148 | Ansible supports loops using: 149 | 1. `with_items` (older approach) 150 | 2. `loop` (recommended approach) 151 | 152 | #### **Example: Installing Multiple Packages** 153 | Using `with_items`: 154 | ```yaml 155 | - name: Install packages 156 | apt: 157 | name: "{{ item }}" 158 | state: present 159 | with_items: 160 | - git 161 | - curl 162 | - nginx 163 | ``` 164 | 165 | Using `loop`: 166 | ```yaml 167 | - name: Install packages 168 | apt: 169 | name: "{{ item }}" 170 | state: present 171 | loop: 172 | - git 173 | - curl 174 | - nginx 175 | ``` 176 | 177 | #### **Looping Over Dictionaries** 178 | ```yaml 179 | - name: Create users with groups 180 | user: 181 | name: "{{ item.name }}" 182 | groups: "{{ item.groups }}" 183 | loop: 184 | - { name: "deepak", groups: "admin" } 185 | - { name: "dev", groups: "dev" } 186 | ``` 187 | This dynamically creates users and assigns them specified groups. 188 | 189 | --- 190 | -------------------------------------------------------------------------------- /DevOps Tools 2 Module/Ansible Vault & Best Practices.md: -------------------------------------------------------------------------------- 1 | # Typed Notes: Ansible Vault & Best Practices 2 | 3 | ## Tags in Ansible 4 | 5 | - **Tags** are labels assigned to tasks, plays, and roles in Ansible. They allow for control over which tasks or plays are executed in a playbook. 6 | - Tags enable running specific parts of a playbook without executing the entire script. This is useful when you need to execute only certain tasks, like installation or configuration. 7 | 8 | ### Example of Tags Usage: 9 | ```yaml 10 | - name: Install Software 11 | yum: 12 | name: my_software 13 | state: present 14 | tags: 15 | - install 16 | 17 | - name: Configure Software 18 | template: 19 | src: config.j2 20 | dest: /etc/my_software/config.conf 21 | tags: 22 | - config 23 | 24 | - name: Deploy Software 25 | shell: /usr/local/bin/deploy.sh 26 | tags: 27 | - deploy 28 | ``` 29 | 30 | To run tasks based on tags: 31 | ```bash 32 | ansible-playbook playbook.yml --tags install # Runs only tasks with the "install" tag 33 | ansible-playbook playbook.yml --skip-tags install # Skips tasks with the "install" tag 34 | ansible-playbook playbook.yml --tags "install,config" # Runs tasks with either "install" or "config" tags 35 | ``` 36 | 37 | ### Why Use Tags: 38 | - They improve efficiency in large playbooks. 39 | - Tags help skip unnecessary tasks and focus on relevant operations, such as install, config, or deploy. 40 | 41 | --- 42 | 43 | ## Handlers in Ansible 44 | 45 | - **Handlers** are special tasks in Ansible that only run when notified by other tasks. These tasks typically perform actions like restarting services or reloading configurations, which should only happen after a change is made. 46 | - Handlers are triggered when a task notifies them, and they ensure that a task is executed only when necessary. 47 | 48 | ### Example of Handlers: 49 | ```yaml 50 | - name: Configure Nginx 51 | hosts: webservers 52 | become: yes # Run tasks with sudo privileges 53 | tasks: 54 | - name: Deploy Nginx configuration 55 | template: 56 | src: nginx.conf.j2 57 | dest: /etc/nginx/nginx.conf 58 | owner: root 59 | group: root 60 | mode: '0644' 61 | notify: Restart Nginx 62 | 63 | handlers: 64 | - name: Restart Nginx 65 | service: 66 | name: nginx 67 | state: restarted 68 | ``` 69 | 70 | ### Practical Use Case: 71 | - Restart services only after configuration changes. 72 | - Handlers help avoid unnecessary service restarts, reducing interruptions in your environment. 73 | 74 | --- 75 | 76 | ## Ansible Vault 77 | 78 | - **Ansible Vault** is a feature that encrypts sensitive data, such as passwords, API keys, and private information, within playbooks or variable files. 79 | - It ensures sensitive data is stored securely, preventing exposure in plain text while still allowing safe sharing and automation. 80 | 81 | ### Basic Commands for Ansible Vault: 82 | - `ansible-vault encrypt `: Encrypt a file. 83 | - `ansible-vault decrypt `: Decrypt a file. 84 | - `ansible-vault edit `: Edit an encrypted file without decrypting it. 85 | 86 | ### Usage Example: 87 | - **Using a password file** to run a playbook with encrypted content: 88 | ```bash 89 | ansible-playbook --vault-password-file path-to-file 90 | ``` 91 | 92 | - Best practice is to **exclude** password files from version control using `.gitignore`. 93 | 94 | --- 95 | 96 | ## Error Handling in Ansible 97 | 98 | 1. **Increasing Verbosity**: 99 | - The `-v`, `-vv`, and `-vvv` flags allow you to control the verbosity of the output when running a playbook. 100 | - `-v`: Basic output. 101 | - `-vv`: More detailed output. 102 | - `-vvv`: The most detailed output, helpful for debugging. 103 | 104 | 2. **Dry Run Mode (`--check`)**: 105 | - The `--check` option allows you to simulate the execution of a playbook without making any changes, which is useful for validation. 106 | - Example: 107 | ```bash 108 | ansible-playbook playbook.yml --check 109 | ``` 110 | 111 | Example Playbook with `--check`: 112 | ```yaml 113 | - name: Install and Start NGINX 114 | hosts: webservers 115 | become: yes 116 | tasks: 117 | - debug: 118 | msg: "This is a DRY RUN, no changes will be made!" 119 | when: ansible_check_mode 120 | 121 | - name: Install NGINX 122 | package: 123 | name: nginx 124 | state: present 125 | 126 | - name: Start and Enable NGINX 127 | service: 128 | name: nginx 129 | state: started 130 | enabled: yes 131 | ``` 132 | 133 | **With `--check`**: The playbook will simulate changes but not apply them. 134 | **Without `--check`**: The playbook will apply the changes (install and start NGINX). 135 | 136 | --- 137 | 138 | ## Best Practices in Ansible 139 | 140 | ### 1. **Directory Layout**: 141 | - **Separation of Environments**: Maintain separate configurations for different environments, like `production` and `staging`. 142 | - **Group and Host Variables**: Use `group_vars` and `host_vars` for environment-specific or host-specific configurations. 143 | - **Roles**: Organize tasks into reusable roles (`common`, `webservers`, `python`) to promote modularity and easier maintenance. 144 | - **Main Playbook**: The `main.yml` serves as the entry point for the automation process, ensuring clarity and structure. 145 | 146 | ### 2. **Reusability and Modularity**: 147 | - Break tasks into reusable, self-contained units using roles. 148 | - Store environment or host-specific configurations in `group_vars` and `host_vars` for better flexibility. 149 | - Use `when` conditions to apply tasks conditionally. 150 | 151 | ### 3. **Performance Optimization**: 152 | - Use `async` and `poll` to handle long-running tasks asynchronously. 153 | 154 | Example: 155 | ```yaml 156 | - name: Restart Nginx 157 | service: 158 | name: nginx 159 | state: restarted 160 | async: 300 # Run asynchronously with a maximum execution time of 300 seconds 161 | poll: 0 # Do not wait for completion, continue with other tasks 162 | ``` 163 | 164 | - **Caching Facts**: Cache facts to speed up the process of gathering data from remote hosts. 165 | -------------------------------------------------------------------------------- /DevOps Tools 2 Module/Efficient Workflows and Security Practices.md: -------------------------------------------------------------------------------- 1 | # Efficient Workflows and Security Practices 2 | 3 | ## Cache and Dependencies 4 | 5 | ### Understanding Caching in CI/CD Workflows 6 | 7 | When building software or installing dependencies, the initial setup can take a significant amount of time. However, subsequent installations tend to be much faster. 8 | 9 | **Question :** Why do dependencies take longer to install initially but run faster in later executions? 10 | **Answer:** The first installation downloads and sets up all required dependencies, but later runs can **reuse cached dependencies** instead of reinstalling them, thereby saving time. This optimization is achieved through **caching**. 11 | 12 | ### What is a Cache? 13 | 14 | A **cache** is a high-speed data storage layer that retains frequently accessed data, making it readily available for future use without needing to fetch it from the original source again. 15 | 16 | ### How Caching Works in GitHub Actions/GitLab CI 17 | 18 | Caching in CI/CD workflows works by storing frequently used files in a **shared location** so they can be accessed across multiple workflow runs. 19 | 20 | **Example:** 21 | 1. **First Run:** 22 | - Dependency Installation: **10 minutes** 23 | - Code Execution: **2 minutes** 24 | - **Total Execution Time:** **12 minutes** 25 | 2. **Subsequent Runs:** 26 | - Dependency Installation **skipped (cached)** 27 | - Code Execution: **2 minutes** 28 | - **Total Execution Time:** **2 minutes** 29 | 30 | By caching dependencies, we avoid redundant installations and significantly improve workflow speed. 31 | 32 | ### Types of Cache Used in CI/CD 33 | 34 | - **Package Dependency Caching** – Stores package manager dependencies like `node_modules`, `pip`, or `maven` dependencies. 35 | - **Build Artifacts** – Reuses compiled code, saving time on future builds. 36 | - **Temporary Files** – Retains logs, reports, and other intermediate files for quick reference. 37 | 38 | ### Example: Caching npm Dependencies in GitHub Actions 39 | 40 | Below is an example of caching **node_modules** to avoid unnecessary dependency installations: 41 | 42 | ![npm script](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/112/637/original/Screenshot_2025-03-06_071255.png?1741225401) 43 | 44 | - The **cache key** is used to create an identifier for stored dependencies. 45 | - If no cache is found for the provided key, the workflow falls back to the **default cache**. 46 | 47 | ### Identifying Your Cache 48 | 49 | **Question :** How do we identify which cache is being used in our workflow? 50 | **Answer:** By referencing the **npm static value** assigned during caching. 51 | 52 | ### Branch-Based Cache Referencing 53 | 54 | GitHub Actions allow caches to be referenced across branches. The diagram below illustrates how different branches can **reuse cache data**: 55 | 56 | ![Cache branches](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/112/639/original/Screenshot_2025-03-06_073317.png?1741226613) 57 | 58 | - **Test B** can reuse cache from **Test A** and the **Main branch**. 59 | - **Test A** can only reference the **Main branch**. 60 | - The **Main branch** can only use its own cache. 61 | 62 | --- 63 | 64 | ## Services 65 | 66 | ### What Are Services in CI/CD Workflows? 67 | 68 | In CI/CD workflows, services refer to **additional containers** that run alongside the primary container, providing essential dependencies required for job execution. 69 | 70 | ### Key Characteristics of Services 71 | 72 | - Services are **started before the primary job**. 73 | - They remain **active throughout** the job execution. 74 | - If a **service fails**, the job relying on it will also **fail**. 75 | - When the **job completes**, the associated **service shuts down automatically**. 76 | 77 | ### Example: Using a Database Service in a CI/CD Pipeline 78 | 79 | Consider a case where a test suite requires a **PostgreSQL database**. Instead of installing and configuring PostgreSQL within the job, a **service container** can be used: 80 | 81 | ![Service Example](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/112/640/original/Screenshot_2025-03-06_074121.png?1741227090) 82 | 83 | 84 | --- 85 | 86 | ## Reusable Workflows 87 | 88 | ### Why Use Reusable Workflows? 89 | 90 | **Reusable workflows** allow teams to: 91 | 92 | - **Encapsulate common t 93 | s** for easy reuse across multiple workflows. 94 | - **Improve maintainability** by centralizing frequently used processes. 95 | - **Keep repositories clean** and modular. 96 | 97 | ### How Do Reusable Workflows Work? 98 | 99 | A **reusable workflow** is a predefined set of jobs that can be **called by other workflows**. However, a reusable workflow **does not execute on its own**; it must be invoked by another workflow. 100 | 101 | ### Example: Using a Reusable Workflow 102 | 103 | #### 1. Creating a Reusable Workflow 104 | 105 | ![Reusable Workflow](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/112/641/original/Screenshot_2025-03-06_080348.png?1741228437) 106 | 107 | #### 2. Calling the Reusable Workflow 108 | 109 | ![Triggering Workflow](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/112/642/original/Screenshot_2025-03-06_080324.png?1741228536) 110 | 111 | - The **calling workflow** specifies **which environment** to use. 112 | 113 | ### Understanding Workflow Dispatch 114 | 115 | - **Workflow Dispatch** defines whether the workflow should be **manually triggered** or executed automatically. 116 | 117 | ### Real-World Example 118 | 119 | ![Reusable Jobs](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/112/643/original/Screenshot_2025-03-06_081328.png?1741229027) 120 | 121 | - Workflows like **Twistloc Scan** and **Block Due Scan** are being **reused** across multiple jobs. 122 | 123 | --- 124 | 125 | ## Workflow Optimizations 126 | 127 | ### Techniques for Optimizing Workflows 128 | 129 | Optimizations help reduce execution time and resource consumption. Common techniques include: 130 | 131 | 1. **Caching Dependencies** – Avoids redundant package installations. 132 | 2. **Matrix Builds** – Runs jobs in **parallel across multiple environments**. 133 | 3. **Limiting Workflow Runs** – Prevents unnecessary executions by using: 134 | - **Path triggers** 135 | - **Event filters** 136 | - **Action conditions** 137 | 4. **Job Timeouts** – Prevents **stuck jobs** by setting execution limits. 138 | 5. **Self-Hosted Runners** – Uses **custom environments** for better control. 139 | 6. **Using Artifacts** – Saves **intermediate results** for reuse in later jobs. 140 | 141 | ### Difference Between Artifacts and Caching 142 | 143 | - **Caching**: Stores large, frequently used dependencies. 144 | - **Artifacts**: Stores intermediate **build outputs** for reuse. 145 | 146 | --- 147 | 148 | ## Limitations of CI/CD Workflows 149 | 150 | ### Key Limitations 151 | 152 | 1. **Limited Compute Resources** 153 | - GitHub-hosted runners provide **2 CPUs, 7GB RAM**. 154 | - Workflow execution is limited to **72 hours**. 155 | 156 | 2. **Storage Constraints** 157 | - **Maximum artifact storage:** **5GB per repository**. 158 | - Artifacts are **deleted after 90 days** unless stored externally. 159 | 160 | 3. **Lack of Stateful Workflow Support** 161 | - No built-in way to track workflow state across multiple runs. 162 | 163 | 4. **Secret Management Limitations** 164 | - Lacks **automatic secret rotation** and **expiration management**. 165 | 166 | --- 167 | -------------------------------------------------------------------------------- /DevOps Tools 2 Module/Github Actions - Advanced Techniques and Best Practices: -------------------------------------------------------------------------------- 1 | # GitHub Workflows, Branching Strategies & Expression Examples 2 | 3 | 4 | # 1. Manual Workflows 5 | 6 | ## Concept 7 | GitHub Actions allow the execution of workflows in multiple ways, including automatic and manual triggers. The `workflow_dispatch` event is specifically used for manually triggering workflows. This is useful in cases where automated triggers are not ideal, such as running deployments on demand or executing administrative tasks that require user input. 8 | 9 | ## Key Features 10 | - **Workflow Dispatch:** Allows users to initiate workflows manually from the GitHub UI. 11 | - **Custom Inputs:** Users can specify parameters (such as region, image, environment, etc.) when triggering the workflow manually. 12 | 13 | ### Example YAML Configuration: 14 | 15 | ```yaml 16 | name: Manual Deployment 17 | 18 | on: 19 | workflow_dispatch: 20 | inputs: 21 | region: 22 | description: 'Target region for deployment' 23 | required: true 24 | default: 'us-east-1' 25 | image: 26 | description: 'Docker image to deploy' 27 | required: true 28 | 29 | jobs: 30 | deploy: 31 | runs-on: ubuntu-latest 32 | steps: 33 | - name: Checkout Code 34 | uses: actions/checkout@v2 35 | - name: Deploy Application 36 | run: | 37 | echo "Deploying image ${{ github.event.inputs.image }} to region ${{ github.event.inputs.region }}" 38 | ``` 39 | 40 | ### Breakdown 41 | - The `workflow_dispatch` keyword defines a manual trigger. 42 | - `inputs` define parameters that users must provide when triggering the workflow. 43 | - The `deploy` job runs on `ubuntu-latest` and executes steps such as checking out the repository and deploying an application. 44 | - `github.event.inputs` is used to reference user-provided values in the workflow. 45 | 46 | # 2. Pipeline Setup 47 | 48 | ## Concept 49 | A **pipeline** is a structured sequence of automated processes that handle software integration, testing, and deployment. Pipelines ensure a streamlined development cycle by automating tasks such as code validation, compilation, testing, and deployment. 50 | 51 | ### Pipeline Keywords 52 | - **CI (Continuous Integration):** Ensures that code changes are automatically built and tested. 53 | - **CD (Continuous Deployment/Delivery):** Automates the deployment of software to production environments. 54 | - **Pipeline Triggers:** Pipelines can be triggered by events such as code pushes, pull requests, schedule-based executions, or manual interventions. 55 | 56 | ## Workflow Types 57 | - **Automated Triggers:** Initiate workflows based on push events, pull requests, or scheduled intervals. 58 | - **Manual Initiation:** Allows users to trigger workflows on demand. 59 | 60 | ### Example YAML Snippet for a CI Pipeline 61 | 62 | ```yaml 63 | name: CI Pipeline 64 | 65 | on: 66 | push: 67 | branches: 68 | - main 69 | - develop 70 | pull_request: 71 | branches: 72 | - main 73 | - develop 74 | 75 | jobs: 76 | build: 77 | runs-on: ubuntu-latest 78 | steps: 79 | - name: Checkout Repository 80 | uses: actions/checkout@v2 81 | - name: Run Tests 82 | run: npm test 83 | - name: Build Application 84 | run: npm run build 85 | ``` 86 | 87 | ### Breakdown 88 | - **Triggers:** The workflow runs on `push` and `pull_request` events for `main` and `develop` branches. 89 | - **Steps:** The pipeline follows a sequence: 90 | - Checking out the repository. 91 | - Running tests (`npm test`). 92 | - Building the application (`npm run build`). 93 | 94 | # 3. Workflow Triggers and Filters 95 | 96 | ## Concept 97 | GitHub Actions allow workflows to be triggered based on specific events and provide filters to control when these workflows execute. 98 | 99 | ## Triggers 100 | - **Push Events:** Triggered when new commits are pushed to the repository. 101 | - **Pull Request Events:** Triggered when pull requests are opened, updated, or merged. 102 | - **Scheduled Triggers:** Run workflows at predefined intervals using cron syntax. 103 | - **Manual Triggers (`workflow_dispatch`)**: Allow users to execute workflows on demand. 104 | 105 | ## Filters 106 | - **Branch Filters:** Restrict workflows to execute only for specific branches. 107 | - **Path Filters:** Allow workflows to run only if changes occur in specified directories or files. 108 | 109 | ### Example: Branch and Path Filters 110 | 111 | ```yaml 112 | name: Filtered Workflow 113 | 114 | on: 115 | push: 116 | branches: 117 | - main 118 | - develop 119 | paths: 120 | - 'src/**' 121 | - 'config/**' 122 | pull_request: 123 | branches: 124 | - main 125 | 126 | jobs: 127 | test: 128 | runs-on: ubuntu-latest 129 | steps: 130 | - uses: actions/checkout@v2 131 | - run: npm run test 132 | ``` 133 | 134 | ### Breakdown 135 | - The workflow runs only when changes occur in `src/` or `config/` directories. 136 | - It applies to `push` and `pull_request` events for specific branches. 137 | 138 | # 4. Expressions and Context Objects 139 | 140 | ## Concept 141 | Expressions in GitHub Actions enable dynamic control over workflows by evaluating conditions, manipulating strings, and applying logic within YAML files. 142 | 143 | ### Features 144 | - **Conditionals:** Run steps based on certain conditions (`if` statements). 145 | - **String Operations:** Validate string content with functions like `contains`, `startsWith`, and `endsWith`. 146 | - **Logical & Mathematical Operations:** Combine multiple conditions or perform arithmetic operations. 147 | - **Ternary Operator:** Shortens conditional assignments. 148 | 149 | ### Example: Using Conditional Expressions 150 | 151 | ```yaml 152 | jobs: 153 | conditional-job: 154 | runs-on: ubuntu-latest 155 | steps: 156 | - name: Check Branch and Run Command 157 | run: | 158 | if [ "${{ github.ref }}" == "refs/heads/main" ]; then 159 | echo "Running on main branch" 160 | else 161 | echo "Not on main branch" 162 | fi 163 | - name: Using Ternary Operator 164 | run: | 165 | echo "The value is ${{ github.event_name == 'push' && 'Push Event' || 'Other Event' }}" 166 | ``` 167 | 168 | ### Breakdown 169 | - The job evaluates whether the workflow is running on the `main` branch. 170 | - A ternary operator is used to print different messages based on the event type. 171 | 172 | # 5. Branching Strategies in Git 173 | 174 | ## Concept 175 | Effective branch management is critical for streamlined development and release processes. 176 | 177 | ### Common Branching Strategies 178 | - **Main Branch:** Contains stable, production-ready code. 179 | - **Develop Branch:** A working branch where features are integrated before merging into `main`. 180 | - **Feature Branches:** Used for developing new features. 181 | - **Release Branches:** Prepare features for production releases. 182 | - **Hotfix Branches:** Address urgent issues in production. 183 | 184 | ### Example: Branch-Based CI/CD Workflow 185 | 186 | ```yaml 187 | name: Branch Strategy Workflow 188 | 189 | on: 190 | push: 191 | branches: 192 | - main 193 | - develop 194 | - 'feature/*' 195 | - 'release/*' 196 | - 'hotfix/*' 197 | 198 | jobs: 199 | build: 200 | runs-on: ubuntu-latest 201 | steps: 202 | - uses: actions/checkout@v2 203 | - name: Run Build 204 | run: echo "Building branch ${{ github.ref }}" 205 | 206 | deploy: 207 | if: github.ref == 'refs/heads/main' 208 | runs-on: ubuntu-latest 209 | steps: 210 | - name: Deploy to Production 211 | run: echo "Deploying main branch to production..." 212 | ``` 213 | 214 | ### Breakdown 215 | - Workflows are triggered based on branch naming conventions. 216 | - A conditional `deploy` job ensures deployments occur only on `main`. 217 | 218 | 219 | -------------------------------------------------------------------------------- /DevOps Tools 2 Module/Introduction to CI CD & Jenkins.md: -------------------------------------------------------------------------------- 1 | ## Typed Notes - Class 1 - Introduction to CI/CD & Jenkins 2 | 3 | ## **CI/CD** 4 | 5 | ### **Continuous Integration (CI)** 6 | - **Definition:** 7 | Continuous Integration (CI) is a development practice where developers frequently integrate code into a shared repository. 8 | Each integration is automatically verified by automated builds and tests. 9 | 10 | - **Purpose:** 11 | - Ensures early detection of integration issues. 12 | - Encourages collaboration among team members. 13 | 14 | - **Process:** 15 | 1. Developers commit code to a version control system (e.g., Git). 16 | 2. An automated build process compiles the code and runs tests. 17 | 3. If the build and tests pass, the changes are merged into the main branch. 18 | 19 | - **Benefits:** 20 | - Faster feedback on code changes. 21 | - Reduces integration challenges during the development cycle. 22 | - Improves code quality through automated testing. 23 | 24 | --- 25 | 26 | ### **Continuous Deployment (CD)** 27 | - **Definition:** 28 | Continuous Deployment is the practice of automatically deploying code changes to the production environment without manual intervention. 29 | 30 | - **Key Features:** 31 | - Ensures faster delivery of new features to users. 32 | - Requires robust automated testing to ensure reliability. 33 | 34 | - **Process:** 35 | 1. Code changes are pushed to a repository. 36 | 2. CI processes validate the changes through builds and tests. 37 | 3. If successful, the code is deployed directly to production. 38 | 39 | - **Benefits:** 40 | - Accelerates software delivery cycles. 41 | - Reduces manual errors during deployment. 42 | - Enhances user satisfaction by delivering updates quickly. 43 | 44 | --- 45 | 46 | ### **Continuous Delivery** 47 | - **Definition:** 48 | Continuous Delivery involves automating the delivery process up to the production environment but requires manual approval to deploy the changes. 49 | 50 | - **Process:** 51 | 1. Automated CI builds and tests code changes. 52 | 2. Artifacts are prepared and validated for deployment. 53 | 3. Deployment to production requires manual approval. 54 | 55 | - **Use Case:** 56 | - Suitable for organizations that require regulatory checks or approvals before production deployment. 57 | 58 | ![image](https://hackmd.io/_uploads/SkH5fIjwke.png) 59 | 60 | --- 61 | 62 | ## **Jenkins Architecture** 63 | 64 | ### **Overview of Jenkins** 65 | - **Purpose:** 66 | Jenkins is an open-source automation server that facilitates the automation of software development processes such as build, test, and deployment. 67 | 68 | - **Features:** 69 | - Automates repetitive tasks in software delivery. 70 | - Integrates with tools like Docker, AWS, GitHub, and more. 71 | 72 | - **Core Components:** 73 | 1. **Jenkins Server:** 74 | - The centralized server managing configurations and jobs. 75 | 2. **Jobs:** 76 | - Represents tasks or projects executed by Jenkins. 77 | 3. **Users:** 78 | - Manages user access and permissions. 79 | 4. **Plugins:** 80 | - Extend Jenkins functionality (e.g., Git, Docker plugins). 81 | 82 | --- 83 | 84 | ![image](https://hackmd.io/_uploads/H1UCmUoDkx.png) 85 | 86 | 87 | ### **Jenkins Architecture** 88 | 1. **Server:** 89 | - Centralized component responsible for orchestrating jobs and managing configurations. 90 | 2. **Agents:** 91 | - Remote executors that offload job execution to improve scalability. 92 | 3. **Plugins:** 93 | - Enhance Jenkins’ capabilities by adding features like integration with third-party tools. 94 | 95 | --- 96 | 97 | ### **Jenkins Installation Requirements** 98 | 1. **System Requirements:** 99 | - Minimum 1 GB RAM (higher recommended for larger projects). 100 | - Minimum 4 GB disk space for Jenkins and build data. 101 | 102 | 2. **Dependencies:** 103 | - **Java:** 104 | - Required to run Jenkins. 105 | - Ensure the appropriate version is installed before setup. 106 | 107 | --- 108 | 109 | ### **Jenkins Configuration** 110 | 1. **Plugin Management:** 111 | - Install required plugins (e.g., GitHub, Docker, Pipeline). 112 | - Use the plugin manager available in the Jenkins UI. 113 | 114 | 2. **Credential Management:** 115 | - Add secure credentials (e.g., API tokens, SSH keys) in Jenkins. 116 | - Use Jenkins’ credential manager for secure storage. 117 | 118 | 3. **Pipeline Setup:** 119 | - Define workflows using `Jenkinsfile`. 120 | - Example stages: `Build → Test → Deploy`. 121 | 122 | 4. **Tool Integration:** 123 | - **Docker:** Automate containerized builds. 124 | - **AWS:** Enable cloud deployment and resource provisioning. 125 | - **GitHub:** Automate code fetching using Git repositories. 126 | 127 | --- 128 | 129 | ### **Types of Jenkins Jobs** 130 | 1. **Freestyle Projects:** 131 | - Basic projects with manual configuration options. 132 | - Suitable for simple automation tasks. 133 | 134 | 2. **Pipeline Projects:** 135 | - Define the entire build process as code. 136 | - Written in a `Jenkinsfile`. 137 | - Provides flexibility and version control. 138 | 139 | 3. **Multi-Configuration Jobs:** 140 | - Useful for testing across multiple environments (e.g., OS, browsers). 141 | 142 | --- 143 | 144 | ### **Pipeline Configuration** 145 | 1. **Pipeline as Code:** 146 | - Written in a `Jenkinsfile`. 147 | - Stored in the version control repository alongside the codebase. 148 | 149 | 2. **Pipeline Stages:** 150 | - Example: 151 | ```groovy 152 | pipeline { 153 | agent any 154 | stages { 155 | stage('Build') { 156 | steps { 157 | echo 'Building the project...' 158 | } 159 | } 160 | stage('Test') { 161 | steps { 162 | echo 'Running tests...' 163 | } 164 | } 165 | stage('Deploy') { 166 | steps { 167 | echo 'Deploying to production...' 168 | } 169 | } 170 | } 171 | } 172 | ``` 173 | 174 | 3. **Integration with Tools:** 175 | - Docker: Build and deploy containers. 176 | - AWS: Cloud deployment and scaling. 177 | - GitHub: Source code integration for CI/CD workflows. 178 | 179 | --- 180 | 181 | ### **Tips and Best Practices** 182 | 1. **Use Plugins Wisely:** 183 | - Select plugins that enhance productivity and suit project requirements. 184 | 185 | 2. **Automate Testing:** 186 | - Include unit, integration, and functional tests in CI/CD pipelines. 187 | 188 | 3. **Secure Credentials:** 189 | - Always use the Jenkins credential manager for storing sensitive information. 190 | 191 | 4. **Monitor Resources:** 192 | - Regularly check server performance (RAM, CPU, disk space). 193 | 194 | 5. **Notifications:** 195 | - Configure email or chat notifications for build statuses to keep stakeholders informed. 196 | 197 | --- 198 | 199 | ### **Quick Reference** 200 | - **Credentials:** 201 | - Securely store tokens, keys, and other secrets using the Jenkins credential manager. 202 | - **Pipeline Syntax:** 203 | - Use declarative pipelines (`Jenkinsfile`) for consistency and maintainability. 204 | - **Scalability:** 205 | - Leverage agents for distributed builds. 206 | - **Notifications:** 207 | - Set up notifications to alert about success, failure, or other events. 208 | 209 | --- 210 | -------------------------------------------------------------------------------- /DevOps Tools 2 Module/Introduction to Chef.md: -------------------------------------------------------------------------------- 1 | ## Introduction to Chef 2 | 3 | ### Ansible Vs. Chef 4 | 5 | - **Ease of Use**: 6 | - **Ansible**: Easy to use, based on YAML. 7 | - **Chef**: Complex setup, uses Ruby DSL. 8 | 9 | - **Architecture**: 10 | - **Ansible**: Agentless. 11 | - **Chef**: Requires agent installation. 12 | 13 | - **Performance**: 14 | - **Ansible**: Lightweight, but has performance concerns. 15 | - **Chef**: Better performance overall. 16 | 17 | - **Language**: 18 | - **Ansible**: Uses YAML for playbooks. 19 | - **Chef**: Uses Ruby-based DSL. 20 | 21 | - **Strengths**: 22 | - **Ansible**: Simple, lightweight, and easy to learn. 23 | - **Chef**: Powerful DSL and good performance. 24 | 25 | - **Weaknesses**: 26 | - **Ansible**: Performance limitations. 27 | - **Chef**: Steep learning curve and complex setup. 28 | 29 | ### Terminologies in Chef 30 | 31 | - **Recipe**: A set of instructions in Ruby (Chef) or YAML (Ansible) to perform automation tasks. 32 | - **Resource**: A unit inside recipes/playbooks defining actions like package installation or file creation. 33 | - **Cookbook**: A collection of recipes (Chef) or tasks (Ansible) used for configuration management. 34 | - **Attribute**: Defines configurable parameters in Chef, similar to variables or facts in Ansible. 35 | - **Data Bag**: Stores JSON-based data, similar to Ansible Vault, used for storing secrets or configurations. 36 | - **Knife**: A command-line tool used to manage Chef resources, similar to Ansible's CLI. 37 | 38 | Chef is idempotent, meaning repeated executions of the same recipe lead to the same system state without unintended changes. It checks the current state before applying updates, ensuring predictability. 39 | 40 | ### Chef and Ansible Terminology Comparison 41 | 42 | | Chef Term | Ansible Term | 43 | |--------------------|----------------------| 44 | | Node Object | Managed Hosts | 45 | | Role | Role | 46 | | Custom Resource | Custom Modules | 47 | 48 | ### Chef Architecture 49 | 50 | Chef has a client-server architecture with three main components: 51 | 52 | 1. **Chef Server**: The central hub where configurations (cookbooks, roles, environments, etc.) are stored. It manages communications between nodes. 53 | 2. **Chef Clients (Nodes)**: Machines (servers, VMs, containers) managed by Chef. The Chef client pulls configuration data from the Chef server and applies it. 54 | 3. **Workstations**: Where users (often system admins) write and manage Chef code. Workstations interact with the Chef server to upload configurations. 55 | 56 | In contrast to Ansible, which is agentless and uses SSH to push configurations, Chef uses a pull-based mechanism where nodes periodically pull configurations from the Chef server. 57 | 58 | ![Chef Architecture](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/113/172/original/Screenshot_2025-03-10_150906.png?1741599900) 59 | 60 | #### Components Breakdown: 61 | - **Workstation (Laptop)**: Where recipes (configuration instructions) are written. 62 | - **Knife**: Tool used to upload recipes from the workstation to the Chef Server. 63 | - **Chef Server**: Stores recipes and configurations. 64 | - **Node (Chef Client)**: Communicates with the Chef Server, applying configurations. 65 | 66 | **Pull-based vs Push-based Mechanism**: 67 | - **Chef (default)**: Pull-based mechanism, where nodes fetch configurations from the server. 68 | - **Push-based**: Using tools like Chef Push Jobs, configurations can be pushed from the server to the nodes. 69 | - **Use Case**: Pull-based is ideal for frequent updates as it avoids pushing updates to thousands of machines. 70 | 71 | ![Chef Workflow](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/113/173/original/Screenshot_2025-03-10_150918.png?1741599955) 72 | 73 | ### Chef Installation 74 | 75 | In this section, you would typically go through the installation and configuration of Chef, covering the following points: 76 | 77 | - **Setting up Chef Server**: Install, configure, and initialize the Chef Server. 78 | - **User and Organization Setup**: Create a user and an organization, then generate the respective `.pem` files for authentication. 79 | - **Workstation Configuration**: 80 | - Install Chef Workstation. 81 | - Generate a default Chef repository. 82 | - Set up the `.chef` directory. 83 | - Generate `.pem` and `config.rb` files. 84 | - Set up a username and password. 85 | - **Workstation-Server Interaction**: Use Knife to establish a connection between the workstation and Chef Server. 86 | 87 | **Execution Steps**: 88 | - Create a cookbook on the workstation. 89 | - Upload the cookbook to the Chef Server. 90 | 91 | ### Roles in Chef 92 | 93 | Roles in Chef allow you to define sets of attributes and recipes to apply to multiple nodes. Roles help organize and streamline the application of configurations across nodes. 94 | 95 | - Roles are stored in the `/chef-repo/roles/` directory. 96 | - Each role includes a set of attributes and a list of recipes. 97 | 98 | ### Environment Files 99 | 100 | - **Environments**: Used to define different configurations based on deployment stages like development, testing, and production. 101 | - **Chef Environments**: Defined in JSON or Ruby. 102 | - **Version Control**: Environments ensure the right version of cookbooks is applied to nodes. 103 | 104 | **Example**: 105 | 106 | ```json 107 | { 108 | "name": "production", 109 | "cookbook_versions": { 110 | "apache2": ">= 3.0.0" 111 | } 112 | } 113 | ``` 114 | 115 | ### Databags 116 | 117 | - **Data Bags**: Store global information that can be shared across nodes, such as secrets, user credentials, and configuration data. 118 | - **Data Bags in Chef** are similar to **Vaults** in Ansible. 119 | -------------------------------------------------------------------------------- /DevOps Tools 2 Module/Introduction to Puppet.md: -------------------------------------------------------------------------------- 1 | ## **Introduction to Puppet** 2 | 3 | Puppet is a **configuration management tool** that automates infrastructure provisioning, deployment, and management using a **declarative language**. 4 | 5 | However, Puppet is not widely adopted due to its **complexity** and **steep learning curve**, as it is based on its own **Domain-Specific Language (DSL)** and requires an understanding of **Ruby**. 6 | 7 | ### **Features of Puppet** 8 | - Manages the **entire lifecycle** of a server—from setup, configuration, management, and retirement. 9 | - Provides **granular control by roles**, allowing configurations to be applied at the **individual node level** and enabling grouping of nodes. 10 | - Ensures that **local changes** on a server are overridden by the **desired state**. 11 | 12 | ## **Key Components of Puppet** 13 | 14 | ### **1. Manifests (.pp files)** 15 | - Core files where the desired system configuration is defined. 16 | - Contains **resources, classes, and definitions**. 17 | - Written in **DSL (Domain-Specific Language)**. 18 | - Equivalent to **Ansible Playbooks**. 19 | 20 | #### **Example:** 21 | ```puppet 22 | package { 'nginx': 23 | ensure => installed; 24 | } 25 | 26 | service { 'nginx': 27 | ensure => running, 28 | enable => true, 29 | } 30 | ``` 31 | 32 | ### **2. Resources** 33 | - The **smallest unit** in Puppet. 34 | - Manages **files, packages, services, and users**. 35 | - Equivalent to **Ansible Tasks**. 36 | 37 | ### **3. Classes** 38 | - **Reusable groups** of multiple resources. 39 | - Equivalent to **Ansible Modules**. 40 | 41 | #### **Example:** 42 | ```puppet 43 | class apache { 44 | package { 'httpd': 45 | ensure => installed, 46 | } 47 | } 48 | 49 | include apache 50 | ``` 51 | 52 | ### **4. Modules** 53 | - **Self-contained units** for organizing manifests, templates, and files. 54 | - Equivalent to **Ansible Roles**. 55 | 56 | ### **5. Facter** 57 | - Tool that collects **system information** (OS, IP, memory, etc.). 58 | - Puppet uses these **facts** to make configuration decisions. 59 | - Equivalent to **Ansible Facts (`ansible_facts`)**. 60 | 61 | #### **Example:** 62 | ```puppet 63 | if $facts['os']['family'] == 'Debian' { 64 | package { 'apache2': ensure => installed } 65 | } 66 | ``` 67 | 68 | ### **6. Hiera** 69 | - **Hierarchical data lookup system** to store configuration values externally. 70 | - No direct equivalent in **Ansible**, but similar to **variables, inventory, and vaults**. 71 | 72 | ## **Puppet Workflow** 73 | 74 | Puppet follows a **client-server (agent-master) architecture**, where configurations are defined on the **master** and applied on **agent nodes**. 75 | 76 | ### **Workflow Steps:** 77 | 1. **Fact Collection** → Puppet agent collects system information using **Facter**. 78 | 2. **Catalog Compilation** → Puppet master receives facts, checks manifests, modules, and Hiera, and compiles a **catalog** (desired system state). 79 | 3. **Catalog Delivery** → Compiled catalog is sent to the agent. 80 | 4. **Configuration Enforcement** → Agent applies the required configuration to the target host. 81 | 5. **Reporting & Logging** → Agent generates a **report** and sends it back to the Puppet master. 82 | 83 | ## **Puppet Installation** 84 | 85 | ### **Key Installation Steps:** 86 | - Create and manage files using **Puppet command-line tools**. 87 | - Create a **simple manifest file** in `/tmp`. 88 | - Create a **complex manifest file** using **modules** in a master-agent architecture. 89 | - Write **modules on the master** and apply them on the agent. 90 | 91 | ## **Modules vs. Classes** 92 | 93 | | **Modules** | **Classes** | 94 | |------------|------------| 95 | | Organize files, manifests, templates, and resources. | Manage configuration and group related resources. | 96 | | Exist at the **directory level** and contain **manifests, files, and templates**. | Define code logic that applies configurations to resources. | 97 | | Can contain **multiple classes and resources**. | Can be used within a module or across multiple modules. | 98 | 99 | ## **Key Takeaways** 100 | - Puppet is a **declarative configuration management tool**. 101 | - It follows a **client-server (Master-Agent) model**. 102 | - Key components include **Manifests, Resources, Classes, Modules, Facter, and Hiera**. 103 | - Puppet ensures **idempotency**, enforcing desired states on nodes. 104 | - Understanding Puppet helps in answering **interview questions** on configuration management tools. 105 | 106 | ### **Comparison with Ansible** 107 | | **Puppet Concept** | **Equivalent in Ansible** | 108 | |--------------------|--------------------| 109 | | Manifest | Playbook | 110 | | Resource | Task | 111 | | Class | Module | 112 | | Module | Role | 113 | | Facter | Facts (`ansible_facts`) | 114 | | Hiera | Variables, Inventory, Vault | 115 | 116 | Understanding **Puppet vs. Ansible** helps in learning **configuration management principles** applicable across tools. 117 | -------------------------------------------------------------------------------- /DevOps Tools 2 Module/Jenkins Miscellaneous: -------------------------------------------------------------------------------- 1 | # Jenkins Miscellaneous 2 | 3 | - **Jenkins Administrator Tasks** 4 | - Backup strategies and plugins 5 | - Managing Jenkins security and user access 6 | - Understanding and monitoring Jenkins logs 7 | - **Groovy Syntax** 8 | - Basic syntax, variables, and data structures 9 | - Loops, conditions, and switch-case 10 | - Error handling in Groovy 11 | - Writing Groovy scripts in Jenkins pipelines 12 | 13 | ## Jenkins Administrator Tasks 14 | 15 | ### Jenkins Backup 16 | 17 | #### Why is Backup Important? 18 | 19 | Maintaining a backup of Jenkins data is crucial for: 20 | 21 | - **Disaster Recovery**: In case of a system failure, a backup allows restoration of Jenkins configurations and jobs. 22 | - **Old Configuration Recovery**: If unwanted changes are made to the Jenkins setup, backups help restore previous configurations. 23 | 24 | #### Jenkins Backup Plugins 25 | 26 | Jenkins provides multiple plugins to manage backups, with **ThinBackup** being one of the most popular choices. 27 | 28 | ### Managing Jenkins Security 29 | 30 | #### User Management 31 | 32 | - Jenkins allows administrators to manage users and their access levels. 33 | - Instead of adding hundreds of users manually, Jenkins provides an option for users to **self-register**. 34 | - This is configured by selecting: 35 | - **"Jenkins' own user database"** 36 | - Enabling **"Allow users to sign up"** 37 | 38 | #### Permissions and Roles 39 | 40 | - Jenkins administrators can assign specific **permissions** to users based on their roles. 41 | - Users can have **read-only access**, **job execution privileges**, or **full administrative control**. 42 | - Jenkins differentiates between: 43 | - **Anonymous users**: Anyone who accesses Jenkins without authentication. 44 | - **Authenticated users**: Users with valid credentials. 45 | 46 | --- 47 | 48 | ### Jenkins Logs 49 | 50 | #### Why Are Logs Important? 51 | 52 | - Logs help track **user activities** and identify **system changes** in Jenkins. 53 | - Logs assist in debugging issues by pinpointing the **source of errors**. 54 | 55 | #### Monitoring System Performance 56 | 57 | - Jenkins provides **Load Statistics** to track system performance and resource usage. 58 | - Jenkins **metrics** offer a detailed view of the system, but how do we access them? 59 | 60 | #### Prometheus Metrics for Jenkins 61 | 62 | - Jenkins can be integrated with **Prometheus** for **real-time monitoring**. 63 | - Steps to enable Prometheus in Jenkins: 64 | 1. Install the **Prometheus Metrics Plugin** from Jenkins plugin manager. 65 | 2. Configure Prometheus settings in Jenkins. 66 | 3. Restart Jenkins and access metrics at: 67 | ``` 68 | http://your-jenkins-url/prometheus 69 | ``` 70 | 4. Prometheus provides insights into: 71 | - **JVM performance** 72 | - **System metrics** (CPU, memory usage) 73 | - **Job execution times** 74 | - **Queued builds** 75 | - **Node availability** 76 | 77 | #### Role of a Jenkins Administrator 78 | 79 | If asked in an interview, a **Jenkins Administrator** has responsibilities such as: 80 | 81 | - Managing **backups** to protect configurations using plugins like **ThinBackup**. 82 | - Controlling **user access**, defining permissions, and ensuring security compliance. 83 | - Monitoring **system logs** to track changes and ensure accountability. 84 | - Using **Prometheus** to analyze Jenkins **performance metrics** and optimize resource usage. 85 | 86 | --- 87 | 88 | ## Groovy Syntax 89 | 90 | ### Introduction 91 | 92 | - **Groovy** is a scripting language used in Jenkins pipelines. 93 | - It is useful for writing **declarative and scripted pipelines**, enabling automation of CI/CD processes. 94 | 95 | --- 96 | 97 | ### Installing and Configuring SonarQube in Jenkins 98 | 99 | - **SonarQube** is a tool that analyzes **code quality** and identifies potential bugs. 100 | - To integrate SonarQube with Jenkins: 101 | 1. Install the **SonarQube Scanner Plugin** from Jenkins plugin manager. 102 | 2. Configure **SonarQube server details** in Jenkins global settings. 103 | 3. Link SonarQube to your pipeline to **analyze code quality** after each build. 104 | 105 | --- 106 | 107 | ### Groovy Basics 108 | 109 | #### Variables and Data Types 110 | 111 | Groovy supports dynamic typing. Variables can store different data types such as: 112 | 113 | ```groovy 114 | def name = "ABC" 115 | def version = 2.0 116 | ``` 117 | 118 | #### Lists and Maps (Dictionaries) 119 | 120 | Groovy provides **lists** and **maps** for structured data storage: 121 | 122 | ```groovy 123 | def students = ["Sovan", "Tejas", "Vikas"] 124 | def users = [name: "John", age: 38] 125 | ``` 126 | 127 | --- 128 | 129 | ### Loops and Conditions 130 | 131 | #### Conditional Statements 132 | 133 | ```groovy 134 | if (version > 1.0) { 135 | println("Version is greater than 1.0") 136 | } else { 137 | println("Version is 1.0 or lower") 138 | } 139 | ``` 140 | 141 | #### Looping Constructs 142 | 143 | **For Loop:** 144 | 145 | ```groovy 146 | for (int i = 0; i < 5; i++) { 147 | println(i) 148 | } 149 | ``` 150 | 151 | **While Loop:** 152 | 153 | ```groovy 154 | def i = 0 155 | while (i < 5) { 156 | println(i) 157 | i++ 158 | } 159 | ``` 160 | 161 | --- 162 | 163 | ### Switch Case in Groovy 164 | 165 | ```groovy 166 | switch(user) { 167 | case 'dev': 168 | println("Development Environment") 169 | break 170 | case 'prod': 171 | println("Production Environment") 172 | break 173 | default: 174 | println("Unknown Environment") 175 | } 176 | ``` 177 | 178 | --- 179 | 180 | ### Error Handling in Groovy 181 | 182 | Error handling ensures that unexpected failures do not break the pipeline. 183 | 184 | - Consider a scenario with three stages: **Build, Test, Deploy**. 185 | - If the **Test stage fails**, the deployment should be **skipped**. 186 | 187 | #### Try-Catch for Exception Handling 188 | 189 | ```groovy 190 | try { 191 | sh 'some-shell-command' 192 | } catch (Exception e) { 193 | echo "Error: ${e.getMessage()}" 194 | } 195 | ``` 196 | 197 | --- 198 | 199 | ### Example Jenkins Pipeline Using Groovy 200 | 201 | Below is a **Jenkins Declarative Pipeline** integrating Groovy concepts: 202 | 203 | ```groovy 204 | pipeline { 205 | agent any 206 | 207 | stages { 208 | stage('Build') { 209 | steps { 210 | echo "Building the application..." 211 | } 212 | } 213 | 214 | stage('Test') { 215 | steps { 216 | try { 217 | echo "Running tests..." 218 | sh 'exit 1' // Simulating a test failure 219 | } catch (Exception e) { 220 | echo "Test failed: ${e.getMessage()}" 221 | error("Stopping pipeline due to test failure") 222 | } 223 | } 224 | } 225 | 226 | stage('Deploy') { 227 | steps { 228 | echo "Deploying the application..." 229 | } 230 | } 231 | } 232 | } 233 | ``` 234 | 235 | - If the **Test stage fails**, the **Deploy stage will not execute**. 236 | - The **Try-Catch block** captures errors and logs them. 237 | 238 | --- 239 | 240 | ## Conclusion 241 | 242 | ### Key Takeaways 243 | 244 | - **Jenkins Administrator Tasks**: 245 | - Backup strategies using **ThinBackup**. 246 | - Managing users and security roles. 247 | - Monitoring logs and system performance using **Prometheus Metrics**. 248 | 249 | - **Groovy Scripting for Jenkins Pipelines**: 250 | - Understanding **basic syntax**, variables, and data structures. 251 | - Using **loops, conditions, and switch-case statements**. 252 | - Implementing **error handling** to manage pipeline failures. 253 | - Writing a **complete Jenkins pipeline using Groovy**. 254 | 255 | These topics are essential for mastering **Jenkins administration and pipeline scripting**. 🚀 256 | -------------------------------------------------------------------------------- /DevOps Tools 2 Module/Templates, Variables & Facts.md: -------------------------------------------------------------------------------- 1 | # Templates, Variables & Facts 2 | 3 | ## Variables 4 | 5 | ### Variables in Ansible 6 | 7 | - Variables in Ansible store values that can be used in playbooks and roles. 8 | - They enable **dynamic configuration**, making playbooks reusable and flexible. 9 | - Variable values can be defined at multiple levels, allowing for parameterization. 10 | 11 | ### Types of Variables in Ansible 12 | 13 | #### **1. Host Variables** 14 | - Host-specific variables apply to individual machines. 15 | - Typically defined in: 16 | - Inventory files (`inventory.ini` or YAML-based inventory) 17 | - `host_vars/` directory with per-host variable files. 18 | - Allow customization of host configurations dynamically. 19 | 20 | #### **2. Group Variables** 21 | - Define common settings for a group of hosts. 22 | - Useful for managing configurations across multiple servers. 23 | - Typically stored in: 24 | - Inventory files under the `[group_name]` section. 25 | - `group_vars/` directory with YAML files for each group. 26 | 27 | --- 28 | 29 | ## Inventory 30 | 31 | ### Introduction to Ansible Inventory 32 | 33 | - An inventory file lists **managed nodes (hosts)** that Ansible controls. 34 | - The default static inventory file is **`/etc/ansible/hosts`**. 35 | - Dynamic inventory scripts (`inventory.ini` or external sources like AWS) can be used. 36 | 37 | ### YAML-Based Inventory 38 | 39 | - An alternative to INI-based inventories is using a YAML-based format. 40 | - YAML format improves readability and enables structured data representation. 41 | 42 | #### **Example of a YAML-Based Inventory** 43 | ```yaml 44 | all: 45 | hosts: 46 | webserver: 47 | ansible_host: 192.168.1.10 48 | database: 49 | ansible_host: 192.168.1.20 50 | children: 51 | web: 52 | hosts: 53 | webserver: 54 | db: 55 | hosts: 56 | database: 57 | ``` 58 | 59 | ### Running Ansible Playbooks with YAML Inventories 60 | ```bash 61 | ansible-playbook -i inventory.yaml playbook.yml 62 | ``` 63 | 64 | ### Variable Precedence in Ansible 65 | 66 | - When multiple variables are defined for the same entity, **precedence** determines which value is applied. 67 | 68 | #### **Hierarchy of Variable Precedence (Lowest to Highest)** 69 | 1. **Default variables** (inside roles) 70 | 2. **Inventory variables** (defined in `inventory.ini` or YAML-based inventory) 71 | 3. **Group variables** (`group_vars/`) 72 | 4. **Host variables** (`host_vars/`) 73 | 5. **Playbook variables** (directly assigned within a playbook) 74 | 6. **Registered variables** (created during playbook execution) 75 | 7. **Extra variables** (passed via CLI using `-e` flag) 76 | 77 | Example of passing extra variables: 78 | ```bash 79 | ansible-playbook playbook.yml -e "var_name=value" 80 | ``` 81 | 82 | --- 83 | 84 | ## Facts 85 | 86 | ### Understanding Facts in Ansible 87 | 88 | - **Facts** are system properties automatically gathered by Ansible from remote hosts. 89 | - Examples of gathered information: 90 | - Network configuration 91 | - Operating system details 92 | - Hardware specifications 93 | - Facts allow conditional execution of tasks based on host characteristics. 94 | 95 | #### **Example of Using Facts in a Playbook** 96 | ```yaml 97 | - name: Install web server based on OS 98 | hosts: all 99 | tasks: 100 | - name: Install httpd on RHEL-based systems 101 | yum: 102 | name: httpd 103 | state: present 104 | when: ansible_os_family == "RedHat" 105 | 106 | - name: Install apache2 on Debian-based systems 107 | apt: 108 | name: apache2 109 | state: present 110 | when: ansible_os_family == "Debian" 111 | ``` 112 | 113 | ### Disabling Fact Gathering 114 | 115 | #### Why Disable Fact Gathering? 116 | - **Optimizes playbook execution time** by skipping fact collection. 117 | - **Reduces system load** on remote machines. 118 | - Suitable when facts are not required for task execution. 119 | 120 | #### **How to Disable Fact Gathering** 121 | - Use `gather_facts: no` in the playbook: 122 | ```yaml 123 | - name: Playbook without facts gathering 124 | hosts: all 125 | gather_facts: no 126 | tasks: 127 | - name: Execute command without facts 128 | command: echo "Hello, Ansible!" 129 | ``` 130 | 131 | --- 132 | 133 | ## Defining Custom Facts 134 | 135 | ### Creating Custom Facts 136 | 137 | - Custom facts allow administrators to define **additional system properties**. 138 | - Can be stored as: 139 | - **Executable scripts** (e.g., Bash or Python) 140 | - **Static JSON files** inside `/etc/ansible/facts.d/` 141 | 142 | #### **Example: Defining a Custom Fact in JSON Format** 143 | 1. Create a custom facts file: 144 | ```bash 145 | mkdir -p /etc/ansible/facts.d/ 146 | echo '{"app_version": "2.0"}' > /etc/ansible/facts.d/custom_fact.json 147 | ``` 148 | 2. Retrieve the fact using Ansible: 149 | ```yaml 150 | - name: Display custom fact 151 | hosts: all 152 | tasks: 153 | - name: Fetch app version 154 | debug: 155 | msg: "Application Version: {{ ansible_local.custom_fact.app_version }}" 156 | ``` 157 | 158 | ### **Use Case: Ensuring Correct Software Version** 159 | - If a new **V2 application** requires **Nginx 1.2**, the playbook can: 160 | - Check which servers run **V2**. 161 | - Install or update Nginx accordingly. 162 | 163 | ### **Alternative: Registering a Custom Fact Dynamically** 164 | Instead of static facts, you can **register a variable** during playbook execution: 165 | 166 | ```yaml 167 | - name: Gather custom facts dynamically 168 | hosts: all 169 | tasks: 170 | - name: Register application version 171 | shell: echo "2.0" 172 | register: app_version 173 | 174 | - name: Print application version 175 | debug: 176 | msg: "App Version: {{ app_version.stdout }}" 177 | ``` 178 | 179 | --- 180 | 181 | ## Jinja2 Templating 182 | 183 | ### Introduction to Jinja2 Templates 184 | 185 | - **Jinja2** is a templating engine used in Ansible for dynamic content generation. 186 | - It allows embedding **variables, loops, and conditions** inside configuration files. 187 | 188 | #### **Basic Syntax** 189 | ```jinja2 190 | {{ variable_name }} # Prints the value of a variable 191 | {% if condition %} # Conditional statements 192 | {% for item in list %} # Looping constructs 193 | ``` 194 | 195 | #### **Example: Using Jinja2 in an Ansible Template** 196 | 1. Create a Jinja2 template (`nginx.conf.j2`): 197 | ```jinja2 198 | server { 199 | listen 80; 200 | server_name {{ server_name }}; 201 | location / { 202 | root {{ document_root }}; 203 | } 204 | } 205 | ``` 206 | 2. Apply the template in a playbook: 207 | ```yaml 208 | - name: Deploy Nginx configuration 209 | hosts: web 210 | tasks: 211 | - name: Render nginx.conf from template 212 | template: 213 | src: nginx.conf.j2 214 | dest: /etc/nginx/nginx.conf 215 | ``` 216 | 217 | --- 218 | -------------------------------------------------------------------------------- /Linux Module/Advanced _Shell _Scripting_ Techniques.md: -------------------------------------------------------------------------------- 1 | 2 | # Advanced Shell Scripting Techniques 3 | 4 | 5 | ## What is the difference between Arrays and Lists? 6 | 7 | ### Arrays: 8 | - An array is a data structure that allows you to store multiple values in a single variable. Each value in an array is identified by a unique index, starting from `0`. This indexed structure makes it easier to manage and retrieve data directly without needing to iterate through the entire collection. 9 | 10 | - **Accessing elements:** 11 | Individual elements in an array can be accessed using the index associated with them. The syntax for accessing an element is as follows: 12 | ```bash 13 | echo ${Array_Name[Index]} 14 | ``` 15 | Here, `Array_Name` is the name of the array, and `Index` refers to the position of the element in the array. 16 | 17 | - **Printing all elements:** 18 | If you want to display all the elements stored in an array, use the following command: 19 | ```bash 20 | echo ${Array_Name[@]} 21 | ``` 22 | This will output all the values in the array in the order they were stored. 23 | 24 | - **Finding the size of an array:** 25 | You can determine the number of elements in an array using the `#` symbol before the array name: 26 | ```bash 27 | echo ${#Array_Name[@]} 28 | ``` 29 | This command will return the total number of elements in the array. 30 | 31 | - **Adding elements to an array:** 32 | To add a new element to an existing array at a specific position (index), you can use the following syntax: 33 | ```bash 34 | Array_Name[Index]=new_element 35 | ``` 36 | This will store the `new_element` at the position specified by `Index`. 37 | 38 | ### Lists: 39 | - A list is another type of collection, but unlike arrays, a list does not have a built-in indexing system. Each element is not identified by a unique index but is instead stored sequentially. 40 | 41 | - **Accessing data in lists:** 42 | To retrieve or work with data in a list, you must traverse the list using a loop. This is because, without an index, the system needs to iterate over all elements to find or manipulate specific data. 43 | 44 | - **Non-continuous memory allocation:** 45 | Lists do not store data in continuous memory blocks. This means that the elements in a list can be scattered across different memory locations, making random access more complicated compared to arrays. 46 | 47 | ### Associative Arrays: 48 | - Associative arrays are a more advanced form of array that stores data in **key-value pairs**. Each element is identified by a key rather than an index. This allows you to map relationships between two different sets of data, which is especially useful for creating lookup tables or dictionaries. 49 | 50 | **Syntax to access a value using a key:** 51 | ```bash 52 | echo ${Array_Name[Key]} 53 | ``` 54 | Here, `Key` is used to access the value it is associated with. 55 | 56 | **Example of an associative array:** 57 | ![Associative Array Example](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/089/945/original/Screenshot_2024-09-17_223319.png?1726592614) 58 | 59 | --- 60 | 61 | ## What is Process Control? 62 | 63 | Process control is the ability to manage the lifecycle of processes—instances of running programs—on a system. It involves starting, stopping, and controlling various aspects of a process to ensure efficient and controlled execution. 64 | 65 | ### Background Jobs: 66 | - Background jobs are processes that run independently of the terminal. They do not block the terminal, allowing you to continue interacting with the system while the background job is executing. 67 | 68 | - **Running jobs in the background:** 69 | A common example of background jobs is using the `sleep` command to delay execution for a set amount of time. To push a job to the background, use `&` at the end of the command: 70 | ```bash 71 | sleep 60 & 72 | ``` 73 | This runs the command in the background for 60 seconds without interfering with other tasks. 74 | 75 | 76 | 77 | **Example:** 78 | ![Background Jobs Example](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/089/954/original/Screenshot_2024-09-18_081548.png?1726627601) 79 | 80 | ### Signals: 81 | - Signals are predefined messages that are sent to processes to trigger specific actions or manage their execution. Signals are fundamental to process control as they allow for communication between the operating system and running processes. 82 | 83 | **Common Signals and Their Usage:** 84 | - **SIGINT (Signal Interrupt):** 85 | This signal is used to interrupt a process, often through user input like `Ctrl + C`. It gracefully stops the process. 86 | 87 | - **SIGTERM (Signal Termination):** 88 | This signal requests the termination of a process, allowing it to clean up resources before exiting. 89 | 90 | - **SIGKILL (Kill Signal):** 91 | Unlike `SIGTERM`, this signal forcibly terminates a process without giving it the opportunity to clean up. 92 | 93 | - **SIGHUP (Hang Up Signal):** 94 | This signal reloads the configuration of a terminal or daemon without fully restarting it. It's used to reapply settings without stopping services. 95 | 96 | - **SIGSTOP:** 97 | This signal temporarily halts the process, which can later be resumed. 98 | 99 | **Example to Terminate a Process:** 100 | ```bash 101 | kill -SIGTERM 102 | ``` 103 | 104 | **Note for Instructor:** 105 | Demonstrate how to list process IDs, and use signals to manage them effectively in a terminal session. 106 | 107 | ### Trap: 108 | - The `trap` command is used to detect and respond to signals. It allows the script to handle interruptions gracefully by executing specific commands when a signal is received. 109 | 110 | **Real-World Analogy:** 111 | Think of `trap` as a contingency plan during a party. If you plan to clean up after the party but interruptions (like 10-minute breaks) happen, `trap` commands ensure that small cleaning tasks are done during breaks to save time later. Similarly, when a script receives a signal, `trap` ensures certain tasks are performed before exiting. 112 | 113 | - The script can store these interruptions in trap files, and upon receiving signals, the trap deletes unnecessary files and restarts necessary parts of the script. 114 | 115 | 116 | ### Cron Jobs: 117 | - Cron jobs are scheduled tasks in Unix-like operating systems that allow commands or scripts to run at specific times or intervals. They are useful for automating system maintenance, backups, and other repetitive tasks. 118 | 119 | **Cron Syntax:** 120 | ![Cron Job Syntax](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/089/958/original/Screenshot_2024-09-18_101419.png?1726634668) 121 | 122 | **Example of a Cron Job:** 123 | ![Cron Job Example](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/089/959/original/Screenshot_2024-09-18_101559.png?1726634769) 124 | 125 | **Note for Instructor:** 126 | Provide examples of more complex cron jobs, such as running commands every 3rd or 4th day. Explain how to schedule tasks for different intervals for better understanding. 127 | 128 | ### Set: 129 | - The `set` command is an essential debugging tool. It enables shell options that make script debugging easier by displaying each command before it is executed. This helps in identifying errors and understanding the flow of the script. 130 | 131 | ### Process Substitution: 132 | - Process substitution is a technique that allows the output of one command to be used as input for another command. This is useful when you want to pass data from one process directly to another without creating intermediate files. 133 | 134 | **Example:** 135 | ![Process Substitution Example](https://d2beiqkhq929f0.cloudfront.net/public_assets/assets/000/089/962/original/Screenshot_2024-09-18_103620.png?1726636004) 136 | 137 | In this example, process substitution is used to compare the differences between files in the `scripts` directory and the `advance` directory. 138 | 139 | -------------------------------------------------------------------------------- /Linux Module/Introduction _to _Operating _Systems _(Continued).md: -------------------------------------------------------------------------------- 1 | 2 | # Introduction to Operating Systems (Continued) 3 | 4 | ## Agenda of the Lecture 5 | ### Overview 6 | Today’s session will cover the following topics: 7 | - A **small quiz** to review previous concepts. 8 | - Introduction to **more Linux commands**. 9 | - Understanding **file permissions** in Linux. 10 | - Learning how to use the **Vim editor**. 11 | 12 | --- 13 | 14 | ## Quizzes 15 | 16 | ### Quiz 1: Linux Filesystem Hierarchy 17 | **Question:** Which of the following directories is NOT part of the Linux filesystem hierarchy? 18 | - /bin 19 | - **/exe** 20 | - /var 21 | - /etc 22 | 23 | **Answer Explanation:** The `/exe` directory does not exist in the Linux filesystem hierarchy. Common directories include `/bin`, which stores essential binaries, `/var` for variable data files, and `/etc`, which contains system configuration files. 24 | 25 | ### Quiz 2: Viewing Directory Contents 26 | **Question:** Which command is used to view the contents of a directory in Linux? 27 | - view 28 | - **ls** 29 | - open 30 | - dir 31 | 32 | **Answer Explanation:** The `ls` command is used in Linux to list the files and directories within a directory. The other options are either incorrect or pertain to different operating systems or applications. 33 | 34 | ### Quiz 3: Functions of the Operating System 35 | **Question:** Which of the following is NOT a function of the Operating System? 36 | - Memory Management 37 | - **Virus Scanning** 38 | - Process Management 39 | - Device Management 40 | 41 | **Answer Explanation:** Virus scanning is typically handled by specialized antivirus software, not by the operating system itself. OS functions include memory management, process management, and device management. 42 | 43 | --- 44 | 45 | # More Commands 46 | 47 | ## Revising Previous Commands 48 | Before learning new commands, let's quickly revise the commands from the last session: 49 | ```bash 50 | ls 51 | pwd 52 | cd /bin 53 | cd .. 54 | mkdir 55 | mkdir -p test/test2/ 56 | ``` 57 | ### Useful Tips: 58 | - **Tab Auto-Completion**: Pressing the Tab key can auto-complete commands or filenames, saving time. 59 | - **Absolute vs. Relative Path**: 60 | - **Absolute Path**: Specifies the complete path from the root directory to the desired file (e.g., `cd /bin`). 61 | - **Relative Path**: Specifies the path relative to the current directory (e.g., `cd bin` when you are in the parent directory). 62 | 63 | ## Learning More Commands 64 | 65 | ### Deleting Directories 66 | - **Remove an Empty Directory**: 67 | ```bash 68 | rmdir test 69 | ``` 70 | - **Recursively and Forcefully Delete a Directory**: 71 | ```bash 72 | rm -rf test 73 | ``` 74 | - **Caution**: This command permanently deletes the directory and all its contents without confirmation. 75 | 76 | ### Copying Files and Directories 77 | - **Copy a File**: 78 | ```bash 79 | cp test.sh /tmp/ 80 | ``` 81 | - Copies `test.sh` to the `/tmp/` directory. 82 | - **Copy a Directory Recursively**: 83 | ```bash 84 | cp -r scalar /tmp/ 85 | ``` 86 | - Copies the `scalar` directory and all its subdirectories to `/tmp/`. 87 | 88 | ### Moving and Renaming Files/Directories 89 | - **Move a Directory**: 90 | ```bash 91 | mv scalar /tmp/ 92 | ``` 93 | - Moves the `scalar` directory to `/tmp/`. 94 | - **Rename a File**: 95 | ```bash 96 | mv test.sh abc.sh 97 | ``` 98 | - Renames `test.sh` to `abc.sh`. 99 | 100 | ### Creating Files 101 | - **Create an Empty File**: 102 | ```bash 103 | touch filename.txt 104 | ``` 105 | - **Create a File with Content**: 106 | ```bash 107 | echo "text" > filename.txt 108 | ``` 109 | - Creates `filename.txt` containing the text "text". 110 | - **Append Content to a File**: 111 | ```bash 112 | echo "additional text" >> filename.txt 113 | ``` 114 | - Adds "additional text" to `filename.txt` without removing existing content. 115 | 116 | ### Reading File Content 117 | - **Display File Content**: 118 | ```bash 119 | cat filename.txt 120 | ``` 121 | - **Show First Two Lines of a File**: 122 | ```bash 123 | head -n 2 filename.txt 124 | ``` 125 | - **Show Last Two Lines of a File**: 126 | ```bash 127 | tail -n 2 filename.txt 128 | ``` 129 | - **Monitor a File in Real-Time**: 130 | ```bash 131 | tail -f filename.txt 132 | ``` 133 | - Continuously monitors `filename.txt` for updates. 134 | 135 | ### Checking Disk and Memory Usage 136 | - **Check Disk Space Usage**: 137 | ```bash 138 | df 139 | df -h 140 | ``` 141 | - `df` displays disk usage in bytes, while `df -h` converts this to a human-readable format (e.g., MB, GB). 142 | - **Check Directory Size**: 143 | ```bash 144 | du -sh 145 | ``` 146 | - Displays the size of the current directory and its subdirectories. 147 | - **Check RAM Usage**: 148 | ```bash 149 | free 150 | free -h 151 | ``` 152 | - `free` shows RAM usage in bytes, while `free -h` gives a human-readable output. This data is gathered from the `/proc/meminfo` file, which provides detailed memory information. 153 | 154 | --- 155 | 156 | # Vim Editor 157 | 158 | ## Introduction to Vim 159 | Vim is a powerful text editor commonly used in Linux. It offers multiple modes and commands for efficient text editing. 160 | 161 | ### Opening Vim 162 | - **Open Vim**: 163 | ```bash 164 | vim 165 | ``` 166 | - **Open a Specific File**: 167 | ```bash 168 | vim filename 169 | ``` 170 | 171 | ### Modes in Vim 172 | Vim operates in several modes: 173 | 1. **Normal Mode**: The default mode for navigation and command execution. Press `Esc` to return to Normal Mode from any other mode. 174 | 2. **Insert Mode**: Used for text input. Enter Insert Mode by pressing any of these keys: `i`, `I`, `a`, `A`, `o`, `O`. 175 | 3. **Visual Mode**: Used for text selection. Enter Visual Mode by pressing `v`, `V`, or `Ctrl-v`. 176 | 4. **Command-Line Mode**: Used to execute commands like saving or searching. Enter this mode by pressing `:`. 177 | 178 | ### Navigating in Vim 179 | - **Mouse Navigation**: 180 | - On Windows: Use *Shift + Left Click*. 181 | - On Mac: Use *Option + Left Click*. 182 | - **Keyboard Navigation**: 183 | - `h`: Move left. 184 | - `j`: Move down. 185 | - `k`: Move up. 186 | - `l`: Move right. 187 | - `0`: Move to the beginning of the line. 188 | - `$`: Move to the end of the line. 189 | - `gg`: Go to the beginning of the file. 190 | - `G`: Go to the end of the file. 191 | - `:n`: Go to line `n`. 192 | 193 | ### Saving and Quitting 194 | - **Save and Quit**: 195 | ```bash 196 | :wq 197 | ``` 198 | - **Quit Without Saving**: 199 | ```bash 200 | :q! 201 | ``` 202 | - **Save File**: 203 | ```bash 204 | :w 205 | ``` 206 | - **Save with a Different Name**: 207 | ```bash 208 | :w newfilename 209 | ``` 210 | 211 | ### Selecting and Manipulating Text 212 | - **Copy Selected Text**: 213 | 1. Enter Visual Mode by pressing `v`. 214 | 2. Select the desired text. 215 | 3. Press `y` to copy. 216 | 217 | ### Deleting Text 218 | - **Delete a Line**: 219 | ```bash 220 | dd 221 | ``` 222 | - **Delete a Character**: 223 | ```bash 224 | x 225 | ``` 226 | - Deletes the character under the cursor. 227 | - **Delete from Cursor to End of Line**: 228 | ```bash 229 | d$ 230 | ``` 231 | 232 | ### Searching in Vim 233 | - **Forward Search**: 234 | ```bash 235 | /pattern 236 | ``` 237 | - **Backward Search**: 238 | ```bash 239 | ?pattern 240 | ``` 241 | - **Next Occurrence**: 242 | ```bash 243 | n 244 | ``` 245 | - **Previous Occurrence**: 246 | ```bash 247 | N 248 | ``` 249 | 250 | ### Find and Replace 251 | - **Replace Text Globally**: 252 | ```bash 253 | :%s/old/new/g 254 | ``` 255 | - Replaces every occurrence of `old` with `new` in the entire file. 256 | -------------------------------------------------------------------------------- /Linux Module/Real_World _Shell_ Scripting _Projects.md: -------------------------------------------------------------------------------- 1 | # Typed Notes on Real-World Shell Scripting Projects 2 | 3 | ## 1. Implementing the Password Script 4 | 5 | ### Objective: 6 | To implement a script that validates passwords based on the following conditions: 7 | - Password must be longer than 8 characters. 8 | - Must contain at least one uppercase letter. 9 | - Must contain at least one lowercase letter. 10 | - Must contain at least one number. 11 | 12 | ### Steps: 13 | 14 | #### Step 1: Taking Input 15 | We will first prompt the user to enter a password using the `read` command with the `-s` flag (to hide the input for privacy). 16 | 17 | ```bash 18 | read -s -p "Enter your password: " user_pass 19 | echo 20 | ``` 21 | 22 | - The `-s` flag hides the password input. 23 | - `echo` creates an empty line after input for better readability. 24 | 25 | #### Step 2: Calling the Validation Function 26 | Next, we call a function to validate the password entered by the user: 27 | 28 | ```bash 29 | validate_password() { 30 | # Function logic will be written here 31 | } 32 | 33 | read -s -p "Enter your password: " user_pass 34 | echo 35 | validate_password "$user_pass" 36 | ``` 37 | 38 | Here, `validate_password` takes the `user_pass` variable as an argument. 39 | 40 | #### Step 3: Defining the `validate_password` Function 41 | 42 | 1. **Local Variable for Password** 43 | Inside the function, we use a local variable `password` to capture the first argument (`$1`) passed to the function. 44 | 45 | ```bash 46 | validate_password() { 47 | local password="$1" 48 | } 49 | ``` 50 | 51 | 2. **Checking if the Password is Greater than 8 Characters** 52 | Use `${#password}` to check the length of the password. If it’s less than 8, the script returns a failure message. 53 | 54 | ```bash 55 | if [ ${#password} -lt 8 ]; then 56 | echo "Weak Password" 57 | return 1 58 | fi 59 | ``` 60 | 61 | 3. **Checking for Uppercase Letters** 62 | Use regular expressions to validate if the password contains at least one uppercase letter: 63 | 64 | ```bash 65 | if ! [[ "$password" =~ [A-Z] ]]; then 66 | echo "Weak Password" 67 | return 1 68 | fi 69 | ``` 70 | 71 | 4. **Checking for Lowercase Letters** 72 | Similarly, we check for lowercase letters: 73 | 74 | ```bash 75 | if ! [[ "$password" =~ [a-z] ]]; then 76 | echo "Weak Password" 77 | return 1 78 | fi 79 | ``` 80 | 81 | 5. **Checking for Numbers** 82 | We check if the password contains at least one numeric character: 83 | 84 | ```bash 85 | if ! [[ "$password" =~ [0-9] ]]; then 86 | echo "Weak Password" 87 | return 1 88 | fi 89 | ``` 90 | 91 | 6. **Returning Success for Strong Passwords** 92 | If all the conditions are met, the script will output a message indicating the password is strong. 93 | 94 | ```bash 95 | echo "Strong Password" 96 | ``` 97 | 98 | ### Final Script: 99 | 100 | ```bash 101 | validate_password() { 102 | local password="$1" 103 | 104 | if [ ${#password} -lt 8 ]; then 105 | echo "Weak Password" 106 | return 1 107 | fi 108 | 109 | if ! [[ "$password" =~ [A-Z] ]]; then 110 | echo "Weak Password" 111 | return 1 112 | fi 113 | 114 | if ! [[ "$password" =~ [a-z] ]]; then 115 | echo "Weak Password" 116 | return 1 117 | fi 118 | 119 | if ! [[ "$password" =~ [0-9] ]]; then 120 | echo "Weak Password" 121 | return 1 122 | fi 123 | 124 | echo "Strong Password" 125 | } 126 | 127 | read -s -p "Enter your password: " user_pass 128 | echo 129 | validate_password "$user_pass" 130 | ``` 131 | 132 | ### Explanation: 133 | - The script checks all conditions for password strength. 134 | - If any condition fails, it returns "Weak Password." 135 | - A "Strong Password" message is displayed if all checks pass. 136 | 137 | --- 138 | 139 | ## 2. Basic Aspects of Scripting 140 | 141 | ### Elements of Scripting: 142 | 1. **Clear Objective**: Before starting, understand the task and what the script needs to accomplish. 143 | 2. **Flowchart**: Plan the flow of the script with a flowchart to better visualize and structure the logic. 144 | 3. **Modular Code**: Break down your script into reusable functions. For example, the `validate_password` function can be reused in other scripts. 145 | 4. **Comments**: Always include comments to explain key parts of the script. This helps in future maintenance and readability. 146 | 147 | --- 148 | 149 | ## 3. Real-World Project: Backup Script 150 | 151 | ### Objective: 152 | To write a script that takes a backup of a user-specified directory in either `zip` or `tar.gz` format, with a timestamp added to the filename. 153 | 154 | ### Key Features of the Script: 155 | 1. **User Input**: The user specifies the directory and the compression method (`zip` or `tar`). 156 | 2. **Timestamp**: A timestamp is added to the backup filename for easy identification of when the backup was created. 157 | 3. **Error Handling**: The script checks if the directory exists and if the compression method is valid. 158 | 159 | ### Final Script: 160 | 161 | ```bash 162 | # Function to display usage information 163 | usage() { 164 | echo "Usage: $0 [-d directory] [-c compression (zip|tar)]" 165 | exit 1 166 | } 167 | 168 | # Parse options 169 | while getopts ":d:c:" opt; do 170 | case $opt in 171 | d) directory=$OPTARG ;; 172 | c) compression=$OPTARG ;; 173 | \?) usage ;; 174 | esac 175 | done 176 | 177 | # Check if directory is specified and exists 178 | if [[ -z "$directory" ]] || [[ ! -d "$directory" ]]; then 179 | echo "Error: Directory is not specified or does not exist." 180 | usage 181 | fi 182 | 183 | # Validate compression method 184 | if [[ "$compression" != "zip" ]] && [[ "$compression" != "tar" ]]; then 185 | echo "Error: Compression method must be 'zip' or 'tar'." 186 | usage 187 | fi 188 | 189 | # Generate a timestamp 190 | timestamp=$(date +'%Y-%m-%d_%H-%M-%S') 191 | 192 | # Set filename based on the chosen compression method 193 | if [[ "$compression" == "zip" ]]; then 194 | archive_name="${directory}_${timestamp}.zip" 195 | # Compress directory with zip 196 | zip -r "$archive_name" "$directory" 197 | elif [[ "$compression" == "tar" ]]; then 198 | archive_name="${directory}_${timestamp}.tar.gz" 199 | # Compress directory with tar 200 | tar -czvf "$archive_name" "$directory" 201 | fi 202 | 203 | # Check if compression was successful 204 | if [[ $? -ne 0 ]]; then 205 | echo "Error: Compression failed." 206 | exit 1 207 | fi 208 | 209 | echo "Archive created: $archive_name" 210 | echo "Backup process completed successfully." 211 | exit 0 212 | ``` 213 | 214 | ### Explanation: 215 | 1. **Usage Function**: The script begins with a `usage` function that shows how to run the script. 216 | 2. **Option Parsing**: `getopts` is used to parse command-line options. The user provides the directory with `-d` and the compression type (`zip` or `tar`) with `-c`. 217 | 3. **Directory Check**: The script checks if the directory exists using `[[ -d "$directory" ]]`. 218 | 4. **Compression Validation**: It ensures the user selects either `zip` or `tar` for the compression method. 219 | 5. **Timestamp Creation**: The `timestamp` variable holds the current date and time to add to the backup file's name. 220 | 6. **File Compression**: Depending on the chosen compression method, either `zip` or `tar` is used. 221 | 7. **Success Check**: The script checks if the compression command was successful using `$?`. 222 | 223 | ### Practice Problem: 224 | A script to automate this backup process to run every day at 11:00 AM, and store the backups in a designated directory. 225 | -------------------------------------------------------------------------------- /Linux Module/System _Resource _Monitoring.md: -------------------------------------------------------------------------------- 1 | # System Resource Monitoring 2 | --- 3 | 4 | ## Introduction to System Resource Monitoring 5 | 6 | **System resource monitoring** involves observing and tracking various system components such as CPU usage, memory consumption, network traffic, and disk I/O. This helps in identifying bottlenecks, optimizing performance, and ensuring system reliability. 7 | 8 | ### Key Discussion Points: 9 | - Display a **Grafana dashboard** for visual reference. 10 | ![Grafana Dashboard](https://hackmd.io/_uploads/S1SSqSvaR.png) 11 | 12 | - **Interactive Discussion:** 13 | - **What is system resource monitoring?** 14 | Monitoring all system resources to understand when and how monitoring should be done. A demo will be provided later in the session. 15 | 16 | - **Why is system monitoring important?** 17 | - Enables preventive measures before issues escalate. 18 | - Optimizes resource usage for efficient system operations. 19 | - Identifies bottlenecks, ensuring smooth system functioning. 20 | 21 | --- 22 | 23 | ## Tools for Monitoring 24 | 25 | System resource monitoring can be done using a variety of tools, categorized broadly into: 26 | 27 | ### 1. **Built-in Tools** 28 | - **Linux:** `top`, `free`, `ps`, `df`, `vmstat` for monitoring CPU, memory, processes, and disk usage. 29 | - **Network Monitoring:** `netstat` for network connections and `tcpdump` for traffic analysis. 30 | 31 | ### 2. **Specialized Tools** 32 | These offer in-depth system analysis, often with enhanced visualization and real-time data collection: 33 | - **Grafana**: Provides a customizable interface for system metrics. 34 | - **AppDynamics, Zabbix, Datadog**: Comprehensive tools for performance management and anomaly detection. 35 | 36 | **Key Considerations for Tool Selection:** 37 | - **System Compliance:** Ensure the tool supports your system. 38 | - **Scalability:** Can it handle growth? 39 | - **Cost:** What are the financial constraints? 40 | 41 | **Interactive Questions:** 42 | - **What should be the selection criteria for choosing a monitoring tool?** 43 | The criteria should include compliance, scalability, and cost-effectiveness. 44 | 45 | - **Do we need to learn all the tools?** 46 | No, it’s more important to grasp the fundamentals of a few key tools rather than mastering all. 47 | 48 | ### Real-Time Analysis with Grafana: 49 | - **Dashboard Analysis:** 50 | ![System Health](https://hackmd.io/_uploads/BJRonrPpA.png) 51 | - The system is healthy, consuming below-average resources. Memory and CPU utilization are low, indicating underutilization. 52 | 53 | ![Increased Load](https://hackmd.io/_uploads/SJWIpBPa0.png) 54 | - Increased system load with a decrease in idle time. 55 | 56 | - **Insight:** 57 | Built-in tools provide basic monitoring, but **specialized tools** like Grafana are needed for detailed analysis. 58 | 59 | --- 60 | 61 | ## Troubleshooting in Monitoring 62 | 63 | Troubleshooting involves identifying the root cause of system issues using monitoring data and a set of diagnostic commands. This process requires real-time data and systematic observation. 64 | 65 | ### Common Metrics for Troubleshooting: 66 | - **CPU Utilization** 67 | - Example: Initially, CPU was idle, but a job increased the load temporarily. After job completion, the system returned to idle. 68 | 69 | - **Memory Utilization** 70 | - Memory was lightly utilized initially. When a process demanded more memory, the system adapted. Upon job completion, swap memory was minimized. 71 | 72 | ### Key Commands for Troubleshooting: 73 | 1. **`top` Command:** 74 | Displays real-time resource usage, highlighting processes consuming excessive CPU or memory. 75 | ![top Command](https://hackmd.io/_uploads/BkJHYcu6A.png) 76 | 77 | 2. **Analyze the Output:** 78 | ![Output Analysis](https://hackmd.io/_uploads/HJ-x5cupC.png) 79 | High CPU and memory utilization are identified. 80 | 81 | 3. **Identify the Process:** 82 | - Use the **`ps -ef | grep `** command to find the specific process causing the issue. 83 | ![Process Identification](https://hackmd.io/_uploads/S17ic9upR.png) 84 | 85 | 4. **Steps after Identifying Process:** 86 | - Kill unnecessary processes. 87 | - For essential processes, adjust priorities to prevent system bottlenecks. 88 | 89 | --- 90 | 91 | ## Remediation Steps 92 | 93 | Once a potential issue is identified, remediation ensures the system continues to function efficiently. 94 | 95 | ### Key Remediation Considerations: 96 | - **How to inform users about overutilization?** 97 | - **Incorrect Approach:** Ignore the issue. 98 | - **Correct Approach:** Ask users how often the process runs: 99 | - If it runs for a few minutes, no action is needed. 100 | - If it runs continuously, consider increasing resources based on usage. 101 | 102 | **Dashboard Example:** 103 | ![Network Traffic Issue](https://hackmd.io/_uploads/rJcXC9O6A.png) 104 | - A possible network traffic issue is highlighted. 105 | 106 | ### Network Traffic Analysis: 107 | - **Graphical Representation:** 108 | ![Network Traffic](https://hackmd.io/_uploads/BkeFAqOpR.png) 109 | - Data transmission and reception increase at certain times, indicating user activity. 110 | 111 | **Real-World Example:** 112 | - Think of Instagram’s traffic, which increases during peak periods or special events like sales. 113 | 114 | ### Monitoring Components: 115 | - **Node Exporter:** Collects system metrics and sends them to Prometheus. 116 | - **Prometheus:** Stores data and allows queries. 117 | - **Grafana:** Visualizes the data, making it easier to interpret system performance. 118 | 119 | --- 120 | 121 | ## Importance of Alerting 122 | 123 | An essential part of system monitoring is **alerting**, which ensures that administrators are notified when issues arise. 124 | 125 | **Why is alerting important?** 126 | - It provides real-time updates when spikes in system activity occur, helping prevent crashes and downtime. 127 | 128 | **Command:** 129 | - `$ systemctl status` 130 | - Used to check the status of key services like Grafana. 131 | 132 | ### Running Applications as Services 133 | - Applications can be managed and monitored using system commands like `systemctl` to ensure networked systems perform efficiently across multiple machines. 134 | --------------------------------------------------------------------------------