├── 00-Introduction.md ├── 01-Hugging-Face.md ├── 02-FastAPI.md ├── 03-Docker.md ├── 04-Trivy.md ├── 05-AWS-ECR.md ├── 06-Kubernetes.md ├── 07-kube-score.md ├── 08-EKS.md ├── 09-kustomize.md ├── 10-github-action.md ├── README.md ├── _config.yml ├── _layouts └── default.html ├── assets └── css │ └── style.css └── img ├── EKS.jpg ├── FastAPI.jpg ├── Trivy.jpg ├── aws-ecr.jpg ├── docker.jpg ├── github_action.jpg ├── hugging-face.jpg ├── hugging_face_book.png ├── kube-score.jpg ├── kubernetes.jpg ├── kustomize.jpg └── llmops_new.jpg /00-Introduction.md: -------------------------------------------------------------------------------- 1 | # Project Overview 2 | 3 | When a user or developer pushes code or raises a Pull Request (PR) to GitHub, specifically to the `main` branch, it triggers a GitHub Action workflow. Here's a high-level outline of what happens next: 4 | 5 | ## Trigger Workflow 6 | Every push or pull request to the `main` branch initiates the GitHub Action workflow. 7 | 8 | ## Build Docker Image 9 | The workflow begins by building a Docker image from the latest code. 10 | 11 | ## Security Scan with Trivy 12 | The built Docker image is scanned using Trivy to detect any **HIGH** or **CRITICAL** vulnerabilities. 13 | 14 | ## Publish to AWS ECR 15 | If the image passes the security scan, it is then pushed to the AWS Elastic Container Registry (ECR) for storage. 16 | 17 | ## Kubernetes Manifest Validation 18 | `kube-score` validates the Kubernetes YAML files to ensure they meet predefined standards and best practices. 19 | 20 | ## Deployment with Kustomize 21 | After validation, `Kustomize` deploys the updated Kubernetes manifests to the AWS Elastic Kubernetes Service (EKS) cluster. 22 | -------------------------------------------------------------------------------- /01-Hugging-Face.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: "Day 1: Hugging Face" 4 | --- 5 | 6 | ![Hugging Face](img/hugging-face.jpg) 7 | 8 | # What is Hugging Face? 9 | 10 | Hugging Face is a leading organization in the field of Natural Language Processing (NLP), known for its open-source libraries and resources that make it easier to build and deploy state-of-the-art machine learning models. Their transformers library provides easy access to thousands of pre-trained models in a standardized format, which can be used for tasks such as text classification, text generation, named entity recognition, and question answering. 11 | 12 | ## What are Transformers? 13 | 14 | Transformers are a type of deep learning model architecture that has revolutionized NLP. They are based on self-attention mechanisms, which allow them to consider the importance of different words in a sequence relative to each other. This architecture has proven extremely effective for tasks that require understanding context, such as translation, summarization, and question answering. 15 | 16 | ## Getting Started 17 | Prerequisites 18 | * Python 3.6 or higher 19 | * transformers library by Hugging Face 20 | * torch (PyTorch) 21 | 22 | ## Installation 23 | You need to install the necessary libraries. You can do this using pip: 24 | ```bash 25 | pip install transformers torch 26 | ``` 27 | > **Note:** Installing these modules manually via pip is just for demonstration purposes. We are going to create `requirements.txt` and install these modules via GitHub Action. 28 | 29 | 30 | ## Using the Question Answering Pipeline 31 | The following code sets up a question-answering pipeline using a pre-trained model from Hugging Face's model hub. 32 | 33 | ```bash 34 | # Initialize the Hugging Face question-answering pipeline 35 | qa_pipeline = pipeline("question-answering", model="distilbert-base-uncased-distilled-squad") 36 | 37 | # Define the context and question 38 | context = """ 39 | Hugging Face is a technology company that provides open-source NLP libraries 40 | and creates tools for machine learning developers. The company is widely 41 | recognized for its transformers library, which offers pre-trained models 42 | for a variety of NLP tasks. 43 | """ 44 | 45 | question = "What does Hugging Face provide?" 46 | 47 | # Get the answer 48 | answer = qa_pipeline(question=question, context=context) 49 | 50 | # Display the result 51 | print(f"Question: {question}") 52 | print(f"Answer: {answer['answer']}") 53 | ``` 54 | 55 | > **Note:** This code is for demonstration purposes; we will use it along with FastAPI. 56 | 57 | ## Example Output 58 | When you run the above code, you should see output similar to: 59 | 60 | ```bash 61 | Question: What does Hugging Face provide? 62 | Answer: open-source NLP libraries 63 | ``` 64 | 65 | ## Explanation 66 | The code demonstrates how to use the Hugging Face transformers library to create a question-answering pipeline. It imports the pipeline function, initializes a question-answering pipeline using the "distilbert-base-uncased-distilled-squad" model, and defines a context and a question. The pipeline is then used to find the best answer to the question based on the context provided. Finally, the code prints both the question and the corresponding answer found by the model. 67 | 68 | [Next →](02-FastAPI) 69 | -------------------------------------------------------------------------------- /02-FastAPI.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: "Day 2: FastAPI" 4 | --- 5 | 6 | ![FastAPI](img/FastAPI.jpg) 7 | 8 | # What is FastAPI? 9 | 10 | FastAPI is a modern, fast (high-performance) web framework for building APIs with Python 3.6+ based on standard Python type hints. It is designed to be easy to use and to provide a good developer experience, while also being one of the fastest frameworks available. FastAPI automatically generates interactive API documentation and is built on top of Starlette for the web parts and Pydantic for the data parts. 11 | 12 | ## Getting Started 13 | Prerequisites 14 | * Python 3.6 or higher 15 | * fastapi library 16 | * uvicorn for running the server 17 | * transformers library by Hugging Face 18 | * pydantic for data validation 19 | 20 | ## Installation 21 | You need to install the necessary libraries. You can do this using pip: 22 | ```bash 23 | pip install fastapi uvicorn transformers pydantic 24 | ``` 25 | > **Note:** Installing these modules manually via pip is just for demonstration purposes. We are going to create `requirements.txt` and install these modules via GitHub Action. 26 | 27 | 28 | ```bash 29 | # Imports the pipeline function from the Hugging Face transformers library. 30 | 31 | from fastapi import FastAPI, HTTPException 32 | from pydantic import BaseModel 33 | from transformers import pipeline 34 | import uvicorn 35 | ``` 36 | 37 | ## Importing Necessary Libraries: 38 | 39 | * FastAPI and HTTPException from fastapi to build the API and handle exceptions. 40 | * BaseModel from pydantic to define request and response data models. 41 | * pipeline from transformers to initialize the Hugging Face question-answering model. 42 | * uvicorn to run the FastAPI server. 43 | 44 | ```bash 45 | app = FastAPI() 46 | ``` 47 | 48 | ## Creating the FastAPI Application: 49 | app = FastAPI() initializes the FastAPI application. 50 | 51 | 52 | # Initialize the Hugging Face question-answering pipeline 53 | qa_pipeline = pipeline("question-answering", model="distilbert-base-uncased-distilled-squad") 54 | 55 | ```bash 56 | # Initialize the Hugging Face question-answering pipeline 57 | qa_pipeline = pipeline("question-answering", model="distilbert-base-uncased-distilled-squad") 58 | ``` 59 | 60 | 61 | # Initializing the Question-Answering Pipeline: 62 | 63 | The qa_pipeline is created using the "distilbert-base-uncased-distilled-squad" model from Hugging Face's model hub. This pipeline will handle the question-answering task. 64 | 65 | > **Note:** For a more detailed explanation on this topic, please check the [Hugging Face](https://github.com/100daysofdevops/end-to-end-llmops-pipeline/blob/main/01-Hugging-Face.md) 66 | 67 | 68 | ```bash 69 | class ChatRequest(BaseModel): 70 | question: str 71 | context: str 72 | 73 | class ChatResponse(BaseModel): 74 | answer: str 75 | ``` 76 | 77 | ## Defining Data Models: 78 | 79 | * ChatRequest and ChatResponse are Pydantic models that define the structure of the request and response data, respectively. 80 | * ChatRequest expects two fields: question (the question to be answered) and context (the context in which to search for the answer). 81 | * ChatResponse contains a single field: answer, which holds the answer found by the model. 82 | 83 | 84 | ```bash 85 | @app.post("/chat", response_model=ChatResponse) 86 | async def chat(request: ChatRequest): 87 | try: 88 | result = qa_pipeline(question=request.question, context=request.context) 89 | return ChatResponse(answer=result['answer']) 90 | except Exception as e: 91 | raise HTTPException(status_code=500, detail=str(e)) 92 | ``` 93 | 94 | ## Creating the /chat Endpoint: 95 | 96 | * The @app.post("/chat") decorator defines an endpoint that accepts POST requests at /chat. 97 | * The chat function takes a ChatRequest object as input and uses the qa_pipeline to find the answer based on the question and context. 98 | * If the process is successful, the function returns a ChatResponse object containing the answer. 99 | * If an error occurs, an HTTP 500 error is raised with the exception message. 100 | 101 | 102 | ```bash 103 | if __name__ == "__main__": 104 | uvicorn.run(app, host="0.0.0.0", port=8000) 105 | ``` 106 | 107 | ## Running the Server: 108 | 109 | * The if __name__ == "__main__": block ensures that the FastAPI app is only run when the script is executed directly. It starts the server using uvicorn.run() on `host="0.0.0.0"` and `port=8000` 110 | 111 | [← Previous](01-Hugging-Face) | [Next →](03-Docker) 112 | -------------------------------------------------------------------------------- /03-Docker.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: "Day 3: Docker" 4 | --- 5 | 6 | ![Docker](img/docker.jpg) 7 | # What is Docker? 8 | 9 | Docker is a platform that allows developers to automate the deployment of applications inside lightweight, portable containers. These containers include everything needed to run an application, including the code, runtime, libraries, and system tools. By using Docker, you can ensure that your application behaves the same, regardless of where it is deployed. 10 | 11 | ## What is a Docker Image? 12 | 13 | A Docker image is a lightweight, standalone, executable package that includes everything needed to run a piece of software. It contains the application code, runtime, libraries, environment variables, and configuration files. Images are built from a Dockerfile and can be stored in Docker Hub or other container registries. 14 | 15 | ## What is a Dockerfile? 16 | 17 | A Dockerfile is a script composed of various instructions (commands) that are executed sequentially to build a Docker image. It defines what goes into your Docker image, such as the base image to use, environment variables, files to include, and commands to run. 18 | 19 | ## Creating the Docker Image 20 | 21 | ### Prerequisites 22 | 23 | - Docker installed on your local machine 24 | - A `main.py` file created in Part 2, which is the FastAPI application we developed earlier 25 | - A `requirements.txt` file listing the dependencies 26 | 27 | ### Dockerfile 28 | 29 | Here's the Dockerfile we'll use to build the Docker image for our FastAPI application: 30 | 31 | ```Dockerfile 32 | # Use the official Python image from the Docker Hub 33 | FROM python:3.10-slim 34 | 35 | # Set the working directory in the container 36 | WORKDIR /app 37 | 38 | # Copy only the requirements file to leverage Docker cache 39 | COPY requirements.txt . 40 | 41 | # Install the dependencies 42 | RUN pip install --no-cache-dir -r requirements.txt 43 | 44 | # Copy the rest of the application code to the working directory 45 | COPY . . 46 | 47 | # Expose port 8000 48 | EXPOSE 8000 49 | 50 | # Use a minimal entrypoint and CMD 51 | ENTRYPOINT ["python"] 52 | CMD ["main.py"] 53 | ``` 54 | 55 | ## Dockerfile Explanation 56 | * Base Image:`FROM python:3.10-slim`: This line specifies the base image to use. Here, we are using the official slim version of Python 3.10. The slim version is a smaller image that contains only the minimal packages needed to run Python, making it more lightweight. 57 | * Working Directory: `WORKDIR /app`: This command sets the working directory inside the container to /app. All subsequent commands in the Dockerfile will be executed within this directory. 58 | * Copying Dependencies: `COPY requirements.txt .`: This command copies the requirements.txt file from your local machine to the container's working directory. This step allows Docker to cache the installed dependencies, speeding up subsequent builds if the dependencies haven't changed. 59 | * Installing Dependencies:`RUN pip install --no-cache-dir -r requirements.txt`: This command installs the Python dependencies listed in the requirements.txt file. The --no-cache-dir option ensures that pip doesn't store any installation files, keeping the Docker image size smaller. 60 | * Copying Application Code: `COPY . .`: This command copies the rest of your application's code into the container's working directory. This includes the main.py file and any other necessary files. 61 | * Exposing Port:`EXPOSE 8000`: This command documents that the container listens on port 8000 at runtime. It does not actually publish the port to the host machine. To make the application accessible from outside the container, you need to map the port using the `-p` option when you run the container. 62 | * Entrypoint and CMD: `ENTRYPOINT ["python"]`: This sets the entrypoint for the container to the python command, meaning the container will execute Python when it starts. `CMD ["main.py"]`: This command specifies the default Python script to run. In this case, it's main.py, which is the FastAPI application. 63 | 64 | ## Building and Running the Docker Image 65 | ### Building the Docker Image: 66 | 67 | * To build the Docker image, use the following command: 68 | bash 69 | ``` 70 | docker build -t fastapi-app . 71 | ``` 72 | This command tells Docker to build the image from the Dockerfile in the current directory and tag it as fastapi-app. 73 | Running the Docker Container: 74 | 75 | Once the image is built, you can run the container using: 76 | 77 | ``` 78 | docker run -p 8000:8000 fastapi-app 79 | ``` 80 | This command runs the container, mapping port 8000 of the container to port 8000 on your host machine. You can now access the FastAPI application at 81 | 82 | http://localhost:8000 83 | 84 | > **Note:** This is only for illustration purposes. We are going to use Kubernetes to run the Docker container. 85 | 86 | [← Previous](02-FastAPI) | [Next →](04-Trivy) 87 | -------------------------------------------------------------------------------- /04-Trivy.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: "Day 4: Trivy" 4 | --- 5 | 6 | ![Trivy](img/Trivy.jpg) 7 | # What is Trivy? 8 | 9 | Trivy is a comprehensive, easy-to-use vulnerability scanner for containers. It detects vulnerabilities in operating system packages (Alpine, RHEL, CentOS, etc.) and application dependencies (Bundler, Composer, npm, yarn, etc.). Trivy is designed to be simple and fast, with a high level of accuracy, and it supports scanning both Docker images and filesystem directories. 10 | 11 | ## Installing Trivy 12 | 13 | Trivy can be easily installed on any system that supports Docker. Below are the steps to install Trivy: 14 | 15 | ### Prerequisites 16 | 17 | - Docker installed on your local machine 18 | 19 | ### Installation 20 | 21 | You can install Trivy using the following command: 22 | 23 | ```bash 24 | curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sudo sh -s -- -b /usr/local/bin 25 | ``` 26 | 27 | This command downloads and installs Trivy on your local machine, placing the executable in /usr/local/bin. 28 | 29 | Alternatively, you can use the Docker container method: 30 | 31 | ``` 32 | docker pull aquasec/trivy:latest 33 | ``` 34 | 35 | ## Scanning a Docker Image 36 | 37 | Once Trivy is installed, you can use it to scan a Docker image for vulnerabilities. 38 | 39 | ``` 40 | trivy image --severity HIGH,CRITICAL 41 | ``` 42 | 43 | This command will scan the image and report only the vulnerabilities with HIGH and CRITICAL severities. 44 | 45 | ## Interpreting the Scan Results 46 | Trivy will output a list of vulnerabilities found in the Docker image, categorized by severity (LOW, MEDIUM, HIGH, CRITICAL). Each entry in the report includes: 47 | 48 | * Vulnerability ID: The unique identifier of the vulnerability. 49 | * Package Name: The affected package or dependency. 50 | * Installed Version: The version of the package that is currently installed in the image. 51 | * Fixed Version: The version of the package in which the vulnerability has been fixed. 52 | * Severity: The level of threat the vulnerability poses (e.g., LOW, MEDIUM, HIGH, CRITICAL). 53 | 54 | ## Example Output 55 | Here's an example of what the output might look like: 56 | 57 | ``` 58 | fastapi-app (alpine 3.13.5) 59 | ============================ 60 | Total: 3 (UNKNOWN: 0, LOW: 1, MEDIUM: 1, HIGH: 1, CRITICAL: 0) 61 | 62 | +---------------+------------------+----------+-------------------+---------------+--------------------------------+ 63 | | VULNERABILITY | PACKAGE | SEVERITY | INSTALLED VERSION | FIXED VERSION | DESCRIPTION | 64 | +---------------+------------------+----------+-------------------+---------------+--------------------------------+ 65 | | CVE-2021-1234 | alpine-foo | HIGH | 1.2.3 | 1.2.4 | Description of the vulnerability| 66 | +---------------+------------------+----------+-------------------+---------------+--------------------------------+ 67 | | CVE-2021-5678 | alpine-bar | MEDIUM | 4.5.6 | 4.5.7 | Description of the vulnerability| 68 | +---------------+------------------+----------+-------------------+---------------+--------------------------------+ 69 | | CVE-2021-9101 | alpine-baz | LOW | 7.8.9 | 7.8.10 | Description of the vulnerability| 70 | +---------------+------------------+----------+-------------------+---------------+--------------------------------+ 71 | 72 | ``` 73 | 74 | > **Note:** The example is for illustration purposes, and the focus will be on integrating Trivy with GitHub Actions to identify vulnerabilities before deployment. 75 | 76 | [← Previous](03-Docker) | [Next →](05-AWS-ECR) 77 | -------------------------------------------------------------------------------- /05-AWS-ECR.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: "Day 5: AWS ECR" 4 | --- 5 | 6 | ![AWSECR](img/aws-ecr.jpg) 7 | # What is AWS Elastic Container Registry (ECR)? 8 | 9 | AWS Elastic Container Registry (ECR) is a fully managed Docker container registry that makes it easy for developers to store, manage, and deploy container images. ECR is integrated with AWS Identity and Access Management (IAM), so you can control who can access your container images. 10 | 11 | ## Prerequisites 12 | 13 | Before you can push Docker images to AWS ECR, you need the following: 14 | 15 | - AWS CLI installed and configured on your local machine. 16 | - Docker installed and running on your local machine. 17 | - An AWS account with appropriate permissions to create and push images to ECR. 18 | 19 | ## Configuring AWS ECR 20 | 21 | To start using AWS ECR, you need to create a repository where your Docker images will be stored. 22 | 23 | ### Step 1: Create an ECR Repository 24 | 25 | You can create an ECR repository using the AWS Management Console, AWS CLI, or an SDK. Here’s how to create one using the AWS CLI: 26 | 27 | ```bash 28 | aws ecr create-repository --repository-name my-repo --region $AWS_REGION 29 | ``` 30 | 31 | Replace `my-repo` with your desired repository name and `$AWS_REGION` with the region where you want to create the repository. 32 | 33 | ## Authenticate Docker with ECR 34 | Before you can push Docker images to ECR, you need to authenticate Docker with your ECR registry. This is done using the AWS CLI. 35 | 36 | ``` 37 | aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.$AWS_REGION.amazonaws.com 38 | ``` 39 | 40 | This command retrieves an authentication token using the AWS CLI. The token is required for Docker to authenticate with the AWS ECR registry. 41 | This command logs in to the ECR registry using the retrieved authentication token. The `--password-stdin` option allows you to pass the password securely from the AWS CLI command via stdin. 42 | 43 | # Tag and Push Docker Image to ECR 44 | 45 | Once Docker is authenticated with ECR, you can tag your Docker image and push it to your ECR repository. 46 | 47 | ```bash 48 | docker push ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPOSITORY:$GITHUB_SHA 49 | ``` 50 | 51 | > **Note:** This is only for illustration purposes. We are going to integrate ECR with GitHub Actions to ensure that Docker images are automatically built and pushed to ECR as part of your CI/CD pipeline. 52 | 53 | [← Previous](04-Trivy) | [Next →](06-Kubernetes) 54 | -------------------------------------------------------------------------------- /06-Kubernetes.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: "Day 6: Kubernetes" 4 | --- 5 | 6 | ![Kubernetes](img/kubernetes.jpg) 7 | # Kubernetes: Deployments and Services 8 | 9 | Kubernetes is an open-source platform designed to automate deploying, scaling, and operating application containers. It allows developers to manage containerized applications across a cluster of machines, providing essential features like scaling, load balancing, and automated rollouts and rollbacks. 10 | 11 | ## Kubernetes Deployments 12 | 13 | A **Deployment** in Kubernetes is a resource object that provides declarative updates to applications. You define the desired state of the application in a YAML file, and the Deployment controller makes the changes to the application’s current state to match the desired state. Deployments are used to create and manage a set of identical pods (containers running an application), ensuring that the specified number of replicas is always running. 14 | 15 | ### Key Features of Deployments: 16 | - **Replica Management**: Ensures that a specified number of replicas of a pod are running at all times. 17 | - **Rolling Updates**: Gradually replaces old versions of pods with new ones, ensuring zero downtime. 18 | - **Rollback**: If something goes wrong, you can roll back to a previous version of the deployment. 19 | 20 | ## Kubernetes Services 21 | 22 | A **Service** in Kubernetes is an abstract way to expose an application running on a set of Pods as a network service. With Kubernetes, you don't need to modify your application to use an unfamiliar service discovery mechanism. Kubernetes gives Pods their own IP addresses and a single DNS name for a set of Pods, and can load-balance across them. 23 | 24 | ### Types of Services: 25 | - **ClusterIP**: Exposes the service on a cluster-internal IP. This is the default ServiceType. 26 | - **NodePort**: Exposes the service on each Node’s IP at a static port (the NodePort). A ClusterIP service, to which the NodePort service routes, is automatically created. 27 | - **LoadBalancer**: Exposes the service externally using a cloud provider’s load balancer. 28 | - **ExternalName**: Maps a Service to a DNS name. 29 | 30 | ## YAML File Explanation 31 | 32 | The provided YAML file defines a Kubernetes Deployment and a Service. 33 | 34 | ### Deployment YAML 35 | 36 | ```yaml 37 | apiVersion: apps/v1 38 | kind: Deployment 39 | metadata: 40 | name: gpt-huggingface 41 | spec: 42 | replicas: 3 43 | selector: 44 | matchLabels: 45 | app: gpt-hf-pod 46 | template: 47 | metadata: 48 | labels: 49 | app: gpt-hf-pod 50 | spec: 51 | containers: 52 | - name: gptcontainer 53 | image: image_name 54 | ports: 55 | - containerPort: 8000 56 | ``` 57 | 58 | - **apiVersion: apps/v1**: Specifies the API version of the Kubernetes resource. For Deployments, it's `apps/v1`. 59 | - **kind: Deployment**: Defines the type of Kubernetes resource being created. 60 | - **metadata**: 61 | - **name: gpt-huggingface**: The name of the Deployment. 62 | - **spec**: 63 | - **replicas: 3**: Specifies that three replicas (instances) of the pod should be running. 64 | - **selector**: 65 | - **matchLabels**: 66 | - **app: gpt-hf-pod**: This label is used to identify the pods managed by this Deployment. 67 | - **template**: Describes the pod to be created. 68 | - **metadata**: 69 | - **labels**: 70 | - **app: gpt-hf-pod**: Labels that the pod will carry. These must match the `selector` to ensure the Deployment manages these pods. 71 | - **spec**: 72 | - **containers**: Defines the container specifications. 73 | - **name: gptcontainer**: The name of the container. 74 | - **image: image_name**: The Docker image to use for this container. Replace `image_name` with the actual image name. 75 | - **ports**: 76 | - **containerPort: 8000**: The port on which the container will listen. 77 | 78 | ```yaml 79 | apiVersion: v1 80 | kind: Service 81 | metadata: 82 | name: gpt-hf-service 83 | spec: 84 | type: NodePort 85 | selector: 86 | app: gpt-hf-pod 87 | ports: 88 | - port: 8000 89 | targetPort: 8000 90 | nodePort: 30007 91 | ``` 92 | 93 | ### Deploying the `deploy.yaml` File to Kubernetes 94 | 95 | To deploy the `deploy.yaml` file to your Kubernetes cluster, follow these steps: 96 | 97 | To deploy the deployment.yaml file, use the kubectl apply command: 98 | 99 | ``` 100 | kubectl apply -f deployment.yaml 101 | ``` 102 | 103 | This command will create the resources defined in the deploy.yaml file, such as the Deployment and Service. 104 | 105 | 106 | ### Verify the Deployment 107 | 108 | You can verify that your deployment was successful by checking the status of the deployment and services: 109 | 110 | ``` 111 | kubectl get deploy 112 | kubectl get services 113 | ``` 114 | 115 | These commands will display the running deployments and services, confirming that your application is deployed and accessible. 116 | 117 | > **Note:** This is for illustration purposes. We are going to deploy this configuration using GitHub Actions, which will automate the deployment process as part of your CI/CD pipeline. 118 | 119 | [← Previous](05-AWS-ECR) | [Next →](07-kube-score) 120 | -------------------------------------------------------------------------------- /07-kube-score.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: "Day 7: kube-score" 4 | --- 5 | 6 | ![kube-score](img/kube-score.jpg) 7 | # Using `kube-score` to Validate Kubernetes Manifests 8 | 9 | `kube-score` is a tool that helps you validate and improve the quality of your Kubernetes manifests. It performs a static analysis of your manifests and provides recommendations for improvements. This tool is particularly useful in CI/CD pipelines to ensure that your Kubernetes configurations follow best practices. 10 | 11 | ## Installing `kube-score` 12 | 13 | You can install `kube-score` by following the instructions on the [kube-score GitHub repository](https://github.com/zegl/kube-score). Alternatively, you can use `brew` on macOS: 14 | 15 | ```bash 16 | brew install kube-score 17 | ``` 18 | 19 | ### Validating Your Kubernetes Manifests with `kube-score` 20 | 21 | To validate your Kubernetes manifests, you can use the `kube-score score` command. This command analyzes your YAML files and provides a score based on various checks, such as pod security, container resource limits, and other best practices. 22 | 23 | ```bash 24 | kube-score score --output-format ci deploy.yaml 25 | ``` 26 | 27 | * kube-score score: This command runs kube-score to analyze the specified Kubernetes YAML file (deploy.yaml in this case). 28 | * --output-format ci: This option formats the output in a way that is suitable for continuous integration (CI) systems. The output will be concise and in a format that can be easily parsed by CI tools. 29 | * deploy.yaml: The file being analyzed by kube-score. Replace this with the path to your actual Kubernetes manifest file if different. 30 | 31 | ### Output 32 | 33 | The command will output a list of checks performed on the manifest, along with any warnings or recommendations for improvement. For example: 34 | 35 | ``` 36 | [WARNING] Container Resources 37 | · gptcontainer -> No resource limits set for container 38 | Resource limits are recommended to avoid resource contention issues in the cluster. 39 | Set resource limits using 'resources.limits.cpu' and 'resources.limits.memory'. 40 | 41 | [OK] Pod Probes 42 | · gptcontainer -> Liveness probe is configured 43 | · gptcontainer -> Readiness probe is configured 44 | ``` 45 | [← Previous](06-Kubernetes) | [Next →](08-EKS) 46 | -------------------------------------------------------------------------------- /08-EKS.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: "Day 8: AWS EKS" 4 | --- 5 | 6 | ![EKS](img/EKS.jpg) 7 | # Deploying and Managing Kubernetes Clusters with AWS EKS 8 | 9 | AWS Elastic Kubernetes Service (EKS) is a fully managed Kubernetes service that simplifies the process of running Kubernetes on AWS. EKS provides highly available and secure clusters that are integrated with AWS services, making it easier to manage Kubernetes deployments in the cloud. 10 | 11 | #### Setting Up an EKS Cluster 12 | 13 | To set up an EKS cluster, you typically follow these steps: 14 | 15 | 1. **Create an EKS Cluster**: Use the AWS Management Console, AWS CLI, or AWS CloudFormation to create an EKS cluster. 16 | 2. **Configure `kubectl`**: Ensure that `kubectl` is configured to interact with your EKS cluster. 17 | 3. **Deploy Kubernetes Resources**: Use `kubectl` to deploy your Kubernetes resources to the EKS cluster. 18 | 19 | #### Example Command to Configure `kubectl` 20 | 21 | Once your EKS cluster is set up, you need to configure `kubectl` to connect to it: 22 | 23 | ```bash 24 | aws eks update-kubeconfig --region $AWS_REGION --name $EKS_CLUSTER_NAME 25 | ``` 26 | 27 | ### Command Explanation 28 | 29 | - **`aws eks update-kubeconfig`**: This command updates your local `kubeconfig` file with the necessary configuration to connect to your EKS cluster. 30 | - **`--region $AWS_REGION`**: Specifies the AWS region where your EKS cluster is running. 31 | - **`--name $EKS_CLUSTER_NAME`**: The name of the EKS cluster you want to connect to. Replace `$EKS_CLUSTER_NAME` with the actual name of your cluster 32 | 33 | [← Previous](07-kube-score) | [Next →](09-kustomize) 34 | -------------------------------------------------------------------------------- /09-kustomize.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: "Day 9: kustomize" 4 | --- 5 | 6 | ![Kustomize](img/kustomize.jpg) 7 | # Working with `kustomize` to Manage Kubernetes Manifests 8 | 9 | `kustomize` is a tool that lets you customize Kubernetes resource definitions without modifying the original YAML files. It works by layering configurations on top of base resources, making it easier to manage different environments (e.g., development, staging, production) with a single set of YAML files. 10 | 11 | #### Example Commands 12 | 13 | 1. **Generate a Kustomization File** 14 | ``` 15 | kustomize create --autodetect 16 | ``` 17 | 18 | kustomize create --autodetect: This command generates a kustomization.yaml file by automatically detecting the Kubernetes resource files in the current directory. This file will define the resources, patches, and other customizations that kustomize will manage. 19 | 20 | 2. **Build Kubernetes Manifests** 21 | ``` 22 | kustomize build . 23 | ``` 24 | kustomize build .: This command builds the final Kubernetes manifests by applying the customizations defined in the kustomization.yaml file to the base resources. The output is a fully rendered set of YAML files that can be applied to your Kubernetes cluster. 25 | 26 | 3. **Apply the Built Manifests to a Kubernetes Cluster** 27 | ``` 28 | kubectl apply -k . 29 | ``` 30 | 31 | kubectl apply -k .: This command applies the customized resources directly to your Kubernetes cluster. The -k flag tells kubectl to use kustomize to build the manifests before applying them. 32 | 33 | **Set a New Image in Your Kustomization** 34 | 35 | ``` 36 | kustomize edit set image image_name=${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPOSITORY:$GITHUB_SHA 37 | ``` 38 | 39 | This command updates the kustomization.yaml file to set a new image for your Kubernetes deployment. The image name is set dynamically using environment variables or secrets provided in a CI/CD pipeline (e.g., GitHub Actions). Here, image_name is replaced with the Docker image hosted in AWS ECR, tagged with the commit SHA ($GITHUB_SHA) for traceability. 40 | 41 | [← Previous](08-EKS) | [Next →](10-github-action) 42 | -------------------------------------------------------------------------------- /10-github-action.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: "Day 10: GitHub Actions" 4 | --- 5 | 6 | ![GitHub Action](img/github_action.jpg) 7 | # GitHub Actions 8 | 9 | This GitHub Actions workflow automates the process of building, scanning, publishing a Docker image to Amazon ECR, and deploying it to an Amazon EKS cluster. 10 | 11 | ## Workflow Overview 12 | 13 | - **Name**: CI/CD Pipeline for EKS Deployment 14 | - **Triggers**: 15 | - Runs on push events to the `main` branch. 16 | - Runs on pull requests targeting the `main` branch. 17 | 18 | ## Environment Variables 19 | 20 | The `env` section defines environment variables that are used throughout the workflow: 21 | 22 | - **AWS_REGION**: AWS region where the ECR repository and EKS cluster are located. 23 | - **ECR_REPOSITORY**: Name of the ECR repository. 24 | - **EKS_CLUSTER_NAME**: Name of the EKS cluster. 25 | - **DEPLOYMENT_NAME**: Name of the Kubernetes deployment. 26 | - **IMAGE**: Name of the Docker image. 27 | 28 | ## Job: Build, Scan, Publish, and Deploy 29 | 30 | This job runs on an `ubuntu-latest` runner and is responsible for the entire CI/CD process. It runs in the `production` environment. 31 | 32 | ### Steps 33 | 34 | 1. **Checkout the Code** 35 | 36 | ```yaml 37 | - name: Checkout 38 | uses: actions/checkout@v2 39 | ``` 40 | 41 | - This step checks out the repository's code, making it available for subsequent steps. 42 | 43 | 2. **Configure AWS Credentials** 44 | 45 | ```yaml 46 | - name: Configure AWS credentials 47 | uses: aws-actions/configure-aws-credentials@v1 48 | with: 49 | aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} 50 | aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} 51 | aws-region: ${{ env.AWS_REGION }} 52 | ``` 53 | 54 | - This step configures the AWS credentials required to interact with AWS services like ECR and EKS. 55 | 56 | 3. **Login to Amazon ECR** 57 | 58 | ```yaml 59 | - name: Login to Amazon ECR 60 | id: login-ecr 61 | run: | 62 | aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.$AWS_REGION.amazonaws.com 63 | ``` 64 | 65 | - This step logs in to the Amazon ECR registry using AWS CLI. The login credentials are passed securely using the `--password-stdin` option. 66 | 67 | 4. **Build Docker Image** 68 | 69 | ```yaml 70 | - name: Build Docker image 71 | run: | 72 | docker build -t $ECR_REPOSITORY:$GITHUB_SHA . 73 | docker tag $ECR_REPOSITORY:$GITHUB_SHA ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPOSITORY:$GITHUB_SHA 74 | ``` 75 | 76 | - This step builds a Docker image from the checked-out code and tags it with the GitHub SHA for traceability. 77 | 78 | 5. **Install Trivy** 79 | 80 | ```yaml 81 | - name: Install Trivy 82 | run: | 83 | curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sudo sh -s -- -b /usr/local/bin 84 | ``` 85 | 86 | - This step installs Trivy, a security scanner for Docker images. 87 | 88 | 6. **Clean Up Disk Space** 89 | 90 | ```yaml 91 | - name: Clean up disk space 92 | run: | 93 | sudo rm -rf /var/lib/apt/lists/* 94 | sudo apt-get clean 95 | ``` 96 | 97 | - This step cleans up disk space to ensure that the runner has enough free space for subsequent operations. 98 | 99 | 7. **Scan Docker Image with Trivy** 100 | 101 | ```yaml 102 | - name: Scan Docker image with Trivy 103 | run: | 104 | trivy image --severity HIGH,CRITICAL $ECR_REPOSITORY:$GITHUB_SHA 105 | ``` 106 | 107 | - This step scans the Docker image for vulnerabilities with high and critical severity using Trivy. 108 | 109 | 8. **Push Docker Image to Amazon ECR** 110 | 111 | ```yaml 112 | - name: Push Docker image to Amazon ECR 113 | run: | 114 | docker push ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPOSITORY:$GITHUB_SHA 115 | ``` 116 | 117 | - This step pushes the Docker image to the Amazon ECR repository. 118 | 119 | 9. **Setup kubectl** 120 | 121 | ```yaml 122 | - name: Setup kubectl 123 | run: | 124 | curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.21.13/2022-06-08/bin/linux/amd64/kubectl 125 | chmod +x ./kubectl 126 | mkdir -p $HOME/bin && cp ./kubectl $HOME/bin/kubectl && export PATH=$HOME/bin:$PATH 127 | echo 'export PATH=$HOME/bin:$PATH' >> ~/.bashrc 128 | ``` 129 | 130 | - This step installs `kubectl`, the Kubernetes command-line tool, and configures the system path to include it. 131 | 132 | 10. **Set up Kustomize** 133 | 134 | ```yaml 135 | - name: Set up Kustomize 136 | run: | 137 | curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash 138 | chmod u+x kustomize 139 | sudo mv kustomize /usr/local/bin/kustomize 140 | ``` 141 | 142 | - This step installs Kustomize, a tool for customizing Kubernetes YAML configurations. 143 | 144 | 11. **Update kubeconfig** 145 | 146 | ```yaml 147 | - name: Update kubeconfig 148 | run: | 149 | aws eks update-kubeconfig --name $EKS_CLUSTER_NAME --region $AWS_REGION 150 | ``` 151 | 152 | - This step updates the kubeconfig file to allow `kubectl` to interact with the EKS cluster. 153 | 154 | 12. **Install kube-score** 155 | 156 | ```yaml 157 | - name: Install kube-score 158 | run: | 159 | curl -L -o kube-score https://github.com/zegl/kube-score/releases/download/v1.11.0/kube-score_1.11.0_linux_amd64 160 | chmod +x kube-score 161 | sudo mv kube-score /usr/local/bin/ 162 | ``` 163 | 164 | - This step installs kube-score, a tool that validates Kubernetes manifests for best practices. 165 | 166 | 13. **Lint Kubernetes Manifests with kube-score** 167 | 168 | ```yaml 169 | - name: Lint Kubernetes manifests with kube-score 170 | continue-on-error: true 171 | run: | 172 | kube-score score --output-format ci deploy.yaml 173 | ``` 174 | 175 | - This step lints the Kubernetes manifests using kube-score. The `continue-on-error` flag allows the workflow to proceed even if this step fails. 176 | 177 | 14. **Deploy with Kustomize** 178 | 179 | ```yaml 180 | - name: Deploy with Kustomize 181 | run: | 182 | kustomize edit set image image_name=${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPOSITORY:$GITHUB_SHA 183 | kustomize build . | kubectl apply -f - 184 | kubectl rollout status deployment/$DEPLOYMENT_NAME 185 | kubectl get services -o wide 186 | ``` 187 | 188 | -This step uses Kustomize to update the image in the Kubernetes manifests, builds the final manifests, and deploys them to the EKS cluster. It also checks the status of the deployment and lists the services. 189 | 190 | [← Previous](09-kustomize) 191 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # End-to-End LLMOps Pipeline from Scratch Using Hugging Face, FastAPI, Docker, Trivy, AWS Elastic Container Registry (ECR), Kubernetes, Kube-score, AWS Elastic Kubernetes Service (EKS), Kustomize, and GitHub Actions 2 | 3 | ## About This Project 4 | 5 | This project is an excerpt from my book. For a more detailed explanation and in-depth coverage of the topics, please check out the book [here](https://pratimuniyal.gumroad.com/l/BuildinganLLMOpsPipelineUsingHuggingFace). 6 | 7 | ![Book](img/hugging_face_book.png) 8 | 9 | 10 | ## Project Overview 11 | 12 | This repository provides a step-by-step guide to building an LLM (Large Language Model) pipeline from scratch. Below is an overview of what the pipeline looks like: 13 | 14 | ![llmops_pipeline](img/llmops_new.jpg) 15 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-minimal 2 | title: End-to-End LLMOps Project from Scratch Using Hugging Face, FastAPI, Docker, Trivy, AWS Elastic Container Registry (ECR), Kubernetes, Kube-score, AWS Elastic Kubernetes Service (EKS), Kustomize, and GitHub Actions 3 | description: This repository provides a step-by-step guide to building an LLM (Large Language Model) pipeline from scratch. 4 | -------------------------------------------------------------------------------- /_layouts/default.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | {{ page.title }} 7 | 8 | 9 | 10 | 11 |
12 |
13 |

End-to-End LLMOps

14 | 26 |
27 | 28 |
29 | {{ content }} 30 |
31 |
32 | 33 | 34 | -------------------------------------------------------------------------------- /assets/css/style.css: -------------------------------------------------------------------------------- 1 | body { 2 | margin: 0; 3 | font-family: Arial, sans-serif; 4 | } 5 | 6 | .wrapper { 7 | display: flex; 8 | } 9 | 10 | .sidebar { 11 | width: 220px; 12 | background-color: #2c3e50; 13 | padding: 20px; 14 | height: 100vh; 15 | box-sizing: border-box; 16 | position: fixed; 17 | top: 0; 18 | left: 0; 19 | color: white; 20 | } 21 | 22 | .sidebar h2 { 23 | color: #ecf0f1; 24 | text-align: center; 25 | } 26 | 27 | .sidebar ul { 28 | list-style-type: none; 29 | padding: 0; 30 | margin: 0; 31 | } 32 | 33 | .sidebar ul li { 34 | margin-bottom: 15px; 35 | } 36 | 37 | .sidebar ul li a { 38 | text-decoration: none; 39 | color: #ecf0f1; 40 | font-size: 16px; 41 | } 42 | 43 | .sidebar ul li a:hover { 44 | color: #3498db; 45 | } 46 | 47 | .sidebar ul li a i { 48 | margin-right: 10px; 49 | } 50 | 51 | .content { 52 | margin-left: 240px; 53 | padding: 20px; 54 | width: calc(100% - 240px); 55 | } 56 | 57 | h1, h2, h3, h4, h5, h6 { 58 | font-size: 18px; 59 | } 60 | 61 | img { 62 | max-width: 100%; 63 | height: auto; 64 | margin-bottom: 20px; 65 | } 66 | -------------------------------------------------------------------------------- /img/EKS.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/100daysofdevops/end-to-end-llmops-pipeline/5ce7b4a1b6e330dbd942d9562ca149887478da07/img/EKS.jpg -------------------------------------------------------------------------------- /img/FastAPI.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/100daysofdevops/end-to-end-llmops-pipeline/5ce7b4a1b6e330dbd942d9562ca149887478da07/img/FastAPI.jpg -------------------------------------------------------------------------------- /img/Trivy.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/100daysofdevops/end-to-end-llmops-pipeline/5ce7b4a1b6e330dbd942d9562ca149887478da07/img/Trivy.jpg -------------------------------------------------------------------------------- /img/aws-ecr.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/100daysofdevops/end-to-end-llmops-pipeline/5ce7b4a1b6e330dbd942d9562ca149887478da07/img/aws-ecr.jpg -------------------------------------------------------------------------------- /img/docker.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/100daysofdevops/end-to-end-llmops-pipeline/5ce7b4a1b6e330dbd942d9562ca149887478da07/img/docker.jpg -------------------------------------------------------------------------------- /img/github_action.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/100daysofdevops/end-to-end-llmops-pipeline/5ce7b4a1b6e330dbd942d9562ca149887478da07/img/github_action.jpg -------------------------------------------------------------------------------- /img/hugging-face.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/100daysofdevops/end-to-end-llmops-pipeline/5ce7b4a1b6e330dbd942d9562ca149887478da07/img/hugging-face.jpg -------------------------------------------------------------------------------- /img/hugging_face_book.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/100daysofdevops/end-to-end-llmops-pipeline/5ce7b4a1b6e330dbd942d9562ca149887478da07/img/hugging_face_book.png -------------------------------------------------------------------------------- /img/kube-score.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/100daysofdevops/end-to-end-llmops-pipeline/5ce7b4a1b6e330dbd942d9562ca149887478da07/img/kube-score.jpg -------------------------------------------------------------------------------- /img/kubernetes.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/100daysofdevops/end-to-end-llmops-pipeline/5ce7b4a1b6e330dbd942d9562ca149887478da07/img/kubernetes.jpg -------------------------------------------------------------------------------- /img/kustomize.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/100daysofdevops/end-to-end-llmops-pipeline/5ce7b4a1b6e330dbd942d9562ca149887478da07/img/kustomize.jpg -------------------------------------------------------------------------------- /img/llmops_new.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/100daysofdevops/end-to-end-llmops-pipeline/5ce7b4a1b6e330dbd942d9562ca149887478da07/img/llmops_new.jpg --------------------------------------------------------------------------------