├── .gitignore ├── flake.lock ├── flake.nix ├── README.md ├── 01-spellbook-cli └── README.md ├── 02-enchanters-chat └── README.md ├── 04-scryer-crawler └── README.md ├── 05-conductor-async-job └── README.md └── 03-sorcerers-eyes └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | /spellbook/ 2 | /enchanters/ 3 | /sorcerer_eye/ 4 | /scryer/ 5 | /conductor/ 6 | -------------------------------------------------------------------------------- /flake.lock: -------------------------------------------------------------------------------- 1 | { 2 | "nodes": { 3 | "nixpkgs": { 4 | "locked": { 5 | "lastModified": 1733261153, 6 | "narHash": "sha256-eq51hyiaIwtWo19fPEeE0Zr2s83DYMKJoukNLgGGpek=", 7 | "owner": "NixOS", 8 | "repo": "nixpkgs", 9 | "rev": "b681065d0919f7eb5309a93cea2cfa84dec9aa88", 10 | "type": "github" 11 | }, 12 | "original": { 13 | "owner": "NixOS", 14 | "ref": "nixos-24.11", 15 | "repo": "nixpkgs", 16 | "type": "github" 17 | } 18 | }, 19 | "root": { 20 | "inputs": { 21 | "nixpkgs": "nixpkgs" 22 | } 23 | } 24 | }, 25 | "root": "root", 26 | "version": 7 27 | } 28 | -------------------------------------------------------------------------------- /flake.nix: -------------------------------------------------------------------------------- 1 | { 2 | inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.11"; 3 | 4 | outputs = {nixpkgs, ...}: let 5 | system = "aarch64-darwin"; 6 | pkgs = import nixpkgs {inherit system;}; 7 | inherit (pkgs.beam.interpreters) erlang_27; 8 | inherit (pkgs.beam) packagesWith; 9 | beam = packagesWith erlang_27; 10 | in { 11 | packages.${system} = { 12 | # 01 - cli 13 | spellbook = null; 14 | # 02 - web chat 15 | enchanters = null; 16 | # 03 - real-time observer 17 | sorcerers_eye = null; 18 | # 04 - web crawler 19 | scryer = null; 20 | # 05 - async job processor 21 | conductor = null; 22 | }; 23 | 24 | devShells.${system}.default = with pkgs; 25 | mkShell { 26 | packages = with pkgs; 27 | [beam.elixir erlang_27 nodejs sqlite] 28 | ++ lib.optional stdenv.isLinux [inotify-tools] 29 | ++ lib.optional stdenv.isDarwin [ 30 | darwin.apple_sdk.frameworks.CoreServices 31 | darwin.apple_sdk.frameworks.CoreFoundation 32 | ]; 33 | }; 34 | }; 35 | } 36 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Arcane Quests (elixir) 2 | 3 | This repository contains readmes for five different [Elixir](elixir) projects designed to explore the unique strengths of the [Elixir](elixir) language and the [BEAM](beam). 4 | 5 | Inside this repository are 5 directories, each representing a single project idea. 6 | 7 | Each of the project ideas contains a README and any supplementary resources or advice to help you build the project. 8 | 9 | ## 01 - Spellbook CLI 10 | 11 | This project is to build a simple CLI tool called `spellbook`. It allows users to manage and execute "spells" (tasks) stored in a local file system. This beginner-friendly project introduces [Elixir's](elixir) functional paradigm, providing a foundation for learning [Elixir's](elixir) core concepts. 12 | 13 | ## 02 - Phoenix Enchanter's Chat 14 | 15 | This project is to build a real-time chat application using [Phoenix LiveView](phoenix), called `enchanters`. The project demonstrates the [BEAM's](beam) ability to handle thousands of concurrent connections and highlights the use of [Erlang's](erlang) actor model for seamless real-time communication. 16 | 17 | ## 03 - Sorcerer's Eye Monitoring 18 | 19 | This project is to build a distributed system monitoring tool called `sorcerers_eye`. It collects and displays metrics across nodes, handling failures gracefully with [GenServers](genserver) and [Supervisors](supervisor). This project highlights [Elixir's](elixir) focus on fault tolerance and distributed systems. 20 | 21 | ## 04 - Mystic Web Crawler 22 | 23 | This project is to build a concurrent web crawler called `scryer`. It asynchronously fetches and processes web pages using multiple processes, showcasing [Elixir's](elixir) concurrency and the actor model for efficient task management. It also highlights the benefits of process supervision to manage failures. 24 | 25 | ## 05 - Alchemist's Task Scheduler 26 | 27 | This project is to build a background job processing system called `conductor`. It demonstrates how to use [OTP](erlang) behaviors to manage queues, workers, and job scheduling. The project emphasizes the [BEAM's](beam) ability to handle concurrency and back-pressure gracefully. 28 | 29 | # General Getting Started Advice 30 | 31 | For all projects, make sure you start off by creating a new Mix project with `mix new --sup ` and using the default OTP app structure. 32 | 33 | Each project is an opportunity to explore [Elixir's](elixir) unique paradigms, combining functional programming, the actor model, and the [BEAM's](beam) resilience. Dive in and enjoy the arcane magic of [Elixir](elixir)! 34 | 35 | ## Thanks and inspirations 36 | - [concept video by Dreams of Code for 5 golang projects](goprojects) 37 | 38 | 39 | [phoenix]: https://phoenixframework.org 40 | [elixir]: https://hexdocs.pm/elixir/introduction.html 41 | [erlang]: https://www.erlang.org 42 | [beam]: https://www.erlang.org/blog/a-brief-beam-primer/ 43 | [nexus]: https://hexdocs.pm/nexus_cli 44 | [genserver]: https://hexdocs.pm/elixir/GenServer.html 45 | [supervisor]: https://hexdocs.pm/elixir/Supervisor.html 46 | [goprojects]: https://youtu.be/gXmznGEW9vo 47 | -------------------------------------------------------------------------------- /01-spellbook-cli/README.md: -------------------------------------------------------------------------------- 1 | # Spellbook CLI 2 | 3 | ## Goal 4 | 5 | Create a CLI application for managing and executing "spells" (tasks) in the terminal, using Elixir. 6 | 7 | ```sh 8 | $ spellbook 9 | ``` 10 | 11 | ## Requirements 12 | 13 | The application should perform CRUD operations on a local data file of spells. Each operation should be accessible via a CLI command. The operations are as follows: 14 | 15 | ### Add 16 | 17 | Create new spells by providing a description via the `add` command: 18 | 19 | ```sh 20 | $ spellbook add 21 | ``` 22 | 23 | For example: 24 | 25 | ```sh 26 | $ spellbook add “Learn Phoenix LiveView basics” 27 | ``` 28 | 29 | should add a new "spell" with the description "Learn Phoenix LiveView basics." 30 | 31 | ### List 32 | 33 | Display all **active** (not completed) spells. Use a flag to display all spells regardless of status: 34 | 35 | ```sh 36 | $ spellbook list 37 | ``` 38 | 39 | For example: 40 | 41 | ```sh 42 | ID Spell Description Created 43 | 1 Learn Phoenix LiveView basics a minute ago 44 | 2 Understand OTP and GenServer 10 seconds ago 45 | ``` 46 | 47 | To show all spells (active and completed): 48 | 49 | ```sh 50 | $ spellbook list -a 51 | ID Spell Description Created Completed 52 | 1 Learn Phoenix LiveView basics 2 minutes ago false 53 | 2 Understand OTP and GenServer a minute ago true 54 | ``` 55 | 56 | ### Complete 57 | 58 | Mark a spell as completed: 59 | 60 | ```sh 61 | $ spellbook complete 62 | ``` 63 | 64 | For example: 65 | 66 | ```sh 67 | $ spellbook complete 1 68 | ``` 69 | 70 | ### Delete 71 | 72 | Remove a spell permanently from the data file: 73 | 74 | ```sh 75 | $ spellbook delete 76 | ``` 77 | 78 | For example: 79 | 80 | ```sh 81 | $ spellbook delete 2 82 | ``` 83 | 84 | ## Notable Packages Used 85 | 86 | - [Nexus](https://hex.pm/packages/nexus_cli) for CLI command parsing and execution. 87 | - [File](https://hexdocs.pm/elixir/File.html) and [IO](https://hexdocs.pm/elixir/IO.html) modules for reading and writing to the local file system. 88 | - [DateTime](https://hexdocs.pm/elixir/DateTime.html) for managing timestamps. 89 | - [Jason](https://hex.pm/packages/jason) for JSON serialization and deserialization if you're using erlang/otp pre v27. If you're using erlang/otp v27+ then you can use the [json](https://www.erlang.org/doc/apps/stdlib/json.html#) module from standard library. 90 | 91 | ## Custom Resources 92 | 93 | ### Example Application 94 | 95 | You can find an example implementation of this CLI tool in the [releases](https://github.com/zoedsoupe/arcane-quests/releases) section. 96 | 97 | ### Example Data File 98 | 99 | A JSON-based data file might look like this: 100 | 101 | ```json 102 | [ 103 | { 104 | "id": 1, 105 | "description": "Learn Phoenix LiveView basics", 106 | "created_at": "2024-12-04T15:00:00Z", 107 | "is_completed": false 108 | }, 109 | { 110 | "id": 2, 111 | "description": "Understand OTP and GenServer", 112 | "created_at": "2024-12-04T15:05:00Z", 113 | "is_completed": true 114 | } 115 | ] 116 | ``` 117 | 118 | ## Technical Considerations 119 | 120 | ### File Locking 121 | 122 | To prevent concurrent read/write issues with the data file, ensure exclusive access using Elixir’s [File.open/2](https://hexdocs.pm/elixir/File.html#open/2) with appropriate modes. 123 | 124 | ### Error Handling 125 | 126 | Provide clear error messages for invalid commands, such as attempting to complete or delete a spell that does not exist. 127 | 128 | ## Extra Features 129 | 130 | - Add a due date field to spells to help prioritize tasks. 131 | - Support a remind command to list spells due soon. 132 | - Implement a search feature to find spells by keywords in their description. 133 | - Enhance the data store from JSON to SQLite for more advanced features. 134 | 135 | 136 | -------------------------------------------------------------------------------- /02-enchanters-chat/README.md: -------------------------------------------------------------------------------- 1 | # Phoenix Enchanter's Chat 2 | 3 | ## Goal 4 | 5 | Create a real-time chat application using [Phoenix LiveView](https://hexdocs.pm/phoenix_live_view), leveraging the BEAM's ability to handle thousands of concurrent connections. 6 | 7 | ```sh 8 | $ mix phx.new enchanters 9 | ``` 10 | 11 | ## Requirements 12 | 13 | The application should allow users to join chat rooms, send messages, and see real-time updates without page reloads. Each feature should be implemented using Phoenix LiveView. 14 | 15 | ### Join a Chat Room 16 | 17 | Users should be able to join a chat room by entering their name and selecting a room: 18 | 19 | - The root page (/) should display a form where users can input their name and choose a room. 20 | - Upon submission, they should be redirected to the room’s live view (/rooms/:room_name). 21 | 22 | ### Send Messages 23 | 24 | Users in a chat room should be able to send messages: 25 | - Provide an input field and a submit button to send messages. 26 | - Display sent messages in real-time to all participants in the room. 27 | 28 | ### Real-Time Updates 29 | 30 | Messages sent in a room should appear immediately for all users without requiring a page refresh. Use [Phoenix PubSub](https://hexdocs.pm/phoenix_pubsub/Phoenix.PubSub.html) to broadcast updates across all connected clients. 31 | 32 | ### Persistent Storage 33 | 34 | Messages can be persisted in a database using Ecto, allowing users to see the chat history when joining a room or in-memory using something like [ETS](https://www.erlang.org/doc/apps/stdlib/ets.html#) 35 | 36 | ### Example Features 37 | 38 | #### Homepage 39 | 40 | A simple form to join a chat room: 41 | 42 | ``` 43 | Welcome to Enchanter's Chat! 44 | Name: [________] 45 | Room: [________] [Join Room] 46 | ``` 47 | 48 | #### Chat Room 49 | 50 | An interactive live page showing the chat history and allowing users to send messages: 51 | 52 | ``` 53 | Room: Magic Circle 54 | [John]: Hello, everyone! 55 | [Jane]: Hi, John! Ready for some magic? 56 | 57 | [__________________________] [Send] 58 | ``` 59 | 60 | ## Notable Packages Used 61 | 62 | - [Phoenix LiveView](https://hexdocs.pm/phoenix_live_view) for real-time updates. 63 | - [Ecto](https://hexdocs.pm/ecto) for data mapping and if you intend to persist messages into database 64 | - [Peri](https://hexdocs.pm/peri) for data mapping and schema validation if you want to validate raw data and persist messages in-memory 65 | - [Phoenix PubSub](https://hexdocs.pm/phoenix_pub_sub) for broadcasting messages. 66 | - Tailwind CSS for simple styling (optional, you can safely use raw CSS). 67 | 68 | ## Custom Resources 69 | 70 | ### Example Application 71 | 72 | You can find an example implementation of this chat application in the [releases]() section. 73 | 74 | ### Example Database Schema 75 | 76 | If you intend to persist messages in database, a simple schema for messages could be: 77 | 78 | ```ex 79 | defmodule Enchanters.Message do 80 | use Ecto.Schema 81 | 82 | schema "messages" do 83 | field :name, :string 84 | field :content, :string 85 | field :room, :string 86 | 87 | timestamps() 88 | end 89 | end 90 | ``` 91 | 92 | ### Example Routes 93 | 94 | ```ex 95 | # lib/enchanters_web/router.ex 96 | 97 | scope "/", EnchantersWeb do 98 | live "/", HomeLive, :index 99 | live "/rooms/:room_name", RoomLive, :index 100 | end 101 | ``` 102 | 103 | ## Technical Considerations 104 | 105 | ### Concurrency 106 | 107 | Leverage the BEAM’s concurrency model to allow multiple users to interact with the system in real-time. Each room can be managed by a separate process, isolating the state and communication for better fault tolerance. 108 | 109 | ### Error Handling 110 | 111 | Provide clear error messages for invalid inputs, such as trying to send an empty message or joining a non-existent room. 112 | 113 | ### Security 114 | 115 | - Implement basic sanitization to prevent XSS attacks. 116 | 117 | ## Extra Features 118 | 119 | - Add user authentication for personalized chat experiences. 120 | - Allow users to create private rooms with passwords. 121 | - Add timestamps to messages for better context. 122 | - Display a list of active users in each chat room. 123 | - Support markdown/html syntax for chat inputs 124 | - Support file or image sharing within the chat. 125 | 126 | 127 | -------------------------------------------------------------------------------- /04-scryer-crawler/README.md: -------------------------------------------------------------------------------- 1 | # Mystic Web Crawler 2 | 3 | ## Goal 4 | 5 | Create a concurrent web crawler called `scryer` to fetch and process web pages asynchronously. This project demonstrates [Elixir's](https://elixir-lang.org/) concurrency model and the actor model for efficient task management. It also highlights the use of [Supervisors](https://hexdocs.pm/elixir/Supervisor.html) to handle process failures gracefully. 6 | 7 | ```sh 8 | $ mix new scryer --sup 9 | ``` 10 | 11 | ## Requirements 12 | 13 | The application should recursively crawl a website, identify dead links (links returning 4xx or 5xx HTTP status codes), and report the results. 14 | 15 | ### Crawling Web Pages 16 | 17 | - Start crawling from a given URL. 18 | - Extract all links from the page. 19 | - Recursively visit links within the same domain. 20 | - Skip links pointing to external domains. 21 | 22 | ### Concurrent Processing 23 | 24 | - Use multiple processes to fetch and process web pages concurrently. 25 | - Limit the number of concurrent processes to avoid overwhelming the target server. 26 | 27 | ### Error Handling 28 | 29 | - Detect and log errors such as: 30 | - Dead links (HTTP 4xx or 5xx). 31 | - Timeouts or connection errors. 32 | - Use a Supervisor to restart failed processes. 33 | 34 | ### Results Aggregation 35 | 36 | - Collect and display the results in a structured format (e.g., a CLI table or JSON file): 37 | - Total number of links. 38 | - Number of dead links. 39 | - List of dead links with their status codes. 40 | 41 | ## Example Features 42 | 43 | ### CLI Input 44 | 45 | Start the crawler with a URL: 46 | 47 | ```sh 48 | $ scryer crawl https://example.com 49 | ``` 50 | 51 | ### CLI Output 52 | 53 | Display a summary of results: 54 | 55 | ``` 56 | Summary: 57 | - Total Links: 50 58 | - Dead Links: 5 59 | 60 | Dead Links: 61 | - https://example.com/broken1 (404) 62 | - https://example.com/broken2 (500) 63 | ``` 64 | 65 | ## Notable Packages Used 66 | 67 | - [Req](https://hexdocs.pm/req) or [Tesla](https://hexdocs.pm/tesla) for HTTP requests. 68 | - [Floki](https://hexdocs.pm/floki) for HTML parsing and extracting links. 69 | - [Task](https://hexdocs.pm/elixir/Task.html) for concurrent processing. 70 | - [Supervisor](https://hexdocs.pm/elixir/Supervisor.html) for process supervision. 71 | 72 | ## Custom Resources 73 | 74 | ### Example Application 75 | 76 | You can find an example implementation of this web crawler in the [releases ]section. 77 | 78 | ### Example URL Filtering 79 | 80 | A simple function to filter links: 81 | 82 | ```ex 83 | defmodule Scryer.Filter do 84 | def filter_links(links, base_url) do 85 | Enum.filter(links, fn link -> 86 | URI.parse(link).host == URI.parse(base_url).host 87 | end) 88 | end 89 | end 90 | ``` 91 | 92 | ### Example Supervisor Tree 93 | 94 | ```ex 95 | defmodule Scryer.Application do 96 | use Application 97 | 98 | def start(_type, _args) do 99 | children = [ 100 | {DynamicSupervisor, strategy: :one_for_one, name: Scryer.CrawlerSupervisor} 101 | ] 102 | 103 | opts = [strategy: :one_for_one, name: Scryer.Supervisor] 104 | Supervisor.start_link(children, opts) 105 | end 106 | end 107 | ``` 108 | 109 | ### Example crawler worker 110 | 111 | ```ex 112 | defmodule Scryer.Worker do 113 | use Task 114 | 115 | def start_link(_opts) do 116 | Task.start_link(fn -> crawl() end) 117 | end 118 | 119 | defp crawl do 120 | # todo... 121 | end 122 | end 123 | ``` 124 | 125 | ## Technical Considerations 126 | 127 | ### Concurrency 128 | 129 | - Use [Task.async_stream/3](https://hexdocs.pm/elixir/Task.html#async_stream/3) to fetch pages concurrently while limiting the number of active tasks. 130 | 131 | ### Link Extraction 132 | 133 | - Parse HTML with Floki to find all `` tags. 134 | - Resolve relative URLs to absolute URLs using [URI.merge/2](https://hexdocs.pm/elixir/URI.html#merge/2). 135 | 136 | ### Error Handling 137 | 138 | - Handle HTTP timeouts (429), connection failures (5xx), and invalid URLs gracefully. 139 | - Log errors and continue processing remaining pages. 140 | 141 | ### Rate Limiting 142 | 143 | - Add a delay between requests to avoid overwhelming the target server. 144 | - Use :timer.sleep/1 or a rate-limiting library. 145 | 146 | ## Extra Features 147 | 148 | - Save results to a file in JSON or CSV format. 149 | - Visualize crawling progress with a spinner or progress bar in the CLI. 150 | - Add support for sitemaps to enhance crawling efficiency. 151 | - Include an option to crawl external domains. 152 | - Integrate with a web dashboard to display crawling results in real-time. 153 | 154 | 155 | -------------------------------------------------------------------------------- /05-conductor-async-job/README.md: -------------------------------------------------------------------------------- 1 | # Alchemist's Task Scheduler 2 | 3 | ## Goal 4 | 5 | Create a background job processing system called `conductor` to manage job queues, workers, and scheduling. This project demonstrates the use of [OTP behaviors](https://hexdocs.pm/elixir/GenServer.html) such as `GenServer` and `Supervisor`, highlighting the [BEAM's](https://www.erlang.org/doc/white_paper.html) ability to handle concurrency and back-pressure efficiently. 6 | 7 | ```sh 8 | $ mix new conductor --sup 9 | ``` 10 | 11 | ## Requirements 12 | 13 | The application should manage a queue of background jobs, execute them with workers, and gracefully handle job failures or retries. 14 | 15 | ### Job Queues 16 | 17 | - Create a queue to store jobs. 18 | - Jobs should include: 19 | - Unique ID 20 | - Payload (e.g., JSON data or Elixir map) 21 | - Status (`pending`, `in_progress`, `completed`, `failed`) 22 | - Retry count and max retries 23 | 24 | ### Worker Processes 25 | 26 | - Spawn worker processes to execute jobs from the queue. 27 | - Limit the number of concurrent workers to control system load. 28 | 29 | ### Job Scheduling 30 | 31 | - Support immediate and scheduled job execution: 32 | - Immediate jobs are executed as soon as a worker is available. 33 | - Scheduled jobs are executed after a specified delay. 34 | 35 | ### Failure Handling and Retries 36 | 37 | - Automatically retry failed jobs up to a configurable maximum. 38 | - Track failure reasons and logs for debugging. 39 | 40 | ### Monitoring and Metrics 41 | 42 | - Provide a CLI command or web dashboard to view job statuses and queue health: 43 | - Total jobs 44 | - Completed jobs 45 | - Failed jobs 46 | - Pending jobs 47 | - In-progress jobs 48 | 49 | ## Example Features 50 | 51 | ### Enqueue a Job 52 | 53 | ```sh 54 | $ conductor enqueue --payload '{ "task": "send_email", "email": "test@example.com" }' 55 | ``` 56 | 57 | ### View Job Queue 58 | 59 | ```sh 60 | $ conductor jobs 61 | ID Task Status Retries 62 | 1 send_email pending 0 63 | 2 data_import in_progress 1 64 | ``` 65 | 66 | ## Notable Packages Used 67 | 68 | - [GenServer](https://hexdocs.pm/elixir/GenServer.html) for job queue and worker processes. 69 | - [DynamicSupervisor](https://hexdocs.pm/elixir/DynamicSupervisor.html) for managing worker processes. 70 | - [Ecto](https://hexdocs.pm/ecto) for persistent job storage (optional). 71 | - [Telemetry](https://hexdocs.pm/telemetry/readme.html) for monitoring job metrics. 72 | - [Oban](https://hexdocs.pm/oban) for inspiration as a async job queue processing (backed with postgresql) 73 | 74 | ## Custom Resources 75 | 76 | ### Example Application 77 | 78 | You can find an example implementation of this task scheduler in the [releases]section. 79 | 80 | ### Example Job Queue Implementation 81 | 82 | A simple GenServer for managing the job queue: 83 | 84 | ```ex 85 | defmodule Conductor.JobQueue do 86 | use GenServer 87 | 88 | def start_link(_) do 89 | GenServer.start_link(__MODULE__, %{}, name: __MODULE__) 90 | end 91 | 92 | def enqueue(job) do 93 | GenServer.call(__MODULE__, {:enqueue, job}) 94 | end 95 | 96 | def fetch_next() do 97 | GenServer.call(__MODULE__, :fetch_next) 98 | end 99 | 100 | def init(_) do 101 | {:ok, []} 102 | end 103 | 104 | def handle_call({:enqueue, job}, _from, state) do 105 | {:reply, :ok, [job | state]} 106 | end 107 | 108 | def handle_call(:fetch_next, _from, [next | rest]) do 109 | {:reply, {:ok, next}, rest} 110 | end 111 | 112 | def handle_call(:fetch_next, _from, []) do 113 | {:reply, :empty, []} 114 | end 115 | end 116 | ``` 117 | 118 | ### Example Worker Implementation 119 | 120 | ```ex 121 | defmodule Conductor.Worker do 122 | use GenServer 123 | 124 | def start_link(job) do 125 | GenServer.start_link(__MODULE__, job, name: via_tuple(job.id)) 126 | end 127 | 128 | def init(job) do 129 | Task.start(fn -> execute(job) end) 130 | {:ok, job} 131 | end 132 | 133 | defp execute(%{payload: payload} = job) do 134 | # Simulate job execution 135 | IO.inspect(payload, label: "Executing Job") 136 | :timer.sleep(1000) 137 | 138 | # Handle success or failure 139 | case :rand.uniform(10) > 2 do 140 | true -> IO.puts("Job #{job.id} completed successfully.") 141 | false -> IO.puts("Job #{job.id} failed.") 142 | end 143 | end 144 | end 145 | ``` 146 | 147 | ## Technical Considerations 148 | 149 | ### Back-Pressure Management 150 | 151 | - Limit the number of concurrent workers using `DynamicSupervisor`. 152 | - Pause job fetching when the system is under heavy load. 153 | 154 | ### Persistent Storage 155 | 156 | - Use `Ecto` or a simple `ETS` table to persist jobs and their statuses. 157 | - Store job metadata for debugging and monitoring. 158 | 159 | ### Resilience 160 | 161 | - Restart failed worker processes with `DynamicSupervisor`. 162 | - Retry failed jobs based on a backoff strategy (e.g., exponential backoff). 163 | 164 | ## Extra Features 165 | 166 | - Add priority levels to jobs (e.g., high, medium, low). 167 | - Implement job cancellation by ID. 168 | - Provide a web dashboard using Phoenix LiveView for monitoring and managing jobs. 169 | - Support cron-like periodic job scheduling. 170 | - Integrate Telemetry events for Prometheus or other monitoring tools. 171 | 172 | 173 | -------------------------------------------------------------------------------- /03-sorcerers-eyes/README.md: -------------------------------------------------------------------------------- 1 | # Sorcerer's Eye Monitoring 2 | 3 | ## Goal 4 | 5 | Create a distributed system monitoring tool called `sorcerers_eye` that collects and displays real-time metrics from nodes in a distributed system. This project showcases [GenServers](https://hexdocs.pm/elixir/GenServer.html), [Supervisors](https://hexdocs.pm/elixir/Supervisor.html), and Elixir's focus on fault tolerance and resilience. 6 | 7 | ```sh 8 | $ mix new sorcerers_eye --sup 9 | ``` 10 | 11 | ## Requirements 12 | 13 | The application should monitor connected nodes, gather system metrics (e.g., memory usage, CPU load, process counts), and display them in real-time via a simple CLI or web interface. 14 | 15 | ### Node Discovery 16 | 17 | Automatically discover nodes in a distributed system: 18 | - Use Elixir’s Node module to connect to other nodes. 19 | - Maintain a registry of connected nodes. 20 | 21 | ### Metrics Collection 22 | 23 | Gather system metrics from each node, including: 24 | - Memory usage 25 | - CPU load 26 | - Process counts 27 | - Uptime 28 | 29 | Use Erlang’s :os module and Elixir’s System module for metric collection. 30 | 31 | ### Real-Time Updates 32 | 33 | Ensure that metrics are updated in real-time: 34 | - Use a GenServer process to poll metrics at regular intervals. 35 | - Broadcast updates to the UI using Phoenix PubSub or a custom mechanism. 36 | 37 | ### Fault Tolerance 38 | 39 | Implement fault-tolerant processes: 40 | - Use Supervisors to restart GenServers that fail. 41 | - Gracefully handle node disconnections and reconnections. 42 | 43 | ### CLI or Web Interface 44 | 45 | Display metrics using: 46 | - A CLI-based interface that lists metrics for each node. 47 | - Alternatively, a Phoenix LiveView web interface for a more visual display. 48 | 49 | ## Example Features 50 | 51 | ### CLI Interface 52 | 53 | ``` 54 | $ sorcerers_eyes fetch 55 | Node: mystic-node-1 56 | Memory Usage: 512MB 57 | CPU Load: 20% 58 | Process Count: 100 59 | Uptime: 2 hours 60 | 61 | Node: mystic-node-2 62 | Memory Usage: 1GB 63 | CPU Load: 50% 64 | Process Count: 200 65 | Uptime: 5 hours 66 | ``` 67 | 68 | ### Web Interface 69 | 70 | Display metrics in a table with real-time updates: 71 | 72 | ``` 73 | Node Memory Usage CPU Load Process Count Uptime 74 | mystic-node-1 512MB 20% 100 2 hours 75 | mystic-node-2 1GB 50% 200 5 hours 76 | ``` 77 | 78 | ## Notable Packages Used 79 | 80 | - [GenServer](https://hexdocs.pm/elixir/GenServer.html) for managing metric collection processes. 81 | - [Supervisor](https://hexdocs.pm/elixir/Supervisor.html) for process supervision. 82 | - [libcluster](https://hexdocs.pm/libcluster/readme.html) for easy distributed node discory 83 | - [Phoenix PubSub](https://hexdocs.pm/phoenix_pub_sub) for real-time communication (optional for the web version). 84 | - [Nexus](https://https://hexdocs.pm/nexus_cli) as CLI framework (optional for the CLI version). 85 | 86 | ## Custom Resources 87 | 88 | ### Example Application 89 | 90 | You can find an example implementation of this monitoring tool in the [releases ]section. 91 | 92 | ### Example GenServer Implementation 93 | 94 | A simple GenServer for collecting metrics: 95 | 96 | ```ex 97 | defmodule SorcerersEye.MetricsCollector do 98 | use GenServer 99 | 100 | def start_link(node_name) do 101 | GenServer.start_link(__MODULE__, node_name, name: via_tuple(node_name)) 102 | end 103 | 104 | @impl true 105 | def init(node_name) do 106 | {:ok, %{node_name: node_name, metrics: %{}}, {:continue, :collect_metrics}} 107 | end 108 | 109 | @impl true 110 | def handle_continue(:collect_metrics, state) do 111 | schedule_metrics_collection() 112 | {:noreply, collect_metrics(state)} 113 | end 114 | 115 | @impl true 116 | def handle_info(:collect_metrics, state) do 117 | schedule_metrics_collection() 118 | {:noreply, collect_metrics(state)} 119 | end 120 | 121 | defp collect_metrics(state) do 122 | metrics = %{ 123 | memory: :os.system_info(:memory), 124 | cpu: :os.system_info(:cpu_load), 125 | process_count: length(Process.list()) 126 | } 127 | 128 | %{state | metrics: metrics} 129 | end 130 | 131 | defp schedule_metrics_collection do 132 | Process.send_after(self(), :collect_metrics, 5_000) 133 | end 134 | end 135 | ``` 136 | 137 | ### Example Supervisor Tree 138 | 139 | ```ex 140 | defmodule SorcerersEye.Application do 141 | use Application 142 | 143 | def start(_type, _args) do 144 | children = [ 145 | {DynamicSupervisor, strategy: :one_for_one, name: SorcerersEye.NodeSupervisor} 146 | ] 147 | 148 | opts = [strategy: :one_for_one, name: SorcerersEye.Supervisor] 149 | Supervisor.start_link(children, opts) 150 | end 151 | end 152 | ``` 153 | 154 | ## Technical Considerations 155 | 156 | ### Distributed Setup 157 | 158 | Run the application across multiple nodes: 159 | - Start nodes with the --name flag: `iex --sname mystic-node-1 -S mix`. 160 | - Use Node.connect/1 to establish connections between nodes. 161 | 162 | ### Resilience 163 | 164 | Ensure processes are supervised to handle failures gracefully: 165 | - Restart failed GenServers. 166 | - Handle node disconnections with retries or fallback behavior. 167 | 168 | ### Scalability 169 | 170 | For large systems, optimize by batching metric collection or reducing polling frequency. 171 | 172 | ## Extra Features 173 | 174 | - Display metrics in a dashboard with graphs (if using Phoenix LiveView). 175 | - Add alerts for high CPU or memory usage. 176 | - Allow filtering metrics by node or resource type. 177 | - Save historical metrics to a database for analysis. 178 | - Implement a CLI command to trigger garbage collection on remote nodes. 179 | - Implement a `observe` command for real-time tracking (if using CLI version) 180 | 181 | 182 | --------------------------------------------------------------------------------