├── .gitmodules ├── makefile ├── .gitignore └── README.md /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "logmink-hub"] 2 | path = logmink-hub 3 | url = https://github.com/mutairibassam/logmink-hub.git 4 | [submodule "logmink-agent"] 5 | path = logmink-agent 6 | url = https://github.com/mutairibassam/logmink-agent.git 7 | -------------------------------------------------------------------------------- /makefile: -------------------------------------------------------------------------------- 1 | init: 2 | # network name should match what it's specified in 3 | # the compose file for both [hub, and agent]. 4 | 5 | # the predefined network is logmink_network which can be changed. 6 | @if [ -z "$$(docker network ls --filter name=logmink_network -q)" ]; then \ 7 | echo "Creating Docker network 'logmink_network'..."; \ 8 | docker network create logmink_network; \ 9 | else \ 10 | echo "Docker network 'logmink_network' already exists."; \ 11 | fi 12 | 13 | # [todo] deploy the hub, and agent to a registry. 14 | docker compose -f logmink-hub/compose.yaml up -d 15 | docker build -t logmink/agent:0.9 ./logmink-agent -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Logs 2 | logs 3 | *.log 4 | npm-debug.log* 5 | yarn-debug.log* 6 | yarn-error.log* 7 | lerna-debug.log* 8 | .pnpm-debug.log* 9 | 10 | node_modules 11 | *.txt 12 | 13 | # Diagnostic reports (https://nodejs.org/api/report.html) 14 | report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json 15 | 16 | # Runtime data 17 | pids 18 | *.pid 19 | *.seed 20 | *.pid.lock 21 | 22 | # Directory for instrumented libs generated by jscoverage/JSCover 23 | lib-cov 24 | 25 | # Coverage directory used by tools like istanbul 26 | coverage 27 | *.lcov 28 | 29 | # nyc test coverage 30 | .nyc_output 31 | 32 | # Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files) 33 | .grunt 34 | 35 | # Bower dependency directory (https://bower.io/) 36 | bower_components 37 | 38 | # node-waf configuration 39 | .lock-wscript 40 | 41 | # Compiled binary addons (https://nodejs.org/api/addons.html) 42 | build/Release 43 | 44 | # Dependency directories 45 | node_modules/ 46 | jspm_packages/ 47 | 48 | # Snowpack dependency directory (https://snowpack.dev/) 49 | web_modules/ 50 | 51 | # TypeScript cache 52 | *.tsbuildinfo 53 | 54 | # Optional npm cache directory 55 | .npm 56 | 57 | # Optional eslint cache 58 | .eslintcache 59 | 60 | # Optional stylelint cache 61 | .stylelintcache 62 | 63 | # Microbundle cache 64 | .rpt2_cache/ 65 | .rts2_cache_cjs/ 66 | .rts2_cache_es/ 67 | .rts2_cache_umd/ 68 | 69 | # Optional REPL history 70 | .node_repl_history 71 | 72 | # Output of 'npm pack' 73 | *.tgz 74 | 75 | # Yarn Integrity file 76 | .yarn-integrity 77 | 78 | # dotenv environment variable files 79 | .env 80 | .env.development.local 81 | .env.test.local 82 | .env.production.local 83 | .env.local 84 | 85 | # parcel-bundler cache (https://parceljs.org/) 86 | .cache 87 | .parcel-cache 88 | 89 | # Next.js build output 90 | .next 91 | out 92 | 93 | # Nuxt.js build / generate output 94 | .nuxt 95 | dist 96 | 97 | # Gatsby files 98 | .cache/ 99 | # Comment in the public line in if your project uses Gatsby and not Next.js 100 | # https://nextjs.org/blog/next-9-1#public-directory-support 101 | # public 102 | 103 | # vuepress build output 104 | .vuepress/dist 105 | 106 | # vuepress v2.x temp and cache directory 107 | .temp 108 | .cache 109 | 110 | # Docusaurus cache and generated files 111 | .docusaurus 112 | 113 | # Serverless directories 114 | .serverless/ 115 | 116 | # FuseBox cache 117 | .fusebox/ 118 | 119 | # DynamoDB Local files 120 | .dynamodb/ 121 | 122 | # TernJS port file 123 | .tern-port 124 | 125 | # Stores VSCode versions used for testing VSCode extensions 126 | .vscode-test 127 | 128 | # yarn v2 129 | .yarn/cache 130 | .yarn/unplugged 131 | .yarn/build-state.yml 132 | .yarn/install-state.gz 133 | .pnp.* 134 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Logmink.hub 2 | 3 | **Logmink.hub** is a centralized logging hub designed to store all logs sent by agents deployed across containers. It aims to provide a seamless, "Plug-N-Play" experience for capturing HTTP logs from containers without requiring any code modifications. This project addresses the challenges of working with other log management tools like Logstash, Filebeat, Loki, Prometheus, Dynatrace, etc., by offering a straightforward setup. 4 | 5 | ### Logmink.hub stack: 6 | 1. **Node.js Server**: The core application that listens on the `/capture` endpoint for incoming log data. 7 | 2. **MongoDB**: A NoSQL database used to store all log data received by `Logmink.hub`. 8 | 3. **Mongo Express (Optional)**: A web-based UI for interacting with the MongoDB database, providing an easy way to view and manage logs. 9 | 10 | 11 | ## How It Works 12 | 1. **Setup**: Deploy Logmink.hub and agents within your containerized environment. 13 | 2. **Agents Capture Logs**: Agents are responsible for capturing all outgoing and incoming HTTP requests from the containers. These agents do not store any data locally. 14 | 3. **Data Sent to Logmink.hub**: Captured logs are sent to the `/capture` endpoint of Logmink.hub. 15 | 4. **Storage**: Logmink.hub stores the logs in a MongoDB database for future analysis and monitoring. 16 | 5. **Visualization**: Optionally, use Mongo Express to visualize and manage the stored logs. 17 | 18 | #### Initial setup: 19 | ```bash 20 | # clone project repo 21 | git clone --recurse-submodules https://github.com/mutairibassam/logmink.git 22 | cd logmink 23 | 24 | # pull latest changes from hub or agent repo (if needed) 25 | # git submodule update --remote 26 | 27 | # `make init` is going to build and run `Logmink.hub` container, then is going to only build the `Logmink-agent` image without running it since agents need to be attached to other containers as (sidecar) for listening. 28 | make init 29 | ``` 30 | #### Attach agent: 31 | 32 | Below compose configuration is a skelton for sidecar implementation for `Logmink.agent`. Below config should be copied and modified to another compose file and attach to a valid service. 33 | 34 | ```yml 35 | service_name: # recommended to start with agent ex; (agent.service_name) 36 | 37 | image: logmink/agent:0.9 # default image 38 | # build: ../http-agent # or local Dockerfile 39 | 40 | container_name: anyname_or_service_name 41 | network_mode: "service:{service_name}" # {service_name} should match the same service_name 42 | depends_on: 43 | - parent_service # agent should depends_on container that we want to capture its http traffic. 44 | 45 | environment: 46 | PORT: 32000 # or any available port 47 | LOGMINK_HUB_URL: http://service_name 48 | 49 | # database can be 1 huge database to store all logs 50 | # or can segregated per app based on need. 51 | 52 | # below example is one database for all dev env logs 53 | # ex; mongodb://mongo.logmink.hub:27017/logdb-dev 54 | 55 | # below example is one database for each application 56 | # ex; mongodb://mongo.logmink.hub:27017/crmlogs-prod 57 | # ex; mongodb://mongo.logmink.hub:27017/portallogs-test 58 | mongoUrl: mongodb://{service_name.hub}:27017/logdb-prod 59 | 60 | entrypoint: ["node", "agent.js"] 61 | 62 | ``` 63 | #### Details: 64 | *Note: You can modify `compose.yaml` file to change default password.* 65 | - mongo express (http://localhost:8081) 66 | - username (default): admin 67 | - password (default): pass 68 | 69 | - Logmink container: 70 | - service name (default): logmink.hub 71 | - service port (default): 32000 72 | 73 | Please contribute. 74 | --------------------------------------------------------------------------------