├── .gitignore ├── .readthedocs.yaml ├── LICENSE.md ├── Makefile ├── README.md ├── api.example.yml ├── checker.example.yml ├── docker-compose.nginx.conf ├── docker-compose.yml ├── filter.example.yml ├── notification-icons.zip ├── notifier.example.yml ├── requirements.txt └── source ├── _static ├── advanced_edit.png ├── advanced_trigger.png ├── advanced_trigger_example_1.png ├── advanced_trigger_example_2.png ├── advanced_trigger_example_3.png ├── advanced_trigger_example_4.png ├── channels.png ├── contact-events-button.png ├── contact-events-per-trigger.png ├── current_state.png ├── delete_metric.png ├── dfd-filter.svg ├── dfd-notifier-events.svg ├── dfd-notifier-notifications.svg ├── event_history.png ├── fancy-email-template.png ├── gsoc-grafana.png ├── gsoc-moira-delivery-channel.png ├── gsoc-moira-delivery-channels.png ├── gsoc-moira-screencast.gif ├── gsoc-retail-ui.png ├── helth-check-email.png ├── helth-check-webui.png ├── main_screen.png ├── maintenance.png ├── moving_average.png ├── nodata.png ├── notifier-toggle.png ├── reset_throttling.png ├── schedule_subscription.png ├── schedule_trigger.png ├── selfstate_full_cycle_WARN_to_ERROR.png ├── selfstate_full_cycle_WARN_to_OK.png ├── simple.png ├── slack-emoji.png ├── slack-state-emoji.png ├── subscription_plotting.png ├── subscription_tags.png ├── system_subscription_switch_select.png ├── system_subscription_tags.png ├── tags.png ├── throttling.png ├── trigger-transition-stats.png ├── triggers.png └── web-ui-example.png ├── conf.py ├── contacts.rst ├── development ├── architecture.rst ├── index.rst ├── notifier.rst └── ui.rst ├── gsoc.rst ├── index.rst ├── installation ├── configuration.rst ├── docker.rst ├── feeding.rst ├── index.rst ├── manual.rst ├── packages.rst ├── security.rst └── webhooks_scripts.rst ├── overview.rst ├── release_notes ├── 2_0.rst ├── 2_1.rst ├── 2_10_0.rst ├── 2_11_0.rst ├── 2_11_1.rst ├── 2_12_0.rst ├── 2_13_0.rst ├── 2_14_0.rst ├── 2_14_1.rst ├── 2_15_0.rst ├── 2_2.rst ├── 2_3.rst ├── 2_4_0.rst ├── 2_5_0.rst ├── 2_5_1.rst ├── 2_6_2.rst ├── 2_7_0.rst ├── 2_7_1.rst ├── 2_7_2.rst ├── 2_8_0.rst ├── 2_8_1.rst ├── 2_8_2.rst ├── 2_8_3.rst ├── 2_8_4.rst ├── 2_9_0.rst ├── 2_9_1.rst └── index.rst └── user_guide ├── advanced.rst ├── contact_events.rst ├── efficient.rst ├── hidden_pages.rst ├── index.rst ├── maintenance.rst ├── nodata.rst ├── schedule.rst ├── selfstate.rst ├── simple.rst ├── subscriptions.rst ├── throttling.rst └── trigger_page.rst /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | .DS_Store 3 | build 4 | source/_build_html 5 | source/_build 6 | .vscode 7 | .idea 8 | openapi/.hypothesis 9 | -------------------------------------------------------------------------------- /.readthedocs.yaml: -------------------------------------------------------------------------------- 1 | version: 2 2 | 3 | build: 4 | os: "ubuntu-22.04" 5 | tools: 6 | python: "3.8" 7 | 8 | sphinx: 9 | configuration: source/conf.py 10 | 11 | python: 12 | install: 13 | - requirements: ./requirements.txt 14 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 Moira 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = sphinx-build 7 | PAPER = 8 | BUILDDIR = build 9 | 10 | # User-friendly check for sphinx-build 11 | ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) 12 | $(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) 13 | endif 14 | 15 | # Internal variables. 16 | PAPEROPT_a4 = -D latex_paper_size=a4 17 | PAPEROPT_letter = -D latex_paper_size=letter 18 | ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source 19 | # the i18n builder cannot share the environment and doctrees with the others 20 | I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source 21 | 22 | .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext 23 | 24 | help: 25 | @echo "Please use \`make ' where is one of" 26 | @echo " html to make standalone HTML files" 27 | @echo " dirhtml to make HTML files named index.html in directories" 28 | @echo " singlehtml to make a single large HTML file" 29 | @echo " pickle to make pickle files" 30 | @echo " json to make JSON files" 31 | @echo " htmlhelp to make HTML files and a HTML help project" 32 | @echo " qthelp to make HTML files and a qthelp project" 33 | @echo " applehelp to make an Apple Help Book" 34 | @echo " devhelp to make HTML files and a Devhelp project" 35 | @echo " epub to make an epub" 36 | @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" 37 | @echo " latexpdf to make LaTeX files and run them through pdflatex" 38 | @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" 39 | @echo " text to make text files" 40 | @echo " man to make manual pages" 41 | @echo " texinfo to make Texinfo files" 42 | @echo " info to make Texinfo files and run them through makeinfo" 43 | @echo " gettext to make PO message catalogs" 44 | @echo " changes to make an overview of all changed/added/deprecated items" 45 | @echo " xml to make Docutils-native XML files" 46 | @echo " pseudoxml to make pseudoxml-XML files for display purposes" 47 | @echo " linkcheck to check all external links for integrity" 48 | @echo " doctest to run all doctests embedded in the documentation (if enabled)" 49 | @echo " coverage to run coverage check of the documentation (if enabled)" 50 | 51 | clean: 52 | rm -rf $(BUILDDIR)/* 53 | 54 | html: 55 | $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html 56 | @echo 57 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." 58 | 59 | dirhtml: 60 | $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml 61 | @echo 62 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." 63 | 64 | singlehtml: 65 | $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml 66 | @echo 67 | @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." 68 | 69 | pickle: 70 | $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle 71 | @echo 72 | @echo "Build finished; now you can process the pickle files." 73 | 74 | json: 75 | $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json 76 | @echo 77 | @echo "Build finished; now you can process the JSON files." 78 | 79 | htmlhelp: 80 | $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp 81 | @echo 82 | @echo "Build finished; now you can run HTML Help Workshop with the" \ 83 | ".hhp project file in $(BUILDDIR)/htmlhelp." 84 | 85 | qthelp: 86 | $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp 87 | @echo 88 | @echo "Build finished; now you can run "qcollectiongenerator" with the" \ 89 | ".qhcp project file in $(BUILDDIR)/qthelp, like this:" 90 | @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/moira-api.qhcp" 91 | @echo "To view the help file:" 92 | @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/moira-api.qhc" 93 | 94 | applehelp: 95 | $(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp 96 | @echo 97 | @echo "Build finished. The help book is in $(BUILDDIR)/applehelp." 98 | @echo "N.B. You won't be able to view it unless you put it in" \ 99 | "~/Library/Documentation/Help or install it in your application" \ 100 | "bundle." 101 | 102 | devhelp: 103 | $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp 104 | @echo 105 | @echo "Build finished." 106 | @echo "To view the help file:" 107 | @echo "# mkdir -p $$HOME/.local/share/devhelp/moira-api" 108 | @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/moira-api" 109 | @echo "# devhelp" 110 | 111 | epub: 112 | $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub 113 | @echo 114 | @echo "Build finished. The epub file is in $(BUILDDIR)/epub." 115 | 116 | latex: 117 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 118 | @echo 119 | @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." 120 | @echo "Run \`make' in that directory to run these through (pdf)latex" \ 121 | "(use \`make latexpdf' here to do that automatically)." 122 | 123 | latexpdf: 124 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 125 | @echo "Running LaTeX files through pdflatex..." 126 | $(MAKE) -C $(BUILDDIR)/latex all-pdf 127 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 128 | 129 | latexpdfja: 130 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 131 | @echo "Running LaTeX files through platex and dvipdfmx..." 132 | $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja 133 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 134 | 135 | text: 136 | $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text 137 | @echo 138 | @echo "Build finished. The text files are in $(BUILDDIR)/text." 139 | 140 | man: 141 | $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man 142 | @echo 143 | @echo "Build finished. The manual pages are in $(BUILDDIR)/man." 144 | 145 | texinfo: 146 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 147 | @echo 148 | @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." 149 | @echo "Run \`make' in that directory to run these through makeinfo" \ 150 | "(use \`make info' here to do that automatically)." 151 | 152 | info: 153 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 154 | @echo "Running Texinfo files through makeinfo..." 155 | make -C $(BUILDDIR)/texinfo info 156 | @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." 157 | 158 | gettext: 159 | $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale 160 | @echo 161 | @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." 162 | 163 | changes: 164 | $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes 165 | @echo 166 | @echo "The overview file is in $(BUILDDIR)/changes." 167 | 168 | linkcheck: 169 | $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck 170 | @echo 171 | @echo "Link check complete; look for any errors in the above output " \ 172 | "or in $(BUILDDIR)/linkcheck/output.txt." 173 | 174 | doctest: 175 | $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest 176 | @echo "Testing of doctests in the sources finished, look at the " \ 177 | "results in $(BUILDDIR)/doctest/output.txt." 178 | 179 | coverage: 180 | $(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage 181 | @echo "Testing of coverage in the sources finished, look at the " \ 182 | "results in $(BUILDDIR)/coverage/python.txt." 183 | 184 | xml: 185 | $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml 186 | @echo 187 | @echo "Build finished. The XML files are in $(BUILDDIR)/xml." 188 | 189 | pseudoxml: 190 | $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml 191 | @echo 192 | @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." 193 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Documentation 2 | 3 | [![Documentation Status](https://readthedocs.org/projects/moira/badge/?version=latest)](http://moira.readthedocs.org/en/latest/?badge=latest) 4 | 5 | If you're new here, better check out our main [README](https://github.com/moira-alert/moira/blob/master/README.md). 6 | 7 | Compiled version of this documentation is available on [Read the Docs](https://moira.readthedocs.io) site. 8 | 9 | ## Thanks 10 | 11 | ![SKB Kontur](https://kontur-inc.com/theme/ver-1633004215/images/logo/kontur-en-36.svg) 12 | 13 | Moira was originally developed and is supported by [SKB Kontur](https://kontur.ru/eng/about), a B2G company based in Ekaterinburg, Russia. We express gratitude to our company for encouraging us to opensource Moira and for giving back to the community that created [Graphite](https://graphite.readthedocs.io) and many other useful DevOps tools. 14 | -------------------------------------------------------------------------------- /checker.example.yml: -------------------------------------------------------------------------------- 1 | # Redis configuration depends on fields specified in redis config section: 2 | # 1. Use field `master_name` to enable Redis Sentinel support 3 | # 2. Specify two or more `addrs` to enable cluster support 4 | # 3. Otherwise, standalone configuration is enabled 5 | redis: 6 | # Sentinel master name 7 | master_name: "" 8 | # address list, format: {host1_name:port},{ip:port} 9 | addrs: "localhost:6379" 10 | # Redis username 11 | username: "username" 12 | # Redis password 13 | password: "password" 14 | # Redis Sentinel username 15 | sentinel_username: "sentinel_username" 16 | # Redis Sentinel password 17 | sentinel_password: "sentinel_password" 18 | # Moira will delete metrics older than this value from Redis. Large values will lead to various problems everywhere. 19 | # See https://github.com/moira-alert/moira/pull/519 20 | metrics_ttl: 3h 21 | # Dial timeout for establishing new connections 22 | # Default is 500 milliseconds 23 | dial_timeout: 1s 24 | # Timeout for socket reads. If reached, commands will fail 25 | # with a timeout instead of blocking. Default is 3 seconds. 26 | # Skip this setting or set 0 for default. 27 | read_timeout: 5s 28 | # Timeout for socket writes. If reached, commands will fail 29 | # with a timeout instead of blocking. Default is ReadTimeout. 30 | # Skip this setting or set 0 for default. 31 | write_timeout: 5s 32 | # Enables read-only commands on slave nodes 33 | # The flag does not work without `route_randomly` or `route_by_latency` set to true 34 | read_only: true 35 | # Allows routing read-only commands to the **random** master or slave node 36 | # It automatically enables ReadOnly 37 | route_randomly: true 38 | # Allows routing read-only commands to the **closest** master or slave node 39 | # It automatically enables ReadOnly 40 | route_by_latency: false 41 | # Minimum backoff between retries. Used to calculate exponential backoff. Default value is 0 42 | min_retry_backoff: 1s 43 | # Maximum backoff between retries. Used to calculate exponential backoff. Default value is 0 44 | max_retry_backoff: 10s 45 | # Max amount of attempts to find alive node (in Redis cluster) 46 | max_redirects: 4 47 | telemetry: 48 | # Common port for all telemetry data: Prometheus scraping, pprof, etc. 49 | listen: ":8091" 50 | pprof: 51 | # If true, pprof will be enabled on common telemetry port. 52 | enabled: false 53 | graphite: 54 | # If true, graphite sender will be enabled. 55 | enabled: true 56 | # If true, runtime stats will be captured and sent to graphite. Note: It takes to call stoptheworld() with configured "graphite.interval" to capture runtime stats (https://golang.org/src/runtime/mstats.go) 57 | runtime_stats: false 58 | # Graphite relay URI, format: ip:port 59 | uri: "graphite-relay:2003" 60 | # Moira metrics prefix. Use 'prefix: {hostname}' to use hostname autoresolver. 61 | prefix: DevOps.moira 62 | # Metrics sending interval 63 | interval: 60s 64 | checker: 65 | # Min period to perform triggers re-check. Note: Reducing of this value leads to increasing of CPU and memory usage values 66 | # Used to be `checker.check_interval` before v2.10 67 | metric_event_trigger_check_interval: 30s 68 | # In Moira 2.4 we add a new entity - Lazy Trigger. This is a regular trigger but without any subscription for it. 69 | # By default Moira treats any trigger equally regardless on its subscriptions number. 70 | # You can change this behaviour using option below. This can reduce CPU usage on your server. 71 | # Lazy triggers checker works if lazy_triggers_check_interval > check_interval. We recommend setting it to 10m. 72 | lazy_triggers_check_interval: 10m 73 | # Period to cancel forced checks for all triggers if no metrics were received (greater than 'local.check_interval') 74 | stop_checking_interval: 3600s 75 | # Time after which the check is considered critically slow. Such checks are logged for debug purposes 76 | # default: 1h 77 | critical_time_of_check: 10m 78 | local: 79 | # Period for every non-lazy trigger to perform a check on 80 | # Used to be `checker.nodata_check_interval` before v2.10 81 | check_interval: 60s 82 | # Equals to the number of processor cores found on Moira host by default or when variable is defined as 0. 83 | max_parallel_checks: 512 84 | # This section configures the list of graphite remote triggers sources. 85 | # See https://moira.readthedocs.io/en/latest/installation/configuration.html#graphite-remote-triggers-checker for further information 86 | # Used to be `remote` and contain only one source before v2.10 87 | graphite_remote: 88 | - # Unique cluster id (no other graphite_remote cluster should have the same one) 89 | # Use `default` for compatibility with old triggers without cluster_id 90 | # Should not be changed if there are any triggers using it 91 | cluster_id: default 92 | # Cluster name to be displayed in UI 93 | cluster_name: Graphite Remote 94 | # URL of Graphite HTTP API: graphite-web, carbonapi, etc. 95 | # Specify full URL including '/render' 96 | url: "http://graphite.example.com/render" 97 | # Auth username. Only Basic-auth supported 98 | user: graphite_admin 99 | # Auth password. Only Basic-auth supported 100 | password: verySecurePassword 101 | # Minimal period to perform triggers re-check. 102 | # Note: Reducing of this value leads to increasing of CPU and memory usage values and extra load on Graphite HTTP API 103 | check_interval: 60s 104 | # Don't fetch metrics older than this value from remote storage 105 | metrics_ttl: 168h 106 | # Maximum timeout for HTTP-request made to Graphite HTTP API 107 | timeout: 60s 108 | # Equals to the number of processor cores found on Moira host by default or when variable is defined as 0. 109 | max_parallel_checks: 0 110 | # From 2.14.0 111 | # Retries configuration for requests, that fetch data. 112 | # Library used for calculating exponential backoff retries: https://pkg.go.dev/github.com/cenkalti/backoff/v4 113 | retries: 114 | # Initial interval for retry attempt 115 | initial_interval: 60s 116 | # Used to calc interval between retries: RandomizedRetryInterval = RetryInterval * 117 | # * (random value from range [1 - randomization_factor, 1 + randomization_factor]) 118 | randomization_factor: 0.5 119 | # For calculating next: RetryInterval = RetryInterval * multiplier 120 | multiplier: 1.5 121 | # Caps RetryInterval (NOT RandomizedRetryInterval) 122 | max_interval: 120s 123 | # If already max_retries_count retries performed, stop retrying. 124 | # At least one of (max_retries_count, max_elapsed_time) should be specified 125 | max_retries_count: 3 126 | # If time passed since first try is more than max_elapsed_time than stop retrying 127 | # At least one of (max_retries_count, max_elapsed_time) should be specified 128 | max_elapsed_time: 360s 129 | # From 2.14.0 130 | # Maximum timeout for healthcheck requests 131 | heathcheck_timeout: 60s 132 | # From 2.14.0 133 | # Retries configuration for healthcheck requests (same fields as for retries) 134 | heathcheck_retries: 135 | # Initial interval for retry attempt 136 | initial_interval: 10s 137 | # Used to calc interval between retries: RandomizedRetryInterval = RetryInterval * 138 | # * (random value from range [1 - randomization_factor, 1 + randomization_factor]) 139 | randomization_factor: 0.5 140 | # For calculating next: RetryInterval = RetryInterval * multiplier 141 | multiplier: 1.5 142 | # Caps RetryInterval (NOT RandomizedRetryInterval) 143 | max_interval: 60s 144 | # If already max_retries_count retries performed, stop retrying. 145 | # At least one of (max_retries_count, max_elapsed_time) should be specified 146 | max_retries_count: 3 147 | # If time passed since first try is more than max_elapsed_time than stop retrying 148 | # At least one of (max_retries_count, max_elapsed_time) should be specified 149 | max_elapsed_time: 200s 150 | # This section configures the list of prometheus remote triggers sources. 151 | # See https://moira.readthedocs.io/en/latest/installation/configuration.html#prometheus-remote-triggers-checker for further information 152 | # Used to be `prometheus` and contain only one source before v2.10 153 | prometheus_remote: 154 | - # Unique cluster id (no other prometheus_remote cluster should have the same one) 155 | # Use `default` for compatibility with old triggers without cluster_id 156 | # Should not be changed if there are any triggers using it 157 | cluster_id: default 158 | # Cluster name to be displayed in UI 159 | cluster_name: Prometheus Remote 160 | # URL of Prometheus HTTP API: Prometheus or VMSelect 161 | # Only domain name must be specified, no URL-path 162 | url: https://prometheus.example.com 163 | # Auth username. Only Basic-auth supported 164 | user: prometheus_admin 165 | # Auth password. Only Basic-auth supported 166 | password: verySecurePassword 167 | # Minimal period to perform triggers re-check. 168 | # Note: Reducing of this value leads to increasing of CPU and memory usage values and extra load on Prometheus HTTP API 169 | check_interval: "60s" 170 | # Maximum timeout for HTTP-request made to Prometheus HTTP API 171 | timeout: "10s" 172 | # Number of times failed request should be retried 173 | retries: 3 174 | # Delay between retries 175 | retry_timeout: "3s" 176 | # Don't fetch metrics older than this value from remote storage 177 | metrics_ttl: "168h" 178 | # Equals to the number of processor cores found on Moira host by default or when variable is defined as 0. 179 | max_parallel_checks: 0 180 | log: 181 | log_file: stdout 182 | log_level: info 183 | -------------------------------------------------------------------------------- /docker-compose.nginx.conf: -------------------------------------------------------------------------------- 1 | server { 2 | listen 0.0.0.0:8083 default; 3 | 4 | location /api/ { 5 | proxy_pass http://api:8081; 6 | } 7 | 8 | location / { 9 | proxy_pass http://web:80; 10 | } 11 | } 12 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | 3 | services: 4 | redis: 5 | image: redis:alpine 6 | 7 | filter: 8 | image: moira/filter 9 | ports: 10 | - "2003:2003" 11 | depends_on: 12 | - redis 13 | links: 14 | - redis:localhost 15 | 16 | checker: 17 | image: moira/checker 18 | depends_on: 19 | - redis 20 | - filter 21 | links: 22 | - redis:localhost 23 | 24 | notifier: 25 | image: moira/notifier 26 | depends_on: 27 | - redis 28 | - checker 29 | links: 30 | - redis:localhost 31 | 32 | api: 33 | image: moira/api 34 | depends_on: 35 | - redis 36 | - checker 37 | links: 38 | - redis:localhost 39 | networks: 40 | - default 41 | - balancer 42 | 43 | web: 44 | image: moira/web2 45 | networks: 46 | - balancer 47 | 48 | balancer: 49 | image: nginx:alpine 50 | ports: 51 | - "8080:8083" 52 | depends_on: 53 | - web 54 | - api 55 | networks: 56 | - balancer 57 | volumes: 58 | - ./docker-compose.nginx.conf:/etc/nginx/conf.d/moira.conf 59 | 60 | networks: 61 | balancer: 62 | -------------------------------------------------------------------------------- /filter.example.yml: -------------------------------------------------------------------------------- 1 | # Redis configuration depends on fields specified in redis config section: 2 | # 1. Use field `master_name` to enable Redis Sentinel support 3 | # 2. Specify two or more `addrs` to enable cluster support 4 | # 3. Otherwise, standalone configuration is enabled 5 | redis: 6 | # Sentinel master name 7 | master_name: "" 8 | # address list, format: {host1_name:port},{ip:port} 9 | addrs: "localhost:6379" 10 | # Redis username 11 | username: "username" 12 | # Redis password 13 | password: "password" 14 | # Redis Sentinel username 15 | sentinel_username: "sentinel_username" 16 | # Redis Sentinel password 17 | sentinel_password: "sentinel_password" 18 | # Moira will delete metrics older than this value from Redis. Large values will lead to various problems everywhere. 19 | # See https://github.com/moira-alert/moira/pull/519 20 | metrics_ttl: 3h 21 | # Dial timeout for establishing new connections 22 | # Default is 500 milliseconds 23 | dial_timeout: 1s 24 | # Timeout for socket reads. If reached, commands will fail 25 | # with a timeout instead of blocking. Default is 3 seconds. 26 | # Skip this setting or set 0 for default. 27 | read_timeout: 5s 28 | # Timeout for socket writes. If reached, commands will fail 29 | # with a timeout instead of blocking. Default is ReadTimeout. 30 | # Skip this setting or set 0 for default. 31 | write_timeout: 5s 32 | # Enables read-only commands on slave nodes 33 | # The flag does not work without `route_randomly` or `route_by_latency` set to true 34 | read_only: true 35 | # Allows routing read-only commands to the **random** master or slave node 36 | # It automatically enables ReadOnly 37 | route_randomly: true 38 | # Allows routing read-only commands to the **closest** master or slave node 39 | # It automatically enables ReadOnly 40 | route_by_latency: false 41 | # Minimum backoff between retries. Used to calculate exponential backoff. Default value is 0 42 | min_retry_backoff: 1s 43 | # Maximum backoff between retries. Used to calculate exponential backoff. Default value is 0 44 | max_retry_backoff: 10s 45 | # Max amount of attempts to find alive node (in Redis cluster) 46 | max_redirects: 4 47 | telemetry: 48 | # Common port for all telemetry data: Prometheus scraping, pprof, etc. 49 | listen: ":8091" 50 | pprof: 51 | # If true, pprof will be enabled on common telemetry port. 52 | enabled: false 53 | graphite: 54 | # If true, graphite sender will be enabled. 55 | enabled: true 56 | # If true, runtime stats will be captured and sent to graphite. Note: It takes to call stoptheworld() with configured "graphite.interval" to capture runtime stats (https://golang.org/src/runtime/mstats.go) 57 | runtime_stats: false 58 | # Graphite relay URI, format: ip:port 59 | uri: "graphite-relay:2003" 60 | # Moira metrics prefix. Use 'prefix: {hostname}' to use hostname autoresolver. 61 | prefix: DevOps.moira 62 | # Metrics sending interval 63 | interval: 60s 64 | filter: 65 | # Metrics listener uri 66 | listen: ":2003" 67 | # Retentions config file path. Simply use your original storage-schemas.conf or create new if you're using Moira without existing Graphite installation. 68 | retention_config: /etc/moira/storage-schemas.conf 69 | # Number of metrics to cache before checking them. 70 | # Note: As this value increases, Redis CPU usage decreases. 71 | # Normally, this value must be an order of magnitude less than graphite.prefix.filter.recevied.matching.count | nonNegativeDerivative() | scaleToSeconds(1) 72 | # For example: with 100 matching metrics, set cache_capacity to 10. With 1000 matching metrics, increase cache_capacity up to 100. 73 | cache_capacity: 10 74 | # Defines number of threads to match incoming graphite-metrics. 75 | # Equals to the number of processor cores found on Moira host by default or when variable is defined as 0. 76 | max_parallel_matches: 0 77 | # Period in which patterns will be reloaded from Redis 78 | patterns_update_period: 1s 79 | # Compatibility with different versions of graphite supported (carbon and clickhouse) 80 | # By default moira is compatible with carbon, but can be configured to be compatible with clickhouse 81 | graphite_compatibility: 82 | # Controls how regices in tag matching are treated 83 | # If false (default value), regex will match start of the string strictly. 'tag~=foo' is equivalent to 'tag~=^foo.*' 84 | # If true, regex will match start of the string loosely. 'tag~=foo' is equivalent to 'tag~=.*foo.*' 85 | allow_regex_loose_start_match: false 86 | # Controls how absent tags are treated 87 | # If true (default value), empty tags in regices will be matched 88 | # If false, empty tags will be discarded 89 | allow_regex_match_empty: true 90 | # Time after which the batch of metrics is forced to be saved, default is 1s 91 | batch_forced_save_timeout: 1s 92 | # Defines the configuration for pattern storage 93 | pattern_storage: 94 | # Determines the size of the pattern matching cache 95 | # It is better to put no less than the number of triggers 96 | pattern_matching_cache_size: 1000 97 | log: 98 | log_file: stdout 99 | log_level: info 100 | -------------------------------------------------------------------------------- /notification-icons.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/notification-icons.zip -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | Pillow 2 | Sphinx==5.0.0 3 | sphinx-rtd-theme 4 | sphinxcontrib-httpdomain 5 | sphinxcontrib-images 6 | pyenchant 7 | recommonmark 8 | txredisapi 9 | mkdocs==1.2.3 10 | jinja2==3.0.0 11 | -------------------------------------------------------------------------------- /source/_static/advanced_edit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/advanced_edit.png -------------------------------------------------------------------------------- /source/_static/advanced_trigger.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/advanced_trigger.png -------------------------------------------------------------------------------- /source/_static/advanced_trigger_example_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/advanced_trigger_example_1.png -------------------------------------------------------------------------------- /source/_static/advanced_trigger_example_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/advanced_trigger_example_2.png -------------------------------------------------------------------------------- /source/_static/advanced_trigger_example_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/advanced_trigger_example_3.png -------------------------------------------------------------------------------- /source/_static/advanced_trigger_example_4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/advanced_trigger_example_4.png -------------------------------------------------------------------------------- /source/_static/channels.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/channels.png -------------------------------------------------------------------------------- /source/_static/contact-events-button.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/contact-events-button.png -------------------------------------------------------------------------------- /source/_static/contact-events-per-trigger.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/contact-events-per-trigger.png -------------------------------------------------------------------------------- /source/_static/current_state.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/current_state.png -------------------------------------------------------------------------------- /source/_static/delete_metric.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/delete_metric.png -------------------------------------------------------------------------------- /source/_static/event_history.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/event_history.png -------------------------------------------------------------------------------- /source/_static/fancy-email-template.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/fancy-email-template.png -------------------------------------------------------------------------------- /source/_static/gsoc-grafana.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/gsoc-grafana.png -------------------------------------------------------------------------------- /source/_static/gsoc-moira-delivery-channel.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/gsoc-moira-delivery-channel.png -------------------------------------------------------------------------------- /source/_static/gsoc-moira-delivery-channels.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/gsoc-moira-delivery-channels.png -------------------------------------------------------------------------------- /source/_static/gsoc-moira-screencast.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/gsoc-moira-screencast.gif -------------------------------------------------------------------------------- /source/_static/gsoc-retail-ui.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/gsoc-retail-ui.png -------------------------------------------------------------------------------- /source/_static/helth-check-email.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/helth-check-email.png -------------------------------------------------------------------------------- /source/_static/helth-check-webui.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/helth-check-webui.png -------------------------------------------------------------------------------- /source/_static/main_screen.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/main_screen.png -------------------------------------------------------------------------------- /source/_static/maintenance.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/maintenance.png -------------------------------------------------------------------------------- /source/_static/moving_average.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/moving_average.png -------------------------------------------------------------------------------- /source/_static/nodata.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/nodata.png -------------------------------------------------------------------------------- /source/_static/notifier-toggle.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/notifier-toggle.png -------------------------------------------------------------------------------- /source/_static/reset_throttling.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/reset_throttling.png -------------------------------------------------------------------------------- /source/_static/schedule_subscription.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/schedule_subscription.png -------------------------------------------------------------------------------- /source/_static/schedule_trigger.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/schedule_trigger.png -------------------------------------------------------------------------------- /source/_static/selfstate_full_cycle_WARN_to_ERROR.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/selfstate_full_cycle_WARN_to_ERROR.png -------------------------------------------------------------------------------- /source/_static/selfstate_full_cycle_WARN_to_OK.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/selfstate_full_cycle_WARN_to_OK.png -------------------------------------------------------------------------------- /source/_static/simple.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/simple.png -------------------------------------------------------------------------------- /source/_static/slack-emoji.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/slack-emoji.png -------------------------------------------------------------------------------- /source/_static/slack-state-emoji.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/slack-state-emoji.png -------------------------------------------------------------------------------- /source/_static/subscription_plotting.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/subscription_plotting.png -------------------------------------------------------------------------------- /source/_static/subscription_tags.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/subscription_tags.png -------------------------------------------------------------------------------- /source/_static/system_subscription_switch_select.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/system_subscription_switch_select.png -------------------------------------------------------------------------------- /source/_static/system_subscription_tags.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/system_subscription_tags.png -------------------------------------------------------------------------------- /source/_static/tags.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/tags.png -------------------------------------------------------------------------------- /source/_static/throttling.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/throttling.png -------------------------------------------------------------------------------- /source/_static/trigger-transition-stats.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/trigger-transition-stats.png -------------------------------------------------------------------------------- /source/_static/triggers.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/triggers.png -------------------------------------------------------------------------------- /source/_static/web-ui-example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/moira-alert/doc/0cf502100e0c51c71bb9ed70fd564f909b119b8b/source/_static/web-ui-example.png -------------------------------------------------------------------------------- /source/conf.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # moira-api documentation build configuration file, created by 4 | # sphinx-quickstart on Mon Jun 29 12:49:57 2015. 5 | # 6 | # This file is execfile()d with the current directory set to its 7 | # containing dir. 8 | # 9 | # Note that not all possible configuration values are present in this 10 | # autogenerated file. 11 | # 12 | # All configuration values have a default; values that are commented out 13 | # serve to show the default. 14 | 15 | import sys 16 | import os 17 | import shlex 18 | import sphinx_rtd_theme 19 | from recommonmark.parser import CommonMarkParser 20 | 21 | from docutils.parsers.rst.directives.admonitions import BaseAdmonition 22 | from sphinx.util import compat 23 | compat.make_admonition = BaseAdmonition 24 | 25 | # If extensions (or modules to document with autodoc) are in another directory, 26 | # add these directories to sys.path here. If the directory is relative to the 27 | # documentation root, use os.path.abspath to make it absolute, like shown here. 28 | # sys.path.insert(0, os.path.abspath('../../worker/bin')) 29 | 30 | # -- General configuration ------------------------------------------------ 31 | 32 | # If your documentation needs a minimal Sphinx version, state it here. 33 | #needs_sphinx = '1.0' 34 | 35 | # Add any Sphinx extension module names here, as strings. They can be 36 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom 37 | # ones. 38 | 39 | extensions = [ 40 | 'sphinx.ext.autodoc', 'sphinxcontrib.httpdomain', 'sphinxcontrib.images' 41 | ] 42 | 43 | images_config = { 44 | 'override_image_directive': True 45 | } 46 | 47 | autodoc_docstring_signature = True 48 | 49 | # Add any paths that contain templates here, relative to this directory. 50 | templates_path = ['_templates'] 51 | 52 | # The suffix(es) of source filenames. 53 | # You can specify multiple suffix as a list of string: 54 | # source_suffix = ['.rst', '.md'] 55 | source_suffix = ['.rst', '.md'] 56 | 57 | source_parsers = { 58 | '.md': CommonMarkParser, 59 | } 60 | 61 | # The encoding of source files. 62 | #source_encoding = 'utf-8-sig' 63 | 64 | # The master toctree document. 65 | master_doc = 'index' 66 | 67 | # General information about the project. 68 | project = u'Moira' 69 | copyright = u'2025, SKB Kontur' 70 | author = u'SKB Kontur' 71 | 72 | # The version info for the project you're documenting, acts as replacement for 73 | # |version| and |release|, also used in various other places throughout the 74 | # built documents. 75 | # 76 | # The short X.Y version. 77 | version = '2.15' 78 | # The full version, including alpha/beta/rc tags. 79 | release = '2.15.0' 80 | 81 | # The language for content autogenerated by Sphinx. Refer to documentation 82 | # for a list of supported languages. 83 | # 84 | # This is also used if you do content translation via gettext catalogs. 85 | # Usually you set "language" from the command line for these cases. 86 | language = 'en' 87 | 88 | # There are two options for replacing |today|: either, you set today to some 89 | # non-false value, then it is used: 90 | #today = '' 91 | # Else, today_fmt is used as the format for a strftime call. 92 | #today_fmt = '%B %d, %Y' 93 | 94 | # List of patterns, relative to source directory, that match files and 95 | # directories to ignore when looking for source files. 96 | exclude_patterns = ['api/*'] 97 | 98 | # The reST default role (used for this markup: `text`) to use for all 99 | # documents. 100 | #default_role = None 101 | 102 | # If true, '()' will be appended to :func: etc. cross-reference text. 103 | #add_function_parentheses = True 104 | 105 | # If true, the current module name will be prepended to all description 106 | # unit titles (such as .. function::). 107 | #add_module_names = True 108 | 109 | # If true, sectionauthor and moduleauthor directives will be shown in the 110 | # output. They are ignored by default. 111 | #show_authors = False 112 | 113 | # The name of the Pygments (syntax highlighting) style to use. 114 | pygments_style = 'sphinx' 115 | 116 | # A list of ignored prefixes for module index sorting. 117 | #modindex_common_prefix = [] 118 | 119 | # If true, keep warnings as "system message" paragraphs in the built documents. 120 | #keep_warnings = False 121 | 122 | # If true, `todo` and `todoList` produce output, else they produce nothing. 123 | todo_include_todos = False 124 | 125 | 126 | # -- Options for HTML output ---------------------------------------------- 127 | 128 | # The theme to use for HTML and HTML Help pages. See the documentation for 129 | # a list of builtin themes. 130 | html_theme = 'sphinx_rtd_theme' 131 | 132 | # Theme options are theme-specific and customize the look and feel of a theme 133 | # further. For a list of options available for each theme, see the 134 | # documentation. 135 | #html_theme_options = {} 136 | 137 | # Add any paths that contain custom themes here, relative to this directory. 138 | html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] 139 | 140 | # The name for this set of Sphinx documents. If None, it defaults to 141 | # " v documentation". 142 | #html_title = None 143 | 144 | # A shorter title for the navigation bar. Default is the same as html_title. 145 | #html_short_title = None 146 | 147 | # The name of an image file (relative to this directory) to place at the top 148 | # of the sidebar. 149 | #html_logo = None 150 | 151 | # The name of an image file (within the static path) to use as favicon of the 152 | # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 153 | # pixels large. 154 | #html_favicon = None 155 | 156 | # Add any paths that contain custom static files (such as style sheets) here, 157 | # relative to this directory. They are copied after the builtin static files, 158 | # so a file named "default.css" will overwrite the builtin "default.css". 159 | html_static_path = ['_static'] 160 | 161 | # Add any extra paths that contain custom files (such as robots.txt or 162 | # .htaccess) here, relative to this directory. These files are copied 163 | # directly to the root of the documentation. 164 | #html_extra_path = [] 165 | 166 | # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, 167 | # using the given strftime format. 168 | #html_last_updated_fmt = '%b %d, %Y' 169 | 170 | # If true, SmartyPants will be used to convert quotes and dashes to 171 | # typographically correct entities. 172 | #html_use_smartypants = True 173 | 174 | # Custom sidebar templates, maps document names to template names. 175 | #html_sidebars = {} 176 | 177 | # Additional templates that should be rendered to pages, maps page names to 178 | # template names. 179 | #html_additional_pages = {} 180 | 181 | # If false, no module index is generated. 182 | #html_domain_indices = True 183 | 184 | # If false, no index is generated. 185 | #html_use_index = True 186 | 187 | # If true, the index is split into individual pages for each letter. 188 | #html_split_index = False 189 | 190 | # If true, links to the reST sources are added to the pages. 191 | #html_show_sourcelink = True 192 | 193 | # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. 194 | #html_show_sphinx = True 195 | 196 | # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. 197 | #html_show_copyright = True 198 | 199 | # If true, an OpenSearch description file will be output, and all pages will 200 | # contain a tag referring to it. The value of this option must be the 201 | # base URL from which the finished HTML is served. 202 | #html_use_opensearch = '' 203 | 204 | # This is the file name suffix for HTML files (e.g. ".xhtml"). 205 | #html_file_suffix = None 206 | 207 | # Language to be used for generating the HTML full-text search index. 208 | # Sphinx supports the following languages: 209 | # 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja' 210 | # 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr' 211 | #html_search_language = 'en' 212 | 213 | # A dictionary with options for the search language support, empty by default. 214 | # Now only 'ja' uses this config value 215 | #html_search_options = {'type': 'default'} 216 | 217 | # The name of a javascript file (relative to the configuration directory) that 218 | # implements a search results scorer. If empty, the default will be used. 219 | #html_search_scorer = 'scorer.js' 220 | 221 | # Output file base name for HTML help builder. 222 | htmlhelp_basename = 'moira-doc' 223 | 224 | # The name of an image file (relative to this directory) to place at the top of 225 | # the title page. 226 | #latex_logo = None 227 | 228 | # For "manual" documents, if this is true, then toplevel headings are parts, 229 | # not chapters. 230 | #latex_use_parts = False 231 | 232 | # If true, show page references after internal links. 233 | #latex_show_pagerefs = False 234 | 235 | # If true, show URL addresses after external links. 236 | #latex_show_urls = False 237 | 238 | # Documents to append as an appendix to all manuals. 239 | #latex_appendices = [] 240 | 241 | # If false, no module index is generated. 242 | #latex_domain_indices = True 243 | 244 | # Documents to append as an appendix to all manuals. 245 | #texinfo_appendices = [] 246 | 247 | # If false, no module index is generated. 248 | #texinfo_domain_indices = True 249 | 250 | # How to display URL addresses: 'footnote', 'no', or 'inline'. 251 | #texinfo_show_urls = 'footnote' 252 | 253 | # If true, do not generate a @detailmenu in the "Top" node's menu. 254 | #texinfo_no_detailmenu = False 255 | -------------------------------------------------------------------------------- /source/contacts.rst: -------------------------------------------------------------------------------- 1 | Contact Moira Developers 2 | ======================== 3 | 4 | .. _Telegram: https://t.me/moira_alert 5 | 6 | The best way to contact us is to visit our Telegram_ chat. 7 | We usually reply within a day, but sometimes immediately :) 8 | -------------------------------------------------------------------------------- /source/development/architecture.rst: -------------------------------------------------------------------------------- 1 | Architecture 2 | ============ 3 | 4 | .. _target: http://graphite.readthedocs.io/en/latest/render_api.html#target 5 | 6 | .. raw:: html 7 | 8 | 42 | 43 | 44 | 45 | Terminology 46 | ----------- 47 | 48 | Pattern 49 | ^^^^^^^ 50 | 51 | A Graphite pattern is a single dot-separated metric name, 52 | possibly containing one or more wildcards. 53 | 54 | Examples: 55 | 56 | .. code-block:: text 57 | 58 | server.web*.load 59 | server.web{1,2,3}.load 60 | server.web1.load 61 | 62 | 63 | Target 64 | ^^^^^^ 65 | 66 | A Graphite target_ is one or more patterns, 67 | possibly combined using Graphite functions. 68 | 69 | Examples: 70 | 71 | .. code-block:: text 72 | 73 | averageSeries(server.web*.load) 74 | 75 | 76 | Metric 77 | ^^^^^^ 78 | 79 | A metric is a single time-series that is a result 80 | of parsing some Graphite target. 81 | 82 | Some targets produce a single metric, for example: 83 | 84 | .. code-block:: text 85 | 86 | server.web1.load 87 | highestCurrent(server.web*.load) 88 | 89 | Some targets produce several metrics, for example: 90 | 91 | .. code-block:: text 92 | 93 | movingAverage(server.web*.load, 10) 94 | 95 | State 96 | ^^^^^ 97 | 98 | Moira stores separate state for every metric. Each metric 99 | can be in only one state at any moment: 100 | 101 | .. raw:: html 102 | 103 |
104 | 105 | OK 106 |
107 | 108 |
109 | 110 | WARN 111 |
112 | 113 |
114 | 115 | ERROR 116 |
117 | 118 |
119 | 120 | NODATA 121 |
122 | 123 |
124 | 125 | EXCEPTION 126 |
127 | 128 |
 
129 | 130 | 131 | Trigger 132 | ^^^^^^^ 133 | 134 | Trigger is a configuration that tells Moira which metrics to watch for. 135 | Triggers consist of: 136 | 137 | - Name. This is just for convenience, user can enter anything here. 138 | - Description. Longer text that gets included in notification 139 | to delivery channels that support long texts. 140 | - One or more targets. 141 | - WARN and ERROR value limits, or a Python expression to calculate state. 142 | - One or more tags. 143 | - TTL value. When new data doesn't arrive for TTL seconds, the metric will switch to the `State`_ set by the user. 144 | - Check schedule. For example, a trigger can be set 145 | to check only during business hours. 146 | 147 | 148 | Last Check 149 | ^^^^^^^^^^ 150 | 151 | When Moira checks a trigger, it stores the following 152 | information on each metric: 153 | 154 | - Current value. 155 | - Current timestamp. 156 | - Current state. 157 | 158 | 159 | Trigger Event 160 | ^^^^^^^^^^^^^ 161 | 162 | When Moira checks a trigger, if any of the metric 163 | states change, Moira generates an event. Events consist of: 164 | 165 | - Trigger ID. 166 | - Metric name (as given by parsed target). 167 | - New state. 168 | - Previous state. 169 | - Current timestamp. 170 | 171 | 172 | Tags 173 | ^^^^ 174 | 175 | Tags are simple string markers for grouping of triggers 176 | and configuring subscriptions. 177 | 178 | 179 | Subscription 180 | ^^^^^^^^^^^^ 181 | 182 | Moira generates notifications for an event only if trigger tags match any 183 | of the user-created subscriptions. Each subscription consists of: 184 | 185 | - One or more tags. 186 | - Contact information. 187 | - Quiet time schedule. 188 | 189 | 190 | Dataflow 191 | -------- 192 | 193 | Filter and Check Incoming Metrics 194 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 195 | 196 | .. image:: ../_static/dfd-filter.svg 197 | :alt: filter and checker 198 | :width: 70% 199 | :align: center 200 | 201 | When user adds a new trigger, Moira parses patterns from targets and saves them 202 | to ``moira-pattern-list`` key in Redis. Filter rereads this list every second. 203 | When a metric value arrives, Filter checks metric name against the list 204 | of patterns. Matching metrics are saved to ``moira-metric:`` keys 205 | in Redis. Redis pub/sub mechanism is used to inform Checker of incoming metric 206 | value that should be checked as soon as possible. 207 | 208 | Checker metrics handler reads triggers by pattern from 209 | ``moira-pattern-triggers:`` and add ``trigger_id`` to Redis set 210 | ``moira-triggers-to-check``. NODATA Checker adds all triggers to Redis set 211 | ``moira-triggers-to-check`` once per ``nodata_check_interval`` setting. 212 | 213 | :ref:`graphite-remote-triggers-checker` gets all graphite remote trigger IDs and adds it to 214 | Redis set ``moira-remote-triggers-to-check`` once per ``remote\check_interval`` 215 | setting. 216 | 217 | :ref:`prometheus-remote-triggers-checker` gets all prometheus remote trigger IDs and adds it to 218 | Redis set ``moira-prometheus-triggers-to-check`` once per ``prometheus\check_interval`` 219 | setting. 220 | 221 | Checker pops ``trigger_id`` from ``moira-triggers-to-check`` and starts 222 | checking procedure. 223 | :ref:`graphite-remote-triggers-checker` does the same, but pops 224 | ``trigger_id`` from ``moira-remote-triggers-to-check`` and starts remote check, 225 | which involve remote Graphite HTTP API. 226 | Same for :ref:`prometheus-remote-triggers-checker`, which pops 227 | ``trigger_id`` from ``moira-prometheus-triggers-to-check`` and starts remote check, 228 | which involve remote Prometheus HTTP API. 229 | 230 | Trigger target can contain one or multiple metrics, so results are written 231 | per metric. ``moira-metric-last-check:`` Redis key contains last 232 | check JSON with metric states. 233 | 234 | When a metric changes its state, a new event is written to 235 | ``moira-trigger-events`` Redis key. This happens only if value timestamp 236 | falls inside time period allowed by trigger schedule. 237 | 238 | If a metric has been in NODATA or ERROR state for a long period, every 24 hours 239 | Moira will issue an additional reminder event. 240 | 241 | Trigger switches to EXCEPTION state, if any exception occurs during 242 | trigger checking. 243 | 244 | 245 | Process Trigger Events 246 | ^^^^^^^^^^^^^^^^^^^^^^ 247 | 248 | .. image:: ../_static/dfd-notifier-events.svg 249 | :alt: notifier events 250 | :width: 70% 251 | :align: center 252 | 253 | Notifier constantly pulls new events from ``moira-trigger-events`` 254 | Redis key and schedules notifications according to subscription schedule 255 | and throttling rules. If a trigger contains *all* of the tags in 256 | a subscription, and only in this case, a notification is created for 257 | this subscription. 258 | 259 | Subscription schedule delays notifications of occurred event to the 260 | beginning of next allowed time interval. Note that this differs from 261 | trigger schedule, which suppresses event generation entirely. 262 | 263 | Throttling rules will delay notifications: 264 | 265 | - If there are more than 10 events per hour, a notification will be sent 266 | at most once per 30 minutes. 267 | - If there are more than 20 events per 3 hours, a notification will be sent 268 | at most once per hour. 269 | 270 | Scheduled notifications are written to ``moira-notifier-notifications`` 271 | Redis key. 272 | 273 | 274 | Process Notifications 275 | ^^^^^^^^^^^^^^^^^^^^^ 276 | 277 | .. image:: ../_static/dfd-notifier-notifications.svg 278 | :alt: notifier notifications 279 | 280 | Notifier constantly pulls scheduled notifications from 281 | ``moira-notifier-notifications`` Redis key. It calls sender for certain contact 282 | type and writes notification back to Redis in case of sender error. 283 | -------------------------------------------------------------------------------- /source/development/index.rst: -------------------------------------------------------------------------------- 1 | Development 2 | =========== 3 | 4 | All services use Redis database to store and exchange data. 5 | Therefore, it is important to maintain an accurate description 6 | of data storage formats and conventions. 7 | 8 | Following topics describe database structure, running tests, 9 | developing notification plugins, etc. 10 | 11 | .. toctree:: 12 | :maxdepth: 1 13 | 14 | architecture 15 | ui 16 | notifier 17 | -------------------------------------------------------------------------------- /source/development/notifier.rst: -------------------------------------------------------------------------------- 1 | Backend 2 | ======= 3 | 4 | .. _Go: https://golang.org 5 | .. _GoConvey: http://goconvey.co 6 | 7 | Project Setup 8 | ------------- 9 | Backend microservices are written in Go_ (with module support, minimal Go lang required is 1.11), this is how you can get started writing your own: 10 | 11 | 1. Create a fork of the project from GitHub 12 | 2. Create a base directory to check out the project into e.g. 13 | .. code-block:: bash 14 | 15 | /Development/go/moira/ 16 | 3. Set your GOPATH to that directory and change into it 17 | .. code-block:: bash 18 | 19 | export GOPATH=/Development/go/moira/ 20 | cd $GOPATH 21 | 4. Get GoConvey_ for tests 22 | .. code-block:: bash 23 | 24 | go get github.com/smartystreets/goconvey 25 | 5. Export your ``GOPATH``'s ``bin`` directory 26 | .. code-block:: bash 27 | 28 | export PATH=$PATH:$GOPATH/bin 29 | 6. Checkout your Moira clone into the root of the ``GOPATH`` 30 | .. code-block:: bash 31 | 32 | git clone https://github.com//moira $GOPATH/src/github.com/moira-alert/moira 33 | 34 | 7. Go into ``$GOPATH/src/github.com/moira-alert/moira`` 35 | 36 | 8. Launch GoConvey_ 37 | .. code-block:: bash 38 | 39 | goconvey 40 | 41 | You are now ready to start hacking. Goconvey_ will keep running in the background, scanning for new test classes to execute. If you don't want to run the entire suite, start Goconvey_ from the root of directory you will be developing your code from. 42 | 43 | Alternatively you can run a quick test on the terminal with 44 | 45 | .. code-block:: bash 46 | 47 | go test -v 48 | 49 | 50 | Writing Your Own Notification Sender 51 | ------------------------------------ 52 | 53 | First, look at built-in senders: 54 | 55 | - senders/slack 56 | - senders/pushover 57 | - senders/mail 58 | 59 | All of them implement interface ``Sender`` from ``interfaces.go``. 60 | Please, note that scheduling and throttling require senders to support 61 | packing several events into one message. 62 | 63 | You should include your new sender in ``RegisterSenders`` 64 | method of ``notifier/registrator.go`` with appropriate type. 65 | 66 | Senders have access to their settings in common config, 67 | which is passed to the ``Init`` method. 68 | -------------------------------------------------------------------------------- /source/development/ui.rst: -------------------------------------------------------------------------------- 1 | UI 2 | == 3 | 4 | .. _RetailUI: https://github.com/skbkontur/retail-ui 5 | .. _React: https://reactjs.org 6 | .. _Storybook: https://storybook.js.org 7 | .. _ESLint: https://eslint.org/ 8 | .. _Flow: https://flow.org/ 9 | 10 | UI is a static web application built with RetailUI_ based on React_. 11 | 12 | Install dependencies. 13 | 14 | .. code-block:: bash 15 | 16 | yarn build 17 | 18 | Starts dev server on port 9000. You'll have to run ``yarn fakeapi`` in separate terminal to provide mock API data. Mock API server starts on port 9002. 19 | 20 | .. code-block:: bash 21 | 22 | yarn start --env.API=local 23 | 24 | Starts dev server with proxy to your API service. Make sure you setup local Moira API service and add it URL to webpack.config.js in devServer.proxy block. 25 | 26 | .. code-block:: bash 27 | 28 | yarn storybook 29 | 30 | Starts Storybook_ on port 9001. 31 | 32 | .. code-block:: bash 33 | 34 | yarn lint 35 | 36 | ESLint_ check. *Recommended to run before commit*. 37 | 38 | .. code-block:: bash 39 | 40 | yarn flow 41 | 42 | Starts Flow_ server for checking types. You can also run ``yarn flow.status`` for status, yarn ``flow.check`` for errors report, ``yarn flow.coverage.html`` to export html report with cute UI. -------------------------------------------------------------------------------- /source/gsoc.rst: -------------------------------------------------------------------------------- 1 | Google Summer of Code 2 | ===================== 3 | 4 | Here is the ideas page for Google Summer of Code 2020. 5 | 6 | We encourage interested students to contact mentors to discuss these ideas or propose 7 | new ones. The Moira team would appreciate your contributions. 8 | 9 | About Moira 10 | ----------- 11 | 12 | Moira is a realtime alerting system based on Graphite data. 13 | 14 | It's key features are: 15 | 16 | * storage independence 17 | * simple and advanced trigger syntax 18 | * tags for triggers and subscriptions 19 | * extendable notification channels 20 | * alarm fatigue protection 21 | 22 | See `overview <./overview.html>`_ for more. 23 | 24 | .. image:: ./_static/gsoc-moira-screencast.gif 25 | 26 | Moira is written in Go, the web UI is written in JavaScript. 27 | 28 | The source code is licensed under MIT. 29 | 30 | Mentors 31 | ------- 32 | 33 | * Alexey Kirpichnikov (`beevee `_) 34 | * Arkady Borovsky (`borovskyav `_) 35 | * Emil Sharifullin (`litleleprikon `_) 36 | * Andrey Kolkov (`androndo `_) 37 | * Sorokin Vladimir (`sorovlad `_) 38 | 39 | Ideas 40 | ----- 41 | 42 | Health checks for delivery channels and contacts 43 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 44 | 45 | **Explanation.** 46 | Moira's users are able to set up new delivery channels and contacts to be used with those channels. 47 | However Moira doesn't check if the channel configuration is valid and alerts can be actually sent. 48 | A user may provide a non-existent Slack user name, block Moira's bot in Telegram, etc. 49 | As a result, such user wouldn't be able to receive alerts. 50 | The bad thing is that sometimes invalid configuration would cause Moira's bots to be banned for a certain period of time. 51 | This effectively means a denial-of-service for alerts which is highly undesirable. 52 | 53 | The aim of this project is to implement health checks when delivery channels and contacts are set up. 54 | To do so, one should enhance the delivery channel and contact setup flow: send a test alert, verify that it's received, don't let to save an invalid configuration otherwise. 55 | Certain modifications of the web UI may be required. 56 | 57 | .. image:: ./_static/gsoc-moira-delivery-channel.png 58 | 59 | **Code reference.** 60 | See `contact API `_ source code and `subscription API `_ source code. 61 | 62 | **Required skills.** 63 | Go skills to add health checks, a bit of JavaScript and React to tune the web UI. 64 | 65 | **Expected outcome.** 66 | Health checks are implemented and released. 67 | 68 | **Mentors.** 69 | Arkady Borovsky (borovskyav@kontur.ru), 70 | Emil Sharifullin (e.sharifullin@kontur.ru). 71 | 72 | RESTify Moira's API 73 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 74 | 75 | **Explanation.** 76 | 77 | Moira designed to be API-first solution and all the setup of alerting must be done via HTTP API. Unfortunately Moira'a API right now is not follow all the principles of `REST `_. This means that HTTP methods somewhere are not used correctly and URL paths somewhere are not describe the resources in a right way. Additionally some of the endpoints provide the data which schema is overcomplicated and contains wrong attributes. The great solution for this type of issues will be to use `JSON API standard `_. 78 | 79 | The aim of this project is to define methods of API that do not follow to the and change it using the REST and JSON API principles. 80 | 81 | **Code reference.** 82 | See `Moira's API `_ source code and the `OpenAPI description `_ source code. 83 | 84 | **Required skills.** 85 | General Go skills. Familiarity with OpenAPI or Swagger. Knowledge of REST and JSON API principles. 86 | 87 | **Expected outcome.** 88 | Moira's API is RESTful. 89 | OpenAPI description is up to date. 90 | Moira's frontend is changed to follow the API changes(optional). 91 | 92 | **Mentors.** 93 | Emil Sharifullin (e.sharifullin@kontur.ru), 94 | Andrey Kolkov (androndo@kontur.ru). 95 | 96 | Moira's Business Metrics 97 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 98 | 99 | **Explanation.** 100 | Moira is a huge and complicated software and it operates with a huge amount of data. Sometimes for statistics and troubleshooting we need to define some metrics that will more precise tell us which amount of load Moira is carrying on. The example of this metrics is: amount of triggers with tagged metrics, amount of triggers with huge amount of metrics, amount of triggers with and without subscriptions, etc. 101 | 102 | To achieve this goal we can create a new microservice that will collect this data from storage and export it to graphite or implement this metrics to existing services. 103 | 104 | **Code reference.** 105 | See `Moira's backend `_ source code. 106 | 107 | **Required skills.** 108 | Go skills. Familiarity with graphite would be a plus. 109 | 110 | **Expected outcome.** 111 | Created a new service that will export business metrics or this metrics export will be added to existing services. 112 | 113 | **Mentors.** 114 | Alexey Kirpichnikov (alexkir@kontur.ru), 115 | Andrey Kolkov (androndo@kontur.ru). 116 | 117 | Complete Moira's mobile web version 118 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 119 | 120 | **Explanation.** 121 | 122 | To provide best user experience Moira's web UI were developed with accuracy and meant to be as much minimalistic and laconic as possible. But still there are exist pages that do not look perfectly in mobile version of web interface. The example of this pages are following pages: 123 | 124 | * Main page and navigation on it 125 | * Subscriptions page 126 | * Trigger page and trigger edit pages 127 | * Teams page 128 | 129 | The aim of this project is to add this pages to mobile version of Moira's web UI and build the UI with best user experience in mobile environment. 130 | 131 | **Code reference.** 132 | See `Moira's web UI `_ source code. 133 | 134 | **Required skills.** 135 | JavaScript, TypeScript and React skills. Knowledge of UX will be a plus. 136 | 137 | **Expected outcome.** 138 | Moira's mobile web UI allows user to use the pages that are listed above. 139 | 140 | **Mentors.** 141 | Sorokin Vladimir (v_sorokin@kontur.ru), 142 | Arkady Borovsky (borovskyav@kontur.ru). 143 | 144 | Noisy trigger analysis tools 145 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 146 | 147 | **Explanation.** 148 | On-call engineers are badly affected by noisy triggers that generate alerts multiple times a day. 149 | Attention to alerts reduces greatly, and chances to miss one important alert grow. 150 | One badly configured flapping trigger can affect the entire workflow. 151 | Our documentation contains an `entire page <./user_guide/efficient.html>`_ dedicated to this problem with some tips on mitigation. 152 | But we can do more. 153 | 154 | The aim of this project is to help Moira users identify noisy triggers. 155 | To do so, one should research and define a metric of trigger noisiness, and then create a UI page that demonstrates worst triggers to the user. 156 | 157 | **Code reference.** 158 | See `Moira's backend `_ source code and `Moira's web UI `_ source code. 159 | 160 | **Required skills.** 161 | Basic Go and JavaScript skills. 162 | 163 | **Expected outcome.** 164 | Moira's web UI allows user to see noisy trigger list, optionally filtered by tags. 165 | 166 | **Mentors.** 167 | Alexey Kirpichnikov (alexkir@kontur.ru), 168 | Emil Sharifullin (e.sharifullin@kontur.ru). 169 | 170 | Done in previous years 171 | ---------------------- 172 | 173 | .. warning:: 174 | Following projects are no longer available. 175 | 176 | OpenAPI description of Moira's API 177 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 178 | Done in 2020 by `Michael Okoko `_. 179 | 180 | **Explanation.** 181 | Moira's web UI is nice and widely used. 182 | However, users don't always want to create triggers, subscriptions, and contacts manually. 183 | They would like to be able to automate routine tasks with the tools like `Ansible `_ which they already use to bootstrap database and application clusters. 184 | For this kind of automation, Moira should have a well-documented API and a number of client libraries for all popular languages. 185 | At this point, Moira doesn't have any API documentation. 186 | To use the API, one should study Moira's source code or an existing client library source code to understand how the API works and reverse-engineer contracts of its methods. 187 | 188 | The aim of this project is to provide an always up-to-date documentation of Moira's API and a few client libraries. 189 | To do so, one should create an `OpenAPI `_ description of API, generate a number of client libraries for popular programming languages with `Swagger tools `_, and setup a process so the documentation and the clients are updated when a new API version is released. 190 | 191 | **Code reference.** 192 | See `Moira's API `_ source code and the `Python client library `_ source code. 193 | 194 | **Required skills.** 195 | General Go or Python skills. Familiarity with OpenAPI or Swagger would be a plus. 196 | 197 | **Expected outcome.** 198 | Moira's documentation has a link to a human-readable API documentation. 199 | Client libraries are released (not required). 200 | There's a process in place to update the documentation and the clients on API changes. 201 | 202 | **Mentors.** 203 | Emil Sharifullin (e.sharifullin@kontur.ru), 204 | Alexey Kirpichnikov (alexkir@kontur.ru). 205 | 206 | Flow to TypeScript migration 207 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 208 | 209 | Done in 2020 by `Gilevich Petr `_. 210 | 211 | **Explanation.** 212 | Nowadays, Moira's web UI is written in JavaScript and `Flow `_ is used as a type checker. 213 | Although we love Flow dearly, TypeScript is adopted widely and has a bigger community. 214 | This makes TypeScript a better choice for Moira's web UI development. 215 | 216 | The aim of this project is to migrate Moira's web UI source code from Flow to TypeScript. 217 | To do so, one should analyze the code base, propose a migration strategy, actually rewrite the code, and change the build process if needed. 218 | 219 | **Code reference.** 220 | See `Moira's web UI `_ source code. 221 | 222 | **Required skills.** 223 | JavaScript and TypeScript skills. Familiarity with Flow would be a plus. 224 | 225 | **Expected outcome.** 226 | Moira's web UI source code is migrated to TypeScript. 227 | A new major version of Moira's web UI is released. 228 | 229 | **Mentors.** 230 | Alexey Kirpichnikov (alexkir@kontur.ru), 231 | Nikolay Kudrin (n.kudrin@kontur.ru). 232 | 233 | Noisy trigger analysis tools 234 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 235 | 236 | Support for additional delivery channels 237 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 238 | 239 | Done in 2019 by `Aswin `_. 240 | 241 | **Explanation.** 242 | Moira supports a number of delivery channels such as email, Slack, Telegram, etc. to inform users that a certain trigger was activated (see :ref:`subscriptions`). 243 | 244 | The aim of this project is to provide support for a number of additional delivery channels. 245 | To do so, one should talk to community and research possible channels to be added, contribute corresponding `senders `_, and tune the web UI to allow users to create subscriptions using new channels. 246 | 247 | .. image:: ./_static/gsoc-moira-delivery-channels.png 248 | 249 | **Code reference.** 250 | See `email sender `_ source code or `Pushover sender `_ source code. 251 | 252 | **Required skills.** 253 | Go skills to add senders, a bit of JavaScript and React to tune the web UI. 254 | 255 | **Expected outcome.** 256 | Some qualitative or quantitative data on channel popularity is collected. 257 | Several delivery channels are added to Moira and released. 258 | 259 | **Mentors.** 260 | Alexey Kirpichnikov (alexkir@kontur.ru), 261 | Alexander Sushko (sushko@kontur.ru). 262 | -------------------------------------------------------------------------------- /source/index.rst: -------------------------------------------------------------------------------- 1 | Moira Documentation 2 | =================== 3 | 4 | Contents 5 | -------- 6 | 7 | .. toctree:: 8 | :maxdepth: 2 9 | 10 | overview 11 | release_notes/index 12 | installation/index 13 | user_guide/index 14 | development/index 15 | contacts 16 | gsoc 17 | 18 | .. include:: overview.rst 19 | -------------------------------------------------------------------------------- /source/installation/configuration.rst: -------------------------------------------------------------------------------- 1 | Configuration 2 | ============= 3 | 4 | By default, microservices will look for ``/etc/moira/.yml``, 5 | but you can change this location by passing your path as a command-line 6 | parameter ``--config``. 7 | 8 | **Important**: Incorrect configuration format may cause the service to crash at startup 9 | 10 | On this page you can find examples of configuration files 11 | for Moira microservices. 12 | 13 | .. _filter-configuration: 14 | 15 | Filter 16 | ------ 17 | 18 | .. _storage-schemas.conf: http://graphite.readthedocs.io/en/latest/config-carbon.html#storage-schemas-conf 19 | 20 | .. literalinclude:: ../../filter.example.yml 21 | :language: yaml 22 | 23 | storage-schemas.conf_ is graphite carbon configuration file that 24 | should match similarly-named file in your Graphite installation. 25 | 26 | .. _checker-configuration: 27 | 28 | Checker 29 | ------- 30 | 31 | .. literalinclude:: ../../checker.example.yml 32 | :language: yaml 33 | 34 | .. _graphite-remote-triggers-checker: 35 | 36 | Remote Triggers Checker 37 | ^^^^^^^^^^^^^^^^^^^^^^^ 38 | 39 | One of Moira key feature is Graphite independance. Some Graphite queries 40 | are *very* ineffective. Tools like Seyren_ multiply this effect every 41 | minute making lots of ineffective queries and overloading your cluster. 42 | Moira relies on the incoming metric stream, and has its own fast cache 43 | for recent data. 44 | 45 | Enabling Remote triggers Checker allows user to create triggers 46 | that relies on Graphite Storage instead of Redis DB. 47 | 48 | .. warning:: Use this feature with caution, because it can create an extra load on Graphite HTTP API. 49 | 50 | .. _lazy-triggers-checker: 51 | 52 | Lazy Triggers Checker 53 | ^^^^^^^^^^^^^^^^^^^^^^ 54 | 55 | In Moira 2.4 we add a new entity - Lazy Trigger. This is a regular trigger 56 | but without any subscription for it. By default Moira treats any trigger 57 | equally regardless on its subscriptions number. You can change this behaviour 58 | using ``lazy_triggers_check_interval`` option in checker section. This can 59 | reduce CPU usage on your server. Lazy triggers checker works if 60 | ``lazy_triggers_check_interval`` > ``check_interval``. We recommend set 61 | it to ``10m`` (10 minutes). 62 | 63 | .. _prometheus-remote-triggers-checker: 64 | 65 | Prometheus Checker 66 | ^^^^^^^^^^^^^^^^^^ 67 | 68 | In Moira 2.9 Prometheus Remote metric source was added. It works like Graphite 69 | remote metric source, but uses prometheus metrics and PromQL instead. It makes 70 | queries to Prometheus api or Victoria Metrics VMSelect api. 71 | 72 | Prometheus Checker can be counfigured to use retries when trying to fetch metrics. 73 | We recommend to use 3 retries with the retry timeout of `10s`. 74 | 75 | .. _notifier-configuration: 76 | 77 | Notifier 78 | -------- 79 | 80 | .. literalinclude:: ../../notifier.example.yml 81 | :language: yaml 82 | 83 | .. _slack-icons: 84 | 85 | Slack icons 86 | ^^^^^^^^^^^ 87 | 88 | .. image:: ../_static/slack-state-emoji.png 89 | :alt: State-specific emoji 90 | :width: 534 91 | :height: 226 92 | 93 | By default Slack sender won't change default icon configured for your bot. 94 | To use state-specific icons in notifications: 95 | 96 | .. image:: ../_static/slack-emoji.png 97 | :alt: Add custom emoji 98 | 99 | - Download and unzip `notification icons `_ 100 | - Add icons from ``..icons/slack`` directory as custom emojis according to their filenames to `Slack `_ 101 | - Set ``use_emoji`` to ``true`` for Slack sender section in 102 | notifier configuration file 103 | 104 | Email Template 105 | ^^^^^^^^^^^^^^ 106 | 107 | By default mail sender will use 'Fancy' template: 108 | 109 | .. image:: ../_static/fancy-email-template.png 110 | :alt: Fancy email template 111 | :width: 400 112 | 113 | Self State Monitor 114 | ^^^^^^^^^^^^^^^^^^ 115 | 116 | If self state monitor is enabled, Moira will periodically check the Redis 117 | connection, the number of incoming metrics in the Moira-Filter and 118 | the number of triggers to be checked by Moira-Checker. 119 | 120 | See :doc:`../user_guide/selfstate` for more details. 121 | 122 | .. _api-and-web-config: 123 | 124 | API and Web 125 | ----------- 126 | 127 | .. literalinclude:: ../../api.example.yml 128 | :language: yaml 129 | 130 | Web contact fields: 131 | 132 | - **type** (any uniq string) *required* — contact type: pushover, slack, mail, 133 | script, telegram, twilio sms, twilio voice, etc.; 134 | - **label** *required* — contact label type. Uses in add/edit 135 | contact form in select control; 136 | - **validation** — regular expression for user contact, 137 | uses for validation in add/edit contact form; 138 | - **placeholder** — hint shown in input field; 139 | - **help** — help text in Markdown_ markup; 140 | 141 | .. image:: ../_static/web-ui-example.png 142 | :alt: WEB UI example 143 | :width: 575 144 | 145 | Remote API 146 | ---------- 147 | 148 | By default, Web uses local API server (both containers are running on the same host). 149 | But if you need to reconfigure Web to interact with API running on remote server then simply set container environment variable MOIRA_API_URI equal to required URI: 150 | 151 | ``MOIRA_API_URI: remoteapi.domain:8081`` 152 | 153 | .. _Markdown: https://daringfireball.net/projects/markdown/syntax 154 | .. _Seyren: https://github.com/scobal/seyren 155 | -------------------------------------------------------------------------------- /source/installation/docker.rst: -------------------------------------------------------------------------------- 1 | Docker 2 | ====== 3 | 4 | .. |Docker Hub| replace:: Docker Hub 5 | .. _Docker Hub: https://hub.docker.com/u/moira/ 6 | 7 | You can quickly test a local Moira installation using Docker containers 8 | from |Docker Hub|_ and docker-compose file in documentation repository. 9 | 10 | .. code-block:: bash 11 | 12 | git clone https://github.com/moira-alert/docker-compose.git 13 | cd docker-compose 14 | docker-compose pull 15 | docker-compose up 16 | 17 | Containers are preconfigured to serve Web UI at ``localhost:8080`` 18 | and accept graphite metrics at ``localhost:2003``. 19 | -------------------------------------------------------------------------------- /source/installation/feeding.rst: -------------------------------------------------------------------------------- 1 | Feeding Metrics to Moira 2 | ======================== 3 | 4 | .. _carbon-c-relay: https://github.com/grobian/carbon-c-relay 5 | 6 | Moira needs to keep its own local copy of your metric data to improve 7 | performance and reduce load on your existing graphite cluster. This means 8 | data needs to be duplicated from your existing stream and sent to your 9 | existing cluster and to your Moira installation. 10 | 11 | Unfortunately, the Carbon-Relay with Graphite does not support duplication 12 | of data to multiple backends, and so you need to use a more feature rich 13 | carbon relay such as carbon-c-relay_. 14 | 15 | The following is a basic example configuration which defines two clusters 16 | and sends all metrics to both at once. One cluster is Moira installation, 17 | and the other uses consistent hashing across a three node cluster of 18 | Carbon servers. 19 | 20 | .. code-block:: text 21 | 22 | cluster moira 23 | forward 24 | moira-host:2003 25 | ; 26 | 27 | cluster graphite 28 | carbon_ch 29 | 127.0.0.1:2006=a 30 | 127.0.0.1:2007=b 31 | 127.0.0.1:2008=c 32 | ; 33 | 34 | match * 35 | send to 36 | moira 37 | graphite 38 | ; 39 | -------------------------------------------------------------------------------- /source/installation/index.rst: -------------------------------------------------------------------------------- 1 | Installation 2 | ============ 3 | 4 | .. toctree:: 5 | :maxdepth: 1 6 | 7 | manual 8 | docker 9 | packages 10 | configuration 11 | webhooks_scripts 12 | feeding 13 | security 14 | -------------------------------------------------------------------------------- /source/installation/manual.rst: -------------------------------------------------------------------------------- 1 | Manual Installation 2 | =================== 3 | 4 | .. _golang: https://golang.org/doc/install 5 | .. _redis: http://redis.io/download 6 | .. _nginx: http://nginx.org/en/download.html 7 | .. _web.json: https://github.com/moira-alert/moira/blob/master/pkg/api/web.json 8 | .. _configuration: https://moira.readthedocs.io/en/latest/installation/configuration.html#api 9 | 10 | 11 | .. tip:: To get Moira running quickly, try :doc:`/installation/docker` version 12 | 13 | There are following components you need to install 14 | before running Moira microservices: 15 | 16 | 1. golang_ version 1.9 or higher 17 | 2. redis_ database version 3.2 or higher 18 | 3. web server e.g. nginx_ 19 | 20 | 21 | Build Moira Microservices 22 | ------------------------- 23 | 24 | .. code-block:: bash 25 | 26 | go get -u github.com/moira-alert/moira 27 | cd $GOPATH/src/github.com/moira-alert/moira 28 | make build 29 | 30 | You will find binaries in ``$GOPATH/src/github.com/moira-alert/moira/build``. 31 | 32 | 33 | Download Web UI Application 34 | --------------------------- 35 | 36 | https://github.com/moira-alert/web2.0/releases/latest 37 | 38 | Download and unpack ``.tar.gz`` file into Nginx static 39 | files directory (e.g. ``/var/local/www/moira``). 40 | 41 | 42 | Configure 43 | --------- 44 | 45 | 1. If you need to override default settings, place configuration 46 | files somewhere on your disk (e.g. ``/etc/moira/``). You can dive into 47 | :doc:`/installation/configuration` syntax on a separate page. 48 | 49 | 2. Place nginx configuration file to proper location 50 | (e.g. ``/etc/nginx/conf.d/moira.conf``): 51 | 52 | .. code-block:: text 53 | 54 | server { 55 | listen 127.0.0.1:80; 56 | location / { 57 | root /var/local/www/moira; 58 | index index.html; 59 | try_files $uri $uri/ /index.html; 60 | } 61 | location /api/ { 62 | proxy_pass http://127.0.0.1:8081; 63 | } 64 | } 65 | 66 | 3. If you need to override UI settings, edit web.json_ file. 67 | You can find its location in API configuration_. 68 | 69 | 70 | Run 71 | --- 72 | 73 | 1. Run nginx and redis-server 74 | 2. Run microservices 75 | 76 | .. code-block:: bash 77 | 78 | $GOPATH/src/github.com/moira-alert/moira/build/cache 79 | $GOPATH/src/github.com/moira-alert/moira/build/checker 80 | $GOPATH/src/github.com/moira-alert/moira/build/notifier 81 | $GOPATH/src/github.com/moira-alert/moira/build/api 82 | 83 | Now you need to feed your metrics to Moira (see :doc:`/installation/feeding`) 84 | on port 2003 and to create alerts in UI (see :doc:`/user_guide/index`). 85 | -------------------------------------------------------------------------------- /source/installation/packages.rst: -------------------------------------------------------------------------------- 1 | RPM and DEB Packages 2 | ==================== 3 | 4 | All stable versions of Moira components are tagged on GitHub. 5 | For every tag, we automatically build RPM and DEB packages. 6 | You can download these packages on each repository release page: 7 | 8 | 1. https://github.com/moira-alert/web2.0/releases 9 | 2. https://github.com/moira-alert/moira/releases 10 | -------------------------------------------------------------------------------- /source/installation/security.rst: -------------------------------------------------------------------------------- 1 | .. _auth_basic_module: http://nginx.org/en/docs/http/ngx_http_auth_basic_module.html 2 | 3 | Security 4 | ======== 5 | 6 | Typically, internal DevOps tools like Graphite are deployed in intranet 7 | without any external access, so you can skip authentication and leave 8 | everything accessible to everyone. But powerful Moira features, like 9 | separate subscriptions for tags, work best when you have some way 10 | to tell apart users. 11 | 12 | Moira doesn't provide any authentication mechanism. It is hard to find 13 | one that fits all situations. Instead, Moira accepts ``X-WebAuth-User`` 14 | header with some user id, like login name. You are free to set up any 15 | reverse proxy and configure it to provide this header. 16 | 17 | If you don't, Moira will assume that user id is "anonymous". 18 | 19 | .. warning:: Even if you do provide authentication header, please note that most parts of Moira are 20 | read and write accessible to every user, and there is no meaningful way of authorization 21 | in Moira. This is by design, because Moira is an internal DevOps tool. Separating users 22 | is a convenience, not protection feature. 23 | 24 | 25 | Example of Nginx Configuration 26 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 27 | 28 | Assuming that Moira UI static files are in ``/var/www/moira-web`` 29 | and API is running on port 8081 30 | 31 | .. code-block:: text 32 | 33 | server { 34 | auth_basic "Moira"; 35 | auth_basic_user_file /etc/nginx/htpasswd; 36 | 37 | listen 0.0.0.0:80 default_server; 38 | 39 | location / { 40 | root /var/www/moira; 41 | index index.html; 42 | try_files $uri $uri/ /index.html; 43 | } 44 | 45 | location /api/ { 46 | proxy_pass http://127.0.0.1:8081; 47 | proxy_set_header X-WebAuth-User $remote_user; 48 | } 49 | } 50 | 51 | Look at auth_basic_module_ if you need more details of Nginx 52 | basic authentication. 53 | 54 | 55 | Webhooks and Custom Scripts 56 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 57 | 58 | When configuring :doc:`/installation/webhooks_scripts`, note that 59 | ``${contact_value}`` is substituted with user input value from web UI. 60 | It means that a malicious user can potentially run anything or make 61 | arbitrary web requests on a server that runs Moira notifier. 62 | Always make sure you can trust your users if you use ``${contact_value}`` 63 | templated parameter in your scripts or webhooks. 64 | -------------------------------------------------------------------------------- /source/installation/webhooks_scripts.rst: -------------------------------------------------------------------------------- 1 | Webhooks and Custom Scripts 2 | =========================== 3 | 4 | Moira has two special kinds of senders: webhooks and scripts. 5 | 6 | Script runs an executable on the same machine that runs notifier instances. 7 | Webhook makes POST requests to a specified URL. Scripts and webhooks are 8 | a flexible way to add integrations with services that are not supported 9 | in Moira natively. 10 | 11 | You may want to add several different scripts for users to choose from. 12 | Next section describes how to implement this. 13 | 14 | 15 | Scripts 16 | ------- 17 | 18 | You can specify executable path and arguments in the notifier 19 | configuration file (see :doc:`/installation/configuration` for details). 20 | 21 | Add a separate section for each script: 22 | 23 | .. code-block:: yaml 24 | 25 | - sender_type: script 26 | contact_type: jira 27 | exec: /usr/bin/post_to_jira --project=${contact_value} 28 | - sender_type: script 29 | contact_type: irc 30 | exec: /opt/myscripts/irc_adapter ${contact_value} ${trigger_id} 31 | 32 | Then, in web UI configuration: 33 | 34 | .. code-block:: json 35 | 36 | { 37 | "contacts": [ 38 | { 39 | "type": "jira", 40 | "label": "Create JIRA issue", 41 | "placeholder": "Project name" 42 | }, 43 | { 44 | "type": "irc", 45 | "label": "Post to IRC channel", 46 | "placeholder": "Channel name" 47 | } 48 | ], 49 | ... 50 | } 51 | 52 | 53 | Templated Parameters 54 | -------------------- 55 | 56 | You may have noted that we use templated parameters like 57 | ``${contact_value}`` in configuration examples. You can use these 58 | parameters in script as well as webhook contacts. Notifier will 59 | replace them with actual values extracted from event. 60 | 61 | ================ ================================================ 62 | Parameter Value 63 | ================ ================================================ 64 | ${contact_id} Contact ID 65 | ${contact_value} Contact value, as specified by user via web UI 66 | ${contact_type} Contact type, as specified in web UI config file 67 | ${trigger_id} Trigger ID 68 | ================ ================================================ 69 | 70 | 71 | Webhooks 72 | -------- 73 | 74 | On each event, Moira will make a POST request to the URL specified 75 | in notifier configuration file with the following JSON payload 76 | if the request `body` in the config is empty. 77 | 78 | 79 | ========= =========== ===================================== 80 | Attribute Type Description 81 | ========= =========== ===================================== 82 | trigger Trigger Trigger data 83 | events Event Array List of events 84 | contact Contact Contact data 85 | plot String Base64 string containing trigger plot 86 | throttled Bool True if notifications are throttled 87 | ========= =========== ===================================== 88 | 89 | 90 | Fields Description 91 | ------------------ 92 | 93 | Trigger 94 | ~~~~~~~ 95 | 96 | =========== ============ ==================== 97 | Attribute Type Description 98 | =========== ============ ==================== 99 | id String Trigger ID 100 | name String Trigger name 101 | description String Trigger description 102 | tags String Array List of trigger tags 103 | =========== ============ ==================== 104 | 105 | 106 | Event 107 | ~~~~~ 108 | 109 | ============= ======= ===================== 110 | Attribute Type Description 111 | ============= ======= ===================== 112 | metric String Metric name 113 | value Float64 Metric value 114 | timestamp Int64 Event timestamp 115 | trigger_event Bool Event type 116 | state String Current metric state 117 | old_state String Previous metric state 118 | ============= ======= ===================== 119 | 120 | 121 | Contact 122 | ~~~~~~~ 123 | 124 | ========= ====== ============== 125 | Attribute Type Description 126 | ========= ====== ============== 127 | type String Contact type 128 | value String Contact value 129 | id String Contact ID 130 | user String Contact Author 131 | ========= ====== ============== 132 | 133 | 134 | HTTP Headers 135 | ------------ 136 | 137 | ============ ================ 138 | Name Value 139 | ============ ================ 140 | User-Agent Moira 141 | Content-Type application/json 142 | ... ... 143 | ============ ================ 144 | 145 | ... - You can set additional headers in the webhook config 146 | (see :doc:`/installation/configuration` for details). 147 | 148 | 149 | Example of a request without explicitly specifying the request body in the config 150 | --------------------------------------------------------------------------------- 151 | 152 | .. code-block:: json 153 | 154 | { 155 | "trigger": { 156 | "id": "triggerID", 157 | "name": "triggerName", 158 | "description": "triggerDescription", 159 | "tags": [ 160 | "triggerTag1", 161 | "triggerTag2" 162 | ] 163 | }, 164 | "events": [ 165 | { 166 | "metric": "metricName1", 167 | "value": 0, 168 | "timestamp": 499165200, 169 | "trigger_event": false, 170 | "state": "OK", 171 | "old_state": "ERROR" 172 | }, 173 | { 174 | "metric": "triggerName", 175 | "value": 0, 176 | "timestamp": 1445412480, 177 | "trigger_event": true, 178 | "state": "OK", 179 | "old_state": "ERROR" 180 | }, 181 | { 182 | "metric": "metricName2", 183 | "value": 0, 184 | "timestamp": -446145720, 185 | "trigger_event": false, 186 | "state": "OK", 187 | "old_state": "ERROR" 188 | } 189 | ], 190 | "contact": { 191 | "type": "webhookContactName", 192 | "value": "https://localhost/webhooks/moira", 193 | "id": "9728adae-1487-4e5b-80f6-8496f59b223e", 194 | "user": "author" 195 | }, 196 | "plot": "", 197 | "throttled": false 198 | } 199 | 200 | You can also explicitly specify the request body using go-templates. 201 | Available fields: .Contact.Value and .Contact.Type 202 | 203 | Example of a configuration with an explicit request body 204 | -------------------------------------------------------- 205 | 206 | .. code-block:: yaml 207 | 208 | - sender_type: webhook 209 | contact_type: test_webhook 210 | url: http://localhost:8080/webhook 211 | body: { "name": "test-name", "value": {{ .Contact.Value | quote }} } 212 | headers: 213 | test-header: test-value 214 | 215 | Example of a request with explicitly specifying the request body in the config 216 | ------------------------------------------------------------------------------ 217 | 218 | .. code-block:: json 219 | 220 | { 221 | "name": "test-name", 222 | "value": "test-contact-value" 223 | } 224 | 225 | Delivery checks 226 | --------------- 227 | As you may notice, Moira will do her best to send notification. 228 | But often successful sending doesn't mean that notification was successfully delivered 229 | (for example if delivering is a long lasting operation). 230 | 231 | In order to solve the problem we add the delivery checks feature to webhook sender. 232 | 233 | First please read the notifier :doc:`/installation/configuration` and pay attention to webhook sender. 234 | 235 | The most important fields for performing delivery checks is ``url_template`` and ``check_template``. 236 | These two fields are both `go templates `_ and support `sprig functions `_. 237 | 238 | 239 | How does it work? 240 | ~~~~~~~~~~~~~~~~~ 241 | 242 | When the notification is sent (HTTP POST request performed) Moira reads response status code and response body. 243 | If response code is greater or equal 200 and less than 300, then for Moira it means that notification is sent ok, otherwise sent failed. 244 | 245 | After successful notification sending ``url_template`` is filled with data to provide a valid url. 246 | URL and other necessary information is stored in the database. 247 | 248 | Separate goroutine reads such info from database (every ``check_timeout`` seconds) and perform delivery check request (HTTP GET request), with: 249 | 250 | * URL got after filling ``url_template`` 251 | * user and password specified in ``delivery_check`` option of config 252 | * headers specified in ``delivery_check`` option of config and headers from **HTTP Headers** above 253 | 254 | If delivery check request succeeds, then the response body and some other fields (you can find details below) is used 255 | to fill ``check_template``. The result of filling ``check_template`` must be one of valid strings 256 | (which represent delivery states, more info could also be found below). 257 | Based on calculated delivery state and count of already performed attempts Moira will do one of the following things: 258 | 259 | * Mark delivery notification ok in metrics 260 | * Mark delivery notification failed in metrics 261 | * Mark that delivery checks is stopped in metrics 262 | * Schedule one more delivery check 263 | 264 | ``url_template`` 265 | ~~~~~~~~~~~~~~~~ 266 | Here is the list of data that is available for use in ``url_template`` 267 | 268 | ================== ============== ============================================== 269 | Attribute Type Description 270 | ================== ============== ============================================== 271 | .Contact.Type string Contact type 272 | .Contact.Value string Contact value 273 | .TriggerID string Trigger id 274 | .SendAlertResponse map[string]any JSON response on POST request decoded into map 275 | ================== ============== ============================================== 276 | 277 | The result of the template filling must be a valid url. 278 | 279 | For example, if we have: 280 | 281 | * Contact.Type: slack 282 | * Contact.Value: some_channel 283 | * TriggerID: some-trigger-id 284 | * And on send alert we have response: 285 | 286 | .. code-block:: json 287 | 288 | { 289 | "some_value": 25, 290 | "another_value": "hello" 291 | } 292 | 293 | And our ``url_template`` is the following: 294 | 295 | .. code-block:: 296 | 297 | "https://example.com/{{ .Contact.Type }}/{{ .Contact.Value }}/{{ .TriggerID }}/{{ .SendAlertResponse.another_value }}" 298 | 299 | Our result URL will be: 300 | 301 | .. code-block:: 302 | 303 | "https://example.com/slack/some_channel/some-trigger-id/hello" 304 | 305 | ``check_template`` 306 | ~~~~~~~~~~~~~~~~~~ 307 | Here is the list of data that available for use in ``check_template`` 308 | 309 | ====================== ============== ============================================= 310 | Attribute Type Description 311 | ====================== ============== ============================================= 312 | .Contact.Type string Contact type 313 | .Contact.Value string Contact value 314 | .TriggerID string Trigger id 315 | .DeliveryCheckResponse map[string]any JSON response on GET request decoded into map 316 | ====================== ============== ============================================= 317 | 318 | The result of the template filling must be one of the strings below: 319 | 320 | ============ =========================================================================== 321 | String value Description 322 | ============ =========================================================================== 323 | OK Should be returned if notification was successfully delivered 324 | FAILED Should be returned if notification definitely was not delivered 325 | PENDING Should be returned if notification has not yet been delivered 326 | EXCEPTION Should be returned if error occurred while evaluating the state of delivery 327 | ============ =========================================================================== 328 | 329 | For example, if we have: 330 | 331 | * Contact.Type: slack 332 | * Contact.Value: some_channel 333 | * TriggerID: some-trigger-id 334 | * And on delivery check request we have response: 335 | 336 | .. code-block:: json 337 | 338 | { 339 | "contact_value": "some_channel", 340 | "some_value": 25, 341 | "important_value": "ok" 342 | } 343 | 344 | And our ``check_template`` is: 345 | 346 | .. code-block:: 347 | 348 | {{-if and 349 | (eq .DeliveryCheckResponse.contact_value .Contact.Value) 350 | (eq .DeliveryCheckResponse.important_value "ok") 351 | -}} 352 | OK 353 | {{- else -}} 354 | FAILED 355 | {{- end -}} 356 | 357 | The result of the template filling will be ``OK``. 358 | For Moira this means that notification was successfully delivered. 359 | 360 | For the same ``check_template`` but following delivery check response body: 361 | 362 | .. code-block:: json 363 | 364 | { 365 | "contact_value": "some_channel", 366 | "some_value": 25, 367 | "important_value": "not ok" 368 | } 369 | 370 | The result of the template filling will be ``FAILED``. 371 | For Moira this means that notification definitely was not delivered. 372 | 373 | **Note** that if the result of filling ``check_template`` is one of ``PENDING``, ``EXCEPTION``, 374 | Moira continues to perform delivery checks until ``max_attempts`` count will be performed. 375 | -------------------------------------------------------------------------------- /source/overview.rst: -------------------------------------------------------------------------------- 1 | Overview 2 | ======== 3 | 4 | .. _Graphite: https://github.com/graphite-project 5 | 6 | Moira is a real-time alerting tool, based on Graphite_ data. 7 | 8 | 9 | Key Features 10 | ------------ 11 | 12 | .. _Seyren: https://github.com/scobal/seyren 13 | .. _Slack: https://slack.com 14 | .. _Pushover: https://pushover.net 15 | .. |Alarm fatigue| replace:: **Alarm fatigue** 16 | .. _Alarm fatigue: https://en.wikipedia.org/wiki/Alarm_fatigue 17 | .. _carbonapi: https://github.com/go-graphite/carbonapi/blob/ccac7217894801a5a6ceb8602a70ea0d79e975cf/cmd/carbonapi/COMPATIBILITY.md#functions 18 | .. _govaluate: https://github.com/Knetic/govaluate/blob/master/MANUAL.md 19 | 20 | * **Graphite storage independence** 21 | 22 | Some Graphite queries are *very* ineffective. Tools like Seyren_ 23 | multiply this effect every minute making lots of ineffective queries 24 | and overloading your cluster. Moira relies on the incoming 25 | metric stream, and has its own fast cache for recent data. 26 | 27 | * **Support for (almost) all Graphite functions** 28 | 29 | Graphite function library (carbonapi_) is embedded directly into 30 | Moira source code. You can use any function and get predictable results, 31 | like in your Graphite or Grafana dashboards. 32 | 33 | * **Support for custom expressions** 34 | 35 | If simple warning/error threshold is not enough, you can write flexible 36 | govaluate_ expressions to calculate trigger state based on metric data. 37 | 38 | * **Tags for triggers and subscriptions** 39 | 40 | When several teams/services share one monitoring tool, it is essential 41 | to provide some way of filtering triggers and subscriptions in the UI. 42 | Moira has a flexible tag system. 43 | 44 | * **Extendable notification channels** 45 | 46 | Moira supports email, Slack_, Pushover_ and many other channels 47 | of notification out-of-the-box. But you can always write your own plugin 48 | in Go and rebuild Moira Notifier microservice. 49 | 50 | * |Alarm fatigue|_ **protection** 51 | 52 | Sometimes one of your triggers goes mad and switches back and forth 53 | between states, sending you hundreds of notifications. Sometimes you 54 | just ignore and delete all messages, accidentally also deleting one that 55 | is actually important. Moira tries to protect you with a feature called 56 | *throttling*. It's simple: if one of your triggers starts to send over 57 | 10 messages per hour, Moira limits this trigger to one message 58 | per 30 minutes. Alerts from this trigger are combined, and not lost - 59 | just packaged into a single message. 60 | 61 | 62 | Limitations 63 | ----------- 64 | 65 | By default, Moira stores metric history for one hour. This ensures 66 | performance under heavy load. You can tweak this in config file, 67 | but note that performance will degrade. 68 | 69 | In order to reduce database load, Moira checks every single trigger 70 | at most once every 5 seconds. Probably, your metrics arrive once 71 | every minute, so you really won't notice this limitation. You can also 72 | tweak this in config file. 73 | 74 | 75 | .. _microservices-architecture: 76 | 77 | Microservices 78 | ------------- 79 | 80 | In spirit of Graphite architecture, Moira consists of several loosely 81 | coupled microservices. You are welcome to replace or to add new ones. 82 | 83 | 84 | Filter 85 | ^^^^^^ 86 | 87 | Filter is a lightweight service responsible for receiving lots of metric 88 | data in Graphite format. It filters received data and saves only metrics 89 | that match any of user triggers. This reduces load on all other parts of Moira. 90 | 91 | 92 | Checker 93 | ^^^^^^^ 94 | 95 | Checker is an application with embedded Graphite functions. Checker watches 96 | for incoming metric values and performs checks according to saved trigger 97 | settings. When state of any trigger changes, Checker generates an event. 98 | 99 | 100 | Notifier 101 | ^^^^^^^^ 102 | 103 | Notifier is an application that watches for generated events. Notifier is 104 | responsible for scheduling and sending notifications, observing quiet hours, 105 | retrying failed notifications, etc. 106 | 107 | 108 | API 109 | ^^^ 110 | 111 | API is an application that serves as a backend for UI. 112 | The Moira API is documented in swagger at https://app.swaggerhub.com/apis/Moira/moira-alert/master 113 | or at the relative path ``/api/swagger/index.html``. 114 | 115 | 116 | Web 2.0 117 | ^^^^^^^ 118 | 119 | Web 2.0 is a frontend React application, it looks like this: 120 | 121 | .. image:: _static/triggers.png 122 | :alt: ui screenshot 123 | -------------------------------------------------------------------------------- /source/release_notes/2_0.rst: -------------------------------------------------------------------------------- 1 | .. _govaluate: https://github.com/Knetic/govaluate 2 | .. _carbonapi: https://github.com/go-graphite/carbonapi/blob/ccac7217894801a5a6ceb8602a70ea0d79e975cf/cmd/carbonapi/COMPATIBILITY.md#functions 3 | .. |supported Graphite functions| replace:: supported Graphite functions 4 | .. _supported Graphite functions: https://github.com/go-graphite/carbonapi/blob/ccac7217894801a5a6ceb8602a70ea0d79e975cf/cmd/carbonapi/COMPATIBILITY.md#functions 5 | 6 | 2.0 7 | --- 8 | 9 | Version 2.0 is fully rewritten in Go instead of Python. This implies lower CPU load in Checker and API microservices, but also changes the list of |supported 10 | Graphite functions|_. 11 | 12 | We also introduce new UI based on React. It is not backwards-compatible with old API, but new API supports both old and new UI. 13 | 14 | 15 | Breaking Changes 16 | ^^^^^^^^^^^^^^^^ 17 | 18 | - New structure of :doc:`../installation/configuration` files. 19 | - New Advanced mode expression format. Moira 2.0 supports govaluate_ expressions instead of Python expressions. Use ``moira-cli -convert-expressions`` to convert. 20 | - API methods URLs do not have trailing slashes anymore. 21 | - API ``/notification`` method returns valid JSON list instead of plain text. 22 | - ``ttl`` parameter in API calls is always a number instead of string. 23 | - API ``PUT`` methods strictly separate create and update operations. 24 | - There is no ``tag maintenance`` entity anymore. 25 | - Error messages return valid JSON instead of plain text. 26 | - Support for Graphite functions changed. See carbonapi_ compatibility list for details. 27 | 28 | 29 | Other Improvements 30 | ^^^^^^^^^^^^^^^^^^ 31 | 32 | - Internal Graphite metric names changed. 33 | - Numerous bugs fixed. Some new were created :) -------------------------------------------------------------------------------- /source/release_notes/2_1.rst: -------------------------------------------------------------------------------- 1 | 2.1 2 | --- 3 | 4 | - Throw an exception if any target except the first one resolves in more than one metric. 5 | - Fix Moira version detection in CI builds. 6 | - Add user login information to API request logs. 7 | - Fix long interval between creating a new trigger and getting data into that trigger. -------------------------------------------------------------------------------- /source/release_notes/2_10_0.rst: -------------------------------------------------------------------------------- 1 | 2.10.0 2 | ===== 3 | 4 | What’s Changed 5 | -------------- 6 | 7 | *Major Changes* 8 | 9 | * Multiple clusters per trigger source. !! Config changes required in checker, notifier and api, 10 | see the docs (:ref:`checker-configuration`, :ref:`notifier-configuration`, :ref:`api-and-web-config`). 11 | https://github.com/moira-alert/moira/pull/981 12 | (by @tetrergeru) 13 | 14 | * Add admin permissions via list of admins in api. 15 | Admins can be configured via config (:ref:`api-and-web-config`) 16 | https://github.com/moira-alert/moira/pull/996 17 | https://github.com/moira-alert/moira/pull/1001 18 | (by @tetrergeru) 19 | 20 | * Fix sending notifications on muted and deleted triggers and muted metrics. 21 | Read more in notifier docs (:ref:`notifier-configuration`) 22 | https://github.com/moira-alert/moira/pull/966 23 | (by @almostinf) 24 | 25 | *Minor Changes* 26 | 27 | * Make triggers with seriesByTag function require at least one argument with a strict equality 28 | https://github.com/moira-alert/moira/pull/988 29 | (by @mchrome) 30 | 31 | * Add metrics for number of contacts 32 | https://github.com/moira-alert/moira/pull/1002 33 | (by @almostinf) 34 | 35 | * Add remove subscriptions command 36 | https://github.com/moira-alert/moira/pull/979 37 | (by @almostinf) 38 | 39 | * Add sentry dsn in web config 40 | https://github.com/moira-alert/moira/pull/973 41 | (by @almostinf) 42 | 43 | * Add platform field to get sentry handler 44 | https://github.com/moira-alert/moira/pull/982 45 | (by @almostinf) 46 | 47 | * Log triggers with critical time of check 48 | https://github.com/moira-alert/moira/pull/987 49 | (by @tetrergeru) 50 | 51 | * Add api route for tags creation 52 | https://github.com/moira-alert/moira/pull/991 53 | (by @almostinf) 54 | 55 | * Add api for getting a single subscription 56 | https://github.com/moira-alert/moira/pull/998 57 | (by @tetrergeru) 58 | 59 | *Fixes* 60 | 61 | * Fix fetching notifications by batch 62 | https://github.com/moira-alert/moira/pull/974 63 | (by @almostinf) 64 | 65 | * Fix nodata when toggling alone metrics checkbox 66 | https://github.com/moira-alert/moira/pull/983 67 | (by @almostinf) 68 | 69 | * Fix panic with alone and regular metrics 70 | https://github.com/moira-alert/moira/pull/984 71 | (by @almostinf) 72 | 73 | * Increase sleep and attempts count for lock 74 | https://github.com/moira-alert/moira/pull/985 75 | https://github.com/moira-alert/moira/pull/986 76 | (by @kissken) 77 | 78 | *Dependencies* 79 | 80 | * Update blackfriday slack to 0.1.2 81 | https://github.com/moira-alert/moira/pull/978 82 | (by @tetrergeru) 83 | 84 | * Bump golang.org/x/crypto from 0.14.0 to 0.17.0 85 | https://github.com/moira-alert/moira/pull/977 86 | (by @dependabot) 87 | 88 | * Bump google.golang.org/protobuf from 1.31.0 to 1.33.0 89 | https://github.com/moira-alert/moira/pull/999 90 | (by @dependabot) 91 | 92 | *Lints* 93 | 94 | * Add decorder, errorlint, loggercheck and noctx lints 95 | https://github.com/moira-alert/moira/pull/960 96 | (by @kissken) 97 | 98 | * Add dots to godocs 99 | https://github.com/moira-alert/moira/pull/997 100 | (by @tetrergeru) 101 | 102 | * Add exportloopref, gocheckcompilerdirectives, goconst, godot, gofumpt, goheader and grouper lints 103 | https://github.com/moira-alert/moira/pull/1000 104 | (by @kissken) -------------------------------------------------------------------------------- /source/release_notes/2_11_0.rst: -------------------------------------------------------------------------------- 1 | 2.11.0 2 | ===== 3 | 4 | What’s Changed 5 | -------------- 6 | 7 | *Incompatible changes* 8 | 9 | * To realize the feature with multiple senders of the same type we have changed the configuration of senders in notifier config and web api config. 10 | * It is necessary to replace the `contacts` field in the web config api with `contacts_template`. 11 | * It is also necessary to replace field `type` with `sender_type` in notifier config and add field `contact_type`, which will refer to `type` in web config `contacts_template`. 12 | 13 | *Major Changes* 14 | 15 | * Add ability to scale senders with same type. 16 | Read more in notifier docs (:ref:`notifier-configuration`) 17 | https://github.com/moira-alert/moira/pull/994 18 | (by @almostinf) 19 | 20 | * Add ability to use template webhook body and custom headers. 21 | Read more in webhook docs 22 | https://github.com/moira-alert/moira/pull/995 23 | (by @almostinf) 24 | 25 | *Minor Changes* 26 | 27 | * Add contacts logo uri to web config. 28 | Read more in api docs (:ref:`api-and-web-config`) 29 | https://github.com/moira-alert/moira/pull/1005 30 | (by @almostinf) 31 | 32 | * Update go to 1.21. 33 | https://github.com/moira-alert/moira/pull/1008 34 | (by @tetrergeru) 35 | 36 | *Fixes* 37 | 38 | * Fix filter connection, heartbeat and create matching handler logs level. 39 | https://github.com/moira-alert/moira/pull/1010 40 | (by @almostinf) 41 | -------------------------------------------------------------------------------- /source/release_notes/2_11_1.rst: -------------------------------------------------------------------------------- 1 | 2.11.1 2 | ===== 3 | 4 | What’s Changed 5 | -------------- 6 | 7 | *Major Changes* 8 | 9 | * Add clean up moira patterns metrics key in cli command ``--cleanup-pattern-metrics``. 10 | Be careful using this command, it can save checker resources a lot, however it changes the default behavior of `transformNull` in Moira, 11 | in fact the moment you use this command the function stops working for metrics that did not come to Moira more than `metrics_ttl`. 12 | If you do a nightly cleanup in cronjob, the behavior of the function becomes similar in Graphite. 13 | https://github.com/moira-alert/moira/pull/1011 14 | https://github.com/moira-alert/moira/pull/1019 15 | (by @almostinf) 16 | 17 | * Add ability to read from slaves in redis cluster. 18 | Read more in the redis configuration of each service (:ref:`checker-configuration`, :ref:`notifier-configuration`, :ref:`api-and-web-config`, :ref:`filter-configuration`). 19 | https://github.com/moira-alert/moira/pull/1025 20 | (by @tetrergeru) 21 | 22 | *Minor Changes* 23 | 24 | * Move batch forced save timeout to filter config. 25 | Read more in filter docs (:ref:`filter-configuration`). 26 | https://github.com/moira-alert/moira/pull/1014 27 | (by @tetrergeru) 28 | 29 | * Add remove future metrics in cli with command ``--cleanup-future-metrics``. 30 | Also added discarding such metrics that are written further into the future than the current time plus `drop_metrics_ttl`. 31 | https://github.com/moira-alert/moira/pull/1012 32 | (by @almostinf) 33 | 34 | * Add ability for admins to create contacts and subscriptions for other users and teams. 35 | https://github.com/moira-alert/moira/pull/1024 36 | (by @tetrergeru) 37 | 38 | * Add custom name field to contacts. 39 | https://github.com/moira-alert/moira/pull/1021 40 | (by @almostinf) 41 | 42 | * Add emoji support in Mattermost notifications. 43 | Read more in notifier docs (:ref:`notifier-configuration`). 44 | https://github.com/moira-alert/moira/pull/1028 45 | (by @kissken) 46 | 47 | * Forbidden create unresolved contact type. 48 | https://github.com/moira-alert/moira/pull/1030 49 | https://github.com/moira-alert/moira/pull/1030 50 | (by @almostinf) 51 | 52 | * Add user authorization info to /api/user/settings. 53 | https://github.com/moira-alert/moira/pull/1034 54 | (by @tetrergeru) 55 | 56 | * Add a cli command ``--remove-unused-triggers-with-ttl`` to remove unsubscribed triggers 57 | that have not been modified within the time passed. 58 | https://github.com/moira-alert/moira/pull/1033 59 | (by @kissken) 60 | 61 | * Remove benchmarks from CI. 62 | https://github.com/moira-alert/moira/pull/1032 63 | (by @almostinf) 64 | 65 | * Improve perfomance of `restoreMetricStringByNameAndLabels` function. 66 | It reduced allocations by more than a factor of two in filters. 67 | https://github.com/moira-alert/moira/pull/1039 68 | https://github.com/moira-alert/moira/pull/1045 69 | (by @almostinf) 70 | 71 | * Add cache for pattern matching handler in filters. 72 | This increased the speed of building the tree by ~20%. 73 | Read more in filter docs (:ref:`filter-configuration`). 74 | https://github.com/moira-alert/moira/pull/1038 75 | (by @almostinf) 76 | 77 | *Fixes* 78 | 79 | * Fix workers lock key in discord and telegram. 80 | https://github.com/moira-alert/moira/pull/1027 81 | (by @almostinf) 82 | 83 | * Increase local `stop_checking_interval`. 84 | https://github.com/moira-alert/moira/pull/1041 85 | (by @almostinf) 86 | 87 | * Don't rewrite error while compare trigger states. 88 | https://github.com/moira-alert/moira/pull/1035 89 | (by @kissken) 90 | 91 | *Dependencies* 92 | 93 | * Update golang image in docker to 1.21. 94 | https://github.com/moira-alert/moira/pull/1026 95 | (by @tetrergeru) 96 | 97 | * Upgrade Mattermost client and improve the look of the notifications in Mattermost. 98 | https://github.com/moira-alert/moira/pull/1023 99 | (by @kissken) 100 | -------------------------------------------------------------------------------- /source/release_notes/2_12_0.rst: -------------------------------------------------------------------------------- 1 | 2.12.0 2 | ===== 3 | 4 | What’s Changed 5 | -------------- 6 | 7 | *Major Changes* 8 | 9 | * Add support group topics in Telegram sender. 10 | The instruction for adding such contacts are the same as in group chats (:ref:`api-and-web-config`). 11 | https://github.com/moira-alert/moira/pull/1044 12 | https://github.com/moira-alert/moira/pull/1022 13 | (by @Gaspero & @almostinf) 14 | 15 | *Minor Changes* 16 | 17 | * Add filling with default values of ``ttl_state``, ``ttl`` and ``sched`` fields in the trigger dto. 18 | https://github.com/moira-alert/moira/pull/1047 19 | (by @AleksandrMatsko) 20 | 21 | * Add validating of ``alone_metrics`` field in trigger dto. 22 | https://github.com/moira-alert/moira/pull/1049 23 | (by @AleksandrMatsko) 24 | 25 | * Accelerate handling of non-strict asterisk tags. 26 | https://github.com/moira-alert/moira/pull/1050 27 | (by @almostinf) 28 | 29 | * Added more info (``clusted_id`` and ``source``) to trigger check failed logs. 30 | https://github.com/moira-alert/moira/pull/1052 31 | (by @kissken) 32 | 33 | * Make configurable rescheduling delay in notifier. 34 | Read more in notifier docs (:ref:`notifier-configuration`). 35 | https://github.com/moira-alert/moira/pull/1046 36 | (by @AleksandrMatsko) 37 | 38 | * Replaced old notifiers logs about plotting with more informative logs with sending notifications. 39 | https://github.com/moira-alert/moira/pull/1054 40 | https://github.com/moira-alert/moira/pull/1057 41 | (by @AleksandrMatsko) 42 | -------------------------------------------------------------------------------- /source/release_notes/2_13_0.rst: -------------------------------------------------------------------------------- 1 | 2.13.0 2 | ===== 3 | 4 | What’s Changed 5 | -------------- 6 | 7 | *Major Changes* 8 | 9 | * Add trigger description to telegram alerts 10 | https://github.com/moira-alert/moira/pull/1075 (by @AleksandrMatsko) 11 | 12 | * fix (api): contact notification history 13 | You need to update with the command ``cli --update --from-version '2.12'``. 14 | https://github.com/moira-alert/moira/pull/1051 (by @AleksandrMatsko) 15 | 16 | 17 | *Minor Changes* 18 | 19 | * fix(filter): fix support custom retention for tagged metrics 20 | https://github.com/moira-alert/moira/pull/950 (by @kissken) 21 | 22 | * feat(api): query params to event history 23 | https://github.com/moira-alert/moira/pull/1072 (by @AleksandrMatsko) 24 | 25 | * fix(notifier): fixed sending of notifications with schedule between different days 26 | https://github.com/moira-alert/moira/pull/1069 (by @almostinf) 27 | 28 | * feat(notifier): add ability to move contacts and subscriptions between users and teams 29 | https://github.com/moira-alert/moira/pull/1067 (by @almostinf) 30 | 31 | * fix(checker): fix skip first metric value 32 | https://github.com/moira-alert/moira/pull/1007 (by @almostinf) 33 | 34 | * fix: counting of trigger events 35 | https://github.com/moira-alert/moira/pull/1076 (by @AleksandrMatsko) 36 | 37 | * fix(config): redis sentinel 38 | https://github.com/moira-alert/moira/pull/1090 (by @almostinf) 39 | 40 | *Dependencies* 41 | 42 | * chore: update go chart version 43 | https://github.com/moira-alert/moira/pull/1070 (by @almostinf) 44 | 45 | * feat(build): Upgrade go 1.22 46 | https://github.com/moira-alert/moira/pull/1062 (by @kissken) 47 | 48 | * build(deps): bump actions/download-artifact from 3 to 4.1.7 in /.github/workflows 49 | https://github.com/moira-alert/moira/pull/1079 (by @dependabot) -------------------------------------------------------------------------------- /source/release_notes/2_14_0.rst: -------------------------------------------------------------------------------- 1 | 2.14.0 2 | ===== 3 | 4 | What’s Changed 5 | -------------- 6 | 7 | *Major* 8 | 9 | * Update cabonapi to v0.17.0 10 | https://github.com/moira-alert/moira/pull/1074 (by @Tetrergeru) 11 | 12 | .. warning:: Please note that triggers with invalid alias expressions (e.g. alias by nonexisting nodes or tags) will break after this update 13 | 14 | * Add remote graphite retries 15 | https://github.com/moira-alert/moira/pull/1085 (by @AleksandrMatsko) 16 | 17 | .. warning:: Please note that retries must be set in configs for each graphite remote, otherwise servises won't start 18 | 19 | * Add team validation 20 | https://github.com/moira-alert/moira/pull/1123 (by @AleksandrMatsko) 21 | 22 | .. warning:: Please note that database migration is required due to this feature. Use ``cli --update --from-version '2.13'`` 23 | 24 | *Features* 25 | 26 | * Add contact template validation 27 | https://github.com/moira-alert/moira/pull/1100 (by @AleksandrMatsko) 28 | 29 | * Add limit for trigger name length 30 | https://github.com/moira-alert/moira/pull/1080 (by @AleksandrMatsko) 31 | 32 | * Optimize hot paths in filter 33 | https://github.com/moira-alert/moira/pull/1114, 34 | https://github.com/moira-alert/moira/pull/1115 35 | (by @Tetrergeru) 36 | 37 | * Update sprig 38 | https://github.com/moira-alert/moira/pull/1125 (by @kissken) 39 | 40 | * Add api to get all teams 41 | https://github.com/moira-alert/moira/pull/1126 (by @AleksandrMatsko) 42 | 43 | * feat: Add retry backoffs and max redirects to database config 44 | https://github.com/moira-alert/moira/pull/1068 45 | https://github.com/moira-alert/moira/pull/1128 (by @Tetrergeru) 46 | 47 | * Add celebration mode 48 | https://github.com/moira-alert/moira/pull/1129 (by @kissken) 49 | 50 | * Add remote graphite trigger validation 51 | https://github.com/moira-alert/moira/pull/1127 (by @AleksandrMatsko) 52 | 53 | *Fixes* 54 | 55 | * Add validation for prometheus target 56 | https://github.com/moira-alert/moira/pull/1077 (by @AleksandrMatsko) 57 | 58 | * Trigger schedule fill 59 | https://github.com/moira-alert/moira/pull/1116 (by @AleksandrMatsko) 60 | -------------------------------------------------------------------------------- /source/release_notes/2_14_1.rst: -------------------------------------------------------------------------------- 1 | 2.14.1 2 | ===== 3 | 4 | What’s Changed 5 | -------------- 6 | 7 | *Deprecation warning* 8 | 9 | * Unify contact dto between /team/settings and /contact (#1138) (by @Tetrergeru) 10 | 11 | .. warning:: Please note, that field ``team`` in contacts dto is deprecated and will be removed in the next version. Please use field ``team_id`` instead. 12 | 13 | *Features* 14 | 15 | * Trigger noisiness 16 | https://github.com/moira-alert/moira/pull/1131 (by @AleksandrMatsko) 17 | 18 | * Validate trigger patterns 19 | https://github.com/moira-alert/moira/pull/1136 (by @AleksandrMatsko) 20 | 21 | * Take pager TTL from config instead of hardcoded const 22 | https://github.com/moira-alert/moira/pull/1139 (by @HeavyPunk) 23 | 24 | *Fixes and chores* 25 | 26 | * Notifier metric alive count 27 | https://github.com/moira-alert/moira/pull/1130 (by @AleksandrMatsko) 28 | 29 | * Bump deps 30 | https://github.com/moira-alert/moira/pull/1132 (by @AleksandrMatsko) 31 | 32 | * Checker handle source unavailable 33 | https://github.com/moira-alert/moira/pull/1134 (by @AleksandrMatsko) 34 | 35 | * Fix swagger type for typeOfProblem 36 | https://github.com/moira-alert/moira/pull/1141 (by @kissken) 37 | 38 | * Change log to warning if remote unavailable 39 | https://github.com/moira-alert/moira/pull/1142 (by @HeavyPunk) 40 | -------------------------------------------------------------------------------- /source/release_notes/2_15_0.rst: -------------------------------------------------------------------------------- 1 | 2.15.0 2 | ===== 3 | 4 | What's Changed 5 | -------------- 6 | 7 | *Features* 8 | 9 | *Major* 10 | 11 | * feat(api): added full trigger validation in api by @Tetrergeru in https://github.com/moira-alert/moira/pull/957 12 | * feat(api): move from swagger v2 to swagger v3 by @kissken 13 | * feat: system subscriptions on moira's selfstate monitor events. Enable moira automatically when selfstate detects no problem (#1166 #1179 #1190 #1152 #1164). By @HeavyPunk 14 | * feat(build): Update go into 1.23 by @kissken in https://github.com/moira-alert/moira/pull/1186 15 | 16 | *Minor* 17 | 18 | * feat: contact noisiness by @AleksandrMatsko in https://github.com/moira-alert/moira/pull/1137 19 | * feat: senders delivery metrics by @AleksandrMatsko in https://github.com/moira-alert/moira/pull/1143 20 | * fix: metric maintenace disappearance by @AleksandrMatsko in https://github.com/moira-alert/moira/pull/1145 21 | * feat: insecure tls option for webhook by @AleksandrMatsko in https://github.com/moira-alert/moira/pull/1154 22 | * feat: db methods for delivery check and new metric for sender by @AleksandrMatsko in https://github.com/moira-alert/moira/pull/1149 23 | * feat: controller for delivery checks by @AleksandrMatsko in https://github.com/moira-alert/moira/pull/1157 24 | * feat: Pass redis connection pool settings to config by @Tetrergeru in https://github.com/moira-alert/moira/pull/1140 25 | * feat(api): Add ability filer who can delete triggers by @kissken in https://github.com/moira-alert/moira/pull/1184 26 | * feat(api): Add ability filer who can change triggers by @kissken in https://github.com/moira-alert/moira/pull/1189 27 | * feat: webhook sender delivery checks by @AleksandrMatsko in https://github.com/moira-alert/moira/pull/1162 28 | * feat(api): Delete notifications filtered by tags by @Tetrergeru in https://github.com/moira-alert/moira/pull/1187 29 | 30 | *Fixes and chores* 31 | 32 | * chore: updated linters by @kissken 33 | * chore: Update mockgen to v0.5.1 by @Tetrergeru in https://github.com/moira-alert/moira/pull/1188 34 | * fix(filter): Fix metric overrides by @Tetrergeru in https://github.com/moira-alert/moira/pull/1183 35 | * build: Update dependency by @kissken in https://github.com/moira-alert/moira/pull/1195 36 | 37 | 38 | 39 | **Full Changelog**: https://github.com/moira-alert/moira/compare/v2.14.1...v2.15.0 40 | -------------------------------------------------------------------------------- /source/release_notes/2_2.rst: -------------------------------------------------------------------------------- 1 | 2.2 2 | --- 3 | 4 | - Add Redis Sentinel support. 5 | - Increase new metric event processing speed by adding a cache on metric patterns. 6 | - Update carbonapi (new functions: map, reduce, delay; updated: asPercent). 7 | - Optimize reading metrics while checking trigger (removed unnecessary Redis transaction). 8 | - Add domain autoresolving for self-metrics sending to Graphite. 9 | - Fix concurrent read/write from expression cache. 10 | - Re-enable Markdown in Slack sender. 11 | - Optimize internal metric collection. 12 | - Replace pseudotags with ordinary checkboxes in Web UI (but not on backend yet). 13 | - Fix bug that allowed to create pseudotags (ERROR, etc.) as ordinary tags. 14 | - Add metrics for each trigger handling time. 15 | - Translate pagination. 16 | - Make sorting by status the default option on trigger page. 17 | - Hide tag list on trigger edit page. 18 | - Sort tags alphabetically everywhere. 19 | - Highlight metric row on mouse hover. 20 | - Automatically add tags from search bar when creating new trigger. 21 | - Add metric name to "Trigger has same timeseries names" error message. 22 | - Update event names in case trigger name had changed. 23 | - Fix bug in triggers with multiple targets. Metrics from targets T2, T3, ... were not deleted properly. 24 | - Fix old-style configuration files in platform-specific packages. 25 | - Fix bug that prevented non-integer timestamps from processing. 26 | - Fix logo image background. 27 | - Fix sorting on -s and 0s. 28 | - Fix UI glitch while setting maintenance time. 29 | - Fix retention scheme parsing for some rare cases with comments. -------------------------------------------------------------------------------- /source/release_notes/2_3.rst: -------------------------------------------------------------------------------- 1 | 2.3 2 | --- 3 | 4 | - Add API methods: ``DELETE /notification/all`` and ``DELETE /event/all`` `moira-alert/moira#73 `_. 5 | - Add notifier config option: DateTime format for email sender `moira-alert/moira#74 `_. 6 | - Add Graphite-API support for remote triggers `moira-alert/moira#75 `_. See more: :ref:`graphite-remote-triggers-checker`. Thanks to `@errx `_. 7 | - Fix newlines in trigger description body for web and email sender `moira-alert/moira#76 `_. 8 | - Add option to enable runtime metrics in Graphite-section of configuration `moira-alert/moira#79 `_. 9 | - Add new fancy email template 🎂 `moira-alert/moira#82 `_. 10 | - Change default trigger state to TTLState option instead of NODATA `moira-alert/moira#83 `_. 11 | - Refactor maintenance logic `moira-alert/moira#87 `_. See more: :doc:`/user_guide/maintenance`. 12 | - Add basic false NODATA protection `moira-alert/moira#90 `_. See more: :doc:`/user_guide/selfstate`. 13 | - Prohibit removal of contact with assigned subscriptions found `moira-alert/moira#91 `_. 14 | - Make trigger exception messages more descriptive `moira-alert/moira#92 `_. 15 | - Make filter cache capacity configurable `moira-alert/moira#93 `_. See more :ref:`Filter Configuration `. 16 | - Fix incorrect behavior in which the trigger did not return from the ``EXCEPTION`` state `moira-alert/moira#94 `_. 17 | - Remove deprecated pseudo-tags, use checkboxes instead `moira-alert/moira#95 `_. See more: :ref:`subscription-states-transitions`. 18 | - Allow to use single-valued thresholds (ex. only ``WARN`` or only ``ERROR``) `moira-alert/moira#96 `_. 19 | - Reduce the useless CPU usage in Moira-Filter `moira-alert/moira#98 `_. Thanks to `@errx `_. 20 | - Add concurrent matching workers in Moira-Filter `moira-alert/moira#99 `_. Thanks to `@errx `_. 21 | - Update Carbonapi to 1.0.0-rc.0 `moira-alert/moira#101 `_. 22 | - Improve checker performance `moira-alert/moira#103 `_. 23 | - Add Markdown support in contact edit modal view `moira-alert/web2.0#138 `_. 24 | - Fix default timezone in trigger `moira-alert/web2.0#173 `_. 25 | - Add ability to type negative numbers in simple trigger edit mode `moira-alert/web2.0#169 `_. 26 | - Fix trailing whitespaces in tag search bar `moira-alert/web2.0#139 `_. 27 | - Update `Moira Client 2.3.4 `_. 28 | - Update `Moira Trigger Role 2.3 `_. 29 | 30 | .. important:: **Redis DB conversion is desirable.** 31 | 32 | Moira 2.3 has some structure changes in Redis DB. 33 | It will work fluently out of the box, but we recommend you to run converter once Moira is updated. 34 | 35 | .. code-block:: bash 36 | 37 | moira-cli -update --config=/etc/moira/cli.yml 38 | 39 | .. code-block:: YAML 40 | :name: cli.yml 41 | :caption: /etc/moira/cli.yml 42 | 43 | redis: 44 | host: localhost 45 | port: "6379" 46 | dbid: 0 47 | log_file: stdout 48 | log_level: debug 49 | 50 | If you would like to downgrade back to Moira 2.2, you should run CLI-converter. 51 | 52 | .. code-block:: bash 53 | 54 | moira-cli -downgrade --config=/etc/moira/cli.yml 55 | 56 | Both cases imply usage of Moira-Cli v.2.3, you can find it on `Release Page `_. 57 | 58 | 59 | 2.3.1 60 | ----- 61 | 62 | - Fix ``last_remote_check_delay`` option in :ref:`Notifier configuration ` `moira-alert/moira#114 `_. -------------------------------------------------------------------------------- /source/release_notes/2_4_0.rst: -------------------------------------------------------------------------------- 1 | 2.4.0 2 | ----- 3 | 4 | - Timeseries graphs in notifications `moira-alert/moira#148 `_. See more :ref:`subscriptions-plotting`. 5 | - Add api method ``GET trigger/{{triggerId}}/render`` to imlement timeseries plotting in api `moira-alert/moira#137 `_. 6 | - Add maintenance for a whole trigger. Add new api method ``PUT trigger/{{triggerId}}/setMaintenance``. ``PUT trigger/{{triggerId}}/maintenance`` is deprecated now `moira-alert/moira#138 `_, `moira-alert/web2.0#199 `_. 7 | - Add extra maintenance intervals: 14 and 30 days `moira-alert/web2.0#198 `_. 8 | - Add option to mute notifications about new metrics in the trigger `moira-alert/moira#120 `_. See more: :doc:`/user_guide/nodata`. 9 | - Allow user to remove all ``NODATA`` metrics from trigger `moira-alert/moira#124 `_. 10 | - Check Lazy triggers (triggers without any subscriptions) less frequently `moira-alert/moira#131 `_. See more :ref:`lazy-triggers-checker`. 11 | - Run single NODATA checker worker at single moment `moira-alert/moira#129 `_. 12 | - Avoid throttling of remote-triggers when trigger switches to ``EXCEPTION`` and back to ``OK`` `moira-alert/moira#121 `_. 13 | - Consider the status of the trigger when rendering the trigger status indicator `moira-alert/web2.0#195 `_. 14 | - Replace useless trigger export button with "Duplicate" `moira-alert/web2.0#189 `_. 15 | - Add Moira-Notifier toggle on :doc:`/user_guide/hidden_pages` `moira-alert/web2.0#191 `_. 16 | **Please, read** :doc:`/user_guide/selfstate` **first**. 17 | - Show contact type icon on :doc:`/user_guide/hidden_pages` `moira-alert/web2.0#196 `_. 18 | - Show TTL and TTLState in Advanced mode `moira-alert/web2.0#197 `_. 19 | - Throw an exception if first target is no longer valid `moira-alert/moira#122 `_. 20 | - Refactor cli. Remove old converters, whiсh were written before moira 2.2 `moira-alert/moira#139 `_. 21 | - Update golang to version 1.11.2 `moira-alert/moira#147 `_. 22 | - Flush trigger events when removing the trigger `moira-alert/moira#116 `_. 23 | - Remove redundant Graphite-metrics that counted the time of check of each single trigger `moira-alert/moira#117 `_. 24 | - Add api method ``GET trigger/search`` to implement full-text trigger search in api, ``GET trigger/page`` is deprecated now `moira-alert/moira#125 `_. 25 | - Fix Redis leakages: some data was not removed properly from Redis storage `moira-alert/moira#129 `_. 26 | - Fix bug in trigger schedule due to which triggers were considered suppressed between 23:59:00 and 00:00:59 `moira-alert/moira#127 `_. 27 | - Fix bug in trigger when specific schedule time didn't work if start time was bigger than end time `moira-alert/moira#119 `_. 28 | - Fix bug in ``Create and test`` button when add new subscription `moira-alert/web2.0#194 `_. 29 | - Fix bug that increases updated last checks count when user create or update trigger from api (or web) `moira-alert/moira#146 `_. 30 | - Fix bug which allowed to use other people's contacts your in subscriptions `moira-alert/moira#145 `_. 31 | - Fix bug that allowed to create and use an empty tag in subscriptions and triggers `moira-alert/moira#144 `_. 32 | - Fix bug when senders didn't resolve ``EXCEPTION`` state `moira-alert/moira#156 `_. 33 | - Update `Moira Client 2.4 `_. 34 | - Update `Moira Trigger Role 2.4 `_. 35 | 36 | .. _moira-cli-2-4-0: 37 | 38 | .. important:: **Redis DB conversion is required.** 39 | 40 | Moira 2.4 has some structure changes in Redis DB. 41 | It will work fluently out of the box, but lazy triggers will still be checked every time on new metrics. 42 | 43 | You can upgrade from moira 2.2 or 2.3 using corresponding flag in ``--from-version`` variable. 44 | 45 | .. code-block:: bash 46 | 47 | moira-cli --config=/etc/moira/cli.yml --update --from-version=2.2/2.3 48 | 49 | If you would like to downgrade back to Moira 2.2 or 2.3, you should run CLI-converter. 50 | 51 | .. code-block:: bash 52 | 53 | moira-cli --config=/etc/moira/cli.yml --downgrade --to-version=2.2/2.3 54 | 55 | Both cases imply usage of Moira-Cli v.2.4, you can find it on `Release Page `_. 56 | -------------------------------------------------------------------------------- /source/release_notes/2_5_0.rst: -------------------------------------------------------------------------------- 1 | .. _important: https://moira.readthedocs.io/en/release-2.5/release_notes/2_4_0.html 2 | .. _upgrading: https://moira.readthedocs.io/en/release-2.5/release_notes/2_5_0.html#upgrading 3 | .. |incompatible changes| replace:: incompatible changes 4 | .. _incompatible changes: https://moira.readthedocs.io/en/release-2.5/release_notes/2_5_0.html#incompatible-changes 5 | .. |version 2.4| replace:: version 2.4 6 | .. _version 2.4: https://github.com/moira-alert/moira/releases/tag/v2.4.0 7 | .. |WEB UI configuration guide| replace:: WEB UI configuration guide 8 | .. _WEB UI configuration guide: https://moira.readthedocs.io/en/release-2.5/installation/configuration.html#web-ui 9 | .. |default WEB UI configuration| replace:: default WEB UI configuration 10 | .. _default WEB UI configuration: https://github.com/moira-alert/moira/blob/b8523885f003fceeefe98ca30be2b42d70032794/pkg/api/web.json 11 | .. |email template| replace:: email template 12 | .. _email template: https://moira.readthedocs.io/en/release-2.5/installation/configuration.html#email-template 13 | .. _documantation: https://moira.readthedocs.io/en/release-2.5/installation/webhooks_scripts.html 14 | 15 | 16 | 2.5.0 17 | ===== 18 | 19 | Upgrading 20 | --------- 21 | 22 | .. warning:: 23 | This release is not compatible with Redis version below 3.2, please upgrade your Redis instance. 24 | 25 | .. warning:: 26 | It is not possible to upgrade from Moira 2.2 to Moira 2.5 directly. To upgrade Moira from version 2.2 or older to 2.5, please first run moira-cli |version 2.4|_ (see :ref:`important ` note). 27 | 28 | Please update your web configuration according to the following rules: 29 | 30 | - Add ``label`` (new required field). You can see default type and label field mappings in |default WEB UI configuration|_. 31 | - Rename ``title`` to ``placeholder``. 32 | 33 | See |WEB UI configuration guide|_ for more information. 34 | 35 | 36 | Incompatible changes 37 | -------------------- 38 | 39 | - Removed deprecated method ``PUT trigger/{{triggerId}}/maintenance``. Use ``PUT trigger/{{triggerId}}/setMaintenance`` instead (request body has not changed). 40 | - Removed deprecated version 2.2 related conversion code. Now if you want to upgrade Moira from version 2.2 or older use moira-cli |version 2.4|_ (see important_ note). 41 | - Fixed contact types configuration. It was hardcoded in the web UI, and now it is configurable via config file (as it should have been originally). 42 | - Renamed the ``title`` field to more semantically correct ``placeholder`` in web UI config. 43 | - Added a new required ``label`` field to web UI config, which is used as a contact label in web UI instead of ``type``. 44 | - Removed ``ERROR_VALUE`` and ``WARN_VALUE`` as valid variables in expression. Old triggers with these variables will still work, but you can not update these triggers until you delete ``ERROR_VALUE`` and ``WARN_VALUE`` variables from the expression. 45 | - Dropped support for all old browsers. Only last 2 major versions of Chrome, Firefox, Safari (all mobile and desktop) and Edge (only desktop) are supported. 46 | 47 | New features 48 | ------------ 49 | 50 | - Added Graphite tags support `#142 `_. 51 | - Reworked trigger search input control in web UI. Fulltext search is now available, as long as the old tag filters `#185 `_. 52 | - Added Webhook sender `#123 `_. For more info see documantation_. 53 | - Added information who and when turned on maintenance mode. You can see it as a hint in web UI near the metric, and in metric alert message `#192 `_. 54 | - Added a meaningful title to all Moira web pages `#177 `_. 55 | - Added environment variable that customizes api URL path for web UI Docker image `#173 `_. 56 | - Added new variables to script sender. Variable ${trigger_name} is now deprecated, removed from documentation and will be removed in the future versions of Moira `#228 `_. For more information about new variables and script configuration see documantation_. 57 | 58 | 59 | Bug fixes and improvements 60 | -------------------------- 61 | 62 | - Limited connection count in Redis connection pool, added a separate pool for remote locks, added ConnectionsLimit config field in Redis configuration `#163 `_. 63 | - Prohibited saving trigger with both ``expression`` and ``warn_value`` + ``error_value``. If you set both of these fields, API will return 400 status code. Web UI saves only fields that are displayed in the open tab (simple or advanced mode) `#172 `_. 64 | - Fixed handling incoming metrics with Windows-style line breaks (/r/n) `#268 `_. 65 | - Fixed checking of triggers that do not have any metrics stored in remote or local storage `#166 `_. 66 | - Fixed execution of self-checks: do it only once, regardless of how many notifier instances are running `#186 `_. 67 | - Fixed response code on invalid requests to update or create trigger (was 500, now 400) `#323 `_. 68 | - Combined telegram alert and plot in one message `#248 `_. 69 | - Added icons in Slack notifications `#180 `_. See more: :ref:`slack-icons`. 70 | - Got rid of old ugly mail template, now we use only new |email template|_. `#181 `_. 71 | - Fixed bug that turned old pseudo-tags ``ERROR`` ``DEGRADATION`` ``HIGH DEGRADATION`` to subscription settings checkboxes `#184 `_. 72 | - Fixed advanced schedule in subscriptions `#162 `_. 73 | - Fixed multibyte splitting bug in graph titles `#179 `_. 74 | - Fixed sending a message "This metric changed its state..." if a state does not change during maintenance interval `#328 `_. 75 | - Removed useless broken links in test and self-state notifications `#178 `_. 76 | - Fixed symbols counting bug in telegram messages `#248 `_. 77 | - Changed mobile detection logic from "get window width" to "parse user agent and detect mobile browser" `#218 `_. 78 | - Fixed 500 status code then trying to update subscription if one of the subscribed triggers was removed `#271 `_. 79 | - Properly encoded parameters that are passed in a web to API requests `#174 `_. 80 | - Fixed layout with long words or URLs in name and description on the trigger web page `#176 `_. 81 | - Fixed showing tags that exist in the user local browser storage, but don't exist in server-side `#175 `_. 82 | - Fixed external loader on non-existing trigger page in a mobile version of the web `#168 `_. 83 | - Removed cancel button and restyled delete button in subscription modal `#221 `_. 84 | - Prohibited creating simple mode trigger with several targets via API `#171 `_. 85 | - Fixed data source toggle that was missing from simple edit trigger mode `#236 `_. 86 | - Fixed rising/falling mode selector when switching between simple and advanced modes `#172 `_. 87 | - Limited Twilio SMS sender to 5 events per SMS `#237 `_. 88 | - Fixed Pushover message priority calculation when sending over five events `#237 `_. 89 | - Added contact type icon in choose subscription contact combobox `#219 `_. 90 | - Changed remove subscription contact icon `#220 `_. 91 | - Improved plotting conditions to render non-empty timeseries only `#197 `_. 92 | - Upgraded NPM dependencies for security reasons `#194 `_. 93 | - Added log message describing the reason why self-state monitor disabled notifications `#323 `_. -------------------------------------------------------------------------------- /source/release_notes/2_5_1.rst: -------------------------------------------------------------------------------- 1 | 2.5.1 2 | ===== 3 | 4 | Upgrading 5 | --------- 6 | 7 | Config for web is moved to config for API. Please read :ref:`api-and-web-config` to detect the changes and merge two configs. Old config for web is not needed anymore. 8 | 9 | Incompatible changes 10 | -------------------- 11 | 12 | - Frontend and web configs are merged to one file `#360 `_. 13 | 14 | New features 15 | ------------ 16 | 17 | - Added ability to subscribe for all triggers without specifying tags `#236 `_. 18 | - Added ability to send markdown for email, fix markdown formatting in slack senders `#353 `_. 19 | - Added new senders: Discord, VictorOps, PagerDuty, OpsGenie. 20 | - ⚡️✨💫🔥🔥🔥 Graphs now support emojis `#333 `_. 21 | - Y-axis graph now uses algorithm to define "beautiful" ticks `#217 `_. 22 | 23 | Bug fixes and improvements 24 | -------------------------- 25 | 26 | - Added support for magic -1 timestamp `#426 `_. 27 | - Fixed incorrect timezone in maintenance notification text `#356 `_. 28 | - Dependency management switched to Go modules mechanism `#423 `_. 29 | - Linter was switched to GolangCI Lint `#436 `_. 30 | - Go version was switched to 1.13.1 `#435 `_. 31 | - Alert which contain NODATA now uses timestamp of NODATA detection instead of data loose time `#355 `_. 32 | - Readyness and liveness probes delay was upgraded in helm chart to fit long triggers indexing in database `#2 `_. 33 | - API now exits with error if unable to index triggers for full-text search `#327 `_. 34 | - Deleting tags that are used in existing subscriptions is now disallowed `#344 `_. 35 | -------------------------------------------------------------------------------- /source/release_notes/2_6_2.rst: -------------------------------------------------------------------------------- 1 | 2.6.2 2 | ===== 3 | 4 | What’s Changed 5 | -------------- 6 | 7 | - Changed naming for feature docker images by @litleleprikon in 8 | https://github.com/moira-alert/moira/pull/440 9 | - Added subscription transfer and contact deletion to CLI by @Nixolay 10 | in https://github.com/moira-alert/moira/pull/443 11 | - Added Msteams support by @imavroukakis in 12 | https://github.com/moira-alert/moira/pull/432 13 | - Upgraded golangci-lint to v1.21.0 by @titusjaka in 14 | https://github.com/moira-alert/moira/pull/439 15 | - Fixed memory leak in Scorch-type index in Bleve by @Nixolay in 16 | https://github.com/moira-alert/moira/pull/444 17 | - Made image cli by @Nixolay in 18 | https://github.com/moira-alert/moira/pull/454 19 | - Checked nil pointer by @Nixolay in 20 | https://github.com/moira-alert/moira/pull/455 21 | - Fixed nil pointer dereference in notifier by @litleleprikon in 22 | https://github.com/moira-alert/moira/pull/452 23 | - Added twimlets support for Twilio sender by @prizov in 24 | https://github.com/moira-alert/moira/pull/450 25 | - Panic in filter by @Nixolay in 26 | https://github.com/moira-alert/moira/pull/467 27 | - Added cleaning moira from fired users by @Nixolay in 28 | https://github.com/moira-alert/moira/pull/458 29 | - Cleaned up metrics interface by @Pliner in 30 | https://github.com/moira-alert/moira/pull/475 31 | - Run goimports by @Pliner in 32 | https://github.com/moira-alert/moira/pull/478 33 | - Metrics facade by @Pliner in 34 | https://github.com/moira-alert/moira/pull/477 35 | - Reworked Counter.Inc and drop NewMetersCollection from Registry by 36 | @Pliner in https://github.com/moira-alert/moira/pull/479 37 | - Set up metrics prefix at start up by @Pliner in 38 | https://github.com/moira-alert/moira/pull/480 39 | - Added CODEOWNERS by @Pliner in 40 | https://github.com/moira-alert/moira/pull/483 41 | - Extracted telemetry config and setup telemetry server by @Pliner in 42 | https://github.com/moira-alert/moira/pull/482 43 | - Added Prometheus endpoint for internal metrics by @Pliner in 44 | https://github.com/moira-alert/moira/pull/474 45 | - Fixed incorrect parsing of multiple equal signs in label by @idoqo in 46 | https://github.com/moira-alert/moira/pull/490 47 | - Added community guides by @litleleprikon in 48 | https://github.com/moira-alert/moira/pull/485 49 | - Moved plot boundaries by @A1bemuth in 50 | https://github.com/moira-alert/moira/pull/495 51 | - Reduced nodes slice capacity by @A1bemuth in 52 | https://github.com/moira-alert/moira/pull/496 53 | - Implemented stable pagination by @litleleprikon in 54 | https://github.com/moira-alert/moira/pull/498 55 | - Enabled pprof heap handler by @A1bemuth in 56 | https://github.com/moira-alert/moira/pull/497 57 | - Handled pager params on /page url by @litleleprikon in 58 | https://github.com/moira-alert/moira/pull/502 59 | - Made filter optimizations by @A1bemuth in 60 | https://github.com/moira-alert/moira/pull/503 61 | - Upgraded to Go 1.14 by @beevee in 62 | https://github.com/moira-alert/moira/pull/504 63 | - Updated golang/mock to 1.4.1 by @A1bemuth in 64 | https://github.com/moira-alert/moira/pull/507 65 | - Updated GolangCI Lint version by @litleleprikon in 66 | https://github.com/moira-alert/moira/pull/509 67 | - Added reviewdog by @litleleprikon in 68 | https://github.com/moira-alert/moira/pull/468 69 | - Improved docker-compose by @litleleprikon in 70 | https://github.com/moira-alert/moira/pull/515 71 | - Fixed pattern matching in filter by @beevee in 72 | https://github.com/moira-alert/moira/pull/517 73 | - Added slave replicas read by @A1bemuth in 74 | https://github.com/moira-alert/moira/pull/510 75 | - Added cap on metric fetching to prevent checker OOM by @beevee in 76 | https://github.com/moira-alert/moira/pull/519 77 | - Added metrics for every possible state transition by @beevee in 78 | https://github.com/moira-alert/moira/pull/527 79 | - Metrics mixed up in graph legend by @Nixolay in 80 | https://github.com/moira-alert/moira/pull/526 81 | - Test on a non-primary database by @Nixolay in 82 | https://github.com/moira-alert/moira/pull/529 83 | - Added templates to trigger description #484 by @Nixolay in 84 | https://github.com/moira-alert/moira/pull/487 85 | - Added limit to FetchNotifications function for read notifications 86 | from db by @ifireice in https://github.com/moira-alert/moira/pull/505 87 | - Improved advanced mode by @litleleprikon in 88 | https://github.com/moira-alert/moira/pull/470 89 | - Rewrote self-state check by @Nixolay in 90 | https://github.com/moira-alert/moira/pull/417 91 | - Changed regexp submatch index by @litleleprikon in 92 | https://github.com/moira-alert/moira/pull/534 93 | - Added alone metrics to get trigger reply by @litleleprikon in 94 | https://github.com/moira-alert/moira/pull/538 95 | - Rewrote validation for empty targets by @litleleprikon in 96 | https://github.com/moira-alert/moira/pull/540 97 | - Now we save all evaluated metrics not only from T1 target to compere 98 | it then save trigger by @borovskyav in 99 | https://github.com/moira-alert/moira/pull/541 100 | - Do not use t1: prefix in trigger alerts that have only one target by 101 | @borovskyav in https://github.com/moira-alert/moira/pull/539 102 | - Fixed alone metrics check error message by @borovskyav in 103 | https://github.com/moira-alert/moira/pull/543 104 | - Improved performance of check by @litleleprikon in 105 | https://github.com/moira-alert/moira/pull/542 106 | - Added test selfstate by @Nixolay in 107 | https://github.com/moira-alert/moira/pull/546 108 | - Improved alone metrics error message by @litleleprikon in 109 | https://github.com/moira-alert/moira/pull/547 110 | - Allowed stale read for pattern metrics by @litleleprikon in 111 | https://github.com/moira-alert/moira/pull/549 112 | - Changed default metric name to T1 by @litleleprikon in 113 | https://github.com/moira-alert/moira/pull/548 114 | - Denied usage of asterisk pattern by @litleleprikon in 115 | https://github.com/moira-alert/moira/pull/555 116 | - Added debug exception by @Nixolay in 117 | https://github.com/moira-alert/moira/pull/552 118 | - Fixed goroutines leak in filter by @litleleprikon in 119 | https://github.com/moira-alert/moira/pull/562 120 | - Added tagging current master branch with latest tag by @beevee in 121 | https://github.com/moira-alert/moira/pull/565 122 | - Fixed incorrect shutdown and conflict of data types during output in 123 | MetricsMatcher. by @JIexa24 in 124 | https://github.com/moira-alert/moira/pull/566 125 | - Fixed multiple connections closing in Moira-Filter caused by PR-562. 126 | by @JIexa24 in https://github.com/moira-alert/moira/pull/570 127 | - Updated golang by @Nixolay in 128 | https://github.com/moira-alert/moira/pull/571 129 | - Added return ‘not found’ when rendering non-existing trigger chart by 130 | @idoqo in https://github.com/moira-alert/moira/pull/572 131 | - Added more linters by @zhelyabuzhsky in 132 | https://github.com/moira-alert/moira/pull/573 133 | - Fixed full-text search if the text is in uppercase by @zhelyabuzhsky 134 | in https://github.com/moira-alert/moira/pull/574 135 | - Added private channels support by @zhelyabuzhsky in 136 | https://github.com/moira-alert/moira/pull/578 137 | - Updated Go to 1.15.2 by @zhelyabuzhsky in 138 | https://github.com/moira-alert/moira/pull/579 139 | - Fixed Telegram group chat response message by @zhelyabuzhsky in 140 | https://github.com/moira-alert/moira/pull/582 141 | - Improved sendAsAlbum Telegram function by @zhelyabuzhsky in 142 | https://github.com/moira-alert/moira/pull/581 143 | - Fixed sending of plots in notifications by @zhelyabuzhsky in 144 | https://github.com/moira-alert/moira/pull/580 145 | - Added support of Slack user-group mentioning in the alert message by 146 | @ArXa1L in https://github.com/moira-alert/moira/pull/585 147 | - Fixed CarbonAPI pow function by @zhelyabuzhsky in 148 | https://github.com/moira-alert/moira/pull/586 149 | - Bumped golangci-lint version by @beevee in 150 | https://github.com/moira-alert/moira/pull/587 151 | - Updated templates by @Nixolay in 152 | https://github.com/moira-alert/moira/pull/536 153 | - Removed “parse” post message argument by @ArXa1L in 154 | https://github.com/moira-alert/moira/pull/588 155 | - Fixed Telegram group chat response message by @zhelyabuzhsky in 156 | https://github.com/moira-alert/moira/pull/589 157 | - Made responding only to messages beginning with /start in Telegram by 158 | @beevee in https://github.com/moira-alert/moira/pull/590 159 | - Allowed targets be single if it not declared by @litleleprikon in 160 | https://github.com/moira-alert/moira/pull/554 161 | - Marked all dangerous Graphite functions as such by @Nixolay in 162 | https://github.com/moira-alert/moira/pull/531 163 | - Improved logging by @androndo in 164 | https://github.com/moira-alert/moira/pull/599 165 | - Cleaned last check on trigger update by @litleleprikon in 166 | https://github.com/moira-alert/moira/pull/596 167 | - Removed populate check in trigger update by @litleleprikon in 168 | https://github.com/moira-alert/moira/pull/602 169 | - Added error logging in notifier by @litleleprikon in 170 | https://github.com/moira-alert/moira/pull/604 171 | - Cloned logger by @androndo in 172 | https://github.com/moira-alert/moira/pull/605 173 | - Fixed logging place by @litleleprikon in 174 | https://github.com/moira-alert/moira/pull/606 175 | - Updated Slack client by @androndo in 176 | https://github.com/moira-alert/moira/pull/608 177 | - Switched to github actions instead of travis CI by @litleleprikon in 178 | https://github.com/moira-alert/moira/pull/610 179 | - Detailed logs by @androndo in 180 | https://github.com/moira-alert/moira/pull/600 181 | - Disabled excluded logs if plots by @androndo in 182 | https://github.com/moira-alert/moira/pull/612 183 | - Changed trigger/check method to PUT and body params by @androndo in 184 | https://github.com/moira-alert/moira/pull/611 185 | - Detected broken contacts by @androndo in 186 | https://github.com/moira-alert/moira/pull/615 187 | - Added metrics export by @litleleprikon in 188 | https://github.com/moira-alert/moira/pull/613 189 | - Made expression not in uppercase only by @balalay12 in 190 | https://github.com/moira-alert/moira/pull/622 191 | - **Added team subscriptions and contacts** by @litleleprikon in 192 | https://github.com/moira-alert/moira/pull/537 193 | - Added pager deletion by @litleleprikon in 194 | https://github.com/moira-alert/moira/pull/623 195 | - Added strings methods to templating functions by @androndo in 196 | https://github.com/moira-alert/moira/pull/624 197 | - Fixed api bugs by @litleleprikon in 198 | https://github.com/moira-alert/moira/pull/628 199 | - Fixed http schema escaping when build url in webhook sender by 200 | @androndo in https://github.com/moira-alert/moira/pull/627 201 | - Moved coverage to codecov by @androndo in 202 | https://github.com/moira-alert/moira/pull/630 203 | - Fixed checker bugs by @litleleprikon in 204 | https://github.com/moira-alert/moira/pull/621 205 | - Fixed prepare test by @litleleprikon in 206 | https://github.com/moira-alert/moira/pull/644 207 | - Simplified ConvertForCheck function by @litleleprikon in 208 | https://github.com/moira-alert/moira/pull/645 209 | - Bumped go to 1.16.5 by @litleleprikon in 210 | https://github.com/moira-alert/moira/pull/642 211 | - Improved speed of metrics matching in filter by @KiskachiMaria in 212 | https://github.com/moira-alert/moira/pull/682 213 | - Added performance test for matching of tagged metrics by 214 | @dmitryanchikov in https://github.com/moira-alert/moira/pull/686 215 | - Updated module github.com/golang/snappy by @zhelyabuzhsky in 216 | https://github.com/moira-alert/moira/pull/698 217 | - Changed Kontur logo by @zhelyabuzhsky in 218 | https://github.com/moira-alert/moira/pull/704 219 | - Added shared test configuration for GoLand by @zhelyabuzhsky in 220 | https://github.com/moira-alert/moira/pull/703 221 | - Updated bleve package by @zhelyabuzhsky in 222 | https://github.com/moira-alert/moira/pull/706 223 | - Fixed redis port exposing by @zhelyabuzhsky in 224 | https://github.com/moira-alert/moira/pull/711 225 | - Added automaxprocs package to filter by @zhelyabuzhsky in 226 | https://github.com/moira-alert/moira/pull/712 227 | - Decreased level of logging for broken contact errors to ‘warning’ by 228 | @dmitryanchikov in https://github.com/moira-alert/moira/pull/716 229 | - Added automaxprocs package to api, checker, cli, notifier by 230 | @zhelyabuzhsky in https://github.com/moira-alert/moira/pull/719 231 | - Added CodeQL analysis by @zhelyabuzhsky in 232 | https://github.com/moira-alert/moira/pull/705 233 | - Fixed plotting error in notifier by @dmitryanchikov in 234 | https://github.com/moira-alert/moira/pull/724 235 | -------------------------------------------------------------------------------- /source/release_notes/2_7_0.rst: -------------------------------------------------------------------------------- 1 | 2.7.0 2 | ===== 3 | 4 | Upgrading 5 | --------- 6 | 7 | .. important:: **Redis config section update is required.** 8 | 9 | Please update redis section in your config files according to the `examples `_. 10 | 11 | .. important:: **Redis DB conversion is required.** 12 | 13 | Moira 2.7 has some structure changes in Redis DB. 14 | 15 | You can upgrade from moira 2.6.2 using corresponding flag in ``--from-version`` variable. 16 | 17 | .. code-block:: bash 18 | 19 | moira-cli --config=/etc/moira/cli.yml --update --from-version=2.6 20 | 21 | If you would like to downgrade back to Moira 2.6.2, you should run CLI-converter. 22 | 23 | .. code-block:: bash 24 | 25 | moira-cli --config=/etc/moira/cli.yml --downgrade --to-version=2.6 26 | 27 | Both cases imply usage of Moira-Cli v.2.7, you can find it on `Release Page `_. 28 | 29 | What’s Changed 30 | -------------- 31 | 32 | - Added Redis Cluster support by @zhelyabuzhsky 33 | in https://github.com/moira-alert/moira/pull/696 34 | -------------------------------------------------------------------------------- /source/release_notes/2_7_1.rst: -------------------------------------------------------------------------------- 1 | 2.7.1 2 | ===== 3 | 4 | What’s Changed 5 | -------------- 6 | 7 | - Updated golang to 1.17.7 by @zhelyabuzhsky 8 | in https://github.com/moira-alert/moira/pull/732 9 | - Extended list of broken contact errors by @dmitryanchikov 10 | in https://github.com/moira-alert/moira/pull/729 11 | - Improved speed of tagged metrics matching in filter by @KiskachiMaria 12 | in https://github.com/moira-alert/moira/pull/726 13 | -------------------------------------------------------------------------------- /source/release_notes/2_7_2.rst: -------------------------------------------------------------------------------- 1 | 2.7.2 2 | ===== 3 | 4 | What’s Changed 5 | -------------- 6 | 7 | *Added* 8 | 9 | - ⚠️ Cleanup (to clean users) named cleanup-users now 10 | - ⚠️ Moira will ignore tag order in tag metrics and will sort them alphabetically by @zhelyabuzhsky 11 | in https://github.com/moira-alert/moira/pull/758 12 | - Click action on tags in trigger list & view by @sol-un 13 | in https://github.com/moira-alert/web2.0/pull/384 14 | - Mattermost as a delivery channel option by @sol-un and @Dimedrolity 15 | in https://github.com/moira-alert/web2.0/pull/396 16 | and https://github.com/moira-alert/moira/pull/791 17 | - Don't write into Redis old metrics by @kissken 18 | in https://github.com/moira-alert/moira/pull/757 19 | - Checker can run in multiple instances by @maksgalimz 20 | in https://github.com/moira-alert/moira/pull/802 21 | 22 | 23 | *Fixed* 24 | 25 | - Markdown not working in the mobile version by @OAPrilepa 26 | in https://github.com/moira-alert/web2.0/pull/374 27 | - Interface inconsistencies in some modals by @sol-un 28 | in https://github.com/moira-alert/web2.0/pull/376 29 | and https://github.com/moira-alert/web2.0/pull/383 30 | - Last event date format by @sol-un #389 31 | in https://github.com/moira-alert/web2.0/pull/389 32 | - Search by non-existent tags by @HolyPrapor and @sol-un 33 | in https://github.com/moira-alert/web2.0/pull/353 34 | and https://github.com/moira-alert/web2.0/pull/394 35 | - Trigger list item styles on hover by @sol-un 36 | in https://github.com/moira-alert/web2.0/pull/390 37 | - A bottleneck in trigger list rendering by @sol-un 38 | in https://github.com/moira-alert/web2.0/pull/391 39 | -------------------------------------------------------------------------------- /source/release_notes/2_8_0.rst: -------------------------------------------------------------------------------- 1 | 2.8.0 2 | ===== 3 | 4 | What’s Changed 5 | -------------- 6 | 7 | *Added and Fixed* 8 | 9 | Breaking 10 | - ⚠️ Update carbonapi to 0.16.0 by @Tetrergeru in https://github.com/moira-alert/moira/pull/822 11 | - ⚠️ Replace flat logs with structured logging by @Tetrergeru in https://github.com/moira-alert/moira/pull/811 and https://github.com/moira-alert/moira/pull/819 12 | 13 | Other 14 | - Fix regex in series by tag parser by @balalay12 in https://github.com/moira-alert/moira/pull/825 15 | - Validate trigger target on create/update by @Dimedrolity in https://github.com/moira-alert/moira/pull/817 16 | - Fix sending plot in slack-threads by @kissken in https://github.com/moira-alert/moira/pull/828 17 | - Add created_by and updated_by for trigger by @kissken in https://github.com/moira-alert/moira/pull/824 18 | - Make strings.Split function available in description templates by @kotbauk in https://github.com/moira-alert/moira/pull/843 19 | - Added cleanup abandoned tags by @Dimedrolity in https://github.com/moira-alert/moira/pull/773 20 | -------------------------------------------------------------------------------- /source/release_notes/2_8_1.rst: -------------------------------------------------------------------------------- 1 | 2.8.1 2 | ===== 3 | 4 | What’s Changed 5 | -------------- 6 | 7 | *Added and Fixed* 8 | 9 | - Updated golang to 1.18 by @Dimedrolity in https://github.com/moira-alert/moira/pull/775 10 | - Upgraded dependencies by @Dimedrolity in https://github.com/moira-alert/moira/pull/776 11 | - Add support for sentinel auth by @DreidSpb in https://github.com/moira-alert/moira/pull/847 -------------------------------------------------------------------------------- /source/release_notes/2_8_2.rst: -------------------------------------------------------------------------------- 1 | 2.8.2 2 | ===== 3 | 4 | What’s Changed 5 | -------------- 6 | 7 | *Added and Fixed* 8 | 9 | * Add func to delete unused triggers without subscriptions by @kissken in https://github.com/moira-alert/moira/pull/851 10 | * Fix incorrect key naming that broke subscriptions and contact of anonymous user in upgrade from 2.7 to 2.8 by @kissken in https://github.com/moira-alert/moira/pull/850 11 | * For fix incorrect behaviour run cli command ``update --from-version=2.7 --to-version=2.8`` 12 | -------------------------------------------------------------------------------- /source/release_notes/2_8_3.rst: -------------------------------------------------------------------------------- 1 | 2.8.3 2 | ===== 3 | 4 | What’s Changed 5 | -------------- 6 | 7 | *Major features* 8 | 9 | * Collect history of sent notifications by @ejuravlev in https://github.com/moira-alert/moira/pull/846 10 | * Change PR Action logic to trigger build from comment and support build on forks @kotbauk in https://github.com/moira-alert/moira/pull/868 11 | 12 | *Other* 13 | 14 | * Trigger with duplicated metrics now has EXCEPTION state instead of ERROR by @almostinf in https://github.com/moira-alert/moira/pull/875 15 | * Add support for sprig library in trigger description templates by @gODeaLoAple https://github.com/moira-alert/moira/pull/876 16 | * Add EXCEPTION state to the list of daily reminded states by @Tetrergeru in https://github.com/moira-alert/moira/pull/878 17 | * Add validation of the list of used variables in trigger expressions by @almostinf in https://github.com/moira-alert/moira/pull/881 18 | * Stricter validation of trigger ids by @almostinf in https://github.com/moira-alert/moira/pull/887 19 | * Add timezone to messages in notifier by @almostinf in https://github.com/moira-alert/moira/pull/885 20 | * Add swagger generation by @almostinf in https://github.com/moira-alert/moira/pull/891 21 | -------------------------------------------------------------------------------- /source/release_notes/2_8_4.rst: -------------------------------------------------------------------------------- 1 | 2.8.4 2 | ===== 3 | 4 | What’s Changed 5 | -------------- 6 | 7 | *Fixed* 8 | 9 | * Uprade moira from 2.7 to 2.8 10 | * Now notifier configs must use exact types (now you can't use `"true"` in boolean fields, only `true`) -------------------------------------------------------------------------------- /source/release_notes/2_9_0.rst: -------------------------------------------------------------------------------- 1 | 2.9.0 2 | ===== 3 | 4 | What’s Changed 5 | -------------- 6 | 7 | *Major Changes* 8 | 9 | * Add Prometheus support. !! Config changes required to use prometheus, 10 | see the docs (:ref:`prometheus-remote-triggers-checker`). 11 | https://github.com/moira-alert/moira/pull/848 12 | (by @tetrergeru) 13 | 14 | * Filter apply regices the same way as graphite, see the docs 15 | (:ref:`filter-configuration`). 16 | https://github.com/moira-alert/moira/pull/923 17 | https://github.com/moira-alert/moira/pull/924 18 | (by @lordvidex and @tetrergeru) 19 | 20 | * Add swagger generated client 21 | https://github.com/moira-alert/client-go 22 | (by @almostinf) 23 | 24 | *Minor Changes* 25 | 26 | * Log failed notification only after several retries 27 | https://github.com/moira-alert/moira/pull/906 28 | (by @kissken) 29 | 30 | * Add metrics for the number of triggers by source 31 | https://github.com/moira-alert/moira/pull/904 32 | (by @tetrergeru) 33 | 34 | * Add prometheus retries 35 | https://github.com/moira-alert/moira/pull/909 36 | (by @tetrergeru) 37 | 38 | * Add notifier is alive metric in notifier-selfcheck 39 | https://github.com/moira-alert/moira/pull/910 40 | (by @kissken) 41 | 42 | * Add api for all unused triggers 43 | https://github.com/moira-alert/moira/pull/915 44 | (by @kissken) 45 | 46 | * Add search by created by parameter 47 | https://github.com/moira-alert/moira/pull/908 48 | (by @almostinf) 49 | 50 | * Add error handling for regex compilation 51 | https://github.com/moira-alert/moira/pull/933 52 | (by @tetrergeru) 53 | 54 | *Dependencies* 55 | 56 | * Switch to a maintained version of carbon-c-relay 57 | https://github.com/moira-alert/moira/pull/925 58 | (by @Pliner) 59 | 60 | * Upgrade mattermost & x/net version 61 | https://github.com/moira-alert/moira/pull/937 62 | (by @kissken) 63 | 64 | * Update go to 1.19 65 | https://github.com/moira-alert/moira/pull/938 66 | (by @kissken) 67 | 68 | *Fixes* 69 | 70 | * Fix switch to maintenance at set del 71 | https://github.com/moira-alert/moira/pull/916 72 | (by @almostinf) 73 | 74 | * Fix panics in checker and enable panic logging 75 | https://github.com/moira-alert/moira/pull/928 76 | https://github.com/moira-alert/moira/pull/929 77 | https://github.com/moira-alert/moira/pull/939 78 | (by @tetrergeru) 79 | 80 | * Don't mock decode err in api 81 | https://github.com/moira-alert/moira/pull/902 82 | (by @kissken) 83 | 84 | * Calculate RemoteAllowed based on remote/prom configs 85 | https://github.com/moira-alert/moira/pull/945 86 | (by @Pliner) 87 | 88 | * Add x-nullable annotation and fix documentaion in config handler 89 | https://github.com/moira-alert/moira/pull/903 90 | (by @almostinf) 91 | -------------------------------------------------------------------------------- /source/release_notes/2_9_1.rst: -------------------------------------------------------------------------------- 1 | 2.9.1 2 | ===== 3 | 4 | What’s Changed 5 | -------------- 6 | 7 | *Fixes* 8 | 9 | * Fix checker settings: add `max_parallel_prometheus_checks` and fix filling default settings 10 | https://github.com/moira-alert/moira/pull/972 11 | (by @almostinf) 12 | -------------------------------------------------------------------------------- /source/release_notes/index.rst: -------------------------------------------------------------------------------- 1 | Release Notes 2 | ============ 3 | 4 | .. toctree:: 5 | :maxdepth: 1 6 | 7 | 2_0 8 | 2_1 9 | 2_2 10 | 2_3 11 | 2_4_0 12 | 2_5_0 13 | 2_5_1 14 | 2_6_2 15 | 2_7_0 16 | 2_7_1 17 | 2_7_2 18 | 2_8_0 19 | 2_8_1 20 | 2_8_2 21 | 2_8_3 22 | 2_8_4 23 | 2_9_0 24 | 2_9_1 25 | 2_10_0 26 | 2_11_0 27 | 2_11_1 28 | 2_12_0 29 | 2_13_0 30 | 2_14_0 31 | 2_14_1 32 | 2_15_0 33 | -------------------------------------------------------------------------------- /source/user_guide/advanced.rst: -------------------------------------------------------------------------------- 1 | Advanced Mode Trigger 2 | ===================== 3 | 4 | .. _govaluate: https://github.com/Knetic/govaluate/blob/master/MANUAL.md 5 | .. _redis: https://redis.io/ 6 | .. _graphite: https://github.com/go-graphite/carbonapi 7 | .. _prometheus: https://prometheus.io/ 8 | 9 | Sometimes a simple trigger (:doc:`/user_guide/simple`) 10 | doesn't provide enough flexibility for your task. 11 | 12 | For example, you may want to receive a notification when 5% of user 13 | requests take up more than a second to process, but only if there are 14 | more than 100 requests per minute. Usually, you will have two separate 15 | metrics for this: 16 | 17 | 1. ``Nginx.requests.process_time.p95`` - 95th percentile 18 | of request processing time in milliseconds 19 | 2. ``Nginx.requests.count`` - request count per minute 20 | 21 | Maybe you can construct a monstrous Graphite expression to reflect 22 | this combination, but Moira's Advanced Mode is better: 23 | 24 | .. image:: ../_static/advanced_edit.png 25 | :alt: advanced trigger edit 26 | 27 | .. image:: ../_static/advanced_trigger.png 28 | :alt: advanced trigger 29 | 30 | `aliasByNode` is needed so that the final name of the metric for two targets is the same 31 | 32 | You can use any govaluate_ expression with predefined constants here: 33 | 34 | - ``t1``, ``t2``, ... are values from your targets 35 | - ``OK``, ``WARN``, ``ERROR``, ``NODATA`` are states that must be 36 | the result of evaluation 37 | - ``PREV_STATE`` is equal to previously set state, and allows you 38 | to prevent frequent state changes 39 | 40 | Any incorrect expressions or bad syntax will result in EXCEPTION trigger state. 41 | 42 | If you do not use alone metrics, it is important that the names of the metrics received at the output 43 | for several targets are the same so that they can be matched. Use `aliasByTags`, `aliasByNode` and other similar functions. 44 | 45 | Example 1. Correct use of Advanced Mode 46 | ~~~~~~~~~~~~~~~~~ 47 | 48 | 1. Suppose you have metrics: 49 | 50 | - ``host1.loadavg`` — Load Average on server host1 51 | - ``host2.loadavg`` 52 | - ``host3.loadavg`` 53 | - ``host1.cpu_count`` — Number of cores on host1 54 | - ``host2.cpu_count`` 55 | - ``host3.cpu_count`` 56 | 57 | 2. You are creating an Advanced Mode trigger: 58 | 59 | - t1 — ``aliasByNode(*.loadavg, 0)`` 60 | - t2 — ``aliasByNode(*.cpu_count, 0)`` 61 | 62 | Expression — ``(t1 / t2) > 1 ? ERROR : OK`` 63 | 64 | 3. This results in three metrics in the trigger, for which the state is tracked separately: 65 | 66 | - ``host1`` — expression is calculated for ``t1 = host1.loadavg, t2 = host1.cpu_count`` 67 | - ``host2`` — expression is calculated for ``t1 = host2.loadavg, t2 = host2.cpu_count`` 68 | - ``host3`` — expression is calculated for ``t1 = host3.loadavg, t2 = host3.cpu_count`` 69 | 70 | .. image:: ../_static/advanced_trigger_example_1.png 71 | :alt: advanced trigger first example 72 | 73 | Example 2. Incorrect use of Advanced Mode 74 | ~~~~~~~~~~~~~~~~~ 75 | 76 | 1. Suppose you have metrics: 77 | 78 | - ``host1.loadavg`` — Load Average on server host1 79 | - ``host1.cpu_count`` — Number of cores on host1 80 | 81 | 2. You are creating an Advanced Mode trigger: 82 | 83 | - t1 — ``aliasByNode(*.loadavg, 1)`` 84 | - t2 — ``aliasByNode(*.cpu_count, 1)`` 85 | 86 | Expression — ``(t1 / t2) > 1 ? ERROR : OK`` 87 | 88 | 3. This results in two metrics in the trigger: 89 | 90 | - ``loadavg`` — expression is calculated for ``t1 = host1.loadavg, t2 = NaN``, ``NaN`` appeared because there are no metrics named ``loadavg`` in the second target. 91 | - ``cpu_count`` — expression is calculated for ``t1 = NaN, t2 = host2.cpu_count``, ``NaN`` appeared because there are no metrics named ``cpu_count`` in the first target. 92 | 93 | ``NaN`` is a series in which there are no points, it will not be possible to correctly calculate the expression with it, 94 | so both final metrics will be in **NODATA**. 95 | 96 | .. image:: ../_static/advanced_trigger_example_2.png 97 | :alt: advanced trigger second example 98 | 99 | Alone Target 100 | ~~~~~~~~~~~~~~~~~ 101 | 102 | Alone targets are targets that result in a single metric, for example, you can use grouping functions 103 | (`groupByNode`, `groupByTags` and other similar ones) for such targets. 104 | 105 | The advantage of such targets is that it is very easy to use them in Advanced Mode, 106 | it is not necessary that the metrics that result from such targets have the same name as the metrics of non alone targets. 107 | 108 | A ``single`` checkbox appears in the Moira web interface **after adding new targets** to mark some target as Alone. 109 | 110 | It is important that the Alone target returns **only one** series of values. If the ``single`` flag is set but more target series are returned, 111 | the trigger will change state to **EXCEPTION**. 112 | 113 | Example 3. Using single and not single targets 114 | ~~~~~~~~~~~~~~~~~ 115 | 116 | 1. Suppose you have metrics: 117 | 118 | - ``host1.loadavg`` — Load Average on server host1 119 | - ``host2.loadavg`` 120 | - ``host3.loadavg`` 121 | - ``all_hosts.cpu_count`` — Number of cores on any of the servers (the same everywhere) 122 | 123 | 2. You are creating an Advanced Mode trigger: 124 | 125 | - t1 — ``aliasByNode(*.loadavg, 0)`` 126 | - t2 — ``all_hosts.cpu_count`` — **alone metric** 127 | 128 | Expression — ``(t1 / t2) > 1 ? ERROR : OK`` 129 | 130 | 3. This results in three metrics in the trigger, for which the state is tracked separately: 131 | 132 | - ``host1`` — expression is calculated for ``t1 = host1.loadavg, t2 = all_hosts.cpu_count`` 133 | - ``host2`` — expression is calculated for ``t1 = host2.loadavg, t2 = all_hosts.cpu_count`` 134 | - ``host3`` — expression is calculated for ``t1 = host3.loadavg, t2 = all_hosts.cpu_count`` 135 | 136 | .. image:: ../_static/advanced_trigger_example_3.png 137 | :alt: advanced trigger third example 138 | 139 | Example 4. Using only single targets 140 | ~~~~~~~~~~~~~~~~~ 141 | 142 | 1. Suppose you have metrics: 143 | 144 | - ``all_hosts.loadavg`` — Load Average on any of the servers (the same everywhere) 145 | - ``all_hosts.cpu_count`` — Number of cores on any of the servers (the same everywhere) 146 | 147 | 2. You are creating an Advanced Mode trigger: 148 | 149 | - t1 — ``all_hosts.loadavg`` — **alone metric** 150 | - t2 — ``all_hosts.cpu_count`` — **alone metric** 151 | 152 | Expression — ``(t1 / t2) < 1 ? OK : ERROR`` 153 | 154 | 3. This results in one metric in the trigger: 155 | 156 | - ``all_hosts.loadavg`` — expression is calculated for ``t1 = all_hosts.loadavg, t2 = all_hosts.cpu_count`` 157 | 158 | If the trigger has **all single** targets, there will always be one metric whose name will be the same as the metric for the first target. 159 | 160 | .. image:: ../_static/advanced_trigger_example_4.png 161 | :alt: advanced trigger fourth example 162 | 163 | Templates 164 | ~~~~~~~~~~~~~~~~~ 165 | 166 | The template is supported by Moira, the template implements data-driven templates for generating textual output. 167 | Information about how to program the templates themselves, see the `documentation. `_ 168 | You can also use functions from `sprig-library. `_ 169 | 170 | 171 | Data you can use: 172 | ~~~~~~~~~~~~~~~~~ 173 | 174 | .. code-block:: text 175 | 176 | Trigger { Name } 177 | Events [ ] { 178 | Metric 179 | MetricElements [ ]string 180 | Timestamp 181 | Value 182 | State 183 | } 184 | 185 | Example: 186 | ``https://grafana.yourhost.com/some-dashboard{{ range $i, $v := .Events }}{{ if ne $i 0 }}&{{ else }}? 187 | {{ end }}var-host={{ $v.Metric }}{{ end }}`` 188 | 189 | Strings manipulations 190 | ~~~~~~~~~~~~~~~~~~~~~ 191 | - ``{{ stringsReplace .Trigger.Name "." "_" -1 }}`` 192 | - ``{{ stringsToLower .Trigger.Name }}`` 193 | - ``{{ stringsToUpper .Trigger.Name }}`` 194 | - ``{{ stringsTrimPrefix .Trigger.Name "remove_me" }}`` 195 | - ``{{ stringsTrimSuffix .Trigger.Name "remove_me" }}`` 196 | - ``{{ stringsSplit .Trigger.Name "sep" }}`` 197 | 198 | 199 | See more about functions and args in golang `strings `. 200 | 201 | Date manipulations 202 | ~~~~~~~~~~~~~~~~~~ 203 | - ``{{ date $v.Timestamp }}`` print date timestamp. 204 | - ``{{ formatDate $v.Timestamp "Mon Jan _2 15:04:05 2006" }}`` format timestamp by pattern, see `more `_. 205 | 206 | Also you can use some methods for events: 207 | 208 | .. code-block:: text 209 | 210 | {{ range $event:= .Events }} 211 | {{ $event.TimestampDecrease 5 }} 212 | {{ end }} 213 | 214 | - ``{{ $event.TimestampDecrease 5 }}`` - decrease event timestamp. 215 | - ``{{ $event.TimestampIncrease 5 }}`` - increase event timestamp. 216 | 217 | 218 | Data source 219 | ~~~~~~~~~~~~~~~~~ 220 | 221 | If :ref:`graphite-remote-triggers-checker` or :ref:`prometheus-remote-triggers-checker` is enabled, you can 222 | choose between following Data Sources: 223 | 224 | - Local_ — Moira database. By default Redis stores data for only several hours. 225 | It covers most of user cases when you need real-time alerting. 226 | - Graphite_ — remote Graphite-like HTTP API. It should be used only when you 227 | need to get metrics for a large period. 228 | (Only if :ref:`graphite-remote-triggers-checker` is enabled) 229 | .. warning:: Please, use this Data Source with caution. It may cause extra load on Graphite HTTP API. 230 | 231 | .. important:: 232 | 233 | Please, keep in mind that functions in Remote and Local triggers can work differently. 234 | To avoid this, make sure you use Carbonapi with the same revision as in Moira. Latest Carbonapi listed in :doc:`../changelog`. 235 | - Prometheus_ — remote Prometheus HTTP API. It could be used to select Prometheus 236 | metrics via PromQL. (Only if :ref:`prometheus-remote-triggers-checker` is enabled) 237 | -------------------------------------------------------------------------------- /source/user_guide/contact_events.rst: -------------------------------------------------------------------------------- 1 | Contact notification events 2 | ================== 3 | 4 | To track notifications in a specific delivery channel, you can use the "Events" button on the /settings page. 5 | 6 | .. image:: ../_static/contact-events-button.png 7 | :alt: contact events button 8 | 9 | On the side page you can see statistics per trigger in particular time range. 10 | 11 | .. image:: ../_static/contact-events-per-trigger.png 12 | :alt: contact events per trigger 13 | 14 | You can also see events groupped by trigger transitions. 15 | 16 | .. image:: ../_static/trigger-transition-stats.png 17 | :alt: trigger transition stats 18 | 19 | Storage time interval can be configured, see :doc:`/installation/configuration` for details. -------------------------------------------------------------------------------- /source/user_guide/efficient.rst: -------------------------------------------------------------------------------- 1 | .. include:: 2 | 3 | .. raw:: html 4 | 5 | 39 | 40 | Efficient Triggers 41 | ================== 42 | 43 | To use Moira efficiently, you should understand its underlying 44 | design decisions. 45 | 46 | We often notice that when new users create their first triggers, 47 | they set thresholds at random, or by intuition. It happens because 48 | when you configure your first 24/7/365 automated monitoring system, 49 | you don't really know how your system works. If you have at least 50 | hundreds of metrics, it's impossible to watch all of them with 51 | your eyes. What are the limits of your system? How often does your 52 | system reach critical resource consumption during a day? Should you 53 | immediately react when metric X reaches value N, or is it a fluctuation 54 | that passes by itself? 55 | 56 | Later, when you learn to understand you system, you will need to adjust 57 | your triggers. And that's when you need to understand Moira. 58 | 59 | 60 | States 61 | ------ 62 | 63 | Unlike many other tools providing several distinct level systems like 64 | "priority" and "severity", Moira supports a single set of states. 65 | Every state has a well-defined meaning, and you should use these states 66 | accordingly. 67 | 68 | 69 | OK 70 | ^^ 71 | 72 | .. raw:: html 73 | 74 |
75 | Color: 76 | 77 |
78 | 79 | This is a basic state, in which all your metrics must spend most of their 80 | time. Just like you keep your autotests green, you should keep your 81 | metrics green. 82 | 83 | 84 | WARN 85 | ^^^^ 86 | 87 | .. raw:: html 88 | 89 |
90 | Color: 91 | 92 |
93 | 94 | This state means that you should do something to prevent ERRORs in the future. 95 | Not immediately: maybe you should order more hardware from your vendor, 96 | or plan to optimize code in the next iteration. You can configure less 97 | intrusive delivery channels here, like email. 98 | 99 | Metrics can be in this state for days or even weeks. 100 | 101 | 102 | ERROR 103 | ^^^^^ 104 | 105 | .. raw:: html 106 | 107 |
108 | Color: 109 | 110 |
111 | 112 | This is a critical condition that requires immediate intervention. Your 113 | datacenter is on fire. All application processes shut down. There is no 114 | disk space left on your database server to process million-dollar 115 | transactions. These notifications are important enough to wake you up 116 | at night. You can still configure schedules to assign shifts to several 117 | engineers, though (see :doc:`/user_guide/schedule`). You should configure 118 | more intrusive delivery channels here, like Pushover. 119 | 120 | Metrics should not be in this state for more than several hours. 121 | 122 | Moira will send you reminders every 24 hours if some of your metrics 123 | remain in this state. 124 | 125 | If a delivery channel supports high-priority messages (like Pushover does), 126 | Moira will try to use them for ERRORs. 127 | 128 | 129 | NODATA 130 | ^^^^^^ 131 | 132 | .. raw:: html 133 | 134 |
135 | Color: 136 | 137 |
138 | 139 | This state means that Moira hasn't been receiving data points for a metric 140 | for some time. See :doc:`/user_guide/nodata` for details. This state 141 | is considered as bad as an ERROR in Moira (because it can actually be 142 | an ERROR - we don't receive any data, so we don't know for sure). 143 | It may be even worse than an ERROR, because users tend to ignore metrics 144 | in this state and leave them hanging in the web interface, greatly 145 | increasing the chance to miss something actually important. You should 146 | delete old unused metrics from Moira when they stop providing data points: 147 | 148 | .. image:: ../_static/delete_metric.png 149 | :alt: delete unused metrics 150 | 151 | In the beginning every metric is in this state. You will receive one 152 | NODATA |rarr| OK notification when the first data point arrives. 153 | 154 | Moira will send you reminders every 24 hours if some of your metrics remain 155 | in this state. 156 | 157 | Moira will set NODATA state only for known metrics - i.e. for metrics that 158 | have sent at least one data point to Moira. 159 | 160 | 161 | EXCEPTION 162 | ^^^^^^^^^ 163 | 164 | .. raw:: html 165 | 166 |
167 | Color: 168 | 169 |
170 | 171 | This is an error inside Moira. Unless you have bad syntax in your 172 | :doc:`/user_guide/advanced` trigger, this has nothing to do with your 173 | metric state. You should try to fix or update Moira, or contact Moira 174 | developers (see :doc:`/contacts`). 175 | 176 | 177 | Dealing With False Positives 178 | ---------------------------- 179 | 180 | Sometimes it's hard to maintain strict rule of keeping your metrics 181 | green, if your triggers switch OK |rarr| ERROR |rarr| OK |rarr| ERROR 182 | for short periods of time several times a day. It can lead to alarm 183 | fatigue and missing actual failures. 184 | 185 | There is no single recipe for eliminating false positives, but here 186 | are some tips. 187 | 188 | 189 | Use Graphite Functions 190 | ^^^^^^^^^^^^^^^^^^^^^^ 191 | 192 | .. _functions: http://graphite.readthedocs.org/en/latest/functions.html 193 | 194 | Graphite provides tons of useful functions_ to process data, and Moira 195 | understands all of them. For example: 196 | 197 | - If you are experiencing peaks on you graphs that lead to unnecessary 198 | state switches, you can alleviate these peaks with ``movingAverage`` or 199 | ``movingMedian``. 200 | 201 | .. image:: ../_static/moving_average.png 202 | :alt: moving average Graphite function 203 | 204 | - If you are interested in aggregate 10-minute values, not single minute 205 | values, use ``movingSum``. 206 | - If you want zeros instead of missing data points, use ``transformNull``. 207 | Also, ``keepLastValue`` is useful when dealing with missing points. 208 | - Avoid functions that show and hide metrics, like ``averageAbove``. 209 | Moira does not consider hidden metrics to be in NODATA state. Instead, 210 | Moira retains last state that the metric had when it was visible. 211 | 212 | 213 | Draw First, Monitor Later 214 | ^^^^^^^^^^^^^^^^^^^^^^^^^ 215 | 216 | Always draw a graph of target(s) you are planning to monitor. Use generic 217 | Graphite web interface or something like Grafana. Look for minumum and maximum 218 | values. Notice, how often and for how long the graph crosses your 219 | planned thresholds. Try to correlate the graph with previous system failures. 220 | Then, copy and paste corrected target to Moira. 221 | 222 | Of course, you may and should remove any functions that make no sense in Moira 223 | (like ``sortByTotal``) and can generate unwanted side effects 224 | (like ``averageAbove``). 225 | -------------------------------------------------------------------------------- /source/user_guide/hidden_pages.rst: -------------------------------------------------------------------------------- 1 | Hidden Pages 2 | ============ 3 | 4 | Some rarely used features of Moira are hidden on pages that are not 5 | linked from anywhere. This is a deliberate design decision to reduce 6 | visual clutter in the main UI. 7 | 8 | You need to type an address of a hidden page manually, like this: ``http://moira.example.com/hidden_page``. 9 | 10 | .. _notifications-hidden-page: 11 | 12 | Notifications 13 | ------------- 14 | 15 | If Moira encounters an error while sending a notification, it will try 16 | again every minute for the next 24 hours. After that period, the notification 17 | is considered lost. You can configure this via ``resending_timeout`` parameter 18 | in notifier yaml config. 19 | 20 | In some cases notifications will never be delivered, for example if a user 21 | specifies invalid contact. 22 | 23 | If you need to interrupt this behavior, you can manually delete failing 24 | notifications at ``/notifications``. 25 | 26 | 27 | Tags 28 | ---- 29 | 30 | Deleting a tag is a rare and dangerous operation, but you still can 31 | do it at ``/tags``. 32 | 33 | Tags list shows how many triggers and subscriptions use a tag. 34 | You can't delete a tag if there is at least one trigger that uses it. 35 | You **can** delete a tag that is used in a subscription. 36 | 37 | 38 | Patterns 39 | -------- 40 | 41 | You can see a list of all Graphite patterns with links to corresponding 42 | triggers and list of all matching metrics at ``/patterns``. 43 | 44 | 45 | Contacts 46 | -------- 47 | 48 | Sometimes you need to find (and/or modify, delete) contact registered in Moira, 49 | which not belongs to you or your team. 50 | For this purpose ``/contacts`` page maybe useful. 51 | If authorization is enabled this page is available only for admins. 52 | 53 | 54 | Teams 55 | -------- 56 | 57 | We recommend to use teams to store contacts and subscriptions 58 | because after creating inside the team they are visible to all team members. 59 | With ``/teams/all`` you can find all the teams exist in Moira. 60 | If authorization is enabled this page is available only for admins. 61 | 62 | 63 | Noisiness 64 | -------- 65 | 66 | As soon as you've set up your triggers, contact and subscriptions you may want to know 67 | what triggers generate more events than others. 68 | With ``/noisiness`` you can find how many events each trigger has generated. 69 | If authorization is enabled this page is available only for admins. -------------------------------------------------------------------------------- /source/user_guide/index.rst: -------------------------------------------------------------------------------- 1 | User Guide 2 | ========== 3 | 4 | This user guide is based on a number of real-life scenarios, 5 | from simple and universal to complicated and specific. 6 | 7 | .. toctree:: 8 | :maxdepth: 1 9 | 10 | simple 11 | subscriptions 12 | efficient 13 | schedule 14 | trigger_page 15 | throttling 16 | nodata 17 | advanced 18 | hidden_pages 19 | maintenance 20 | selfstate 21 | contact_events 22 | -------------------------------------------------------------------------------- /source/user_guide/maintenance.rst: -------------------------------------------------------------------------------- 1 | Maintenance 2 | =========== 3 | 4 | Maintenance is a proper way to mute alerting on specific metrics 5 | or triggers. It can be useful during planned work. E.g., you are 6 | going to move server from one data center to another and don't want 7 | Moira to disturb you. 8 | 9 | .. image:: ../_static/maintenance.png 10 | :alt: maintenance mode 11 | 12 | Examples 13 | ------------------------------------- 14 | 15 | When you switch a metric or trigger into maintenance, Moira will 16 | mute all state changes during that period. You will receive notification 17 | about every metric, if the state before and after maintenance turn out 18 | to be different. 19 | 20 | Example 1. Maintenance metric, alert will not be sent 21 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 22 | 23 | * metric ``awesomeMetric1`` is in ``OK`` state; 24 | * Rick_ switches metric into maintenance for an hour; 25 | * within the hour metric changes its state several times: 26 | 27 | - ``OK`` → ``WARN``, 28 | - ``WARN`` → ``ERROR``, 29 | - ``ERROR`` → ``OK``; 30 | 31 | * after one-hour maintenance ends, metric is in ``OK`` state; 32 | * Moira checks if metric state changed during maintenance: 33 | 34 | - ``awesomeMetric1`` state before maintenance: ``OK``; 35 | - ``awesomeMetric1`` state after maintenance ``OK``; 36 | * nothing to notify about: the state remained the same as it was 37 | before the maintenance period. 38 | 39 | Example 2. Maintenance metric, alert will be sent 40 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 41 | 42 | * metric ``awesomeMetric2`` is in ``OK`` state; 43 | * Rick_ switches metric into maintenance for an hour; 44 | * within the hour metric changes its state several times: 45 | 46 | - ``OK`` → ``WARN``, 47 | - ``WARN`` → ``ERROR``, 48 | - ``ERROR`` → ``OK``, 49 | - ``OK`` → ``ERROR``; 50 | 51 | * after one-hour maintenance ends, metric is in ``ERROR`` state; 52 | * Moira checks if metric state changed during maintenance: 53 | 54 | - ``awesomeMetric2`` state before maintenance: ``OK``; 55 | - ``awesomeMetric2`` state after maintenance ``ERROR``; 56 | 57 | * Moira sends message to user: the state has changed from that which 58 | was before the maintenance period. 59 | 60 | Example 3. Maintenance trigger, alert will be sent 61 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 62 | 63 | * metric ``awesomeMetric1`` is in ``WARN`` state; 64 | * metric ``awesomeMetric2`` is in ``OK`` state; 65 | * Rick_ switches trigger with this metrics into maintenance for an hour; 66 | * within the hour metric ``awesomeMetric2`` changes its state several times: 67 | 68 | - ``OK`` → ``WARN``, 69 | - ``WARN`` → ``ERROR``, 70 | - ``ERROR`` → ``OK``, 71 | - ``OK`` → ``ERROR``; 72 | 73 | * after one-hour maintenance ends, metric is in ``ERROR`` state; 74 | * Moira checks if metric state changed during maintenance: 75 | 76 | - ``awesomeMetric1`` state before maintenance: ``WARN``; 77 | - ``awesomeMetric1`` state after maintenance ``WARN``; 78 | - ``awesomeMetric2`` state before maintenance: ``OK``; 79 | - ``awesomeMetric2`` state after maintenance ``ERROR``; 80 | 81 | * Moira sends message about ``awesomeMetric2`` metric to user: the state 82 | has changed from that which was before the maintenance period. 83 | 84 | .. _Rick: https://www.youtube.com/watch?v=dQw4w9WgXcQ 85 | -------------------------------------------------------------------------------- /source/user_guide/nodata.rst: -------------------------------------------------------------------------------- 1 | Dealing with NODATA 2 | =================== 3 | 4 | If you have a simple trigger (like the one described in 5 | :doc:`/user_guide/simple`), you probably know what happens when 6 | a metric has a very high or a very low value. Free disk space 7 | is too low? You get a notification. 8 | 9 | But what if your metric has *no* value? Literally, what if Moira 10 | is not receiving any data for your metric? How can you know, 11 | whether you have enough disk space left or not? In this case, 12 | a trigger setting defines the behavior: 13 | 14 | .. image:: ../_static/nodata.png 15 | :alt: NODATA settings 16 | :width: 400 17 | 18 | When Moira hasn't been receiving data for more than default 600 seconds, 19 | it will set a special NODATA state for this metric. 20 | You can set any other state or change time delay here. For example, 21 | if you have an error metric, and no data means no errors, you should 22 | set this to OK. 23 | 24 | .. note:: 25 | 26 | Checkbox ``Mute new metrics notifications`` defines whether Moira should notify you about new metrics or mute those notifications. 27 | If box is unchecked, Moira will send you ``NODATA`` → ``OK`` event for every new metric in the trigger. 28 | 29 | Muting notifications about new metrics could be useful if you have trigger with lots of metric in it. 30 | 31 | You can also select DEL here to automatically delete all metrics that 32 | no longer provide data. A simple use case is when you often rename metrics 33 | and Moira quickly becomes flooded with old irrelevant metric names. 34 | 35 | .. warning:: DEL is a dangerous setting, you can easily miss a real notification if your system stops sending metric data. 36 | 37 | You will receive notifications when your metric goes in and out of 38 | NODATA state, just like any other state. 39 | -------------------------------------------------------------------------------- /source/user_guide/schedule.rst: -------------------------------------------------------------------------------- 1 | Schedules 2 | ========= 3 | 4 | Moira provides two ways of defining allowed time intervals for notifications. 5 | 6 | 7 | Subscription Schedule 8 | --------------------- 9 | 10 | If a metric is not that important to wake you up in the middle of the night, 11 | you can set a schedule for subscription: 12 | 13 | .. image:: ../_static/schedule_subscription.png 14 | :alt: subscription schedule 15 | :width: 400 16 | 17 | Notifications generated by this subscription will arrive only on weekdays, 18 | from 08:00 to 17:59 local time. 19 | 20 | If an event happens on weekend, you will receive a notification at 08:00 21 | on Monday. So notifications are not skipped, you just receive them later. 22 | Events will still appear on the event history page at the time when they 23 | happened (see :doc:`/user_guide/trigger_page`). 24 | 25 | 26 | Trigger Watch Time 27 | ------------------ 28 | 29 | Let's say, you have a popular website, that serves over 1000 page views 30 | per second during a day. You can set up a trigger to notify you when you 31 | have less than 50 page views per second - obviously, something is wrong. 32 | You also need to disable this trigger for the night, because at night 33 | all of your users sleep, and this metric is irrelevant. 34 | 35 | Of course, you can set up a subscription schedule - but your history will 36 | become riddled with false night "events", and you will still receive 37 | notifications in the morning. In this case, you need to set up a trigger 38 | watch time: 39 | 40 | .. image:: ../_static/schedule_trigger.png 41 | :alt: trigger schedule 42 | :width: 400 43 | 44 | No events will be recorded for this trigger outside of watch time - you will 45 | receive no notifications, and the event history page will be empty 46 | (see :doc:`/user_guide/trigger_page`). 47 | -------------------------------------------------------------------------------- /source/user_guide/selfstate.rst: -------------------------------------------------------------------------------- 1 | Self State Monitor 2 | ================== 3 | 4 | Self State Monitor is a built-in mechanism designed to protect 5 | end user from false ``NODATA`` notifications and notify administrator 6 | and end user about issues in Moira and/or Graphite systems. 7 | 8 | 9 | Why Self State Monitor 10 | ----------------------- 11 | 12 | A situation is possible when Graphite Relay, Redis DB or Moira-Filter 13 | service breaks down. This leads to the fact that Moira doesn't receive 14 | any metrics from Graphite. In this case, Moira has no metrics on which 15 | it could check state of the triggers. According to the Moira logic, 16 | it should switch triggers to ``NODATA`` state and send alert messages to users. 17 | 18 | To handle this situation properly, we recommend turning on the Self 19 | State Monitor. In this case, Moira will **prevent itself from sending 20 | alert messages to end users but will notify administrators of the existing 21 | problem or end users who have subscribed to Self State alerts (see :ref:`system-subscriptions-description`)**. 22 | 23 | .. warning:: 24 | 25 | When Self State Monitor detects a problem, it disables notifications to end users by their triggers, but can send notifications about the problem via system-subscriptions. When the problem resolved, Self State Monitor notifies admins about it and end users via system-subscriptions and turn back sending notifications automatically. 26 | 27 | Please, read this manual before using Self State Monitor in production. 28 | 29 | .. seealso:: 30 | 31 | For a better understanding, look at the architecture of the :ref:`Moira microservices `. 32 | 33 | 34 | .. _when-monitor-helps: 35 | 36 | When Self State Monitor Helps 37 | ----------------------------- 38 | 39 | Self state monitor checks these situations: 40 | 41 | 1. If there is no connection between Moira and Redis for longer 42 | than ``redis_disconect_delay``. 43 | 2. If Moira-Filter receive no metrics for longer than 44 | ``last_metric_received_delay``. 45 | 3. If Moira-Checker checks no triggers for longer than ``last_check_delay``. 46 | 47 | .. seealso:: 48 | 49 | All the above configuration parametres can be found in the :ref:`Moira-Notifier section ` 50 | on configuration page. 51 | 52 | 53 | How Self State Monitor Works 54 | ---------------------------- 55 | 56 | When you turn Self State Monitor on, it works this way: 57 | 58 | * Self State Monitor checks :ref:`Moira state ` 59 | every 10 seconds. 60 | 61 | * Something breaks down. It can be Graphite-Relay, connection 62 | to Redis DB or crashed Moira-Filter docker container. 63 | 64 | * Self State send alarm message to administrator with issue discription and switch own state to ``WARN`` 65 | 66 | *Here is an example of message*: 67 | 68 | .. image:: ../_static/helth-check-email.png 69 | :alt: email alarm message 70 | 71 | * Self State Monitor turns Moira-Notifier service off, 72 | switching it in ``ERROR`` state. 73 | 74 | .. note:: 75 | 76 | When Moira-Notifier switches to ``ERROR`` state, it mutes all messages to end users and only alerts administrators about Moira health issues. 77 | You need to fix existing problems and then manually switch Moira-Notifier back to ``OK`` using :ref:`API `. 78 | 79 | *When Moira-Notifier not in* ``OK`` *state, Moira will show you an error in Web UI*: 80 | 81 | .. image:: ../_static/helth-check-webui.png 82 | :alt: WEB UI error notification 83 | 84 | * Self State Monitor waits for ``user_notifications_interval`` and switches own state from ``WARN`` to ``ERROR`` if problem persists. 85 | Then users will be notified about the problem via their system-subscriptions. 86 | 87 | * If problem disappears, the Self State Monitor switches own state to ``OK`` and sends notifications according to these rules. 88 | 89 | - Self State Monitor state mutates from ``WARN`` to ``OK`` then notification about Moira normalize are sent only to admins. 90 | - Self State Monitor state mutates from ``ERROR`` to ``OK`` then notification sends to both admins and users via system-subscriptions. 91 | 92 | .. note:: 93 | For a better understanding, look at pictures below 94 | 95 | If Self State detects a problem, then it possible 2 ways to next actions: 96 | 97 | * If problem persists a much more than ``user_notifications_interval`` 98 | .. image:: ../_static/selfstate_full_cycle_WARN_to_ERROR.png 99 | :alt: Self State sends notifications to admins and users 100 | 101 | * If problem persists a less than ``user_notifications_interval`` 102 | .. image:: ../_static/selfstate_full_cycle_WARN_to_OK.png 103 | :alt: Self State sends notifications to admins only 104 | 105 | ----- 106 | 107 | .. _notifier-state-api: 108 | 109 | Turn Moira Notifier On and Off 110 | ------------------------------ 111 | 112 | You can reveal current Moira-Notifier state or change it 113 | on a hidden ``/notifications`` page. 114 | 115 | .. image:: ../_static/notifier-toggle.png 116 | :alt: Notifier toggle 117 | 118 | .. warning:: 119 | 120 | Please, note this toggle changes Moira-Notifier state, not user notifications preferences. 121 | 122 | When you disable notifications with this toggle, Moira-Notifier stops sending messages to all users. 123 | -------------------------------------------------------------------------------- /source/user_guide/simple.rst: -------------------------------------------------------------------------------- 1 | Simple Threshold Trigger 2 | ======================== 3 | 4 | .. _documentation: http://graphite.readthedocs.org/en/latest/functions.html 5 | 6 | Let's say you measure how much free space is left on your HDD and store 7 | this value as ``DevOps.my_server.hdd.freespace_mbytes`` in Graphite. 8 | Maybe you want to get an email when you have less than 50 GB left (it's not 9 | a big problem), and a Pushover notification when you have less than 10 | 1 GB left (you really need to delete something asap). 11 | 12 | You can easily accomplish this by adding a trigger in Moira's Simple Mode: 13 | 14 | .. image:: ../_static/simple.png 15 | :alt: simple trigger 16 | 17 | 18 | Graphite Target 19 | --------------- 20 | 21 | You can specify a single metric like we did here: 22 | ``DevOps.my_server.hdd.freespace_mbytes``. 23 | 24 | You can also specify multiple metrics like ``DevOps.*.hdd.freespace_mbytes``. 25 | All metrics will be monitored separately, and you will get separate 26 | notifications for each metric. 27 | 28 | You can even use Graphite functions like 29 | ``movingAverage(DevOps.my_server.hdd.freespace_mbytes, 10)``. Moira understands 30 | everything that Graphite itself understands. See appropriate documentation_. 31 | 32 | 33 | Thresholds 34 | ---------- 35 | 36 | In simple mode you need to at least one threshold values: WARNING and ERROR. 37 | In our example we set both, lower values are bad, so we set warning threshold 38 | greater than error threshold. In this case, Moira will consider any value less 39 | than 50000 a warning and less than 1000 an error, which is what we want. 40 | In other cases, you may need to consider large values a problem - then you 41 | should make error threshold greater than warning and select 42 | ``Watch for value rising`` option. 43 | 44 | 45 | .. seealso:: 46 | 47 | You can set only one threshold. For example, you can set WARNING equal to 50000, omit ERROR and select ``Watch for value falling``. 48 | In this case you will receive only WARNING messages when free space goes under 50GB and never receive ERROR messages. 49 | You can also do vise versa: set ERROR and omit WARNING. 50 | 51 | 52 | Tags 53 | ---- 54 | 55 | In Moira, you cannot subscribe to a single trigger. Instead, you should 56 | categorize your triggers by tags and subscribe to a tag. It may look like 57 | an overkill here, but when you have dozens of triggers, you are much better 58 | off with tags, because you don't have to enter your contact information over 59 | and over again. Tags also help to filter information on main screen: 60 | 61 | .. image:: ../_static/main_screen.png 62 | :alt: main screen with filter 63 | 64 | You can add as many tags as you want. 65 | 66 | 67 | Subscriptions 68 | ------------- 69 | 70 | Proceed to the :doc:`/user_guide/subscriptions` page to learn how to 71 | set up a subscription to your trigger. 72 | -------------------------------------------------------------------------------- /source/user_guide/subscriptions.rst: -------------------------------------------------------------------------------- 1 | .. include:: 2 | 3 | .. _subscriptions: 4 | 5 | Setting Up Your Subscriptions 6 | ============================= 7 | 8 | By now you should have at least one trigger saved. If you don't, 9 | go back to the :doc:`/user_guide/simple` page. 10 | 11 | First, add some delivery channels: 12 | 13 | .. image:: ../_static/channels.png 14 | :alt: delivery channel settings 15 | 16 | If your Moira installation is configured with separate user accounts, 17 | you will see only your channels and subscriptions on this page. Otherwise, 18 | every user will see the same page with the same channels and subscriptions. 19 | 20 | Consult :doc:`/installation/security` page for instructions on separating 21 | user accounts. 22 | 23 | Once you have at least one channel, you can create a subscription. 24 | Press ``+ Add subscription`` button: 25 | 26 | .. image:: ../_static/subscription_plotting.png 27 | :alt: subscription plotting settings 28 | 29 | .. _subscriptions-plotting: 30 | 31 | Plotting 32 | -------- 33 | 34 | According to two existing polling approaches: 35 | 36 | - Local triggers are best to analyze realtime metrics 37 | - Remote triggers allows to use wider time windows to fetch 38 | historical data directly from Graphite 39 | 40 | there is also two different time ranges will be used according to trigger type: 41 | 42 | - Notification based on events generated by local trigger will contain 43 | graph with timeseries for the last 30 minuntes wheter is throttled 44 | or it was scheduled earlier because of subscription's own time limits. 45 | - Notification based on events generated by remote trigger will contain 46 | graph with timeseries for not less than 30 minuntes until last event occured. 47 | Otherwise first and last events times will form the window. 48 | 49 | .. image:: ../_static/subscription_tags.png 50 | :alt: subscription tags 51 | 52 | 53 | Tags 54 | ---- 55 | 56 | Add required tags into subscription to receive notifications 57 | from triggers with these tags. 58 | 59 | Matching rule is: Notification will be sent if trigger 60 | contains **ALL** of selected tags. 61 | 62 | For example: 63 | 64 | - If subscription has only one tag, you will receive notifications 65 | from any trigger with this tag. 66 | 67 | - Create Triggger1 with tags: ``["DevOps", "Moira-duty"]`` 68 | - Create Triggger2 with tags: ``["DevOps"]`` 69 | - Create Subscription1 with tags: ``["DevOps"]`` 70 | 71 | By using Subscription1 you will receive events for 72 | both Triggger1 and Triggger2 73 | 74 | 75 | - If subscription has multiple tags, you will receive notifications 76 | only from triggers which include all these tags. 77 | 78 | - Create Subscription2 with tags: ``["DevOps", "Moira-duty"]`` 79 | 80 | By using Subscription2 you will receive events only for Trigger1 81 | 82 | 83 | .. _system-tags-description: 84 | 85 | System-tags 86 | ----------- 87 | 88 | You can create a subscription using special tags provided by Moira Self State Monitor. 89 | System tags can only be defined in Moira's config and cannot be mutated by any users or admins in Moira's UI. 90 | 91 | Users can only subscribe to system tags to receive notifications from Moira Self State Monitor (read about it :doc:`/user_guide/selfstate`). 92 | 93 | .. note:: 94 | 95 | System tags should be defined in a notifier config. Example can be found in the :ref:`Moira-Notifier section ` 96 | 97 | .. _system-subscriptions-description: 98 | 99 | System-subscriptions 100 | -------------------- 101 | 102 | When the Self State Monitor detects a failure, it sends a notification to admins. However, if end users want to receive notifications about Moira's problems, they can subscribe to Self State Monitor events. 103 | These subscriptions are called system-subscriptions. System-subscriptions are subset of Moira's subscriptions that should have only system-tags in the tags list. 104 | 105 | System-subscriptions cannot be mapped to any triggers because they are used by Moira Self State Monitor. 106 | 107 | You can create the system-subscription on subscription creation modal view. 108 | The first, switch tags selector to "System tags" 109 | 110 | .. image:: ../_static/system_subscription_switch_select.png 111 | 112 | The second, select subscription tags and delivery channels 113 | 114 | .. image:: ../_static/system_subscription_tags.png 115 | 116 | Now, you will receive all system notifications from Self State Monitor designed for end-users! 117 | 118 | 119 | .. _subscription-states-transitions: 120 | 121 | Ignore Specific States Transitions 122 | ---------------------------------- 123 | 124 | You also can reduce number of notifications ignoring unnecessary event. 125 | For this purpose use following check boxes: 126 | 127 | - Send notifications when triggers degraded only 128 | 129 | Only following states transitions will require notifications: 130 | 131 | - ``OK`` |rarr| ``WARN`` 132 | - ``OK`` |rarr| ``ERROR`` 133 | - ``OK`` |rarr| ``NODATA`` 134 | - ``WARN`` |rarr| ``ERROR`` 135 | - ``WARN`` |rarr| ``NODATA`` 136 | - ``ERROR`` |rarr| ``NODATA`` 137 | 138 | - Do not send WARN notifications 139 | 140 | Following states transitions will be ignored: 141 | 142 | - ``OK`` |rarr| ``WARN`` 143 | - ``WARN`` |rarr| ``OK`` 144 | 145 | 146 | Create and Test 147 | --------------- 148 | 149 | You can just save your subscription, but if you want to be 100% sure 150 | it works, you should immediately test it. Dummy notification message 151 | will arrive shortly. 152 | -------------------------------------------------------------------------------- /source/user_guide/throttling.rst: -------------------------------------------------------------------------------- 1 | Throttling 2 | ========== 3 | 4 | Throttling is a distinctive and controversial feature of Moira. 5 | If you are experiencing a delay or any other strange behavior 6 | of notifications, chances are, it is because of throttling. 7 | 8 | To understand throttling, imagine two triggers: 9 | 10 | 1. Send notification if CPU load on any of your servers is more than 75%. 11 | 2. Send notification if there is a fire in your server room. 12 | 13 | It is a busy day, your servers are overloaded, and you are receiving 14 | a ton of notifications about CPU load. Probably, you already have several 15 | dozens of notifications in your inbox. You will likely delete all of them 16 | at once, and you probably won't notice that one of these hundreds of letters 17 | was about a fire in your server room. 18 | 19 | So, the problem is: one misconfigured trigger spoils everything by spamming 20 | your inbox with irrelevant notifications. Moira provides a protection 21 | mechanism called throttling. Simple rules: 22 | 23 | 1. If a trigger sends more than 10 notifications per 1 hour, 24 | limit this trigger to 1 message per 30 minutes. 25 | 2. If a trigger sends more than 20 notifications per 3 hours, 26 | limit this trigger to 1 message per 1 hour. 27 | 28 | It works like this: 29 | 30 | - First notification is delivered immediately. 31 | - Second notification is delivered immediately. 32 | - ... 33 | - Tenth notification is delivered immediately, and you get a warning: 34 | "Please, fix your system or tune this trigger to generate less events." 35 | - Next notifications are delayed so that you receive one message 36 | per 30 minutes/1 hour. Nothing is lost, you just receive one message 37 | with a pack of events. Every message contains a warning: "Please, fix 38 | your system or tune this trigger to generate less events." 39 | 40 | *Moira will enable and disable throttling automatically based 41 | on frequency of events.* 42 | 43 | 44 | Disabling Throttling 45 | -------------------- 46 | 47 | There are four ways to disable throttling for a specific trigger: 48 | 49 | 1. Obey the warning message. That is, fix your system to generate less events. 50 | Or change trigger thresholds. Or use Graphite functions like 51 | ``movingAverage`` to remove spikes from your metric graph. This is the best 52 | method to deal with throttling. 53 | 2. Enable :doc:`maintenance` mode for some of your metrics. This will 54 | temporarily disable checking of a metric and give you time to fix 55 | the system: 56 | 57 | .. image:: ../_static/maintenance.png 58 | :alt: maintenance mode 59 | 60 | 3. Manually reset throttling for your trigger. This basically means that 61 | you've fixed the system and would like to resume operation normally. 62 | It won't help if your trigger is still spamming notifications: 63 | 64 | .. image:: ../_static/reset_throttling.png 65 | :alt: reset throttling 66 | 67 | 4. Entirely disable throttling for a subscription. *This is not recommended*, 68 | unless you really know what you are doing: 69 | 70 | .. image:: ../_static/throttling.png 71 | :alt: disable throttling 72 | :width: 400 73 | -------------------------------------------------------------------------------- /source/user_guide/trigger_page.rst: -------------------------------------------------------------------------------- 1 | Current State and Event History 2 | =============================== 3 | 4 | By clicking on a saved trigger, you can see current state and 5 | event history of this trigger. 6 | 7 | 8 | Current State 9 | ------------- 10 | 11 | Moira shows current state, current value and time of last event 12 | for every separate metric that matches the trigger. 13 | 14 | .. image:: ../_static/current_state.png 15 | :alt: current state 16 | 17 | 18 | Event History 19 | ------------- 20 | 21 | On this tab you can see a chronologically sorted list of events 22 | for each separate metric. Each event includes time, old and new 23 | values. Please, note that the left (old) value is taken from 24 | the previous event, and does not represent metric value just 25 | before the event. 26 | 27 | .. image:: ../_static/event_history.png 28 | :alt: event history 29 | --------------------------------------------------------------------------------