├── README.md ├── docker-compose.yml ├── prometheus ├── Dockerfile └── prometheus.yml ├── screenshot └── screenshot.png └── sentry.conf.py /README.md: -------------------------------------------------------------------------------- 1 | ## Monitor hosted sentry with prometheus 2 | 3 | Since sentry does not support prometheus as a monitoring backend, we can send `StatsdMetricsBackend` metrics to 4 | [statsd_exporter] and configure prometheus to scrape them. 5 | 6 | [statsd_exporter]: https://github.com/prometheus/statsd_exporter 7 | 8 | 9 | ### Demo: 10 | 11 | ![sentry metrics](./screenshot/screenshot.png) 12 | 13 | ### TODO 14 | 15 | - [ ] Add grafana dashboard 16 | - [ ] Deploy to kubernetes/minikube 17 | 18 | ### Test 19 | 20 | ```shell 21 | docker-compose build 22 | docker-compose up 23 | ``` 24 | 25 | - to view the scraped metrics in prometheus navigate to `http://localhost:9090/` 26 | 27 | - to view the translated statsd metrics navigate to 28 | `http://localhost:9102/metrics` 29 | 30 | ### Notes: 31 | 32 | - The first time you do `docker-compose up`, you need to run: 33 | - `docker exec -it bash` 34 | 35 | - `sentry upgrade` (we need to create database tables, and create a superuser account). 36 | - `docker-compose stop` 37 | - `docker-compose up` 38 | 39 | - The statsd metrics backend is configured in `sentry.conf.py`. 40 | Ref: (https://github.com/getsentry/sentry/blob/master/docs/internal-metrics.rst) 41 | 42 | ```python 43 | # Sentry statsd exporter config 44 | SENTRY_METRICS_BACKEND = 'sentry.metrics.statsd.StatsdMetricsBackend' 45 | SENTRY_METRICS_OPTIONS = { 46 | 'host': os.environ.get('SENTRY_METRICS_HOST'), 47 | 'port': os.environ.get('SENTRY_METRICS_PORT') 48 | } 49 | ``` 50 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '2' 2 | services: 3 | prometheus: 4 | build: 5 | context: ./prometheus 6 | ports: 7 | - "9090:9090" 8 | command: 9 | - '--config.file=/etc/prometheus/prometheus.yml' 10 | 11 | redis: 12 | image: redis 13 | 14 | postgres: 15 | image: postgres 16 | environment: 17 | POSTGRES_PASSWORD: secret 18 | POSTGRES_USER: sentry 19 | 20 | sentry-web: 21 | image: sentry:8.19 22 | command: > 23 | bash -c "pip uninstall -y redis \ 24 | && pip install redis==2.10.5 \ 25 | && pip install 'botocore<1.8.0,>=1.7.0' \ 26 | && sentry run web" 27 | ports: 28 | - 8080:9000 29 | links: 30 | - redis 31 | - postgres 32 | environment: 33 | SENTRY_SECRET_KEY: 'notasecretkey' 34 | SENTRY_USE_SSL: 'false' 35 | SENTRY_EMAIL_HOST: 'mail.example.com' 36 | SENTRY_EMAIL_PASSWORD: '' 37 | SENTRY_EMAIL_PORT: 587 38 | SENTRY_EMAIL_USER: 'notify' 39 | SENTRY_EMAIL_USE_TLS: 'true' 40 | SENTRY_SERVER_EMAIL: 'notify@example.com' 41 | SENTRY_REDIS_HOST: redis 42 | SENTRY_METRICS_HOST: prometheusexporter 43 | SENTRY_METRICS_PORT: 8125 44 | DATABASE_URL: postgresql://sentry:secret@postgres:5432/sentry 45 | volumes: 46 | - ./sentry.conf.py:/etc/sentry/sentry.conf.py:ro 47 | 48 | sentry-cron: 49 | image: sentry:8.19 50 | links: 51 | - redis 52 | - postgres 53 | command: > 54 | bash -c "pip uninstall -y redis \ 55 | && pip install redis==2.10.5 \ 56 | && pip install 'botocore<1.8.0,>=1.7.0' \ 57 | && sentry run cron" 58 | environment: 59 | SENTRY_SECRET_KEY: 'notasecretkey' 60 | SENTRY_REDIS_HOST: redis 61 | DATABASE_URL: postgresql://sentry:secret@postgres:5432/sentry 62 | SENTRY_METRICS_HOST: prometheusexporter 63 | SENTRY_METRICS_PORT: 8125 64 | volumes: 65 | - ./sentry.conf.py:/etc/sentry/sentry.conf.py:ro 66 | 67 | sentry-worker: 68 | image: sentry:8.19 69 | links: 70 | - redis 71 | - postgres 72 | command: > 73 | bash -c "pip uninstall -y redis \ 74 | && pip install redis==2.10.5 \ 75 | && pip install 'botocore<1.8.0,>=1.7.0' \ 76 | && sentry run worker" 77 | environment: 78 | SENTRY_SECRET_KEY: 'notasecretkey' 79 | SENTRY_REDIS_HOST: redis 80 | DATABASE_URL: postgresql://sentry:secret@postgres:5432/sentry 81 | # prom statd exporter host 82 | SENTRY_METRICS_HOST: prometheusexporter 83 | # port to send statsd metrics 84 | SENTRY_METRICS_PORT: 8125 85 | # prevent worker from exiting, NSFW 86 | C_FORCE_ROOT: "true" 87 | volumes: 88 | - ./sentry.conf.py:/etc/sentry/sentry.conf.py:ro 89 | 90 | prometheusexporter: 91 | image: prom/statsd-exporter:latest 92 | command: -statsd.listen-udp=:8125 93 | ports: 94 | - 9102:9102 95 | - 9125:9125 96 | - 125:9125/udp 97 | # port to receive statsd metrics 98 | - 8125:8125/udp 99 | 100 | -------------------------------------------------------------------------------- /prometheus/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM prom/prometheus:v2.0.0 2 | 3 | COPY prometheus.yml /etc/prometheus/prometheus.yml 4 | -------------------------------------------------------------------------------- /prometheus/prometheus.yml: -------------------------------------------------------------------------------- 1 | global: 2 | scrape_interval: 60s 3 | scrape_timeout: 10s 4 | evaluation_interval: 10s 5 | 6 | 7 | scrape_configs: 8 | 9 | - job_name: 'prometheus-sentry-exporter' 10 | static_configs: 11 | - targets: 12 | - 'prometheusexporter:9102' 13 | 14 | -------------------------------------------------------------------------------- /screenshot/screenshot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/saady/sentry-prometheus/56c93997b39cbca5639316ba2447fa6b80c1de1c/screenshot/screenshot.png -------------------------------------------------------------------------------- /sentry.conf.py: -------------------------------------------------------------------------------- 1 | # This file is just Python, with a touch of Django which means 2 | # you can inherit and tweak settings to your hearts content. 3 | 4 | # For Docker, the following environment variables are supported: 5 | # SENTRY_POSTGRES_HOST 6 | # SENTRY_POSTGRES_PORT 7 | # SENTRY_DB_NAME 8 | # SENTRY_DB_USER 9 | # SENTRY_DB_PASSWORD 10 | # SENTRY_RABBITMQ_HOST 11 | # SENTRY_RABBITMQ_USERNAME 12 | # SENTRY_RABBITMQ_PASSWORD 13 | # SENTRY_RABBITMQ_VHOST 14 | # SENTRY_REDIS_HOST 15 | # SENTRY_REDIS_PASSWORD 16 | # SENTRY_REDIS_PORT 17 | # SENTRY_REDIS_DB 18 | # SENTRY_MEMCACHED_HOST 19 | # SENTRY_MEMCACHED_PORT 20 | # SENTRY_FILESTORE_DIR 21 | # SENTRY_SERVER_EMAIL 22 | # SENTRY_EMAIL_HOST 23 | # SENTRY_EMAIL_PORT 24 | # SENTRY_EMAIL_USER 25 | # SENTRY_EMAIL_PASSWORD 26 | # SENTRY_EMAIL_USE_TLS 27 | # SENTRY_ENABLE_EMAIL_REPLIES 28 | # SENTRY_SMTP_HOSTNAME 29 | # SENTRY_MAILGUN_API_KEY 30 | # SENTRY_SINGLE_ORGANIZATION 31 | # SENTRY_SECRET_KEY 32 | # GITHUB_APP_ID 33 | # GITHUB_API_SECRET 34 | # BITBUCKET_CONSUMER_KEY 35 | # BITBUCKET_CONSUMER_SECRET 36 | from sentry.conf.server import * # NOQA 37 | 38 | import os 39 | import os.path 40 | 41 | CONF_ROOT = os.path.dirname(__file__) 42 | 43 | DATABASE_URL = os.environ.get('DATABASE_URL') 44 | 45 | # You should not change this setting after your database has been created 46 | # unless you have altered all schemas first 47 | SENTRY_USE_BIG_INTS = True 48 | 49 | # If you're expecting any kind of real traffic on Sentry, we highly recommend 50 | # configuring the CACHES and Redis settings 51 | 52 | ########### 53 | # General # 54 | ########### 55 | 56 | # Instruct Sentry that this install intends to be run by a single organization 57 | # and thus various UI optimizations should be enabled. 58 | SENTRY_SINGLE_ORGANIZATION = env('SENTRY_SINGLE_ORGANIZATION', True) 59 | 60 | ######### 61 | # Redis # 62 | ######### 63 | 64 | # Generic Redis configuration used as defaults for various things including: 65 | # Buffers, Quotas, TSDB 66 | 67 | redis = env('SENTRY_REDIS_HOST') or (env('REDIS_PORT_6379_TCP_ADDR') and 'redis') 68 | if not redis: 69 | raise Exception('Error: REDIS_PORT_6379_TCP_ADDR (or SENTRY_REDIS_HOST) is undefined, did you forget to `--link` a redis container?') 70 | 71 | redis_password = env('SENTRY_REDIS_PASSWORD') or '' 72 | redis_port = env('SENTRY_REDIS_PORT') or '6379' 73 | redis_db = env('SENTRY_REDIS_DB') or '0' 74 | 75 | SENTRY_OPTIONS.update({ 76 | 'redis.clusters': { 77 | 'default': { 78 | 'hosts': { 79 | 0: { 80 | 'host': redis, 81 | 'password': redis_password, 82 | 'port': redis_port, 83 | 'db': redis_db, 84 | }, 85 | }, 86 | }, 87 | }, 88 | }) 89 | 90 | ######### 91 | # Cache # 92 | ######### 93 | 94 | # Sentry currently utilizes two separate mechanisms. While CACHES is not a 95 | # requirement, it will optimize several high throughput patterns. 96 | 97 | memcached = env('SENTRY_MEMCACHED_HOST') or (env('MEMCACHED_PORT_11211_TCP_ADDR') and 'memcached') 98 | if memcached: 99 | memcached_port = ( 100 | env('SENTRY_MEMCACHED_PORT') 101 | or '11211' 102 | ) 103 | CACHES = { 104 | 'default': { 105 | 'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache', 106 | 'LOCATION': [memcached + ':' + memcached_port], 107 | 'TIMEOUT': 3600, 108 | } 109 | } 110 | 111 | # A primary cache is required for things such as processing events 112 | SENTRY_CACHE = 'sentry.cache.redis.RedisCache' 113 | 114 | ######### 115 | # Queue # 116 | ######### 117 | 118 | # See https://docs.getsentry.com/on-premise/server/queue/ for more 119 | # information on configuring your queue broker and workers. Sentry relies 120 | # on a Python framework called Celery to manage queues. 121 | 122 | rabbitmq = env('SENTRY_RABBITMQ_HOST') or (env('RABBITMQ_PORT_5672_TCP_ADDR') and 'rabbitmq') 123 | 124 | if rabbitmq: 125 | BROKER_URL = ( 126 | 'amqp://' + ( 127 | env('SENTRY_RABBITMQ_USERNAME') 128 | or env('RABBITMQ_ENV_RABBITMQ_DEFAULT_USER') 129 | or 'guest' 130 | ) + ':' + ( 131 | env('SENTRY_RABBITMQ_PASSWORD') 132 | or env('RABBITMQ_ENV_RABBITMQ_DEFAULT_PASS') 133 | or 'guest' 134 | ) + '@' + rabbitmq + '/' + ( 135 | env('SENTRY_RABBITMQ_VHOST') 136 | or env('RABBITMQ_ENV_RABBITMQ_DEFAULT_VHOST') 137 | or '/' 138 | ) 139 | ) 140 | else: 141 | BROKER_URL = 'redis://:' + redis_password + '@' + redis + ':' + redis_port + '/' + redis_db 142 | 143 | 144 | ############### 145 | # Rate Limits # 146 | ############### 147 | 148 | # Rate limits apply to notification handlers and are enforced per-project 149 | # automatically. 150 | 151 | SENTRY_RATELIMITER = 'sentry.ratelimits.redis.RedisRateLimiter' 152 | 153 | ################## 154 | # Update Buffers # 155 | ################## 156 | 157 | # Buffers (combined with queueing) act as an intermediate layer between the 158 | # database and the storage API. They will greatly improve efficiency on large 159 | # numbers of the same events being sent to the API in a short amount of time. 160 | # (read: if you send any kind of real data to Sentry, you should enable buffers) 161 | 162 | SENTRY_BUFFER = 'sentry.buffer.redis.RedisBuffer' 163 | 164 | ########## 165 | # Quotas # 166 | ########## 167 | 168 | # Quotas allow you to rate limit individual projects or the Sentry install as 169 | # a whole. 170 | 171 | SENTRY_QUOTAS = 'sentry.quotas.redis.RedisQuota' 172 | 173 | ######## 174 | # TSDB # 175 | ######## 176 | 177 | # The TSDB is used for building charts as well as making things like per-rate 178 | # alerts possible. 179 | 180 | SENTRY_TSDB = 'sentry.tsdb.redis.RedisTSDB' 181 | 182 | ########### 183 | # Digests # 184 | ########### 185 | 186 | # The digest backend powers notification summaries. 187 | 188 | SENTRY_DIGESTS = 'sentry.digests.backends.redis.RedisBackend' 189 | 190 | ################ 191 | # File storage # 192 | ################ 193 | 194 | # Uploaded media uses these `filestore` settings. The available 195 | # backends are either `filesystem` or `s3`. 196 | 197 | SENTRY_OPTIONS['filestore.backend'] = 'filesystem' 198 | SENTRY_OPTIONS['filestore.options'] = { 199 | 'location': env('SENTRY_FILESTORE_DIR'), 200 | } 201 | 202 | ############## 203 | # Web Server # 204 | ############## 205 | 206 | # If you're using a reverse SSL proxy, you should enable the X-Forwarded-Proto 207 | # header and set `SENTRY_USE_SSL=1` 208 | 209 | if env('SENTRY_USE_SSL', False): 210 | SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https') 211 | SESSION_COOKIE_SECURE = True 212 | CSRF_COOKIE_SECURE = True 213 | SOCIAL_AUTH_REDIRECT_IS_HTTPS = True 214 | 215 | SENTRY_FEATURES['auth:register'] = False 216 | 217 | SENTRY_WEB_HOST = '0.0.0.0' 218 | SENTRY_WEB_PORT = 9000 219 | SENTRY_WEB_OPTIONS = { 220 | # 'workers': 3, # the number of web workers 221 | } 222 | 223 | ############### 224 | # Mail Server # 225 | ############### 226 | 227 | 228 | email = env('SENTRY_EMAIL_HOST') or (env('SMTP_PORT_25_TCP_ADDR') and 'smtp') 229 | if email: 230 | SENTRY_OPTIONS['mail.backend'] = 'smtp' 231 | SENTRY_OPTIONS['mail.host'] = email 232 | SENTRY_OPTIONS['mail.password'] = env('SENTRY_EMAIL_PASSWORD') or '' 233 | SENTRY_OPTIONS['mail.username'] = env('SENTRY_EMAIL_USER') or '' 234 | SENTRY_OPTIONS['mail.port'] = int(env('SENTRY_EMAIL_PORT') or 25) 235 | SENTRY_OPTIONS['mail.use-tls'] = env('SENTRY_EMAIL_USE_TLS', False) 236 | else: 237 | SENTRY_OPTIONS['mail.backend'] = 'dummy' 238 | 239 | # The email address to send on behalf of 240 | SENTRY_OPTIONS['mail.from'] = env('SENTRY_SERVER_EMAIL') or 'root@localhost' 241 | 242 | # If you're using mailgun for inbound mail, set your API key and configure a 243 | # route to forward to /api/hooks/mailgun/inbound/ 244 | SENTRY_OPTIONS['mail.mailgun-api-key'] = env('SENTRY_MAILGUN_API_KEY') or '' 245 | 246 | # If you specify a MAILGUN_API_KEY, you definitely want EMAIL_REPLIES 247 | if SENTRY_OPTIONS['mail.mailgun-api-key']: 248 | SENTRY_OPTIONS['mail.enable-replies'] = True 249 | else: 250 | SENTRY_OPTIONS['mail.enable-replies'] = env('SENTRY_ENABLE_EMAIL_REPLIES', False) 251 | 252 | if SENTRY_OPTIONS['mail.enable-replies']: 253 | SENTRY_OPTIONS['mail.reply-hostname'] = env('SENTRY_SMTP_HOSTNAME') or '' 254 | 255 | # If this value ever becomes compromised, it's important to regenerate your 256 | # SENTRY_SECRET_KEY. Changing this value will result in all current sessions 257 | # being invalidated. 258 | secret_key = env('SENTRY_SECRET_KEY') 259 | if not secret_key: 260 | raise Exception('Error: SENTRY_SECRET_KEY is undefined, run `generate-secret-key` and set to -e SENTRY_SECRET_KEY') 261 | 262 | if 'SENTRY_RUNNING_UWSGI' not in os.environ and len(secret_key) < 32: 263 | print('!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!') 264 | print('!! CAUTION !!') 265 | print('!! Your SENTRY_SECRET_KEY is potentially insecure. !!') 266 | print('!! We recommend at least 32 characters long. !!') 267 | print('!! Regenerate with `generate-secret-key`. !!') 268 | print('!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!') 269 | 270 | SENTRY_OPTIONS['system.secret-key'] = secret_key 271 | 272 | if 'GITHUB_APP_ID' in os.environ: 273 | GITHUB_EXTENDED_PERMISSIONS = ['repo'] 274 | GITHUB_APP_ID = env('GITHUB_APP_ID') 275 | GITHUB_API_SECRET = env('GITHUB_API_SECRET') 276 | 277 | if 'BITBUCKET_CONSUMER_KEY' in os.environ: 278 | BITBUCKET_CONSUMER_KEY = env('BITBUCKET_CONSUMER_KEY') 279 | BITBUCKET_CONSUMER_SECRET = env('BITBUCKET_CONSUMER_SECRET') 280 | 281 | # Sentry statsd exporter config 282 | SENTRY_METRICS_BACKEND = 'sentry.metrics.statsd.StatsdMetricsBackend' 283 | SENTRY_METRICS_OPTIONS = { 284 | 'host': os.environ.get('SENTRY_METRICS_HOST'), 285 | 'port': os.environ.get('SENTRY_METRICS_PORT') 286 | } 287 | --------------------------------------------------------------------------------