├── .gitignore ├── LICENSE ├── README.md ├── django_traffic ├── __init__.py ├── middleware.py └── models.py ├── example_project ├── example_project │ ├── __init__.py │ ├── settings.py │ ├── urls.py │ └── wsgi.py └── manage.py ├── requirements.txt ├── setup.cfg └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | env/ 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | .eggs/ 18 | lib/ 19 | lib64/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | *.egg-info/ 24 | .installed.cfg 25 | *.egg 26 | 27 | # PyInstaller 28 | # Usually these files are written by a python script from a template 29 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 30 | *.manifest 31 | *.spec 32 | 33 | # Installer logs 34 | pip-log.txt 35 | pip-delete-this-directory.txt 36 | 37 | # Unit test / coverage reports 38 | htmlcov/ 39 | .tox/ 40 | .coverage 41 | .coverage.* 42 | .cache 43 | nosetests.xml 44 | coverage.xml 45 | *,cover 46 | .hypothesis/ 47 | 48 | # Translations 49 | *.mo 50 | *.pot 51 | 52 | # Django stuff: 53 | *.log 54 | local_settings.py 55 | 56 | # Flask stuff: 57 | instance/ 58 | .webassets-cache 59 | 60 | # Scrapy stuff: 61 | .scrapy 62 | 63 | # Sphinx documentation 64 | docs/_build/ 65 | 66 | # PyBuilder 67 | target/ 68 | 69 | # IPython Notebook 70 | .ipynb_checkpoints 71 | 72 | # pyenv 73 | .python-version 74 | 75 | # celery beat schedule file 76 | celerybeat-schedule 77 | 78 | # dotenv 79 | .env 80 | 81 | # virtualenv 82 | venv/ 83 | ENV/ 84 | 85 | # Spyder project settings 86 | .spyderproject 87 | 88 | # Rope project settings 89 | .ropeproject 90 | 91 | .idea 92 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2016 Konstantinos Livieratos 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # django-traffic 2 | A Django middleware that helps visualize your app's traffic in Kibana 3 | 4 | In a nutshell, by using this middleware you need no more effort to stream your app's traffic in your 5 | ElasticSearch host(s) and use Kibana for visualizations around it. 6 | 7 | Request information transmitted to ElasticSearch: 8 | - `timestamp` | type: `date` 9 | - `location` | type: `geo_point` 10 | - `method` | type: `string` 11 | - `body` | type: `string` 12 | - `path` | type: `string` 13 | - `path_info` | type: `string` 14 | - `scheme` | type: `string` 15 | - `encoding` | type: `string` 16 | - `encoding_type` | type: `string`, supported only in `Django 1.10` and later 17 | - `ip` | type: `ip` 18 | 19 | Geolocation is achieved by using `django.contrib.gis.geoip2.GeoIP2` wrapper, included in Django 1.9 and latter. 20 | 21 | # Quick Start 22 | **1. Install using pip:** 23 | 24 | ``` 25 | pip install django-traffic 26 | ``` 27 | To install the latest version directly from GitHub: 28 | 29 | ``` 30 | pip install git+https://github.com/koslibpro/django-traffic 31 | ``` 32 | 33 | **2. Include "django-traffic" in your INSTALLED_APPS:** 34 | 35 | ``` 36 | INSTALLED_APPS = [ 37 | ... 38 | 'django_traffic', 39 | ] 40 | ``` 41 | 42 | **3. Include "ESTrafficInfoMiddleware" to your MIDDLEWARE_CLASSES:** 43 | 44 | ``` 45 | MIDDLEWARE_CLASSES = ( 46 | ... 47 | 'django_traffic.middleware.ESTrafficInfoMiddleware', 48 | ) 49 | ``` 50 | 51 | The middleware is created in a way that supports both `MIDDLEWARE_CLASSES` on Django older versions and `MIDDLEWARE` in latest Django versions. 52 | 53 | # Configuration 54 | There are some variables required in your `settings.py` file to function normally. 55 | - `TRAFFIC_INDEX_NAME`: this is the name of the index that will be used in ElasticSearch. 56 | If you leave this empty or do not define it, the default index name will be applied: `django-traffic`. 57 | 58 | - `ES_CLIENT`: if you already have an ElasticSearch() client instance ready in your app, we'll use this one be default. 59 | 60 | - `ES_HOST`: this is required only if you don't have an `ES_CLIENT` defined. Here the lib expects to find a hostname 61 | (including the port) of your ElasticSearch instance. 62 | 63 | - `GEO_DB_PATH`: if you don't have `GEOIP_PATH` already defined, you need to define the path where django can find your 64 | geolocation database. 65 | 66 | - `LOG_WITHOUT_LOCATION`: (default `False`) log data even when `ip` cannot be translated to `location` 67 | 68 | After you deploy your project or locally run your django server, requests and traffic to your web app will be sent to 69 | the ElasticSearch hosts defined. Practically you are ready to create a Kibana map-tile visualization and start watching 70 | where traffic is flooding you in. 71 | 72 | # Contributing - Error Reporting 73 | This lib was made with my own needs in mind, so it's uncommon to fit everyone's project out there. 74 | For Error Reporting, open an issue in Github and I will try to take care of it as soon as possible. 75 | If you are able to contribute to this lib and make it better, feel free to fork it and adjust it to your use-case. 76 | 77 | -------------------------------------------------------------------------------- /django_traffic/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koslib/django-traffic/014ac1928decd9015b76c58bf63033b73f9e3c2b/django_traffic/__init__.py -------------------------------------------------------------------------------- /django_traffic/middleware.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # Author: Konstantinos Livieratos 3 | # https://github.com/koslibpro/django-traffic/ 4 | 5 | import logging 6 | 7 | from django.conf import settings 8 | from django.utils import timezone 9 | from django.utils.deprecation import MiddlewareMixin 10 | from django.contrib.gis.geoip2 import GeoIP2 11 | 12 | from elasticsearch import Elasticsearch 13 | 14 | 15 | def _load_geo_db(): 16 | geo_db_path = getattr(settings, 'GEO_DB_PATH', None) 17 | if geo_db_path is None: 18 | if hasattr(settings, 'GEOIP_PATH'): 19 | g = GeoIP2() 20 | return g 21 | logging.error('[django-traffic] GEOIP_PATH or GEO_DB_PATH should be present in settings.py') 22 | return None 23 | return GeoIP2(path=geo_db_path) 24 | 25 | 26 | class ESTrafficInfoMiddleware(MiddlewareMixin): 27 | def __init__(self, get_response=None): 28 | self.get_response = get_response 29 | 30 | if hasattr(settings, 'TRAFFIC_INDEX_NAME'): 31 | self.index_name = getattr(settings, 'TRAFFIC_INDEX_NAME') 32 | else: 33 | self.index_name = "django-traffic" 34 | 35 | # if settings.ES_CLIENT: 36 | if hasattr(settings, 'ES_CLIENT'): 37 | self.es = settings.ES_CLIENT 38 | else: 39 | assert settings.ES_HOST, 'ES_HOST definition in settings.py is required' 40 | self.es = Elasticsearch( 41 | hosts=[settings.ES_HOST] 42 | ) 43 | 44 | super(ESTrafficInfoMiddleware, self).__init__(get_response=get_response) 45 | 46 | def process_request(self, request, *args, **kwargs): 47 | self.es_upstream(request) 48 | return None 49 | 50 | def ip_to_cordinates(self, device_ip): 51 | g = _load_geo_db() 52 | try: 53 | lat, lng = g.lat_lon(device_ip) 54 | except Exception as e: 55 | logging.error("[django-traffic] Error while getting lan/lon from GEOIP2: %s " % e) 56 | return None, None 57 | else: 58 | return lat, lng 59 | 60 | def es_upstream(self, request): 61 | device_ip = request.META.get('HTTP_X_FORWARDED_FOR') or request.META.get('REMOTE_ADDR') 62 | device_ip = str(device_ip).split(',')[0] 63 | logging.info("[django-traffic] Device IP: %s" % device_ip) 64 | 65 | if not self.es.indices.exists(index=self.index_name): 66 | mapping = { 67 | "mappings": { 68 | "request-info": { 69 | "properties": { 70 | "timestamp": { 71 | "type": "date" 72 | }, 73 | "text": { 74 | "type": "string" 75 | }, 76 | "location": { 77 | "type": "geo_point" 78 | }, 79 | "method": { 80 | "type": "string" 81 | }, 82 | "body": { 83 | "type": "string" 84 | }, 85 | "path": { 86 | "type": "string" 87 | }, 88 | "path_info": { 89 | "type": "string" 90 | }, 91 | "scheme": { 92 | "type": "string" 93 | }, 94 | "encoding": { 95 | "type": "string" 96 | }, 97 | "encoding_type": { 98 | "type": "string" 99 | }, 100 | "ip_addr": { 101 | "type": "ip" 102 | } 103 | } 104 | } 105 | } 106 | } 107 | logging.info("[django-traffic] Creating new elasticsearch indice...") 108 | self.es.indices.create(index=self.index_name, body=mapping) 109 | 110 | doc = { 111 | "timestamp": timezone.now(), 112 | "text": "django-traffic geo-point object", 113 | "method": request.method, 114 | "body": request.body, 115 | "path": request.path, 116 | "path_info": request.path_info, 117 | "scheme": request.scheme, 118 | "encoding": request.encoding, 119 | "encoding_type": getattr(request, 'encoding_type', ''), 120 | "ip_addr": device_ip 121 | } 122 | 123 | lat, lng = self.ip_to_cordinates(device_ip) 124 | if lat and lng: 125 | doc["location"] = { 126 | "lat": lat, 127 | "lon": lng 128 | } 129 | else: 130 | if not getattr(settings, 'LOG_WITHOUT_LOCATION', False): 131 | return 132 | 133 | res = self.es.index(index=self.index_name, doc_type='request-info', body=doc) 134 | if res['created']: 135 | logging.info("[django-traffic] request indexed") 136 | -------------------------------------------------------------------------------- /django_traffic/models.py: -------------------------------------------------------------------------------- 1 | from django.db import models 2 | 3 | # Create your models here. -------------------------------------------------------------------------------- /example_project/example_project/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koslib/django-traffic/014ac1928decd9015b76c58bf63033b73f9e3c2b/example_project/example_project/__init__.py -------------------------------------------------------------------------------- /example_project/example_project/settings.py: -------------------------------------------------------------------------------- 1 | """ 2 | Django settings for example_project project. 3 | 4 | Generated by 'django-admin startproject' using Django 1.10.2. 5 | 6 | For more information on this file, see 7 | https://docs.djangoproject.com/en/1.10/topics/settings/ 8 | 9 | For the full list of settings and their values, see 10 | https://docs.djangoproject.com/en/1.10/ref/settings/ 11 | """ 12 | 13 | import os 14 | 15 | # Build paths inside the project like this: os.path.join(BASE_DIR, ...) 16 | BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) 17 | 18 | 19 | # Quick-start development settings - unsuitable for production 20 | # See https://docs.djangoproject.com/en/1.10/howto/deployment/checklist/ 21 | 22 | # SECURITY WARNING: keep the secret key used in production secret! 23 | SECRET_KEY = '5nh86-(yn1loxq5rcb3zlc-gsirjbuh*5a(exc)8z%ako(imrv' 24 | 25 | # SECURITY WARNING: don't run with debug turned on in production! 26 | DEBUG = True 27 | 28 | ALLOWED_HOSTS = [] 29 | 30 | 31 | # Application definition 32 | 33 | INSTALLED_APPS = [ 34 | 'django.contrib.admin', 35 | 'django.contrib.auth', 36 | 'django.contrib.contenttypes', 37 | 'django.contrib.sessions', 38 | 'django.contrib.messages', 39 | 'django.contrib.staticfiles', 40 | 41 | 'django_traffic' 42 | ] 43 | 44 | MIDDLEWARE = [ 45 | 'django.middleware.security.SecurityMiddleware', 46 | 'django.contrib.sessions.middleware.SessionMiddleware', 47 | 'django.middleware.common.CommonMiddleware', 48 | 'django.middleware.csrf.CsrfViewMiddleware', 49 | 'django.contrib.auth.middleware.AuthenticationMiddleware', 50 | 'django.contrib.messages.middleware.MessageMiddleware', 51 | 'django.middleware.clickjacking.XFrameOptionsMiddleware', 52 | 53 | 'django_traffic.middleware.ESTrafficInfoMiddleware' 54 | ] 55 | 56 | ROOT_URLCONF = 'example_project.urls' 57 | 58 | TEMPLATES = [ 59 | { 60 | 'BACKEND': 'django.template.backends.django.DjangoTemplates', 61 | 'DIRS': [], 62 | 'APP_DIRS': True, 63 | 'OPTIONS': { 64 | 'context_processors': [ 65 | 'django.template.context_processors.debug', 66 | 'django.template.context_processors.request', 67 | 'django.contrib.auth.context_processors.auth', 68 | 'django.contrib.messages.context_processors.messages', 69 | ], 70 | }, 71 | }, 72 | ] 73 | 74 | WSGI_APPLICATION = 'example_project.wsgi.application' 75 | 76 | 77 | # Database 78 | # https://docs.djangoproject.com/en/1.10/ref/settings/#databases 79 | 80 | DATABASES = { 81 | 'default': { 82 | 'ENGINE': 'django.db.backends.sqlite3', 83 | 'NAME': os.path.join(BASE_DIR, 'db.sqlite3'), 84 | } 85 | } 86 | 87 | 88 | # Password validation 89 | # https://docs.djangoproject.com/en/1.10/ref/settings/#auth-password-validators 90 | 91 | AUTH_PASSWORD_VALIDATORS = [ 92 | { 93 | 'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator', 94 | }, 95 | { 96 | 'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator', 97 | }, 98 | { 99 | 'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator', 100 | }, 101 | { 102 | 'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator', 103 | }, 104 | ] 105 | 106 | 107 | # Internationalization 108 | # https://docs.djangoproject.com/en/1.10/topics/i18n/ 109 | 110 | LANGUAGE_CODE = 'en-us' 111 | 112 | TIME_ZONE = 'UTC' 113 | 114 | USE_I18N = True 115 | 116 | USE_L10N = True 117 | 118 | USE_TZ = True 119 | 120 | 121 | # Static files (CSS, JavaScript, Images) 122 | # https://docs.djangoproject.com/en/1.10/howto/static-files/ 123 | 124 | STATIC_URL = '/static/' 125 | 126 | # django-traffic config 127 | TRAFFIC_INDEX_NAME = 'my-index-name' 128 | ES_HOST = 'https://example.com:9200' 129 | GEO_DB_PATH = 'GeoLite2-City.mmdb' 130 | -------------------------------------------------------------------------------- /example_project/example_project/urls.py: -------------------------------------------------------------------------------- 1 | """example_project URL Configuration 2 | 3 | The `urlpatterns` list routes URLs to views. For more information please see: 4 | https://docs.djangoproject.com/en/1.10/topics/http/urls/ 5 | Examples: 6 | Function views 7 | 1. Add an import: from my_app import views 8 | 2. Add a URL to urlpatterns: url(r'^$', views.home, name='home') 9 | Class-based views 10 | 1. Add an import: from other_app.views import Home 11 | 2. Add a URL to urlpatterns: url(r'^$', Home.as_view(), name='home') 12 | Including another URLconf 13 | 1. Import the include() function: from django.conf.urls import url, include 14 | 2. Add a URL to urlpatterns: url(r'^blog/', include('blog.urls')) 15 | """ 16 | from django.conf.urls import url 17 | from django.contrib import admin 18 | 19 | urlpatterns = [ 20 | url(r'^admin/', admin.site.urls), 21 | ] 22 | -------------------------------------------------------------------------------- /example_project/example_project/wsgi.py: -------------------------------------------------------------------------------- 1 | """ 2 | WSGI config for example_project project. 3 | 4 | It exposes the WSGI callable as a module-level variable named ``application``. 5 | 6 | For more information on this file, see 7 | https://docs.djangoproject.com/en/1.10/howto/deployment/wsgi/ 8 | """ 9 | 10 | import os 11 | 12 | from django.core.wsgi import get_wsgi_application 13 | 14 | os.environ.setdefault("DJANGO_SETTINGS_MODULE", "example_project.settings") 15 | 16 | application = get_wsgi_application() 17 | -------------------------------------------------------------------------------- /example_project/manage.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import os 3 | import sys 4 | 5 | if __name__ == "__main__": 6 | os.environ.setdefault("DJANGO_SETTINGS_MODULE", "example_project.settings") 7 | try: 8 | from django.core.management import execute_from_command_line 9 | except ImportError: 10 | # The above import may fail for some other reason. Ensure that the 11 | # issue is really that Django is missing to avoid masking other 12 | # exceptions on Python 2. 13 | try: 14 | import django 15 | except ImportError: 16 | raise ImportError( 17 | "Couldn't import Django. Are you sure it's installed and " 18 | "available on your PYTHONPATH environment variable? Did you " 19 | "forget to activate a virtual environment?" 20 | ) 21 | raise 22 | execute_from_command_line(sys.argv) 23 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | Django>=1.9 2 | elasticsearch==5.0.0 3 | geoip2>=2.3.0 4 | ipaddress==1.0.17 5 | maxminddb>=1.2.1 6 | requests==2.32.2 7 | urllib3==1.26.5 8 | wheel==0.38.1 9 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | description-file = README.md -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from distutils.core import setup 2 | 3 | setup( 4 | name='django-traffic', 5 | packages=['django_traffic'], 6 | version='1.2.4', 7 | description='Django middleware that helps visualize your app\'s traffic in Kibana', 8 | author='Konstantinos Livieratos', 9 | author_email='livieratos.konstantinos@gmail.com', 10 | url='https://github.com/koslibpro/django-traffic', 11 | download_url='https://github.com/koslibpro/django-traffic/tarball/1.2.4', 12 | keywords=['django', 'kibana', 'elasticsearch', 'traffic'], 13 | classifiers=[], 14 | ) 15 | --------------------------------------------------------------------------------