├── .github └── ISSUE_TEMPLATE │ ├── bug_report.md │ └── feature_request.md ├── .gitignore ├── LICENSE ├── README.rst ├── app ├── __init__.py ├── core │ ├── __init__.py │ ├── api.py │ └── generic.py ├── resources │ ├── asn │ │ ├── as_names.json │ │ └── ribs │ │ │ └── __init__.py │ ├── config.json │ └── geoip │ │ └── __init__.py ├── tasks │ └── __init__.py ├── templates │ └── index.html └── utils │ ├── __init__.py │ └── helpers.py ├── requirements.txt ├── scripts ├── fetch.py └── fetch_geo.py ├── server.py ├── service ├── README.rst ├── netinfo.service └── netinfod.service ├── worker.py └── wsgi.ini /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Describe the bug** 11 | A clear and concise description of what the bug is. 12 | 13 | **To Reproduce** 14 | Steps to reproduce the behavior: 15 | 1. Go to '...' 16 | 2. Click on '....' 17 | 3. Scroll down to '....' 18 | 4. See error 19 | 20 | **Expected behavior** 21 | A clear and concise description of what you expected to happen. 22 | 23 | **Screenshots** 24 | If applicable, add screenshots to help explain your problem. 25 | 26 | **Desktop (please complete the following information):** 27 | - OS: [e.g. iOS] 28 | - Browser [e.g. chrome, safari] 29 | - Version [e.g. 22] 30 | 31 | **Smartphone (please complete the following information):** 32 | - Device: [e.g. iPhone6] 33 | - OS: [e.g. iOS8.1] 34 | - Browser [e.g. stock browser, safari] 35 | - Version [e.g. 22] 36 | 37 | **Additional context** 38 | Add any other context about the problem here. 39 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is your feature request related to a problem? Please describe.** 11 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 12 | 13 | **Describe the solution you'd like** 14 | A clear and concise description of what you want to happen. 15 | 16 | **Describe alternatives you've considered** 17 | A clear and concise description of any alternative solutions or features you've considered. 18 | 19 | **Additional context** 20 | Add any other context or screenshots about the feature request here. 21 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | .DS_Store 3 | ._* 4 | Thumbs.db 5 | *.sublime-project 6 | *.sublime-workspace 7 | .jshintrc 8 | bkp 9 | ssl-cert 10 | docs/s3_website.yml 11 | extension/Archive.zip 12 | venv/* 13 | venv3/* 14 | dump.rdb 15 | .notes 16 | tools/* 17 | celerybeat-schedule.db 18 | *.bz2 19 | current -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright Brandon Dixon 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 4 | 5 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 6 | 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | NetInfo 2 | ======= 3 | NetInfo is a simple IP enrichment service to provide additional data related to an IP address. The primary utility of NetInfo is to serve as a API wrapper and management system for the PyASN and MaxMind GeoIP libraries. NetInfo will automatically seek and download new update files, ensuring the databases are always up-to-date. The local API queries the PyASN and GeoIP instance and returns back enrichment data for an IP address. 4 | 5 | Getting Started 6 | --------------- 7 | Grab the dependencies:: 8 | 9 | $ apt-get install redis-server rabbitmq-server 10 | 11 | Check out netinfo to `/opt/`:: 12 | 13 | $ cd /opt && git clone https://github.com/9b/netinfo.git 14 | 15 | Change directory to the `netinfo` working directory:: 16 | 17 | $ cd netinfo/ 18 | 19 | Setup the virtualenv:: 20 | 21 | $ virtualenv -p python3 venv3 22 | 23 | Activate the virtualenv:: 24 | 25 | $ source venv3/bin/activate 26 | 27 | Install the requirements:: 28 | 29 | $ (venv3) pip install -r requirements.txt 30 | 31 | Install the services and start them. 32 | 33 | See `Services README`_. 34 | 35 | Or start the Celery Daemon:: 36 | 37 | $ (venv3) /opt/netinfo/venv3/bin/celery worker -A worker.celery --loglevel=info -B 38 | 39 | And start the server:: 40 | 41 | $ (venv3) /opt/netinfo/venv3/bin/uwsgi --ini wsgi.ini 42 | 43 | You can then access the API through http://localhost:7777 for more details. 44 | 45 | .. _Services README: https://github.com/9b/netinfo/blob/master/service/README.rst 46 | 47 | Sample Output 48 | ------------- 49 | When calling http://localhost:7777/lookup?ip=74.96.192.82:: 50 | 51 | { 52 | "as_name": "UUNET - MCI Communications Services, Inc. d/b/a Verizon Business, US", 53 | "as_num": 701, 54 | "city": "Vienna", 55 | "country_iso": "US", 56 | "country_name": "United States", 57 | "ip": "74.96.192.82", 58 | "ip_hex": "0x4a60c052", 59 | "ip_version": 4, 60 | "latitude": 38.8977, 61 | "longitude": -77.288, 62 | "network": "74.96.0.0/16", 63 | "network_broadcast": "74.96.255.255", 64 | "network_hostmask": "0.0.255.255", 65 | "network_netmask": "255.255.0.0", 66 | "network_size": 65536, 67 | "postal_code": "22181", 68 | "region_iso": "VA", 69 | "region_name": "Virginia" 70 | } 71 | 72 | Unlike the standard PyASN library, NetInfo will add the AS name to the response, additional network data and the original IP address that was requested. 73 | 74 | API Endpoints 75 | ------------- 76 | The following endpoints are available within the NetInfo service. 77 | 78 | **/lookup?ip=8.8.8.8** 79 | 80 | Get back AS, network information and geolocation for an IP address. 81 | 82 | **/network-addresses?cidr=8.8.8.0/24** 83 | 84 | Get back all IP addresses as part of a network range. 85 | 86 | **/prefixes?asn=15169** 87 | 88 | Get back all prefixes advertised for a specific AS network. 89 | 90 | **/as-name?asn=15169** 91 | 92 | Get back the name of the AS network. 93 | 94 | **/geolocation?ip=8.8.8.8** 95 | 96 | Get back geolocation information for an IP address. 97 | -------------------------------------------------------------------------------- /app/__init__.py: -------------------------------------------------------------------------------- 1 | """.""" 2 | import geoip2.database 3 | import json 4 | import logging 5 | import os 6 | import pyasn 7 | import socket 8 | import sys 9 | 10 | from celery import Celery 11 | from celery.schedules import crontab 12 | from flask import Flask, redirect, url_for, render_template, request 13 | from flask_pymongo import PyMongo 14 | from functools import wraps 15 | from flask import current_app as app 16 | from app.utils.helpers import now_time, load_time 17 | from werkzeug.contrib.cache import MemcachedCache 18 | 19 | APP_NAME = 'netinfo' 20 | APP_BASE = os.path.dirname(os.path.realpath(__file__)) 21 | REFRESH_TIME = 1800 22 | 23 | mongo = PyMongo() 24 | cache = MemcachedCache(['127.0.0.1:11211']) 25 | celery = Celery(APP_NAME) 26 | logger = logging.getLogger(APP_NAME) 27 | logger.setLevel(logging.DEBUG) 28 | shandler = logging.StreamHandler(sys.stdout) 29 | fmt = '\033[1;32m%(levelname)-5s %(module)s:%(funcName)s():' 30 | fmt += '%(lineno)d %(asctime)s\033[0m| %(message)s' 31 | shandler.setFormatter(logging.Formatter(fmt)) 32 | logger.addHandler(shandler) 33 | 34 | 35 | def server_error(e): 36 | """500 handler.""" 37 | logger.error("500 triggered: %s" % (str(e))) 38 | return "500" 39 | 40 | 41 | def page_not_found(e): 42 | """404 handler.""" 43 | logger.info("404 triggered: Path %s" % (request.path)) 44 | return "404" 45 | 46 | 47 | def check_asndb(f): 48 | """Check if the ASN database should be updated. 49 | 50 | This wraps any call to the API to ensure the version of the database is 51 | always the most current. The PyASN database remains in a global variable 52 | exposed by Flask. Celery will update the configuration file after a new 53 | RIB has been downloaded and processed. That serves as the trigger data in 54 | order to reload the database or not. 55 | """ 56 | @wraps(f) 57 | def decorated_function(*args, **kwargs): 58 | config = json.load(open('%s/resources/config.json' % APP_BASE)) 59 | delta = (now_time() - load_time(config['asn']['last_update'])).seconds 60 | if delta > REFRESH_TIME or not app.config['ASNDB']: 61 | try: 62 | app.config['ASNDB'] = pyasn.pyasn('%s/resources/asn/current' % APP_BASE, 63 | as_names_file='%s/resources/asn/as_names.json' % APP_BASE) 64 | app.config['ASNDB'].loaded = config['asn']['last_rib_file'] 65 | except Exception as e: 66 | raise Exception("Database has not been initialized.") 67 | return f(*args, **kwargs) 68 | return decorated_function 69 | 70 | 71 | def check_geoip(f): 72 | """Check if the GeoIP database should be updated. 73 | """ 74 | @wraps(f) 75 | def decorated_function(*args, **kwargs): 76 | config = json.load(open('%s/resources/config.json' % APP_BASE)) 77 | delta = (now_time() - load_time(config['geoip']['last_update'])).seconds 78 | if delta > REFRESH_TIME or not app.config['GEOIPDB']: 79 | try: 80 | app.config['GEOIPDB'] = geoip2.database.Reader('%s/resources/geoip/current' % APP_BASE) 81 | except Exception as e: 82 | print(e) 83 | raise Exception("Database has not been initialized.") 84 | return f(*args, **kwargs) 85 | return decorated_function 86 | 87 | 88 | def housekeeping(): 89 | """Check if the services we need are running.""" 90 | try: 91 | s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 92 | s.bind(('0.0.0.0', 6379)) 93 | raise Exception("[!] Redis does not appear to be running") 94 | return False 95 | except Exception as e: 96 | pass 97 | 98 | try: 99 | s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 100 | s.bind(('0.0.0.0', 5672)) 101 | raise Exception("[!] RabbitMQ does not appear to be running") 102 | return False 103 | except Exception as e: 104 | pass 105 | 106 | return True 107 | 108 | 109 | def create_app(debug=False): 110 | """Create an application context with blueprints.""" 111 | state = housekeeping() 112 | if not state: 113 | sys.exit(1) 114 | app = Flask(__name__, static_folder='./resources') 115 | app.config['SECRET_KEY'] = 'tRSn3mh2bY3@1$W2T9aQ' 116 | app.config['MONGO_DBNAME'] = 'netinfo' 117 | app.config['MONGO_HOST'] = 'localhost' 118 | app.config['ASNDB'] = None 119 | app.config['GEOIPDB'] = None 120 | app.config['DEBUG'] = debug 121 | muri = "mongodb://%s:27017/%s" % (app.config['MONGO_HOST'], 122 | app.config['MONGO_DBNAME']) 123 | app.config['MONGO_URI'] = muri 124 | mongo.init_app(app) 125 | app.config.update( 126 | CELERY_BROKER_URL='redis://localhost:6379', 127 | CELERY_RESULT_BACKEND='redis://localhost:6379', 128 | CELERYBEAT_SCHEDULE={ 129 | 'fetch-rib': { 130 | 'task': 'fetch-rib', 131 | 'schedule': crontab(minute='*/5') 132 | }, 133 | 'fetch-as-name': { 134 | 'task': 'fetch-as-names', 135 | 'schedule': crontab(hour="*/12") 136 | }, 137 | 'fetch-geo': { 138 | 'task': 'fetch_geoip', 139 | 'schedule': crontab(hour=7, minute=30, day_of_week=1) 140 | } 141 | } 142 | ) 143 | celery.conf.update(app.config) 144 | 145 | config_file = '%s/resources/config.json' % APP_BASE 146 | if not os.path.exists(config_file): 147 | config = {'asn': {'last_rib_file': None, 'last_update': None}, 148 | 'geoip': {'last_update': None}} 149 | json.dump(config, open(config_file, 'w'), indent=4) 150 | 151 | from .core import core as core_blueprint 152 | app.register_blueprint(core_blueprint) 153 | app.register_error_handler(404, page_not_found) 154 | app.register_error_handler(500, server_error) 155 | 156 | return app 157 | -------------------------------------------------------------------------------- /app/core/__init__.py: -------------------------------------------------------------------------------- 1 | """Initialize the blueprint with the global context. 2 | 3 | Despite it appearing useless, this file is needed in order to import all of 4 | the routes, so the parent context understands what we have defined. 5 | """ 6 | from flask import Blueprint 7 | 8 | core = Blueprint('core', __name__) 9 | 10 | from . import ( 11 | api, 12 | generic 13 | ) 14 | -------------------------------------------------------------------------------- /app/core/api.py: -------------------------------------------------------------------------------- 1 | """Generic calls within the application.""" 2 | import os 3 | import pyasn 4 | 5 | from . import core 6 | from .. import mongo, logger, celery, check_asndb, check_geoip, cache 7 | from flask import (jsonify, request) 8 | from flask import current_app as app 9 | from netaddr import IPAddress, IPNetwork 10 | 11 | 12 | @core.route('/lookup', methods=['GET']) 13 | @check_asndb 14 | @check_geoip 15 | def lookup(): 16 | """Enrich IP address.""" 17 | ip_addr = request.args.get('ip') 18 | cached = cache.get(ip_addr) 19 | if cached: 20 | return jsonify(cached) 21 | __ip_addr = IPAddress(ip_addr) 22 | 23 | record = { 24 | 'as_num': None, 25 | 'network': None, 26 | 'as_name': None, 27 | 'ip': ip_addr, 28 | 'ip_version': int(__ip_addr.version), 29 | 'ip_hex': hex(__ip_addr), 30 | 'network_broadcast': None, 31 | 'network_netmask': None, 32 | 'network_hostmask': None, 33 | 'network_size': None} 34 | 35 | data = app.config['ASNDB'].lookup(ip_addr) 36 | if data: 37 | __network = IPNetwork(data[1]) 38 | obj = {'as_num': int(data[0]), 'network': data[1], 39 | 'as_name': str(app.config['ASNDB'].get_as_name(data[0])), 40 | 'network_broadcast': str(__network.broadcast), 41 | 'network_netmask': str(__network.netmask), 42 | 'network_hostmask': str(__network.hostmask), 43 | 'network_size': int(__network.size)} 44 | record.update(obj) 45 | 46 | if app.config['GEOIPDB']: 47 | response = app.config['GEOIPDB'].city(ip_addr) 48 | geo = {'country_name': response.country.name, 49 | 'country_iso': response.country.iso_code, 50 | 'latitude': response.location.latitude, 51 | 'longitude': response.location.longitude, 52 | 'region_name': response.subdivisions.most_specific.name, 53 | 'region_iso': response.subdivisions.most_specific.iso_code, 54 | 'city': response.city.name, 55 | 'postal_code': response.postal.code} 56 | record.update(geo) 57 | if app.config['DEBUG']: 58 | mongo.db.queries.insert(record) 59 | _ = record.pop('_id', None) 60 | cache.set(ip_addr, record, timeout=3600) 61 | return jsonify(record) 62 | 63 | 64 | @core.route('/network-addresses', methods=['GET']) 65 | def network_addresses(): 66 | """Enrich IP address.""" 67 | cidr = request.args.get('cidr') 68 | __network = IPNetwork(cidr) 69 | addresses = [str(x) for x in list(__network)] 70 | record = {'cidr': cidr, 'network_addresses': addresses, 71 | 'network_size': int(__network.size)} 72 | if app.config['DEBUG']: 73 | mongo.db.queries.insert(record) 74 | _ = record.pop('_id', None) 75 | return jsonify(record) 76 | 77 | 78 | @core.route('/as', methods=['GET']) 79 | @check_asndb 80 | def as_enrich(): 81 | """Enrich the AS.""" 82 | asn = request.args.get('asn') 83 | prefixes = list(app.config['ASNDB'].get_as_prefixes(asn)) 84 | record = {'as_num': int(asn), 'prefixes': prefixes, 85 | 'prefix_count': len(prefixes), 86 | 'as_name': str(app.config['ASNDB'].get_as_name(int(asn)))} 87 | if app.config['DEBUG']: 88 | mongo.db.queries.insert(record) 89 | _ = record.pop('_id', None) 90 | return jsonify(record) 91 | 92 | 93 | @core.route('/prefixes', methods=['GET']) 94 | @check_asndb 95 | def prefixes(): 96 | """Enrich IP address.""" 97 | asn = request.args.get('asn') 98 | data = app.config['ASNDB'].get_as_prefixes(asn) 99 | record = {'as_num': int(asn), 'prefixes': list(data), 'count': len(data)} 100 | if app.config['DEBUG']: 101 | mongo.db.queries.insert(record) 102 | _ = record.pop('_id', None) 103 | return jsonify(record) 104 | 105 | 106 | @core.route('/as-name', methods=['GET']) 107 | @check_asndb 108 | def as_name(): 109 | """Enrich IP address.""" 110 | asn = request.args.get('asn') 111 | record = {'as_name': str(app.config['ASNDB'].get_as_name(int(asn))), 112 | 'as_num': asn} 113 | if app.config['DEBUG']: 114 | mongo.db.queries.insert(record) 115 | _ = record.pop('_id', None) 116 | return jsonify(record) 117 | 118 | 119 | @core.route('/geolocation', methods=['GET']) 120 | @check_geoip 121 | def geolocation(): 122 | """Enrich IP address with geolocation.""" 123 | ip_addr = request.args.get('ip') 124 | response = app.config['GEOIPDB'].city(ip_addr) 125 | record = {'country_name': response.country.name, 126 | 'country_iso': response.country.iso_code, 127 | 'latitude': response.location.latitude, 128 | 'longitude': response.location.longitude, 129 | 'region_name': response.subdivisions.most_specific.name, 130 | 'region_iso': response.subdivisions.most_specific.iso_code, 131 | 'city': response.city.name, 132 | 'postal_code': response.postal.code} 133 | if app.config['DEBUG']: 134 | mongo.db.queries.insert(record) 135 | _ = record.pop('_id', None) 136 | return jsonify(record) -------------------------------------------------------------------------------- /app/core/generic.py: -------------------------------------------------------------------------------- 1 | """Generic calls within the application.""" 2 | import json 3 | import os 4 | 5 | from . import core 6 | from .. import mongo, logger, celery 7 | from flask import ( 8 | render_template, redirect, url_for, jsonify, request, Response 9 | ) 10 | from flask import current_app as app 11 | 12 | 13 | APP_BASE = os.path.dirname(os.path.realpath(__file__)).replace('/core', '') 14 | 15 | 16 | @core.route('/') 17 | def root(): 18 | """Render the index page.""" 19 | config = json.load(open('%s/resources/config.json' % APP_BASE)) 20 | return render_template('index.html', config=config) 21 | 22 | 23 | @core.route('/force-geo') 24 | def force_geo(): 25 | """Force a fetching of the latest database.""" 26 | logger.debug("Fetch the latest copy of the database") 27 | celery.send_task('fetch-geoip') 28 | return jsonify({'success': True, 29 | 'message': 'This will take a few minutes to complete'}) 30 | 31 | 32 | @core.route('/force-db') 33 | def force_db(): 34 | """Force a fetching of the latest database.""" 35 | logger.debug("Fetch the latest copy of the database") 36 | celery.send_task('fetch-geoip') 37 | celery.send_task('fetch-as-names') 38 | celery.send_task('fetch-rib', kwargs={'force': True}) 39 | return jsonify({'success': True, 40 | 'message': 'This will take a few minutes to complete'}) 41 | -------------------------------------------------------------------------------- /app/resources/asn/ribs/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/9b/netinfo/6fcab55eaeabfd005a9192ace0c2b04122ab3df4/app/resources/asn/ribs/__init__.py -------------------------------------------------------------------------------- /app/resources/config.json: -------------------------------------------------------------------------------- 1 | { 2 | "asn": { 3 | "last_rib_file": "rib.20181228.0000.bz2", 4 | "last_update": "2018-12-27 19:11:26" 5 | }, 6 | "geoip": { 7 | "last_update": "2018-12-26 16:05:29" 8 | } 9 | } -------------------------------------------------------------------------------- /app/resources/geoip/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/9b/netinfo/6fcab55eaeabfd005a9192ace0c2b04122ab3df4/app/resources/geoip/__init__.py -------------------------------------------------------------------------------- /app/tasks/__init__.py: -------------------------------------------------------------------------------- 1 | """Tasks related to asynchronous processing. 2 | 3 | For all intents and purposes, this is where the magic of the database syncing 4 | takes place. There's some extra functions in here that should really move over 5 | to the utils file, but it was easier to keep the logic in one spot during 6 | testing. 7 | 8 | Each of the celery decorated functions are tasks that can be called directly 9 | from the Flask web application aka the API or via some scheduler. These tasks 10 | will all run as a non-blocking call if made through the API or scheduler; they 11 | just run and log out to the Celery handler. 12 | 13 | If you're using the service wrappers, this is netinfod. And if it wasn't clear, 14 | this needs to be running in order for any real processing to take place. 15 | """ 16 | from .. import mongo, logger 17 | import celery 18 | import datetime 19 | import json 20 | import os 21 | import re 22 | import requests 23 | import tarfile 24 | import shutil 25 | from pyasn import mrtx 26 | import codecs 27 | from urllib.request import urlopen 28 | 29 | from ..utils.helpers import str_now_time 30 | 31 | 32 | APP_BASE = os.path.dirname(os.path.realpath(__file__)).replace('/tasks', '') 33 | ASNAMES_URL = 'http://www.cidr-report.org/as2.0/autnums.html' 34 | HTML_FILENAME = "autnums.html" 35 | EXTRACT_ASNAME_C = re.compile(r"AS(?P.+?)\s*\s*(?P.*)", re.U) 36 | 37 | 38 | def __parse_asname_line(line): 39 | match = EXTRACT_ASNAME_C.match(line) 40 | return match.groups() 41 | 42 | 43 | def _html_to_dict(data): 44 | """Translates an HTML string available at `ASNAMES_URL` into a dict.""" 45 | split = data.split("\n") 46 | split = filter(lambda line: line.startswith(" 2 | 3 |

Status

4 | {% if not config['file'] %} 5 |

Not configured. Click here to initialize the database.

6 | {% else %} 7 |

Ready to rock.

8 |

IP Enrichment: /lookup?ip=8.8.8.8

9 |

Network Addresses: /network-addresses?cidr=8.8.8.0/24

10 |

ASN Prefixes: /prefixes?asn=15169

11 |

ASN Name: /as-name?asn=15169

12 |

Geolocation: /geolocation?ip=8.8.8.8

13 | {% endif %} 14 | 15 |

Configuration

16 |

ASN Last Updated: {{config['asn']['last_update']}}

17 |

ASN Last File: {{config['asn']['last_rib_file']}}

18 |

GEOIP Last Updated: {{config['geoip']['last_update']}}

19 | 20 | -------------------------------------------------------------------------------- /app/utils/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/9b/netinfo/6fcab55eaeabfd005a9192ace0c2b04122ab3df4/app/utils/__init__.py -------------------------------------------------------------------------------- /app/utils/helpers.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | 3 | 4 | def str_now_time(): 5 | """Get current time as a string.""" 6 | return datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") 7 | 8 | 9 | def now_time(): 10 | """Get current time.""" 11 | return datetime.datetime.now() 12 | 13 | 14 | def load_time(str_time): 15 | """Load a string date as a normal datetime.""" 16 | return datetime.datetime.strptime(str_time, "%Y-%m-%d %H:%M:%S") 17 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | amqp==2.3.2 2 | billiard==3.5.0.5 3 | celery==4.2.1 4 | certifi==2018.11.29 5 | chardet==3.0.4 6 | Click==7.0 7 | Flask==1.0.2 8 | Flask-PyMongo==2.2.0 9 | geoip2==2.9.0 10 | idna==2.8 11 | itsdangerous==1.1.0 12 | Jinja2==2.10 13 | kombu==4.2.2.post1 14 | MarkupSafe==1.1.0 15 | maxminddb==1.4.1 16 | netaddr==0.7.19 17 | pyasn==1.6.0b1 18 | pymongo==3.7.2 19 | python-memcached==1.59 20 | pytz==2018.7 21 | redis==3.0.1 22 | requests==2.21.0 23 | six==1.12.0 24 | urllib3==1.24.1 25 | uWSGI==2.0.17.1 26 | vine==1.1.4 27 | Werkzeug==0.14.1 -------------------------------------------------------------------------------- /scripts/fetch.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | import requests 3 | 4 | from pyasn import mrtx, __version__ 5 | 6 | if __name__ == '__main__': 7 | base = "http://archive.routeviews.org//bgpdata/" 8 | now = datetime.datetime.utcnow() 9 | slug = now.strftime('%Y.%m') 10 | fname = now.strftime('rib.%Y%m%d.%H00.bz2') 11 | hour = int(now.strftime('%H')) 12 | if not hour % 2 == 0: 13 | fname = now.strftime('rib.%Y%m%d.') 14 | fname = fname + str(hour - 1) + '00.bz2' 15 | url = "%s/%s/RIBS/%s" % (base, slug, fname) 16 | 17 | response = requests.get(url) 18 | path = 'app/resources/ribs/%s' % (fname) 19 | open(path, 'wb').write(response.content) 20 | 21 | current = 'app/resources/current' 22 | prefixes = mrtx.parse_mrt_file(path, 23 | print_progress=False, 24 | skip_record_on_error=True) 25 | mrtx.dump_prefixes_to_file(prefixes, current, path) 26 | -------------------------------------------------------------------------------- /scripts/fetch_geo.py: -------------------------------------------------------------------------------- 1 | import tarfile 2 | import requests 3 | import shutil 4 | 5 | from pyasn import mrtx, __version__ 6 | 7 | APP_BASE = "/tmp" 8 | 9 | if __name__ == '__main__': 10 | url = "https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz" 11 | response = requests.get(url) 12 | path = '%s/GeoLite2-City.tar.gz' % (APP_BASE) 13 | open(path, 'wb').write(response.content) 14 | tar = tarfile.open(path) 15 | files = tar.getmembers() 16 | tar.extractall(path='/tmp/') 17 | tar.close() 18 | 19 | for file in files: 20 | if not file.name.endswith('.mmdb'): 21 | continue 22 | shutil.move('/tmp/%s' % file.name, '/tmp/GeoLite2-City.mmdb') -------------------------------------------------------------------------------- /server.py: -------------------------------------------------------------------------------- 1 | """Run the server and begin hosting.""" 2 | from app import create_app 3 | from argparse import ArgumentParser 4 | 5 | parser = ArgumentParser() 6 | subs = parser.add_subparsers(dest='cmd') 7 | setup_parser = subs.add_parser('run') 8 | setup_parser.add_argument('--debug', action='store_true', 9 | help='Run in debug mode.') 10 | args = parser.parse_args() 11 | kwargs = {'debug': args.debug} 12 | app = create_app(**kwargs) 13 | 14 | if __name__ == '__main__': 15 | app.run(host="localhost", port=7777) 16 | -------------------------------------------------------------------------------- /service/README.rst: -------------------------------------------------------------------------------- 1 | Services 2 | ======== 3 | Running NetInfo as a service is the most optimal setup as it ensures a process is not killed. There are two key services, an API and a celery worker to run asynchronous tasks. 4 | 5 | Setup 6 | ----- 7 | Copy the files to `/etc/systemd/system/`:: 8 | 9 | $ cp -r service/*.service /etc/systemd/system/. 10 | 11 | Start each service:: 12 | 13 | $ sudo systemctl start netinfo && sudo systemctl start netinfod 14 | 15 | Enable the services:: 16 | 17 | $ sudo systemctl enable netinfo && sudo systemctl enable netinfod 18 | 19 | Check the status of each service to ensure it's running:: 20 | 21 | $ service status 22 | -------------------------------------------------------------------------------- /service/netinfo.service: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=NetInfo webserver and API 3 | After=network.target 4 | 5 | [Service] 6 | User=root 7 | Group=root 8 | WorkingDirectory=/opt/netinfo 9 | Environment="PATH=/opt/netinfo/venv3/bin" 10 | ExecStart=/opt/netinfo/venv3/bin/uwsgi --ini wsgi.ini 11 | KillSignal=SIGQUIT 12 | SyslogIdentifier=netinfo 13 | StandardOutput=syslog 14 | StandardError=syslog 15 | RemainAfterExit=yes 16 | Restart=always 17 | RestartSec=3 18 | 19 | [Install] 20 | WantedBy=multi-user.target -------------------------------------------------------------------------------- /service/netinfod.service: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=NetInfo celery daemon 3 | After=network.target 4 | 5 | [Service] 6 | User=root 7 | Group=root 8 | WorkingDirectory=/opt/netinfo 9 | Environment="PATH=/opt/netinfo/venv3/bin" 10 | ExecStart=/opt/netinfo/venv3/bin/celery worker -A worker.celery --loglevel=info -B -n netinfod 11 | KillSignal=SIGQUIT 12 | SyslogIdentifier=netinfod 13 | StandardOutput=syslog 14 | StandardError=syslog 15 | RemainAfterExit=yes 16 | Restart=always 17 | RestartSec=3 18 | 19 | [Install] 20 | WantedBy=multi-user.target -------------------------------------------------------------------------------- /worker.py: -------------------------------------------------------------------------------- 1 | """Run the Celery jobs.""" 2 | import os 3 | from app import celery, create_app 4 | import app.tasks 5 | 6 | flask_app = create_app(os.getenv('FLASK_CONFIG') or 'default') 7 | flask_app.app_context().push() 8 | -------------------------------------------------------------------------------- /wsgi.ini: -------------------------------------------------------------------------------- 1 | [uwsgi] 2 | plugins = python3 3 | virtualenv = ./venv3 4 | 5 | module = app:create_app() 6 | 7 | master = true 8 | processes = 5 9 | enable-threads = true 10 | disable-logging = true 11 | 12 | socket = netinfo.sock 13 | chmod-socket = 666 14 | vacuum = true 15 | 16 | die-on-term = true --------------------------------------------------------------------------------