├── .gitignore ├── MANIFEST.in ├── Makefile ├── README.md ├── diagramly-r3.xml ├── r3-web-1.jpg ├── r3-web-2.jpg ├── r3-web-3.jpg ├── r3-web-4.jpg ├── r3.png ├── r3 ├── __init__.py ├── app │ ├── __init__.py │ ├── app.py │ ├── config.py │ ├── handlers │ │ ├── __init__.py │ │ ├── healthcheck.py │ │ ├── index.py │ │ └── stream.py │ ├── keys.py │ ├── server.py │ ├── templates │ │ └── index.html │ └── utils.py ├── version.py ├── web │ ├── __init__.py │ ├── app.py │ ├── config.py │ ├── extensions.py │ ├── server.py │ ├── static │ │ ├── css │ │ │ ├── bootstrap.min.css │ │ │ ├── progress.css │ │ │ ├── reset.css │ │ │ └── style.css │ │ ├── img │ │ │ └── logo.png │ │ └── js │ │ │ ├── bootstrap.min.js │ │ │ ├── jquery-1.7.2.min.js │ │ │ ├── progress.js │ │ │ └── tabs.js │ └── templates │ │ ├── failed.html │ │ ├── index.html │ │ ├── job-types.html │ │ ├── mappers.html │ │ ├── master.html │ │ ├── show_key.html │ │ └── stats.html └── worker │ ├── __init__.py │ └── mapper.py ├── redis.conf ├── requirements.txt ├── setup.py └── test ├── __init__.py ├── app_config.py ├── chekhov.txt ├── count_words_mapper.py ├── count_words_reducer.py ├── count_words_stream.py ├── small-chekhov.txt ├── test_count_words.py └── test_sync.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.py[co] 2 | 3 | # Packages 4 | *.egg 5 | *.egg-info 6 | dist 7 | build 8 | eggs 9 | parts 10 | bin 11 | var 12 | sdist 13 | develop-eggs 14 | .installed.cfg 15 | 16 | # Installer logs 17 | pip-log.txt 18 | 19 | # Unit test / coverage reports 20 | .coverage 21 | .tox 22 | 23 | #Translations 24 | *.mo 25 | 26 | #Mr Developer 27 | .mr.developer.cfg 28 | .DS_Store 29 | *.geany 30 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | prune dist 2 | prune build 3 | prune test 4 | recursive-include r3 *.py 5 | recursive-include r3 *.gif 6 | recursive-include r3 *.png 7 | recursive-include r3 *.jpg 8 | recursive-include r3 *.jpeg 9 | recursive-include r3 *.html 10 | recursive-include r3 *.htm 11 | recursive-include r3 *.js 12 | recursive-include r3 *.css 13 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # %%%%%%%%%%%%%% SERVICE %%%%%%%%%%%%%% 2 | run: 3 | @PYTHONPATH=$$PYTHONPATH:.:./test python r3/app/server.py --redis-port=7778 --redis-pass=r3 --config-file="./test/app_config.py" --debug 4 | 5 | 6 | # %%%%%%%%%%%%%% WORKER %%%%%%%%%%%%%% 7 | worker: 8 | @PYTHONPATH=$$PYTHONPATH:. python r3/worker/mapper.py --mapper-key="${KEY}" --mapper-class="test.count_words_mapper.CountWordsMapper" --redis-port=7778 --redis-pass=r3 9 | 10 | 11 | # %%%%%%%%%%%%%% WEB %%%%%%%%%%%%%% 12 | web: 13 | @PYTHONPATH=$$PYTHONPATH:.:./test python r3/web/server.py --redis-port=7778 --redis-pass=r3 --config-file=./r3/web/config.py --debug 14 | 15 | 16 | # %%%%%%%%%%%%%% REDIS %%%%%%%%%%%%%% 17 | kill_redis: 18 | @ps aux | awk '(/redis-server/ && $$0 !~ /awk/){ system("kill -9 "$$2) }' 19 | 20 | redis: kill_redis 21 | @mkdir -p /tmp/r3/db 22 | @redis-server redis.conf & 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | r³ 2 | == 3 | 4 | r³ is a map reduce engine written in python using a redis backend. It's purpose 5 | is to be simple. 6 | 7 | r³ has only three concepts to grasp: input streams, mappers and reducers. 8 | 9 | The diagram below relates how they interact: 10 | 11 | ![r³ components interaction](https://github.com/heynemann/r3/raw/master/r3.png) 12 | 13 | If the diagram above is a little too much to grasp right now, don't worry. Keep 14 | reading and use this diagram later for reference. 15 | 16 | A fairly simple map-reduce example to solve is counting the number of 17 | occurrences of each word in an extensive document. We'll use this scenario as 18 | our example. 19 | 20 | Installing 21 | ---------- 22 | 23 | Installing r³ is as easy as: 24 | 25 | pip install r3 26 | 27 | After successful installation, you'll have three new commands: `r3-app`, 28 | `r3-map` and `r3-web`. 29 | 30 | Running the App 31 | --------------- 32 | 33 | In order to use r³ you must have a redis database running. Getting one up in 34 | your system is beyond the scope of this document. 35 | 36 | We'll assume you have one running at 127.0.0.1, port 7778 and configured to 37 | require the password 'r3' using database 0. 38 | 39 | The service that is at the heart of r³ is `r3-app`. It is the web-server that 40 | will receive requests for map-reduce jobs and return the results. 41 | 42 | To run `r3-app`, given the above redis back-end, type: 43 | 44 | r3-app --redis-port=7778 --redis-pass=r3 -c config.py 45 | 46 | We'll learn more about the configuration file below. 47 | 48 | Given that you have a proper configuration file, your r3 service will be 49 | available at `http://localhost:9999`. 50 | 51 | As to how we actually perform a map-reduce operation, we'll see that after the 52 | `Running Mappers` section. 53 | 54 | App Configuration 55 | ----------------- 56 | 57 | In the above section we specified a file called `config.py` as configuration. 58 | Now we'll see what that file contains. 59 | 60 | The configuration file that we pass to the `r3-app` command is responsible for 61 | specifying `input stream processors` and `reducers` that should be enabled. 62 | 63 | Let's see a sample configuration file: 64 | 65 | INPUT_STREAMS = [ 66 | 'test.count_words_stream.CountWordsStream' 67 | ] 68 | 69 | REDUCERS = [ 70 | 'test.count_words_reducer.CountWordsReducer' 71 | ] 72 | 73 | This configuration specifies that there should be a `CountWordsStream` input 74 | stream processor and a `CountWordsReducer` reducer. Both will be used by the 75 | `stream` service to perform a map-reduce operation. 76 | 77 | We'll learn more about `input streams` and `reducers` in the sections below. 78 | 79 | The input stream 80 | ---------------- 81 | 82 | The input stream processor is the class responsible for creating the input 83 | streams upon which the mapping will occur. 84 | 85 | In our counting words in a document sample, the input stream processor class 86 | should open the document, read the lines in the document and then return each 87 | line to `r3-app`. 88 | 89 | Let's see a possible implementation: 90 | 91 | from os.path import abspath, dirname, join 92 | 93 | class CountWordsStream: 94 | job_type = 'count-words' 95 | group_size = 1000 96 | 97 | def process(self, app, arguments): 98 | with open(abspath(join(dirname(__file__), 'chekhov.txt'))) as f: 99 | contents = f.readlines() 100 | 101 | return [line.lower() for line in contents] 102 | 103 | The `job_type` property is required and specifies the relationship that this 104 | input stream has with mappers and with a specific reducer. 105 | 106 | The `group_size` property specifies how big is an input stream. In the above 107 | example, our input stream processor returns all the lines in the document, but 108 | r³ will group the resulting lines in batches of 1000 lines to be processed by 109 | each mapper. How big is your group size varies wildly depending on what your 110 | mapping consists of. 111 | 112 | Running Mappers 113 | --------------- 114 | 115 | `Input stream processors` and `reducers` are sequential and thus run in-process 116 | in the r³ app. Mappers, on the other hand, are inherently parallel and are run 117 | on their own as independent worker units. 118 | 119 | Considering the above example of input stream and reducer, we'll use a 120 | `CountWordsMapper` class to run our mapper. 121 | 122 | We can easily start the mapper with: 123 | 124 | r3-map --redis-port=7778 --redis-pass=r3 --mapper-key=mapper-1 --mapper-class="test.count_words_mapper.CountWordsMapper" 125 | 126 | The `redis-port` and `redis-pass` arguments require no further explanation. 127 | 128 | The `mapper-key` argument specifies a unique key for this mapper. This key 129 | should be the same once this mapper restarts. 130 | 131 | The `mapper-class` is the class r³ will use to map input streams. 132 | 133 | Let's see what this map class looks like. If we are mapping lines (what we got 134 | out of the input stream steap), we should return each word and how many times 135 | it occurs. 136 | 137 | from r3.worker.mapper import Mapper 138 | 139 | class CountWordsMapper(Mapper): 140 | job_type = 'count-words' 141 | 142 | def map(self, lines): 143 | return list(self.split_words(lines)) 144 | 145 | def split_words(self, lines): 146 | for line in lines: 147 | for word in line.split(): 148 | yield word, 1 149 | 150 | The `job_type` property is required and specifies the relationship that this 151 | mapper has with a specific input stream and with a specific reducer. 152 | 153 | Reducing 154 | -------- 155 | 156 | After all input streams have been mapped, it is time to reduce our data to one 157 | coherent value. This is what the reducer does. 158 | 159 | In the case of counting word occurrences, a sample implementation is as 160 | follows: 161 | 162 | from collections import defaultdict 163 | 164 | class CountWordsReducer: 165 | job_type = 'count-words' 166 | 167 | def reduce(self, app, items): 168 | word_freq = defaultdict(int) 169 | for line in items: 170 | for word, frequency in line: 171 | word_freq[word] += frequency 172 | 173 | return word_freq 174 | 175 | The `job_type` property is required and specifies the relationship that this 176 | reducer has with mappers and with a specific input stream. 177 | 178 | This reducer will return a dictionary that contains all the words and the 179 | frequency with which they occur in the given file. 180 | 181 | Testing our Solution 182 | ------------------- 183 | 184 | To test the above solution, just clone r³'s repository and run the commands 185 | from the directory you just cloned. 186 | 187 | Given that we have the above working, we should have `r3-app` running at 188 | `http://localhost:9999`. In order to access our `count-words` job we'll point 189 | our browser to: 190 | 191 | http://localhost:9999/count-words 192 | 193 | This should return a JSON document with the resulting occurrences of words in 194 | the sample document. 195 | 196 | Creating my own Reducers 197 | ------------------------ 198 | 199 | As you have probably guessed, creating new jobs of mapping and reducing is as 200 | simple as implementing your own `input stream processor`, `mapper` and 201 | `reducer`. 202 | 203 | After they are implemented, just include the processor and reducer in the 204 | config file and fire up as many mappers as you want. 205 | 206 | Monitoring r³ 207 | ------------- 208 | 209 | We talked about three available commands: `r3-app`, `r3-map` and `r3-web`. 210 | 211 | The last one fires up a monitoring interface that helps you in understanding 212 | how your r³ farm is working. 213 | 214 | Some screenshots of the monitoring application: 215 | 216 | ![r³ web monitoring interface](https://github.com/heynemann/r3/raw/master/r3-web-4.jpg) 217 | 218 | Failed jobs monitoring: 219 | 220 | ![r³ web monitoring interface](https://github.com/heynemann/r3/raw/master/r3-web-2.jpg) 221 | 222 | Stats: 223 | 224 | ![r³ web monitoring interface](https://github.com/heynemann/r3/raw/master/r3-web-3.jpg) 225 | 226 | License 227 | ------- 228 | 229 | r³ is licensed under the MIT License: 230 | 231 | > The MIT License 232 | 233 | > Copyright (c) 2012 Bernardo Heynemann 234 | 235 | > Permission is hereby granted, free of charge, to any person obtaining a copy 236 | > of this software and associated documentation files (the "Software"), to deal 237 | > in the Software without restriction, including without limitation the rights 238 | > to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 239 | > copies of the Software, and to permit persons to whom the Software is 240 | > furnished to do so, subject to the following conditions: 241 | 242 | > The above copyright notice and this permission notice shall be included in 243 | > all copies or substantial portions of the Software. 244 | 245 | > THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 246 | > IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 247 | > FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 248 | > AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 249 | > LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 250 | > OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 251 | > THE SOFTWARE. -------------------------------------------------------------------------------- /diagramly-r3.xml: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /r3-web-1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/heynemann/r3/cc6b4eb55c7ae30a8f75af2be165504565dbeb79/r3-web-1.jpg -------------------------------------------------------------------------------- /r3-web-2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/heynemann/r3/cc6b4eb55c7ae30a8f75af2be165504565dbeb79/r3-web-2.jpg -------------------------------------------------------------------------------- /r3-web-3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/heynemann/r3/cc6b4eb55c7ae30a8f75af2be165504565dbeb79/r3-web-3.jpg -------------------------------------------------------------------------------- /r3-web-4.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/heynemann/r3/cc6b4eb55c7ae30a8f75af2be165504565dbeb79/r3-web-4.jpg -------------------------------------------------------------------------------- /r3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/heynemann/r3/cc6b4eb55c7ae30a8f75af2be165504565dbeb79/r3.png -------------------------------------------------------------------------------- /r3/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/heynemann/r3/cc6b4eb55c7ae30a8f75af2be165504565dbeb79/r3/__init__.py -------------------------------------------------------------------------------- /r3/app/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/heynemann/r3/cc6b4eb55c7ae30a8f75af2be165504565dbeb79/r3/app/__init__.py -------------------------------------------------------------------------------- /r3/app/app.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | import tornado.web 5 | import tornado.ioloop 6 | 7 | from r3.app.handlers.healthcheck import HealthcheckHandler 8 | from r3.app.handlers.stream import StreamHandler 9 | from r3.app.handlers.index import IndexHandler 10 | from r3.app.utils import kls_import 11 | 12 | class R3ServiceApp(tornado.web.Application): 13 | 14 | def __init__(self, redis, config, log_level, debug, show_index_page): 15 | self.redis = redis 16 | self.log_level = log_level 17 | self.config = config 18 | self.debug = debug 19 | 20 | handlers = [ 21 | (r'/healthcheck', HealthcheckHandler), 22 | ] 23 | 24 | if show_index_page: 25 | handlers.append( 26 | (r'/', IndexHandler) 27 | ) 28 | 29 | handlers.append( 30 | (r'/stream/(?P.+)/?', StreamHandler), 31 | ) 32 | 33 | self.redis.delete('r3::mappers') 34 | 35 | self.load_input_streams() 36 | self.load_reducers() 37 | 38 | super(R3ServiceApp, self).__init__(handlers, debug=debug) 39 | 40 | def load_input_streams(self): 41 | self.input_streams = {} 42 | 43 | if hasattr(self.config, 'INPUT_STREAMS'): 44 | for stream_class in self.config.INPUT_STREAMS: 45 | stream = kls_import(stream_class) 46 | self.input_streams[stream.job_type] = stream() 47 | 48 | def load_reducers(self): 49 | self.reducers = {} 50 | 51 | if hasattr(self.config, 'REDUCERS'): 52 | for reducer_class in self.config.REDUCERS: 53 | reducer = kls_import(reducer_class) 54 | self.reducers[reducer.job_type] = reducer() 55 | 56 | 57 | -------------------------------------------------------------------------------- /r3/app/config.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | from os.path import isabs, abspath 5 | import imp 6 | 7 | class Config: 8 | def __init__(self, path): 9 | if not isabs(path): 10 | self.path = abspath(path) 11 | else: 12 | self.path = path 13 | 14 | self.load() 15 | 16 | def load(self): 17 | with open(self.path) as config_file: 18 | name = 'configuration' 19 | code = config_file.read() 20 | module = imp.new_module(name) 21 | exec code in module.__dict__ 22 | 23 | for name, value in module.__dict__.iteritems(): 24 | setattr(self, name, value) 25 | 26 | 27 | -------------------------------------------------------------------------------- /r3/app/handlers/__init__.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | import tornado.web 5 | 6 | from r3.app.utils import logger 7 | 8 | class BaseHandler(tornado.web.RequestHandler): 9 | def _error(self, status, msg=None): 10 | self.set_status(status) 11 | if msg is not None: 12 | logger.error(msg) 13 | self.finish() 14 | 15 | @property 16 | def redis(self): 17 | return self.application.redis 18 | 19 | 20 | -------------------------------------------------------------------------------- /r3/app/handlers/healthcheck.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | from r3.app.handlers import BaseHandler 5 | 6 | class HealthcheckHandler(BaseHandler): 7 | def get(self): 8 | self.write('WORKING') 9 | 10 | -------------------------------------------------------------------------------- /r3/app/handlers/index.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | from r3.app.handlers import BaseHandler 5 | from r3.app.keys import MAPPERS_KEY 6 | from r3.version import __version__ 7 | 8 | class IndexHandler(BaseHandler): 9 | def get(self): 10 | has_reducers = len(self.application.reducers.keys()) > 0 11 | 12 | self.render( 13 | "../templates/index.html", 14 | title="", 15 | r3_version=__version__, 16 | input_streams=self.application.input_streams.keys(), 17 | has_reducers=has_reducers, 18 | mappers=self.get_mappers() 19 | ) 20 | 21 | def get_mappers(self): 22 | return self.redis.smembers(MAPPERS_KEY) 23 | 24 | 25 | -------------------------------------------------------------------------------- /r3/app/handlers/stream.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | import time 5 | import logging 6 | from uuid import uuid4 7 | from ujson import dumps, loads 8 | from datetime import datetime 9 | 10 | import tornado.web 11 | import tornado.gen 12 | 13 | from r3.app.handlers import BaseHandler 14 | from r3.app.utils import DATETIME_FORMAT 15 | from r3.app.keys import PROCESSED, PROCESSED_FAILED, PROCESSED_SUCCESS, JOB_TYPE_KEY, MAPPER_INPUT_KEY, MAPPER_OUTPUT_KEY, MAPPER_ERROR_KEY 16 | 17 | class StreamHandler(BaseHandler): 18 | def group_items(self, stream_items, group_size): 19 | items = [] 20 | current_item = [] 21 | items.append(current_item) 22 | for stream_item in stream_items: 23 | if len(current_item) == group_size: 24 | current_item = [] 25 | items.append(current_item) 26 | current_item.append(stream_item) 27 | return items 28 | 29 | @tornado.web.asynchronous 30 | def get(self, job_key): 31 | arguments = self.request.arguments 32 | job_id = uuid4() 33 | job_date = datetime.now() 34 | 35 | job_type_input_queue = JOB_TYPE_KEY % job_key 36 | self.redis.sadd(job_type_input_queue, str(job_id)) 37 | 38 | try: 39 | start = time.time() 40 | input_stream = self.application.input_streams[job_key] 41 | items = input_stream.process(self.application, arguments) 42 | if hasattr(input_stream, 'group_size'): 43 | items = self.group_items(items, input_stream.group_size) 44 | 45 | mapper_input_queue = MAPPER_INPUT_KEY % job_key 46 | mapper_output_queue = MAPPER_OUTPUT_KEY % (job_key, job_id) 47 | mapper_error_queue = MAPPER_ERROR_KEY % job_key 48 | 49 | with self.redis.pipeline() as pipe: 50 | start = time.time() 51 | 52 | for item in items: 53 | msg = { 54 | 'output_queue': mapper_output_queue, 55 | 'job_id': str(job_id), 56 | 'job_key': job_key, 57 | 'item': item, 58 | 'date': job_date.strftime(DATETIME_FORMAT), 59 | 'retries': 0 60 | } 61 | pipe.rpush(mapper_input_queue, dumps(msg)) 62 | pipe.execute() 63 | logging.debug("input queue took %.2f" % (time.time() - start)) 64 | 65 | start = time.time() 66 | results = [] 67 | errored = False 68 | while (len(results) < len(items)): 69 | key, item = self.redis.blpop(mapper_output_queue) 70 | json_item = loads(item) 71 | if 'error' in json_item: 72 | json_item['retries'] -= 1 73 | self.redis.hset(mapper_error_queue, json_item['job_id'], dumps(json_item)) 74 | errored = True 75 | break 76 | results.append(loads(json_item['result'])) 77 | 78 | self.redis.delete(mapper_output_queue) 79 | logging.debug("map took %.2f" % (time.time() - start)) 80 | 81 | if errored: 82 | self.redis.incr(PROCESSED) 83 | self.redis.incr(PROCESSED_FAILED) 84 | self._error(500, 'Mapping failed. Check the error queue.') 85 | else: 86 | start = time.time() 87 | reducer = self.application.reducers[job_key] 88 | result = reducer.reduce(self.application, results) 89 | logging.debug("reduce took %.2f" % (time.time() - start)) 90 | 91 | self.set_header('Content-Type', 'application/json') 92 | 93 | self.write(dumps(result)) 94 | 95 | self.redis.incr(PROCESSED) 96 | self.redis.incr(PROCESSED_SUCCESS) 97 | 98 | self.finish() 99 | finally: 100 | self.redis.srem(job_type_input_queue, str(job_id)) 101 | 102 | -------------------------------------------------------------------------------- /r3/app/keys.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | ALL_KEYS = 'r3::*' 5 | 6 | # MAPPER KEYS 7 | MAPPERS_KEY = 'r3::mappers' 8 | MAPPER_INPUT_KEY = 'r3::jobs::%s::input' 9 | MAPPER_OUTPUT_KEY = 'r3::jobs::%s::%s::output' 10 | MAPPER_ERROR_KEY = 'r3::jobs::%s::errors' 11 | MAPPER_WORKING_KEY = 'r3::jobs::%s::working' 12 | LAST_PING_KEY = 'r3::mappers::%s::last-ping' 13 | 14 | # JOB TYPES KEYS 15 | JOB_TYPES_KEY = 'r3::job-types' 16 | JOB_TYPES_ERRORS_KEY = 'r3::jobs::*::errors' 17 | JOB_TYPE_KEY = 'r3::job-types::%s' 18 | 19 | # STATS KEYS 20 | PROCESSED = 'r3::stats::processed' 21 | PROCESSED_SUCCESS = 'r3::stats::processed::success' 22 | PROCESSED_FAILED = 'r3::stats::processed::fail' 23 | 24 | -------------------------------------------------------------------------------- /r3/app/server.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | import sys 5 | import logging 6 | import argparse 7 | 8 | import tornado.ioloop 9 | from tornado.httpserver import HTTPServer 10 | import redis 11 | 12 | from r3.app.app import R3ServiceApp 13 | from r3.app.config import Config 14 | 15 | 16 | def main(arguments=None): 17 | '''Runs r³ server with the specified arguments.''' 18 | 19 | parser = argparse.ArgumentParser(description='runs the application that processes stream requests for r³') 20 | parser.add_argument('-b', '--bind', type=str, default='0.0.0.0', help='the ip that r³ will bind to') 21 | parser.add_argument('-p', '--port', type=int, default=9999, help='the port that r³ will bind to') 22 | parser.add_argument('-l', '--loglevel', type=str, default='warning', help='the log level that r³ will run under') 23 | parser.add_argument('-i', '--hide-index-page', action='store_true', default=False, help='indicates whether r³ app should show the help page') 24 | parser.add_argument('-d', '--debug', action='store_true', default=False, help='indicates whether r³ app should run in debug mode') 25 | parser.add_argument('--redis-host', type=str, default='0.0.0.0', help='the ip that r³ will use to connect to redis') 26 | parser.add_argument('--redis-port', type=int, default=6379, help='the port that r³ will use to connect to redis') 27 | parser.add_argument('--redis-db', type=int, default=0, help='the database that r³ will use to connect to redis') 28 | parser.add_argument('--redis-pass', type=str, default='', help='the password that r³ will use to connect to redis') 29 | parser.add_argument('-c', '--config-file', type=str, help='the config file that r³ will use to load input stream classes and reducers', required=True) 30 | 31 | args = parser.parse_args(arguments) 32 | 33 | cfg = Config(args.config_file) 34 | 35 | c = redis.StrictRedis(host=args.redis_host, port=args.redis_port, db=args.redis_db, password=args.redis_pass) 36 | 37 | logging.basicConfig(level=getattr(logging, args.loglevel.upper())) 38 | 39 | application = R3ServiceApp(redis=c, config=cfg, log_level=args.loglevel.upper(), debug=args.debug, show_index_page=not args.hide_index_page) 40 | 41 | server = HTTPServer(application) 42 | server.bind(args.port, args.bind) 43 | server.start(1) 44 | 45 | try: 46 | logging.debug('r³ service app running at %s:%d' % (args.bind, args.port)) 47 | tornado.ioloop.IOLoop.instance().start() 48 | except KeyboardInterrupt: 49 | print 50 | print "-- r³ service app closed by user interruption --" 51 | 52 | if __name__ == "__main__": 53 | main(sys.argv[1:]) 54 | -------------------------------------------------------------------------------- /r3/app/templates/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | r³ - map reduce service 5 | 169 | 170 | 171 |
172 |
173 |

r³ - map reduce service

 174 |
175 |
176 | 177 |
178 |
179 |
180 |

Welcome to r³

181 | 182 | {% if not input_streams %} 183 |
184 |

It appears you didn't setup any input streams!

185 |

Input Streams are classes that generate units-of-work that your mappers will work on.

186 |

Creating them is as simple as creating a class that has a process method and a job_type argument:

187 |
188 |

After you create your input stream, just add it to a config.py file:

189 |
190 |

Then pass the file path as an argument to r3-app like this:

191 |
192 |

For more information check the documentation online.

193 |
194 | {% end %} 195 | 196 | {% if not mappers %} 197 |
198 |

It appears you didn't setup any mappers!

199 |

Setting mappers to run your map/reduce tasks is an integral part of r³ and is as simple as creating a class that inherits from Mapper:

200 |
201 |

Running the mappers is pretty simple as well. Say you want to run four different mappers:

202 |
203 |

For more information check the documentation online.

204 |
205 | {% end %} 206 | 207 | {% if not has_reducers %} 208 |
209 |

It appears you didn't setup any reducers!

210 |

Reducers are the classes that get the mapped units-of-work generated by your mappers and process them into a single coherent result.

211 |

Creating them is as simple as creating a class that has a reduce method and a job_type argument:

212 |
213 |

After you create your input stream, just add it to a config.py file:

214 |
215 |

Then pass the file path as an argument to r3-app like this:

216 |
217 |

For more information check the documentation online.

218 |
219 | {% end %} 220 | 221 | {% if mappers %} 222 |
223 |

Available Mappers ({{ len(mappers) }})

224 |
    225 | {% for mapper in mappers %} 226 |
  • * {{ mapper }}
    • 227 | {% end %} 228 |
229 |
230 | {% end %} 231 | 232 | {% if input_streams %} 233 |
234 |

Available Streams

235 |

Please be advised that the link below may not work if your input stream requires additional arguments in it's URL.

236 |
    237 | {% for stream in input_streams %} 238 |
  • {{ stream }}
    • 239 | {% end %} 240 |
241 |
242 | {% end %} 243 | 244 |
245 |
246 |
247 | 248 | 253 | 254 | 255 | -------------------------------------------------------------------------------- /r3/app/utils.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | import logging 5 | from datetime import datetime 6 | 7 | DATETIME_FORMAT = '%Y-%m-%d %H:%M:%S' 8 | TIMEOUT = 15 9 | 10 | def real_import(name): 11 | if '.' in name: 12 | return reduce(getattr, name.split('.')[1:], __import__(name)) 13 | return __import__(name) 14 | 15 | logger = logging.getLogger('R3ServiceApp') 16 | 17 | def flush_dead_mappers(redis, mappers_key, ping_key): 18 | mappers = redis.smembers(mappers_key) 19 | for mapper in mappers: 20 | last_ping = redis.get(ping_key % mapper) 21 | if last_ping: 22 | now = datetime.now() 23 | last_ping = datetime.strptime(last_ping, DATETIME_FORMAT) 24 | if ((now - last_ping).seconds > TIMEOUT): 25 | logging.warning('MAPPER %s found to be inactive after %d seconds of not pinging back' % (mapper, TIMEOUT)) 26 | redis.srem(mappers_key, mapper) 27 | redis.delete(ping_key % mapper) 28 | 29 | 30 | def kls_import(fullname): 31 | if not '.' in fullname: 32 | return __import__(fullname) 33 | 34 | name_parts = fullname.split('.') 35 | klass_name = name_parts[-1] 36 | module_parts = name_parts[:-1] 37 | module = reduce(getattr, module_parts[1:], __import__('.'.join(module_parts))) 38 | klass = getattr(module, klass_name) 39 | return klass 40 | 41 | 42 | -------------------------------------------------------------------------------- /r3/version.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | __version__ = '0.2.0' 5 | version = __version__ 6 | VERSION = __version__ 7 | -------------------------------------------------------------------------------- /r3/web/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/heynemann/r3/cc6b4eb55c7ae30a8f75af2be165504565dbeb79/r3/web/__init__.py -------------------------------------------------------------------------------- /r3/web/app.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | from flask import Flask, render_template, g, redirect, url_for 5 | from ujson import loads 6 | 7 | from r3.version import __version__ 8 | from r3.app.utils import flush_dead_mappers 9 | from r3.app.keys import MAPPERS_KEY, JOB_TYPES_KEY, JOB_TYPE_KEY, LAST_PING_KEY, MAPPER_ERROR_KEY, MAPPER_WORKING_KEY, JOB_TYPES_ERRORS_KEY, ALL_KEYS, PROCESSED, PROCESSED_FAILED 10 | 11 | app = Flask(__name__) 12 | 13 | def server_context(): 14 | return { 15 | 'r3_service_status': 'running', 16 | 'r3_version': __version__ 17 | } 18 | 19 | @app.before_request 20 | def before_request(): 21 | g.config = app.config 22 | g.server = server_context() 23 | g.job_types = app.db.connection.smembers(JOB_TYPES_KEY) 24 | g.jobs = get_all_jobs(g.job_types) 25 | g.mappers = get_mappers() 26 | 27 | def get_mappers(): 28 | all_mappers = app.db.connection.smembers(MAPPERS_KEY) 29 | mappers_status = {} 30 | for mapper in all_mappers: 31 | key = MAPPER_WORKING_KEY % mapper 32 | working = app.db.connection.lrange(key, 0, -1) 33 | if not working: 34 | mappers_status[mapper] = None 35 | else: 36 | mappers_status[mapper] = loads(working[0]) 37 | 38 | return mappers_status 39 | 40 | def get_all_jobs(all_job_types): 41 | all_jobs = {} 42 | for job_type in all_job_types: 43 | job_type_jobs = app.db.connection.smembers(JOB_TYPE_KEY % job_type) 44 | all_jobs[job_type] = [] 45 | if job_type_jobs: 46 | all_jobs[job_type] = job_type_jobs 47 | 48 | return all_jobs 49 | 50 | def get_errors(): 51 | errors = [] 52 | for job_type in g.job_types: 53 | errors = [loads(item) for key, item in app.db.connection.hgetall(MAPPER_ERROR_KEY % job_type).iteritems()] 54 | 55 | return errors 56 | 57 | @app.route("/") 58 | def index(): 59 | error_queues = app.db.connection.keys(JOB_TYPES_ERRORS_KEY) 60 | 61 | has_errors = False 62 | for queue in error_queues: 63 | if app.db.connection.hlen(queue) > 0: 64 | has_errors = True 65 | 66 | flush_dead_mappers(app.db.connection, MAPPERS_KEY, LAST_PING_KEY) 67 | 68 | return render_template('index.html', failed_warning=has_errors) 69 | 70 | @app.route("/mappers") 71 | def mappers(): 72 | flush_dead_mappers(app.db.connection, MAPPERS_KEY, LAST_PING_KEY) 73 | return render_template('mappers.html') 74 | 75 | @app.route("/failed") 76 | def failed(): 77 | return render_template('failed.html', errors=get_errors()) 78 | 79 | @app.route("/failed/delete") 80 | def delete_all_failed(): 81 | for job_type in g.job_types: 82 | key = MAPPER_ERROR_KEY % job_type 83 | app.db.connection.delete(key) 84 | 85 | return redirect(url_for('failed')) 86 | 87 | @app.route("/failed/delete/") 88 | def delete_failed(job_id): 89 | for job_type in g.job_types: 90 | key = MAPPER_ERROR_KEY % job_type 91 | if app.db.connection.hexists(key, job_id): 92 | app.db.connection.hdel(key, job_id) 93 | 94 | return redirect(url_for('failed')) 95 | 96 | @app.route("/job-types") 97 | def job_types(): 98 | return render_template('job-types.html') 99 | 100 | @app.route("/stats") 101 | def stats(): 102 | info = app.db.connection.info() 103 | key_names = app.db.connection.keys(ALL_KEYS) 104 | 105 | keys = [] 106 | for key in key_names: 107 | key_type = app.db.connection.type(key) 108 | 109 | if key_type == 'list': 110 | size = app.db.connection.llen(key) 111 | elif key_type == 'set': 112 | size = app.db.connection.scard(key) 113 | else: 114 | size = 1 115 | 116 | keys.append({ 117 | 'name': key, 118 | 'size': size, 119 | 'type': key_type 120 | }) 121 | 122 | processed = app.db.connection.get(PROCESSED) 123 | processed_failed = app.db.connection.get(PROCESSED_FAILED) 124 | 125 | return render_template('stats.html', info=info, keys=keys, processed=processed, failed=processed_failed) 126 | 127 | @app.route("/stats/keys/") 128 | def key(key): 129 | key_type = app.db.connection.type(key) 130 | 131 | if key_type == 'list': 132 | value = app.db.connection.lrange(key, 0, -1) 133 | multi = True 134 | elif key_type == 'set': 135 | value = app.db.connection.smembers(key) 136 | multi = True 137 | else: 138 | value = app.db.connection.get(key) 139 | multi = False 140 | 141 | return render_template('show_key.html', key=key, multi=multi, value=value) 142 | 143 | @app.route("/stats/keys//delete") 144 | def delete_key(key): 145 | app.db.connection.delete(key) 146 | return redirect(url_for('stats')) 147 | 148 | #if __name__ == "__main__": 149 | #app.config.from_object('r3.web.config') 150 | #db = RedisDB(app) 151 | #app.run(debug=True, host=app.config['WEB_HOST'], port=app.config['WEB_PORT']) 152 | 153 | -------------------------------------------------------------------------------- /r3/web/config.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | DEBUG = True 5 | SECRET_KEY = 'development key' 6 | 7 | WEB_HOST = '0.0.0.0' 8 | WEB_PORT = 8888 9 | 10 | REDIS_HOST = 'localhost' 11 | REDIS_PORT = 7778 12 | REDIS_PASS = 'r3' 13 | -------------------------------------------------------------------------------- /r3/web/extensions.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | import redis 5 | 6 | from flask import _app_ctx_stack as stack 7 | 8 | class RedisDB(object): 9 | 10 | def __init__(self, app=None): 11 | if app is not None: 12 | self.app = app 13 | self.init_app(self.app) 14 | else: 15 | self.app = None 16 | 17 | def init_app(self, app): 18 | app.config.setdefault('REDIS_HOST', '0.0.0.0') 19 | app.config.setdefault('REDIS_PORT', 6379) 20 | app.config.setdefault('REDIS_DB', 0) 21 | app.config.setdefault('REDIS_PASS', None) 22 | 23 | # Use the newstyle teardown_appcontext if it's available, 24 | # otherwise fall back to the request context 25 | if hasattr(app, 'teardown_appcontext'): 26 | app.teardown_appcontext(self.teardown) 27 | else: 28 | app.teardown_request(self.teardown) 29 | 30 | def connect(self): 31 | options = { 32 | 'host': self.app.config['REDIS_HOST'], 33 | 'port': self.app.config['REDIS_PORT'], 34 | 'db': self.app.config['REDIS_DB'] 35 | } 36 | 37 | if self.app.config['REDIS_PASS']: 38 | options['password'] = self.app.config['REDIS_PASS'] 39 | 40 | conn = redis.StrictRedis(**options) 41 | return conn 42 | 43 | def teardown(self, exception): 44 | ctx = stack.top 45 | if hasattr(ctx, 'redis_db'): 46 | del ctx.redis_db 47 | 48 | @property 49 | def connection(self): 50 | ctx = stack.top 51 | if ctx is not None: 52 | if not hasattr(ctx, 'redis_db'): 53 | ctx.redis_db = self.connect() 54 | return ctx.redis_db 55 | -------------------------------------------------------------------------------- /r3/web/server.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | import sys 5 | import os 6 | from os.path import abspath, isabs, join 7 | import logging 8 | import argparse 9 | 10 | from r3.web.app import app 11 | from r3.web.extensions import RedisDB 12 | 13 | def main(arguments=None): 14 | '''Runs r³ web app with the specified arguments.''' 15 | 16 | parser = argparse.ArgumentParser(description='runs the web admin that helps in monitoring r³ usage') 17 | parser.add_argument('-b', '--bind', type=str, default='0.0.0.0', help='the ip that r³ will bind to') 18 | parser.add_argument('-p', '--port', type=int, default=8888, help='the port that r³ will bind to') 19 | parser.add_argument('-l', '--loglevel', type=str, default='warning', help='the log level that r³ will run under') 20 | parser.add_argument('--redis-host', type=str, default='0.0.0.0', help='the ip that r³ will use to connect to redis') 21 | parser.add_argument('--redis-port', type=int, default=6379, help='the port that r³ will use to connect to redis') 22 | parser.add_argument('--redis-db', type=int, default=0, help='the database that r³ will use to connect to redis') 23 | parser.add_argument('--redis-pass', type=str, default='', help='the password that r³ will use to connect to redis') 24 | parser.add_argument('-c', '--config-file', type=str, default='', help='the configuration file that r³ will use') 25 | parser.add_argument('-d', '--debug', default=False, action='store_true', help='indicates that r³ will be run in debug mode') 26 | 27 | args = parser.parse_args(arguments) 28 | 29 | logging.basicConfig(level=getattr(logging, args.loglevel.upper())) 30 | 31 | if args.config_file: 32 | config_path = args.config_file 33 | if not isabs(args.config_file): 34 | config_path = abspath(join(os.curdir, args.config_file)) 35 | 36 | app.config.from_pyfile(config_path, silent=False) 37 | else: 38 | app.config.from_object('r3.web.config') 39 | 40 | app.db = RedisDB(app) 41 | try: 42 | logging.debug('r³ web app running at %s:%d' % (args.bind, args.port)) 43 | app.run(debug=args.debug, host=app.config['WEB_HOST'], port=app.config['WEB_PORT']) 44 | except KeyboardInterrupt: 45 | print 46 | print "-- r³ web app closed by user interruption --" 47 | 48 | if __name__ == "__main__": 49 | main(sys.argv[1:]) 50 | -------------------------------------------------------------------------------- /r3/web/static/css/bootstrap.min.css: -------------------------------------------------------------------------------- 1 | /*! 2 | * Bootstrap v2.0.4 3 | * 4 | * Copyright 2012 Twitter, Inc 5 | * Licensed under the Apache License v2.0 6 | * http://www.apache.org/licenses/LICENSE-2.0 7 | * 8 | * Designed and built with all the love in the world @twitter by @mdo and @fat. 9 | */ 10 | .clearfix{*zoom:1;}.clearfix:before,.clearfix:after{display:table;content:"";} 11 | .clearfix:after{clear:both;} 12 | .hide-text{font:0/0 a;color:transparent;text-shadow:none;background-color:transparent;border:0;} 13 | .input-block-level{display:block;width:100%;min-height:28px;-webkit-box-sizing:border-box;-moz-box-sizing:border-box;-ms-box-sizing:border-box;box-sizing:border-box;} 14 | code,pre{padding:0 3px 2px;font-family:Menlo,Monaco,Consolas,"Courier New",monospace;font-size:12px;color:#333333;-webkit-border-radius:3px;-moz-border-radius:3px;border-radius:3px;} 15 | code{padding:2px 4px;color:#d14;background-color:#f7f7f9;border:1px solid #e1e1e8;} 16 | pre{display:block;padding:8.5px;margin:0 0 9px;font-size:12.025px;line-height:18px;word-break:break-all;word-wrap:break-word;white-space:pre;white-space:pre-wrap;background-color:#f5f5f5;border:1px solid #ccc;border:1px solid rgba(0, 0, 0, 0.15);-webkit-border-radius:4px;-moz-border-radius:4px;border-radius:4px;}pre.prettyprint{margin-bottom:18px;} 17 | pre code{padding:0;color:inherit;background-color:transparent;border:0;} 18 | .pre-scrollable{max-height:340px;overflow-y:scroll;} 19 | table{max-width:100%;background-color:transparent;border-collapse:collapse;border-spacing:0;} 20 | .table{width:100%;margin-bottom:18px;}.table th,.table td{padding:8px;line-height:18px;text-align:left;vertical-align:top;border-top:1px solid #dddddd;} 21 | .table th{font-weight:bold;} 22 | .table thead th{vertical-align:bottom;} 23 | .table caption+thead tr:first-child th,.table caption+thead tr:first-child td,.table colgroup+thead tr:first-child th,.table colgroup+thead tr:first-child td,.table thead:first-child tr:first-child th,.table thead:first-child tr:first-child td{border-top:0;} 24 | .table tbody+tbody{border-top:2px solid #dddddd;} 25 | .table-condensed th,.table-condensed td{padding:4px 5px;} 26 | .table-bordered{border:1px solid #dddddd;border-collapse:separate;*border-collapse:collapsed;border-left:0;-webkit-border-radius:4px;-moz-border-radius:4px;border-radius:4px;}.table-bordered th,.table-bordered td{border-left:1px solid #dddddd;} 27 | .table-bordered caption+thead tr:first-child th,.table-bordered caption+tbody tr:first-child th,.table-bordered caption+tbody tr:first-child td,.table-bordered colgroup+thead tr:first-child th,.table-bordered colgroup+tbody tr:first-child th,.table-bordered colgroup+tbody tr:first-child td,.table-bordered thead:first-child tr:first-child th,.table-bordered tbody:first-child tr:first-child th,.table-bordered tbody:first-child tr:first-child td{border-top:0;} 28 | .table-bordered thead:first-child tr:first-child th:first-child,.table-bordered tbody:first-child tr:first-child td:first-child{-webkit-border-top-left-radius:4px;border-top-left-radius:4px;-moz-border-radius-topleft:4px;} 29 | .table-bordered thead:first-child tr:first-child th:last-child,.table-bordered tbody:first-child tr:first-child td:last-child{-webkit-border-top-right-radius:4px;border-top-right-radius:4px;-moz-border-radius-topright:4px;} 30 | .table-bordered thead:last-child tr:last-child th:first-child,.table-bordered tbody:last-child tr:last-child td:first-child{-webkit-border-radius:0 0 0 4px;-moz-border-radius:0 0 0 4px;border-radius:0 0 0 4px;-webkit-border-bottom-left-radius:4px;border-bottom-left-radius:4px;-moz-border-radius-bottomleft:4px;} 31 | .table-bordered thead:last-child tr:last-child th:last-child,.table-bordered tbody:last-child tr:last-child td:last-child{-webkit-border-bottom-right-radius:4px;border-bottom-right-radius:4px;-moz-border-radius-bottomright:4px;} 32 | .table-striped tbody tr:nth-child(odd) td,.table-striped tbody tr:nth-child(odd) th{background-color:#f9f9f9;} 33 | .table tbody tr:hover td,.table tbody tr:hover th{background-color:#f5f5f5;} 34 | table .span1{float:none;width:44px;margin-left:0;} 35 | table .span2{float:none;width:124px;margin-left:0;} 36 | table .span3{float:none;width:204px;margin-left:0;} 37 | table .span4{float:none;width:284px;margin-left:0;} 38 | table .span5{float:none;width:364px;margin-left:0;} 39 | table .span6{float:none;width:444px;margin-left:0;} 40 | table .span7{float:none;width:524px;margin-left:0;} 41 | table .span8{float:none;width:604px;margin-left:0;} 42 | table .span9{float:none;width:684px;margin-left:0;} 43 | table .span10{float:none;width:764px;margin-left:0;} 44 | table .span11{float:none;width:844px;margin-left:0;} 45 | table .span12{float:none;width:924px;margin-left:0;} 46 | table .span13{float:none;width:1004px;margin-left:0;} 47 | table .span14{float:none;width:1084px;margin-left:0;} 48 | table .span15{float:none;width:1164px;margin-left:0;} 49 | table .span16{float:none;width:1244px;margin-left:0;} 50 | table .span17{float:none;width:1324px;margin-left:0;} 51 | table .span18{float:none;width:1404px;margin-left:0;} 52 | table .span19{float:none;width:1484px;margin-left:0;} 53 | table .span20{float:none;width:1564px;margin-left:0;} 54 | table .span21{float:none;width:1644px;margin-left:0;} 55 | table .span22{float:none;width:1724px;margin-left:0;} 56 | table .span23{float:none;width:1804px;margin-left:0;} 57 | table .span24{float:none;width:1884px;margin-left:0;} 58 | .btn{display:inline-block;*display:inline;*zoom:1;padding:4px 10px 4px;margin-bottom:0;font-size:13px;line-height:18px;*line-height:20px;color:#333333;text-align:center;text-shadow:0 1px 1px rgba(255, 255, 255, 0.75);vertical-align:middle;cursor:pointer;background-color:#f5f5f5;background-image:-moz-linear-gradient(top, #ffffff, #e6e6e6);background-image:-ms-linear-gradient(top, #ffffff, #e6e6e6);background-image:-webkit-gradient(linear, 0 0, 0 100%, from(#ffffff), to(#e6e6e6));background-image:-webkit-linear-gradient(top, #ffffff, #e6e6e6);background-image:-o-linear-gradient(top, #ffffff, #e6e6e6);background-image:linear-gradient(top, #ffffff, #e6e6e6);background-repeat:repeat-x;filter:progid:DXImageTransform.Microsoft.gradient(startColorstr='#ffffff', endColorstr='#e6e6e6', GradientType=0);border-color:#e6e6e6 #e6e6e6 #bfbfbf;border-color:rgba(0, 0, 0, 0.1) rgba(0, 0, 0, 0.1) rgba(0, 0, 0, 0.25);*background-color:#e6e6e6;filter:progid:DXImageTransform.Microsoft.gradient(enabled = false);border:1px solid #cccccc;*border:0;border-bottom-color:#b3b3b3;-webkit-border-radius:4px;-moz-border-radius:4px;border-radius:4px;*margin-left:.3em;-webkit-box-shadow:inset 0 1px 0 rgba(255,255,255,.2), 0 1px 2px rgba(0,0,0,.05);-moz-box-shadow:inset 0 1px 0 rgba(255,255,255,.2), 0 1px 2px rgba(0,0,0,.05);box-shadow:inset 0 1px 0 rgba(255,255,255,.2), 0 1px 2px rgba(0,0,0,.05);}.btn:hover,.btn:active,.btn.active,.btn.disabled,.btn[disabled]{background-color:#e6e6e6;*background-color:#d9d9d9;} 59 | .btn:active,.btn.active{background-color:#cccccc \9;} 60 | .btn:first-child{*margin-left:0;} 61 | .btn:hover{color:#333333;text-decoration:none;background-color:#e6e6e6;*background-color:#d9d9d9;background-position:0 -15px;-webkit-transition:background-position 0.1s linear;-moz-transition:background-position 0.1s linear;-ms-transition:background-position 0.1s linear;-o-transition:background-position 0.1s linear;transition:background-position 0.1s linear;} 62 | .btn:focus{outline:thin dotted #333;outline:5px auto -webkit-focus-ring-color;outline-offset:-2px;} 63 | .btn.active,.btn:active{background-color:#e6e6e6;background-color:#d9d9d9 \9;background-image:none;outline:0;-webkit-box-shadow:inset 0 2px 4px rgba(0,0,0,.15), 0 1px 2px rgba(0,0,0,.05);-moz-box-shadow:inset 0 2px 4px rgba(0,0,0,.15), 0 1px 2px rgba(0,0,0,.05);box-shadow:inset 0 2px 4px rgba(0,0,0,.15), 0 1px 2px rgba(0,0,0,.05);} 64 | .btn.disabled,.btn[disabled]{cursor:default;background-color:#e6e6e6;background-image:none;opacity:0.65;filter:alpha(opacity=65);-webkit-box-shadow:none;-moz-box-shadow:none;box-shadow:none;} 65 | .btn-large{padding:9px 14px;font-size:15px;line-height:normal;-webkit-border-radius:5px;-moz-border-radius:5px;border-radius:5px;} 66 | .btn-large [class^="icon-"]{margin-top:1px;} 67 | .btn-small{padding:5px 9px;font-size:11px;line-height:16px;} 68 | .btn-small [class^="icon-"]{margin-top:-1px;} 69 | .btn-mini{padding:2px 6px;font-size:11px;line-height:14px;} 70 | .btn-primary,.btn-primary:hover,.btn-warning,.btn-warning:hover,.btn-danger,.btn-danger:hover,.btn-success,.btn-success:hover,.btn-info,.btn-info:hover,.btn-inverse,.btn-inverse:hover{color:#ffffff;text-shadow:0 -1px 0 rgba(0, 0, 0, 0.25);} 71 | .btn-primary.active,.btn-warning.active,.btn-danger.active,.btn-success.active,.btn-info.active,.btn-inverse.active{color:rgba(255, 255, 255, 0.75);} 72 | .btn{border-color:#ccc;border-color:rgba(0, 0, 0, 0.1) rgba(0, 0, 0, 0.1) rgba(0, 0, 0, 0.25);} 73 | .btn-primary{background-color:#0074cc;background-image:-moz-linear-gradient(top, #0088cc, #0055cc);background-image:-ms-linear-gradient(top, #0088cc, #0055cc);background-image:-webkit-gradient(linear, 0 0, 0 100%, from(#0088cc), to(#0055cc));background-image:-webkit-linear-gradient(top, #0088cc, #0055cc);background-image:-o-linear-gradient(top, #0088cc, #0055cc);background-image:linear-gradient(top, #0088cc, #0055cc);background-repeat:repeat-x;filter:progid:DXImageTransform.Microsoft.gradient(startColorstr='#0088cc', endColorstr='#0055cc', GradientType=0);border-color:#0055cc #0055cc #003580;border-color:rgba(0, 0, 0, 0.1) rgba(0, 0, 0, 0.1) rgba(0, 0, 0, 0.25);*background-color:#0055cc;filter:progid:DXImageTransform.Microsoft.gradient(enabled = false);}.btn-primary:hover,.btn-primary:active,.btn-primary.active,.btn-primary.disabled,.btn-primary[disabled]{background-color:#0055cc;*background-color:#004ab3;} 74 | .btn-primary:active,.btn-primary.active{background-color:#004099 \9;} 75 | .btn-warning{background-color:#faa732;background-image:-moz-linear-gradient(top, #fbb450, #f89406);background-image:-ms-linear-gradient(top, #fbb450, #f89406);background-image:-webkit-gradient(linear, 0 0, 0 100%, from(#fbb450), to(#f89406));background-image:-webkit-linear-gradient(top, #fbb450, #f89406);background-image:-o-linear-gradient(top, #fbb450, #f89406);background-image:linear-gradient(top, #fbb450, #f89406);background-repeat:repeat-x;filter:progid:DXImageTransform.Microsoft.gradient(startColorstr='#fbb450', endColorstr='#f89406', GradientType=0);border-color:#f89406 #f89406 #ad6704;border-color:rgba(0, 0, 0, 0.1) rgba(0, 0, 0, 0.1) rgba(0, 0, 0, 0.25);*background-color:#f89406;filter:progid:DXImageTransform.Microsoft.gradient(enabled = false);}.btn-warning:hover,.btn-warning:active,.btn-warning.active,.btn-warning.disabled,.btn-warning[disabled]{background-color:#f89406;*background-color:#df8505;} 76 | .btn-warning:active,.btn-warning.active{background-color:#c67605 \9;} 77 | .btn-danger{background-color:#da4f49;background-image:-moz-linear-gradient(top, #ee5f5b, #bd362f);background-image:-ms-linear-gradient(top, #ee5f5b, #bd362f);background-image:-webkit-gradient(linear, 0 0, 0 100%, from(#ee5f5b), to(#bd362f));background-image:-webkit-linear-gradient(top, #ee5f5b, #bd362f);background-image:-o-linear-gradient(top, #ee5f5b, #bd362f);background-image:linear-gradient(top, #ee5f5b, #bd362f);background-repeat:repeat-x;filter:progid:DXImageTransform.Microsoft.gradient(startColorstr='#ee5f5b', endColorstr='#bd362f', GradientType=0);border-color:#bd362f #bd362f #802420;border-color:rgba(0, 0, 0, 0.1) rgba(0, 0, 0, 0.1) rgba(0, 0, 0, 0.25);*background-color:#bd362f;filter:progid:DXImageTransform.Microsoft.gradient(enabled = false);}.btn-danger:hover,.btn-danger:active,.btn-danger.active,.btn-danger.disabled,.btn-danger[disabled]{background-color:#bd362f;*background-color:#a9302a;} 78 | .btn-danger:active,.btn-danger.active{background-color:#942a25 \9;} 79 | .btn-success{background-color:#5bb75b;background-image:-moz-linear-gradient(top, #62c462, #51a351);background-image:-ms-linear-gradient(top, #62c462, #51a351);background-image:-webkit-gradient(linear, 0 0, 0 100%, from(#62c462), to(#51a351));background-image:-webkit-linear-gradient(top, #62c462, #51a351);background-image:-o-linear-gradient(top, #62c462, #51a351);background-image:linear-gradient(top, #62c462, #51a351);background-repeat:repeat-x;filter:progid:DXImageTransform.Microsoft.gradient(startColorstr='#62c462', endColorstr='#51a351', GradientType=0);border-color:#51a351 #51a351 #387038;border-color:rgba(0, 0, 0, 0.1) rgba(0, 0, 0, 0.1) rgba(0, 0, 0, 0.25);*background-color:#51a351;filter:progid:DXImageTransform.Microsoft.gradient(enabled = false);}.btn-success:hover,.btn-success:active,.btn-success.active,.btn-success.disabled,.btn-success[disabled]{background-color:#51a351;*background-color:#499249;} 80 | .btn-success:active,.btn-success.active{background-color:#408140 \9;} 81 | .btn-info{background-color:#49afcd;background-image:-moz-linear-gradient(top, #5bc0de, #2f96b4);background-image:-ms-linear-gradient(top, #5bc0de, #2f96b4);background-image:-webkit-gradient(linear, 0 0, 0 100%, from(#5bc0de), to(#2f96b4));background-image:-webkit-linear-gradient(top, #5bc0de, #2f96b4);background-image:-o-linear-gradient(top, #5bc0de, #2f96b4);background-image:linear-gradient(top, #5bc0de, #2f96b4);background-repeat:repeat-x;filter:progid:DXImageTransform.Microsoft.gradient(startColorstr='#5bc0de', endColorstr='#2f96b4', GradientType=0);border-color:#2f96b4 #2f96b4 #1f6377;border-color:rgba(0, 0, 0, 0.1) rgba(0, 0, 0, 0.1) rgba(0, 0, 0, 0.25);*background-color:#2f96b4;filter:progid:DXImageTransform.Microsoft.gradient(enabled = false);}.btn-info:hover,.btn-info:active,.btn-info.active,.btn-info.disabled,.btn-info[disabled]{background-color:#2f96b4;*background-color:#2a85a0;} 82 | .btn-info:active,.btn-info.active{background-color:#24748c \9;} 83 | .btn-inverse{background-color:#414141;background-image:-moz-linear-gradient(top, #555555, #222222);background-image:-ms-linear-gradient(top, #555555, #222222);background-image:-webkit-gradient(linear, 0 0, 0 100%, from(#555555), to(#222222));background-image:-webkit-linear-gradient(top, #555555, #222222);background-image:-o-linear-gradient(top, #555555, #222222);background-image:linear-gradient(top, #555555, #222222);background-repeat:repeat-x;filter:progid:DXImageTransform.Microsoft.gradient(startColorstr='#555555', endColorstr='#222222', GradientType=0);border-color:#222222 #222222 #000000;border-color:rgba(0, 0, 0, 0.1) rgba(0, 0, 0, 0.1) rgba(0, 0, 0, 0.25);*background-color:#222222;filter:progid:DXImageTransform.Microsoft.gradient(enabled = false);}.btn-inverse:hover,.btn-inverse:active,.btn-inverse.active,.btn-inverse.disabled,.btn-inverse[disabled]{background-color:#222222;*background-color:#151515;} 84 | .btn-inverse:active,.btn-inverse.active{background-color:#080808 \9;} 85 | button.btn,input[type="submit"].btn{*padding-top:2px;*padding-bottom:2px;}button.btn::-moz-focus-inner,input[type="submit"].btn::-moz-focus-inner{padding:0;border:0;} 86 | button.btn.btn-large,input[type="submit"].btn.btn-large{*padding-top:7px;*padding-bottom:7px;} 87 | button.btn.btn-small,input[type="submit"].btn.btn-small{*padding-top:3px;*padding-bottom:3px;} 88 | button.btn.btn-mini,input[type="submit"].btn.btn-mini{*padding-top:1px;*padding-bottom:1px;} 89 | .nav{margin-left:0;margin-bottom:18px;list-style:none;} 90 | .nav>li>a{display:block;} 91 | .nav>li>a:hover{text-decoration:none;background-color:#eeeeee;} 92 | .nav>.pull-right{float:right;} 93 | .nav .nav-header{display:block;padding:3px 15px;font-size:11px;font-weight:bold;line-height:18px;color:#999999;text-shadow:0 1px 0 rgba(255, 255, 255, 0.5);text-transform:uppercase;} 94 | .nav li+.nav-header{margin-top:9px;} 95 | .nav-list{padding-left:15px;padding-right:15px;margin-bottom:0;} 96 | .nav-list>li>a,.nav-list .nav-header{margin-left:-15px;margin-right:-15px;text-shadow:0 1px 0 rgba(255, 255, 255, 0.5);} 97 | .nav-list>li>a{padding:3px 15px;} 98 | .nav-list>.active>a,.nav-list>.active>a:hover{color:#ffffff;text-shadow:0 -1px 0 rgba(0, 0, 0, 0.2);background-color:#0088cc;} 99 | .nav-list [class^="icon-"]{margin-right:2px;} 100 | .nav-list .divider{*width:100%;height:1px;margin:8px 1px;*margin:-5px 0 5px;overflow:hidden;background-color:#e5e5e5;border-bottom:1px solid #ffffff;} 101 | .nav-tabs,.nav-pills{*zoom:1;}.nav-tabs:before,.nav-pills:before,.nav-tabs:after,.nav-pills:after{display:table;content:"";} 102 | .nav-tabs:after,.nav-pills:after{clear:both;} 103 | .nav-tabs>li,.nav-pills>li{float:left;} 104 | .nav-tabs>li>a,.nav-pills>li>a{padding-right:12px;padding-left:12px;margin-right:2px;line-height:14px;} 105 | .nav-tabs{border-bottom:1px solid #ddd;} 106 | .nav-tabs>li{margin-bottom:-1px;} 107 | .nav-tabs>li>a{padding-top:8px;padding-bottom:8px;line-height:18px;border:1px solid transparent;-webkit-border-radius:4px 4px 0 0;-moz-border-radius:4px 4px 0 0;border-radius:4px 4px 0 0;}.nav-tabs>li>a:hover{border-color:#eeeeee #eeeeee #dddddd;} 108 | .nav-tabs>.active>a,.nav-tabs>.active>a:hover{color:#555555;background-color:#ffffff;border:1px solid #ddd;border-bottom-color:transparent;cursor:default;} 109 | .nav-pills>li>a{padding-top:8px;padding-bottom:8px;margin-top:2px;margin-bottom:2px;-webkit-border-radius:5px;-moz-border-radius:5px;border-radius:5px;} 110 | .nav-pills>.active>a,.nav-pills>.active>a:hover{color:#ffffff;background-color:#0088cc;} 111 | .nav-stacked>li{float:none;} 112 | .nav-stacked>li>a{margin-right:0;} 113 | .nav-tabs.nav-stacked{border-bottom:0;} 114 | .nav-tabs.nav-stacked>li>a{border:1px solid #ddd;-webkit-border-radius:0;-moz-border-radius:0;border-radius:0;} 115 | .nav-tabs.nav-stacked>li:first-child>a{-webkit-border-radius:4px 4px 0 0;-moz-border-radius:4px 4px 0 0;border-radius:4px 4px 0 0;} 116 | .nav-tabs.nav-stacked>li:last-child>a{-webkit-border-radius:0 0 4px 4px;-moz-border-radius:0 0 4px 4px;border-radius:0 0 4px 4px;} 117 | .nav-tabs.nav-stacked>li>a:hover{border-color:#ddd;z-index:2;} 118 | .nav-pills.nav-stacked>li>a{margin-bottom:3px;} 119 | .nav-pills.nav-stacked>li:last-child>a{margin-bottom:1px;} 120 | .nav-tabs .dropdown-menu{-webkit-border-radius:0 0 5px 5px;-moz-border-radius:0 0 5px 5px;border-radius:0 0 5px 5px;} 121 | .nav-pills .dropdown-menu{-webkit-border-radius:4px;-moz-border-radius:4px;border-radius:4px;} 122 | .nav-tabs .dropdown-toggle .caret,.nav-pills .dropdown-toggle .caret{border-top-color:#0088cc;border-bottom-color:#0088cc;margin-top:6px;} 123 | .nav-tabs .dropdown-toggle:hover .caret,.nav-pills .dropdown-toggle:hover .caret{border-top-color:#005580;border-bottom-color:#005580;} 124 | .nav-tabs .active .dropdown-toggle .caret,.nav-pills .active .dropdown-toggle .caret{border-top-color:#333333;border-bottom-color:#333333;} 125 | .nav>.dropdown.active>a:hover{color:#000000;cursor:pointer;} 126 | .nav-tabs .open .dropdown-toggle,.nav-pills .open .dropdown-toggle,.nav>li.dropdown.open.active>a:hover{color:#ffffff;background-color:#999999;border-color:#999999;} 127 | .nav li.dropdown.open .caret,.nav li.dropdown.open.active .caret,.nav li.dropdown.open a:hover .caret{border-top-color:#ffffff;border-bottom-color:#ffffff;opacity:1;filter:alpha(opacity=100);} 128 | .tabs-stacked .open>a:hover{border-color:#999999;} 129 | .tabbable{*zoom:1;}.tabbable:before,.tabbable:after{display:table;content:"";} 130 | .tabbable:after{clear:both;} 131 | .tab-content{overflow:auto;} 132 | .tabs-below>.nav-tabs,.tabs-right>.nav-tabs,.tabs-left>.nav-tabs{border-bottom:0;} 133 | .tab-content>.tab-pane,.pill-content>.pill-pane{display:none;} 134 | .tab-content>.active,.pill-content>.active{display:block;} 135 | .tabs-below>.nav-tabs{border-top:1px solid #ddd;} 136 | .tabs-below>.nav-tabs>li{margin-top:-1px;margin-bottom:0;} 137 | .tabs-below>.nav-tabs>li>a{-webkit-border-radius:0 0 4px 4px;-moz-border-radius:0 0 4px 4px;border-radius:0 0 4px 4px;}.tabs-below>.nav-tabs>li>a:hover{border-bottom-color:transparent;border-top-color:#ddd;} 138 | .tabs-below>.nav-tabs>.active>a,.tabs-below>.nav-tabs>.active>a:hover{border-color:transparent #ddd #ddd #ddd;} 139 | .tabs-left>.nav-tabs>li,.tabs-right>.nav-tabs>li{float:none;} 140 | .tabs-left>.nav-tabs>li>a,.tabs-right>.nav-tabs>li>a{min-width:74px;margin-right:0;margin-bottom:3px;} 141 | .tabs-left>.nav-tabs{float:left;margin-right:19px;border-right:1px solid #ddd;} 142 | .tabs-left>.nav-tabs>li>a{margin-right:-1px;-webkit-border-radius:4px 0 0 4px;-moz-border-radius:4px 0 0 4px;border-radius:4px 0 0 4px;} 143 | .tabs-left>.nav-tabs>li>a:hover{border-color:#eeeeee #dddddd #eeeeee #eeeeee;} 144 | .tabs-left>.nav-tabs .active>a,.tabs-left>.nav-tabs .active>a:hover{border-color:#ddd transparent #ddd #ddd;*border-right-color:#ffffff;} 145 | .tabs-right>.nav-tabs{float:right;margin-left:19px;border-left:1px solid #ddd;} 146 | .tabs-right>.nav-tabs>li>a{margin-left:-1px;-webkit-border-radius:0 4px 4px 0;-moz-border-radius:0 4px 4px 0;border-radius:0 4px 4px 0;} 147 | .tabs-right>.nav-tabs>li>a:hover{border-color:#eeeeee #eeeeee #eeeeee #dddddd;} 148 | .tabs-right>.nav-tabs .active>a,.tabs-right>.nav-tabs .active>a:hover{border-color:#ddd #ddd #ddd transparent;*border-left-color:#ffffff;} 149 | .pagination{height:36px;margin:18px 0;} 150 | .pagination ul{display:inline-block;*display:inline;*zoom:1;margin-left:0;margin-bottom:0;-webkit-border-radius:3px;-moz-border-radius:3px;border-radius:3px;-webkit-box-shadow:0 1px 2px rgba(0, 0, 0, 0.05);-moz-box-shadow:0 1px 2px rgba(0, 0, 0, 0.05);box-shadow:0 1px 2px rgba(0, 0, 0, 0.05);} 151 | .pagination li{display:inline;} 152 | .pagination a{float:left;padding:0 14px;line-height:34px;text-decoration:none;border:1px solid #ddd;border-left-width:0;} 153 | .pagination a:hover,.pagination .active a{background-color:#f5f5f5;} 154 | .pagination .active a{color:#999999;cursor:default;} 155 | .pagination .disabled span,.pagination .disabled a,.pagination .disabled a:hover{color:#999999;background-color:transparent;cursor:default;} 156 | .pagination li:first-child a{border-left-width:1px;-webkit-border-radius:3px 0 0 3px;-moz-border-radius:3px 0 0 3px;border-radius:3px 0 0 3px;} 157 | .pagination li:last-child a{-webkit-border-radius:0 3px 3px 0;-moz-border-radius:0 3px 3px 0;border-radius:0 3px 3px 0;} 158 | .pagination-centered{text-align:center;} 159 | .pagination-right{text-align:right;} 160 | .pager{margin-left:0;margin-bottom:18px;list-style:none;text-align:center;*zoom:1;}.pager:before,.pager:after{display:table;content:"";} 161 | .pager:after{clear:both;} 162 | .pager li{display:inline;} 163 | .pager a{display:inline-block;padding:5px 14px;background-color:#fff;border:1px solid #ddd;-webkit-border-radius:15px;-moz-border-radius:15px;border-radius:15px;} 164 | .pager a:hover{text-decoration:none;background-color:#f5f5f5;} 165 | .pager .next a{float:right;} 166 | .pager .previous a{float:left;} 167 | .pager .disabled a,.pager .disabled a:hover{color:#999999;background-color:#fff;cursor:default;} 168 | .alert{padding:8px 35px 8px 14px;margin-bottom:18px;text-shadow:0 1px 0 rgba(255, 255, 255, 0.5);background-color:#fcf8e3;border:1px solid #fbeed5;-webkit-border-radius:4px;-moz-border-radius:4px;border-radius:4px;color:#c09853;} 169 | .alert-heading{color:inherit;} 170 | .alert .close{position:relative;top:-2px;right:-21px;line-height:18px;} 171 | .alert-success{background-color:#dff0d8;border-color:#d6e9c6;color:#468847;} 172 | .alert-danger,.alert-error{background-color:#f2dede;border-color:#eed3d7;color:#b94a48;} 173 | .alert-info{background-color:#d9edf7;border-color:#bce8f1;color:#3a87ad;} 174 | .alert-block{padding-top:14px;padding-bottom:14px;} 175 | .alert-block>p,.alert-block>ul{margin-bottom:0;} 176 | .alert-block p+p{margin-top:5px;} 177 | -------------------------------------------------------------------------------- /r3/web/static/css/progress.css: -------------------------------------------------------------------------------- 1 | .meter { 2 | height: 20px; /* Can be anything */ 3 | position: relative; 4 | margin: 20px 0 20px 0; /* Just for demo spacing */ 5 | background: #EFEFEF; 6 | -moz-border-radius: 25px; 7 | -webkit-border-radius: 25px; 8 | border-radius: 25px; 9 | padding: 8px; 10 | -webkit-box-shadow: inset 0 -1px 1px rgba(255,255,255,0.3); 11 | -moz-box-shadow : inset 0 -1px 1px rgba(255,255,255,0.3); 12 | box-shadow : inset 0 -1px 1px rgba(255,255,255,0.3); 13 | } 14 | .meter > span { 15 | display: block; 16 | height: 100%; 17 | -webkit-border-top-right-radius: 8px; 18 | -webkit-border-bottom-right-radius: 8px; 19 | -moz-border-radius-topright: 8px; 20 | -moz-border-radius-bottomright: 8px; 21 | border-top-right-radius: 8px; 22 | border-bottom-right-radius: 8px; 23 | -webkit-border-top-left-radius: 20px; 24 | -webkit-border-bottom-left-radius: 20px; 25 | -moz-border-radius-topleft: 20px; 26 | -moz-border-radius-bottomleft: 20px; 27 | border-top-left-radius: 20px; 28 | border-bottom-left-radius: 20px; 29 | background-color: rgb(43,194,83); 30 | background-image: -webkit-gradient( 31 | linear, 32 | left bottom, 33 | left top, 34 | color-stop(0, rgb(43,194,83)), 35 | color-stop(1, rgb(84,240,84)) 36 | ); 37 | background-image: -moz-linear-gradient( 38 | center bottom, 39 | rgb(43,194,83) 37%, 40 | rgb(84,240,84) 69% 41 | ); 42 | -webkit-box-shadow: 43 | inset 0 2px 9px rgba(255,255,255,0.3), 44 | inset 0 -2px 6px rgba(0,0,0,0.4); 45 | -moz-box-shadow: 46 | inset 0 2px 9px rgba(255,255,255,0.3), 47 | inset 0 -2px 6px rgba(0,0,0,0.4); 48 | box-shadow: 49 | inset 0 2px 9px rgba(255,255,255,0.3), 50 | inset 0 -2px 6px rgba(0,0,0,0.4); 51 | position: relative; 52 | overflow: hidden; 53 | } 54 | .meter > span:after, .animate > span > span { 55 | content: ""; 56 | position: absolute; 57 | top: 0; left: 0; bottom: 0; right: 0; 58 | background-image: 59 | -webkit-gradient(linear, 0 0, 100% 100%, 60 | color-stop(.25, rgba(255, 255, 255, .2)), 61 | color-stop(.25, transparent), color-stop(.5, transparent), 62 | color-stop(.5, rgba(255, 255, 255, .2)), 63 | color-stop(.75, rgba(255, 255, 255, .2)), 64 | color-stop(.75, transparent), to(transparent) 65 | ); 66 | background-image: 67 | -moz-linear-gradient( 68 | -45deg, 69 | rgba(255, 255, 255, .2) 25%, 70 | transparent 25%, 71 | transparent 50%, 72 | rgba(255, 255, 255, .2) 50%, 73 | rgba(255, 255, 255, .2) 75%, 74 | transparent 75%, 75 | transparent 76 | ); 77 | z-index: 1; 78 | -webkit-background-size: 50px 50px; 79 | -moz-background-size: 50px 50px; 80 | background-size: 50px 50px; 81 | -webkit-animation: move 2s linear infinite; 82 | -moz-animation: move 2s linear infinite; 83 | -webkit-border-top-right-radius: 8px; 84 | -webkit-border-bottom-right-radius: 8px; 85 | -moz-border-radius-topright: 8px; 86 | -moz-border-radius-bottomright: 8px; 87 | border-top-right-radius: 8px; 88 | border-bottom-right-radius: 8px; 89 | -webkit-border-top-left-radius: 20px; 90 | -webkit-border-bottom-left-radius: 20px; 91 | -moz-border-radius-topleft: 20px; 92 | -moz-border-radius-bottomleft: 20px; 93 | border-top-left-radius: 20px; 94 | border-bottom-left-radius: 20px; 95 | overflow: hidden; 96 | } 97 | 98 | .animate > span:after { 99 | display: none; 100 | } 101 | 102 | @-webkit-keyframes move { 103 | 0% { 104 | background-position: 0 0; 105 | } 106 | 100% { 107 | background-position: 50px 50px; 108 | } 109 | } 110 | 111 | @-moz-keyframes move { 112 | 0% { 113 | background-position: 0 0; 114 | } 115 | 100% { 116 | background-position: 50px 50px; 117 | } 118 | } 119 | 120 | .blue > span { 121 | background-color: #4d7cb5; 122 | background-image: -moz-linear-gradient(top, #4d7cb5, #dbebff); 123 | background-image: -webkit-gradient(linear,left top,left bottom,color-stop(0, #4d7cb5),color-stop(1, #dbebff)); 124 | background-image: -webkit-linear-gradient(#4d7cb5, #dbebff); 125 | } 126 | 127 | .orange > span { 128 | background-color: #f1a165; 129 | background-image: -moz-linear-gradient(top, #f1a165, #f36d0a); 130 | background-image: -webkit-gradient(linear,left top,left bottom,color-stop(0, #f1a165),color-stop(1, #f36d0a)); 131 | background-image: -webkit-linear-gradient(#f1a165, #f36d0a); 132 | } 133 | 134 | .red > span { 135 | background-color: #f0a3a3; 136 | background-image: -moz-linear-gradient(top, #f0a3a3, #f42323); 137 | background-image: -webkit-gradient(linear,left top,left bottom,color-stop(0, #f0a3a3),color-stop(1, #f42323)); 138 | background-image: -webkit-linear-gradient(#f0a3a3, #f42323); 139 | } 140 | 141 | .nostripes > span > span, .nostripes > span:after { 142 | -webkit-animation: none; 143 | -moz-animation: none; 144 | background-image: none; 145 | } 146 | 147 | -------------------------------------------------------------------------------- /r3/web/static/css/reset.css: -------------------------------------------------------------------------------- 1 | /* http://meyerweb.com/eric/tools/css/reset/ 2 | v2.0 | 20110126 3 | License: none (public domain) 4 | */ 5 | 6 | html, body, div, span, applet, object, iframe, 7 | h1, h2, h3, h4, h5, h6, p, blockquote, pre, 8 | a, abbr, acronym, address, big, cite, code, 9 | del, dfn, em, img, ins, kbd, q, s, samp, 10 | small, strike, strong, sub, sup, tt, var, 11 | b, u, i, center, 12 | dl, dt, dd, ol, ul, li, 13 | fieldset, form, label, legend, 14 | table, caption, tbody, tfoot, thead, tr, th, td, 15 | article, aside, canvas, details, embed, 16 | figure, figcaption, footer, header, hgroup, 17 | menu, nav, output, ruby, section, summary, 18 | time, mark, audio, video { 19 | margin: 0; 20 | padding: 0; 21 | border: 0; 22 | font-size: 100%; 23 | font: inherit; 24 | vertical-align: baseline; 25 | } 26 | /* HTML5 display-role reset for older browsers */ 27 | article, aside, details, figcaption, figure, 28 | footer, header, hgroup, menu, nav, section { 29 | display: block; 30 | } 31 | body { 32 | line-height: 1; 33 | } 34 | ol, ul { 35 | list-style: none; 36 | } 37 | blockquote, q { 38 | quotes: none; 39 | } 40 | blockquote:before, blockquote:after, 41 | q:before, q:after { 42 | content: ''; 43 | content: none; 44 | } 45 | table { 46 | border-collapse: collapse; 47 | border-spacing: 0; 48 | } 49 | 50 | -------------------------------------------------------------------------------- /r3/web/static/css/style.css: -------------------------------------------------------------------------------- 1 | ul { 2 | list-style: none; 3 | } 4 | 5 | body { 6 | font-family: arial; 7 | background-color: #4d7cb5; 8 | } 9 | 10 | .container { 11 | width: 95%; 12 | margin: 0 auto; 13 | } 14 | 15 | .body { 16 | background-color: #fff; 17 | padding-bottom: 20px; 18 | } 19 | 20 | header { 21 | height: 102px; 22 | overflow: hidden; 23 | } 24 | 25 | header a { 26 | display: block; 27 | float: left; 28 | } 29 | 30 | header a h1 { 31 | background: url(../img/logo.png) no-repeat; 32 | width: 162px; 33 | height: 62px; 34 | margin-top: 20px; 35 | margin-left: 40px; 36 | text-indent: -999999px; 37 | } 38 | 39 | header .menu { 40 | float: right; 41 | margin-top: 26px; 42 | } 43 | 44 | header nav { 45 | overflow: hidden; 46 | height: 18px; 47 | } 48 | 49 | header nav ul { 50 | float: right; 51 | } 52 | 53 | header nav li { 54 | padding: 0 8px; 55 | border-right: solid 1px #8a9db5; 56 | float: left; 57 | } 58 | 59 | header nav li:last-child { 60 | border-right: none; 61 | padding-right: 0; 62 | } 63 | 64 | header nav li a { 65 | color: #fff; 66 | font-weight: bold; 67 | text-decoration: none; 68 | } 69 | 70 | header nav li a:hover { 71 | text-decoration: underline; 72 | } 73 | 74 | header aside { 75 | text-align: right; 76 | margin-top: 12px; 77 | font-size: 13px; 78 | color: #c6cfd9; 79 | } 80 | 81 | header aside .running, 82 | header aside .active-workers, 83 | header aside .active-job-types, 84 | header aside .redis-url { 85 | font-weight: bold; 86 | color: #fff; 87 | } 88 | 89 | header aside p:first-child { 90 | margin-bottom: 4px; 91 | } 92 | 93 | h1 { 94 | font-size: 16px; 95 | font-weight: bold; 96 | color: #333; 97 | } 98 | 99 | .main .section { 100 | padding-top: 34px; 101 | padding-bottom: 15px; 102 | position: relative; 103 | } 104 | 105 | .main .section h1 { 106 | padding-bottom: 4px; 107 | /*margin-bottom: 20px;*/ 108 | border-bottom: solid 1px #d5d5d5; 109 | } 110 | 111 | .main .section .explain { 112 | position: absolute; 113 | right: 0; 114 | top: 34px; 115 | font-size: 13px; 116 | font-weight: bold; 117 | color: #999; 118 | } 119 | 120 | .main .current-jobs table { 121 | margin-left: 10px; 122 | width: 600px !important; 123 | } 124 | 125 | .main .current-jobs table .job-id-col { 126 | width: 250px; 127 | } 128 | 129 | .main .failed-jobs .filters { 130 | margin-top: 30px; 131 | } 132 | 133 | .main .mappers .filters { 134 | margin-top: 30px; 135 | } 136 | 137 | .main .failed-jobs table .job-id-col { 138 | width: 250px; 139 | text-align: center; 140 | } 141 | 142 | .main .failed-jobs table .job-date-col { 143 | width: 150px; 144 | text-align: center; 145 | } 146 | 147 | .main .failed-jobs table .job-type-col { 148 | width: 100px; 149 | text-align: center; 150 | } 151 | 152 | .main .failed-jobs table .job-actions-col { 153 | width: 80px; 154 | text-align: center; 155 | } 156 | 157 | .main .failed-jobs .actions { 158 | float: left; 159 | margin-bottom: 15px; 160 | margin-top: 15px; 161 | } 162 | 163 | .main .stats .actions { 164 | margin-top: 15px; 165 | margin-bottom: 10px; 166 | } 167 | 168 | .main .stats .delete { 169 | width: 70px; 170 | text-align: center; 171 | } 172 | 173 | .main .stats .delete a { 174 | color: #fff; 175 | font-size: 11px; 176 | width: auto; 177 | } 178 | 179 | .main .stats .delete a:hover { 180 | text-decoration: none; 181 | } 182 | 183 | .main .job pre { 184 | margin-top: 15px; 185 | } 186 | 187 | .main .section table a { 188 | font-size: 13px; 189 | text-decoration: none; 190 | color: #4d7cb5; 191 | } 192 | 193 | .main .section table a:hover { 194 | text-decoration: underline; 195 | } 196 | 197 | .main .section h2 { 198 | margin-top: 15px; 199 | } 200 | 201 | .main .section h2 { 202 | font-size: 12px; 203 | font-weight: bold; 204 | margin-bottom: 10px; 205 | color: #333; 206 | text-decoration: none; 207 | display: inline-block; 208 | } 209 | 210 | .main .stats h2 { 211 | margin-bottom: 10px; 212 | } 213 | 214 | .main .section h3 { 215 | font-size: 11px; 216 | color: #999; 217 | display: block; 218 | margin-top: -8px; 219 | margin-bottom: 10px; 220 | } 221 | 222 | .main .filters { 223 | float: right; 224 | margin-bottom: 4px; 225 | } 226 | 227 | .main .filters li { 228 | float: left; 229 | margin-right: 6px; 230 | } 231 | 232 | .main .filters li a { 233 | font-size: 11px; 234 | font-weight: bold; 235 | color: #4d7cb5; 236 | text-decoration: none; 237 | display: block; 238 | } 239 | 240 | .main .filters li a:hover { 241 | text-decoration: underline; 242 | } 243 | 244 | table tr th { 245 | font-size: 11px; 246 | background-color: #4d7cb5; 247 | color: white; 248 | } 249 | 250 | table tbody tr td { 251 | font-size: 13px; 252 | vertical-align: middle !important; 253 | } 254 | 255 | footer { 256 | width: 100%; 257 | height: 100%; 258 | background-color: #4d7cb5; 259 | text-align: center; 260 | overflow: hidden; 261 | display: table; 262 | } 263 | 264 | footer .cell { 265 | display: table-cell; 266 | vertical-align: middle; 267 | text-align: center; 268 | } 269 | 270 | footer a { 271 | color: #fff; 272 | font-weight: bold; 273 | text-decoration: none; 274 | font-size: 11px; 275 | display: block; 276 | margin: 20px 0; 277 | } 278 | 279 | footer a:hover { 280 | text-decoration: underline; 281 | } 282 | 283 | .table { 284 | margin-bottom: 0 !important; 285 | } 286 | 287 | .notice { 288 | padding-top: 15px; 289 | } 290 | 291 | .notice .alert { 292 | width: 75%; 293 | margin: 0 auto; 294 | } 295 | 296 | .notice .alert .close { 297 | text-decoration: none; 298 | } 299 | 300 | .stats .redis { 301 | width: auto; 302 | } 303 | 304 | .stats .redis .key { 305 | width: 190px; 306 | } 307 | 308 | .stats .redis .value { 309 | min-width: 300px; 310 | } 311 | 312 | .stats .keys .no-keys { 313 | height: 90px; 314 | vertical-align: middle; 315 | text-align: center; 316 | font-size: 28px; 317 | font-weight: bold; 318 | color: #aaa; 319 | } 320 | 321 | .stats .nav { 322 | margin-top: 10px; 323 | margin-bottom: 10px; 324 | } 325 | 326 | .stats .nav a { 327 | font-size: 13px; 328 | text-decoration: none; 329 | color: #4d7cb5; 330 | } 331 | 332 | .stats .nav a:hover { 333 | text-decoration: underline; 334 | } 335 | 336 | .alert { 337 | position: relative; 338 | font-size: 13px; 339 | font-weight: bold; 340 | } 341 | 342 | .alert .failed { 343 | text-decoration: none; 344 | color: #4d7cb5; 345 | } 346 | 347 | .alert .failed:hover { 348 | text-decoration: underline; 349 | } 350 | 351 | .alert .close { 352 | position: absolute; 353 | top: 6px; 354 | right: 10px; 355 | color: #333; 356 | } 357 | 358 | .alert .close:hover { 359 | color: #999; 360 | } 361 | 362 | table.no-jobs { 363 | margin-top: 20px; 364 | } 365 | 366 | .no-jobs, 367 | .no-fails, 368 | .no-mappers { 369 | height: 90px; 370 | vertical-align: middle !important; 371 | text-align: center !important; 372 | font-size: 28px; 373 | font-weight: bold; 374 | color: #aaa; 375 | } 376 | 377 | .mappers-table .mapper-name { 378 | width: 200px; 379 | } 380 | 381 | .main .failed-jobs table a { 382 | color: #fff; 383 | } 384 | 385 | .main .failed-jobs .actions a { 386 | color: #fff; 387 | text-decoration: none; 388 | font-weight: bold; 389 | } 390 | 391 | .main .show-key a { 392 | text-decoration: none; 393 | color: #4d7cb5; 394 | font-weight: bold; 395 | } 396 | 397 | .main .show-key a:hover { 398 | text-decoration: underline; 399 | } 400 | 401 | .main .show-key h1 { 402 | margin-bottom: 15px; 403 | } 404 | -------------------------------------------------------------------------------- /r3/web/static/img/logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/heynemann/r3/cc6b4eb55c7ae30a8f75af2be165504565dbeb79/r3/web/static/img/logo.png -------------------------------------------------------------------------------- /r3/web/static/js/bootstrap.min.js: -------------------------------------------------------------------------------- 1 | /** 2 | * Bootstrap.js by @fat & @mdo 3 | * plugins: bootstrap-transition.js, bootstrap-modal.js, bootstrap-dropdown.js, bootstrap-scrollspy.js, bootstrap-tab.js, bootstrap-tooltip.js, bootstrap-popover.js, bootstrap-alert.js, bootstrap-button.js, bootstrap-collapse.js, bootstrap-carousel.js, bootstrap-typeahead.js 4 | * Copyright 2012 Twitter, Inc. 5 | * http://www.apache.org/licenses/LICENSE-2.0.txt 6 | */ 7 | !function(a){a(function(){a.support.transition=function(){var a=function(){var a=document.createElement("bootstrap"),b={WebkitTransition:"webkitTransitionEnd",MozTransition:"transitionend",OTransition:"oTransitionEnd",msTransition:"MSTransitionEnd",transition:"transitionend"},c;for(c in b)if(a.style[c]!==undefined)return b[c]}();return a&&{end:a}}()})}(window.jQuery),!function(a){function c(){var b=this,c=setTimeout(function(){b.$element.off(a.support.transition.end),d.call(b)},500);this.$element.one(a.support.transition.end,function(){clearTimeout(c),d.call(b)})}function d(a){this.$element.hide().trigger("hidden"),e.call(this)}function e(b){var c=this,d=this.$element.hasClass("fade")?"fade":"";if(this.isShown&&this.options.backdrop){var e=a.support.transition&&d;this.$backdrop=a('
').appendTo(document.body),this.options.backdrop!="static"&&this.$backdrop.click(a.proxy(this.hide,this)),e&&this.$backdrop[0].offsetWidth,this.$backdrop.addClass("in"),e?this.$backdrop.one(a.support.transition.end,b):b()}else!this.isShown&&this.$backdrop?(this.$backdrop.removeClass("in"),a.support.transition&&this.$element.hasClass("fade")?this.$backdrop.one(a.support.transition.end,a.proxy(f,this)):f.call(this)):b&&b()}function f(){this.$backdrop.remove(),this.$backdrop=null}function g(){var b=this;this.isShown&&this.options.keyboard?a(document).on("keyup.dismiss.modal",function(a){a.which==27&&b.hide()}):this.isShown||a(document).off("keyup.dismiss.modal")}var b=function(b,c){this.options=c,this.$element=a(b).delegate('[data-dismiss="modal"]',"click.dismiss.modal",a.proxy(this.hide,this))};b.prototype={constructor:b,toggle:function(){return this[this.isShown?"hide":"show"]()},show:function(){var b=this,c=a.Event("show");this.$element.trigger(c);if(this.isShown||c.isDefaultPrevented())return;a("body").addClass("modal-open"),this.isShown=!0,g.call(this),e.call(this,function(){var c=a.support.transition&&b.$element.hasClass("fade");b.$element.parent().length||b.$element.appendTo(document.body),b.$element.show(),c&&b.$element[0].offsetWidth,b.$element.addClass("in"),c?b.$element.one(a.support.transition.end,function(){b.$element.trigger("shown")}):b.$element.trigger("shown")})},hide:function(b){b&&b.preventDefault();var e=this;b=a.Event("hide"),this.$element.trigger(b);if(!this.isShown||b.isDefaultPrevented())return;this.isShown=!1,a("body").removeClass("modal-open"),g.call(this),this.$element.removeClass("in"),a.support.transition&&this.$element.hasClass("fade")?c.call(this):d.call(this)}},a.fn.modal=function(c){return this.each(function(){var d=a(this),e=d.data("modal"),f=a.extend({},a.fn.modal.defaults,d.data(),typeof c=="object"&&c);e||d.data("modal",e=new b(this,f)),typeof c=="string"?e[c]():f.show&&e.show()})},a.fn.modal.defaults={backdrop:!0,keyboard:!0,show:!0},a.fn.modal.Constructor=b,a(function(){a("body").on("click.modal.data-api",'[data-toggle="modal"]',function(b){var c=a(this),d,e=a(c.attr("data-target")||(d=c.attr("href"))&&d.replace(/.*(?=#[^\s]+$)/,"")),f=e.data("modal")?"toggle":a.extend({},e.data(),c.data());b.preventDefault(),e.modal(f)})})}(window.jQuery),!function(a){function d(){a(b).parent().removeClass("open")}var b='[data-toggle="dropdown"]',c=function(b){var c=a(b).on("click.dropdown.data-api",this.toggle);a("html").on("click.dropdown.data-api",function(){c.parent().removeClass("open")})};c.prototype={constructor:c,toggle:function(b){var c=a(this),e,f,g;if(c.is(".disabled, :disabled"))return;return f=c.attr("data-target"),f||(f=c.attr("href"),f=f&&f.replace(/.*(?=#[^\s]*$)/,"")),e=a(f),e.length||(e=c.parent()),g=e.hasClass("open"),d(),g||e.toggleClass("open"),!1}},a.fn.dropdown=function(b){return this.each(function(){var d=a(this),e=d.data("dropdown");e||d.data("dropdown",e=new c(this)),typeof b=="string"&&e[b].call(d)})},a.fn.dropdown.Constructor=c,a(function(){a("html").on("click.dropdown.data-api",d),a("body").on("click.dropdown",".dropdown form",function(a){a.stopPropagation()}).on("click.dropdown.data-api",b,c.prototype.toggle)})}(window.jQuery),!function(a){function b(b,c){var d=a.proxy(this.process,this),e=a(b).is("body")?a(window):a(b),f;this.options=a.extend({},a.fn.scrollspy.defaults,c),this.$scrollElement=e.on("scroll.scroll.data-api",d),this.selector=(this.options.target||(f=a(b).attr("href"))&&f.replace(/.*(?=#[^\s]+$)/,"")||"")+" .nav li > a",this.$body=a("body"),this.refresh(),this.process()}b.prototype={constructor:b,refresh:function(){var b=this,c;this.offsets=a([]),this.targets=a([]),c=this.$body.find(this.selector).map(function(){var b=a(this),c=b.data("target")||b.attr("href"),d=/^#\w/.test(c)&&a(c);return d&&c.length&&[[d.position().top,c]]||null}).sort(function(a,b){return a[0]-b[0]}).each(function(){b.offsets.push(this[0]),b.targets.push(this[1])})},process:function(){var a=this.$scrollElement.scrollTop()+this.options.offset,b=this.$scrollElement[0].scrollHeight||this.$body[0].scrollHeight,c=b-this.$scrollElement.height(),d=this.offsets,e=this.targets,f=this.activeTarget,g;if(a>=c)return f!=(g=e.last()[0])&&this.activate(g);for(g=d.length;g--;)f!=e[g]&&a>=d[g]&&(!d[g+1]||a<=d[g+1])&&this.activate(e[g])},activate:function(b){var c,d;this.activeTarget=b,a(this.selector).parent(".active").removeClass("active"),d=this.selector+'[data-target="'+b+'"],'+this.selector+'[href="'+b+'"]',c=a(d).parent("li").addClass("active"),c.parent(".dropdown-menu")&&(c=c.closest("li.dropdown").addClass("active")),c.trigger("activate")}},a.fn.scrollspy=function(c){return this.each(function(){var d=a(this),e=d.data("scrollspy"),f=typeof c=="object"&&c;e||d.data("scrollspy",e=new b(this,f)),typeof c=="string"&&e[c]()})},a.fn.scrollspy.Constructor=b,a.fn.scrollspy.defaults={offset:10},a(function(){a('[data-spy="scroll"]').each(function(){var b=a(this);b.scrollspy(b.data())})})}(window.jQuery),!function(a){var b=function(b){this.element=a(b)};b.prototype={constructor:b,show:function(){var b=this.element,c=b.closest("ul:not(.dropdown-menu)"),d=b.attr("data-target"),e,f,g;d||(d=b.attr("href"),d=d&&d.replace(/.*(?=#[^\s]*$)/,""));if(b.parent("li").hasClass("active"))return;e=c.find(".active a").last()[0],g=a.Event("show",{relatedTarget:e}),b.trigger(g);if(g.isDefaultPrevented())return;f=a(d),this.activate(b.parent("li"),c),this.activate(f,f.parent(),function(){b.trigger({type:"shown",relatedTarget:e})})},activate:function(b,c,d){function g(){e.removeClass("active").find("> .dropdown-menu > .active").removeClass("active"),b.addClass("active"),f?(b[0].offsetWidth,b.addClass("in")):b.removeClass("fade"),b.parent(".dropdown-menu")&&b.closest("li.dropdown").addClass("active"),d&&d()}var e=c.find("> .active"),f=d&&a.support.transition&&e.hasClass("fade");f?e.one(a.support.transition.end,g):g(),e.removeClass("in")}},a.fn.tab=function(c){return this.each(function(){var d=a(this),e=d.data("tab");e||d.data("tab",e=new b(this)),typeof c=="string"&&e[c]()})},a.fn.tab.Constructor=b,a(function(){a("body").on("click.tab.data-api",'[data-toggle="tab"], [data-toggle="pill"]',function(b){b.preventDefault(),a(this).tab("show")})})}(window.jQuery),!function(a){var b=function(a,b){this.init("tooltip",a,b)};b.prototype={constructor:b,init:function(b,c,d){var e,f;this.type=b,this.$element=a(c),this.options=this.getOptions(d),this.enabled=!0,this.options.trigger!="manual"&&(e=this.options.trigger=="hover"?"mouseenter":"focus",f=this.options.trigger=="hover"?"mouseleave":"blur",this.$element.on(e,this.options.selector,a.proxy(this.enter,this)),this.$element.on(f,this.options.selector,a.proxy(this.leave,this))),this.options.selector?this._options=a.extend({},this.options,{trigger:"manual",selector:""}):this.fixTitle()},getOptions:function(b){return b=a.extend({},a.fn[this.type].defaults,b,this.$element.data()),b.delay&&typeof b.delay=="number"&&(b.delay={show:b.delay,hide:b.delay}),b},enter:function(b){var c=a(b.currentTarget)[this.type](this._options).data(this.type);if(!c.options.delay||!c.options.delay.show)return c.show();clearTimeout(this.timeout),c.hoverState="in",this.timeout=setTimeout(function(){c.hoverState=="in"&&c.show()},c.options.delay.show)},leave:function(b){var c=a(b.currentTarget)[this.type](this._options).data(this.type);this.timeout&&clearTimeout(this.timeout);if(!c.options.delay||!c.options.delay.hide)return c.hide();c.hoverState="out",this.timeout=setTimeout(function(){c.hoverState=="out"&&c.hide()},c.options.delay.hide)},show:function(){var a,b,c,d,e,f,g;if(this.hasContent()&&this.enabled){a=this.tip(),this.setContent(),this.options.animation&&a.addClass("fade"),f=typeof this.options.placement=="function"?this.options.placement.call(this,a[0],this.$element[0]):this.options.placement,b=/in/.test(f),a.remove().css({top:0,left:0,display:"block"}).appendTo(b?this.$element:document.body),c=this.getPosition(b),d=a[0].offsetWidth,e=a[0].offsetHeight;switch(b?f.split(" ")[1]:f){case"bottom":g={top:c.top+c.height,left:c.left+c.width/2-d/2};break;case"top":g={top:c.top-e,left:c.left+c.width/2-d/2};break;case"left":g={top:c.top+c.height/2-e/2,left:c.left-d};break;case"right":g={top:c.top+c.height/2-e/2,left:c.left+c.width}}a.css(g).addClass(f).addClass("in")}},isHTML:function(a){return typeof a!="string"||a.charAt(0)==="<"&&a.charAt(a.length-1)===">"&&a.length>=3||/^(?:[^<]*<[\w\W]+>[^>]*$)/.exec(a)},setContent:function(){var a=this.tip(),b=this.getTitle();a.find(".tooltip-inner")[this.isHTML(b)?"html":"text"](b),a.removeClass("fade in top bottom left right")},hide:function(){function d(){var b=setTimeout(function(){c.off(a.support.transition.end).remove()},500);c.one(a.support.transition.end,function(){clearTimeout(b),c.remove()})}var b=this,c=this.tip();c.removeClass("in"),a.support.transition&&this.$tip.hasClass("fade")?d():c.remove()},fixTitle:function(){var a=this.$element;(a.attr("title")||typeof a.attr("data-original-title")!="string")&&a.attr("data-original-title",a.attr("title")||"").removeAttr("title")},hasContent:function(){return this.getTitle()},getPosition:function(b){return a.extend({},b?{top:0,left:0}:this.$element.offset(),{width:this.$element[0].offsetWidth,height:this.$element[0].offsetHeight})},getTitle:function(){var a,b=this.$element,c=this.options;return a=b.attr("data-original-title")||(typeof c.title=="function"?c.title.call(b[0]):c.title),a},tip:function(){return this.$tip=this.$tip||a(this.options.template)},validate:function(){this.$element[0].parentNode||(this.hide(),this.$element=null,this.options=null)},enable:function(){this.enabled=!0},disable:function(){this.enabled=!1},toggleEnabled:function(){this.enabled=!this.enabled},toggle:function(){this[this.tip().hasClass("in")?"hide":"show"]()}},a.fn.tooltip=function(c){return this.each(function(){var d=a(this),e=d.data("tooltip"),f=typeof c=="object"&&c;e||d.data("tooltip",e=new b(this,f)),typeof c=="string"&&e[c]()})},a.fn.tooltip.Constructor=b,a.fn.tooltip.defaults={animation:!0,placement:"top",selector:!1,template:'
',trigger:"hover",title:"",delay:0}}(window.jQuery),!function(a){var b=function(a,b){this.init("popover",a,b)};b.prototype=a.extend({},a.fn.tooltip.Constructor.prototype,{constructor:b,setContent:function(){var a=this.tip(),b=this.getTitle(),c=this.getContent();a.find(".popover-title")[this.isHTML(b)?"html":"text"](b),a.find(".popover-content > *")[this.isHTML(c)?"html":"text"](c),a.removeClass("fade top bottom left right in")},hasContent:function(){return this.getTitle()||this.getContent()},getContent:function(){var a,b=this.$element,c=this.options;return a=b.attr("data-content")||(typeof c.content=="function"?c.content.call(b[0]):c.content),a},tip:function(){return this.$tip||(this.$tip=a(this.options.template)),this.$tip}}),a.fn.popover=function(c){return this.each(function(){var d=a(this),e=d.data("popover"),f=typeof c=="object"&&c;e||d.data("popover",e=new b(this,f)),typeof c=="string"&&e[c]()})},a.fn.popover.Constructor=b,a.fn.popover.defaults=a.extend({},a.fn.tooltip.defaults,{placement:"right",content:"",template:'

'})}(window.jQuery),!function(a){var b='[data-dismiss="alert"]',c=function(c){a(c).on("click",b,this.close)};c.prototype.close=function(b){function f(){e.trigger("closed").remove()}var c=a(this),d=c.attr("data-target"),e;d||(d=c.attr("href"),d=d&&d.replace(/.*(?=#[^\s]*$)/,"")),e=a(d),b&&b.preventDefault(),e.length||(e=c.hasClass("alert")?c:c.parent()),e.trigger(b=a.Event("close"));if(b.isDefaultPrevented())return;e.removeClass("in"),a.support.transition&&e.hasClass("fade")?e.on(a.support.transition.end,f):f()},a.fn.alert=function(b){return this.each(function(){var d=a(this),e=d.data("alert");e||d.data("alert",e=new c(this)),typeof b=="string"&&e[b].call(d)})},a.fn.alert.Constructor=c,a(function(){a("body").on("click.alert.data-api",b,c.prototype.close)})}(window.jQuery),!function(a){var b=function(b,c){this.$element=a(b),this.options=a.extend({},a.fn.button.defaults,c)};b.prototype.setState=function(a){var b="disabled",c=this.$element,d=c.data(),e=c.is("input")?"val":"html";a+="Text",d.resetText||c.data("resetText",c[e]()),c[e](d[a]||this.options[a]),setTimeout(function(){a=="loadingText"?c.addClass(b).attr(b,b):c.removeClass(b).removeAttr(b)},0)},b.prototype.toggle=function(){var a=this.$element.parent('[data-toggle="buttons-radio"]');a&&a.find(".active").removeClass("active"),this.$element.toggleClass("active")},a.fn.button=function(c){return this.each(function(){var d=a(this),e=d.data("button"),f=typeof c=="object"&&c;e||d.data("button",e=new b(this,f)),c=="toggle"?e.toggle():c&&e.setState(c)})},a.fn.button.defaults={loadingText:"loading..."},a.fn.button.Constructor=b,a(function(){a("body").on("click.button.data-api","[data-toggle^=button]",function(b){var c=a(b.target);c.hasClass("btn")||(c=c.closest(".btn")),c.button("toggle")})})}(window.jQuery),!function(a){var b=function(b,c){this.$element=a(b),this.options=a.extend({},a.fn.collapse.defaults,c),this.options.parent&&(this.$parent=a(this.options.parent)),this.options.toggle&&this.toggle()};b.prototype={constructor:b,dimension:function(){var a=this.$element.hasClass("width");return a?"width":"height"},show:function(){var b,c,d,e;if(this.transitioning)return;b=this.dimension(),c=a.camelCase(["scroll",b].join("-")),d=this.$parent&&this.$parent.find("> .accordion-group > .in");if(d&&d.length){e=d.data("collapse");if(e&&e.transitioning)return;d.collapse("hide"),e||d.data("collapse",null)}this.$element[b](0),this.transition("addClass",a.Event("show"),"shown"),this.$element[b](this.$element[0][c])},hide:function(){var b;if(this.transitioning)return;b=this.dimension(),this.reset(this.$element[b]()),this.transition("removeClass",a.Event("hide"),"hidden"),this.$element[b](0)},reset:function(a){var b=this.dimension();return this.$element.removeClass("collapse")[b](a||"auto")[0].offsetWidth,this.$element[a!==null?"addClass":"removeClass"]("collapse"),this},transition:function(b,c,d){var e=this,f=function(){c.type=="show"&&e.reset(),e.transitioning=0,e.$element.trigger(d)};this.$element.trigger(c);if(c.isDefaultPrevented())return;this.transitioning=1,this.$element[b]("in"),a.support.transition&&this.$element.hasClass("collapse")?this.$element.one(a.support.transition.end,f):f()},toggle:function(){this[this.$element.hasClass("in")?"hide":"show"]()}},a.fn.collapse=function(c){return this.each(function(){var d=a(this),e=d.data("collapse"),f=typeof c=="object"&&c;e||d.data("collapse",e=new b(this,f)),typeof c=="string"&&e[c]()})},a.fn.collapse.defaults={toggle:!0},a.fn.collapse.Constructor=b,a(function(){a("body").on("click.collapse.data-api","[data-toggle=collapse]",function(b){var c=a(this),d,e=c.attr("data-target")||b.preventDefault()||(d=c.attr("href"))&&d.replace(/.*(?=#[^\s]+$)/,""),f=a(e).data("collapse")?"toggle":c.data();a(e).collapse(f)})})}(window.jQuery),!function(a){var b=function(b,c){this.$element=a(b),this.options=c,this.options.slide&&this.slide(this.options.slide),this.options.pause=="hover"&&this.$element.on("mouseenter",a.proxy(this.pause,this)).on("mouseleave",a.proxy(this.cycle,this))};b.prototype={cycle:function(b){return b||(this.paused=!1),this.options.interval&&!this.paused&&(this.interval=setInterval(a.proxy(this.next,this),this.options.interval)),this},to:function(b){var c=this.$element.find(".active"),d=c.parent().children(),e=d.index(c),f=this;if(b>d.length-1||b<0)return;return this.sliding?this.$element.one("slid",function(){f.to(b)}):e==b?this.pause().cycle():this.slide(b>e?"next":"prev",a(d[b]))},pause:function(a){return a||(this.paused=!0),clearInterval(this.interval),this.interval=null,this},next:function(){if(this.sliding)return;return this.slide("next")},prev:function(){if(this.sliding)return;return this.slide("prev")},slide:function(b,c){var d=this.$element.find(".active"),e=c||d[b](),f=this.interval,g=b=="next"?"left":"right",h=b=="next"?"first":"last",i=this,j=a.Event("slide");this.sliding=!0,f&&this.pause(),e=e.length?e:this.$element.find(".item")[h]();if(e.hasClass("active"))return;if(a.support.transition&&this.$element.hasClass("slide")){this.$element.trigger(j);if(j.isDefaultPrevented())return;e.addClass(b),e[0].offsetWidth,d.addClass(g),e.addClass(g),this.$element.one(a.support.transition.end,function(){e.removeClass([b,g].join(" ")).addClass("active"),d.removeClass(["active",g].join(" ")),i.sliding=!1,setTimeout(function(){i.$element.trigger("slid")},0)})}else{this.$element.trigger(j);if(j.isDefaultPrevented())return;d.removeClass("active"),e.addClass("active"),this.sliding=!1,this.$element.trigger("slid")}return f&&this.cycle(),this}},a.fn.carousel=function(c){return this.each(function(){var d=a(this),e=d.data("carousel"),f=a.extend({},a.fn.carousel.defaults,typeof c=="object"&&c);e||d.data("carousel",e=new b(this,f)),typeof c=="number"?e.to(c):typeof c=="string"||(c=f.slide)?e[c]():f.interval&&e.cycle()})},a.fn.carousel.defaults={interval:5e3,pause:"hover"},a.fn.carousel.Constructor=b,a(function(){a("body").on("click.carousel.data-api","[data-slide]",function(b){var c=a(this),d,e=a(c.attr("data-target")||(d=c.attr("href"))&&d.replace(/.*(?=#[^\s]+$)/,"")),f=!e.data("modal")&&a.extend({},e.data(),c.data());e.carousel(f),b.preventDefault()})})}(window.jQuery),!function(a){var b=function(b,c){this.$element=a(b),this.options=a.extend({},a.fn.typeahead.defaults,c),this.matcher=this.options.matcher||this.matcher,this.sorter=this.options.sorter||this.sorter,this.highlighter=this.options.highlighter||this.highlighter,this.updater=this.options.updater||this.updater,this.$menu=a(this.options.menu).appendTo("body"),this.source=this.options.source,this.shown=!1,this.listen()};b.prototype={constructor:b,select:function(){var a=this.$menu.find(".active").attr("data-value");return this.$element.val(this.updater(a)).change(),this.hide()},updater:function(a){return a},show:function(){var b=a.extend({},this.$element.offset(),{height:this.$element[0].offsetHeight});return this.$menu.css({top:b.top+b.height,left:b.left}),this.$menu.show(),this.shown=!0,this},hide:function(){return this.$menu.hide(),this.shown=!1,this},lookup:function(b){var c=this,d,e;return this.query=this.$element.val(),this.query?(d=a.grep(this.source,function(a){return c.matcher(a)}),d=this.sorter(d),d.length?this.render(d.slice(0,this.options.items)).show():this.shown?this.hide():this):this.shown?this.hide():this},matcher:function(a){return~a.toLowerCase().indexOf(this.query.toLowerCase())},sorter:function(a){var b=[],c=[],d=[],e;while(e=a.shift())e.toLowerCase().indexOf(this.query.toLowerCase())?~e.indexOf(this.query)?c.push(e):d.push(e):b.push(e);return b.concat(c,d)},highlighter:function(a){var b=this.query.replace(/[\-\[\]{}()*+?.,\\\^$|#\s]/g,"\\$&");return a.replace(new RegExp("("+b+")","ig"),function(a,b){return""+b+""})},render:function(b){var c=this;return b=a(b).map(function(b,d){return b=a(c.options.item).attr("data-value",d),b.find("a").html(c.highlighter(d)),b[0]}),b.first().addClass("active"),this.$menu.html(b),this},next:function(b){var c=this.$menu.find(".active").removeClass("active"),d=c.next();d.length||(d=a(this.$menu.find("li")[0])),d.addClass("active")},prev:function(a){var b=this.$menu.find(".active").removeClass("active"),c=b.prev();c.length||(c=this.$menu.find("li").last()),c.addClass("active")},listen:function(){this.$element.on("blur",a.proxy(this.blur,this)).on("keypress",a.proxy(this.keypress,this)).on("keyup",a.proxy(this.keyup,this)),(a.browser.webkit||a.browser.msie)&&this.$element.on("keydown",a.proxy(this.keypress,this)),this.$menu.on("click",a.proxy(this.click,this)).on("mouseenter","li",a.proxy(this.mouseenter,this))},keyup:function(a){switch(a.keyCode){case 40:case 38:break;case 9:case 13:if(!this.shown)return;this.select();break;case 27:if(!this.shown)return;this.hide();break;default:this.lookup()}a.stopPropagation(),a.preventDefault()},keypress:function(a){if(!this.shown)return;switch(a.keyCode){case 9:case 13:case 27:a.preventDefault();break;case 38:if(a.type!="keydown")break;a.preventDefault(),this.prev();break;case 40:if(a.type!="keydown")break;a.preventDefault(),this.next()}a.stopPropagation()},blur:function(a){var b=this;setTimeout(function(){b.hide()},150)},click:function(a){a.stopPropagation(),a.preventDefault(),this.select()},mouseenter:function(b){this.$menu.find(".active").removeClass("active"),a(b.currentTarget).addClass("active")}},a.fn.typeahead=function(c){return this.each(function(){var d=a(this),e=d.data("typeahead"),f=typeof c=="object"&&c;e||d.data("typeahead",e=new b(this,f)),typeof c=="string"&&e[c]()})},a.fn.typeahead.defaults={source:[],items:8,menu:'
    ',item:'
    • '},a.fn.typeahead.Constructor=b,a(function(){a("body").on("focus.typeahead.data-api",'[data-provide="typeahead"]',function(b){var c=a(this);if(c.data("typeahead"))return;b.preventDefault(),c.typeahead(c.data())})})}(window.jQuery) -------------------------------------------------------------------------------- /r3/web/static/js/progress.js: -------------------------------------------------------------------------------- 1 | $(function() { 2 | $(".meter > span").each(function() { 3 | $(this) 4 | .data("origWidth", $(this).width()) 5 | .width(0) 6 | .animate({ 7 | width: $(this).data("origWidth") 8 | }, 1200); 9 | }); 10 | }); 11 | 12 | -------------------------------------------------------------------------------- /r3/web/static/js/tabs.js: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /r3/web/templates/failed.html: -------------------------------------------------------------------------------- 1 | {% extends "master.html" %} 2 | 3 | {% block title %} - Failed{% endblock %} 4 | 5 | {% block body %} 6 |
    7 |
    8 |

    Failed Jobs

    9 | 10 |
    11 |
      12 |
    • all
      • 13 | {% for job_type in g.job_types %} 14 |
    • {{ job_type }}
      • 15 | {% endfor %} 16 |
    17 |
    18 | 19 |
    20 | delete all 21 |
    22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | {% for error in errors %} 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | {% else %} 43 | 44 | 45 | 46 | {% endfor %} 47 | 48 |
    Job TypeJob IDDateMessage
    {{ error['job_key'] }}{{ error['job_id'] }}{{ error['date'] }}{{ error['error'] }}delete
    No errors happened so far. Yay!!!
    49 | 50 | click retry or delete to change a failed job 51 |
    52 | 53 |
    54 | {% endblock %} 55 | -------------------------------------------------------------------------------- /r3/web/templates/index.html: -------------------------------------------------------------------------------- 1 | {% extends "master.html" %} 2 | 3 | {% block title %} - Overview{% endblock %} 4 | 5 | {% block body %} 6 | 7 | {% if failed_warning %} 8 |
    9 |
    10 | × 11 | Some messages have failed. Click here to take action. 12 |
    13 |
    14 | {% endif %} 15 | 16 |
    17 |
    18 |

    Jobs in Progress

    19 | 20 | {% for job_type in g.job_types %} 21 |

    {{ job_type }}

    22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | {% for job in g.jobs[job_type] %} 31 | 32 | 33 | 34 | 35 | {% else %} 36 | 37 | 38 | 39 | {% endfor %} 40 | 41 |
    Job IDPhase
    {{ job }}Mapping
    Nothing happening here right now...
    42 | {% else %} 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 |
    Job IDPhase
    Nothing happening here right now...
    56 | {% endfor %} 57 | 58 | click a job type to view its activity 59 |
    60 | 61 |
    62 |

    Mappers

    63 | 64 |
    65 |
      66 |
    • all
      • 67 | {% for job_type in g.job_types %} 68 |
    • {{ job_type }}
      • 69 | {% endfor %} 70 |
    71 |
    72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | {% for mapper, status in g.mappers.iteritems() %} 82 | 83 | 84 | {% if status %} 85 | 86 | {% else %} 87 | 88 | {% endif %} 89 | 90 | {% else %} 91 | 92 | 93 | 94 | {% endfor %} 95 | 96 |
    NameWorking on
    {{ mapper }}Processing job {{ status['job_id'] }}Waiting for a new job...
    No mappers registered so far...
    97 | 98 | click a mapper to view its activity 99 |
    100 |
    101 | {% endblock %} 102 | -------------------------------------------------------------------------------- /r3/web/templates/job-types.html: -------------------------------------------------------------------------------- 1 | {% extends "master.html" %} 2 | 3 | {% block title %} - Overview{% endblock %} 4 | 5 | {% block body %} 6 |
    7 |
    8 |

    Jobs in Progress

    9 | 10 | {% for job_type in g.job_types %} 11 |

    {{ job_type }}

    12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | {% for job in g.jobs[job_type] %} 21 | 22 | 23 | 24 | 25 | {% else %} 26 | 27 | 28 | 29 | {% endfor %} 30 | 31 |
    Job IDPhase
    {{ job }}Mapping
    Nothing happening here right now...
    32 | {% else %} 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 |
    Job IDPhase
    Nothing happening here right now...
    46 | {% endfor %} 47 | 48 | click a job type to view its activity 49 |
    50 |
    51 | {% endblock %} 52 | -------------------------------------------------------------------------------- /r3/web/templates/mappers.html: -------------------------------------------------------------------------------- 1 | {% extends "master.html" %} 2 | 3 | {% block title %} - Mappers{% endblock %} 4 | 5 | {% block body %} 6 |
    7 |
    8 |

    Mappers

    9 | 10 |
    11 |
      12 |
    • all
      • 13 | {% for job_type in g.job_types %} 14 |
    • {{ job_type }}
      • 15 | {% endfor %} 16 |
    17 |
    18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | {% for mapper, status in g.mappers.iteritems() %} 28 | 29 | 30 | {% if status %} 31 | 32 | {% else %} 33 | 34 | {% endif %} 35 | 36 | {% else %} 37 | 38 | 39 | 40 | {% endfor %} 41 | 42 |
    NameWorking on
    {{ mapper }}Processing job {{ status }}Waiting for a new job...
    No mappers registered so far...
    43 | 44 | click a mapper to view its activity 45 |
    46 |
    47 | {% endblock %} 48 | -------------------------------------------------------------------------------- /r3/web/templates/master.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | r³{% block title %}{% endblock %} 6 | {% block css %} 7 | 8 | 9 | 10 | 11 | {% endblock %} 12 | 13 | 14 |
    15 |
    16 |

    r³ - map reduce service

     17 |
    18 | 27 | 31 | 32 |
    33 |
    34 |
    35 |
    36 | {% block body %} 37 | {% endblock %} 38 |
    39 | 40 | 45 | 46 | {% block js %} 47 | 48 | 49 | 50 | {% endblock %} 51 | 52 | 53 | -------------------------------------------------------------------------------- /r3/web/templates/show_key.html: -------------------------------------------------------------------------------- 1 | {% extends "master.html" %} 2 | 3 | {% block title %} - Overview{% endblock %} 4 | 5 | {% block css %} 6 | {{ super() }} 7 | 10 | {% endblock %} 11 | 12 | {% block body %} 13 |
    14 |
    15 |

    Statistics - {{ key }}

    16 | 17 | {% if multi %} 18 |
      19 | {% for v in value %} 20 | 
    • {{ v|safe }}
      • 21 | {% endfor %} 22 |
    23 | {% else %} 24 | 
    {{ value|safe }}
    25 | {% endif %} 26 |
    27 |
    28 | {% endblock %} 29 | -------------------------------------------------------------------------------- /r3/web/templates/stats.html: -------------------------------------------------------------------------------- 1 | {% extends "master.html" %} 2 | 3 | {% block title %} - Overview{% endblock %} 4 | 5 | {% block body %} 6 |
    7 |
    8 |

    Statistics

    9 | 10 |
    11 | 16 | 17 |
    18 |
    19 |

    r³ info

    20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 |
    job-types{{ g.job_types|count }}
    redis server{{ g.config['REDIS_HOST'] }}:{{ g.config['REDIS_PORT'] }}
    failed jobs{{ failed }}
    processed{{ processed }}
    mappers{{ g.mappers|count }}
    44 |
    45 | 46 |
    47 |

    keys owned by r³

    48 |

    (All keys are actually prefixed with "r3:")

    49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | {% for key in keys %} 62 | 63 | 64 | 65 | 66 | 67 | 68 | {% else %} 69 | 70 | 71 | 72 | {% endfor %} 73 | 74 |
    key nametypesize
    {{ key['name'] }}{{ key['type'] }}{{ key['size'] }}delete
    No keys owned by r³
    75 |
    76 | 77 |
    78 |

    redis.r3.globoi.com:20001

    79 | 80 | 81 | 82 | {% for key, value in info.items() %} 83 | 84 | 85 | 86 | 87 | {% endfor %} 88 | 89 |
    {{ key }}{{ value }}
    90 |
    91 |
    92 |
    93 |
    94 |
    95 | {% endblock %} 96 | -------------------------------------------------------------------------------- /r3/worker/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/heynemann/r3/cc6b4eb55c7ae30a8f75af2be165504565dbeb79/r3/worker/__init__.py -------------------------------------------------------------------------------- /r3/worker/mapper.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | from datetime import datetime 5 | import signal 6 | import logging 7 | import sys 8 | import os 9 | import argparse 10 | 11 | import redis 12 | from ujson import loads, dumps 13 | 14 | from r3.app.utils import DATETIME_FORMAT, kls_import 15 | from r3.app.keys import MAPPERS_KEY, JOB_TYPES_KEY, MAPPER_INPUT_KEY, MAPPER_WORKING_KEY, LAST_PING_KEY 16 | 17 | class JobError(RuntimeError): 18 | pass 19 | 20 | class CrashError(JobError): 21 | pass 22 | 23 | class TimeoutError(JobError): 24 | pass 25 | 26 | class Mapper: 27 | def __init__(self, job_type, mapper_key, redis_host, redis_port, redis_db, redis_pass): 28 | self.job_type = job_type 29 | self.mapper_key = mapper_key 30 | self.full_name = '%s::%s' % (self.job_type, self.mapper_key) 31 | self.timeout = None 32 | self.input_queue = MAPPER_INPUT_KEY % self.job_type 33 | self.working_queue = MAPPER_WORKING_KEY % self.full_name 34 | self.max_retries = 5 35 | 36 | self.redis = redis.StrictRedis(host=redis_host, port=redis_port, db=redis_db, password=redis_pass) 37 | 38 | logging.basicConfig(level=getattr(logging, 'WARNING')) 39 | self.initialize() 40 | logging.debug("Mapper UP - pid: %s" % os.getpid()) 41 | logging.debug("Input Q: %s" % self.input_queue) 42 | logging.debug("Working Q: %s" % self.working_queue) 43 | 44 | def handle_signal(self, number, stack): 45 | self.unregister() 46 | sys.exit(number) 47 | 48 | def initialize(self): 49 | signal.signal(signal.SIGTERM, self.handle_signal) 50 | self.ping() 51 | 52 | item = self.redis.rpop(self.working_queue) 53 | if item: 54 | json_item = loads(item) 55 | json_item['retries'] += 1 56 | 57 | if json_item['retries'] > self.max_retries: 58 | json_item['error'] = '%s errored out after %d retries.' % (self.full_name, json_item['retries']) 59 | self.redis.rpush(json_item['output_queue'], dumps(json_item)) 60 | else: 61 | item = dumps(json_item) 62 | self.map_item(item, json_item) 63 | 64 | def map(self): 65 | raise NotImplementedError() 66 | 67 | def run_block(self): 68 | try: 69 | while True: 70 | self.ping() 71 | logging.debug('waiting to process next item...') 72 | values = self.redis.brpop(self.input_queue, timeout=5) 73 | if values: 74 | key, item = values 75 | json_item = loads(item) 76 | self.map_item(item, json_item) 77 | finally: 78 | self.unregister() 79 | 80 | def unregister(self): 81 | self.redis.srem(MAPPERS_KEY, self.full_name) 82 | self.redis.delete(LAST_PING_KEY % self.full_name) 83 | 84 | def ping(self): 85 | self.redis.delete(MAPPER_WORKING_KEY % self.full_name) 86 | self.redis.sadd(JOB_TYPES_KEY, self.job_type) 87 | self.redis.sadd(MAPPERS_KEY, self.full_name) 88 | self.redis.set(LAST_PING_KEY % self.full_name, datetime.now().strftime(DATETIME_FORMAT)) 89 | 90 | def map_item(self, item, json_item): 91 | self.redis.set('r3::mappers::%s::working' % self.full_name, json_item['job_id']) 92 | self.redis.rpush(self.working_queue, item) 93 | result = dumps(self.map(json_item['item'])) 94 | self.redis.rpush(json_item['output_queue'], dumps({ 95 | 'result': result 96 | })) 97 | self.redis.delete(self.working_queue) 98 | self.redis.delete('r3::mappers::%s::working' % self.full_name) 99 | 100 | def main(arguments=None): 101 | if not arguments: 102 | arguments = sys.argv[1:] 103 | 104 | parser = argparse.ArgumentParser(description='runs the application that processes stream requests for r³') 105 | parser.add_argument('-l', '--loglevel', type=str, default='warning', help='the log level that r³ will run under') 106 | parser.add_argument('--redis-host', type=str, default='0.0.0.0', help='the ip that r³ will use to connect to redis') 107 | parser.add_argument('--redis-port', type=int, default=6379, help='the port that r³ will use to connect to redis') 108 | parser.add_argument('--redis-db', type=int, default=0, help='the database that r³ will use to connect to redis') 109 | parser.add_argument('--redis-pass', type=str, default='', help='the password that r³ will use to connect to redis') 110 | parser.add_argument('--mapper-key', type=str, help='the unique identifier for this mapper', required=True) 111 | parser.add_argument('--mapper-class', type=str, help='the fullname of the class that this mapper will run', required=True) 112 | 113 | args = parser.parse_args(arguments) 114 | 115 | if not args.mapper_key: 116 | raise RuntimeError('The --mapper_key argument is required.') 117 | 118 | logging.basicConfig(level=getattr(logging, args.loglevel.upper())) 119 | 120 | try: 121 | klass = kls_import(args.mapper_class) 122 | except Exception, err: 123 | print "Could not import the specified %s class. Error: %s" % (args.mapper_class, err) 124 | raise 125 | 126 | mapper = klass(klass.job_type, args.mapper_key, redis_host=args.redis_host, redis_port=args.redis_port, redis_db=args.redis_db, redis_pass=args.redis_pass) 127 | try: 128 | mapper.run_block() 129 | except KeyboardInterrupt: 130 | print 131 | print "-- r³ mapper closed by user interruption --" 132 | 133 | 134 | if __name__ == '__main__': 135 | main(sys.argv[1:]) 136 | -------------------------------------------------------------------------------- /redis.conf: -------------------------------------------------------------------------------- 1 | # Redis configuration file example 2 | 3 | # Note on units: when memory size is needed, it is possible to specify 4 | # it in the usual form of 1k 5GB 4M and so forth: 5 | # 6 | # 1k => 1000 bytes 7 | # 1kb => 1024 bytes 8 | # 1m => 1000000 bytes 9 | # 1mb => 1024*1024 bytes 10 | # 1g => 1000000000 bytes 11 | # 1gb => 1024*1024*1024 bytes 12 | # 13 | # units are case insensitive so 1GB 1Gb 1gB are all the same. 14 | 15 | # By default Redis does not run as a daemon. Use 'yes' if you need it. 16 | # Note that Redis will write a pid file in /var/run/redis.pid when daemonized. 17 | daemonize no 18 | 19 | # When running daemonized, Redis writes a pid file in /var/run/redis.pid by 20 | # default. You can specify a custom pid file location here. 21 | pidfile /var/run/redis.pid 22 | 23 | # Accept connections on the specified port, default is 6379. 24 | # If port 0 is specified Redis will not listen on a TCP socket. 25 | port 7778 26 | 27 | # If you want you can bind a single interface, if the bind option is not 28 | # specified all the interfaces will listen for incoming connections. 29 | # 30 | # bind 127.0.0.1 31 | 32 | # Specify the path for the unix socket that will be used to listen for 33 | # incoming connections. There is no default, so Redis will not listen 34 | # on a unix socket when not specified. 35 | # 36 | # unixsocket /tmp/redis.sock 37 | # unixsocketperm 755 38 | 39 | # Close the connection after a client is idle for N seconds (0 to disable) 40 | timeout 0 41 | 42 | # Set server verbosity to 'debug' 43 | # it can be one of: 44 | # debug (a lot of information, useful for development/testing) 45 | # verbose (many rarely useful info, but not a mess like the debug level) 46 | # notice (moderately verbose, what you want in production probably) 47 | # warning (only very important / critical messages are logged) 48 | loglevel debug 49 | 50 | # Specify the log file name. Also 'stdout' can be used to force 51 | # Redis to log on the standard output. Note that if you use standard 52 | # output for logging but daemonize, logs will be sent to /dev/null 53 | logfile /dev/null 54 | 55 | # To enable logging to the system logger, just set 'syslog-enabled' to yes, 56 | # and optionally update the other syslog parameters to suit your needs. 57 | # syslog-enabled no 58 | 59 | # Specify the syslog identity. 60 | # syslog-ident redis 61 | 62 | # Specify the syslog facility. Must be USER or between LOCAL0-LOCAL7. 63 | # syslog-facility local0 64 | 65 | # Set the number of databases. The default database is DB 0, you can select 66 | # a different one on a per-connection basis using SELECT where 67 | # dbid is a number between 0 and 'databases'-1 68 | databases 16 69 | 70 | ################################ SNAPSHOTTING ################################# 71 | # 72 | # Save the DB on disk: 73 | # 74 | # save 75 | # 76 | # Will save the DB if both the given number of seconds and the given 77 | # number of write operations against the DB occurred. 78 | # 79 | # In the example below the behaviour will be to save: 80 | # after 900 sec (15 min) if at least 1 key changed 81 | # after 300 sec (5 min) if at least 10 keys changed 82 | # after 60 sec if at least 10000 keys changed 83 | # 84 | # Note: you can disable saving at all commenting all the "save" lines. 85 | # 86 | # It is also possible to remove all the previously configured save 87 | # points by adding a save directive with a single empty string argument 88 | # like in the following example: 89 | # 90 | # save "" 91 | 92 | # save 900 1 93 | # save 300 10 94 | # save 60 10000 95 | 96 | # Compress string objects using LZF when dump .rdb databases? 97 | # For default that's set to 'yes' as it's almost always a win. 98 | # If you want to save some CPU in the saving child set it to 'no' but 99 | # the dataset will likely be bigger if you have compressible values or keys. 100 | rdbcompression yes 101 | 102 | # The filename where to dump the DB 103 | dbfilename dump.rdb 104 | 105 | # The working directory. 106 | # 107 | # The DB will be written inside this directory, with the filename specified 108 | # above using the 'dbfilename' configuration directive. 109 | # 110 | # Also the Append Only File will be created inside this directory. 111 | # 112 | # Note that you must specify a directory here, not a file name. 113 | dir /tmp/r3/db 114 | 115 | ################################# REPLICATION ################################# 116 | 117 | # Master-Slave replication. Use slaveof to make a Redis instance a copy of 118 | # another Redis server. Note that the configuration is local to the slave 119 | # so for example it is possible to configure the slave to save the DB with a 120 | # different interval, or to listen to another port, and so on. 121 | # 122 | # slaveof 123 | 124 | # If the master is password protected (using the "requirepass" configuration 125 | # directive below) it is possible to tell the slave to authenticate before 126 | # starting the replication synchronization process, otherwise the master will 127 | # refuse the slave request. 128 | # 129 | # masterauth 130 | 131 | # When a slave lost the connection with the master, or when the replication 132 | # is still in progress, the slave can act in two different ways: 133 | # 134 | # 1) if slave-serve-stale-data is set to 'yes' (the default) the slave will 135 | # still reply to client requests, possibly with out of date data, or the 136 | # data set may just be empty if this is the first synchronization. 137 | # 138 | # 2) if slave-serve-stale data is set to 'no' the slave will reply with 139 | # an error "SYNC with master in progress" to all the kind of commands 140 | # but to INFO and SLAVEOF. 141 | # 142 | slave-serve-stale-data yes 143 | 144 | # Slaves send PINGs to server in a predefined interval. It's possible to change 145 | # this interval with the repl_ping_slave_period option. The default value is 10 146 | # seconds. 147 | # 148 | # repl-ping-slave-period 10 149 | 150 | # The following option sets a timeout for both Bulk transfer I/O timeout and 151 | # master data or ping response timeout. The default value is 60 seconds. 152 | # 153 | # It is important to make sure that this value is greater than the value 154 | # specified for repl-ping-slave-period otherwise a timeout will be detected 155 | # every time there is low traffic between the master and the slave. 156 | # 157 | # repl-timeout 60 158 | 159 | ################################## SECURITY ################################### 160 | 161 | # Require clients to issue AUTH before processing any other 162 | # commands. This might be useful in environments in which you do not trust 163 | # others with access to the host running redis-server. 164 | # 165 | # This should stay commented out for backward compatibility and because most 166 | # people do not need auth (e.g. they run their own servers). 167 | # 168 | # Warning: since Redis is pretty fast an outside user can try up to 169 | # 150k passwords per second against a good box. This means that you should 170 | # use a very strong password otherwise it will be very easy to break. 171 | # 172 | # requirepass foobared 173 | 174 | requirepass r3 175 | 176 | # Command renaming. 177 | # 178 | # It is possible to change the name of dangerous commands in a shared 179 | # environment. For instance the CONFIG command may be renamed into something 180 | # of hard to guess so that it will be still available for internal-use 181 | # tools but not available for general clients. 182 | # 183 | # Example: 184 | # 185 | # rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52 186 | # 187 | # It is also possible to completely kill a command renaming it into 188 | # an empty string: 189 | # 190 | # rename-command CONFIG "" 191 | 192 | ################################### LIMITS #################################### 193 | 194 | # Set the max number of connected clients at the same time. By default 195 | # this limit is set to 10000 clients, however if the Redis server is not 196 | # able ot configure the process file limit to allow for the specified limit 197 | # the max number of allowed clients is set to the current file limit 198 | # minus 32 (as Redis reserves a few file descriptors for internal uses). 199 | # 200 | # Once the limit is reached Redis will close all the new connections sending 201 | # an error 'max number of clients reached'. 202 | # 203 | # maxclients 10000 204 | 205 | # Don't use more memory than the specified amount of bytes. 206 | # When the memory limit is reached Redis will try to remove keys with an 207 | # EXPIRE set. It will try to start freeing keys that are going to expire 208 | # in little time and preserve keys with a longer time to live. 209 | # Redis will also try to remove objects from free lists if possible. 210 | # 211 | # If all this fails, Redis will start to reply with errors to commands 212 | # that will use more memory, like SET, LPUSH, and so on, and will continue 213 | # to reply to most read-only commands like GET. 214 | # 215 | # WARNING: maxmemory can be a good idea mainly if you want to use Redis as a 216 | # 'state' server or cache, not as a real DB. When Redis is used as a real 217 | # database the memory usage will grow over the weeks, it will be obvious if 218 | # it is going to use too much memory in the long run, and you'll have the time 219 | # to upgrade. With maxmemory after the limit is reached you'll start to get 220 | # errors for write operations, and this may even lead to DB inconsistency. 221 | # 222 | # maxmemory 223 | 224 | # MAXMEMORY POLICY: how Redis will select what to remove when maxmemory 225 | # is reached? You can select among five behavior: 226 | # 227 | # volatile-lru -> remove the key with an expire set using an LRU algorithm 228 | # allkeys-lru -> remove any key accordingly to the LRU algorithm 229 | # volatile-random -> remove a random key with an expire set 230 | # allkeys->random -> remove a random key, any key 231 | # volatile-ttl -> remove the key with the nearest expire time (minor TTL) 232 | # noeviction -> don't expire at all, just return an error on write operations 233 | # 234 | # Note: with all the kind of policies, Redis will return an error on write 235 | # operations, when there are not suitable keys for eviction. 236 | # 237 | # At the date of writing this commands are: set setnx setex append 238 | # incr decr rpush lpush rpushx lpushx linsert lset rpoplpush sadd 239 | # sinter sinterstore sunion sunionstore sdiff sdiffstore zadd zincrby 240 | # zunionstore zinterstore hset hsetnx hmset hincrby incrby decrby 241 | # getset mset msetnx exec sort 242 | # 243 | # The default is: 244 | # 245 | # maxmemory-policy volatile-lru 246 | 247 | # LRU and minimal TTL algorithms are not precise algorithms but approximated 248 | # algorithms (in order to save memory), so you can select as well the sample 249 | # size to check. For instance for default Redis will check three keys and 250 | # pick the one that was used less recently, you can change the sample size 251 | # using the following configuration directive. 252 | # 253 | # maxmemory-samples 3 254 | 255 | ############################## APPEND ONLY MODE ############################### 256 | 257 | # By default Redis asynchronously dumps the dataset on disk. If you can live 258 | # with the idea that the latest records will be lost if something like a crash 259 | # happens this is the preferred way to run Redis. If instead you care a lot 260 | # about your data and don't want to that a single record can get lost you should 261 | # enable the append only mode: when this mode is enabled Redis will append 262 | # every write operation received in the file appendonly.aof. This file will 263 | # be read on startup in order to rebuild the full dataset in memory. 264 | # 265 | # Note that you can have both the async dumps and the append only file if you 266 | # like (you have to comment the "save" statements above to disable the dumps). 267 | # Still if append only mode is enabled Redis will load the data from the 268 | # log file at startup ignoring the dump.rdb file. 269 | # 270 | # IMPORTANT: Check the BGREWRITEAOF to check how to rewrite the append 271 | # log file in background when it gets too big. 272 | 273 | appendonly no 274 | 275 | # The name of the append only file (default: "appendonly.aof") 276 | # appendfilename appendonly.aof 277 | 278 | # The fsync() call tells the Operating System to actually write data on disk 279 | # instead to wait for more data in the output buffer. Some OS will really flush 280 | # data on disk, some other OS will just try to do it ASAP. 281 | # 282 | # Redis supports three different modes: 283 | # 284 | # no: don't fsync, just let the OS flush the data when it wants. Faster. 285 | # always: fsync after every write to the append only log . Slow, Safest. 286 | # everysec: fsync only if one second passed since the last fsync. Compromise. 287 | # 288 | # The default is "everysec" that's usually the right compromise between 289 | # speed and data safety. It's up to you to understand if you can relax this to 290 | # "no" that will let the operating system flush the output buffer when 291 | # it wants, for better performances (but if you can live with the idea of 292 | # some data loss consider the default persistence mode that's snapshotting), 293 | # or on the contrary, use "always" that's very slow but a bit safer than 294 | # everysec. 295 | # 296 | # If unsure, use "everysec". 297 | 298 | # appendfsync always 299 | appendfsync everysec 300 | # appendfsync no 301 | 302 | # When the AOF fsync policy is set to always or everysec, and a background 303 | # saving process (a background save or AOF log background rewriting) is 304 | # performing a lot of I/O against the disk, in some Linux configurations 305 | # Redis may block too long on the fsync() call. Note that there is no fix for 306 | # this currently, as even performing fsync in a different thread will block 307 | # our synchronous write(2) call. 308 | # 309 | # In order to mitigate this problem it's possible to use the following option 310 | # that will prevent fsync() from being called in the main process while a 311 | # BGSAVE or BGREWRITEAOF is in progress. 312 | # 313 | # This means that while another child is saving the durability of Redis is 314 | # the same as "appendfsync none", that in practical terms means that it is 315 | # possible to lost up to 30 seconds of log in the worst scenario (with the 316 | # default Linux settings). 317 | # 318 | # If you have latency problems turn this to "yes". Otherwise leave it as 319 | # "no" that is the safest pick from the point of view of durability. 320 | no-appendfsync-on-rewrite no 321 | 322 | # Automatic rewrite of the append only file. 323 | # Redis is able to automatically rewrite the log file implicitly calling 324 | # BGREWRITEAOF when the AOF log size will growth by the specified percentage. 325 | # 326 | # This is how it works: Redis remembers the size of the AOF file after the 327 | # latest rewrite (or if no rewrite happened since the restart, the size of 328 | # the AOF at startup is used). 329 | # 330 | # This base size is compared to the current size. If the current size is 331 | # bigger than the specified percentage, the rewrite is triggered. Also 332 | # you need to specify a minimal size for the AOF file to be rewritten, this 333 | # is useful to avoid rewriting the AOF file even if the percentage increase 334 | # is reached but it is still pretty small. 335 | # 336 | # Specify a percentage of zero in order to disable the automatic AOF 337 | # rewrite feature. 338 | 339 | # auto-aof-rewrite-percentage 100 340 | # auto-aof-rewrite-min-size 64mb 341 | 342 | ################################ LUA SCRIPTING ############################### 343 | 344 | # Max execution time of a Lua script in milliseconds. 345 | # 346 | # If the maximum execution time is reached Redis will log that a script is 347 | # still in execution after the maximum allowed time and will start to 348 | # reply to queries with an error. 349 | # 350 | # When a long running script exceed the maximum execution time only the 351 | # SCRIPT KILL and SHUTDOWN NOSAVE commands are available. The first can be 352 | # used to stop a script that did not yet called write commands. The second 353 | # is the only way to shut down the server in the case a write commands was 354 | # already issue by the script but the user don't want to wait for the natural 355 | # termination of the script. 356 | # 357 | # Set it to 0 or a negative value for unlimited execution without warnings. 358 | # lua-time-limit 5000 359 | 360 | ################################ REDIS CLUSTER ############################### 361 | # 362 | # Normal Redis instances can't be part of a Redis Cluster, only nodes that are 363 | # started as cluster nodes can. In order to start a Redis instance as a 364 | # cluster node enable the cluster support uncommenting the following: 365 | # 366 | # cluster-enabled yes 367 | 368 | # Every cluster node has a cluster configuration file. This file is not 369 | # intended to be edited by hand. It is created and updated by Redis nodes. 370 | # Every Redis Cluster node requires a different cluster configuration file. 371 | # Make sure that instances running in the same system does not have 372 | # overlapping cluster configuration file names. 373 | # 374 | # cluster-config-file nodes-6379.conf 375 | 376 | # In order to setup your cluster make sure to read the documentation 377 | # available at http://redis.io web site. 378 | 379 | ################################## SLOW LOG ################################### 380 | 381 | # The Redis Slow Log is a system to log queries that exceeded a specified 382 | # execution time. The execution time does not include the I/O operations 383 | # like talking with the client, sending the reply and so forth, 384 | # but just the time needed to actually execute the command (this is the only 385 | # stage of command execution where the thread is blocked and can not serve 386 | # other requests in the meantime). 387 | # 388 | # You can configure the slow log with two parameters: one tells Redis 389 | # what is the execution time, in microseconds, to exceed in order for the 390 | # command to get logged, and the other parameter is the length of the 391 | # slow log. When a new command is logged the oldest one is removed from the 392 | # queue of logged commands. 393 | 394 | # The following time is expressed in microseconds, so 1000000 is equivalent 395 | # to one second. Note that a negative number disables the slow log, while 396 | # a value of zero forces the logging of every command. 397 | # slowlog-log-slower-than 10000 398 | 399 | # There is no limit to this length. Just be aware that it will consume memory. 400 | # You can reclaim memory used by the slow log with SLOWLOG RESET. 401 | # slowlog-max-len 1024 402 | 403 | ############################### ADVANCED CONFIG ############################### 404 | 405 | # Hashes are encoded in a special way (much more memory efficient) when they 406 | # have at max a given number of elements, and the biggest element does not 407 | # exceed a given threshold. You can configure this limits with the following 408 | # configuration directives. 409 | hash-max-zipmap-entries 512 410 | hash-max-zipmap-value 64 411 | 412 | # Similarly to hashes, small lists are also encoded in a special way in order 413 | # to save a lot of space. The special representation is only used when 414 | # you are under the following limits: 415 | list-max-ziplist-entries 512 416 | list-max-ziplist-value 64 417 | 418 | # Sets have a special encoding in just one case: when a set is composed 419 | # of just strings that happens to be integers in radix 10 in the range 420 | # of 64 bit signed integers. 421 | # The following configuration setting sets the limit in the size of the 422 | # set in order to use this special memory saving encoding. 423 | set-max-intset-entries 512 424 | 425 | # Similarly to hashes and lists, sorted sets are also specially encoded in 426 | # order to save a lot of space. This encoding is only used when the length and 427 | # elements of a sorted set are below the following limits: 428 | # zset-max-ziplist-entries 128 429 | # zset-max-ziplist-value 64 430 | 431 | # Active rehashing uses 1 millisecond every 100 milliseconds of CPU time in 432 | # order to help rehashing the main Redis hash table (the one mapping top-level 433 | # keys to values). The hash table implementation Redis uses (see dict.c) 434 | # performs a lazy rehashing: the more operation you run into an hash table 435 | # that is rehashing, the more rehashing "steps" are performed, so if the 436 | # server is idle the rehashing is never complete and some more memory is used 437 | # by the hash table. 438 | # 439 | # The default is to use this millisecond 10 times every second in order to 440 | # active rehashing the main dictionaries, freeing memory when possible. 441 | # 442 | # If unsure: 443 | # use "activerehashing no" if you have hard latency requirements and it is 444 | # not a good thing in your environment that Redis can reply form time to time 445 | # to queries with 2 milliseconds delay. 446 | # 447 | # use "activerehashing yes" if you don't have such hard requirements but 448 | # want to free memory asap when possible. 449 | activerehashing yes 450 | 451 | ################################## INCLUDES ################################### 452 | 453 | # Include one or more other config files here. This is useful if you 454 | # have a standard template that goes to all Redis server but also need 455 | # to customize a few per-server settings. Include files can include 456 | # other files, so use this wisely. 457 | # 458 | # include /path/to/local.conf 459 | # include /path/to/other.conf 460 | 461 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | redis 2 | tornado-redis 3 | http://gevent.googlecode.com/files/gevent-1.0b2.tar.gz 4 | tornado-pyvows 5 | ujson 6 | flask 7 | argparse 8 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | from setuptools import setup 5 | from r3.version import __version__ 6 | 7 | setup( 8 | name = 'r3', 9 | version = __version__, 10 | description = "r3 is a map-reduce engine written in python", 11 | long_description = """ 12 | r3 is a map-reduce engine that uses redis as a backend. It's very simple to use. 13 | """, 14 | keywords = 'map reduce', 15 | author = 'Bernardo Heynemann', 16 | author_email = 'heynemann@gmail.com', 17 | url = 'http://heynemann.github.com/r3/', 18 | license = 'MIT', 19 | classifiers = ['Development Status :: 3 - Alpha', 20 | 'Intended Audience :: Developers', 21 | 'License :: OSI Approved :: MIT License', 22 | 'Natural Language :: English', 23 | 'Operating System :: MacOS', 24 | 'Operating System :: POSIX :: Linux', 25 | 'Programming Language :: Python :: 2.6', 26 | 'Programming Language :: Python :: 2.7', 27 | ], 28 | 29 | include_package_data = True, 30 | package_data = { 31 | '': ['*.gif', '*.png', '*.jpg', '*.jpeg', '*.css', '*.js', '*.html'], 32 | }, 33 | 34 | packages = ['r3', 'r3'], 35 | package_dir = {"r3": "r3"}, 36 | 37 | install_requires=[ 38 | 'redis', 39 | 'tornado-redis', 40 | 'tornado', 41 | 'ujson', 42 | 'flask', 43 | 'argparse', 44 | 'hiredis' 45 | ], 46 | 47 | entry_points = { 48 | 'console_scripts': [ 49 | 'r3-app=r3.app.server:main', 50 | 'r3-web=r3.web.server:main', 51 | 'r3-map=r3.worker.mapper:main' 52 | ], 53 | } 54 | ) 55 | 56 | -------------------------------------------------------------------------------- /test/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/heynemann/r3/cc6b4eb55c7ae30a8f75af2be165504565dbeb79/test/__init__.py -------------------------------------------------------------------------------- /test/app_config.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | INPUT_STREAMS = [ 5 | 'test.count_words_stream.CountWordsStream' 6 | ] 7 | 8 | REDUCERS = [ 9 | 'test.count_words_reducer.CountWordsReducer' 10 | ] 11 | -------------------------------------------------------------------------------- /test/chekhov.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/heynemann/r3/cc6b4eb55c7ae30a8f75af2be165504565dbeb79/test/chekhov.txt -------------------------------------------------------------------------------- /test/count_words_mapper.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | from r3.worker.mapper import Mapper 5 | 6 | class CountWordsMapper(Mapper): 7 | job_type = 'count-words' 8 | 9 | def map(self, lines): 10 | #time.sleep(0.5) 11 | return list(self.split_words(lines)) 12 | 13 | def split_words(self, lines): 14 | for line in lines: 15 | for word in line.split(): 16 | yield word.strip().strip('.').strip(','), 1 17 | -------------------------------------------------------------------------------- /test/count_words_reducer.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | from collections import defaultdict 5 | 6 | class CountWordsReducer: 7 | job_type = 'count-words' 8 | 9 | def reduce(self, app, items): 10 | word_freq = defaultdict(int) 11 | for line in items: 12 | for word, frequency in line: 13 | word_freq[word] += frequency 14 | 15 | return word_freq 16 | -------------------------------------------------------------------------------- /test/count_words_stream.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | from os.path import abspath, dirname, join 5 | 6 | class CountWordsStream: 7 | job_type = 'count-words' 8 | group_size = 1000 9 | 10 | def process(self, app, arguments): 11 | with open(abspath(join(dirname(__file__), 'chekhov.txt'))) as f: 12 | contents = f.readlines() 13 | 14 | return [line.lower() for line in contents] 15 | 16 | 17 | -------------------------------------------------------------------------------- /test/small-chekhov.txt: -------------------------------------------------------------------------------- 1 | Project Gutenberg's Plays by Chekhov, Second Series, by Anton Chekhov 2 | 3 | This eBook is for the use of anyone anywhere at no cost and with 4 | almost no restrictions whatsoever. You may copy it, give it away or 5 | re-use it under the terms of the Project Gutenberg License included 6 | with this eBook or online at www.gutenberg.org 7 | 8 | 9 | Title: Plays by Chekhov, Second Series 10 | On the High Road, The Proposal, The Wedding, The Bear, A 11 | Tragedian In Spite of Himself, The Anniversary, The Three 12 | Sisters, The Cherry Orchard 13 | 14 | Author: Anton Chekhov 15 | 16 | Release Date: April, 2005 [EBook #7986] 17 | Posting Date: August 8, 2009 18 | 19 | Language: English 20 | 21 | Character set encoding: ISO-8859-1 22 | 23 | *** START OF THIS PROJECT GUTENBERG EBOOK PLAYS BY CHEKHOV, SECOND SERIES *** 24 | 25 | 26 | 27 | 28 | Produced by James Rusk and Nicole Apostola 29 | 30 | 31 | 32 | 33 | 34 | PLAYS BY ANTON CHEKHOV, SECOND SERIES 35 | 36 | By Anton Chekhov 37 | 38 | Translated, with an Introduction, by Julius West 39 | 40 | [The First Series Plays have been previously published 41 | by Project Gutenberg in etext numbers: 1753 through 1756] 42 | 43 | 44 | CONTENTS 45 | 46 | INTRODUCTION 47 | ON THE HIGH ROAD 48 | THE PROPOSAL 49 | THE WEDDING 50 | THE BEAR 51 | A TRAGEDIAN IN SPITE OF HIMSELF 52 | THE ANNIVERSARY 53 | THE THREE SISTERS 54 | THE CHERRY ORCHARD 55 | 56 | 57 | 58 | 59 | INTRODUCTION 60 | 61 | The last few years have seen a large and generally unsystematic mass of 62 | translations from the Russian flung at the heads and hearts of English 63 | readers. The ready acceptance of Chekhov has been one of the few 64 | successful features of this irresponsible output. He has been welcomed 65 | by British critics with something like affection. Bernard Shaw has 66 | several times remarked: "Every time I see a play by Chekhov, I want to 67 | chuck all my own stuff into the fire." Others, having no such valuable 68 | property to sacrifice on the altar of Chekhov, have not hesitated 69 | to place him side by side with Ibsen, and the other established 70 | institutions of the new theatre. For these reasons it is pleasant to 71 | -------------------------------------------------------------------------------- /test/test_count_words.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -*- 3 | 4 | import time 5 | 6 | from count_words_stream import CountWordsStream 7 | from count_words_reducer import CountWordsReducer 8 | 9 | class CountWordsMapper: 10 | def map(self, lines): 11 | #time.sleep(0.5) 12 | return list(self.split_words(lines)) 13 | 14 | def split_words(self, lines): 15 | for line in lines: 16 | for word in line.split(): 17 | yield word.strip().strip('.').strip(','), 1 18 | 19 | 20 | def main(): 21 | start = time.time() 22 | items = CountWordsStream().process(None, None) 23 | print "input stream took %.2f" % (time.time() - start) 24 | 25 | start = time.time() 26 | mapper = CountWordsMapper() 27 | results = [] 28 | for item in items: 29 | results.append(mapper.map(item)) 30 | print "mapping took %.2f" % (time.time() - start) 31 | 32 | start = time.time() 33 | CountWordsReducer().reduce(None, results) 34 | print "reducing took %.2f" % (time.time() - start) 35 | 36 | if __name__ == '__main__': 37 | main() 38 | -------------------------------------------------------------------------------- /test/test_sync.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/heynemann/r3/cc6b4eb55c7ae30a8f75af2be165504565dbeb79/test/test_sync.py --------------------------------------------------------------------------------