├── .gitignore ├── .travis.yml ├── README.rst ├── setup.cfg ├── setup.py ├── test-requirements.txt ├── ucron.png ├── ucron.tab └── ucron ├── __init__.py ├── __main__.py ├── cli.py ├── conf.py ├── db.py ├── ext.py ├── server.py ├── stpl.py ├── utils.py └── worker.py /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | env/ 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | .eggs/ 18 | lib/ 19 | lib64/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *.cover 47 | .hypothesis/ 48 | 49 | # Translations 50 | *.mo 51 | *.pot 52 | 53 | # Django stuff: 54 | *.log 55 | local_settings.py 56 | 57 | # Flask stuff: 58 | instance/ 59 | .webassets-cache 60 | 61 | # Scrapy stuff: 62 | .scrapy 63 | 64 | # Sphinx documentation 65 | docs/_build/ 66 | 67 | # PyBuilder 68 | target/ 69 | 70 | # Jupyter Notebook 71 | .ipynb_checkpoints 72 | 73 | # pyenv 74 | .python-version 75 | 76 | # celery beat schedule file 77 | celerybeat-schedule 78 | 79 | # SageMath parsed files 80 | *.sage.py 81 | 82 | # dotenv 83 | .env 84 | 85 | # virtualenv 86 | .venv 87 | venv/ 88 | ENV/ 89 | 90 | # Spyder project settings 91 | .spyderproject 92 | .spyproject 93 | 94 | # Rope project settings 95 | .ropeproject 96 | 97 | # mkdocs documentation 98 | /site 99 | 100 | # mypy 101 | .mypy_cache/ 102 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | python: 3 | - "3.5" 4 | install: "pip install -r test-requirements.txt" 5 | script: 6 | - flake8 7 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | uCron 2 | ===== 3 | 4 | .. image:: https://img.shields.io/travis/akgnah/ucron/master.svg 5 | :target: https://travis-ci.org/akgnah/ucron 6 | 7 | .. image:: https://img.shields.io/pypi/v/ucron.svg 8 | :target: https://pypi.python.org/pypi/ucron 9 | 10 | .. image:: https://img.shields.io/pypi/l/ucron.svg 11 | :target: https://pypi.python.org/pypi/ucron 12 | 13 | .. image:: https://img.shields.io/badge/code_style-pep8-orange.svg 14 | :target: https://www.python.org/dev/peps/pep-0008 15 | 16 | 17 | uCron 是一个微型的执行定时任务（Crontab）和任务队列（Task queue）的小工具，易于安装和运行，配置简单且依赖少。 18 | 19 | 特别地，如果你使用 Windows，没有 Crontab，这个小工具能为你提供类似的功能。当然它也可在 Linux 上工作。 20 | 21 | 安装 22 | ---- 23 | 24 | .. code-block:: bash 25 | 26 | $ pip install ucron 27 | 28 | 程序依赖 bottle，在 Win10 上，Python2.7 和 3.5+ 测试通过，在 Archlinux 上，Python2.7 和 3.3+ 测试通过。 29 | 30 | 使用 31 | ---- 32 | 33 | .. code-block:: bash 34 | 35 | $ python -m ucron 36 | 37 | 这是最简单的使用方法，使用浏览器访问 http://127.0.0.1:8089/ 将会看到一个简单的页面。 38 | 39 | .. image:: ucron.png 40 | 41 | 运行 python -m ucron -h 可查看全部可用参数，简单说明如下： 42 | 43 | :: 44 | 45 | --host 指定程序运行的 IP，默认为 127.0.0.1。 46 | --port 指定程序运行的端口，默认为 8089。 47 | --cron 指定定时任务的配置文件，格式见 ucron.tab 或下文。 48 | --dbn 指定文件用于 SQLite，默认为 :memory:，即内存模式。 49 | --log 指定日志文件，默认为当前目录下的 ucron.log。 50 | --max 指定日志文件的最大行数，默认为 10240。 51 | --utc 指定时区，默认为 +8。 52 | --quiet 指定 bottle 的 quiet 选项，默认为 False。 53 | --reload 指定是否自动重载 ucron.tab，默认为 False。 54 | --tab 指定清理日志文件的执行周期，默认为每天早上 5 点。 55 | --add 增加队列，需指定端口、队列名和类型。 56 | --cls 清空队列，需指定端口和队列名。 57 | --del 删除队列，需指定端口和队列名。 58 | --list 列出全部队列和它们的剩余长度，需指定端口。 59 | 60 | 请注意，当 SQLite 使用内存模式，即 --dbn 使用默认值时，程序关闭时会丢失队列。 61 | 62 | 典型的使用方法可能是这样： 63 | 64 | .. code-block:: bash 65 | 66 | $ python -m ucron --cron ucron.tab 67 | 68 | 这会读取当前目录下的 ucron.tab 增加定时任务。指定的文件可使用相对或绝对路径。 69 | 70 | 定时任务 71 | ^^^^^^^^ 72 | 73 | :: 74 | 75 | */2 * * * * https://httpbin.org/post hello=world&text=test POST 76 | 77 | 这是 ucron.tab 中的一行，它使用和 Crontab 类似的格式，每行为一个任务，每个任务有四个部分，使用空格分隔，最后两个部分均为可选。 78 | 79 | 第一部分是执行周期，使用和标准 Crontab 一致的格式；第二部分是要访问的地址；第三部分是提供给地址的数据，使用 key1=value1&key2=value2 的格式；最后一部分是访问方法，可为 GET 或 POST，默认为 GET。第三、四部分均为可选，也可忽略第三部分直接提供第四部分。 80 | 81 | 请注意各个部分使用空格来分隔，也正因为如此，所以 URL 或参数中不能含有空格。读取配置时会忽略以 # 开始的行。 82 | 83 | 如果在运行中修改了该配置文件，并且没有指定 --reload 为真，那么需要访问 http://127.0.0.1:8089/reload 以使配置生效（假设使用了默认端口 8089，下同）。还有一点很重要，请使用 UTF-8 编码保存 ucron.tab。 84 | 85 | 如果你不熟悉 Crontab 格式，这个在线 `Crontab 编辑器 `_ 可能会有帮助。 86 | 87 | 88 | 任务队列 89 | ^^^^^^^^ 90 | 91 | 要添加任务到队列中很简单： 92 | 93 | .. code-block:: python 94 | 95 | from ucron import add_task 96 | 97 | body = {'page': 1, 'text': '测试'} 98 | resp = add_task('http://setq.me', body, method='GET') 99 | print(resp) 100 | 101 | add_task 方法接受的参数有 path、args、method、name、port、host 和 json，其中只有 path 是必需的，其他均为可选参数。 102 | 103 | path 是要访问的地址，args 为发送给地址的数据，接受一个字典，默认为空字符串，method 可为 GET 或 POST，默认为 GET。 104 | 105 | name 是队列名，默认为 default_seq，这个参数在下面会有详细说明。port 默认为 8089，host 默认为 127.0.0.1。 106 | 107 | 在最新版本中，你还可以把字典传给 add_task 的 json 参数，此时会把 Content-Type header 设置为 application/json 并发送 POST 请求。 108 | 109 | 110 | 队列类型 111 | ^^^^^^^^ 112 | 113 | uCron 中的任务队列有顺序队列和并发队列两种类型。程序在运行的时候会新建两个默认的队列，default_seq 和 default_con，分别对应着顺序队列和并发队列。你也可以自己增加队列并在 add_task 中指定 name 参数。 114 | 115 | 有三种方法创建队列： 116 | 117 | .. code-block:: bash 118 | 119 | $ python -m ucron --add 8089 bot_seq seq 120 | 121 | 这会向 8089 端口运行着的 uCron 增加一个名为 bot_seq 的顺序（seq）队列。 122 | 123 | .. code-block:: python 124 | 125 | from ucron import add_queue 126 | 127 | add_queue('bot_con', 'con', port='8089') 128 | 129 | 这会向 8089 端口运行着的 uCron 增加一个名为 bot_con 的并发（con）队列。 130 | 131 | 向新建的队列添加任务： 132 | 133 | .. code-block:: python 134 | 135 | from ucron import add_task 136 | 137 | for i in range(500): # 顺序队列 138 | body = {'page': i} 139 | add_task('http://setq.me', body, name='bot_seq') 140 | 141 | for i in range(500): # 并发队列 142 | body = {'page': i} 143 | add_task('http://setq.me', body, name='bot_con') 144 | 145 | 同样也有三种方法删除队列： 146 | 147 | .. code-block:: bash 148 | 149 | $ python -m ucron --del 8089 bot_seq 150 | 151 | 这会删除刚才增加的 bot_seq 队列。 152 | 153 | .. code-block:: python 154 | 155 | from ucron import del_queue 156 | 157 | del_queue('bot_con', port='8089') 158 | 159 | 这会删除刚才增加的 bot_con 队列。 160 | 161 | 第二种方法为代码在运行时临时增删队列提供了便利。第三种方法是使用浏览器访问 http://127.0.0.1:8089/ 进行操作。提醒一下，并发队列不保证严格有序。 162 | 163 | add_task、add_queue 和 del_queue 均定义在 ucron/ext.py 中，它们都很简单，请查看以了解更多详情。 164 | 165 | 166 | 杂项 167 | ---- 168 | 这个小工具产生于我写饭否机器人的过程中，由于当时不知道已经存在很多优秀的类似工具，如 `schedule `_ 和 `Celery `_ ，所以重复造了轮子，但造轮子还是挺开心的。 169 | 170 | 最后，感谢使用，若有疑问或建议请 Email 联系我。 171 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [bdist_wheel] 2 | universal=1 3 | [flake8] 4 | ignore = E501,E402,E731 -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -* 3 | from setuptools import setup 4 | from ucron import __version__ 5 | 6 | 7 | long_description = open('README.rst', 'rb').read().decode('utf8') 8 | 9 | setup(name='ucron', 10 | version=__version__, 11 | description='A micro Crontab & Task Queue for Python Web.', 12 | author='Akgnah', 13 | author_email='1024@setq.me', 14 | url='http://github.com/akgnah/ucron', 15 | packages=['ucron'], 16 | install_requires=['bottle'], 17 | long_description=long_description, 18 | license='MIT', 19 | platforms=['any'], 20 | keywords='crontab task queue') 21 | -------------------------------------------------------------------------------- /test-requirements.txt: -------------------------------------------------------------------------------- 1 | flake8 2 | -------------------------------------------------------------------------------- /ucron.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/akgnah/ucron/6c93a7c2b6b5fb302b3c0c84e938a153ce9fa34a/ucron.png -------------------------------------------------------------------------------- /ucron.tab: -------------------------------------------------------------------------------- 1 | */5 * * * * https://httpbin.org/get 2 | */2 * * * * https://httpbin.org/post hello=world&text=test POST -------------------------------------------------------------------------------- /ucron/__init__.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -* 3 | from __future__ import absolute_import 4 | 5 | from ucron.ext import add_task, add_queue, del_queue 6 | 7 | __version__ = '0.1.3' 8 | __author__ = 'akgnah <1024@setq.me>' 9 | __all__ = ['add_task', 'add_queue', 'del_queue'] 10 | -------------------------------------------------------------------------------- /ucron/__main__.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -* 3 | from __future__ import absolute_import 4 | 5 | from ucron import db, cli, server, worker 6 | 7 | if __name__ == '__main__': 8 | cli.parse_args() 9 | db.initalize() # initalize database 10 | worker.start() # start worker threads 11 | server.start() # start bottle server 12 | -------------------------------------------------------------------------------- /ucron/cli.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -* 3 | from __future__ import absolute_import 4 | 5 | import argparse 6 | 7 | from ucron import conf 8 | from ucron.utils import request, urlencode, URLError, loads 9 | 10 | 11 | class Check_Quiet(argparse.Action): 12 | def __init__(self, option_strings, dest, nargs=None, **kwargs): 13 | argparse.Action.__init__(self, option_strings, dest, nargs, **kwargs) 14 | 15 | def __call__(self, parser, namespace, values, option_string=None): 16 | if values.lower() == 'false': 17 | setattr(namespace, self.dest, False) 18 | else: 19 | setattr(namespace, self.dest, True) 20 | 21 | 22 | class Check_Reload(argparse.Action): 23 | def __init__(self, option_strings, dest, nargs=None, **kwargs): 24 | argparse.Action.__init__(self, option_strings, dest, nargs, **kwargs) 25 | 26 | def __call__(self, parser, namespace, values, option_string=None): 27 | if values.lower() == 'false': 28 | setattr(namespace, self.dest, False) 29 | else: 30 | setattr(namespace, self.dest, True) 31 | 32 | 33 | class Check_Tab(argparse.Action): 34 | def __init__(self, option_strings, dest, nargs=None, **kwargs): 35 | argparse.Action.__init__(self, option_strings, dest, nargs, **kwargs) 36 | 37 | def __call__(self, parser, namespace, values, option_string=None): 38 | if len(values.split(' ')) != 5: 39 | print('The crontab rule is invalid.') 40 | print("Example:\n --tab '0 5 * * *'") 41 | print('Note that argument included in single quotes.') 42 | exit(1) 43 | else: 44 | setattr(namespace, self.dest, values) 45 | 46 | 47 | class Add_Queue(argparse.Action): 48 | def __init__(self, option_strings, dest, nargs=None, **kwargs): 49 | argparse.Action.__init__(self, option_strings, dest, nargs, **kwargs) 50 | 51 | def __call__(self, parser, namespace, values, option_string=None): 52 | port, name, mode = values 53 | if mode not in ['con', 'seq']: 54 | print('The mode must be either con or seq.') 55 | exit(1) 56 | if not name.strip(): 57 | print('The name is invalid.') 58 | exit(1) 59 | data = {'name': name, 'mode': mode, 'opt': 'add', 'cli': 'true'} 60 | url = 'http://127.0.0.1:%s/taskq?%s' % (port, urlencode(data)) 61 | try: 62 | resp = request.urlopen(url) 63 | print(resp.read().decode()) 64 | except(URLError): 65 | print('Connection refused. Please check port.') 66 | exit(0) 67 | 68 | 69 | class Del_Queue(argparse.Action): 70 | def __init__(self, option_strings, dest, nargs=None, **kwargs): 71 | argparse.Action.__init__(self, option_strings, dest, nargs, **kwargs) 72 | 73 | def __call__(self, parser, namespace, values, option_string=None): 74 | port, name = values 75 | data = {'name': name, 'opt': 'del', 'cli': 'true'} 76 | url = 'http://127.0.0.1:%s/taskq?%s' % (port, urlencode(data)) 77 | try: 78 | resp = request.urlopen(url) 79 | print(resp.read().decode()) 80 | except(URLError): 81 | print('Connection refused. Please check port.') 82 | exit(0) 83 | 84 | 85 | class Cls_Queue(argparse.Action): # Clear 86 | def __init__(self, option_strings, dest, nargs=None, **kwargs): 87 | argparse.Action.__init__(self, option_strings, dest, nargs, **kwargs) 88 | 89 | def __call__(self, parser, namespace, values, option_string=None): 90 | port, name = values 91 | data = {'name': name, 'opt': 'cls', 'cli': 'true'} 92 | url = 'http://127.0.0.1:%s/taskq?%s' % (port, urlencode(data)) 93 | try: 94 | resp = request.urlopen(url) 95 | print(resp.read().decode()) 96 | except(URLError): 97 | print('Connection refused. Please check port.') 98 | exit(0) 99 | 100 | 101 | class List_Queue(argparse.Action): 102 | def __init__(self, option_strings, dest, nargs=None, **kwargs): 103 | argparse.Action.__init__(self, option_strings, dest, nargs, **kwargs) 104 | 105 | def __call__(self, parser, namespace, values, option_string=None): 106 | url = 'http://127.0.0.1:%s/taskq?opt=list' % values # port 107 | try: 108 | resp = request.urlopen(url) 109 | for item in loads(resp.read().decode()): 110 | print('%-16s%4s%4s' % tuple(item)) 111 | except(URLError): 112 | print('Connection refused. Please check port.') 113 | exit(0) 114 | 115 | 116 | def parse_args(): 117 | parser = argparse.ArgumentParser(description='uCron - Micro(toy) Crontab and Task Queue', prog='python -m ucron', 118 | epilog='See https://github.com/akgnah/ucron/ for more details.') 119 | 120 | parser.add_argument('--host', default='127.0.0.1', metavar='host', 121 | help='specify host, default is 127.0.0.1') 122 | parser.add_argument('--port', default='8089', metavar='port', 123 | help='specify port number, default is 8089') 124 | parser.add_argument('--cron', metavar='file', 125 | help='specify crontab file, required by crontab') 126 | parser.add_argument('--dbn', default=':memory:', metavar='file', 127 | help='specify db file or :memory:, default is :memory:') 128 | parser.add_argument('--log', default='ucron.log', metavar='file', 129 | help='specify log file, default is ucron.log in pwd') 130 | parser.add_argument('--max', default=10240, type=int, metavar='number', 131 | help='specify maximum rows for log, default is 10240') 132 | parser.add_argument('--utc', default=8, type=int, metavar='tzinfo', 133 | help='specify tzinfo for UTC, default is +8') 134 | parser.add_argument('--quiet', default=False, metavar='true/false', action=Check_Quiet, 135 | help='specify value for bottle --quiet, default is False') 136 | parser.add_argument('--reload', default=False, metavar='true/false', action=Check_Reload, 137 | help='specify value for reload ucron.tab, default is False') 138 | parser.add_argument('--tab', default='0 5 * * *', metavar="'x x x x x'", action=Check_Tab, 139 | help="specify cron for clean log, default is '0 5 * * *'") 140 | parser.add_argument('--add', nargs=3, metavar=('port', 'name', 'mode'), action=Add_Queue, 141 | help='specify port, name and mode to add a queue and exit') 142 | parser.add_argument('--cls', nargs=2, metavar=('port', 'name'), action=Cls_Queue, 143 | help='specify port and name to clear a queue and exit') 144 | parser.add_argument('--del', nargs=2, metavar=('port', 'name'), action=Del_Queue, 145 | help='specify port and name to delete a queue and exit') 146 | parser.add_argument('--list', metavar='port', action=List_Queue, 147 | help='specify port to list all queues and exit') 148 | 149 | args = parser.parse_args() 150 | conf.save(args) # save args to conf.py 151 | -------------------------------------------------------------------------------- /ucron/conf.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -* 3 | from __future__ import absolute_import 4 | 5 | import os 6 | 7 | 8 | def save(args): 9 | for key in ['cron', 'dbn', 'log']: 10 | value = getattr(args, key) 11 | if value and value != ':memory:': 12 | setattr(args, key, os.path.join(os.getcwd(), value)) 13 | setattr(args, 'local', 'http://127.0.0.1:%s' % args.port) 14 | globals().update(args.__dict__) 15 | -------------------------------------------------------------------------------- /ucron/db.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -* 3 | from __future__ import absolute_import 4 | 5 | import sqlite3 6 | from threading import Thread 7 | 8 | from ucron import conf 9 | from ucron.utils import Queue, iterbetter, dumps, loads 10 | 11 | 12 | class DB(Thread): 13 | def __init__(self, dbn=':memory:'): 14 | Thread.__init__(self) 15 | self.dbn = dbn 16 | self.reqs = Queue() 17 | self.daemon = True 18 | self.start() 19 | 20 | def run(self): 21 | self.con = sqlite3.connect(self.dbn, check_same_thread=False) 22 | self.cur = self.con.cursor() 23 | 24 | while True: 25 | req, arg, res = self.reqs.get() 26 | if req == '--close--': 27 | break 28 | if req == '--commit--': 29 | self.con.commit() 30 | 31 | try: 32 | self.cur.execute(req, arg) 33 | except Exception: 34 | self.con.rollback() 35 | 36 | if self.cur.description: 37 | for row in self.cur: 38 | res.put(row) 39 | else: 40 | res.put(self.cur.rowcount) 41 | res.put('--no more--') 42 | 43 | self.con.close() 44 | 45 | def execute(self, req, arg=tuple()): 46 | res = Queue() 47 | self.reqs.put((req, arg, res)) 48 | 49 | def query(self, req, arg=tuple()): 50 | res = Queue() 51 | self.reqs.put((req, arg, res)) 52 | 53 | def iterwrapper(): 54 | while True: 55 | row = res.get() 56 | if row == '--no more--': 57 | break 58 | yield row 59 | 60 | return iterbetter(iterwrapper()) 61 | 62 | def close(self): 63 | self.execute('--close--') 64 | 65 | def commit(self): 66 | self.execute('--commit--') 67 | 68 | 69 | class Cron(DB): 70 | def __init__(self, dbn): 71 | DB.__init__(self, dbn) 72 | self.execute("create table cron (path blob, args text, \ 73 | method blob, schedule text, id blob primary key)") 74 | self.commit() 75 | 76 | def push(self, _id, path, args, method, schedule): 77 | rowcount = self.query("insert into cron (id, path, args, method, schedule) \ 78 | values (?, ?, ?, ?, ?)", (_id, path, args, method, dumps(schedule))).first() 79 | self.commit() 80 | return rowcount 81 | 82 | def fetchall(self): 83 | rows = [] 84 | for row in self.query("select id, path, args, method, schedule from cron"): 85 | job = dict(zip(['id', 'path', 'args', 'method'], row[:-1])) 86 | job.update(loads(row[-1])) 87 | rows.append(job) 88 | return rows 89 | 90 | def empty(self): 91 | rowcount = self.query("delete from cron").first() 92 | self.commit() 93 | return rowcount 94 | 95 | 96 | class Status(DB): 97 | def __init__(self, dbn): 98 | DB.__init__(self, dbn) # status: [time] - status 99 | self.execute("create table status (id blob primary key, \ 100 | schedule blob, status text default '[None] - None')") 101 | self.commit() 102 | 103 | def push(self, _id, schedule): 104 | rowcount = self.query("insert into status (id, schedule) \ 105 | values (?, ?)", (_id, schedule)).first() 106 | self.commit() 107 | return rowcount 108 | 109 | def update(self, _id, status): 110 | rowcount = self.query("update status set status = ? \ 111 | where id = ?", (status, _id)).first() 112 | self.commit() 113 | return rowcount 114 | 115 | def fetch(self, _id): 116 | return self.query("select schedule, status from status \ 117 | where id = ?", (_id,)).first() 118 | 119 | 120 | class TaskQ(DB): 121 | def __init__(self, dbn): 122 | DB.__init__(self, dbn) 123 | self.execute("create table taskq (name blob primary key, mode blob)") 124 | self.commit() 125 | 126 | def push(self, name, mode): # mode: 'seq' or 'con' 127 | rowcount = self.query("insert into taskq (name, mode) \ 128 | values (?, ?)", (name, mode)).first() 129 | self.commit() 130 | return rowcount 131 | 132 | def fetchall(self): 133 | return self.query("select name, mode from taskq") 134 | 135 | def delete(self, name): 136 | rowcount = self.query("delete from taskq where name = ?", (name,)).first() 137 | self.commit() 138 | return rowcount 139 | 140 | 141 | class Task(DB): 142 | def __init__(self, dbn): 143 | DB.__init__(self, dbn) 144 | self.execute("create table task (path blob, args text, method blob, \ 145 | name blob, json integer, id integer primary key)") 146 | self.commit() 147 | 148 | def push(self, path, args, method, name, json): 149 | rowcount = self.query("insert into task (path, args, method, name, json) \ 150 | values (?, ?, ?, ?, ?)", (path, args, method, name, json)).first() 151 | self.commit() 152 | return rowcount 153 | 154 | def pop(self, name): 155 | row = self.query("select path, args, method, json, id from task \ 156 | where name = ? order by id", (name,)).first() 157 | if row: 158 | self.query("delete from task where id = ?", (row[-1],)) 159 | self.commit() 160 | return row 161 | 162 | def fetchall(self, name): 163 | rows = self.query("select path, args, method, json, id from task \ 164 | where name = ? order by id", (name,)) 165 | if rows: 166 | self.query("delete from task where name = ?", (name,)) 167 | self.commit() 168 | return rows 169 | 170 | def delete(self, name): 171 | rowcount = self.query("delete from task where name = ?", (name,)).first() 172 | self.commit() 173 | return rowcount 174 | 175 | def length(self, name='%'): 176 | return self.query("select count(*) from task \ 177 | where name = ?", (name,)).first()[0] 178 | 179 | 180 | def initalize(): 181 | global cron, task, taskq, status 182 | cron = Cron(conf.dbn) 183 | task = Task(conf.dbn) 184 | taskq = TaskQ(conf.dbn) 185 | status = Status(conf.dbn) 186 | -------------------------------------------------------------------------------- /ucron/ext.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -* 3 | from __future__ import absolute_import 4 | 5 | from ucron.utils import request, urlencode, URLError, dumps 6 | 7 | 8 | def add_task(path, args='', method='GET', name='default_seq', port='8089', host='127.0.0.1', json=None): 9 | if args != '' and not isinstance(args, dict): 10 | raise Exception('TypeError: Argument args should be dict.') 11 | headers = {'Content-Type': 'application/json'} 12 | req = request.Request('http://%s:%s/add_task' % (host, port), headers=headers) 13 | data = {'path': path, 'args': args, 'method': method, 'name': name, 'json': json} 14 | try: 15 | resp = request.urlopen(req, dumps(data).encode('utf8')) 16 | return resp.read().decode() 17 | except(URLError): 18 | raise Exception('Connection refused. Please check host or port.') 19 | 20 | 21 | def add_queue(name, mode='seq', port='8089', host='127.0.0.1'): 22 | data = {'name': name, 'mode': mode, 'opt': 'add', 'cli': 'true'} 23 | url = 'http://%s:%s/taskq?%s' % (host, port, urlencode(data)) 24 | try: 25 | resp = request.urlopen(url) 26 | return resp.read().decode() 27 | except(URLError): 28 | raise Exception('Connection refused. Please check host or port.') 29 | 30 | 31 | def del_queue(name, port='8089', host='127.0.0.1'): 32 | data = {'name': name, 'opt': 'del', 'cli': 'true'} 33 | url = 'http://%s:%s/taskq?%s' % (host, port, urlencode(data)) 34 | try: 35 | resp = request.urlopen(url) 36 | return resp.read().decode() 37 | except(URLError): 38 | raise Exception('Connection refused. Please check host or port.') 39 | -------------------------------------------------------------------------------- /ucron/server.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -* 3 | from __future__ import absolute_import 4 | 5 | import os 6 | import threading 7 | 8 | from bottle import Bottle, request, template, redirect, __version__ 9 | 10 | from ucron import db, conf, stpl, worker, __version__ as version 11 | from ucron.utils import dict_encode, urlencode, dumps 12 | 13 | app = Bottle() 14 | ctx = threading.local() 15 | 16 | 17 | def flash(): 18 | tmp = getattr(ctx, 'notice', '') 19 | ctx.notice = '' 20 | return tmp 21 | 22 | 23 | @app.route('/clean') 24 | def clean_log(): 25 | worker.clean_log() 26 | return 'OK' 27 | 28 | 29 | @app.route('/reload') 30 | def reload_cron(): 31 | worker.load_crontab() 32 | ctx.notice = '重新加载成功' 33 | redirect('/status') 34 | 35 | 36 | @app.route('/add_task', 'POST') 37 | def add_task(): 38 | path = request.json.get('path') 39 | args = request.json.get('args') 40 | method = request.json.get('method') 41 | name = request.json.get('name') 42 | json = request.json.get('json') 43 | 44 | if args or json: 45 | args = dumps(args or json) if json else urlencode(dict_encode(args)) 46 | 47 | rowcount = db.task.push(path, args, method, name, int(bool(json))) 48 | return 'OK' if rowcount > 0 else 'Not Modified' 49 | 50 | 51 | @app.route('/taskq') 52 | def taskq(): 53 | opt = request.query.get('opt', 'list') 54 | rowcount = -1 55 | 56 | if opt == 'list': 57 | task = [] 58 | for item in db.taskq.fetchall(): 59 | length = db.task.length(item[0]) 60 | task.append(list(item) + [length]) 61 | return dumps(task) 62 | elif opt == 'add': 63 | name = request.query.name.strip() 64 | mode = request.query.get('mode', 'seq').strip() 65 | if name: 66 | rowcount = db.taskq.push(name, mode) 67 | if rowcount > 0: 68 | ctx.notice = '增加队列成功' 69 | else: 70 | ctx.notice = '已有同名队列' 71 | else: 72 | ctx.notice = '名称不能为空' 73 | elif opt == 'del': 74 | name = request.query.name.strip() 75 | rowcount = db.taskq.delete(name) 76 | if rowcount > 0: 77 | db.task.delete(name) 78 | ctx.notice = '删除队列成功' 79 | else: 80 | ctx.notice = '删除队列失败' 81 | elif opt == 'cls': 82 | name = request.query.name.strip() 83 | rowcount = db.task.delete(name) 84 | if rowcount > 0 or db.task.length(name) == 0: 85 | rowcount = 1 # fix cli 86 | ctx.notice = '清空队列成功' 87 | else: 88 | ctx.notice = '清空队列失败' 89 | 90 | if request.query.get('cli'): 91 | flash() 92 | return 'OK' if rowcount > 0 else 'Not Modified' 93 | 94 | redirect('/status') 95 | 96 | 97 | @app.route('/') 98 | @app.route('/status') 99 | def status(): 100 | cron = [] 101 | task = [] 102 | 103 | for item in db.cron.fetchall(): 104 | plan, status = db.status.fetch(item['id']) 105 | last = status[1:status.find(']')] 106 | status = status.split(' - ')[-1] 107 | cron.append([item['path'], plan, last, status]) 108 | 109 | for item in db.taskq.fetchall(): 110 | length = db.task.length(item[0]) 111 | task.append(list(item) + [length]) 112 | 113 | context = { 114 | 'title': '查看状态', 115 | 'cron': cron, 116 | 'task': task, 117 | 'conf': conf, 118 | 'notice': flash(), 119 | 'nav': ('/log', '查看日志'), 120 | 'version': version 121 | } 122 | 123 | return template(stpl.status, context) 124 | 125 | 126 | @app.route('/log') 127 | def log(): 128 | mode = request.query.get('mode', 'cron') 129 | sort = request.query.get('sort', 'new') 130 | page = int(request.query.page or 1) 131 | 132 | data = [] 133 | if os.path.exists(conf.log): 134 | with open(conf.log, 'rb') as f: 135 | lines = map(lambda s: s.decode('utf8'), f.readlines()) 136 | data = [line for line in lines 137 | if line.startswith(mode.title())] 138 | data = data[::-1] if sort == 'new' else data 139 | 140 | neg_sort = { 141 | 'new': {'sort': 'old', 'title': '反序查看'}, 142 | 'old': {'sort': 'new', 'title': '正序查看'} 143 | } 144 | 145 | neg_mode = { 146 | 'cron': {'mode': 'task', 'title': '队列任务'}, 147 | 'task': {'mode': 'cron', 'title': '定时任务'} 148 | } 149 | 150 | context = { 151 | 'title': '查看 %s 日志' % mode.title(), 152 | 'data': data[(page - 1) * 10: page * 10], 153 | 'mode': mode, 154 | 'page': page, 155 | 'count': len(data), 156 | 'sort': neg_sort[sort], 157 | 'other': neg_mode[mode], 158 | 'nav': ('/status', '查看状态'), 159 | 'version': version 160 | } 161 | 162 | return template(stpl.log, context) 163 | 164 | 165 | def start(): 166 | if conf.quiet: 167 | print('Bottle v%s server starting up (using WSGIRefServer())...' % __version__) 168 | print('Listening on http://%s:%s/' % (conf.host, conf.port)) 169 | print('Hit Ctrl-C to quit.') 170 | 171 | app.run(host=conf.host, port=conf.port, quiet=conf.quiet) # start bottle 172 | -------------------------------------------------------------------------------- /ucron/stpl.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -* 3 | from __future__ import absolute_import 4 | 5 | github = ''' 6 | 14 | ''' # Svg copy from httpbin.org 15 | 16 | style = '''*{-webkit-box-sizing: border-box;-moz-box-sizing: border-box;box-sizing: border-box;} 17 | a{color:#0072E3;text-decoration:none;} 18 | a:hover{text-decoration:underline;} 19 | .container{position:relative;max-width:1080px;margin:auto;} 20 | nav{position:absolute;left:0px;width:200px;} 21 | section{margin-left:210px;} 22 | .title{font-size:36px;margin:0;font-family:Open Sans,sans-serif;color:#3b4151;} 23 | .title small{font-size:10px;position:relative;top:-5px;display:inline-block;margin:0 0 0 5px;padding:2px 4px;vertical-align:super;border-radius:57px;background:#7d8492;} 24 | .title small pre{margin:0;font-family:Titillium Web,sans-serif;color:#fff;}''' 25 | 26 | nav = '''

uCron 27 | 
{{ version }}
 28 |

29 |

A micro Crontab & Task Queue for Python Web.

30 |
31 | 队列名称： 32 |
33 | 队列类型： 34 | 38 |

39 |

40 |
41 |

{{ nav[1] }} | 重载 Cron

''' 42 | 43 | status = ''' 44 | 45 | 46 | {{ title }} - uCron 47 | 55 | 56 | 57 | #github# 58 |
59 | 62 |
63 | % if notice: 64 |

{{ notice }}

65 | % end 66 | % if cron: 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | % for item in cron: 75 | 76 | 77 | 78 | % end 79 | 80 |
执行 URL 执行周期（Schedule） 上次执行时间 上次执行状态
{{ item[0] }}{{ item[1] }}{{ item[2] }}{{ item[3] }}
81 | % end 82 | % if not conf.cron: 83 |

暂无用户定时任务，请指定 --cron 参数

84 | % end 85 |
86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | % for item in task: 94 | 95 | 96 | 98 | 99 | % end 100 | 101 |
队列名称 队列类型 剩余长度（估值） 操作
{{ item[0] }}{{ item[1] }}{{ item[2] }}删除 | 97 | 清空
102 |
103 |
104 | 105 | '''.replace('#nav#', nav).replace('#style#', style).replace('#github#', github) 106 | 107 | log = ''' 108 | 109 | 110 | {{ title }} - uCron 111 | 116 | 117 | 118 | #github# 119 |
120 | 123 |
124 | % if not data: 125 |

暂无日志

126 | % end 127 | % for i, item in enumerate(data, 1): 128 | {{ item.strip() }}

129 | % end 130 |

131 | % if count > 10 * page: 132 | 下一页 | 133 | % end 134 | % if page > 1: 135 | 上一页 | 136 | % end 137 | {{ sort['title'] }} | 138 | {{ other['title'] }}

139 |
140 |
141 | 142 | '''.replace('#nav#', nav).replace('#style#', style).replace('#github#', github) 143 | -------------------------------------------------------------------------------- /ucron/utils.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -* 3 | from __future__ import absolute_import 4 | 5 | import sys 6 | from numbers import Number 7 | try: 8 | from ujson import dumps, loads 9 | except ImportError: 10 | from json import dumps, loads 11 | 12 | py3k = sys.version_info.major > 2 13 | 14 | # Lots of stdlib and builtin differences. 15 | if py3k: 16 | from urllib import request 17 | from urllib.parse import urlencode 18 | from urllib.parse import parse_qsl 19 | from urllib.error import URLError 20 | from queue import Queue 21 | unicode = str 22 | else: # 2.x 23 | import urllib2 as request 24 | from urllib import urlencode 25 | from urlparse import parse_qsl 26 | from urllib2 import URLError 27 | from Queue import Queue 28 | unicode = unicode 29 | 30 | 31 | # Some helpers for string/bytes handling 32 | def to_bytes(s, encoding='utf8'): 33 | if isinstance(s, unicode): 34 | return s.encode(encoding) 35 | if isinstance(s, (bool, Number)): 36 | return str(s).encode(encoding) 37 | return bytes('' if s is None else s) 38 | 39 | 40 | def to_unicode(s, encoding='utf8', errors='strict'): 41 | if isinstance(s, bytes): 42 | return s.decode(encoding, errors) 43 | return unicode('' if s is None else s) 44 | 45 | 46 | to_string = to_unicode if py3k else to_bytes 47 | 48 | 49 | def dict_encode(d, encoding='utf8'): 50 | return dict({(to_bytes(k, encoding), to_bytes(v, encoding)) 51 | for k, v in d.items()}) 52 | 53 | 54 | # Copy from https://github.com/webpy/webpy/blob/master/web/utils.py 55 | class IterBetter: 56 | """ 57 | Returns an object that can be used as an iterator 58 | but can also be used via __getitem__ (although it 59 | cannot go backwards -- that is, you cannot request 60 | `iterbetter[0]` after requesting `iterbetter[1]`). 61 | 62 | >>> import itertools 63 | >>> c = iterbetter(itertools.count()) 64 | >>> c[1] 65 | 1 66 | >>> c[5] 67 | 5 68 | >>> c[3] 69 | Traceback (most recent call last): 70 | ... 71 | IndexError: already passed 3 72 | It is also possible to get the first value of the iterator or None. 73 | >>> c = iterbetter(iter([3, 4, 5])) 74 | >>> print(c.first()) 75 | 3 76 | >>> c = iterbetter(iter([])) 77 | >>> print(c.first()) 78 | None 79 | For boolean test, IterBetter peeps at first value in the itertor without effecting the iteration. 80 | >>> c = iterbetter(iter(range(5))) 81 | >>> bool(c) 82 | True 83 | >>> list(c) 84 | [0, 1, 2, 3, 4] 85 | >>> c = iterbetter(iter([])) 86 | >>> bool(c) 87 | False 88 | >>> list(c) 89 | [] 90 | """ 91 | def __init__(self, iterator): 92 | self.i, self.c = iterator, 0 93 | 94 | def first(self, default=None): 95 | """Returns the first element of the iterator or None when there are no 96 | elements. 97 | If the optional argument default is specified, that is returned instead 98 | of None when there are no elements. 99 | """ 100 | try: 101 | return next(iter(self)) 102 | except StopIteration: 103 | return default 104 | 105 | def list(self): 106 | return list(self) 107 | 108 | def __iter__(self): 109 | if hasattr(self, "_head"): 110 | yield self._head 111 | 112 | while 1: 113 | try: 114 | yield next(self.i) 115 | except StopIteration: 116 | return 117 | self.c += 1 118 | 119 | def __getitem__(self, i): 120 | # todo: slices 121 | if i < self.c: 122 | raise IndexError("already passed " + str(i)) 123 | try: 124 | while i > self.c: 125 | next(self.i) 126 | self.c += 1 127 | # now self.c == i 128 | self.c += 1 129 | return next(self.i) 130 | except StopIteration: 131 | raise IndexError(str(i)) 132 | 133 | def __nonzero__(self): 134 | if hasattr(self, "__len__"): 135 | return self.__len__() != 0 136 | elif hasattr(self, "_head"): 137 | return True 138 | else: 139 | try: 140 | self._head = next(self.i) 141 | except StopIteration: 142 | return False 143 | else: 144 | return True 145 | 146 | __bool__ = __nonzero__ 147 | 148 | 149 | iterbetter = IterBetter 150 | 151 | 152 | __all__ = ['dumps', 'loads', 'py3k', 'request', 'urlencode', 'parse_qsl', 'URLError', 'Queue', 153 | 'unicode', 'to_bytes', 'to_unicode', 'to_string', 'dict_encode', 'iterbetter'] 154 | -------------------------------------------------------------------------------- /ucron/worker.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*- coding: utf-8 -* 3 | from __future__ import absolute_import 4 | 5 | import time 6 | import uuid 7 | import threading 8 | from datetime import tzinfo, timedelta, datetime 9 | 10 | from ucron import conf, db, __version__ 11 | from ucron.utils import request, urlencode, parse_qsl, Queue, to_string 12 | 13 | stdout_q = Queue() 14 | lock = threading.Lock() 15 | 16 | 17 | class UTC(tzinfo): 18 | def __init__(self, offset=0): 19 | self._offset = offset 20 | 21 | def utcoffset(self, dt): 22 | return timedelta(hours=self._offset) 23 | 24 | def tzname(self, dt): 25 | return 'UTC %+d:00' % self._offset 26 | 27 | def dst(self, dt): 28 | return timedelta(hours=self._offset) 29 | 30 | 31 | class Job: 32 | def __init__(self, args): 33 | self.path = args[0] 34 | if len(args) > 2: 35 | self.args = urlencode(parse_qsl(args[1])) 36 | self.method = args[2] 37 | elif len(args) == 2: 38 | if args[1].upper() in ['GET', 'POST']: 39 | self.args = '' 40 | self.method = args[1] 41 | else: 42 | self.args = urlencode(parse_qsl(args[1])) 43 | self.method = 'GET' 44 | else: 45 | self.args = '' 46 | self.method = 'GET' 47 | 48 | 49 | def parse_schedule(node, attr, scope, schedule): 50 | if node == '*': 51 | schedule[attr] = list(range(*scope)) 52 | elif '/' in node: 53 | node, step = node.split('/') 54 | if '-' in node: 55 | scope = list(map(int, node.split('-'))) 56 | scope = [scope[0], scope[1] + 1] 57 | scope.append(int(step)) 58 | schedule[attr] = list(range(*scope)) 59 | elif '-' in node: 60 | scope = list(map(int, node.split('-'))) 61 | scope = [scope[0], scope[1] + 1] 62 | schedule[attr] = list(range(*scope)) 63 | elif ',' in node: 64 | schedule[attr] = list(map(int, node.split(','))) 65 | else: 66 | schedule[attr] = [int(node)] 67 | 68 | 69 | def parse_crontab(line): 70 | args = line.split(' ') 71 | job_id = uuid.uuid5(uuid.NAMESPACE_DNS, line).hex 72 | cron, job = args[:5], Job(args[5:]) 73 | db.status.push(job_id, ' '.join(cron)) 74 | schedule = {} 75 | parse_schedule(cron[0], 'minute', [0, 60], schedule) 76 | parse_schedule(cron[1], 'hour', [0, 24], schedule) 77 | parse_schedule(cron[2], 'day', [1, 32], schedule) 78 | parse_schedule(cron[3], 'month', [1, 13], schedule) 79 | parse_schedule(cron[4], 'weekday', [0, 7], schedule) 80 | db.cron.push(job_id, job.path, job.args, job.method, schedule) 81 | 82 | 83 | def urlopen(path, args, method, json=False): 84 | now = datetime.now(UTC(conf.utc)).strftime('%d/%b/%Y %H:%M:%S') 85 | try: 86 | headers = {'User-Agent': 'uCron v%s' % __version__} 87 | path += '/' if path.count('/') < 3 else '' 88 | if method.upper() == 'POST' or json: 89 | method = 'POST' 90 | data = args.encode('utf8') if args else b'' 91 | if json: 92 | headers['Content-Type'] = 'application/json' 93 | else: 94 | data = None 95 | path += '?' + args if args else '' 96 | resp = request.urlopen(request.Request(path, headers=headers), data) 97 | return '[%s] %s %s - %s' % (now, path, method, resp.code) 98 | except Exception as common_ex: 99 | return '[%s] %s %s - %s' % (now, path, method, common_ex) 100 | 101 | 102 | def now(): 103 | now = datetime.now(UTC(conf.utc)) 104 | now = { 105 | 'minute': now.minute, 106 | 'hour': now.hour, 107 | 'day': now.day, 108 | 'month': now.month, 109 | 'weekday': (now.weekday() + 1) % 7 # Monday == 1 ... Saturday == 6, Sunday == 0 110 | } 111 | return now 112 | 113 | 114 | def sub_in(now, job): 115 | for key in now.keys(): 116 | if now[key] not in job[key]: 117 | return False 118 | return True 119 | 120 | 121 | def daemon_cron(): 122 | while True: 123 | for job in db.cron.fetchall(): 124 | if sub_in(now(), job): 125 | resp = urlopen(job['path'], job['args'], job['method']) 126 | db.status.update(job['id'], resp) 127 | stdout_q.put('Cron %s' % resp) 128 | time.sleep(60.1 - datetime.now().second) 129 | 130 | 131 | def run_task(task): 132 | resp = urlopen(*task[:-1]) 133 | stdout_q.put('Task %s' % resp) 134 | 135 | 136 | def select_task(name, mode): 137 | if mode == 'seq': 138 | while True: 139 | task = db.task.pop(name) 140 | if not task: 141 | break 142 | run_task(task) 143 | else: 144 | threads = [] 145 | for task in db.task.fetchall(name): 146 | threads.append(threading.Thread(target=run_task, args=(task,))) 147 | for t in threads: 148 | t.start() 149 | 150 | 151 | def daemon_task(): 152 | while True: 153 | threads = [] 154 | for name, mode in db.taskq.fetchall(): 155 | threads.append(threading.Thread(target=select_task, args=(name, mode))) 156 | for t in threads: 157 | t.start() 158 | time.sleep(0.01) 159 | 160 | 161 | def _stdout(): 162 | prefix = '%s' if conf.quiet else '* %s' 163 | while True: 164 | line = stdout_q.get() 165 | with open(conf.log, 'ab') as f: 166 | f.write(('%s\n' % line).encode('utf8')) 167 | print(prefix % line) 168 | 169 | 170 | def clean_log(): 171 | lock.acquire() 172 | with open(conf.log, 'rb') as f: 173 | lines = f.readlines() 174 | with open(conf.log, 'wb') as f: 175 | f.writelines(lines[-conf.max:]) 176 | lock.release() 177 | 178 | 179 | def load_crontab(): 180 | db.cron.empty() # empty old cron 181 | parse_crontab('%s %s/clean' % (conf.tab, conf.local)) # add clean task 182 | 183 | if not conf.cron: # nothing to do 184 | return 185 | 186 | if conf.reload: 187 | parse_crontab('* * * * * %s/reload' % conf.local) # add reload task 188 | 189 | with open(conf.cron, 'rb') as f: 190 | for line in f.readlines(): 191 | line = to_string(line).strip() 192 | if not line: 193 | break 194 | if line.startswith('#'): 195 | continue 196 | parse_crontab(line) 197 | 198 | 199 | def start(): 200 | print('uCron v%s server starting up ...' % __version__) 201 | 202 | load_crontab() 203 | db.taskq.push('default_seq', 'seq') # add default sequence queue 204 | db.taskq.push('default_con', 'con') # add default concurrence queue 205 | 206 | threads = [] 207 | for func in (daemon_cron, daemon_task, _stdout): 208 | threads.append(threading.Thread(target=func)) 209 | for t in threads: 210 | t.daemon = True 211 | t.start() 212 | --------------------------------------------------------------------------------