├── src ├── tests │ ├── __init__.py │ ├── test_utils.py │ └── test_rsmq.py └── rsmq │ ├── __init__.py │ ├── cmd │ ├── list_queues.py │ ├── __init__.py │ ├── pop_message.py │ ├── delete_queue.py │ ├── receive_message.py │ ├── change_message_visibility.py │ ├── set_queue_attributes.py │ ├── create_queue.py │ ├── delete_message.py │ ├── exceptions.py │ ├── get_queue_attributes.py │ ├── send_message.py │ ├── utils.py │ └── base_command.py │ ├── retry_delay_handler.py │ ├── const.py │ ├── rsmq.py │ └── consumer.py ├── _config.yml ├── requirements.txt ├── examples ├── run-example ├── docker-compose.yaml ├── README.md ├── example.py ├── producer.py ├── consumer.py └── consumer_thread.py ├── requirements-tests.txt ├── MANIFEST.in ├── .coveragerc ├── Pipfile ├── .project ├── package.cfg ├── .pydevproject ├── setup.py ├── .gitignore ├── .github └── workflows │ └── python-package.yml ├── LICENSE └── README.md /src/tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-leap-day -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | # Requirements 2 | redis 3 | -------------------------------------------------------------------------------- /examples/run-example: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | pip install /src 4 | 5 | cd "$(dirname "${0}")" 6 | 7 | python ${@} -------------------------------------------------------------------------------- /requirements-tests.txt: -------------------------------------------------------------------------------- 1 | # Unit test reqirements 2 | fakeredis == 1.7.1 3 | mock 4 | pylint 5 | pytest-cov 6 | six >=1.12.0 7 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include package.cfg 2 | include requirements.txt 3 | recursive-include examples *.txt *.py 4 | prune examples/sample?/build 5 | -------------------------------------------------------------------------------- /.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | branch = true 3 | omit = */test_*.py,*/*_test.py 4 | data_file = .coverage 5 | 6 | [report] 7 | 8 | 9 | [html] 10 | directory = ./reports/coverage 11 | 12 | [xml] 13 | output = ./reports/coverage.xml 14 | 15 | -------------------------------------------------------------------------------- /src/rsmq/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | Python Implementation of the Redis Simple Message Queue serverless(*) queue service 3 | 4 | (* requires only Redis server) 5 | 6 | Based on: 7 | 8 | * Original Node.js version: https://github.com/smrchy/rsmq 9 | * Java port: https://github.com/igr/jrsmq 10 | 11 | """ 12 | 13 | from .rsmq import RedisSMQ 14 | -------------------------------------------------------------------------------- /examples/docker-compose.yaml: -------------------------------------------------------------------------------- 1 | 2 | services: 3 | redis: 4 | image: redis 5 | 6 | producer: &python 7 | image: python:3.9 8 | volumes: 9 | - ..:/src 10 | - .:/examples 11 | command: [ "/examples/run-example", "producer.py", "-H", "redis" ] 12 | 13 | consumer: 14 | <<: *python 15 | command: [ "/examples/run-example", "consumer.py", "-H", "redis" ] -------------------------------------------------------------------------------- /Pipfile: -------------------------------------------------------------------------------- 1 | [[source]] 2 | name = "pypi" 3 | url = "https://pypi.org/simple" 4 | verify_ssl = true 5 | 6 | [scripts] 7 | test = 'python -m pytest --junitxml ./reports/results.xml --cov-config .coveragerc --cov=src .' 8 | 9 | [dev-packages] 10 | fakeredis = "1.7.1" 11 | pytest-cov = "*" 12 | pylint = "*" 13 | mock = "*" 14 | 15 | [packages] 16 | redis = ">=3.0.0" 17 | 18 | [requires] 19 | -------------------------------------------------------------------------------- /.project: -------------------------------------------------------------------------------- 1 | 2 | 3 | PyRSMQ 4 | 5 | 6 | 7 | 8 | 9 | org.python.pydev.PyDevBuilder 10 | 11 | 12 | 13 | 14 | 15 | org.python.pydev.pythonNature 16 | 17 | 18 | -------------------------------------------------------------------------------- /package.cfg: -------------------------------------------------------------------------------- 1 | [Package] 2 | version = 0.6.1 3 | name = PyRSMQ 4 | description = Python Implementation of Redis SMQ 5 | url = https://mlasevich.github.io/PyRSMQ/ 6 | author = Michael Lasevich 7 | author_email = mlasevich+pyrsmq@gmail.com 8 | license = Apache 2.0 9 | 10 | [FindPackages] 11 | where = ./src 12 | exclude = *.tests, 13 | *.tests.* 14 | tests.* 15 | tests 16 | *_test.py 17 | test_* 18 | -------------------------------------------------------------------------------- /examples/README.md: -------------------------------------------------------------------------------- 1 | # Examples: 2 | 3 | Example code 4 | 5 | ## Producer 6 | 7 | Producer example send messages into the queue by creating a RedisSMQ controller and using 8 | `sendMessage()` calls 9 | 10 | Use -h flag in producer command line to see all the options 11 | 12 | ## Consumer 13 | 14 | Consumer example uses RedisSMQConsumer processor and creates a thread to control it 15 | 16 | Use -h flag in producer command line to see all the options -------------------------------------------------------------------------------- /.pydevproject: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | /${PROJECT_DIR_NAME}/src 5 | 6 | python 3.6 7 | PyRSMQ (pipenv) 8 | 9 | -------------------------------------------------------------------------------- /src/rsmq/cmd/list_queues.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | """ 4 | 5 | from .base_command import BaseRSMQCommand 6 | 7 | 8 | class ListQueuesCommand(BaseRSMQCommand): 9 | """ 10 | List Queues. 11 | 12 | On execution returns a set of queues already existing on the redis in this namespace 13 | """ 14 | 15 | PARAMS = {"quiet": {"required": False, "value": False}} 16 | 17 | def exec_command(self): 18 | """Exec Command""" 19 | client = self.client 20 | ret = client.smembers(self.queue_set) 21 | return ret 22 | -------------------------------------------------------------------------------- /src/rsmq/cmd/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | Commands 3 | """ 4 | 5 | from .change_message_visibility import ChangeMessageVisibilityCommand 6 | from .create_queue import CreateQueueCommand 7 | from .delete_message import DeleteMessageCommand 8 | from .delete_queue import DeleteQueueCommand 9 | from .exceptions import * 10 | from .get_queue_attributes import GetQueueAttributesCommand 11 | from .list_queues import ListQueuesCommand 12 | from .pop_message import PopMessageCommand 13 | from .receive_message import ReceiveMessageCommand 14 | from .send_message import SendMessageCommand 15 | from .set_queue_attributes import SetQueueAttributesCommand 16 | -------------------------------------------------------------------------------- /src/rsmq/cmd/pop_message.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | """ 4 | 5 | from .base_command import BaseRSMQCommand 6 | from .exceptions import NoMessageInQueue 7 | 8 | 9 | class PopMessageCommand(BaseRSMQCommand): 10 | """ 11 | Receive Message and delete it 12 | """ 13 | 14 | PARAMS = { 15 | "qname": {"required": True, "value": None}, 16 | "quiet": {"required": False, "value": False}, 17 | } 18 | 19 | def exec_command(self): 20 | """Execute""" 21 | client = self.client 22 | 23 | queue_base = self.queue_base 24 | queue = self.queue_def() 25 | 26 | ts = int(queue["ts"]) 27 | result = client.evalsha(self.popMessageSha1, 2, queue_base, ts) 28 | if not result: 29 | raise NoMessageInQueue(self.get_qname) 30 | [message_id, message, rc, ts] = result 31 | return {"id": message_id, "message": message, "rc": rc, "ts": int(ts)} 32 | -------------------------------------------------------------------------------- /src/rsmq/retry_delay_handler.py: -------------------------------------------------------------------------------- 1 | """ 2 | Utility to handle exp retry delay 3 | """ 4 | import logging 5 | import time 6 | 7 | LOG = logging.getLogger(__name__) 8 | 9 | 10 | class RetryDelayHandler: 11 | """ Tool to manage exponential retry delays """ 12 | 13 | def __init__(self, min_delay: float = 0, max_delay: float = 60): 14 | """ Initialize """ 15 | self.current_delay = min_delay 16 | self.min_delay = min_delay 17 | self.max_delay = max_delay 18 | 19 | def reset(self): 20 | """ Reset delay """ 21 | self.current_delay = self.min_delay 22 | 23 | def delay(self): 24 | """ Perform delay """ 25 | if self.current_delay: 26 | LOG.debug("Retrying in %s seconds", self.current_delay) 27 | time.sleep(self.current_delay) 28 | self.current_delay = min(self.current_delay * 2, self.max_delay) 29 | else: 30 | self.current_delay = 1.0 31 | -------------------------------------------------------------------------------- /src/rsmq/cmd/delete_queue.py: -------------------------------------------------------------------------------- 1 | """ 2 | Delete Queue Command 3 | """ 4 | 5 | from .base_command import BaseRSMQCommand 6 | from .exceptions import QueueDoesNotExist 7 | 8 | 9 | class DeleteQueueCommand(BaseRSMQCommand): 10 | """ 11 | Delete Queue if it exists 12 | """ 13 | 14 | PARAMS = { 15 | "qname": {"required": True, "value": None}, 16 | "quiet": {"required": False, "value": False}, 17 | } 18 | 19 | def exec_command(self): 20 | """Exec Command""" 21 | client = self.client 22 | 23 | queue_name = self.queue_base 24 | queue_key = self.queue_key 25 | tx = client.pipeline(transaction=True) 26 | tx.delete(queue_key) 27 | tx.delete(queue_name) 28 | tx.srem(self.queue_set, self.get_qname) 29 | results = tx.execute() 30 | if True not in results: 31 | raise QueueDoesNotExist(self.get_qname) 32 | self.log.debug("Deleted Queue %s", self.queue_base) 33 | return True 34 | -------------------------------------------------------------------------------- /src/rsmq/cmd/receive_message.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | """ 4 | 5 | from .base_command import BaseRSMQCommand 6 | from .exceptions import NoMessageInQueue 7 | 8 | 9 | class ReceiveMessageCommand(BaseRSMQCommand): 10 | """ 11 | Receive Message 12 | """ 13 | 14 | PARAMS = { 15 | "qname": {"required": True, "value": None}, 16 | "vt": {"required": False, "value": None}, 17 | "quiet": {"required": False, "value": False}, 18 | } 19 | 20 | def exec_command(self): 21 | """Execute""" 22 | client = self.client 23 | 24 | queue_base = self.queue_base 25 | queue = self.queue_def() 26 | 27 | ts = int(queue["ts"]) 28 | vt = float(queue["vt"] if self.get_vt is None else self.get_vt) 29 | vtimeout = ts + int(round(vt * 1000)) 30 | result = client.evalsha(self.receiveMessageSha1, 3, queue_base, ts, vtimeout) 31 | if not result: 32 | raise NoMessageInQueue(self.get_qname) 33 | [message_id, message, rc, ts] = result 34 | return {"id": message_id, "message": message, "rc": rc, "ts": int(ts)} 35 | -------------------------------------------------------------------------------- /src/rsmq/cmd/change_message_visibility.py: -------------------------------------------------------------------------------- 1 | """ 2 | Change Message Visibility 3 | """ 4 | 5 | from .base_command import BaseRSMQCommand 6 | from .exceptions import NoMessageInQueue 7 | 8 | 9 | class ChangeMessageVisibilityCommand(BaseRSMQCommand): 10 | """ 11 | Change Message Visibility Timeout Command 12 | """ 13 | 14 | PARAMS = { 15 | "qname": {"required": True, "value": None}, 16 | "id": {"required": True, "value": None}, 17 | "vt": {"required": False, "value": None}, 18 | "quiet": {"required": False, "value": False}, 19 | } 20 | 21 | def exec_command(self): 22 | """Execute""" 23 | client = self.client 24 | 25 | queue_base = self.queue_base 26 | queue = self.queue_def() 27 | 28 | ts = int(queue["ts"]) 29 | vt = float(queue["vt"] if self.get_vt is None else self.get_vt) 30 | vtimeout = ts + int(round(vt * 1000)) 31 | result = client.evalsha( 32 | self.changeMessageVisibilitySha1, 3, queue_base, self.get_id, vtimeout 33 | ) 34 | if not result: 35 | raise NoMessageInQueue(self.get_qname) 36 | return result == 1 37 | -------------------------------------------------------------------------------- /src/rsmq/cmd/set_queue_attributes.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | """ 4 | import time 5 | from .base_command import BaseRSMQCommand 6 | 7 | 8 | class SetQueueAttributesCommand(BaseRSMQCommand): 9 | """ 10 | Get Queue Attributes if does not exist 11 | """ 12 | 13 | PARAMS = { 14 | "qname": {"required": True, "value": None}, 15 | "vt": {"required": False, "value": None}, 16 | "delay": {"required": False, "value": None}, 17 | "maxsize": {"required": False, "value": None}, 18 | "quiet": {"required": False, "value": False}, 19 | } 20 | 21 | def exec_command(self): 22 | """Exec Command""" 23 | now = int(time.time()) 24 | 25 | queue_key = self.queue_key 26 | 27 | tx = self.client.pipeline(transaction=True) 28 | for param in ["vt", "delay", "maxsize"]: 29 | value = self.param_get(param, default_value=None) 30 | if value is not None: 31 | tx.hset(queue_key, param, value) 32 | if tx: 33 | self.log.debug("Applying queue attribute changes") 34 | tx.hset(queue_key, "modified", now) 35 | else: 36 | self.log.debug("No queue attribute changes") 37 | _result = tx.execute() 38 | 39 | return self.parent.getQueueAttributes().qname(self.get_qname).execute() 40 | -------------------------------------------------------------------------------- /src/rsmq/cmd/create_queue.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | """ 4 | import time 5 | 6 | from .. import const 7 | from .base_command import BaseRSMQCommand 8 | from .exceptions import QueueAlreadyExists 9 | 10 | 11 | class CreateQueueCommand(BaseRSMQCommand): 12 | """ 13 | Create Queue if does not exist 14 | """ 15 | 16 | PARAMS = { 17 | "qname": {"required": True, "value": None}, 18 | "vt": {"required": True, "value": const.VT_DEFAULT}, 19 | "delay": {"required": True, "value": const.DELAY_DEFAULT}, 20 | "maxsize": {"required": True, "value": const.MAXSIZE_DEFAULT}, 21 | "quiet": {"required": False, "value": False}, 22 | } 23 | 24 | def exec_command(self): 25 | """Exec Command""" 26 | client = self.client 27 | now = int(time.time()) 28 | 29 | key = self.queue_key 30 | tx = client.pipeline(transaction=True) 31 | tx.hsetnx(key, "vt", self.get_vt) 32 | tx.hsetnx(key, "delay", self.get_delay) 33 | tx.hsetnx(key, "maxsize", self.get_maxsize) 34 | tx.hsetnx(key, "created", now) 35 | tx.hsetnx(key, "modified", now) 36 | results = tx.execute() 37 | if True not in results: 38 | raise QueueAlreadyExists(self.get_qname) 39 | client.sadd(self.queue_set, self.get_qname) 40 | return True 41 | -------------------------------------------------------------------------------- /src/rsmq/cmd/delete_message.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | """ 4 | 5 | from .base_command import BaseRSMQCommand 6 | 7 | 8 | class DeleteMessageCommand(BaseRSMQCommand): 9 | """ 10 | Delete Message if it exists 11 | """ 12 | 13 | PARAMS = { 14 | "qname": {"required": True, "value": None}, 15 | "id": {"required": True, "value": None}, 16 | "quiet": {"required": False, "value": False}, 17 | } 18 | 19 | def exec_command(self): 20 | """Exec Command""" 21 | result = self.get_transaction().execute() 22 | 23 | # 1 key deleted from zset 24 | # 3 keys deleted from hash (message itself, receive count, first receive field) 25 | if int(result[0]) == 1 and int(result[1]) == 3: 26 | return True 27 | 28 | return False 29 | 30 | def get_transaction(self): 31 | """Returns a transaction (pipeline), pre-populated with the deleteMessage commands""" 32 | client = self.client 33 | queue_base = self.queue_base 34 | queue_key = self.queue_key 35 | message_id = self.get_id 36 | # decode to string when provided as bytes 37 | if isinstance(message_id, bytes): 38 | message_id = message_id.decode('utf-8') 39 | tx = client.pipeline(transaction=True) 40 | tx.zrem(queue_base, message_id) 41 | tx.hdel(queue_key, message_id, "%s:rc" % message_id, "%s:fr" % message_id) 42 | return tx 43 | -------------------------------------------------------------------------------- /examples/example.py: -------------------------------------------------------------------------------- 1 | from pprint import pprint 2 | import time 3 | 4 | from rsmq.rsmq import RedisSMQ 5 | 6 | 7 | # Create controller. 8 | # In this case we are specifying the host and default queue name 9 | queue = RedisSMQ(host="127.0.0.1", qname="myqueue") 10 | 11 | 12 | # Delete Queue if it already exists, ignoring exceptions 13 | queue.deleteQueue().exceptions(False).execute() 14 | 15 | # Create Queue with default visibility timeout of 20 and delay of 0 16 | # demonstrating here both ways of setting parameters 17 | queue.createQueue(delay=0).vt(20).execute() 18 | 19 | # Send a message with a 2 second delay 20 | message_id = queue.sendMessage(delay=2).message("Hello World").execute() 21 | 22 | pprint({"queue_status": queue.getQueueAttributes().execute()}) 23 | 24 | # Try to get a message - this will not succeed, as our message has a delay and no other 25 | # messages are in the queue 26 | msg = queue.receiveMessage().exceptions(False).execute() 27 | 28 | # Message should be False as we got no message 29 | pprint({"Message": msg}) 30 | 31 | print("Waiting for our message to become visible") 32 | # Wait for our message to become visible 33 | time.sleep(2) 34 | 35 | pprint({"queue_status": queue.getQueueAttributes().execute()}) 36 | # Get our message 37 | msg = queue.receiveMessage().execute() 38 | 39 | # Message should now be there 40 | pprint({"Message": msg}) 41 | 42 | # Delete Message 43 | queue.deleteMessage(id=msg["id"]) 44 | 45 | pprint({"queue_status": queue.getQueueAttributes().execute()}) 46 | # delete our queue 47 | queue.deleteQueue().execute() 48 | 49 | # No action 50 | queue.quit() 51 | -------------------------------------------------------------------------------- /src/rsmq/cmd/exceptions.py: -------------------------------------------------------------------------------- 1 | """ 2 | Command Exceptions 3 | """ 4 | 5 | 6 | class RedisSMQException(Exception): 7 | """Base Exception for all RQSM Commands""" 8 | 9 | def __init__(self, msg=None): 10 | """Default constructor""" 11 | if msg is None: 12 | msg = self.__doc__ 13 | super(RedisSMQException, self).__init__(msg) 14 | 15 | 16 | class CommandNotImplementedException(RedisSMQException): 17 | """Command is not yet implemented""" 18 | 19 | 20 | class InvalidParameterValue(RedisSMQException): 21 | """Invalid Parameter Value""" 22 | 23 | def __init__(self, name, value): 24 | """Constructor""" 25 | super(InvalidParameterValue, self).__init__( 26 | "Value '%s' is not valid for parameter '%s'" % (value, name) 27 | ) 28 | 29 | 30 | class QueueAlreadyExists(RedisSMQException): 31 | """Queue Already Exists""" 32 | 33 | def __init__(self, name): 34 | """Constructor""" 35 | super(QueueAlreadyExists, self).__init__("Queue '%s' already exists" % name) 36 | 37 | 38 | class QueueDoesNotExist(RedisSMQException): 39 | """Queue Already Exists""" 40 | 41 | def __init__(self, name): 42 | """Constructor""" 43 | super(QueueDoesNotExist, self).__init__("Queue '%s' does not exist" % name) 44 | 45 | 46 | class NoMessageInQueue(RedisSMQException): 47 | """Queue Already Exists""" 48 | 49 | def __init__(self, name): 50 | """Constructor""" 51 | super(NoMessageInQueue, self).__init__( 52 | "Queue '%s' has no messages waiting" % name 53 | ) 54 | -------------------------------------------------------------------------------- /src/rsmq/cmd/get_queue_attributes.py: -------------------------------------------------------------------------------- 1 | """ 2 | GetQueueAttributes Command 3 | """ 4 | 5 | import time 6 | 7 | from .base_command import BaseRSMQCommand 8 | from .exceptions import QueueDoesNotExist 9 | 10 | 11 | class GetQueueAttributesCommand(BaseRSMQCommand): 12 | """ 13 | Get Queue Attributes from existing queue 14 | """ 15 | 16 | PARAMS = { 17 | "qname": {"required": True, "value": None}, 18 | "quiet": {"required": False, "value": False}, 19 | } 20 | 21 | def exec_command(self): 22 | """Exec Command""" 23 | secs, usecs = self.client.time() 24 | now = secs * 1000 + int(usecs / 1000) 25 | 26 | queue_base = self.queue_base 27 | queue_key = self.queue_key 28 | 29 | tx = self.client.pipeline(transaction=True) 30 | tx.hmget( 31 | queue_key, 32 | "vt", 33 | "delay", 34 | "maxsize", 35 | "totalrecv", 36 | "totalsent", 37 | "created", 38 | "modified", 39 | ) 40 | tx.zcard(queue_base) 41 | tx.zcount(queue_base, now, "+inf") 42 | 43 | results = tx.execute() 44 | 45 | if not results or results[0][0] is None: 46 | raise QueueDoesNotExist(self.get_qname) 47 | 48 | stats = results[0] 49 | 50 | return { 51 | "vt": float(stats[0]), 52 | "delay": float(stats[1]), 53 | "maxsize": int(stats[2]), 54 | "totalrecv": int(stats[3] or 0), 55 | "totalsent": int(stats[4] or 0), 56 | "created": int(stats[5]), 57 | "modified": int(stats[6]), 58 | "msgs": results[1], 59 | "hiddenmsgs": results[2], 60 | } 61 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | """ 2 | Note that bulk of the configuation is in the package.cfg file 3 | """ 4 | try: 5 | import ConfigParser as configparser 6 | except ImportError: 7 | import configparser 8 | 9 | import os.path 10 | from pprint import pprint 11 | 12 | from setuptools import setup, find_packages 13 | 14 | 15 | BASEDIR = os.path.dirname(__file__) 16 | 17 | requirements = [] 18 | with open(os.path.join(BASEDIR, "requirements.txt"), "r") as f: 19 | requirements = f.readlines() 20 | 21 | with open("README.md", "r") as fh: 22 | long_description = fh.read() 23 | 24 | 25 | # Read Package info from a CONFIG file package.cfg 26 | CONFIG = configparser.ConfigParser({}) 27 | CONFIG.add_section("Package") 28 | CONFIG.add_section("FindPackages") 29 | CONFIG.read(os.path.join(BASEDIR, "package.cfg")) 30 | PKG_INFO = dict(CONFIG.items("Package")) 31 | 32 | FIND_PKGS = dict(CONFIG.items("FindPackages")) 33 | 34 | for item in ["include", "exclude"]: 35 | if item in FIND_PKGS: 36 | FIND_PKGS[item] = FIND_PKGS[item].split(",") 37 | 38 | PKG_INFO.update( 39 | { 40 | "packages": find_packages(**(FIND_PKGS)), 41 | "package_dir": {"": "src"}, 42 | "package_data": {"": ["/package.cfg*"]}, 43 | "install_requires": [ 44 | req.strip() for req in requirements if not req.startswith("#") 45 | ], 46 | "long_description": long_description, 47 | "long_description_content_type": "text/markdown", 48 | "python_requires": '>=3.6', 49 | "classifiers": [ 50 | "Programming Language :: Python :: 3", 51 | "License :: OSI Approved :: Apache Software License", 52 | "Operating System :: OS Independent", 53 | ], 54 | } 55 | ) 56 | 57 | setup(**PKG_INFO) 58 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | *.egg-info/ 24 | .installed.cfg 25 | *.egg 26 | MANIFEST 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | /reports/ 40 | htmlcov/ 41 | .tox/ 42 | .coverage 43 | .coverage.* 44 | .cache 45 | nosetests.xml 46 | coverage.xml 47 | *.cover 48 | .hypothesis/ 49 | .pytest_cache/ 50 | 51 | # Translations 52 | *.mo 53 | *.pot 54 | 55 | # Django stuff: 56 | *.log 57 | local_settings.py 58 | db.sqlite3 59 | 60 | # Flask stuff: 61 | instance/ 62 | .webassets-cache 63 | 64 | # Scrapy stuff: 65 | .scrapy 66 | 67 | # Sphinx documentation 68 | docs/_build/ 69 | 70 | # PyBuilder 71 | target/ 72 | 73 | # Jupyter Notebook 74 | .ipynb_checkpoints 75 | 76 | # pyenv 77 | .python-version 78 | 79 | # celery beat schedule file 80 | celerybeat-schedule 81 | 82 | # SageMath parsed files 83 | *.sage.py 84 | 85 | # Environments 86 | .env 87 | .venv 88 | env/ 89 | venv/ 90 | ENV/ 91 | env.bak/ 92 | venv.bak/ 93 | 94 | # Spyder project settings 95 | .spyderproject 96 | .spyproject 97 | 98 | # Rope project settings 99 | .ropeproject 100 | 101 | # mkdocs documentation 102 | /site 103 | 104 | # mypy 105 | .mypy_cache/ 106 | .venv* 107 | -------------------------------------------------------------------------------- /src/rsmq/cmd/send_message.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | """ 4 | 5 | from .base_command import BaseRSMQCommand 6 | from .utils import make_message_id, encode_message 7 | 8 | 9 | class SendMessageCommand(BaseRSMQCommand): 10 | """ 11 | Create Queue if does not exist 12 | """ 13 | 14 | PARAMS = { 15 | "qname": {"required": True, "value": None}, 16 | "message": {"required": True, "value": None}, 17 | "delay": {"required": False, "value": None}, 18 | "quiet": {"required": False, "value": False}, 19 | "encode": {"required": False, "value": False}, 20 | } 21 | 22 | def exec_command(self): 23 | """ 24 | Execute command 25 | 26 | @raise QueueDoesNotExist if queue does not exist 27 | """ 28 | queue = self.queue_def() 29 | message_id = make_message_id(queue.get("ts_usec", None)) 30 | 31 | queue_key = self.queue_key 32 | queue_base = self.queue_base 33 | 34 | ts = int(queue["ts"]) 35 | 36 | delay = self.get_delay 37 | if delay is None: 38 | delay = queue.get("delay", 0) 39 | delay = float(delay or 0) 40 | 41 | message = self.get_message 42 | if self.get_encode or not isinstance(message, (str, bytes)): 43 | message = encode_message(message) 44 | self.log.debug("Encoded message: %s", message) 45 | 46 | tx = self.client.pipeline(transaction=True) 47 | timestamp = ts + int(round(delay * 1000)) 48 | self.log.debug("tx.zadd(%s, %s, %s)", queue_base, timestamp, message_id) 49 | tx.zadd(queue_base, {message_id: timestamp}) 50 | 51 | tx.hset(queue_key, message_id, message) 52 | tx.hincrby(queue_key, "totalsent", 1) 53 | _results = tx.execute() 54 | 55 | return message_id 56 | -------------------------------------------------------------------------------- /src/rsmq/const.py: -------------------------------------------------------------------------------- 1 | """ 2 | Constant values 3 | """ 4 | 5 | # max length of queue name 6 | QNAME_MAX_LEN = 64 7 | 8 | QNAME_INVALID_CHARS_RE = r"[^A-Za-z0-9._-]" 9 | 10 | # Suffix to append to the queue 11 | QUEUE_SUFFUX = ":Q" 12 | 13 | # suffix to append to the queue set 14 | QUEUES = "QUEUES" 15 | 16 | # minimum VT - seconds 17 | VT_MIN = 0 18 | 19 | # maximum VT - seconds 20 | VT_MAX = 9999999 21 | 22 | VT_DEFAULT = 30 23 | 24 | 25 | # minimum DELAY - seconds 26 | DELAY_MIN = 0 27 | 28 | # maximum DELAY - seconds 29 | DELAY_MAX = 9999999 30 | 31 | DELAY_DEFAULT = 0 32 | 33 | 34 | MAXSIZE_MIN = 1024 35 | MAXSIZE_MAX = 65565 36 | MAXSIZE_DEFAULT = MAXSIZE_MAX 37 | 38 | 39 | # pylint: disable = C0301 40 | SCRIPT_POPMESSAGE = 'local msg = redis.call("ZRANGEBYSCORE", KEYS[1], "-inf", KEYS[2], "LIMIT", "0", "1") if #msg == 0 then return {} end redis.call("HINCRBY", KEYS[1] .. ":Q", "totalrecv", 1) local mbody = redis.call("HGET", KEYS[1] .. ":Q", msg[1]) local rc = redis.call("HINCRBY", KEYS[1] .. ":Q", msg[1] .. ":rc", 1) local o = {msg[1], mbody, rc} if rc==1 then table.insert(o, KEYS[2]) else local fr = redis.call("HGET", KEYS[1] .. ":Q", msg[1] .. ":fr") table.insert(o, fr) end redis.call("ZREM", KEYS[1], msg[1]) redis.call("HDEL", KEYS[1] .. ":Q", msg[1], msg[1] .. ":rc", msg[1] .. ":fr") return o' 41 | SCRIPT_RECEIVEMESSAGE = 'local msg = redis.call("ZRANGEBYSCORE", KEYS[1], "-inf", KEYS[2], "LIMIT", "0", "1") if #msg == 0 then return {} end redis.call("ZADD", KEYS[1], KEYS[3], msg[1]) redis.call("HINCRBY", KEYS[1] .. ":Q", "totalrecv", 1) local mbody = redis.call("HGET", KEYS[1] .. ":Q", msg[1]) local rc = redis.call("HINCRBY", KEYS[1] .. ":Q", msg[1] .. ":rc", 1) local o = {msg[1], mbody, rc} if rc==1 then redis.call("HSET", KEYS[1] .. ":Q", msg[1] .. ":fr", KEYS[2]) table.insert(o, KEYS[2]) else local fr = redis.call("HGET", KEYS[1] .. ":Q", msg[1] .. ":fr") table.insert(o, fr) end return o' 42 | SCRIPT_CHANGEMESSAGEVISIBILITY = 'local msg = redis.call("ZSCORE", KEYS[1], KEYS[2]) if not msg then return 0 end redis.call("ZADD", KEYS[1], KEYS[3], KEYS[2]) return 1' 43 | -------------------------------------------------------------------------------- /src/rsmq/cmd/utils.py: -------------------------------------------------------------------------------- 1 | """ Utilities """ 2 | import json 3 | import random 4 | 5 | DEFAULT_CHARSET = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789" 6 | 7 | 8 | def validate_int(value, min_value=None, max_value=None, logger=None, name=None): 9 | """Validate value is integer and between min and max values (if specified)""" 10 | if value is None: 11 | if logger and name: 12 | logger.debug("%s value is not set", name) 13 | return False 14 | try: 15 | int_value = int(value) 16 | if min_value is not None and int_value < min_value: 17 | if logger and name: 18 | logger.debug( 19 | "%s value %s is less than minimum (%s)", name, int_value, min_value 20 | ) 21 | return False 22 | if max_value is not None and int_value > max_value: 23 | if logger and name: 24 | logger.debug( 25 | "%s value %s is greater than maximum (%s)", 26 | name, 27 | int_value, 28 | max_value, 29 | ) 30 | return False 31 | except ValueError: 32 | if logger and name: 33 | logger.debug("%s value (%s) is not an integer", name, value) 34 | return False 35 | return True 36 | 37 | 38 | def baseXencode(value, chars="0123456789abcdefghijklmnopqrstuvwxyz"): 39 | """ 40 | Converts an integer to a base X string using charset. 41 | Base is implied by charset = base36 by default 42 | 43 | Based on: https://en.wikipedia.org/wiki/Base36 44 | 45 | raises ValueError if value cannot be converted to integer 46 | """ 47 | 48 | base = len(chars) 49 | integer = int(value) 50 | sign = "-" if integer < 0 else "" 51 | integer = abs(integer) 52 | result = "" 53 | 54 | while integer > 0: 55 | integer, remainder = divmod(integer, base) 56 | result = chars[remainder] + result 57 | 58 | return sign + result 59 | 60 | 61 | def random_string(length, charset=None): 62 | """generate a random string of characters from charset""" 63 | if not charset: 64 | charset = DEFAULT_CHARSET 65 | 66 | string = "" 67 | for _ in range(length): 68 | string += random.choice(charset) 69 | return string 70 | 71 | 72 | def make_message_id(usec): 73 | """Create a message id based on Redis time""" 74 | return baseXencode(usec) + random_string(22) 75 | 76 | 77 | def encode_message(msg): 78 | """Encode message to JSON if not already string""" 79 | if isinstance(msg, str): 80 | return msg 81 | return json.dumps(msg) 82 | 83 | 84 | def decode_message(msg): 85 | """Decode message from JSON if decodable, else return message as is""" 86 | if isinstance(msg, str): 87 | try: 88 | return json.loads(msg) 89 | except json.decoder.JSONDecodeError: 90 | pass 91 | return msg 92 | -------------------------------------------------------------------------------- /.github/workflows/python-package.yml: -------------------------------------------------------------------------------- 1 | # This workflow will install Python dependencies, run tests and lint with a variety of Python versions 2 | # For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python 3 | 4 | name: Python package 5 | 6 | on: 7 | push: 8 | branches: [ "master" ] 9 | tags: 10 | - 'v*' 11 | 12 | pull_request: 13 | branches: [ "master" ] 14 | 15 | jobs: 16 | build: 17 | runs-on: ubuntu-latest 18 | strategy: 19 | fail-fast: false 20 | matrix: 21 | python-version: ["3.9"] #, "3.10", "3.11"] 22 | 23 | steps: 24 | - uses: actions/checkout@v4 25 | - name: Set up Python ${{ matrix.python-version }} 26 | uses: actions/setup-python@v3 27 | with: 28 | python-version: ${{ matrix.python-version }} 29 | - name: Install dependencies 30 | run: | 31 | python -m pip install --upgrade pip 32 | pip install -r requirements-tests.txt 33 | pip install -r requirements.txt 34 | # pip install coveralls 35 | #python -m pip install flake8 pytest 36 | #if [ -f requirements.txt ]; then pip install -r requirements.txt; fi 37 | #- name: Lint with flake8 38 | # run: | 39 | # # stop the build if there are Python syntax errors or undefined names 40 | # #flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics 41 | # # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide 42 | # #flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics 43 | - name: Test with pytest 44 | run: | 45 | python -m pytest --junitxml ./reports/results.xml --cov-config .coveragerc --cov=src 46 | - name: Install pypa/build 47 | run: >- 48 | python3 -m 49 | pip install 50 | build 51 | --user 52 | - name: Build a binary wheel and a source tarball 53 | run: python3 -m build 54 | - name: Store the distribution packages 55 | uses: actions/upload-artifact@v4 56 | with: 57 | name: python-package-distributions 58 | path: dist/ 59 | 60 | publish-to-pypi: 61 | name: Upload release to PyPI 62 | if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags') 63 | runs-on: ubuntu-latest 64 | environment: 65 | name: pypi 66 | url: https://pypi.org/p/PyRSMQ 67 | permissions: 68 | id-token: write # IMPORTANT: this permission is mandatory for trusted publishing 69 | steps: 70 | # retrieve your distributions here 71 | - name: Download all the dists 72 | uses: actions/download-artifact@v4.1.7 73 | with: 74 | name: python-package-distributions 75 | path: dist/ 76 | - name: Publish package distributions to PyPI 77 | uses: pypa/gh-action-pypi-publish@release/v1 78 | 79 | # github-release: 80 | # if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags') 81 | # name: >- 82 | # Sign the Python 🐍 distribution 📦 with Sigstore 83 | # and upload them to GitHub Release 84 | # needs: 85 | # - publish-to-pypi 86 | # runs-on: ubuntu-latest 87 | 88 | # permissions: 89 | # contents: write # IMPORTANT: mandatory for making GitHub Releases 90 | # id-token: write # IMPORTANT: mandatory for sigstore 91 | 92 | # steps: 93 | # - name: Download all the dists 94 | # uses: actions/download-artifact@v4.1.7 95 | # with: 96 | # name: python-package-distributions 97 | # path: dist/ 98 | # - name: Sign the dists with Sigstore 99 | # uses: sigstore/gh-action-sigstore-python@v2.1.1 100 | # with: 101 | # inputs: >- 102 | # ./dist/*.tar.gz 103 | # ./dist/*.whl 104 | # - name: Create GitHub Release 105 | # env: 106 | # GITHUB_TOKEN: ${{ github.token }} 107 | # run: >- 108 | # gh release create 109 | # '${{ github.ref_name }}' 110 | # --repo '${{ github.repository }}' 111 | # --notes "" 112 | # - name: Upload artifact signatures to GitHub Release 113 | # env: 114 | # GITHUB_TOKEN: ${{ github.token }} 115 | # # Upload to GitHub Release using the `gh` CLI. 116 | # # `dist/` contains the built packages, and the 117 | # # sigstore-produced signatures and certificates. 118 | # run: >- 119 | # gh release upload 120 | # '${{ github.ref_name }}' dist/** 121 | # --repo '${{ github.repository }}' 122 | -------------------------------------------------------------------------------- /examples/producer.py: -------------------------------------------------------------------------------- 1 | """ 2 | Example Producer for a queue 3 | 4 | 5 | In this example we create RedisSMQ controller and using it to send messages 6 | to that at regular intervals 7 | 8 | """ 9 | import argparse 10 | import logging 11 | import os 12 | import sys 13 | import time 14 | 15 | from rsmq import RedisSMQ 16 | 17 | LOG = logging.getLogger("Producer") 18 | 19 | MAX_DELAY = 30 20 | 21 | redis_host = os.environ.get("REDIS_HOST", "127.0.0.1") 22 | 23 | 24 | def create_message(msg_number): 25 | """Create a message to send""" 26 | return {"item": msg_number, "ts": time.time()} 27 | 28 | 29 | def produce(rsmq, long_qname, count, interval): 30 | LOG.info( 31 | "Starting producer for queue '%s' - sending %s messages every %s " 32 | "seconds...", 33 | long_qname, 34 | count, 35 | interval, 36 | ) 37 | 38 | msg_count = 0 39 | while True: 40 | msg_count += 1 41 | msg = create_message(msg_count) 42 | rsmq.sendMessage(message=msg, encode=True).execute() 43 | if count > 0 and msg_count >= count: 44 | # Stop if we are done 45 | break 46 | LOG.debug("Waiting %s seconds to send next message", interval) 47 | time.sleep(interval) 48 | LOG.info("Ended producer after sending %s messages.", msg_count) 49 | 50 | 51 | def loop(argv=None): 52 | if argv is None: 53 | argv = sys.argv 54 | """ Parse args and run producer """ 55 | parser = argparse.ArgumentParser() 56 | parser.add_argument( 57 | "-q", 58 | "--queue", 59 | dest="queue", 60 | action="store", 61 | default="queue", 62 | help="queue name [default: %(default)s]", 63 | ) 64 | parser.add_argument( 65 | "-n", 66 | "--namespace", 67 | dest="ns", 68 | action="store", 69 | default="test", 70 | help="queue namespace [default: %(default)s]", 71 | ) 72 | 73 | parser.add_argument( 74 | "-c", 75 | "--count", 76 | dest="count", 77 | action="store", 78 | type=int, 79 | default=0, 80 | help="number of messages to send. If less than 1, send continuously " 81 | + "[default: %(default)s]", 82 | ) 83 | 84 | parser.add_argument( 85 | "-i", 86 | "--interval", 87 | dest="interval", 88 | type=float, 89 | default=1.5, 90 | help="Interval, in seconds, to send [default: %(default)s]", 91 | ) 92 | 93 | parser.add_argument( 94 | "-d", 95 | "--delay", 96 | dest="delay", 97 | type=int, 98 | default=0, 99 | help="delay, in seconds, to send message with [default: %(default)s]", 100 | ) 101 | 102 | parser.add_argument( 103 | "-v", 104 | "--visibility_timeout", 105 | dest="vt", 106 | type=int, 107 | default=None, 108 | help="Visibility Timeout[default: %(default)s]", 109 | ) 110 | 111 | parser.add_argument( 112 | "--delete", 113 | dest="delete", 114 | action="store_true", 115 | default=False, 116 | help="If set, delete queue first", 117 | ) 118 | 119 | parser.add_argument( 120 | "--no-trace", 121 | dest="trace", 122 | action="store_false", 123 | default=True, 124 | help="If set, hide trace messages", 125 | ) 126 | 127 | parser.add_argument( 128 | "-H", dest="host", default="127.0.0.1", help="Redis Host [default: %(default)s]" 129 | ) 130 | parser.add_argument( 131 | "-P", 132 | dest="port", 133 | type=int, 134 | default=6379, 135 | help="Redis Port [default: %(default)s]", 136 | ) 137 | # Parse command line args` 138 | args = parser.parse_args() 139 | 140 | # Create RedisSMQ queue controller 141 | LOG.info( 142 | "Creating RedisSMQ controller for redis at %s:%s, using default queue: %s:%s", 143 | args.host, 144 | args.port, 145 | args.ns, 146 | args.queue, 147 | ) 148 | rsqm = RedisSMQ( 149 | qname=args.queue, 150 | host=args.host, 151 | port=args.port, 152 | ns=args.ns, 153 | vt=args.vt, 154 | delay=args.delay, 155 | trace=args.trace, 156 | ) 157 | 158 | if args.delete: 159 | rsqm.deleteQueue(qname=args.queue, quiet=True).exceptions( 160 | False).execute() 161 | 162 | # Create queue if it is missing. Swallow errors if already exists 163 | rsqm.createQueue(qname=args.queue, quiet=True).exceptions(False).execute() 164 | 165 | # Start Producing 166 | produce(rsqm, "%s:%s" % (args.ns, args.queue), args.count, args.interval) 167 | 168 | 169 | def main(argv=None): 170 | """ Main with retry """ 171 | delay = 1 172 | while True: 173 | try: 174 | loop(argv) 175 | delay = 1 176 | except Exception as ex: 177 | LOG.error("Error: %s", ex) 178 | LOG.info("Sleeping %s seconds before restarting...", delay) 179 | time.sleep(delay) 180 | delay *= 2 181 | delay = min(delay, MAX_DELAY) 182 | 183 | 184 | if __name__ == "__main__": 185 | logging.basicConfig(level=logging.DEBUG) 186 | main(sys.argv) 187 | -------------------------------------------------------------------------------- /src/tests/test_utils.py: -------------------------------------------------------------------------------- 1 | """ 2 | Unit Tests for utils 3 | """ 4 | import logging 5 | import os 6 | import unittest 7 | 8 | from rsmq.cmd import utils 9 | 10 | 11 | BASE_DIR = os.path.dirname(os.path.dirname(__file__)) 12 | 13 | 14 | ENCODE_TESTS = {"a": "a", '{"a": "b"}': {"a": "b"}} 15 | 16 | # Each test is value, min, max, sucess 17 | VALIDATE_INT_TESTS = [ 18 | (0, 0, 100, True), 19 | (1, 0, 100, True), 20 | (1, 0, 1, True), 21 | (-1, -100, 0, True), 22 | (0, -100, 0, True), 23 | (1, -100, 0, False), 24 | (1, 0, 1, True), 25 | (0, 1, 2, False), 26 | ] 27 | 28 | # Base charsets 29 | BASE_10 = "0123456789" 30 | BASE_16 = "0123456789ABCDEF" 31 | BASE_36 = "0123456789abcdefghijklmnopqrstuvwxyz" 32 | 33 | # Each test is (raw, encoded, base_charset) 34 | BASE_X_ENCODED_TESTS = [ 35 | (10000, "10000", BASE_10), 36 | (10000, "2710", BASE_16), 37 | (10000, "7ps", BASE_36), 38 | (10000000, "5yc1s", BASE_36), 39 | ] 40 | 41 | 42 | class MockLogger(object): 43 | def __init__(self): 44 | self.last_level = None 45 | self.last_msg = None 46 | 47 | def reset(self): 48 | self.last_level = None 49 | self.last_msg = None 50 | 51 | def log(self, level, msg, *params): 52 | self.last_level = level 53 | self.last_msg = msg % params 54 | 55 | def debug(self, msg, *params): 56 | self.log(logging.DEBUG, msg, *params) 57 | 58 | def info(self, msg, *params): 59 | self.log(logging.INFO, msg, *params) 60 | 61 | 62 | class CmdUtilsUnitTests(unittest.TestCase): 63 | """Unit Tests for utils""" 64 | 65 | def setUp(self): 66 | """Setup""" 67 | 68 | def test_encode_object(self): 69 | """Check Encode Message""" 70 | for string, obj in ENCODE_TESTS.items(): 71 | self.assertEqual(string, utils.encode_message(obj)) 72 | 73 | def test_encode_string(self): 74 | """Check Encode String Message""" 75 | for string, _obj in ENCODE_TESTS.items(): 76 | self.assertEqual(string, utils.encode_message(string)) 77 | 78 | def test_decode_object(self): 79 | """Check Decode Message Object""" 80 | for _string, obj in ENCODE_TESTS.items(): 81 | self.assertEqual(obj, utils.decode_message(obj)) 82 | 83 | def test_decode_string(self): 84 | """Check Decode Message String""" 85 | for string, obj in ENCODE_TESTS.items(): 86 | self.assertEqual(obj, utils.decode_message(string)) 87 | 88 | def test_validate_int(self): 89 | """Test validate_int""" 90 | for value, min_value, max_value, expected in VALIDATE_INT_TESTS: 91 | self.assertEqual(expected, utils.validate_int(value, min_value, max_value)) 92 | 93 | def test_validate_int_invalid(self): 94 | """Test validate_int""" 95 | self.assertFalse(utils.validate_int(None, 0, 100)) 96 | self.assertFalse(utils.validate_int("A", 0, 100)) 97 | 98 | def test_validate_int_logging_none(self): 99 | """Test validate_int""" 100 | logger = MockLogger() 101 | self.assertFalse(utils.validate_int(None, 0, 100, logger, "testname")) 102 | self.assertEqual("testname value is not set", logger.last_msg) 103 | 104 | def test_validate_int_logging_too_low(self): 105 | """Test validate_int""" 106 | logger = MockLogger() 107 | self.assertFalse(utils.validate_int(-1, 0, 100, logger, "testname")) 108 | self.assertEqual("testname value -1 is less than minimum (0)", logger.last_msg) 109 | 110 | def test_validate_int_logging_too_high(self): 111 | """Test validate_int""" 112 | logger = MockLogger() 113 | self.assertFalse(utils.validate_int(101, 0, 100, logger, "testname")) 114 | self.assertEqual( 115 | "testname value 101 is greater than maximum (100)", logger.last_msg 116 | ) 117 | 118 | def test_validate_int_logging_invalid(self): 119 | """Test validate_int""" 120 | logger = MockLogger() 121 | self.assertFalse(utils.validate_int("A", 0, 100, logger, "testname")) 122 | self.assertEqual("testname value (A) is not an integer", logger.last_msg) 123 | 124 | def test_random_string(self): 125 | """Test random_string""" 126 | charset_default = set(utils.DEFAULT_CHARSET) 127 | charset1 = "abcdefgh" 128 | charset1_set = set(charset1) 129 | 130 | for i in range(32): 131 | for _ in range(10): 132 | rstr = utils.random_string(i, charset1) 133 | self.assertEqual( 134 | i, len(rstr), "Expected length is %s, got %s" % (i, len(rstr)) 135 | ) 136 | self.assertLessEqual( 137 | set(rstr), charset1_set, "Expected all characters to be in set" 138 | ) 139 | for _ in range(10): 140 | rstr = utils.random_string(i) 141 | self.assertEqual( 142 | i, len(rstr), "Expected length is %s, got %s" % (i, len(rstr)) 143 | ) 144 | self.assertLessEqual( 145 | set(rstr), charset_default, "Expected all characters to be in set" 146 | ) 147 | 148 | def test_make_message_id(self): 149 | """Test make_message_id""" 150 | for _ in range(32): 151 | mid = utils.make_message_id(10000000) 152 | # length is always 22 plus encoded value of 10000000 = which is 153 | self.assertEqual(27, len(mid)) 154 | 155 | def test_baseXencode(self): 156 | """Test baseXencode""" 157 | for raw, encoded, charset in BASE_X_ENCODED_TESTS: 158 | self.assertEqual( 159 | encoded, 160 | utils.baseXencode(raw, charset), 161 | "Invalid encoded value for base %s" % len(charset), 162 | ) 163 | 164 | 165 | if __name__ == "__main__": 166 | unittest.main() 167 | -------------------------------------------------------------------------------- /src/rsmq/rsmq.py: -------------------------------------------------------------------------------- 1 | """ 2 | Python Redis Simple Queue Manager 3 | """ 4 | 5 | from redis import Redis 6 | 7 | from . import const 8 | from .cmd import ChangeMessageVisibilityCommand 9 | from .cmd import CreateQueueCommand, DeleteQueueCommand, ListQueuesCommand 10 | from .cmd import DeleteMessageCommand 11 | from .cmd import SendMessageCommand, ReceiveMessageCommand, PopMessageCommand 12 | from .cmd import SetQueueAttributesCommand, GetQueueAttributesCommand 13 | 14 | 15 | DEFAULT_REDIS_OPTIONS = {"encoding": "utf-8", "decode_responses": True} 16 | 17 | DEFAULT_OPTIONS = {"ns": "rsmq", "realtime": False, "exceptions": True} 18 | 19 | 20 | class RedisSMQ: 21 | """ 22 | Redis Simple Message Queue implementation in Python 23 | """ 24 | 25 | def __init__( 26 | self, client=None, host="127.0.0.1", port="6379", options=None, **kwargs 27 | ): 28 | """ 29 | Constructor: 30 | 31 | @param client: if provided, redis client object to use 32 | @param host: if client is not provided, redis hostname 33 | @param port: if client is not provided, redis port 34 | @param options: if client is not provided, additional options for redis client creation 35 | 36 | Additional arguments: 37 | 38 | @param ns: namespace 39 | @param realtime: if true, use realtime comms (pubsub)(default is False) 40 | @param exceptions: if true, throw exceptions on errors, else return False(default is True) 41 | 42 | Remaining params are automatically passed to commands 43 | 44 | """ 45 | 46 | # redis_client 47 | self._client = client 48 | 49 | # Redis Options 50 | self.redis_options = dict(DEFAULT_REDIS_OPTIONS) 51 | self.redis_options["host"] = host 52 | self.redis_options["port"] = port 53 | if options: 54 | self.redis_options.update(options) 55 | 56 | # RSMQ global options 57 | self.options = dict(DEFAULT_OPTIONS) 58 | to_remove = [] 59 | for param, value in kwargs.items(): 60 | if param in self.options: 61 | self.options[param] = kwargs[param] 62 | to_remove.append(param) 63 | elif value is None: 64 | # Remove default set to None 65 | to_remove.append(param) 66 | 67 | # Remove unnecessary kwargs 68 | for param in to_remove: 69 | del kwargs[param] 70 | 71 | # Everything else is passed through to commands 72 | self._default_params = kwargs 73 | 74 | self._popMessageSha1 = None 75 | self._receiveMessageSha1 = None 76 | self._changeMessageVisibilitySha1 = None 77 | 78 | @property 79 | def popMessageSha1(self): 80 | """Get Pop Message Script SHA1""" 81 | if self._popMessageSha1 is None: 82 | client = self.client 83 | self._popMessageSha1 = client.script_load(const.SCRIPT_POPMESSAGE) 84 | return self._popMessageSha1 85 | 86 | @property 87 | def receiveMessageSha1(self): 88 | """Get Received Message Script SHA1""" 89 | if self._receiveMessageSha1 is None: 90 | client = self.client 91 | self._receiveMessageSha1 = client.script_load(const.SCRIPT_RECEIVEMESSAGE) 92 | return self._receiveMessageSha1 93 | 94 | @property 95 | def changeMessageVisibilitySha1(self): 96 | """Get Change Message Visibilities Script SHA1""" 97 | if self._changeMessageVisibilitySha1 is None: 98 | client = self.client 99 | self._changeMessageVisibilitySha1 = client.script_load( 100 | const.SCRIPT_CHANGEMESSAGEVISIBILITY 101 | ) 102 | return self._changeMessageVisibilitySha1 103 | 104 | @property 105 | def client(self): 106 | """get Redis client. Create one if one does not exist""" 107 | if not self._client: 108 | self._client = Redis(**self.redis_options) 109 | return self._client 110 | 111 | def exceptions(self, enabled=True): 112 | """Set global exceptions flag""" 113 | self.options["exceptions"] = enabled == True 114 | return self 115 | 116 | def setClient(self, client): 117 | """Set Redis Client""" 118 | self._client = client 119 | return self 120 | 121 | def _command(self, command, **kwargs): 122 | """Run command""" 123 | args = dict(self._default_params) 124 | args.update(kwargs) 125 | return command(self, **args) 126 | 127 | def createQueue(self, **kwargs): 128 | """Create Queue""" 129 | return self._command(CreateQueueCommand, **kwargs) 130 | 131 | def deleteQueue(self, **kwargs): 132 | """Create Queue""" 133 | return self._command(DeleteQueueCommand, **kwargs) 134 | 135 | def setQueueAttributes(self, **kwargs): 136 | """setQueueAttributesCommand()""" 137 | return self._command(SetQueueAttributesCommand, **kwargs) 138 | 139 | def getQueueAttributes(self, **kwargs): 140 | """getQueueAttributesCommand()""" 141 | return self._command(GetQueueAttributesCommand, **kwargs) 142 | 143 | def listQueues(self, **kwargs): 144 | """List Queues""" 145 | return self._command(ListQueuesCommand, **kwargs) 146 | 147 | def changeMessageVisibility(self, **kwargs): 148 | """ChangeMessageVisibilityCommand""" 149 | return self._command(ChangeMessageVisibilityCommand, **kwargs) 150 | 151 | def sendMessage(self, **kwargs): 152 | """Send Message Command""" 153 | return self._command(SendMessageCommand, **kwargs) 154 | 155 | def receiveMessage(self, **kwargs): 156 | """Receive Message Command""" 157 | return self._command(ReceiveMessageCommand, **kwargs) 158 | 159 | def popMessage(self, **kwargs): 160 | """Pop Message Command""" 161 | return self._command(PopMessageCommand, **kwargs) 162 | 163 | def deleteMessage(self, **kwargs): 164 | """Delete Message Command""" 165 | return self._command(DeleteMessageCommand, **kwargs) 166 | 167 | def quit(self): 168 | """Quit - here for compatibility purposes""" 169 | self._client = None 170 | 171 | def reset_scripts(self): 172 | """ Reset scripts to forse re-load """ 173 | self._popMessageSha1 = None 174 | self._receiveMessageSha1 = None 175 | self._changeMessageVisibilitySha1 = None -------------------------------------------------------------------------------- /examples/consumer.py: -------------------------------------------------------------------------------- 1 | """ 2 | Example Consumer for a queue 3 | 4 | 5 | In this example we create an queue consumer that gets one message at a time, and processes 6 | it using a processor. To emulate some of the complexities, we randomly fail processing and adding 7 | delays to processing. 8 | 9 | """ 10 | import argparse 11 | import logging 12 | import random 13 | import sys 14 | import time 15 | 16 | from rsmq.consumer import RedisSMQConsumerThread 17 | 18 | 19 | LOG = logging.getLogger("Consumer") 20 | 21 | 22 | class Processor(object): 23 | """Dummy processor class to fake complexity of a real processor""" 24 | 25 | def __init__(self, delay, success_rate): 26 | self.delay = delay 27 | self.success_rate = success_rate 28 | 29 | @property 30 | def wait_delay(self): 31 | """ Get random wait delay """ 32 | return random.randint(0, self.delay*1000)/1000 33 | 34 | def random_result(self): 35 | """ 36 | Produce a random boolean which is True "success_rate" percent of the time 37 | 38 | random.random() produces a float in range [0.0, 1.0) 39 | """ 40 | return random.random() < self.success_rate 41 | 42 | def process(self, id, message, rc, ts): 43 | """Actual method that processes the message""" 44 | LOG.info( 45 | "Got message: id: %s, retry count: %s, ts: %s, msg: %s" 46 | % (id, rc, ts, message) 47 | ) 48 | result = self.random_result() 49 | delay = self.wait_delay 50 | if delay > 0: 51 | # Add occasional long delay 52 | # if random.randint(0, 10) == 0: 53 | # delay = 60 54 | LOG.info("Processing message: %s for %s seconds", id, delay) 55 | time.sleep(delay) 56 | LOG.info("Random Result: %s", "Success" if result else "Failure") 57 | return result 58 | 59 | 60 | def consume(consumer, long_qname, exit_after): 61 | """Example of consuming using a thread""" 62 | LOG.info("Starting consumption on queue: %s", long_qname) 63 | # thread = Thread(target=consumer.run, name="QueueConsumer", daemon=False) 64 | # thread.start() 65 | end = time.time() + exit_after if exit_after else 0 66 | while True: 67 | if exit_after and time.time() > end: 68 | LOG.info("Attempting to stop the consumer...") 69 | consumer.stop(15) 70 | break 71 | LOG.info( 72 | "Exited queue consumer for '%s'. Thread is %s", 73 | long_qname, 74 | ("running" if consumer.is_alive() else "stopped"), 75 | ) 76 | 77 | 78 | def probablity(x): 79 | """An arg validator that Require a float number between 0.0 and 1.0""" 80 | x = float(x) 81 | if not 0.0 <= x <= 1.0: 82 | raise argparse.ArgumentTypeError("%r not in range [0.0, 1.0]" % (x,)) 83 | return x 84 | 85 | 86 | def main(argv=None): 87 | if argv is None: 88 | argv = sys.argv 89 | """ Parse args and run producer """ 90 | parser = argparse.ArgumentParser() 91 | parser.add_argument( 92 | "-q", 93 | "--queue", 94 | dest="queue", 95 | action="store", 96 | default="queue", 97 | help="queue name [default: %(default)s]", 98 | ) 99 | parser.add_argument( 100 | "-n", 101 | "--namespace", 102 | dest="ns", 103 | action="store", 104 | default="test", 105 | help="queue namespace [default: %(default)s]", 106 | ) 107 | 108 | parser.add_argument( 109 | "-r", 110 | "--success_rate", 111 | dest="success_rate", 112 | action="store", 113 | type=probablity, 114 | default=1.0, 115 | help="Probability of success when processing messages." 116 | + "1.0 means all are successful, 0.0 means all fail" 117 | + "[default: %(default)s]", 118 | ) 119 | 120 | parser.add_argument( 121 | "-e", 122 | "--empty_delay", 123 | dest="empty_queue_delay", 124 | type=float, 125 | default=4.0, 126 | help="delay in seconds when queue is empty[default: %(default)s]", 127 | ) 128 | 129 | parser.add_argument( 130 | "-d", 131 | "--delay", 132 | dest="delay", 133 | type=float, 134 | default=2.0, 135 | help="additional delay in seconds, during consumption[default: %(default)s]", 136 | ) 137 | 138 | parser.add_argument( 139 | "-x", 140 | "--exit_after", 141 | dest="exit_after", 142 | type=float, 143 | default=0.0, 144 | help="If set, exit after this many seconds[default: %(default)s]", 145 | ) 146 | 147 | parser.add_argument( 148 | "-v", 149 | "--visibility_timeout", 150 | dest="vt", 151 | type=int, 152 | default=None, 153 | help="Visibility Timeout[default: %(default)s]", 154 | ) 155 | 156 | parser.add_argument( 157 | "--no-trace", 158 | dest="trace", 159 | action="store_false", 160 | default=True, 161 | help="If set, hide trace messages", 162 | ) 163 | 164 | parser.add_argument( 165 | "-H", dest="host", default="127.0.0.1", help="Redis Host [default: %(default)s]" 166 | ) 167 | parser.add_argument( 168 | "-P", 169 | dest="port", 170 | type=int, 171 | default=6379, 172 | help="Redis Port [default: %(default)s]", 173 | ) 174 | 175 | # Parse command line args` 176 | args = parser.parse_args() 177 | 178 | # Create Processor 179 | processor = Processor(delay=args.delay, success_rate=args.success_rate) 180 | 181 | # Create RedisSMQ queue consumer controller 182 | LOG.info( 183 | "Creating RedisSMQ Consumer Controller for redis at %s:%s, using queue: %s:%s", 184 | args.host, 185 | args.port, 186 | args.ns, 187 | args.queue, 188 | ) 189 | LOG.info("Starting consumer thread") 190 | 191 | rsqm_consumer = RedisSMQConsumerThread( 192 | qname=args.queue, 193 | processor=processor.process, 194 | host=args.host, 195 | port=args.port, 196 | ns=args.ns, 197 | vt=args.vt, 198 | empty_queue_delay=args.empty_queue_delay, 199 | trace=args.trace, 200 | ) 201 | rsqm_consumer.start() 202 | while rsqm_consumer.is_alive(): 203 | time.sleep(1) 204 | LOG.info("Complete consumer thread") 205 | # Start Consumption 206 | #consume(rsqm_consumer, "%s:%s" % (args.ns, args.queue), args.exit_after) 207 | 208 | 209 | if __name__ == "__main__": 210 | logging.basicConfig(level=logging.DEBUG) 211 | main(sys.argv) 212 | -------------------------------------------------------------------------------- /examples/consumer_thread.py: -------------------------------------------------------------------------------- 1 | """ 2 | Example Consumer for a queue 3 | 4 | 5 | In this example we create an queue consumer that gets one message at a time, and processes 6 | it using a processor. To emulate some of the complexities, we randomly fail processing and adding 7 | delays to processing. 8 | 9 | This example is similar to consumer.py but uses RedisSMQConsumerThread which is a versioin 10 | of RedisSMQConsumer that extends the Thread class 11 | """ 12 | import argparse 13 | import logging 14 | import random 15 | import sys 16 | import time 17 | 18 | from rsmq.consumer import RedisSMQConsumerThread 19 | 20 | 21 | LOG = logging.getLogger("Consumer") 22 | 23 | 24 | class Processor(object): 25 | """Dummy processor class to fake complexity of a real processor""" 26 | 27 | def __init__(self, delay, success_rate): 28 | self.delay = delay 29 | self.success_rate = success_rate 30 | 31 | def random_result(self): 32 | """ 33 | Produce a random boolean which is True "success_rate" percent of the time 34 | 35 | random.random() produces a float in range [0.0, 1.0) 36 | """ 37 | return random.random() < self.success_rate 38 | 39 | def process(self, id, message, rc, ts): 40 | """Actual method that processes the message""" 41 | LOG.info( 42 | "Got message: id: %s, retry count: %s, ts: %s, msg: %s (%s)" 43 | % (id, rc, ts, message, type(message)) 44 | ) 45 | result = self.random_result() 46 | delay = self.delay 47 | if delay > 0: 48 | # Add occasional long delay 49 | # if random.randint(0, 10) == 0: 50 | # delay = 60 51 | LOG.info("Processing message: %s for %s seconds", id, delay) 52 | time.sleep(delay) 53 | # LOG.info("Random Result: %s", "Success" if result else "Failure") 54 | return result 55 | 56 | 57 | def consume(consumer, long_qname, exit_after): 58 | """Example of consuming using a thread""" 59 | LOG.info("Starting consumption on queue: %s", long_qname) 60 | consumer.start() 61 | end = time.time() + exit_after if exit_after else 0 62 | while True: 63 | if exit_after and time.time() > end: 64 | LOG.info("Attempting to stop the consumer...") 65 | consumer.stop(15) 66 | break 67 | LOG.info( 68 | "Exited queue consumer for '%s'. Thread is %s", 69 | long_qname, 70 | ("running" if consumer.is_alive() else "stopped"), 71 | ) 72 | 73 | 74 | def probablity(x): 75 | """An arg validator that Require a float number between 0.0 and 1.0""" 76 | x = float(x) 77 | if not (0.0 <= x <= 1.0): 78 | raise argparse.ArgumentTypeError("%r not in range [0.0, 1.0]" % (x,)) 79 | return x 80 | 81 | 82 | def main(argv=None): 83 | if argv is None: 84 | argv = sys.argv 85 | """ Parse args and run producer """ 86 | parser = argparse.ArgumentParser() 87 | parser.add_argument( 88 | "-q", 89 | "--queue", 90 | dest="queue", 91 | action="store", 92 | default="queue", 93 | help="queue name [default: %(default)s]", 94 | ) 95 | parser.add_argument( 96 | "-n", 97 | "--namespace", 98 | dest="ns", 99 | action="store", 100 | default="test", 101 | help="queue namespace [default: %(default)s]", 102 | ) 103 | 104 | parser.add_argument( 105 | "-r", 106 | "--success_rate", 107 | dest="success_rate", 108 | action="store", 109 | type=probablity, 110 | default=1.0, 111 | help="Probability of success when processing messages." 112 | + "1.0 means all are successful, 0.0 means all fail" 113 | + "[default: %(default)s]", 114 | ) 115 | 116 | parser.add_argument( 117 | "-e", 118 | "--empty_delay", 119 | dest="empty_queue_delay", 120 | type=float, 121 | default=4.0, 122 | help="delay in seconds when queue is empty[default: %(default)s]", 123 | ) 124 | 125 | parser.add_argument( 126 | "-d", 127 | "--delay", 128 | dest="delay", 129 | type=float, 130 | default=2.0, 131 | help="additional delay in seconds, during consumption[default: %(default)s]", 132 | ) 133 | 134 | parser.add_argument( 135 | "-x", 136 | "--exit_after", 137 | dest="exit_after", 138 | type=float, 139 | default=0.0, 140 | help="If set, exit after this many seconds[default: %(default)s]", 141 | ) 142 | 143 | parser.add_argument( 144 | "-v", 145 | "--visibility_timeout", 146 | dest="vt", 147 | type=int, 148 | default=None, 149 | help="Visibility Timeout[default: %(default)s]", 150 | ) 151 | 152 | parser.add_argument( 153 | "--no-trace", 154 | dest="trace", 155 | action="store_false", 156 | default=True, 157 | help="If set, hide trace messages", 158 | ) 159 | 160 | parser.add_argument( 161 | "-D", 162 | "--decode", 163 | dest="decode", 164 | action="store_true", 165 | default=False, 166 | help="If set, decode messages from JSON", 167 | ) 168 | 169 | parser.add_argument( 170 | "-H", dest="host", default="127.0.0.1", help="Redis Host [default: %(default)s]" 171 | ) 172 | parser.add_argument( 173 | "-P", 174 | dest="port", 175 | type=int, 176 | default=6379, 177 | help="Redis Port [default: %(default)s]", 178 | ) 179 | 180 | # Parse command line args` 181 | args = parser.parse_args() 182 | 183 | # Create Processor 184 | processor = Processor(delay=args.delay, success_rate=args.success_rate) 185 | 186 | # Create RedisSMQ queue consumer controller 187 | LOG.info( 188 | "Creating RedisSMQ Consumer Controller for redis at %s:%s, using queue: %s:%s", 189 | args.host, 190 | args.port, 191 | args.ns, 192 | args.queue, 193 | ) 194 | rsqm_consumer = RedisSMQConsumerThread( 195 | qname=args.queue, 196 | processor=processor.process, 197 | host=args.host, 198 | port=args.port, 199 | ns=args.ns, 200 | vt=args.vt, 201 | empty_queue_delay=args.empty_queue_delay, 202 | decode=args.decode, 203 | trace=args.trace, 204 | ) 205 | 206 | # Start Consumption 207 | consume(rsqm_consumer, "%s:%s" % (args.ns, args.queue), args.exit_after) 208 | 209 | 210 | if __name__ == "__main__": 211 | logging.basicConfig(level=logging.DEBUG) 212 | main(sys.argv) 213 | -------------------------------------------------------------------------------- /src/rsmq/cmd/base_command.py: -------------------------------------------------------------------------------- 1 | """ 2 | Base Command for all the commands 3 | """ 4 | 5 | import logging 6 | import re 7 | 8 | from .. import const 9 | from .exceptions import CommandNotImplementedException 10 | from .exceptions import InvalidParameterValue 11 | from .exceptions import QueueDoesNotExist 12 | from .exceptions import RedisSMQException 13 | from .utils import validate_int 14 | 15 | # REGEX matching invalid QNAME characters 16 | QNAME_INVALID_RE = re.compile(const.QNAME_INVALID_CHARS_RE) 17 | 18 | 19 | class BaseRSMQCommand: 20 | """Base for all RQMS commands""" 21 | 22 | PARAMS = { 23 | "qname": {"required": True, "value": None}, 24 | "quiet": {"required": False, "value": False}, 25 | } 26 | 27 | def __init__(self, rsmq, **kwargs): 28 | self.log = logging.getLogger(f"rsmq.cmd.{self.__class__.__name__}") 29 | self.parent = rsmq 30 | self._params = {} 31 | # Load Defaults 32 | for name, value in self._param_defaults().items(): 33 | self._params[name] = value 34 | # load args 35 | for name, value in kwargs.items(): 36 | self._set_param(name, value) 37 | self.__exceptions = None 38 | 39 | @property 40 | def popMessageSha1(self): # pylint: disable=C0103 41 | """popMessageSha1""" 42 | return self.parent.popMessageSha1 43 | 44 | @property 45 | def receiveMessageSha1(self): # pylint: disable=C0103 46 | """receiveMessageSha1""" 47 | return self.parent.receiveMessageSha1 48 | 49 | @property 50 | def changeMessageVisibilitySha1(self): # pylint: disable=C0103 51 | """changeMessageVisibilitySha1""" 52 | return self.parent.changeMessageVisibilitySha1 53 | 54 | @property 55 | def _exceptions(self): 56 | """Returns true if exceptions are enabled""" 57 | if self.__exceptions is None: 58 | return self.parent.options.get("exceptions", True) 59 | return self.__exceptions 60 | 61 | def exceptions(self, enabled=True): 62 | """Control exceptions""" 63 | self.__exceptions = enabled is not False 64 | return self 65 | 66 | @property 67 | def client(self): 68 | """Get Redis client""" 69 | return self.parent.client 70 | 71 | def _param_defaults(self): 72 | """Get dict of default parameters and their values""" 73 | defaults = {} 74 | for name, value in self.PARAMS.items(): 75 | defaults[name] = value.get("value", None) 76 | return defaults 77 | 78 | def _required_params(self): 79 | """List of parameters that are required""" 80 | return [ 81 | param 82 | for param, definition in self.PARAMS.items() 83 | if definition.get("required", False) is True 84 | ] 85 | 86 | def config(self, key, default_value): 87 | """Get value from global config""" 88 | return self.parent.options.get(key, default_value) 89 | 90 | @property 91 | def namespace(self): 92 | """Get namespace""" 93 | namespace = self.config("ns", "") 94 | if namespace: 95 | return namespace + ":" 96 | return "" 97 | 98 | @property 99 | def queue_base(self): 100 | """Get base name of the queue""" 101 | return self.namespace + self.get_qname 102 | 103 | @property 104 | def queue_key(self): 105 | """Get Full queue name""" 106 | return self.queue_base + const.QUEUE_SUFFUX 107 | 108 | @property 109 | def queue_set(self): 110 | """Get Full Queue Set""" 111 | return self.namespace + const.QUEUES 112 | 113 | def param_get(self, param, default_value=None): 114 | """Get parameter by name""" 115 | return self._params.get(param, self.PARAMS[param].get("default", default_value)) 116 | 117 | def __getattr__(self, name): 118 | """ 119 | Create setters for parameters on the fly 120 | """ 121 | if name in self.PARAMS: 122 | 123 | def setter(value): 124 | if self._validate_param(name, value): 125 | self._params[name] = value 126 | elif self._exceptions: 127 | raise InvalidParameterValue(name, value) 128 | elif self.get_quiet is not True: 129 | self.log.info("Invalid value for '%s': '%s'", name, value) 130 | return self 131 | 132 | return setter 133 | if name.startswith("get_"): 134 | param = name[4:] 135 | if param in self.PARAMS: 136 | return self.param_get(param, default_value=None) 137 | 138 | raise AttributeError( 139 | f"'{self.__class__.__name__}' object has no attribute '{name}'" 140 | ) 141 | 142 | def _validate_param(self, name, value): 143 | """Validate parameter value""" 144 | if hasattr(self, f"_validate_{name}"): 145 | validator = getattr(self, f"_validate_{name}") 146 | return validator(value) 147 | return self._default_validator(name, value) 148 | 149 | def _default_validator(self, name, value): 150 | """Default Validator""" 151 | if not name: 152 | return False 153 | if not value: 154 | return False 155 | return True 156 | 157 | def _validate_qname(self, qname): 158 | """Validate Queue Name""" 159 | # Check we have a value at all 160 | if not qname: 161 | return False 162 | # check length 163 | if len(qname) > const.QNAME_MAX_LEN: 164 | return False 165 | # check if we have invalid characters 166 | if ":" in qname: 167 | print("Detected :") 168 | return False 169 | return True 170 | 171 | def _validate_vt(self, vt): 172 | """Validate visibility timeout parameter""" 173 | return validate_int(vt, const.VT_MIN, const.VT_MAX, logger=self.log, name="vt") 174 | 175 | def _validate_delay(self, delay): 176 | """Validate delay parameter""" 177 | return validate_int( 178 | delay, const.DELAY_MIN, const.DELAY_MAX, logger=self.log, name="delay" 179 | ) 180 | 181 | def _validate_maxsize(self, maxsize): 182 | """Validate maxsize parameter""" 183 | return maxsize == -1 or validate_int( 184 | maxsize, 185 | const.MAXSIZE_MIN, 186 | const.MAXSIZE_MAX, 187 | logger=self.log, 188 | name="maxsize", 189 | ) 190 | 191 | def _set_param(self, name, value): 192 | """set parameter, with validation""" 193 | if name in self.PARAMS: 194 | if self._validate_param(name, value): 195 | self._params[name] = value 196 | return True 197 | return False 198 | 199 | def ready(self): 200 | """Check if we are ready to execute""" 201 | for param in self._required_params(): 202 | if not self._validate_param(param, self._params[param]): 203 | self.log.info("Invalid value for parameter '%s'", param) 204 | return False 205 | return True 206 | 207 | def execute(self): 208 | """Execute Command""" 209 | if self._exceptions: 210 | return self._exec() 211 | try: 212 | ret = self._exec() 213 | except RedisSMQException as ex: 214 | if self.get_quiet is not True: 215 | self.log.warning( 216 | "%s: Exception while processing %s: %s", 217 | ex.__class__.__name__, 218 | self.__class__.__name__, 219 | ex, 220 | ) 221 | return False 222 | return ret 223 | 224 | def _exec(self): 225 | if self.ready(): 226 | return self.exec_command() 227 | return False 228 | 229 | def exec_command(self): 230 | """Execute Command""" 231 | raise CommandNotImplementedException() 232 | 233 | def queue_def(self): 234 | """Get Queue Definition""" 235 | queue_key = self.queue_key 236 | tx = self.client.pipeline(transaction=True) 237 | tx.hmget(queue_key, "vt", "delay", "maxsize") 238 | tx.time() 239 | 240 | results = tx.execute() 241 | 242 | if not results or results[0][0] is None: 243 | raise QueueDoesNotExist(self.get_qname) 244 | stats = results[0] 245 | redis_time = results[1] 246 | 247 | ts_usec = redis_time[0] * 1000000 + redis_time[1] 248 | ts = int(ts_usec / 1000) 249 | 250 | return { 251 | "qname": self.get_qname, 252 | "vt": stats[0], 253 | "delay": stats[1], 254 | "maxsize": stats[2], 255 | "ts": ts, 256 | "ts_usec": ts_usec, 257 | } 258 | -------------------------------------------------------------------------------- /src/tests/test_rsmq.py: -------------------------------------------------------------------------------- 1 | """ 2 | Baseline Unit Test 3 | """ 4 | import difflib 5 | import os.path 6 | import pprint 7 | import unittest 8 | from unittest.util import safe_repr 9 | 10 | import fakeredis 11 | 12 | from rsmq.rsmq import RedisSMQ 13 | 14 | 15 | BASE_DIR = os.path.dirname(os.path.dirname(__file__)) 16 | 17 | 18 | class TestUnitTests(unittest.TestCase): 19 | """Unit Test RedisSMQ""" 20 | 21 | def setUp(self): 22 | """Setup""" 23 | 24 | def assertQueueAttributes(self, expected, actual, msg=None): 25 | """assert queue attributes are set""" 26 | self.assertIsInstance(expected, dict, "First argument is not a dictionary") 27 | self.assertIsInstance(actual, dict, "Second argument is not a dictionary") 28 | actual_short = {key: actual[key] for key in expected.keys()} 29 | if not expected.items() <= actual.items(): 30 | diff = "\n" + "\n".join( 31 | difflib.ndiff( 32 | pprint.pformat(expected).splitlines(), 33 | pprint.pformat(actual_short).splitlines(), 34 | ) 35 | ) 36 | standardMsg = "%s !<= %s" % (safe_repr(expected), safe_repr(actual_short)) 37 | standardMsg = self._truncateMessage(standardMsg, diff) 38 | msg = self._formatMessage(msg, standardMsg) 39 | raise AssertionError(msg) 40 | 41 | def test_unittest(self): 42 | """Check Unit Test Framework""" 43 | unittest_loaded = True 44 | self.assertTrue(unittest_loaded) 45 | 46 | def test_qname_good_validation(self): 47 | """Test Good QName validation""" 48 | client = fakeredis.FakeStrictRedis(decode_responses=True) 49 | client.flushall() 50 | queue = RedisSMQ(client=client, exceptions=False) 51 | GOOD_QUEUE_NAMES = [ 52 | "simple", 53 | "with_underscore", 54 | "with-dash", 55 | "-start", 56 | "with\x01\x02\x03binary", 57 | ] 58 | 59 | for qname in GOOD_QUEUE_NAMES: 60 | self.assertTrue(queue.getQueueAttributes().qname(qname).ready()) 61 | 62 | def test_qname_bad_validation(self): 63 | """Test bad QName validation""" 64 | client = fakeredis.FakeStrictRedis(decode_responses=True) 65 | client.flushall() 66 | queue = RedisSMQ(client=client, exceptions=False) 67 | BAD_QUEUE_NAMES = ["", None, "something:else", ":", "something:", ":else"] 68 | 69 | for qname in BAD_QUEUE_NAMES: 70 | self.assertFalse(queue.getQueueAttributes().qname(qname).ready()) 71 | 72 | def test_create_queue_default(self): 73 | """Test Creating of the Queue""" 74 | client = fakeredis.FakeStrictRedis(decode_responses=True) 75 | client.flushall() 76 | queue = RedisSMQ(client=client) 77 | queue_name = "test-queue" 78 | queue_key = "rsmq:%s:Q" % queue_name 79 | queue.createQueue().qname(queue_name).execute() 80 | keys = client.keys("*") 81 | self.assertListEqual(sorted([queue_key, "rsmq:QUEUES"]), sorted(keys)) 82 | queues = client.smembers("rsmq:QUEUES") 83 | self.assertSetEqual(set([queue_name]), queues) 84 | queue_details = client.hgetall(queue_key) 85 | self.assertTrue( 86 | {"vt": "30", "delay": "0", "maxsize": "65565"}.items() 87 | <= queue_details.items() 88 | ) 89 | 90 | def test_create_queue_custom(self): 91 | """Test Creating of the Queue""" 92 | client = fakeredis.FakeStrictRedis(decode_responses=True) 93 | client.flushall() 94 | queue = RedisSMQ(client=client) 95 | queue_name = "test-queue-2" 96 | queue_key = "rsmq:%s:Q" % queue_name 97 | queue.createQueue(qname=queue_name, vt=15).delay(10).execute() 98 | keys = client.keys("*") 99 | self.assertListEqual(sorted([queue_key, "rsmq:QUEUES"]), sorted(keys)) 100 | queues = client.smembers("rsmq:QUEUES") 101 | self.assertSetEqual(set([queue_name]), queues) 102 | queue_details = client.hgetall(queue_key) 103 | 104 | self.assertTrue( 105 | {"vt": "15", "delay": "10", "maxsize": "65565"}.items() 106 | <= queue_details.items() 107 | ) 108 | 109 | def test_delete_queue(self): 110 | """Test Deleting of the Queue""" 111 | client = fakeredis.FakeStrictRedis(decode_responses=True) 112 | client.flushall() 113 | queue = RedisSMQ(client=client) 114 | queue_name = "test-queue-delete" 115 | queue_key = "rsmq:%s:Q" % queue_name 116 | queue.createQueue().qname(queue_name).execute() 117 | keys = client.keys("*") 118 | self.assertListEqual(sorted([queue_key, "rsmq:QUEUES"]), sorted(keys)) 119 | queues = client.smembers("rsmq:QUEUES") 120 | self.assertSetEqual(set([queue_name]), queues) 121 | queue_details = client.hgetall(queue_key) 122 | self.assertTrue( 123 | {"vt": "30", "delay": "0", "maxsize": "65565"}.items() 124 | <= queue_details.items() 125 | ) 126 | result = queue.deleteQueue(qname=queue_name).execute() 127 | self.assertEqual(result, True) 128 | queue_details = client.hgetall(queue_key) 129 | self.assertEqual(queue_details, {}) 130 | 131 | def test_queue_attributes(self): 132 | """Test Getting/Setting of the Queue Attribues""" 133 | client = fakeredis.FakeStrictRedis(decode_responses=True) 134 | client.flushall() 135 | queue_name = "test-queue-attr" 136 | queue = RedisSMQ(client=client, qname=queue_name) 137 | queue_key = "rsmq:%s:Q" % queue_name 138 | queue.createQueue(vt=15, delay=10).execute() 139 | keys = client.keys("*") 140 | self.assertListEqual(sorted([queue_key, "rsmq:QUEUES"]), sorted(keys)) 141 | queues = client.smembers("rsmq:QUEUES") 142 | self.assertSetEqual(set([queue_name]), queues) 143 | queue_details = client.hgetall(queue_key) 144 | self.assertQueueAttributes( 145 | {"vt": "15", "delay": "10", "maxsize": "65565"}, queue_details 146 | ) 147 | attributes = queue.getQueueAttributes(qname=queue_name).execute() 148 | self.assertQueueAttributes( 149 | { 150 | "vt": 15, 151 | "delay": 10, 152 | "maxsize": 65565, 153 | "totalrecv": 0, 154 | "totalsent": 0, 155 | "msgs": 0, 156 | "hiddenmsgs": 0, 157 | }, 158 | attributes, 159 | ) 160 | attributes_1 = queue.setQueueAttributes(vt=30, maxsize=1024, delay=1).execute() 161 | attributes = queue.getQueueAttributes(qname=queue_name).execute() 162 | self.assertDictEqual(attributes, attributes_1) 163 | self.assertQueueAttributes( 164 | { 165 | "vt": 30, 166 | "delay": 1, 167 | "maxsize": 1024, 168 | "totalrecv": 0, 169 | "totalsent": 0, 170 | "msgs": 0, 171 | "hiddenmsgs": 0, 172 | }, 173 | attributes, 174 | ) 175 | 176 | def test_invalid_queue_attributes(self): 177 | """Test Getting/Setting of the Queue Attribues""" 178 | client = fakeredis.FakeStrictRedis(decode_responses=True) 179 | client.flushall() 180 | queue_name = "test-queue-attr" 181 | queue = RedisSMQ(client=client, qname=queue_name) 182 | queue_key = "rsmq:%s:Q" % queue_name 183 | queue.createQueue(vt=15, delay=10).execute() 184 | keys = client.keys("*") 185 | self.assertListEqual(sorted([queue_key, "rsmq:QUEUES"]), sorted(keys)) 186 | queues = client.smembers("rsmq:QUEUES") 187 | self.assertSetEqual(set([queue_name]), queues) 188 | queue_details = client.hgetall(queue_key) 189 | self.assertQueueAttributes( 190 | {"vt": "15", "delay": "10", "maxsize": "65565"}, queue_details 191 | ) 192 | attributes = queue.getQueueAttributes(qname=queue_name).execute() 193 | self.assertQueueAttributes( 194 | { 195 | "vt": 15, 196 | "delay": 10, 197 | "maxsize": 65565, 198 | "totalrecv": 0, 199 | "totalsent": 0, 200 | "msgs": 0, 201 | "hiddenmsgs": 0, 202 | }, 203 | attributes, 204 | ) 205 | attributes_1 = queue.setQueueAttributes(vt=-3, maxsize=-2, delay=-1).execute() 206 | attributes = queue.getQueueAttributes(qname=queue_name).execute() 207 | self.assertDictEqual(attributes, attributes_1) 208 | self.assertQueueAttributes( 209 | { 210 | "vt": 15, 211 | "delay": 10, 212 | "maxsize": 65565, 213 | "totalrecv": 0, 214 | "totalsent": 0, 215 | "msgs": 0, 216 | "hiddenmsgs": 0, 217 | }, 218 | attributes, 219 | ) 220 | 221 | 222 | if __name__ == "__main__": 223 | unittest.main() 224 | -------------------------------------------------------------------------------- /src/rsmq/consumer.py: -------------------------------------------------------------------------------- 1 | """ 2 | Python Redis Simple Queue Manager Consumer 3 | """ 4 | 5 | import logging 6 | import time 7 | from threading import Thread 8 | 9 | from redis.exceptions import NoScriptError 10 | 11 | from .cmd import utils 12 | from .cmd.exceptions import NoMessageInQueue 13 | from .cmd.exceptions import RedisSMQException 14 | from .retry_delay_handler import RetryDelayHandler 15 | from .rsmq import RedisSMQ 16 | from .rsmq import const 17 | 18 | LOG = logging.getLogger(__name__) 19 | 20 | 21 | class RedisSMQConsumer: 22 | """ 23 | RSMQ Consumer Worker 24 | """ 25 | 26 | # local parameters and their value 27 | LOCAL_PARAMS = {"retry_delay": 0, "empty_queue_delay": 2.0, "decode": True} 28 | 29 | class VisibilityTimeoutExtender(Thread): 30 | """ Thread that keeps the visibility """ 31 | 32 | def __init__(self, consumer, message_id): 33 | """Initialize""" 34 | self.consumer = consumer 35 | self.rsqm = consumer.rsqm 36 | 37 | self.vt = consumer.vt 38 | self.message_id = message_id 39 | thread_name = "VTExtender:%s" % message_id 40 | super(RedisSMQConsumer.VisibilityTimeoutExtender, self).__init__( 41 | name=thread_name 42 | ) 43 | 44 | self.daemon = True 45 | self._request_stop = False 46 | 47 | # How often to check time 48 | self.interval = self.vt / 4 49 | 50 | self.next_extension = time.time() + self.vt / 2 51 | self.set_next_extension() 52 | 53 | def set_next_extension(self): 54 | """set next extension""" 55 | self.next_extension = time.time() + self.vt / 2 56 | 57 | def trace(self, msg, *args): 58 | """Passthrough for trace messages""" 59 | self.consumer.trace("VisibilityTimeoutExtender: %s" % msg, args) 60 | 61 | def extend(self): 62 | """extend visibility timeout""" 63 | try: 64 | self.trace("Extending visibility by %s seconds...", self.vt) 65 | self.rsqm.changeMessageVisibility( 66 | vt=self.vt, id=self.message_id 67 | ).execute() 68 | self.set_next_extension() 69 | except RedisSMQException as ex: 70 | LOG.warning( 71 | "Failed to extend message visibility for %s: %s", 72 | self.message_id, 73 | ex, 74 | ) 75 | 76 | def stop(self): 77 | """Stop this thread""" 78 | self._request_stop = True 79 | 80 | def run(self): 81 | """Run""" 82 | while not self._request_stop: 83 | time.sleep(self.interval) 84 | if not self._request_stop and time.time() > self.next_extension: 85 | self.extend() 86 | self.set_next_extension() 87 | 88 | def __init__(self, qname, processor, **rsmq_params): 89 | """ 90 | initialize 91 | 92 | Required Parameters: 93 | 94 | @param qname: queue name 95 | @param processor: processor for each item 96 | 97 | Optional Parameters: 98 | @param retry_delay: amount of time, in seconds, before failed item is 99 | retried 100 | @param empty_queue_delay: (float) Amount of time, in seconds, to wait 101 | if no items in queue 102 | 103 | Remaining args are passed to RedisSMQ() 104 | 105 | """ 106 | self._request_stop = False 107 | self.rsqm = RedisSMQ(qname=qname, **rsmq_params) 108 | self.qname = qname 109 | self.processor = processor 110 | 111 | self._trace = rsmq_params.get("trace", False) 112 | 113 | # Separate local params from kwargs 114 | self.params = {} 115 | for param, value in self.LOCAL_PARAMS.items(): 116 | self.params[param] = rsmq_params.get(param, value) 117 | if param in rsmq_params: 118 | del rsmq_params[param] 119 | 120 | # get vt from kwargs, if set 121 | self.vt = rsmq_params.get("vt", None) 122 | # get vt from queue if not set in kwargs and queue exists 123 | self._get_vt() 124 | 125 | def _param(self, param, default_value=None): 126 | """get local param""" 127 | return self.params.get(param, 128 | self.LOCAL_PARAMS.get(param, default_value)) 129 | 130 | @property 131 | def retry_delay(self): 132 | """retry delay""" 133 | return self._param("retry_delay") 134 | 135 | @property 136 | def empty_queue_delay(self): 137 | """empty_queue_delay""" 138 | return self._param("empty_queue_delay") 139 | 140 | @property 141 | def decode(self): 142 | """decode, if true, attempt to decode the output from JSON""" 143 | return self._param("decode") 144 | 145 | def _get_vt(self): 146 | """Get VT from the queue info if not set""" 147 | if self.vt is None: 148 | queue_info = ( 149 | self.rsqm.getQueueAttributes() 150 | .qname(self.qname) 151 | .exceptions(False) 152 | .execute() 153 | ) 154 | if queue_info and "vt" in queue_info: 155 | self.vt = int(queue_info.get("vt", const.VT_DEFAULT)) 156 | else: 157 | self.vt = const.VT_DEFAULT 158 | 159 | def stop(self, wait=None): 160 | """Stop""" 161 | self._request_stop = True 162 | if wait: 163 | wait_until = time.time() + wait 164 | while not time.time() > wait_until: 165 | time.sleep(0.5) 166 | if self._request_stop is None: 167 | break 168 | return self._request_stop is None 169 | 170 | def on_success(self, msg): 171 | """Run on success""" 172 | # delete the item 173 | if msg and "id" in msg: 174 | self.trace("Processed message %s", msg["id"]) 175 | self.rsqm.deleteMessage(qname=self.qname, id=msg["id"]).execute() 176 | 177 | def on_failure(self, msg): 178 | """Run on success""" 179 | 180 | if msg and "id" in msg: 181 | LOG.warning("Failed to process message %s", msg["id"]) 182 | self.rsqm.changeMessageVisibility( 183 | vt=self.retry_delay, qname=self.qname, id=msg["id"] 184 | ).execute() 185 | 186 | def create_queue(self): 187 | """Create queue if it does not exists""" 188 | self.rsqm.createQueue(qname=self.qname, quiet=True).exceptions( 189 | False).execute() 190 | 191 | def trace(self, fmt, *args): 192 | """Print trace log messages for debugging, if enabled""" 193 | if self._trace is not False: 194 | LOG.debug(fmt, *args) 195 | 196 | def run(self): 197 | """main loop of the thread""" 198 | self.trace("Starting Queue Consumer for %s", self.qname) 199 | self.create_queue() 200 | retry_delay = RetryDelayHandler(0, 60) 201 | while not self._request_stop: 202 | try: 203 | msg = self.rsqm.receiveMessage().execute() 204 | if msg and "id" in msg: 205 | extender = RedisSMQConsumer.VisibilityTimeoutExtender( 206 | self, msg["id"] 207 | ) 208 | extender.start() 209 | if self.decode and "message" in msg: 210 | msg["message"] = utils.decode_message(msg["message"]) 211 | if self.processor(**msg): 212 | self.on_success(msg) 213 | else: 214 | self.on_failure(msg) 215 | extender.stop() 216 | retry_delay.reset() 217 | else: 218 | raise RedisSMQException( 219 | "Invalid message in queue '%s': %s" % (self.qname, msg) 220 | ) 221 | except NoMessageInQueue: 222 | delay = self.empty_queue_delay 223 | self.trace("No message in queue, waiting %s seconds", delay) 224 | if delay: 225 | time.sleep(delay) 226 | except RedisSMQException as ex: 227 | LOG.warning("Exception while processing queue `%s`: %s", 228 | self.qname, ex) 229 | except NoScriptError as ex: 230 | LOG.warning("Exception while processing queue `%s`: %s", 231 | self.qname, ex) 232 | retry_delay.delay() 233 | LOG.warning("Resetting the client") 234 | self.rsqm.reset_scripts() 235 | except ConnectionError as ex: 236 | LOG.warning("Connection error while processing queue `%s`: %s", 237 | self.qname, ex) 238 | retry_delay.delay() 239 | except Exception as ex: 240 | LOG.warning("Unexpected error while processing queue `%s`: %s", 241 | self.qname, ex) 242 | retry_delay.delay() 243 | self.rsqm.reset_scripts() 244 | 245 | self._request_stop = None 246 | self.trace("Ended Queue Consumer for %s", self.qname) 247 | 248 | 249 | class RedisSMQConsumerThread(RedisSMQConsumer, Thread): 250 | """Version of a RedisSMQConsumer implemented as a self-contained thread""" 251 | 252 | def __init__(self, qname, processor, **rsmq_params): 253 | """Constructor""" 254 | RedisSMQConsumer.__init__(self, qname, processor, **rsmq_params) 255 | Thread.__init__(self, name="RedisSMQConsumer:%s" % qname, daemon=True) 256 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ![RSMQ: Redis Simple Message Queue for Node.js](https://img.webmart.de/rsmq_wide.png) 2 | 3 | [![Build Status](https://travis-ci.org/mlasevich/PyRSMQ.svg?branch=master)](https://travis-ci.org/mlasevich/PyRSMQ) 4 | [![Coverage Status](https://coveralls.io/repos/github/mlasevich/PyRSMQ/badge.svg?branch=master)](https://coveralls.io/github/mlasevich/PyRSMQ?branch=master) 5 | [![PyPI version](https://badge.fury.io/py/PyRSMQ.svg)](https://badge.fury.io/py/PyRSMQ) 6 | 7 | # Redis Simple Message Queue 8 | 9 | A lightweight message queue for Python that requires no dedicated queue server. Just a Redis server. 10 | 11 | This is a Python implementation of [https://github.com/smrchy/rsmq](https://github.com/smrchy/rsmq)) 12 | 13 | 14 | ## PyRSMQ Release Notes 15 | 16 | * 0.6.1 17 | * Bugfix: Fix incomplete delete of queue data when redis auto-decode is off (PR[#23](https://github.com/mlasevich/PyRSMQ/pull/23)) (@ChuckHend) 18 | 19 | * 0.6.0 20 | * Bugfix: Allow for recovery in RedisSMQConsumerThread when redis is temporarily unavailable. 21 | * 0.5.1 22 | * Bugfix: Fix crash on non-existent queue name (Thanks @rwl4) 23 | 24 | * 0.5.0 25 | * Require Python 3.6+ 26 | * Code cleanup 27 | * Fix for scenario where consumer breaks if redis is restarted (#4) 28 | 29 | * 0.4.5 30 | * Re-release to push to PyPi 31 | 32 | * 0.4.4 33 | * Allow extending the transaction for deleteMessage to perform other actions in same transaction ([#9](https://github.com/mlasevich/PyRSMQ/issues/9)) (@yehonatanz) 34 | * Use redis timestamp in milliseconds instead of local in seconds ([#11](https://github.com/mlasevich/PyRSMQ/pull/11)) (@yehonatanz) 35 | * Convert queue attributes to numbers when elligible ([#12](https://github.com/mlasevich/PyRSMQ/pull/12)) (@yehonatanz) 36 | 37 | 38 | * 0.4.3 39 | * Don't encode sent message if it is of type bytes ([#6](https://github.com/mlasevich/PyRSMQ/issues/6)) (@yehonatanz) 40 | * Allow delay and vt to be float (round only after converting to millis) ([#7](https://github.com/mlasevich/PyRSMQ/issues/7)) (@yehonatanz) 41 | * Convert ts from str/bytes to int in receive/pop message ([#8](https://github.com/mlasevich/PyRSMQ/issues/8)) (@yehonatanz) 42 | 43 | * 0.4.2 44 | * Fix typo in `setClient` method [#3](https://github.com/mlasevich/PyRSMQ/issues/3) 45 | * Note this is a breaking change if you use this method, (which seems like nobody does) 46 | 47 | * 0.4.1 48 | * Add auto-decode option for messages from JSON (when possible) in Consumer (on by default) 49 | 50 | * 0.4.0 51 | * Ability to import `RedisSMQ` from package rather than from the module (i.e. you can now use `from rsmq import RedisSMQ` instead of `from rsmq.rsmq import RedisSMQ`) 52 | * Add quiet option to most commands to allow to hide errors if exceptions are disabled 53 | * Additional unit tests 54 | * Auto-encoding of non-string messages to JSON for sendMessage 55 | * Add `RedisSMQConsumer` and `RedisSMQConsumerThread` for easier creation of queue consumers 56 | * Add examples for simple producers/consumers 57 | 58 | * 0.3.1 59 | * Fix message id generation match RSMQ algorithm 60 | 61 | * 0.3.0 62 | * Make message id generation match RSMQ algorithm 63 | * Allow any character in queue name other than `:` 64 | 65 | * 0.2.1 66 | * Allow uppercase characters in queue names 67 | 68 | * 0.2.0 - Adding Python 2 support 69 | * Some Python 2 support 70 | * Some Unit tests 71 | * Change `.exec()` to `.execute()` for P2 compatibility 72 | 73 | * 0.1.0 - initial version 74 | * Initial port 75 | * Missing "Realtime" mode 76 | * Missing unit tests 77 | 78 | ## Quick Intro to RSMQ 79 | 80 | RSMQ is trying to emulate Amazon's SQS-like functionality, where there is a named queue (name 81 | consists of "namespace" and "qname") that is backed by Redis. Queue must be created before used. 82 | Once created, _Producers_ will place messages in queue and _Consumers_ will retrieve them. 83 | Messages have a property of "visibility" - where any "visible" message may be consumed, but 84 | "invisbile" messages stay in the queue until they become visible or deleted. 85 | 86 | Once queue exists, a _Producer_ can push messages into it. When pushing to queue, message gets a 87 | unique ID that is used to track the message. The ID can be used to delete message by _Producer_ or 88 | _Consumer_ or to control its "visibility" 89 | 90 | During insertion, a message may have a `delay` associated with it. "Delay" will mark message 91 | "invisible" for specified delay duration, and thus prevent it from being consumed. Delay may be 92 | specified at time of message creation or, if not specified, default value set in queue attributes 93 | is used. 94 | 95 | _Consumer_ will retrieve next message in queue via either `receiveMessage()` or `popMessage()` 96 | command. If we do not care about reliability of the message beyond delivery, a good and simple way 97 | to retrieve it is via `popMessage()`. When using `popMessage()` the message is automatically 98 | deleted at the same time it is received. 99 | 100 | However in many cases we want to ensure that the message is not only received, but is also 101 | processed before being deleted. For this, `receiveMessage()` is best. When using `receiveMessage()`, 102 | the message is kept in the queue, but is marked "invisible" for some amount of time. The amount of 103 | time is specified by queue attribute `vt`(visibility timeout), which may also be overridden by 104 | specifying a custom `vt` value in `receiveMessage()` call. When using `receiveMessage()`, 105 | _Consumer_' is responsible for deleting the message before `vt` timeout occurs, otherwise the 106 | message may be picked up by another _Consumer_. _Consumer_ can also extend the timeout if it needs 107 | more time, or clear the timeout if processing failed. 108 | 109 | A "Realtime" mode can be specified when using the RSMQ queue. "Realtime" mode adds a Redis PUBSUB 110 | based notification that would allow _Consumers_ to be notified whenever a new message is added to 111 | the queue. This can remove the need for _Consumer_ to constantly poll the queue when it is empty 112 | (*NOTE:* as of this writing, "Realtime" is not yet implemented in python version) 113 | 114 | ## Python Implementation Notes 115 | 116 | *NOTE* This project is written for Python 3.x. While some attempts to get Python2 support were made 117 | I am not sure how stable it would be under Python 2 118 | 119 | This version is heavily based on Java version (https://github.com/igr/jrsmq), which in turn is 120 | based on the original Node.JS version. 121 | 122 | ### API 123 | To start with, best effort is made to maintain same method/parameter/usablity named of both version 124 | (which, admittedly, resulted in a not very pythonic API) 125 | 126 | Although much of the original API is still present, some alternatives are added to make life a bit 127 | easier. 128 | 129 | For example, while you can set any available parameter to command using the "setter" method, you can 130 | also simply specify the parameters when creating the command. So these two commands do same thing: 131 | 132 | rqsm.createQueue().qname("my-queue").vt(20).execute() 133 | 134 | rqsm.createQueue(qname="my-queue", vt=20).execute() 135 | 136 | In addition, when creating a main controller, any non-controller parameters specified will become 137 | defaults for all commands created via this controller - so, for example, you if you plan to work 138 | with only one queue using this controller, you can specify the qname parameter during creation of 139 | the controller and not need to specify it in every command. 140 | 141 | ### A "Consumer" Service Utility 142 | 143 | In addition to all the APIs in the original RSMQ project, a simple to use consumer implementation 144 | is included in this project as `RedisSMQConsumer` and `RedisSMQConsumerThread` classes. 145 | 146 | #### RedisSMQConsumer 147 | The `RedisSMQConsumer` instance wraps an RSMQ Controller and is configured with a processor method 148 | which is called every time a new message is received. The processor method returns true or false 149 | to indicate if message was successfully received and the message is deleted or returned to the 150 | queue based on that. The consumer auto-extends the visibility timeout as long as the processor is 151 | running, reducing the concern that item will become visible again if processing takes too long and 152 | visibility timeout elapses. 153 | 154 | NOTE: Since currently the `realtime` functionality is not implemented, Consumer implementation is 155 | currently using polling to check for queue items. 156 | 157 | Example usage: 158 | 159 | ``` 160 | from rsmq.consumer import RedisSMQConsumer 161 | 162 | # define Processor 163 | def processor(id, message, rc, ts): 164 | ''' process the message ''' 165 | # Do something 166 | return True 167 | 168 | # create consumer 169 | consumer = RedisSMQConsumer('my-queue', processor, host='127.0.0.1') 170 | 171 | # run consumer 172 | consumer.run() 173 | ``` 174 | 175 | For a more complete example, see examples directory. 176 | 177 | 178 | #### RedisSMQConsumerThread 179 | 180 | `RedisSMQConsumerThread` is simply a version of `RedisSMQConsumer` that extends Thread class. 181 | 182 | Once created you can start it like any other thread, or stop it using `stop(wait)` method, where 183 | wait specifies maximum time to wait for the thread to stop before returning (the thread would still 184 | be trying to stop if the `wait` time expires) 185 | 186 | Note that the thread is by default set to be a `daemon` thread, so on exit of your main thread it 187 | will be stopped. If you wish to disable daemon flag, just disable it before starting the thread as 188 | with any other thread 189 | 190 | Example usage: 191 | 192 | ``` 193 | from rsmq.consumer import RedisSMQConsumerThread 194 | 195 | # define Processor 196 | def processor(id, message, rc, ts): 197 | ''' process the message ''' 198 | # Do something 199 | return True 200 | 201 | # create consumer 202 | consumer = RedisSMQConsumerThread('my-queue', processor, host='127.0.0.1') 203 | 204 | # start consumer 205 | consumer.start() 206 | 207 | # do what else you need to, then stop the consumer 208 | # (waiting for 10 seconds for it to stop): 209 | consumer.stop(10) 210 | 211 | ``` 212 | 213 | For a more complete example, see examples directory. 214 | 215 | ### General Usage Approach 216 | 217 | As copied from other versions, the general approach is to create a controller object and use that 218 | object to create, configure and then execute commands 219 | 220 | ### Error Handling 221 | 222 | Commands follow the pattern of other versions and throw exceptions on error. 223 | 224 | Exceptions are all extending `RedisSMQException()` and include: 225 | 226 | * `InvalidParameterValue()` - Invalid Parameter specified 227 | * `QueueAlreadyExists()` - attempt to create queue which already exists 228 | * `QueueDoesNotExist()` - attempt to use/delete queue that does not exist 229 | * `NoMessageInQueue()` - attempt to retrieve message from queue that has no visible messaged 230 | 231 | However, if you do not wish to use exceptions, you can turn them off on per-command or 232 | per-controller basis by using `.exceptions(False)` on the relevant object. For example, the 233 | following will create Queue only if it does not exist without throwing an exception: 234 | 235 | rsmq.createQueue().exceptions(False).execute() 236 | 237 | 238 | ## Usage 239 | 240 | ### Example Usage 241 | 242 | In this example we will create a new queue named "my-queue", deleting previous version, if one 243 | exists, and then send a message with a 2 second delay. We will then demonstrate both the lack of 244 | message before delay expires and getting the message after timeout 245 | 246 | 247 | from pprint import pprint 248 | import time 249 | 250 | from rsmq import RedisSMQ 251 | 252 | 253 | # Create controller. 254 | # In this case we are specifying the host and default queue name 255 | queue = RedisSMQ(host="127.0.0.1", qname="myqueue") 256 | 257 | 258 | # Delete Queue if it already exists, ignoring exceptions 259 | queue.deleteQueue().exceptions(False).execute() 260 | 261 | # Create Queue with default visibility timeout of 20 and delay of 0 262 | # demonstrating here both ways of setting parameters 263 | queue.createQueue(delay=0).vt(20).execute() 264 | 265 | # Send a message with a 2 second delay 266 | message_id = queue.sendMessage(delay=2).message("Hello World").execute() 267 | 268 | pprint({'queue_status': queue.getQueueAttributes().execute()}) 269 | 270 | # Try to get a message - this will not succeed, as our message has a delay and no other 271 | # messages are in the queue 272 | msg = queue.receiveMessage().exceptions(False).execute() 273 | 274 | # Message should be False as we got no message 275 | pprint({"Message": msg}) 276 | 277 | print("Waiting for our message to become visible") 278 | # Wait for our message to become visible 279 | time.sleep(2) 280 | 281 | pprint({'queue_status': queue.getQueueAttributes().execute()}) 282 | # Get our message 283 | msg = queue.receiveMessage().execute() 284 | 285 | # Message should now be there 286 | pprint({"Message": msg}) 287 | 288 | # Delete Message 289 | queue.deleteMessage(id=msg['id']) 290 | 291 | pprint({'queue_status': queue.getQueueAttributes().execute()}) 292 | # delete our queue 293 | queue.deleteQueue().execute() 294 | 295 | # No action 296 | queue.quit() 297 | 298 | 299 | 300 | ### RedisSMQ Controller API Usage 301 | 302 | Usage: `rsmq.rqsm.RedisSMQ([options])` 303 | 304 | * Options (all options are provided as keyword options): 305 | * Redis Connection arguments: 306 | * `client` - provide an existing, configured redis client 307 | or 308 | * `host` - redis hostname (Default: `127.0.0.1`) 309 | * `port` - redis port (Default: `6379`) 310 | * `options` - additional redis client options. Defaults: 311 | * `encoding`: `utf-8` 312 | * `decode_responses`: `True` 313 | * Controller Options 314 | * `ns` - namespace - all redis keys are prepended with `:`. Default: `rsmq` 315 | * `realtime` - if set to True, enables realtime option. Default: `False` 316 | * `exceptions` - if set to True, throw exceptions for all commands. Default: `True` 317 | * Default Command Options. Anything else is passed to each command as defaults. Examples: 318 | * `qname` - default Queue Name 319 | 320 | #### Controller Methods 321 | 322 | * `exceptions(True/False)` - enable/disable exceptions 323 | * `setClient(client)` - specify new redis client object 324 | * `ns(namespace)` - set new namespace 325 | * `quit()` - disconnect from redis. This is mainly for compatibility with other versions. Does not do much 326 | 327 | #### Controller Commands 328 | 329 | * `createQueue()` - Create new queue 330 | * **Parameters:** 331 | * `qname` - (Required) name of the queue 332 | * `vt` - default visibility timeout in seconds. Default: `30` 333 | * `delay` - default delay (visibility timeout on insert). Default: `0` 334 | * `maxsize` - maximum message size (1024-65535, Default: 65535) 335 | * `quiet` - if set to `True` and exceptions are disabled, do not produce error log entries 336 | * **Returns**: 337 | * `True` if queue was created 338 | 339 | * `deleteQueue()` - Delete Existing queue 340 | * Parameters: 341 | * `qname` - (Required) name of the queue 342 | * `quiet` - if set to `True` and exceptions are disabled, do not produce error log entries 343 | * **Returns**: 344 | * `True` if queue was deleted 345 | 346 | * `setQueueAttributes()` - Update queue attributes. If value is not specified, it is not updated. 347 | * **Parameters:** 348 | * `qname` - (Required) name of the queue 349 | * `vt` - default visibility timeout in seconds. Default: `30` 350 | * `delay` - default delay (visibility timeout on insert). Default: `0` 351 | * `maxsize` - maximum message size (1024-65535, Default: 65535) 352 | * `quiet` - if set to `True` and exceptions are disabled, do not produce error log entries 353 | * **Returns**: 354 | * output of `getQueueAttributes()` call 355 | 356 | * `getQueueAttributes()` - Get Queue Attributes and statistics 357 | * Parameters: 358 | * `qname` - (Required) name of the queue 359 | * `quiet` - if set to `True` and exceptions are disabled, do not produce error log entries 360 | * **Returns** a dictionary with following fields: 361 | * `vt` - default visibility timeout 362 | * `delay` - default insertion delay 363 | * `maxsize` - max size of the message 364 | * `totalrecv` - number of messages consumed. Note, this is incremented each time message is retrieved, so if it was not deleted and made visible again, it will show up here multiple times. 365 | * `totalsent` - number of messages sent to queue. 366 | * `created` - unix timestamp (seconds since epoch) of when the queue was created 367 | * `modified` - unix timestamp (seconds since epoch) of when the queue was last updated 368 | * `msgs` - Total number of messages currently in the queue 369 | * `hiddenmsgs` - Number of messages in queue that are not visible 370 | 371 | * `listQueues()` - List all queues in this namespace 372 | * **Parameters:** 373 | * **Returns**: 374 | * All queue names in this namespace as a `set()` 375 | 376 | * `changeMessageVisibility()` - Change Message Visibility 377 | * **Parameters:** 378 | * `qname` - (Required) name of the queue 379 | * `id` - (Required) message id 380 | * `quiet` - if set to `True` and exceptions are disabled, do not produce error log entries 381 | * ??? 382 | * **Returns**: 383 | * ??? 384 | 385 | * `sendMessage()` - Send message into queue 386 | * **Parameters:** 387 | * `qname` - (Required) name of the queue 388 | * `message` - (Required) message id 389 | * `delay` - Optional override of the `delay` for this message (If not specified, default for queue is used) 390 | * `quiet` - if set to `True` and exceptions are disabled, do not produce error log entries 391 | * `encode` if set to `True`, force encode message as JSON string. If False, try to auto-detect if message needs to be encoded 392 | * **Returns**: 393 | * message id of the sent message 394 | 395 | * `receiveMessage()` - Receive Message from queue and mark it invisible 396 | * **Parameters:** 397 | * `qname` - (Required) name of the queue 398 | * `vt` - Optional override for visibility timeout for this message (If not specified, default for queue is used) 399 | * `quiet` - if set to `True` and exceptions are disabled, do not produce error log entries 400 | * **Returns** dictionary for following fields: 401 | * `id` - message id 402 | * `message` - message content 403 | * `rc` - receive count - how many times this message was received 404 | * `ts` - unix timestamp of when the message was originally sent 405 | 406 | * `popMessage()` - Receive Message from queue and delete it from queue 407 | * **Parameters:** 408 | * `qname` - (Required) name of the queue 409 | * `quiet` - if set to `True` and exceptions are disabled, do not produce error log entries 410 | * **Returns** dictionary for following fields: 411 | * `id` - message id 412 | * `message` - message content 413 | * `rc` - receive count - how many times this message was received 414 | * `ts` - unix timestamp of when the message was originally sent 415 | 416 | * `deleteMessage()` - Delete Message from queue 417 | * **Parameters:** 418 | * `qname` - (Required) name of the queue 419 | * `id` - (Required) message id 420 | * `quiet` - if set to `True` and exceptions are disabled, do not produce error log entries 421 | * **Returns**: 422 | * `True` if message was deleted 423 | --------------------------------------------------------------------------------