Welcome!
\n" 16 | "This is the home page. It's nothing special, but we have logged your page view. " 17 | "Thank you for visiting!
\n" 18 | ) 19 | 20 | html += "Total views
" 21 | 22 | try: 23 | total_views = bus.analytics.get_total(url=current_url) 24 | html += f"There have been {total_views} views for this page
" 25 | except LightbusTimeout: 26 | html += ( 27 | f"The bus did not respond in a timely fashion. Have you started the "
28 | f"Lightbus worker process using lightbus run?
Online pet store
" 26 | 27 | for pet_num, image_url in enumerate(PETS): 28 | resized_url = bus.image.resize(url=image_url, width=200, height=200) 29 | html += f'' f'
Pet {pet_num}
" 41 | html = f'' 42 | return html 43 | -------------------------------------------------------------------------------- /docs/explanation/schema.md: -------------------------------------------------------------------------------- 1 | # Schema 2 | 3 | Lightbus creates a schema for each of your APIs using the type hints 4 | specified on the API. This schema is shared on the bus for consumption 5 | by other Lightbus clients. This provides a number of features: 6 | 7 | * The availability of a particular API can be detected by remote clients 8 | * RPCs, results, and events transmitted on the bus can be validated by both the sender and receiver 9 | * Tooling can load the schema to provide additional functionality. For example, you can 10 | [dump your production schema](https://lightbus.org/reference/command-line-use/dumpschema/) 11 | and run your development environment and tests against it. 12 | 13 | Note that an API's schema will only be available on the bus while there is a worker 14 | running to provides it. Once the worker process for an API shuts down the schema on the 15 | bus will be cleaned up shortly thereafter. 16 | 17 | ## See also 18 | 19 | See the [schema reference](../reference/schema.md) section for details on how this works in practice. 20 | 21 | The schema is created using the [JSON schema] format, see the [schema protocol] for details of 22 | the transmission format. 23 | 24 | 25 | [JSON schema]: https://json-schema.org/ 26 | [schema protocol]: ../reference/protocols/schema.md 27 | -------------------------------------------------------------------------------- /docs/explanation/internal-architecture.md: -------------------------------------------------------------------------------- 1 | # Internal Architecture 2 | 3 | Lightbus' internal workings are composed of: 4 | 5 | * **The user-facing API**. This is provided by the `BusClient` class, which then makes use of the `EventClient` and 6 | `RpcResultClient` classes. This is a friendly API that issues helpful errors where appropriate. This also 7 | orchestrates system startup and shutdown. 8 | * **An internal message queuing system**. This includes four separate internal message queues plus the `EventDock` 9 | & `RpcResultDock` classes. The message queues provide the internal communication medium between the 10 | user-facing API and the Lightbus backend. The `EventDock` & `RpcResultDock` classes convert these messages 11 | into a simplified API for implementation by the transports. The `EventDock` contains the `EventTransport`, 12 | and the `RpcResultDock` contains both the `RpcTransport` and `RpcResultTransport`. 13 | * **The Event & RPC transports** implement Lightbus functionality for a specific backend (e.g. Redis). The main transports 14 | shipped with Lightbus are the `RedisEventTransport`, `RedisRpcTransport`, and `RedisResultTransport`. 15 | 16 | ## Diagram 17 | 18 | ![Internal Architecture Diagram][diagram] 19 | 20 | [diagram]: ../static/images/internal-architecture.png 21 | 22 | -------------------------------------------------------------------------------- /lightbus_examples/ex06_django/bus.py: -------------------------------------------------------------------------------- 1 | import os 2 | from datetime import datetime 3 | 4 | import django 5 | import lightbus 6 | from lightbus.utilities.django import uses_django_db 7 | 8 | bus = lightbus.create() 9 | 10 | 11 | class AnalyticsApi(lightbus.Api): 12 | page_view = lightbus.Event( 13 | parameters=( 14 | lightbus.Parameter("pk", int), 15 | lightbus.Parameter("viewed_at", datetime), 16 | lightbus.Parameter("url", str), 17 | lightbus.Parameter("user_agent", str), 18 | ) 19 | ) 20 | 21 | @uses_django_db 22 | def get_total(self, url: str) -> int: 23 | from lightbus_examples.ex06_django.example_app.models import PageView 24 | 25 | return PageView.objects.filter(url=url).count() 26 | 27 | class Meta: 28 | name = "analytics" 29 | 30 | 31 | # Tell the client to respond to this API 32 | bus.client.register_api(AnalyticsApi()) 33 | 34 | 35 | @bus.client.on_start() 36 | async def bus_start(**kwargs): 37 | # Setup the default DJANGO_SETTINGS_MODULE 38 | # (as we also do in manage.py and wsgi.py) 39 | os.environ.setdefault("DJANGO_SETTINGS_MODULE", "lightbus_examples.ex06_django.settings") 40 | 41 | # Sets up django. We must do this before importing any models 42 | django.setup() 43 | -------------------------------------------------------------------------------- /docs/explanation/performance.md: -------------------------------------------------------------------------------- 1 | # Performance 2 | 3 | !!! note "Caveats" 4 | 5 | Lightbus has yet to undergo any profiling or optimisation, therefore it is 6 | reasonable to expect performance to improve with time. 7 | 8 | The performance of Lightbus is primarily governed by the [transports](transports.md) used. Lightbus 9 | currently only ships with Redis support. 10 | 11 | ## Individual process performance 12 | 13 | Simple benchmarking[^1] on a 2018 MacBook Pro indicates the following execution times (plugins disabled, schema 14 | and validation enabled, no event/RPC parameters): 15 | 16 | * Firing an event: ≈ 1.7ms (±10%) 17 | * Performing a remote procedure call: ≈ 6.9ms (±10%) 18 | 19 | ## Redis performance 20 | 21 | The Redis server has the potential to be a central bottleneck in Lightbus' performance. You may start to run into 22 | these limits if you are sending tens thousands of events per second. 23 | 24 | In these cases you can either: 25 | 26 | * Scale via Redis by setting up [Redis Cluster](https://redis.io/topics/cluster-tutorial) 27 | * Scale via Lightbus by specifying a different Redis instance per-API. See 28 | [configuration](../reference/configuration.md) 29 | 30 | [^1]: See [how to modify Lightbus](../howto/modify-lightbus.md) for details on how to run the benchmarks via `pytest` 31 | -------------------------------------------------------------------------------- /tests/config/test_structure.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | from lightbus.config.structure import make_transport_selector_structure, ApiConfig, RootConfig 4 | 5 | pytestmark = pytest.mark.unit 6 | 7 | 8 | def test_make_transport_config_structure(): 9 | EventTransportSelector = make_transport_selector_structure("event") 10 | assert "redis" in EventTransportSelector.__annotations__ 11 | 12 | 13 | def test_make_api_config_structure(): 14 | assert "event_transport" in ApiConfig.__annotations__ 15 | assert "rpc_transport" in ApiConfig.__annotations__ 16 | assert "result_transport" in ApiConfig.__annotations__ 17 | assert "validate" in ApiConfig.__annotations__ 18 | 19 | 20 | def test_root_config_service_name(): 21 | service_name = RootConfig().service_name 22 | assert service_name 23 | assert type(service_name) == str 24 | assert len(service_name) > 3 25 | # No format parameters in there, should have been formatted upon instantiation 26 | assert "{" not in service_name 27 | 28 | 29 | def test_root_config_process_name(): 30 | process_name = RootConfig().process_name 31 | assert process_name 32 | assert type(process_name) == str 33 | assert len(process_name) > 3 34 | # No format parameters in there, should have been formatted upon instantiation 35 | assert "{" not in process_name 36 | -------------------------------------------------------------------------------- /lightbus_vendored/aioredis/commands/scripting.py: -------------------------------------------------------------------------------- 1 | from lightbus_vendored.aioredis.util import wait_ok 2 | 3 | 4 | class ScriptingCommandsMixin: 5 | """Set commands mixin. 6 | 7 | For commands details see: http://redis.io/commands#scripting 8 | """ 9 | 10 | def eval(self, script, keys=[], args=[]): 11 | """Execute a Lua script server side.""" 12 | return self.execute(b'EVAL', script, len(keys), *(keys + args)) 13 | 14 | def evalsha(self, digest, keys=[], args=[]): 15 | """Execute a Lua script server side by its SHA1 digest.""" 16 | return self.execute(b'EVALSHA', digest, len(keys), *(keys + args)) 17 | 18 | def script_exists(self, digest, *digests): 19 | """Check existence of scripts in the script cache.""" 20 | return self.execute(b'SCRIPT', b'EXISTS', digest, *digests) 21 | 22 | def script_kill(self): 23 | """Kill the script currently in execution.""" 24 | fut = self.execute(b'SCRIPT', b'KILL') 25 | return wait_ok(fut) 26 | 27 | def script_flush(self): 28 | """Remove all the scripts from the script cache.""" 29 | fut = self.execute(b"SCRIPT", b"FLUSH") 30 | return wait_ok(fut) 31 | 32 | def script_load(self, script): 33 | """Load the specified Lua script into the script cache.""" 34 | return self.execute(b"SCRIPT", b"LOAD", script) 35 | -------------------------------------------------------------------------------- /lightbus/utilities/importing.py: -------------------------------------------------------------------------------- 1 | import importlib 2 | import logging 3 | import sys 4 | from typing import Sequence, Tuple, Callable 5 | 6 | if sys.version_info < (3, 10): 7 | from importlib.metadata import entry_points as _entry_points 8 | 9 | def entry_points(group): 10 | return _entry_points()[group] 11 | 12 | else: 13 | from importlib.metadata import entry_points 14 | 15 | 16 | logger = logging.getLogger(__name__) 17 | 18 | 19 | def import_module_from_string(name): 20 | """Import a module if necessary, otherwise return it from the list of already imported modules""" 21 | if name in sys.modules: 22 | return sys.modules[name] 23 | else: 24 | return importlib.import_module(name) 25 | 26 | 27 | def import_from_string(name): 28 | components = name.split(".") 29 | mod = __import__(components[0]) 30 | for comp in components[1:]: 31 | mod = getattr(mod, comp) 32 | return mod 33 | 34 | 35 | def load_entrypoint_classes(entrypoint_name) -> Sequence[Tuple[str, str, Callable]]: 36 | """Load classes specified in an entrypoint 37 | 38 | Entrypoints are specified in pyproject.toml, and Lightbus uses them to 39 | discover plugins & transports. 40 | """ 41 | found_classes = [] 42 | for entrypoint in entry_points(group=entrypoint_name): 43 | class_ = entrypoint.load() 44 | found_classes.append((entrypoint.module, entrypoint.name, class_)) 45 | return found_classes 46 | -------------------------------------------------------------------------------- /.github/workflows/docs.yaml: -------------------------------------------------------------------------------- 1 | name: Docs 2 | 3 | on: 4 | push: 5 | paths: 6 | - '.github/workflows/docs.yaml' 7 | - 'docs/**' 8 | 9 | # TODO: Trigger on release tag creation 10 | # TODO: Also create workflow for creating releases 11 | 12 | jobs: 13 | docs: 14 | name: Build docs 15 | runs-on: ubuntu-22.04 16 | 17 | steps: 18 | - name: Checkout 19 | uses: actions/checkout@master 20 | 21 | - uses: actions/setup-python@v3 22 | with: 23 | python-version: 3.9 24 | 25 | - name: "Configure git" 26 | run: | 27 | # Mike needs the git config user.name & user.email values set 28 | git config --global user.name "Adam Charnock" 29 | git config --global user.email "adam@adamcharnock.com" 30 | 31 | - name: "Pull the latest docs" 32 | run: | 33 | git fetch origin master gh-pages 34 | git checkout gh-pages 35 | git pull origin gh-pages 36 | git checkout ${{ github.sha }} 37 | 38 | - name: "Install poetry and lightbus" 39 | run: | 40 | pip install poetry 41 | poetry install 42 | 43 | - name: "Dump config schema" 44 | run: | 45 | poetry run lightbus dumpconfigschema > docs/static/default-config-schema.json 46 | 47 | - name: "Build and push docs" 48 | run: | 49 | poetry run mike deploy --message="Automated docs build [ci skip]" dev 50 | git push --force origin gh-pages 51 | 52 | -------------------------------------------------------------------------------- /lightbus/client/subclients/base.py: -------------------------------------------------------------------------------- 1 | import asyncio 2 | 3 | from lightbus.client.internal_messaging.consumer import InternalConsumer 4 | from lightbus.client.internal_messaging.producer import InternalProducer 5 | from lightbus.client.utilities import ErrorQueueType 6 | from lightbus.hooks import HookRegistry 7 | from lightbus.schema import Schema 8 | from lightbus.api import ApiRegistry 9 | from lightbus.config import Config 10 | from lightbus.utilities.internal_queue import InternalQueue 11 | 12 | 13 | class BaseSubClient: 14 | def __init__( 15 | self, 16 | api_registry: ApiRegistry, 17 | hook_registry: HookRegistry, 18 | config: Config, 19 | schema: Schema, 20 | error_queue: ErrorQueueType, 21 | consume_from: InternalQueue, 22 | produce_to: InternalQueue, 23 | ): 24 | self.api_registry = api_registry 25 | self.hook_registry: HookRegistry = hook_registry 26 | self.config = config 27 | self.schema = schema 28 | self.error_queue = error_queue 29 | self.producer = InternalProducer( 30 | name=self.__class__.__name__, queue=produce_to, error_queue=error_queue 31 | ) 32 | self.consumer = InternalConsumer( 33 | name=self.__class__.__name__, queue=consume_from, error_queue=error_queue 34 | ) 35 | 36 | self.producer.start() 37 | self.consumer.start(self.handle) 38 | 39 | async def handle(self, command): 40 | raise NotImplementedError() 41 | 42 | async def close(self): 43 | pass 44 | -------------------------------------------------------------------------------- /lightbus/utilities/frozendict.py: -------------------------------------------------------------------------------- 1 | """ An immutable dictionary 2 | 3 | This has been vendored from [python-frozendict](https://github.com/slezica/python-frozendict) 4 | and subsequently modified. 5 | """ 6 | import collections.abc 7 | 8 | 9 | class frozendict(collections.abc.Mapping): 10 | """ 11 | An immutable wrapper around dictionaries that implements the complete :py:class:`collections.Mapping` 12 | interface. It can be used as a drop-in replacement for dictionaries where immutability is desired. 13 | """ 14 | 15 | dict_cls = dict 16 | 17 | def __init__(self, *args, **kwargs): 18 | self._dict = self.dict_cls(*args, **kwargs) 19 | self._hash = None 20 | 21 | def __getitem__(self, key): 22 | return self._dict[key] 23 | 24 | def __contains__(self, key): 25 | return key in self._dict 26 | 27 | def copy(self, **add_or_replace): 28 | return self.__class__(self, **add_or_replace) 29 | 30 | def __iter__(self): 31 | return iter(self._dict) 32 | 33 | def __len__(self): 34 | return len(self._dict) 35 | 36 | def __repr__(self): 37 | return "<%s %r>" % (self.__class__.__name__, self._dict) 38 | 39 | def __hash__(self): 40 | if self._hash is None: 41 | h = 0 42 | for key, value in self._dict.items(): 43 | h ^= hash((key, value)) 44 | self._hash = h 45 | return self._hash 46 | 47 | 48 | class FrozenOrderedDict(frozendict): 49 | """ 50 | A frozendict subclass that maintains key order 51 | """ 52 | 53 | dict_cls = collections.OrderedDict 54 | -------------------------------------------------------------------------------- /lightbus/commands/version.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import logging 3 | import sys 4 | from pathlib import Path 5 | 6 | from importlib.metadata import version as importlib_version 7 | 8 | if sys.version_info < (3, 10): 9 | from importlib.metadata import entry_points as _entry_points 10 | 11 | def entry_points(group): 12 | return _entry_points()[group] 13 | 14 | else: 15 | from importlib.metadata import entry_points 16 | 17 | 18 | logger = logging.getLogger(__name__) 19 | 20 | 21 | class Command: 22 | def setup(self, parser, subparsers): 23 | parser_version = subparsers.add_parser( 24 | "version", help="Show the currently installed Lightbus version" 25 | ) 26 | # Read version directly out of pyproject.toml. Useful for the release process 27 | parser_version.add_argument("--pyproject", action="store_true", help=argparse.SUPPRESS) 28 | # Show the version to be used for creating the docs 29 | parser_version.add_argument("--docs", action="store_true", help=argparse.SUPPRESS) 30 | parser_version.set_defaults(func=self.handle) 31 | 32 | def handle(self, args): 33 | if args.pyproject: 34 | import lightbus 35 | import toml 36 | 37 | file_path = Path(lightbus.__file__).parent.parent / "pyproject.toml" 38 | with file_path.open() as f: 39 | version = toml.load(f)["tool"]["poetry"]["version"] 40 | if args.docs: 41 | version = ".".join(version.split(".")[:2]) 42 | print(version) 43 | else: 44 | print(importlib_version("lightbus")) 45 | -------------------------------------------------------------------------------- /lightbus_vendored/aioredis/locks.py: -------------------------------------------------------------------------------- 1 | import asyncio 2 | import sys 3 | 4 | from asyncio.locks import Lock as _Lock 5 | 6 | # Fixes an issue with all Python versions that leaves pending waiters 7 | # without being awakened when the first waiter is canceled. 8 | # Code adapted from the PR https://github.com/python/cpython/pull/1031 9 | # Waiting once it is merged to make a proper condition to relay on 10 | # the stdlib implementation or this one patched 11 | 12 | 13 | class Lock(_Lock): 14 | 15 | if sys.version_info < (3, 7, 0): 16 | async def acquire(self): 17 | """Acquire a lock. 18 | This method blocks until the lock is unlocked, then sets it to 19 | locked and returns True. 20 | """ 21 | if not self._locked and all(w.cancelled() for w in self._waiters): 22 | self._locked = True 23 | return True 24 | 25 | fut = self._loop.create_future() 26 | 27 | self._waiters.append(fut) 28 | try: 29 | await fut 30 | self._locked = True 31 | return True 32 | except asyncio.CancelledError: 33 | if not self._locked: # pragma: no cover 34 | self._wake_up_first() 35 | raise 36 | finally: 37 | self._waiters.remove(fut) 38 | 39 | def _wake_up_first(self): 40 | """Wake up the first waiter who isn't cancelled.""" 41 | for fut in self._waiters: 42 | if not fut.done(): 43 | fut.set_result(True) 44 | break 45 | -------------------------------------------------------------------------------- /lightbus/client/commands.py: -------------------------------------------------------------------------------- 1 | import logging 2 | 3 | from typing import NamedTuple, Optional, List, Tuple 4 | 5 | from lightbus.api import Api 6 | from lightbus.message import EventMessage, RpcMessage, ResultMessage 7 | from lightbus.utilities.internal_queue import InternalQueue 8 | 9 | logger = logging.getLogger(__name__) 10 | 11 | 12 | class SendEventCommand(NamedTuple): 13 | message: EventMessage 14 | options: dict = {} 15 | 16 | 17 | class ConsumeEventsCommand(NamedTuple): 18 | events: List[Tuple[str, str]] 19 | listener_name: str 20 | destination_queue: InternalQueue[EventMessage] 21 | options: dict = {} 22 | 23 | 24 | class AcknowledgeEventCommand(NamedTuple): 25 | message: EventMessage 26 | options: dict = {} 27 | 28 | 29 | class CallRpcCommand(NamedTuple): 30 | message: RpcMessage 31 | options: dict = {} 32 | 33 | 34 | class ConsumeRpcsCommand(NamedTuple): 35 | api_names: List[str] 36 | options: dict = {} 37 | 38 | 39 | class ExecuteRpcCommand(NamedTuple): 40 | """An RPC call has been received and must be executed locally""" 41 | 42 | message: RpcMessage 43 | 44 | 45 | class PublishApiSchemaCommand(NamedTuple): 46 | api: Api 47 | 48 | 49 | class CloseCommand(NamedTuple): 50 | pass 51 | 52 | 53 | class SendResultCommand(NamedTuple): 54 | rpc_message: RpcMessage 55 | message: ResultMessage 56 | 57 | 58 | class ReceiveResultCommand(NamedTuple): 59 | message: RpcMessage 60 | destination_queue: InternalQueue 61 | options: dict 62 | 63 | 64 | class ReceiveSchemaUpdateCommand(NamedTuple): 65 | schema: dict 66 | 67 | 68 | class ShutdownCommand(NamedTuple): 69 | exception: Optional[BaseException] 70 | -------------------------------------------------------------------------------- /lightbus_vendored/aioredis/__init__.py: -------------------------------------------------------------------------------- 1 | from .connection import RedisConnection, create_connection 2 | from .commands import ( 3 | Redis, create_redis, 4 | create_redis_pool, 5 | # GeoPoint, GeoMember, 6 | ) 7 | from .pool import ConnectionsPool, create_pool 8 | from .pubsub import Channel 9 | from .sentinel import RedisSentinel, create_sentinel 10 | from .errors import ( 11 | ConnectionClosedError, 12 | ConnectionForcedCloseError, 13 | MasterNotFoundError, 14 | MultiExecError, 15 | PipelineError, 16 | ProtocolError, 17 | ReadOnlyError, 18 | RedisError, 19 | ReplyError, 20 | MaxClientsError, 21 | AuthError, 22 | ChannelClosedError, 23 | WatchVariableError, 24 | PoolClosedError, 25 | SlaveNotFoundError, 26 | MasterReplyError, 27 | SlaveReplyError, 28 | ) 29 | 30 | 31 | __version__ = '1.3.1' 32 | 33 | __all__ = [ 34 | # Factories 35 | 'create_connection', 36 | 'create_pool', 37 | 'create_redis', 38 | 'create_redis_pool', 39 | 'create_sentinel', 40 | # Classes 41 | 'RedisConnection', 42 | 'ConnectionsPool', 43 | 'Redis', 44 | 'GeoPoint', 45 | 'GeoMember', 46 | 'Channel', 47 | 'RedisSentinel', 48 | # Errors 49 | 'RedisError', 50 | 'ReplyError', 51 | 'MaxClientsError', 52 | 'AuthError', 53 | 'ProtocolError', 54 | 'PipelineError', 55 | 'MultiExecError', 56 | 'WatchVariableError', 57 | 'ConnectionClosedError', 58 | 'ConnectionForcedCloseError', 59 | 'PoolClosedError', 60 | 'ChannelClosedError', 61 | 'MasterNotFoundError', 62 | 'SlaveNotFoundError', 63 | 'ReadOnlyError', 64 | 'MasterReplyError', 65 | 'SlaveReplyError', 66 | ] 67 | -------------------------------------------------------------------------------- /lightbus/commands/dump_config_schema.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import logging 3 | 4 | from lightbus.commands import utilities as command_utilities 5 | from lightbus.config.config import config_as_json_schema 6 | from lightbus.plugins import PluginRegistry 7 | from lightbus.schema.encoder import json_encode 8 | 9 | logger = logging.getLogger(__name__) 10 | 11 | 12 | class Command: 13 | def setup(self, parser, subparsers): 14 | parser_dumpconfigschema = subparsers.add_parser( 15 | "dumpconfigschema", 16 | help=( 17 | "Dumps the lightbus configuration json schema. Can be useful " 18 | "in validating your config. This is not the same as your " 19 | "bus' API schema, for that see the more commonly used 'dumpschema' " 20 | "command" 21 | ), 22 | formatter_class=argparse.ArgumentDefaultsHelpFormatter, 23 | ) 24 | group = parser_dumpconfigschema.add_argument_group( 25 | title="Dump config schema command arguments" 26 | ) 27 | group.add_argument( 28 | "--out", 29 | "-o", 30 | help=( 31 | "File to write config schema to. " 32 | "If omitted the schema will be written to standard out." 33 | ), 34 | metavar="FILE", 35 | ) 36 | parser_dumpconfigschema.set_defaults(func=self.handle) 37 | 38 | def handle(self, args): 39 | schema = json_encode(config_as_json_schema(), indent=2, sort_keys=True) 40 | 41 | if args.out: 42 | with open(args.out, "w", encoding="utf8") as f: 43 | f.write(schema) 44 | else: 45 | print(schema) 46 | -------------------------------------------------------------------------------- /lightbus/client/docks/base.py: -------------------------------------------------------------------------------- 1 | import asyncio 2 | 3 | from lightbus.client.internal_messaging.consumer import InternalConsumer 4 | from lightbus.client.internal_messaging.producer import InternalProducer 5 | from lightbus.api import ApiRegistry 6 | from lightbus.client.utilities import ErrorQueueType 7 | from lightbus.config import Config 8 | from lightbus.transports.registry import TransportRegistry 9 | from lightbus.utilities.internal_queue import InternalQueue 10 | 11 | 12 | class BaseDock: 13 | """The base dock 14 | 15 | A dock is responsible for interfacing a transport with Lightbus' internal 16 | messaging system. 17 | """ 18 | 19 | def __init__( 20 | self, 21 | transport_registry: TransportRegistry, 22 | api_registry: ApiRegistry, 23 | config: Config, 24 | error_queue: ErrorQueueType, 25 | consume_from: InternalQueue, 26 | produce_to: InternalQueue, 27 | ): 28 | self.transport_registry = transport_registry 29 | self.api_registry = api_registry 30 | self.config = config 31 | self.error_queue = error_queue 32 | self.producer = InternalProducer( 33 | name=self.__class__.__name__, queue=produce_to, error_queue=error_queue 34 | ) 35 | self.consumer = InternalConsumer( 36 | name=self.__class__.__name__, queue=consume_from, error_queue=error_queue 37 | ) 38 | 39 | self.producer.start() 40 | self.consumer.start(self.handle) 41 | 42 | async def handle(self, command): 43 | raise NotImplementedError() 44 | 45 | async def wait_until_ready(self): 46 | await self.producer.wait_until_ready() 47 | await self.consumer.wait_until_ready() 48 | -------------------------------------------------------------------------------- /tests/transports/redis/benchmark_integration_redis.py: -------------------------------------------------------------------------------- 1 | import inspect 2 | import pytest 3 | 4 | import lightbus 5 | import lightbus.path 6 | from lightbus.transports.redis.event import StreamUse 7 | 8 | pytestmark = pytest.mark.integration 9 | 10 | stream_use_test_data = [StreamUse.PER_EVENT, StreamUse.PER_API] 11 | 12 | pytestmark = pytest.mark.benchmark 13 | 14 | 15 | class BenchmarkApi(lightbus.Api): 16 | fire_me = lightbus.Event() 17 | 18 | class Meta: 19 | name = "benchmark" 20 | 21 | def call_me(self): 22 | return True 23 | 24 | 25 | BUS_MODULE_CONTENT = f""" 26 | bus = lightbus.create(plugins=[]) 27 | 28 | {inspect.getsource(BenchmarkApi)} 29 | 30 | bus.client.register_api(BenchmarkApi()) 31 | """ 32 | 33 | 34 | @pytest.fixture() 35 | def run_lightbus(run_lightbus_command, make_test_bus_module): 36 | """Run lightbus in a background process""" 37 | run_lightbus_command( 38 | "run", "--bus", make_test_bus_module(code=BUS_MODULE_CONTENT), env={"LIGHTBUS_MODULE": ""} 39 | ) 40 | 41 | 42 | @pytest.fixture() 43 | def bus(redis_config_file): 44 | """Get a BusPath instance so we can use the bus""" 45 | bus = lightbus.create(config_file=redis_config_file) 46 | yield bus 47 | bus.client.close() 48 | 49 | 50 | @pytest.mark.benchmark(group="network") 51 | def benchmark_call_rpc(run_lightbus, bus, benchmark): 52 | def benchmark_me(): 53 | assert bus.benchmark.call_me() 54 | 55 | benchmark.pedantic(benchmark_me, rounds=20, warmup_rounds=1) 56 | 57 | 58 | @pytest.mark.benchmark(group="network") 59 | def benchmark_fire_event(bus, benchmark): 60 | bus.client.register_api(BenchmarkApi()) 61 | benchmark.pedantic(bus.benchmark.fire_me.fire, rounds=20, warmup_rounds=1) 62 | -------------------------------------------------------------------------------- /docs/explanation/services.md: -------------------------------------------------------------------------------- 1 | # Services 2 | 3 | A *service* is one or more processes handling a common task. 4 | These processes operate as a tightly-coupled whole. 5 | 6 | All processes in a service will generally: 7 | 8 | * Share the same API class definitions 9 | * Moreover, they will normally share the same codebase 10 | * Create a single instance of the bus client in `bus.py` using 11 | `bus = lightbus.create()`. 12 | 13 | --- 14 | 15 | For example, your company may have the following: 16 | 17 | * An online store 18 | * A price monitoring script 19 | * An image resizing resizing process 20 | 21 | Each of these would be a service. 22 | 23 | The store service would have a web process and a 24 | Lightbus process. The image resizing service & and price monitoring services 25 | would each likely have a Lightbus process only. 26 | 27 | A simple lightbus deployment could therefore look something like this: 28 | 29 | ![A simple Lightbus deployment][simple-processes] 30 | 31 | [simple-processes]: /static/images/simple-processes.png 32 | 33 | In this example the following actions would take place: 34 | 35 | * Django reads from the web service database in order to serve web content 36 | * The online shop's Lightbus process receives pricing events from the 37 | price monitoring service. It updates products in the database using 38 | this new pricing data. 39 | * When the Django app receives an image upload, it performs a RPC to the 40 | image resizing service to resize the image[^1]. 41 | 42 | 43 | 44 | [^1]: Making the Django process wait for an RPC to respond is 45 | probably a bad idea in this case, but it illustrates how it 46 | *could* be done. Using an event (which is fire-and-forget) 47 | could be more suitable in reality. 48 | -------------------------------------------------------------------------------- /tests/test_creaton.py: -------------------------------------------------------------------------------- 1 | import inspect 2 | 3 | import pytest 4 | 5 | from lightbus import import_bus_module, BusPath 6 | from lightbus.creation import get_bus 7 | from lightbus.exceptions import FailedToImportBusModule 8 | 9 | pytestmark = pytest.mark.unit 10 | 11 | 12 | # import_bus_module() 13 | 14 | 15 | def test_import_bus_module_ok(make_test_bus_module): 16 | bus_module_name = make_test_bus_module() 17 | bus_module = import_bus_module(bus_module_name) 18 | 19 | assert inspect.ismodule(bus_module) 20 | assert isinstance(bus_module.bus, BusPath) 21 | 22 | 23 | def test_import_bus_module_does_not_exist(): 24 | bus_module_name = "does_not_exist" 25 | with pytest.raises(FailedToImportBusModule) as e: 26 | import_bus_module(bus_module_name) 27 | 28 | assert "failed to import" in str(e.value).lower() 29 | 30 | 31 | def test_import_bus_module_does_not_contain_bus(make_test_bus_module): 32 | bus_module_name = make_test_bus_module(code="") 33 | with pytest.raises(FailedToImportBusModule) as e: 34 | import_bus_module(bus_module_name) 35 | 36 | assert "attribute" in str(e.value).lower() 37 | 38 | 39 | def test_import_bus_module_contains_bus_but_wrong_type(make_test_bus_module): 40 | bus_module_name = make_test_bus_module(code="bus = 123") 41 | with pytest.raises(FailedToImportBusModule) as e: 42 | import_bus_module(bus_module_name) 43 | 44 | assert "invalid value" in str(e.value).lower() 45 | assert "int" in str(e.value).lower() 46 | 47 | 48 | # get_bus() - a wrapper around import_bus_module() 49 | 50 | 51 | def test_get_bus(make_test_bus_module): 52 | bus_module_name = make_test_bus_module() 53 | bus = get_bus(bus_module_name) 54 | assert isinstance(bus, BusPath) 55 | -------------------------------------------------------------------------------- /docs/howto/metrics.md: -------------------------------------------------------------------------------- 1 | # How to use Lightbus for metrics 2 | 3 | When we talk about metrics we may mean all or any of the following: 4 | 5 | * Current information is most important 6 | * Previous events will become irrelevant as soon as new data is received 7 | * Lost events are therefore tolerable, as long as we keep up with new events 8 | * Events may be high volume, so optimisations may be needed 9 | 10 | !!! note 11 | 12 | Your needs may not precisely match this scenario, so be prepared to tweak the following configuration to your needs. 13 | 14 | For the above metrics-based scenario, a sample Lightbus [configuration](../reference/configuration.md) may look like 15 | something like this: 16 | 17 | ```yaml 18 | # Lightbus config for metrics 19 | 20 | bus: 21 | schema: 22 | transport: 23 | redis: 24 | url: "redis://redis_host:6379/0" 25 | 26 | apis: 27 | 28 | # Here we specify the default for your entire bus, but you could 29 | # also specify the config for a specific API by using the API's name 30 | # instead of 'default'. 31 | default: 32 | 33 | # Disable validation to enhance performance 34 | validate: 35 | outgoing: false 36 | incoming: false 37 | 38 | # Assume we will be transmitting simple types, so we can bypass casting for performance 39 | cast_values: false 40 | 41 | event_transport: 42 | redis: 43 | url: 'redis://redis_host:6379/0' 44 | 45 | # Load in many events at once for performance improvements 46 | batch_size: 100 47 | 48 | # No need to keep many historical events around 49 | max_stream_length: 10000 50 | 51 | # Per-event streams, to allow selective consumption of metrics 52 | stream_use: "per_event" 53 | ``` 54 | -------------------------------------------------------------------------------- /lightbus/serializers/blob.py: -------------------------------------------------------------------------------- 1 | """ Serializers suitable for transports which require a single serialised value 2 | 3 | These serializers handle moving data to/from a string-based format. 4 | 5 | """ 6 | from typing import Union, TYPE_CHECKING 7 | 8 | from lightbus.serializers.base import ( 9 | decode_bytes, 10 | sanity_check_metadata, 11 | MessageSerializer, 12 | MessageDeserializer, 13 | ) 14 | 15 | if TYPE_CHECKING: 16 | # pylint: disable=unused-import,cyclic-import 17 | from lightbus import Message 18 | 19 | 20 | class BlobMessageSerializer(MessageSerializer): 21 | def __call__(self, message: "Message") -> str: 22 | """Takes a message object and returns a serialised string representation""" 23 | return self.encoder({"metadata": message.get_metadata(), "kwargs": message.get_kwargs()}) 24 | 25 | 26 | class BlobMessageDeserializer(MessageDeserializer): 27 | def __call__(self, serialized: Union[str, dict], *, native_id=None, **extra): 28 | """ Takes a serialised string representation and returns a Message object 29 | 30 | Reverse of BlobMessageSerializer 31 | """ 32 | # Allow for receiving dicts on the assumption that this will be 33 | # json which has already been decoded. 34 | if isinstance(serialized, dict): 35 | decoded = serialized 36 | else: 37 | serialized = decode_bytes(serialized) 38 | decoded = self.decoder(serialized) 39 | 40 | metadata = decoded.get("metadata", {}) 41 | kwargs = decoded.get("kwargs", {}) 42 | 43 | sanity_check_metadata(self.message_class, metadata) 44 | 45 | return self.message_class.from_dict( 46 | metadata=metadata, kwargs=kwargs, native_id=native_id, **extra 47 | ) 48 | -------------------------------------------------------------------------------- /lightbus_experiments/kombu_zmq/producer.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import os 3 | import timeit 4 | from random import randint 5 | from socket import gethostbyname, gethostname 6 | 7 | import zmq 8 | from kombu import Connection 9 | from timer import Timer 10 | 11 | TOTAL_MESSAGES = int(sys.argv[1]) 12 | 13 | context = zmq.Context() 14 | socket = context.socket(zmq.PULL) 15 | PORT = socket.bind_to_random_port("tcp://*") 16 | HOST = gethostbyname(gethostname()) 17 | 18 | 19 | amqp_timer = Timer() 20 | zmq_timer = Timer() 21 | 22 | 23 | def log(msg): 24 | pass 25 | # print(msg) 26 | 27 | 28 | log("Listening for results on {}:{}".format(HOST, PORT)) 29 | 30 | with Connection("amqp://guest:guest@localhost:5672//") as conn: 31 | simple_queue = conn.SimpleQueue("simple_queue") 32 | simple_queue.put("kick-off!") 33 | 34 | def put(): 35 | message_id = str(randint(100, 999)) 36 | with amqp_timer: 37 | simple_queue.put( 38 | b"x" * 1024, 39 | headers={"reply-to": "tcp://{}:{}".format(HOST, PORT), "id": message_id}, 40 | ) 41 | 42 | log("Message {} has been put. Waiting for response...".format(message_id)) 43 | with zmq_timer: 44 | content = socket.recv() 45 | 46 | # Make sure we got the response for the message we sent 47 | assert message_id == str(content, "utf8").split(" ", 1)[0] 48 | 49 | log("Response received") 50 | 51 | seconds = timeit.timeit(put, number=TOTAL_MESSAGES) 52 | print("Time per put: {}ms".format(round(seconds * 1000 / TOTAL_MESSAGES, 2))) 53 | print("Puts per second: {}".format(round(TOTAL_MESSAGES / seconds, 2))) 54 | print("ZeroMQ time: {}".format(zmq_timer)) 55 | print("AMQP time: {}".format(amqp_timer)) 56 | simple_queue.close() 57 | -------------------------------------------------------------------------------- /lightbus/commands/dump_schema.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import logging 3 | 4 | import sys 5 | from pathlib import Path 6 | 7 | from lightbus.commands import utilities as command_utilities 8 | from lightbus.plugins import PluginRegistry 9 | from lightbus.utilities.async_tools import block 10 | 11 | logger = logging.getLogger(__name__) 12 | 13 | 14 | class Command: 15 | def setup(self, parser, subparsers): 16 | parser_dumpschema = subparsers.add_parser( 17 | "dumpschema", 18 | help="Dumps all currently present bus schemas to a file or directory", 19 | formatter_class=argparse.ArgumentDefaultsHelpFormatter, 20 | ) 21 | group = parser_dumpschema.add_argument_group(title="Dump config schema command arguments") 22 | group.add_argument( 23 | "--out", 24 | "-o", 25 | help=( 26 | "File or directory to write schema to. If a directory is " 27 | "specified one schema file will be created for each API. " 28 | "If omitted the schema will be written to standard out." 29 | ), 30 | metavar="FILE_OR_DIRECTORY", 31 | ) 32 | command_utilities.setup_common_arguments(parser_dumpschema) 33 | parser_dumpschema.set_defaults(func=self.handle) 34 | 35 | def handle(self, args, config, plugin_registry: PluginRegistry): 36 | command_utilities.setup_logging(args.log_level or "warning", config) 37 | 38 | bus_module, bus = command_utilities.import_bus(args) 39 | block(bus.client.lazy_load_now()) 40 | bus.schema.save_local(args.out) 41 | 42 | if args.out: 43 | sys.stderr.write( 44 | "Schema for {} APIs saved to {}\n".format( 45 | len(bus.schema.api_names), Path(args.out).resolve() 46 | ) 47 | ) 48 | -------------------------------------------------------------------------------- /docs/explanation/lightbus-vs-celery.md: -------------------------------------------------------------------------------- 1 | # Lightbus vs Celery 2 | 3 | Lightbus was conceived as a result of using Celery to communicate 4 | between multiple Python services. 5 | 6 | ## Differences in principle 7 | 8 | Celery is a task queue: 9 | 10 | * A task queue is tightly coupled. The dispatching code must know what needs to be done 11 | * A task queue typically doesn't return results 12 | 13 | Lightbus is a bus: 14 | 15 | * A bus provides loose coupling. The dispatching code says what did happen, not what should happen (events) 16 | * A bus provides bi-directional communication (remote procedure calls) 17 | 18 | ## Differences in practice 19 | 20 | A number of pain points were identified with Celery that Lightbus 21 | aims to address. In particular: 22 | 23 | * Single vs multi-[service] – Celery is targeted as being a task queue for a service, rather than a means for multiple services to interact. 24 | * Conceptual overlap – The mapping between concepts in Celery and the underlying broker (AMQP at the time) is both unclear and overlapping. 25 | Lightbus provides a limited set of well defined concepts to avoid this confusion. 26 | * Non-sane defaults – Some Celery settings have non-sane defaults, making setup somewhat perilous at times. 27 | Lightbus provides sane defaults for most circumstances, and documentation specifically geared to certain use cases ([metrics], [event sourcing]) 28 | * Tight coupling (as discussed above) – Celery tasks define the action to take, not what happened. Lightbus uses events, 29 | which describe happened, and listening services decide the action to take. 30 | * General feeling – Celery feels large and opaque, debugging issues was challenging. Lightbus aims to feel lightweight, with clear 31 | logging and debugging tools. 32 | 33 | [service]: ../explanation/services.md 34 | [metrics]: ../howto/metrics.md 35 | [event sourcing]: ../howto/event-sourcing.md 36 | -------------------------------------------------------------------------------- /tests/client/test_utilities_unit.py: -------------------------------------------------------------------------------- 1 | import asyncio 2 | import traceback 3 | 4 | import pytest 5 | 6 | from lightbus.client.utilities import queue_exception_checker, Error, ErrorQueueType 7 | from lightbus.utilities.internal_queue import InternalQueue 8 | 9 | pytestmark = pytest.mark.unit 10 | 11 | 12 | class ExampleException(Exception): 13 | pass 14 | 15 | 16 | @pytest.fixture 17 | def erroring_coroutine(): 18 | async def erroring_coroutine_(): 19 | raise ExampleException 20 | 21 | return erroring_coroutine_ 22 | 23 | 24 | @pytest.fixture 25 | def ok_coroutine(): 26 | async def ok_coroutine_(): 27 | pass 28 | 29 | return ok_coroutine_ 30 | 31 | 32 | @pytest.fixture 33 | def error_queue(): 34 | return InternalQueue() 35 | 36 | 37 | @pytest.mark.asyncio 38 | async def test_queue_exception_checker_directly(erroring_coroutine, error_queue: ErrorQueueType): 39 | coroutine = queue_exception_checker(erroring_coroutine(), error_queue) 40 | 41 | with pytest.raises(ExampleException): 42 | await coroutine 43 | 44 | assert error_queue.qsize() == 1 45 | error: Error = error_queue.get_nowait() 46 | 47 | assert error.type == ExampleException 48 | assert isinstance(error.value, ExampleException) 49 | assert "test_utilities_unit.py" in str(error) 50 | assert "ExampleException" in str(error) 51 | 52 | 53 | def test_queue_exception_checker_in_task(erroring_coroutine, error_queue: ErrorQueueType): 54 | coroutine = queue_exception_checker(erroring_coroutine(), error_queue) 55 | 56 | with pytest.raises(ExampleException): 57 | asyncio.run(coroutine) 58 | 59 | assert error_queue.qsize() == 1 60 | error: Error = error_queue.get_nowait() 61 | 62 | assert error.type == ExampleException 63 | assert isinstance(error.value, ExampleException) 64 | assert "test_utilities_unit.py" in str(error) 65 | assert "ExampleException" in str(error) 66 | -------------------------------------------------------------------------------- /docs/howto/access-your-bus-client.md: -------------------------------------------------------------------------------- 1 | # How to access your bus client 2 | 3 | You create your bus client in your bus module 4 | (typically called `bus.py`) with the line: 5 | 6 | ```python3 7 | # Creating your bus client in your bus.py file 8 | import lightbus 9 | 10 | bus = lightbus.create() 11 | ``` 12 | 13 | However, you will often need to make use of your bus client 14 | in other areas of your codebase. For example, you may need to 15 | fire an event when a web form is submitted. 16 | 17 | You can access the bus client in two ways. 18 | 19 | ## Method 1: Direct import (recommended) 20 | 21 | The first approach is to import your bus client directly from 22 | your bus module, in the same way you would import anything else 23 | in your codebase: 24 | 25 | ```python3 26 | # For example 27 | from bus import bus 28 | 29 | # ...or if your service has its own package 30 | from my_service.bus import bus 31 | ``` 32 | 33 | You should use this approach in code which is specific to your 34 | service (i.e. non-shared/non-library code). This approach is more 35 | explicit (good), but hard codes the path to your bus module (bad for shared code). 36 | 37 | ## Method 2: `get_bus()` 38 | 39 | The second approach uses the `lightbus.get_bus()` function. This will 40 | use the [module loading configuration] to determine the bus module location. 41 | If the bus module has already been imported, then the module's 42 | `bus` attribute will simply be returned. Otherwise the bus module will be 43 | imported first. 44 | 45 | ```python3 46 | # Anywhere in your codebase 47 | import lightbus 48 | 49 | bus = lightbus.get_bus() 50 | ``` 51 | 52 | This approach is best suited to when you do not know where your bus module 53 | will be at runtime. This could be the case when: 54 | 55 | * Writing shared code which is used by multiple services 56 | * Writing a third party library 57 | 58 | 59 | [module loading configuration]: ../reference/configuration.md#1-module-loading 60 | -------------------------------------------------------------------------------- /docs/explanation/rpcs.md: -------------------------------------------------------------------------------- 1 | # Remote Procedure Calls (RPCs) 2 | 3 | A remote procedure call is where you call a procedure available on the bus. 4 | The sequence of events is: 5 | 6 | * You call the RPC, `bus.auth.check_password()` 7 | * An autoratitive process for the `auth` API handles the request and sends the response. 8 | * You receive the result 9 | 10 | --- 11 | 12 | Remote Procedure Calls are useful when: 13 | 14 | 1. You require information from a service [^1] 15 | 2. You wish to wait until a remote procedure has completed an action 16 | 17 | You can perform an RPC as follows: 18 | 19 | ```python3 20 | support_case = bus.support.case.get(pk=123) 21 | ``` 22 | 23 | RPCs do not provide a *fire and forget* mode of operation. 24 | Consider using [events] if you need this feature. 25 | 26 | ## At most once semantics 27 | 28 | Remote procedure calls will be processed at most once. In some situations the call will 29 | never be processed, in which case the client will raise a `LigutbusTimeout` exception. 30 | 31 | ## Considerations 32 | 33 | Whether to use RPCs or events for communication will depend upon your project's particular needs. 34 | Some considerations are: 35 | 36 | * RPCs are **conceptually simple**. You call a procedure and wait for a response. You do not need to 37 | store any state locally, you can simply request data on demand (performance considerations aside). 38 | * RPCs can be **fragile**. Any errors in the remote service will propagate to the client's service. 39 | You should handle these if possible. 40 | * Their use within a codebase may be non-obvious, leading to poor performance. 41 | Lightbus tries to alleviate this somewhat by using the 42 | `bus.api.method()` calling format, making it clear that this is a 43 | bus-based operation. 44 | 45 | [^1]: This is also achievable with [events]. However you will need to listen 46 | for the events and likely store the data locally. See the [events] 47 | section for further discussion. 48 | 49 | 50 | [events]: events.md 51 | -------------------------------------------------------------------------------- /tests/serializers/test_blob.py: -------------------------------------------------------------------------------- 1 | import json 2 | 3 | import pytest 4 | 5 | from lightbus.message import EventMessage 6 | from lightbus.serializers.blob import BlobMessageSerializer, BlobMessageDeserializer 7 | 8 | pytestmark = pytest.mark.unit 9 | 10 | 11 | def test_blob_serializer(): 12 | serializer = BlobMessageSerializer() 13 | serialized = serializer( 14 | EventMessage(api_name="my.api", event_name="my_event", id="123", kwargs={"field": "value"}) 15 | ) 16 | assert json.loads(serialized) == { 17 | "metadata": {"api_name": "my.api", "event_name": "my_event", "id": "123", "version": 1}, 18 | "kwargs": {"field": "value"}, 19 | } 20 | 21 | 22 | def test_blob_deserializer(): 23 | deserializer = BlobMessageDeserializer(EventMessage) 24 | message = deserializer( 25 | json.dumps( 26 | { 27 | "metadata": { 28 | "api_name": "my.api", 29 | "event_name": "my_event", 30 | "id": "123", 31 | "version": 1, 32 | }, 33 | "kwargs": {"field": "value"}, 34 | } 35 | ), 36 | native_id="456", 37 | ) 38 | assert message.api_name == "my.api" 39 | assert message.event_name == "my_event" 40 | assert message.id == "123" 41 | assert message.kwargs == {"field": "value"} 42 | assert message.native_id == "456" 43 | 44 | 45 | def test_blob_deserializer_dict(): 46 | deserializer = BlobMessageDeserializer(EventMessage) 47 | message = deserializer( 48 | { 49 | "metadata": {"api_name": "my.api", "event_name": "my_event", "id": "123", "version": 1}, 50 | "kwargs": {"field": "value"}, 51 | }, 52 | native_id="456", 53 | ) 54 | assert message.api_name == "my.api" 55 | assert message.event_name == "my_event" 56 | assert message.id == "123" 57 | assert message.kwargs == {"field": "value"} 58 | assert message.native_id == "456" 59 | -------------------------------------------------------------------------------- /tests/serializers/test_by_field.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | from lightbus.message import EventMessage 4 | from lightbus.serializers.by_field import ByFieldMessageSerializer, ByFieldMessageDeserializer 5 | 6 | pytestmark = pytest.mark.unit 7 | 8 | 9 | def test_by_field_serializer(): 10 | serializer = ByFieldMessageSerializer() 11 | serialized = serializer( 12 | EventMessage( 13 | api_name="my.api", event_name="my_event", kwargs={"field": "value"}, id="123", version=2 14 | ) 15 | ) 16 | assert serialized == { 17 | "api_name": "my.api", 18 | "event_name": "my_event", 19 | ":field": '"value"', 20 | "id": "123", 21 | "version": 2, 22 | } 23 | 24 | 25 | def test_by_field_deserializer(): 26 | deserializer = ByFieldMessageDeserializer(EventMessage) 27 | message = deserializer( 28 | { 29 | "api_name": "my.api", 30 | "event_name": "my_event", 31 | "id": "123", 32 | "version": "2", 33 | ":field": '"value"', 34 | }, 35 | native_id="456", 36 | ) 37 | assert message.api_name == "my.api" 38 | assert message.event_name == "my_event" 39 | assert message.id == "123" 40 | assert message.kwargs == {"field": "value"} 41 | assert message.version == 2 42 | assert message.native_id == "456" 43 | 44 | 45 | def test_by_field_deserializer_empty_keys_and_values(): 46 | deserializer = ByFieldMessageDeserializer(EventMessage) 47 | message = deserializer( 48 | { 49 | "api_name": "my.api", 50 | "event_name": "my_event", 51 | "id": "123", 52 | "version": "2", 53 | ":field": '"value"', 54 | "": "", 55 | }, 56 | native_id="456", 57 | ) 58 | assert message.api_name == "my.api" 59 | assert message.event_name == "my_event" 60 | assert message.id == "123" 61 | assert message.kwargs == {"field": "value"} 62 | assert message.version == 2 63 | -------------------------------------------------------------------------------- /lightbus_experiments/custom_loop.py: -------------------------------------------------------------------------------- 1 | import asyncio 2 | import logging 3 | 4 | 5 | # Logging setup 6 | import threading 7 | import time 8 | from concurrent.futures import ThreadPoolExecutor 9 | 10 | 11 | class AsyncioLoggingFilter(logging.Filter): 12 | def filter(self, record): 13 | task = asyncio.Task.current_task() 14 | 15 | record.task = f'[task {id(task)}]' if task else '[NOLOOP ]' 16 | return True 17 | 18 | 19 | logger = logging.getLogger(__name__) 20 | logger.addFilter(AsyncioLoggingFilter()) 21 | logging.getLogger('asyncio').setLevel(logging.CRITICAL) 22 | 23 | 24 | logging.basicConfig(level=logging.INFO, format="%(msecs)f %(threadName)s %(task)s %(msg)s") 25 | 26 | 27 | class ThreadSerializedTask(asyncio.Task): 28 | _lock = threading.Lock() 29 | 30 | def _wakeup(self, *args, **kwargs): 31 | logger.debug("Acquire lock") 32 | ThreadSerializedTask._lock.acquire() 33 | 34 | super()._wakeup(*args, **kwargs) 35 | 36 | logger.debug("Releasing lock") 37 | ThreadSerializedTask._lock.release() 38 | 39 | 40 | def task_factory(loop, coro): 41 | return ThreadSerializedTask(coro, loop=loop) 42 | 43 | 44 | async def one(): 45 | await asyncio.sleep(0.01) 46 | logger.debug("-> One") 47 | await two() 48 | await asyncio.sleep(0.01) 49 | logger.debug("-> Exiting one") 50 | 51 | 52 | async def two(): 53 | await asyncio.sleep(0.01) 54 | 55 | logger.info("--> Should not be interleaved with other threads") 56 | time.sleep(0.01) 57 | logger.info("--> Should not be interleaved with other threads") 58 | time.sleep(0.01) 59 | logger.info("--> Should not be interleaved with other threads") 60 | 61 | 62 | def run_loop(): 63 | loop = asyncio.new_event_loop() 64 | loop.set_task_factory(task_factory) 65 | loop.run_until_complete(one()) 66 | 67 | 68 | if __name__ == '__main__': 69 | 70 | threads = [] 71 | for _ in range(0, 5): 72 | thread = threading.Thread(target=run_loop) 73 | thread.start() 74 | threads.append(thread) 75 | 76 | [t.join() for t in threads] 77 | -------------------------------------------------------------------------------- /lightbus/commands/shell.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import logging 3 | import sys 4 | from inspect import isclass 5 | 6 | import lightbus 7 | from lightbus.commands import utilities as command_utilities 8 | from lightbus.plugins import PluginRegistry 9 | from lightbus.utilities.async_tools import block 10 | 11 | logger = logging.getLogger(__name__) 12 | 13 | 14 | class Command: 15 | def setup(self, parser, subparsers): 16 | parser_shell = subparsers.add_parser( 17 | "shell", 18 | help="Provide an interactive Lightbus shell", 19 | formatter_class=argparse.ArgumentDefaultsHelpFormatter, 20 | ) 21 | command_utilities.setup_common_arguments(parser_shell) 22 | parser_shell.set_defaults(func=self.handle) 23 | 24 | def handle(self, args, config, plugin_registry: PluginRegistry, fake_it=False): 25 | command_utilities.setup_logging(args.log_level or "warning", config) 26 | 27 | try: 28 | # pylint: disable=unused-import,cyclic-import,import-outside-toplevel 29 | import bpython 30 | from bpython.curtsies import main as bpython_main 31 | except ImportError: # pragma: no cover 32 | print("Lightbus shell requires bpython. Run `pip install bpython` to install bpython.") 33 | sys.exit(1) 34 | return # noqa 35 | 36 | lightbus_logger = logging.getLogger("lightbus") 37 | lightbus_logger.setLevel(logging.WARNING) 38 | 39 | bus_module, bus = command_utilities.import_bus(args) 40 | block(bus.client.lazy_load_now()) 41 | 42 | objects = {k: v for k, v in lightbus.__dict__.items() if isclass(v)} 43 | objects.update(bus=bus) 44 | 45 | block(plugin_registry.execute_hook("receive_args", args=args), timeout=5) 46 | 47 | # Ability to not start up the repl is useful for testing 48 | if not fake_it: 49 | bpython_main( 50 | args=["-i", "-q"], 51 | locals_=objects, 52 | welcome_message="Welcome to the Lightbus shell. Use `bus` to access your bus.", 53 | ) 54 | -------------------------------------------------------------------------------- /tests/transports/redis/test_unit_redis_utils.py: -------------------------------------------------------------------------------- 1 | from datetime import datetime, timezone 2 | 3 | import pytest 4 | 5 | from lightbus.transports.redis.utilities import ( 6 | redis_stream_id_subtract_one, 7 | redis_steam_id_to_datetime, 8 | datetime_to_redis_steam_id, 9 | redis_stream_id_add_one, 10 | ) 11 | 12 | pytestmark = pytest.mark.unit 13 | 14 | 15 | def test_redis_stream_id_subtract_one(): 16 | assert redis_stream_id_subtract_one("1514028809812-0") == "1514028809811-9999" 17 | assert redis_stream_id_subtract_one("1514028809812-10") == "1514028809812-9" 18 | assert redis_stream_id_subtract_one("0000000000000-0") == "0000000000000-0" 19 | 20 | 21 | def test_redis_stream_id_add_one(): 22 | assert redis_stream_id_add_one("1514028809812-0") == "1514028809812-1" 23 | assert redis_stream_id_add_one("1514028809812-10") == "1514028809812-11" 24 | assert redis_stream_id_add_one("0000000000000-0") == "0000000000000-1" 25 | 26 | 27 | def test_redis_steam_id_to_datetime(): 28 | assert redis_steam_id_to_datetime("0000000000000-0") == datetime( 29 | 1970, 1, 1, 0, 0, tzinfo=timezone.utc 30 | ) 31 | assert redis_steam_id_to_datetime("0000000000000-1") == datetime( 32 | 1970, 1, 1, 0, 0, 0, 1, tzinfo=timezone.utc 33 | ) 34 | assert redis_steam_id_to_datetime("1514028809812-10") == datetime( 35 | 2017, 12, 23, 11, 33, 29, 812_010, tzinfo=timezone.utc 36 | ) 37 | assert redis_steam_id_to_datetime(b"0000000000000-0") == datetime( 38 | 1970, 1, 1, 0, 0, tzinfo=timezone.utc 39 | ) 40 | 41 | 42 | def test_datetime_to_redis_steam_id(): 43 | assert ( 44 | datetime_to_redis_steam_id(datetime(1970, 1, 1, 0, 0, tzinfo=timezone.utc)) 45 | == "0000000000000-0" 46 | ) 47 | # Microseconds don't get added on as the sequence number 48 | assert ( 49 | datetime_to_redis_steam_id(datetime(1970, 1, 1, 0, 0, 0, 99, tzinfo=timezone.utc)) 50 | == "0000000000000-0" 51 | ) 52 | assert ( 53 | datetime_to_redis_steam_id(datetime(2017, 12, 23, 11, 33, 29, 812_010, tzinfo=timezone.utc)) 54 | == "1514028809812-0" 55 | ) 56 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # What is Lightbus? 2 | 3 | [](https://app.codacy.com/gh/adamcharnock/lightbus/dashboard) 4 | [](https://lightbus.org/reference/code-of-conduct/) 5 | 6 | Lightbus allows your backend processes to communicate, run background 7 | tasks, and expose internal APIs. 8 | 9 | Lightbus uses Redis as its underlying transport, although support for 10 | other platforms may eventually be added. 11 | 12 | Lightbus requires Python 3.9 or above. 13 | 14 | **Full documentation can be found at https://lightbus.org** 15 | 16 | **[Adam Charnock](https://adamcharnock.com) is available for freelance/contract work.** 17 | 18 | ## Designed for ease of use 19 | 20 | Lightbus is designed with developers in mind. The syntax aims to be 21 | intuitive and familiar, and common problems are caught with clear and 22 | helpful error messages. 23 | 24 | For example, a naïve authentication API: 25 | 26 | ``` python3 27 | class AuthApi(Api): 28 | user_registered = Event(parameters=('username', 'email')) 29 | 30 | class Meta: 31 | name = 'auth' 32 | 33 | def check_password(self, user, password): 34 | return ( 35 | user == 'admin' 36 | and password == 'secret' 37 | ) 38 | ``` 39 | 40 | This can be called as follows: 41 | 42 | ``` python3 43 | import lightbus 44 | 45 | bus = lightbus.create() 46 | 47 | bus.auth.check_password( 48 | user='admin', 49 | password='secret' 50 | ) 51 | # Returns true 52 | ``` 53 | 54 | You can also listen for events: 55 | 56 | ``` python3 57 | import lightbus 58 | 59 | bus = lightbus.create() 60 | 61 | def send_signup_email(event_message, 62 | username, email): 63 | send_mail(email, 64 | subject=f'Welcome {username}' 65 | ) 66 | 67 | @bus.client.on_start() 68 | def bus_start(client): 69 | bus.auth.user_registered.listen( 70 | send_signup_email 71 | ) 72 | ``` 73 | 74 | **To get started checkout the documentation at https://lightbus.org.** 75 | -------------------------------------------------------------------------------- /docs/howto/schedule-recurring-tasks.md: -------------------------------------------------------------------------------- 1 | # How to schedule recurring tasks 2 | 3 | Recurring tasks can be scheduled in two ways: 4 | 5 | * The `@bus.client.every()` decorator – Will execute a function or coroutine at a given interval 6 | * The `@bus.client.schedule()` decorator – Similar to `every()`, but takes complex schedules as provided by the [schedule] library. 7 | 8 | ## Simple recurring tasks using `@bus.client.every()` 9 | 10 | Lightbus natively supports simple recurring tasks using the `@bus.client.every()` decorator: 11 | 12 | ```python3 13 | # bus.py 14 | import lightbus 15 | 16 | bus = lightbus.create() 17 | 18 | @bus.client.every(seconds=1) 19 | def do_it(): 20 | print("Hello!") 21 | 22 | ``` 23 | 24 | The interval can be specified using the `seconds`, `minutes`, `hours`, and `days` keys. 25 | Pass `also_run_immediately=True` to execute the function/coroutine immediately, as well as 26 | at the given interval. 27 | 28 | ## Complex schedules using `@bus.client.schedule()` 29 | 30 | Lightbus also supports using schedules specified using 31 | the [schedule] library. This allows for schedules 32 | such as 'every Monday at 1am', rather than simple intervals. 33 | For example: 34 | 35 | ```python3 36 | import lightbus 37 | import schedule 38 | 39 | bus = lightbus.create() 40 | 41 | # Run the task every 1-3 seconds, varying randomly 42 | @bus.client.schedule(schedule.every(1).to(3).seconds) 43 | def do_it(): 44 | print("Hello using schedule library") 45 | 46 | ``` 47 | 48 | ## Long running tasks 49 | 50 | If your tasks are long running you may prefer to handle these in a separate 51 | Lightbus process. This will avoid blocking the processing of incoming events and RPCs. 52 | 53 | For example, the default RPC timeout is 5 seconds. Any task which runs for longer than 54 | this has the possibility of causing incoming RPCs to timeout. 55 | 56 | You can move your task processing to a separate process as follows: 57 | 58 | ``` 59 | # Process 1: Handles scheduled tasks only 60 | lightbus run --only tasks 61 | 62 | # Process 2: Handles everything else (events and rpcs) 63 | lightbus run --skip tasks 64 | ``` 65 | 66 | [schedule]: https://github.com/dbader/schedule 67 | -------------------------------------------------------------------------------- /lightbus/utilities/config.py: -------------------------------------------------------------------------------- 1 | import inspect 2 | import logging 3 | import secrets 4 | import string 5 | 6 | from typing import Type, NamedTuple # pylint: disable=unused-import,cyclic-import 7 | 8 | import itertools 9 | 10 | logger = logging.getLogger(__name__) 11 | 12 | 13 | def make_from_config_structure(class_name, from_config_method, extra_parameters=tuple()) -> Type: 14 | """ 15 | Create a new named tuple based on the method signature of from_config_method. 16 | 17 | This is useful when dynamically creating the config structure for Transports 18 | and Plugins. 19 | """ 20 | # pylint: disable=exec-used 21 | 22 | code = f"class {class_name}Config(NamedTuple):\n pass\n" 23 | variables = dict(p={}) 24 | 25 | parameters = inspect.signature(from_config_method).parameters.values() 26 | for parameter in itertools.chain(parameters, extra_parameters): 27 | if parameter.name == "config": 28 | # The config parameter is always passed to from_config() in order to 29 | # give it access to the global configuration (useful for setting 30 | # sensible defaults) 31 | continue 32 | 33 | if parameter.kind in (parameter.POSITIONAL_ONLY, parameter.VAR_POSITIONAL): 34 | logger.warning( 35 | f"Positional-only arguments are not supported in from_config() on class {class_name}" 36 | ) 37 | elif parameter.kind in (parameter.VAR_KEYWORD,): 38 | logger.warning( 39 | f"**kwargs-style parameters are not supported in from_config() on class {class_name}" 40 | ) 41 | else: 42 | name = parameter.name 43 | variables["p"][name] = parameter 44 | code += f" {name}: p['{name}'].annotation = p['{name}'].default\n" 45 | 46 | globals_ = globals().copy() 47 | globals_.update(variables) 48 | exec(code, globals_) # nosec 49 | return globals_[f"{class_name}Config"] 50 | 51 | 52 | def random_name(length: int) -> str: 53 | """Get a random string suitable for a processes/consumer name""" 54 | return "".join(secrets.choice(string.ascii_lowercase) for _ in range(length)) 55 | -------------------------------------------------------------------------------- /lightbus/hooks.py: -------------------------------------------------------------------------------- 1 | import asyncio 2 | from collections import defaultdict 3 | from typing import Dict, NamedTuple, List, Callable, Optional 4 | 5 | from lightbus.client.utilities import queue_exception_checker, ErrorQueueType 6 | from lightbus.utilities.async_tools import run_user_provided_callable 7 | 8 | 9 | class CallbackKey(NamedTuple): 10 | name: str 11 | run_before_plugins: bool 12 | 13 | 14 | class HookRegistry: 15 | def __init__( 16 | self, 17 | error_queue: ErrorQueueType, 18 | execute_plugin_hooks: Callable, 19 | extra_parameters: Optional[dict] = None, 20 | ): 21 | self._hook_callbacks: Dict[CallbackKey, List[Callable]] = defaultdict(list) 22 | self.error_queue = error_queue 23 | self.execute_plugin_hooks = execute_plugin_hooks 24 | self.extra_parameters = extra_parameters or {} 25 | 26 | def set_extra_parameter(self, name, value): 27 | self.extra_parameters[name] = value 28 | 29 | async def execute(self, name, **kwargs): 30 | # Hooks that need to run before plugins 31 | key = CallbackKey(name, run_before_plugins=True) 32 | for callback in self._hook_callbacks[key]: 33 | await queue_exception_checker( 34 | run_user_provided_callable( 35 | callback, 36 | args=[], 37 | kwargs=dict(**self.extra_parameters, **kwargs), 38 | type_name="hook", 39 | ), 40 | self.error_queue, 41 | ) 42 | 43 | await self.execute_plugin_hooks(name, **self.extra_parameters, **kwargs) 44 | 45 | # Hooks that need to run after plugins 46 | key = CallbackKey(name, run_before_plugins=False) 47 | for callback in self._hook_callbacks[key]: 48 | await run_user_provided_callable( 49 | callback, 50 | args=[], 51 | kwargs=dict(**self.extra_parameters, **kwargs), 52 | type_name="hook", 53 | ) 54 | 55 | def register_callback(self, name, fn, before_plugins=False): 56 | key = CallbackKey(name, bool(before_plugins)) 57 | self._hook_callbacks[key].append(fn) 58 | -------------------------------------------------------------------------------- /lightbus/serializers/base.py: -------------------------------------------------------------------------------- 1 | import inspect 2 | import json 3 | from typing import Union, TypeVar, Type 4 | 5 | from lightbus.exceptions import InvalidMessage, InvalidSerializerConfiguration 6 | from lightbus.schema.encoder import json_encode 7 | 8 | 9 | def decode_bytes(b: Union[str, bytes]): 10 | return b.decode("utf8") if isinstance(b, bytes) else b 11 | 12 | 13 | def sanity_check_metadata(message_class, metadata): 14 | """Takes unserialized metadata and checks it looks sane 15 | 16 | This relies upon the required_metadata of each Message class 17 | to provide a list of metadata fields that are required. 18 | """ 19 | for required_key in message_class.required_metadata: 20 | if required_key not in metadata: 21 | raise InvalidMessage( 22 | "Required key '{key}' missing in {cls} metadata. " 23 | "Found keys: {keys}".format( 24 | key=required_key, keys=", ".join(metadata.keys()), cls=message_class.__name__ 25 | ) 26 | ) 27 | elif not metadata.get(required_key): 28 | raise InvalidMessage( 29 | "Required key '{key}' present in {cls} metadata but value was empty" 30 | "".format(key=required_key, cls=message_class.__name__) 31 | ) 32 | 33 | 34 | SerialisedData = TypeVar("SerialisedData") 35 | 36 | 37 | class MessageSerializer: 38 | def __init__(self, encoder=json_encode): 39 | self.encoder = encoder 40 | 41 | def __call__(self, message: "lightbus.Message") -> SerialisedData: 42 | raise NotImplementedError() 43 | 44 | 45 | class MessageDeserializer: 46 | def __init__(self, message_class: Type["lightbus.Message"], decoder=json.loads): 47 | if not inspect.isclass(message_class): 48 | raise InvalidSerializerConfiguration( 49 | "The message_class value provided to JsonMessageDeserializer was not a class, " 50 | "it was actually: {}".format(message_class) 51 | ) 52 | 53 | self.message_class = message_class 54 | self.decoder = decoder 55 | 56 | def __call__(self, serialized: SerialisedData, *, native_id=None) -> "lightbus.Message": 57 | raise NotImplementedError() 58 | -------------------------------------------------------------------------------- /tests/transports/redis/conftest.py: -------------------------------------------------------------------------------- 1 | import asyncio 2 | import logging 3 | 4 | import pytest 5 | 6 | import lightbus 7 | import lightbus.creation 8 | import lightbus.transports.redis.event 9 | import lightbus.transports.redis.result 10 | import lightbus.transports.redis.rpc 11 | import lightbus.transports.redis.schema 12 | from lightbus.exceptions import BusAlreadyClosed 13 | from lightbus.transports.redis.event import StreamUse 14 | 15 | logger = logging.getLogger(__name__) 16 | 17 | 18 | @pytest.fixture 19 | async def redis_rpc_transport(new_redis_pool, loop): 20 | return lightbus.transports.redis.rpc.RedisRpcTransport( 21 | redis_pool=await new_redis_pool(maxsize=10000) 22 | ) 23 | 24 | 25 | @pytest.fixture 26 | async def redis_result_transport(new_redis_pool, loop): 27 | return lightbus.transports.redis.result.RedisResultTransport( 28 | redis_pool=await new_redis_pool(maxsize=10000) 29 | ) 30 | 31 | 32 | @pytest.fixture 33 | async def redis_event_transport(new_redis_pool, loop): 34 | transport = lightbus.transports.redis.event.RedisEventTransport( 35 | redis_pool=await new_redis_pool(maxsize=10000), 36 | service_name="test_service", 37 | consumer_name="test_consumer", 38 | # This used to be the default, so we still test against it here 39 | stream_use=StreamUse.PER_EVENT, 40 | ) 41 | yield transport 42 | await transport.close() 43 | 44 | 45 | @pytest.fixture 46 | async def redis_schema_transport(new_redis_pool, loop): 47 | return lightbus.transports.redis.schema.RedisSchemaTransport( 48 | redis_pool=await new_redis_pool(maxsize=10000) 49 | ) 50 | 51 | 52 | @pytest.fixture 53 | async def bus(new_bus): 54 | bus = new_bus() 55 | 56 | yield bus 57 | 58 | try: 59 | await bus.client.stop_worker() 60 | await bus.client.close_async() 61 | except BusAlreadyClosed: 62 | pass 63 | 64 | 65 | @pytest.fixture(name="fire_dummy_events") 66 | def fire_dummy_events_fixture(bus): 67 | async def fire_dummy_events(total, initial_delay=0.1): 68 | await asyncio.sleep(initial_delay) 69 | for x in range(0, total): 70 | await bus.my.dummy.my_event.fire_async(field=str(x)) 71 | logger.warning("TEST: fire_dummy_events() completed") 72 | 73 | return fire_dummy_events 74 | -------------------------------------------------------------------------------- /lightbus_experiments/kombu_zmq/consumer.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import timeit 3 | 4 | import zmq 5 | from kombu import Connection 6 | from timer import Timer 7 | 8 | TOTAL_MESSAGES = int(sys.argv[1]) 9 | 10 | 11 | amqp_timer = Timer() 12 | zmq_timer = Timer() 13 | 14 | 15 | def log(msg): 16 | pass 17 | # print(msg) 18 | 19 | 20 | def main(): 21 | context = zmq.Context() 22 | sockets = {} 23 | 24 | with Connection("amqp://guest:guest@127.0.0.1:5672//") as conn: 25 | simple_queue = conn.SimpleQueue("simple_queue") 26 | # Block until we get the 'ready to start' message 27 | print("Waiting for kick-off message from producer") 28 | simple_queue.get(block=True).ack() 29 | print("Got it! Let's go...") 30 | 31 | def get(): 32 | nonlocal sockets 33 | 34 | with amqp_timer: 35 | message = simple_queue.get(block=True) 36 | message_id = message.headers.get("id") 37 | addr = message.headers.get("reply-to") 38 | 39 | if not addr: 40 | with amqp_timer: 41 | message.ack() 42 | log("Message with no reply-to header. Ignoring.") 43 | return 44 | 45 | if addr not in sockets: 46 | log("Opening socket to: {}".format(addr)) 47 | with zmq_timer: 48 | socket = context.socket(zmq.PUSH) 49 | socket.connect(addr) 50 | sockets[addr] = socket 51 | 52 | socket = sockets[addr] 53 | log("Sending response for {} to: {}".format(message_id, addr)) 54 | 55 | # Send the message ID back plus some data 56 | with zmq_timer: 57 | socket.send(bytes(message_id, "utf8") + b" x" * 1024) 58 | 59 | log("Sent") 60 | 61 | with amqp_timer: 62 | message.ack() 63 | 64 | seconds = timeit.timeit(get, number=TOTAL_MESSAGES) 65 | print("Time per get: {}ms".format(round(seconds * 1000 / TOTAL_MESSAGES, 2))) 66 | print("Gets per second: {}".format(round(TOTAL_MESSAGES / seconds, 2))) 67 | print("ZeroMQ time: {}".format(zmq_timer)) 68 | print("AMQP time: {}".format(amqp_timer)) 69 | simple_queue.close() 70 | 71 | 72 | if __name__ == "__main__": 73 | main() 74 | -------------------------------------------------------------------------------- /lightbus_experiments/potential_use.py: -------------------------------------------------------------------------------- 1 | # Example use of API once it has been defined 2 | 3 | 4 | from mycompany.common.auth import api 5 | 6 | # Blocking calls 7 | user_info = api.get_user(username="testuser") 8 | password_ok = api.check_passsword(password="Passw0rd1") 9 | api.user_registered.listen(my_callback) 10 | 11 | # Parallel calls 12 | async_result1 = api.get_user.async(username="testuser") 13 | async_result2 = api.check_passsword.async(password="Passw0rd1") 14 | user_info = async_result1.wait() 15 | password_ok = async_result2.wait() 16 | 17 | 18 | # Pro 19 | # - Presence of 'api' indicates that this is an external service 20 | # - As api is an object, one cannot access the methods/events without the prefix 21 | # - Simple, readable 22 | # Con 23 | # - Use of 'api' is by convention only 24 | # - ...but we're all consenting adults. 25 | 26 | 27 | # Developer tools 28 | 29 | 30 | api.debug.enable() 31 | api.debug.disable() 32 | # Enables/disables debugging for calls using this api instance 33 | 34 | api.debug.info("get_user") 35 | # Shows: number of consumers, consumer information, last call timestamp, last call args, last call response, 36 | # last handled by 37 | 38 | api.debug.trace("get_user", username="testuser") 39 | # Shows: 40 | # - "WARNING: Only works when working with other warren consumers & producers" 41 | # - Total number of handlers listening for this api call/event [1] 42 | # - Raw message as sent to broker 43 | # - Expecting response to tcp://10.1.2.3:54142 44 | # - Raw message received by rabbitmq [1] 45 | # - "Waiting for debug information from handlers..." [3] 46 | # - Consumer ---> Message received from by PID at HOST 47 | # - Consumer ---> Raw message as received from broker: ... 48 | # - Consumer ---> Message being handled by implementation mycompany.common.auth.AuthApi.get_user() 49 | # - Consumer ---> Implementation returned result: ... 50 | # - Consumer ---> Returning result to tcp://10.1.2.3:54142 51 | # - Consumer ---> Acknowledgement sent to broker 52 | # - Consumer ---> All done 53 | # - Response received from consumer: ... 54 | # - Done 55 | 56 | # [1] Requires a debug back chanel of some sort (i.e. a debug exchange) 57 | # [2] Creates a temporary queue on the relevant exchange. 58 | # [3] The debugger sends the message with a 'debug-to' header, which logs back 59 | # via ZeroMQ 60 | -------------------------------------------------------------------------------- /docs/explanation/marshalling.md: -------------------------------------------------------------------------------- 1 | Lightbus has four stages of data marshalling: 2 | 3 | * Encode / Decode 4 | * Serialize / Deserialize 5 | * Validation 6 | * Deform / Cast 7 | 8 | An **inbound message** will go through this process from **top to bottom**. 9 | An **outbound message** will go through this process from **bottom to top**. 10 | 11 | ## Inbound flow 12 | 13 | Messages arriving from the bus go through the following stages 14 | in order to prepare the data for use: 15 | 16 | 1. **Decode:** Decode the incoming data (JSON decoding by default) 17 | 2. **Deserialise:** Convert decoded data into a `Message` object 18 | 3. **Validate:** Validate the incoming message against the JSON schema 19 | available on the bus. 20 | 4. **Cast:** Best effort casting of parameters/results based on 21 | the locally available type hinting. This can be disabled with the 22 | [`cast_values` configuration option](../reference/configuration.md#api-config). 23 | 24 | 25 | ## Outbound flow 26 | 27 | This is the reverse of the inbound flow. Messages being 28 | sent will go through the following process in order to 29 | prepare the data for transmission on bus: 30 | 31 | 1. **Deform:** Lightbus handles [NamedTuples], [dataclasses] 32 | and [other classes] by converting 33 | them into dictionaries. Other common types such as 34 | datetimes, Decimals etc are converted into strings. 35 | Internally this is referred to as the *deform* process and is 36 | the inverse of the *cast* process. 37 | 2. **Validate:** Validate the outgoing message against the JSON schema 38 | available on the bus. 39 | 3. **Serialize:** Structures the data in a way suitable for the 40 | transport. 41 | 4. **Encode:** Converts the data to a form suitable for transmission. 42 | This typically means stringifying it, for which lightbus 43 | uses JSON encoding by default. 44 | 45 | ## About casting 46 | 47 | Casting is separate from validation, although both rely on type hints. 48 | Whereas validation uses a shared 49 | bus-wide schema to check data validity, casting uses any Python type hints 50 | available in the **local codebase** to marshall event and RPC parameters 51 | into a format useful to the service's developer. 52 | 53 | 54 | 55 | 56 | [NamedTuples]: ../reference/typing.md#namedtuple-example 57 | [dataclasses]: ../reference/typing.md#dataclass-example 58 | [other classes]: ../reference/typing.md#custom-class-example 59 | -------------------------------------------------------------------------------- /tests/plugins/test_unit_plugin_base.py: -------------------------------------------------------------------------------- 1 | """Test the infrastructure for loading and calling plugins""" 2 | import pytest 3 | from collections import OrderedDict 4 | 5 | from lightbus.config import Config 6 | from lightbus.plugins import LightbusPlugin, PluginRegistry 7 | from lightbus.plugins.metrics import MetricsPlugin 8 | from lightbus.plugins.state import StatePlugin 9 | 10 | 11 | pytestmark = pytest.mark.unit 12 | 13 | 14 | def test_manually_set_plugins(plugin_registry: PluginRegistry): 15 | assert not plugin_registry._plugins 16 | p1 = LightbusPlugin() 17 | p2 = LightbusPlugin() 18 | plugin_registry.set_plugins([p1, p2]) 19 | assert plugin_registry._plugins == [p1, p2] 20 | 21 | 22 | def test_autoload_plugins(plugin_registry: PluginRegistry): 23 | config = Config.load_dict( 24 | {"plugins": {"internal_state": {"enabled": True}, "internal_metrics": {"enabled": True}}} 25 | ) 26 | assert not plugin_registry._plugins 27 | assert plugin_registry.autoload_plugins(config) 28 | assert [type(p) for p in plugin_registry._plugins] == [StatePlugin, MetricsPlugin] 29 | 30 | 31 | @pytest.mark.asyncio 32 | async def test_execute_hook(mocker, plugin_registry: PluginRegistry): 33 | """Ensure calling execute_hook() calls the method on the plugin""" 34 | assert not plugin_registry._plugins 35 | plugin = LightbusPlugin() 36 | plugin_registry.set_plugins([plugin]) 37 | 38 | mocker.spy(plugin, "before_worker_start") 39 | 40 | await plugin_registry.execute_hook("before_worker_start", client=None) 41 | assert plugin.before_worker_start.called 42 | 43 | 44 | def test_is_plugin_loaded(plugin_registry: PluginRegistry): 45 | assert plugin_registry.is_plugin_loaded(LightbusPlugin) == False 46 | plugin_registry.set_plugins([LightbusPlugin()]) 47 | assert plugin_registry.is_plugin_loaded(LightbusPlugin) == True 48 | 49 | 50 | def test_plugin_config(): 51 | # Is the Config attached to the plugin class by the 52 | # base plugin's metaclass? 53 | class PluginWithConfig(LightbusPlugin): 54 | @classmethod 55 | def from_config(cls, config, first: int = 123): 56 | pass 57 | 58 | assert PluginWithConfig.Config 59 | assert type(PluginWithConfig.Config) == type 60 | assert "config" not in PluginWithConfig.Config.__annotations__ 61 | assert "first" in PluginWithConfig.Config.__annotations__ 62 | assert PluginWithConfig.Config().first == 123 63 | -------------------------------------------------------------------------------- /lightbus/commands/utilities.py: -------------------------------------------------------------------------------- 1 | import logging 2 | 3 | from typing import Tuple, Any 4 | 5 | import lightbus.creation 6 | from lightbus import configure_logging, BusPath 7 | import lightbus.client 8 | from lightbus.config import Config 9 | from lightbus.exceptions import NoBusFoundInBusModule 10 | 11 | logger = logging.getLogger(__name__) 12 | 13 | 14 | def import_bus(args) -> Tuple[Any, BusPath]: 15 | bus_module = lightbus.creation.import_bus_module(args.bus_module_name) 16 | try: 17 | return bus_module, bus_module.bus 18 | except AttributeError: 19 | raise NoBusFoundInBusModule( 20 | f"Bus module at {bus_module.__file__} contains no variable named 'bus'. " 21 | f"Your bus module should contain the line 'bus = lightbus.create()'." 22 | ) 23 | 24 | 25 | def setup_logging(override: str, config: Config): 26 | configure_logging(log_level=(override or config.bus().log_level.value).upper()) 27 | 28 | 29 | def setup_common_arguments(parser): 30 | """Set common arguments needed by all commands""" 31 | general_argument_group = parser.add_argument_group(title="Common arguments") 32 | general_argument_group.add_argument( 33 | "--bus", 34 | "-b", 35 | dest="bus_module_name", 36 | metavar="BUS_MODULE", 37 | help=( 38 | "The bus module to import. Example 'bus', 'my_project.bus'. Defaults to " 39 | "the value of the LIGHTBUS_MODULE environment variable, or 'bus'" 40 | ), 41 | ) 42 | general_argument_group.add_argument( 43 | "--service-name", 44 | "-s", 45 | help="Name of service in which this process resides. YOU SHOULD " 46 | "LIKELY SET THIS IN PRODUCTION. Can also be set using the " 47 | "LIGHTBUS_SERVICE_NAME environment. Will default to a random string.", 48 | ) 49 | general_argument_group.add_argument( 50 | "--process-name", 51 | "-p", 52 | help="A unique name of this process within the service. Can also be set using the " 53 | "LIGHTBUS_PROCESS_NAME environment. Will default to a random string.", 54 | ) 55 | general_argument_group.add_argument( 56 | "--config", dest="config_file", help="Config file to load, JSON or YAML", metavar="FILE" 57 | ) 58 | general_argument_group.add_argument( 59 | "--log-level", 60 | help="Set the log level. Overrides any value set in config. " 61 | "One of debug, info, warning, critical, exception.", 62 | metavar="LOG_LEVEL", 63 | ) 64 | -------------------------------------------------------------------------------- /tests/transports/redis/test_reliability_redis_events.py: -------------------------------------------------------------------------------- 1 | import asyncio 2 | import logging 3 | from asyncio import CancelledError 4 | 5 | import pytest 6 | 7 | import lightbus 8 | import lightbus.path 9 | from lightbus import EventMessage 10 | from lightbus.utilities.async_tools import cancel 11 | from tests.conftest import Worker 12 | 13 | pytestmark = pytest.mark.reliability 14 | 15 | logger = logging.getLogger(__name__) 16 | 17 | 18 | @pytest.mark.asyncio 19 | async def test_listener_failures( 20 | bus: lightbus.path.BusPath, new_bus, caplog, redis_client, dummy_api, worker: Worker 21 | ): 22 | """Keep killing bus clients and check that we don't loose any events regardless""" 23 | 24 | caplog.set_level(logging.ERROR) 25 | event_ok_ids = dict() 26 | history = [] 27 | 28 | async def listener(event_message: EventMessage, field, **kwargs): 29 | call_id = int(field) 30 | event_ok_ids.setdefault(call_id, 0) 31 | event_ok_ids[call_id] += 1 32 | await asyncio.sleep(0.1) 33 | 34 | # Put a lot of events onto the bus (we'll pull them off shortly) 35 | bus.client.register_api(dummy_api) 36 | for n in range(0, 50): 37 | await bus.my.dummy.my_event.fire_async(field=str(n)) 38 | 39 | # Now pull the events off, and sometimes kill a worker early. 40 | # We kill 20% of listeners, so run 20% extra workers (we don't kill 41 | # any listeners in that extra 20% because these are just mop-up) 42 | for n in range(0, int(50 * 1.2)): 43 | cursed_bus = new_bus() 44 | cursed_bus.my.dummy.my_event.listen( 45 | listener, listener_name="test", bus_options={"since": "0"} 46 | ) 47 | 48 | async with worker(cursed_bus): 49 | logger.debug(f"Worker {n}") 50 | await asyncio.sleep(0.05) 51 | if n % 5 == 0 and n < 50: 52 | # Cancel 1 in every 5 attempts at handling the event 53 | tasks = cursed_bus.client.event_client._event_listener_tasks 54 | await cancel(list(tasks)[0]) 55 | await asyncio.sleep(0.15) 56 | 57 | await asyncio.sleep(0.1) 58 | 59 | info = await redis_client.xinfo_groups(stream="my.dummy.my_event:stream") 60 | 61 | duplicate_calls = [n for n, v in event_ok_ids.items() if v > 1] 62 | assert len(event_ok_ids) == 50, event_ok_ids 63 | assert len(duplicate_calls) > 0 64 | 65 | assert ( 66 | len(info) == 1 67 | ), "There should only be one consumer group which was reused by every listener above" 68 | assert info[0][b"pending"] == 0 69 | -------------------------------------------------------------------------------- /lightbus/client/validator.py: -------------------------------------------------------------------------------- 1 | import logging 2 | from typing import Union 3 | 4 | from lightbus.exceptions import UnknownApi 5 | from lightbus.message import EventMessage, RpcMessage, ResultMessage 6 | from lightbus.schema import Schema 7 | from lightbus.config import Config 8 | 9 | logger = logging.getLogger(__name__) 10 | 11 | 12 | def validate_incoming( 13 | config: Config, schema: Schema, message: Union[EventMessage, RpcMessage, ResultMessage] 14 | ): 15 | return _validate(config, schema, message, "incoming") 16 | 17 | 18 | def validate_outgoing( 19 | config: Config, schema: Schema, message: Union[EventMessage, RpcMessage, ResultMessage] 20 | ): 21 | return _validate(config, schema, message, "outgoing") 22 | 23 | 24 | def _validate( 25 | config, schema, message: Union[EventMessage, RpcMessage, ResultMessage], direction: str 26 | ): 27 | if direction not in ("incoming", "outgoing"): 28 | raise AssertionError("Invalid direction specified") 29 | 30 | # Result messages do not carry the api or procedure name, so allow them to be 31 | # specified manually 32 | api_name = message.api_name 33 | event_or_rpc_name = getattr(message, "procedure_name", None) or getattr(message, "event_name") 34 | api_config = config.api(api_name) 35 | strict_validation = api_config.strict_validation 36 | 37 | if not getattr(api_config.validate, direction): 38 | return 39 | 40 | if api_name not in schema: 41 | if strict_validation: 42 | raise UnknownApi( 43 | f"Validation is enabled for API named '{api_name}', but there is no schema present for this API. " 44 | f"Validation is therefore not possible. You are also seeing this error because the " 45 | f"'strict_validation' setting is enabled. Disabling this setting will turn this exception " 46 | f"into a warning. " 47 | ) 48 | else: 49 | logger.warning( 50 | f"Validation is enabled for API named '{api_name}', but there is no schema present for this API. " 51 | f"Validation is therefore not possible. You can force this to be an error by enabling " 52 | f"the 'strict_validation' config option. You can silence this message by disabling validation " 53 | f"for this API using the 'validate' option." 54 | ) 55 | return 56 | 57 | if isinstance(message, (RpcMessage, EventMessage)): 58 | schema.validate_parameters(api_name, event_or_rpc_name, message.kwargs) 59 | elif isinstance(message, ResultMessage): 60 | schema.validate_response(api_name, event_or_rpc_name, message.result) 61 | -------------------------------------------------------------------------------- /docs/reference/apis.md: -------------------------------------------------------------------------------- 1 | APIs specify the functionality available on the bus. To do this you 2 | define API classes within your `bus.py` file. You can also define 3 | your API elsewhere and import it into your `bus.py` file. 4 | 5 | **For further discussion of APIs [see the concepts section](../explanation/apis.md).** 6 | 7 | ## An example API 8 | 9 | ```python3 10 | # An example API. You can define this in your bus.py, 11 | # or import into your bus.py file from elsewhere 12 | 13 | class SupportCaseApi(Api): 14 | # An event, 15 | # available at bus.support.case.case_created 16 | case_created = Event(parameters=('id', 'sender', 'subject', 'body')) 17 | 18 | # Options for this API 19 | class Meta: 20 | # API name on the bus 21 | name = 'support.case' 22 | 23 | # Will be available as a remote procedure call at 24 | # bus.support.case.get() 25 | def get(self, id): 26 | return get_case_from_db(pk=id) 27 | ``` 28 | 29 | A service can define zero or more APIs, and each API can contain 30 | zero or more events and zero or more procedures. 31 | 32 | The `Meta` class specifies options regarding the API, with `name` being 33 | the only required option. The name specifies how the API will be 34 | accessed on the bus. 35 | 36 | You could call an RPC on the above API as follows: 37 | 38 | ```python3 39 | bus = lightbus.create() 40 | 41 | # Call the get_case() RPC. 42 | case = bus.support.case.get_case(id=123) 43 | ``` 44 | 45 | You can also fire an event on this API: 46 | 47 | ```python3 48 | bus = lightbus.create() 49 | 50 | # Fire the case_created event 51 | bus.support.case.case_created.fire( 52 | id=123, 53 | sender='Joe', 54 | subject='I need support please!', 55 | body='...', 56 | ) 57 | ``` 58 | 59 | ## Options 60 | 61 | ### `name (str)` 62 | 63 | Specifies the name of the API. This will determine how the API is addressed 64 | on the bus. See [naming](#naming-your-apis), below. 65 | 66 | `name` is a required option. 67 | 68 | ## Naming your APIs 69 | 70 | As you can from the `Meta.name` option in the example above, API names 71 | can contain periods which allow you 72 | to structure your bus in a suitable form for your situation. 73 | Some example API naming schemes may look like: 74 | 75 | ```yaml 76 | # Example API naming schemes for use within Meta.name 77 | Format: