├── .gitignore ├── 1.0.0-alpha-release-note.md ├── 1.0.0-preview-release-note.md ├── LICENSE ├── README.md ├── assets └── images │ ├── domain-design-0.1.0-alpha.png │ ├── domain-design-1.0.0-preview.png │ ├── landscape-0.1.0-alpha.png │ ├── landscape-0.2.0-alpha.png │ ├── landscape-1.0.0-preview.png │ └── use_cases │ ├── 15078677455707.jpg │ ├── 15078677592272.jpg │ ├── 15078677775543.jpg │ ├── 15078678095515.jpg │ ├── 15078678261228.jpg │ ├── 15078678568510.jpg │ ├── 15078678981991.jpg │ ├── 15078679330640.jpg │ ├── 15078679950623.jpg │ ├── 15078680221779.jpg │ ├── 15078680617233.jpg │ ├── 15078680954998.jpg │ └── 15078681271290.jpg ├── domain_architecture.md ├── extensionHeader.md ├── oms_access_point_schema.md ├── specification-schema.md └── usecase.md /.gitignore: -------------------------------------------------------------------------------- 1 | .classpath 2 | .project 3 | .settings/ 4 | target/ 5 | devenv 6 | *.log* 7 | *.iml 8 | .idea/ 9 | *.versionsBackup 10 | !NOTICE-BIN 11 | !LICENSE-BIN 12 | .DS_Store 13 | -------------------------------------------------------------------------------- /1.0.0-alpha-release-note.md: -------------------------------------------------------------------------------- 1 | # Status of 1.0.0-alpha Version of OpenMessaging 2 | 3 | 4 | ## Overview 5 | The OpenMessaging Community has recently approved and submitted the specification for 30-days public: 6 | 7 | OpenMessaging Specification Version: 1.0.0-preview. 8 | 2018/09/05 9 | 10 | This version modified mainly according to the public review opinions of 1.0.0-alpha, and includes some feature enhancements. 11 | 2019/02/28 12 | 13 | ## Specification Overview 14 | 15 | OpenMessaging is vendor-neutral and language-independent, provides industry guidelines for areas of finance, e-commerce, 16 | IoT and big-data, and aimed to develop messaging and streaming applications across heterogeneous systems and platforms. 17 | 18 | ### Spec. major changes 19 | [ISSUE-12](https://github.com/openmessaging/specification/issues/12) Simplify specification based on public review. 20 | [ISSUE-16](https://github.com/openmessaging/specification/issues/16) Add extension fields in specification. 21 | 22 | 23 | ### Runtime interface(java) major changes 24 | [ISSUE-33](https://github.com/openmessaging/openmessaging-java/issues/33) Support optional fields and interfaces. 25 | [ISSUE-35](https://github.com/openmessaging/openmessaging-java/issues/35) Improve Interceptor APIs to supprt java8 26 | lambda style. 27 | [ISSUE-37](https://github.com/openmessaging/openmessaging-java/issues/37) The method name of setDurability/getDurability 28 | in the Header class should be setReliability/getReliability, finaly used qos instead. 29 | [ISSUE-40](https://github.com/openmessaging/openmessaging-java/issues/40) Add boolean type to KeyValue. 30 | [ISSUE-41](https://github.com/openmessaging/openmessaging-java/issues/41) Fix KeyValue short type. 31 | [ISSUE-42](https://github.com/openmessaging/openmessaging-java/issues/42) Add cancel method in the Future class. 32 | [ISSUE-43](https://github.com/openmessaging/openmessaging-java/issues/43) Add ACCOUNT_KEY default field to OMSBuiltinKeys. 33 | [ISSUE-48](https://github.com/openmessaging/openmessaging-java/issues/48) A new MessageReceipt interface is introduced. 34 | -------------------------------------------------------------------------------- /1.0.0-preview-release-note.md: -------------------------------------------------------------------------------- 1 | # Status of 1.0.0-preview Version of OpenMessaging 2 | 3 | 4 | ## Overview 5 | The OpenMessaging Community has recently approved and submitted the specification for 30-days public: 6 | 7 | OpenMessaging Specification Version: 1.0.0-preview 8 | 2018/09/05 9 | 10 | ## Specification Overview 11 | 12 | OpenMessaging is vendor-neutral and language-independent, provides industry guidelines for areas of finance, e-commerce, IoT and big-data, and aimed to develop messaging and streaming applications across heterogeneous systems and platforms. 13 | 14 | ### Message model & Spec. major changes 15 | 1) Changed OpenMessaging model to queue based model. Queue based model is more flexible than the previous model, and queue is a logic destination of messaging. 16 | 2) Added a type system. 17 | 3) Made the definition of message schema, improved the message attribute definition. 18 | More details can be found: [spec.](https://github.com/openmessaging/specification/blob/master/specification-schema.md) 19 | 20 | 21 | ### Runtime interface(java&cpp) major changes 22 | 1) Removed some unclear routing method in resourceManager interface, and replaced it by more clearly defined routing and filtering operations. 23 | 2) Merged pullconsumer and pushconsumer interface. 24 | 3) Merged BatchProducer and Producer interface. 25 | 4) Defined error codes and openmessaging exceptions. 26 | 5) Temporarily removed the Streaming related interface. 27 | 6) Redefined the transaction interface. 28 | 7) Modified message interface, removed key-value header and added a Header interface. 29 | 8) Removed most of the key-value attributes. 30 | 9) Renamed some interfaces. 31 | More details can be found: [API](http://openmessaging.cloud/openmessaging-java/) 32 | 33 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "{}" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright {yyyy} {name of copyright owner} 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # The OpenMessaging Specification repository 2 | 3 | This repository is a place to document (and discuss) the OpenMessaging specification itself (independent of any particular language or platform). 4 | 5 | # Goals 6 | OpenMessaging is vendor-neutral and language-independent, provides industry guidelines for areas of finance, e-commerce, IoT and big-data, and aimed to develop messaging and streaming applications across heterogeneous systems and platforms. 7 | 8 | # Landscape 9 | ![landscape](assets/images/landscape-1.0.0-preview.png) 10 | 11 | # OpenMessaging 12 | Please see http://openmessaging.cloud/. 13 | 14 | # Schema 15 | ```json 16 | { 17 | "message": { 18 | "version":"1.0.0", 19 | "header": { 20 | "messageId": "7F00000100002873000000000004F49C", 21 | "destination": "orderQueue", 22 | "bornTimestamp": 1533780827824, 23 | "bornHost": "172.24.0.101:10035", 24 | "compression": "gzip", 25 | "qos": 2 26 | }, 27 | "extensionHeader": { 28 | "storeTimestamp": 1533780827825, 29 | "storeHost": "172.24.0.102:52511", 30 | "messageKey": "orderId-103368921567", 31 | "correlationId": "7F00000100002873000000000004F2B4", 32 | "delayTime": 30000, 33 | "transactionId": "1E0578887D3F18B4AAC22B64D2B40A62", 34 | "expireTime": 1533780830000, 35 | "traceId": "1E0578887D3F18B4AAC22B64D2B00A5E", 36 | "priority": 1 37 | }, 38 | "properties": { 39 | "service": "helloService" 40 | }, 41 | "data": {} 42 | } 43 | } 44 | ``` 45 | 46 | 47 | 48 | # Proposal 49 | #### [specification](specification-schema.md) 50 | 51 | # Release note 52 | 53 | #### [Status of 1.0.0-preview Version of OpenMessaging](1.0.0-preview-release-note.md) 54 | #### [Status of 1.0.0-alpha Version of OpenMessaging](1.0.0-alpha-release-note.md) 55 | 56 | # Communications 57 | Chat with us on the OpenMessaging Slack in the [#general channel:](https://openmessaging.herokuapp.com/) 58 | -------------------------------------------------------------------------------- /assets/images/domain-design-0.1.0-alpha.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openmessaging/specification/99569ec31ff268dc4031b34cfc133c3ceb7d01f9/assets/images/domain-design-0.1.0-alpha.png -------------------------------------------------------------------------------- /assets/images/domain-design-1.0.0-preview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openmessaging/specification/99569ec31ff268dc4031b34cfc133c3ceb7d01f9/assets/images/domain-design-1.0.0-preview.png -------------------------------------------------------------------------------- /assets/images/landscape-0.1.0-alpha.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openmessaging/specification/99569ec31ff268dc4031b34cfc133c3ceb7d01f9/assets/images/landscape-0.1.0-alpha.png -------------------------------------------------------------------------------- /assets/images/landscape-0.2.0-alpha.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openmessaging/specification/99569ec31ff268dc4031b34cfc133c3ceb7d01f9/assets/images/landscape-0.2.0-alpha.png -------------------------------------------------------------------------------- /assets/images/landscape-1.0.0-preview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openmessaging/specification/99569ec31ff268dc4031b34cfc133c3ceb7d01f9/assets/images/landscape-1.0.0-preview.png -------------------------------------------------------------------------------- /assets/images/use_cases/15078677455707.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openmessaging/specification/99569ec31ff268dc4031b34cfc133c3ceb7d01f9/assets/images/use_cases/15078677455707.jpg -------------------------------------------------------------------------------- /assets/images/use_cases/15078677592272.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openmessaging/specification/99569ec31ff268dc4031b34cfc133c3ceb7d01f9/assets/images/use_cases/15078677592272.jpg -------------------------------------------------------------------------------- /assets/images/use_cases/15078677775543.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openmessaging/specification/99569ec31ff268dc4031b34cfc133c3ceb7d01f9/assets/images/use_cases/15078677775543.jpg -------------------------------------------------------------------------------- /assets/images/use_cases/15078678095515.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openmessaging/specification/99569ec31ff268dc4031b34cfc133c3ceb7d01f9/assets/images/use_cases/15078678095515.jpg -------------------------------------------------------------------------------- /assets/images/use_cases/15078678261228.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openmessaging/specification/99569ec31ff268dc4031b34cfc133c3ceb7d01f9/assets/images/use_cases/15078678261228.jpg -------------------------------------------------------------------------------- /assets/images/use_cases/15078678568510.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openmessaging/specification/99569ec31ff268dc4031b34cfc133c3ceb7d01f9/assets/images/use_cases/15078678568510.jpg -------------------------------------------------------------------------------- /assets/images/use_cases/15078678981991.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openmessaging/specification/99569ec31ff268dc4031b34cfc133c3ceb7d01f9/assets/images/use_cases/15078678981991.jpg -------------------------------------------------------------------------------- /assets/images/use_cases/15078679330640.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openmessaging/specification/99569ec31ff268dc4031b34cfc133c3ceb7d01f9/assets/images/use_cases/15078679330640.jpg -------------------------------------------------------------------------------- /assets/images/use_cases/15078679950623.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openmessaging/specification/99569ec31ff268dc4031b34cfc133c3ceb7d01f9/assets/images/use_cases/15078679950623.jpg -------------------------------------------------------------------------------- /assets/images/use_cases/15078680221779.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openmessaging/specification/99569ec31ff268dc4031b34cfc133c3ceb7d01f9/assets/images/use_cases/15078680221779.jpg -------------------------------------------------------------------------------- /assets/images/use_cases/15078680617233.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openmessaging/specification/99569ec31ff268dc4031b34cfc133c3ceb7d01f9/assets/images/use_cases/15078680617233.jpg -------------------------------------------------------------------------------- /assets/images/use_cases/15078680954998.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openmessaging/specification/99569ec31ff268dc4031b34cfc133c3ceb7d01f9/assets/images/use_cases/15078680954998.jpg -------------------------------------------------------------------------------- /assets/images/use_cases/15078681271290.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openmessaging/specification/99569ec31ff268dc4031b34cfc133c3ceb7d01f9/assets/images/use_cases/15078681271290.jpg -------------------------------------------------------------------------------- /domain_architecture.md: -------------------------------------------------------------------------------- 1 | # OpenMessaging Domain Architecture 2 | 3 | ## Overview 4 | 5 | ![](/assets/images/domain-design-1.0.0-preview.png) 6 | 7 | Above is the domain architecture of Open Messaging, see JavaDoc for details. 8 | 9 | OpenMessaging has released first alpha version recently, Apache RocketMQ has provided a partial implementation for OpenMessaging-0.1.0-alpha. 10 | 11 | ## Namespace 12 | 13 | **Namespace** likes a cgroup namespace, to create an isolated space with security guarantee. Each namespace has its own set of producer, consumer, topic, queue and so on. OpenMessaging uses **MessagingAccessPoint** to access/read/write the **resources** of a specified Namespace. 14 | 15 | ## Producer 16 | 17 | OpenMessaging defines two kinds of Producer: **Producer** and **SequenceProducer**. 18 | 19 | * Producer, provides various send methods to send a message to a specified destination, Topic or Queue. Three ways are supported: synchronous, asynchronous and oneway. 20 | * SequenceProducer, focuses on speed, the implementation can adopt the batch way, send many messages and then commit at once. 21 | 22 | ## Consumer 23 | 24 | OpenMessaging defines two kinds of Consumer: **PullConsumer**, **PushConsumer** and **StreamingConsumer**. Each consumer only supports consume messages from the **Queue**. 25 | 26 | * **PullConsumer**, pulls messages from the specified queue, supports submit the consume result by acknowledgement at any time. One PullConsumer only can pull messages from one fixed queue. 27 | * **PushConsumer**, receives messages from multiple queues, these messages are pushed from the MOM server. PushConsumer can attach to multiple queues with separate MessageListener and submit consume result through **ReceivedMessageContext** at any time. 28 | 29 | ## Topic Queue and Routing 30 | 31 | These three concepts are closely connected, although **Topic** and **Queue** have different responsibilities, they are confusing. 32 | 33 | ### Topic 34 | 35 | Topic, the carrier of origin messages, is responsibility for holding messages. The distribution and sequential of messages in Topic is undefined. 36 | 37 | ### Routing 38 | 39 | The messages in Topic are original, waiting for processing, which always can't arouse the interests of consumers. In a word, the messages in Topic is producer-oriented, not consumer--oriented. 40 | 41 | So the Routing is in charge of processing the original messages in Topic, and routing to Queue. Each Routing has a operator pipeline, consists of a series of operators. The messages will flow through the operator pipeline from Topic and Queue. 42 | 43 | A **operator** is used to handle the flowing messages in Routing. There are many kinds of operator, expression operator, deduplicator operator, joiner operator, filter operator, rpc operator, and so on. 44 | 45 | What’s more? Routing can cross the network, message can be routed from a network partition to another partition. 46 | 47 | ## Queue 48 | 49 | The messages have been routed to **Queue**, now can be consumed by consumers. 50 | It is noteworthy that a Queue should be divided into **partitions**, a message will be routed to a specified partition by MessageHeader#SHARDING_KEY. 51 | 52 | Queue also accepts messages from Producer directly, sometimes, we want to the shortest path from Producer to Consumer, for performance. 53 | 54 | ## Topic vs. Queue 55 | 56 | * Both Topic and Queue are the carrier of messages. 57 | * Topic is producer-oriented, while Queue is consumer-oriented. 58 | * Messages in Topic are from Producer, while messages in Queue is from Topic or Producer. 59 | * Queue is divided into partitions, while the arch of Topic is undefined. 60 | * Queue is a little sub set of a Topic in most cases. 61 | * Create or destroy a Queue is easy and producer irrelevant. 62 | -------------------------------------------------------------------------------- /extensionHeader.md: -------------------------------------------------------------------------------- 1 | # OpenMessaging Extension Header 2 | The [OpenMessaging Specification](./specification-schema.md) defines the core fields of for messaging and 3 | streaming, which can meet the needs of most common message middleware 4 | message transmissions, but there are still many fields related to the message 5 | domain, and are also well known to many developers, but not produced by every 6 | message. but not so much commonly used as the fields in OpenMessaging 7 | Specification, but these fields are still very meaningful and can help message 8 | middleware achieve more features and improve its visibility. 9 | 10 | So we officially defined some of the following attributes in `extensionHeader`: 11 | 12 | ## storeTimestamp 13 | - Type: `Long` 14 | - Description: when the `durability` The timestamp that a message stored by server. when a 15 | message is stored by the server, this field will be set with the current 16 | timestamp of the server. 17 | It is represented as a long value which is defined as the difference, 18 | measured in milliseconds, between this time and midnight, January 1, 1970 19 | UTC. 20 | - Constraints: OPTIONAL 21 | 22 | ## storeHost 23 | - Type: `String` 24 | - Description: The host info of the server that stores this message. when a 25 | message is stored by the server, this field will be set with the host info of 26 | server. 27 | - Constraints: OPTIONAL 28 | 29 | ## correlationId 30 | - Type: `String` 31 | - Description: A client can use the correlationId header field to link one 32 | message with another. A typical use is to link a response message with its 33 | request message. 34 | - Constraints: OPTIONAL 35 | 36 | ## messageKey 37 | - Type: `String` 38 | - Description: This key that specifies that a message belongs to a specific 39 | message group, and this key can be used for the server to shard or 40 | dispatch messages. 41 | - Constraints: OPTIONAL 42 | 43 | ## delayTime 44 | - Type: `Long` 45 | - Description: The message will be delivered after `delayTime` milliseconds 46 | starting from `bornTimeStamp` . 47 | When this filed isn't set explicitly, this means this message should be 48 | delivered immediately. 49 | - Constraints: OPTIONAL 50 | 51 | ## expireTime 52 | - Type: `Long` 53 | - Description: This field represents the discard time of the message, if an 54 | undelivered message's `expireTime` is reached, the message would be 55 | destroyed. If an earlier timestamp is set than `expireTime` or isn't set 56 | , that means the message will not be expired. 57 | It is represented as a long value which is defined as the difference, 58 | measured in milliseconds, between this time and midnight, January 1, 1970 59 | UTC. 60 | - Constraints: OPTIONAL 61 | 62 | 63 | ## traceId 64 | - Type: `String` 65 | - Description: This identifier represents a global and unique 66 | identification, to associate key events in the whole lifecycle of a message, 67 | like sent by who, stored at where, and received by who. And, the messaging 68 | system only plays exchange role in a distributed system in most cases, 69 | so the TraceID can be used to trace the whole call link with other parts in 70 | the whole system. 71 | - Constraints: OPTIONAL 72 | 73 | ## transactionId 74 | - Type: `String` 75 | - Description: This field is used in transactional message, and it can be 76 | used to trace a transaction. So the same `transactionId` will be appeared 77 | not only in prepare message, but also in commit message, and consumer 78 | received message also contains this field. 79 | - Constraints: OPTIONAL 80 | 81 | ## deliveryCount 82 | - Type: `Integer` 83 | - Description: This field indicates the times of the message was 84 | delivered, when a consumer consumes a message failed, it will be resend to 85 | the server for consume it later. and this field records the consumed times 86 | during the consume process. 87 | - Constraints: OPTIONAL 88 | 89 | 90 | ## priority 91 | - Type: `Integer` 92 | - Description: OpenMessaging defines a ten level priority value with 1 as 93 | the lowest priority and 10 as the highest, and the default priority is 5. 94 | The priority beyond this region will be ignored. 95 | OpenMessaging does not require or provide any guarantee that the message 96 | should be delivered in priority order strictly, but the vendor should 97 | provide a best effort to deliver expedited messages ahead of normal 98 | messages. 99 | - Constraints: OPTIONAL 100 | 101 | ## partition 102 | - Type: `Integer` 103 | - Description: This field in extension header contains the partition of target destination which the message 104 | is being sent. When a Message is set with this value, this message will be delivered to specified partition, but the 105 | premise is that the implementation of the server side is dependent on the partition or a queue-like storage mechanism. 106 | - Constraints: OPTIONAL -------------------------------------------------------------------------------- /oms_access_point_schema.md: -------------------------------------------------------------------------------- 1 | ## OMS Access Point Schema 2 | 3 | This document describes the access point format for defining connections between applications and OpenMessaging providers. 4 | 5 | ### Standard Connection String Schema 6 | 7 | This section describes the standard schema of the OpenMessaging access point URI used to connect to a OMS provider. The format is the same for all OMS implementation. 8 | 9 | The following is the standard URI connection scheme: 10 | 11 | ``` 12 | oms:://[account_id@]host1[:port1][,host2[:port2],...[,hostN[:portN]]]/ 13 | ``` 14 | 15 | The components of this string are: 16 | 17 | | Component | Description | 18 | | --- | --- | 19 | | oms | A required prefix to identify that this is a string in the standard OMS access point format. | 20 | | driver_type | Required. It identifies a specific OMS provider, like RocketMQ. | 21 | | account_id | Optional. The ID of the specific account system that owns the resource. | 22 | | host1 | Required. It identifies a server address to connect to. It identifies either a hostname or IP address. | 23 | | port1 | Optional. The default value is 80 if not specified. | 24 | | hostN | Optional. You can specify as many hosts as necessary. You would specify multiple hosts. | 25 | | portN | Optional. The default value is 80 if not specified. | 26 | | region | Required. The target region. | 27 | 28 | ### Access Point Example 29 | 30 | 1. Single host 31 | 32 | The following example connects user alice to a rocketmq implementation in region us-east: 33 | 34 | ``` 35 | oms:rocketmq://alice@rocketmq.apache.org/us-east 36 | ``` 37 | 38 | 2. Multiple hosts 39 | 40 | The multiple hosts are often used to load balancing. 41 | 42 | ``` 43 | oms:rocketmq://alice@rocketmq.apache.org, openmessaging.io/us-east 44 | ``` 45 | -------------------------------------------------------------------------------- /specification-schema.md: -------------------------------------------------------------------------------- 1 | # OpenMessaging Specification 2 | 3 | ## Table of Contents 4 | - [License](#license) 5 | - [Overview](#0-overview) 6 | - [Type System](#1-type-system) 7 | - [Message Model](#2-message-model) 8 | - [UseCases](#openmessaging-common-usecases) 9 | - [Appendix](#appendix) 10 | ## License 11 | ## 0 Overview 12 | ### 0.1 What is OpenMessaging? 13 | OpenMessaging is a cloud-native, vendor-neutral open standard for distributed messaging. 14 | 15 | ### 0.2 Why OpenMessaging? 16 | #### 0.2.1 Goals 17 | Messaging products have been widely used in modern architecture and data processing, for decoupling, queuing, buffering, 18 | ordering, replicating, etc. But when data transfers across different messaging and streaming platforms, the compatibility 19 | problem arises, which always means much additional work. Although JMS was a good solution during the past decade, it 20 | is limited in java environment, lacks specified guidelines for load balance/fault-tolerance, administration, security, 21 | and streaming feature, which make it not good at satisfying modern cloud-native messaging and streaming applications. 22 | While OpenMessaging aims at : 23 | 24 | - Language-agnostic and platform independence. message standard support multiple platforms, architectures or systems. 25 | - Global, cloud-native, vendor-neutral industry standard for distributed messaging. 26 | - Facilitating a standard benchmark for testing applications. 27 | - Targeting cloud data streaming and messaging requirements with scalability, flexibility, isolation, and security 28 | built in Fostering a growing community of contributing developers. 29 | 30 | #### 0.2.1 Non-Goals 31 | The following will not be part of the specification: 32 | - Language-specific runtime APIs. 33 | - Benchmark Interface for evaluating performance. 34 | - Connector Interface for data stream exchange with other systems. 35 | 36 | ### 0.3 OpenMessaging Terminologies 37 | #### 0.3.1 Topic 38 | An administered object that encapsulates the identity of a message destination for messaging. 39 | 40 | #### 0.3.2 Producer 41 | An object that sending a message to all consumers of a topic. 42 | 43 | #### 0.3.3 Consumer 44 | An object that is used for receiving messages sent to a topic. 45 | 46 | #### 0.3.4 Queue 47 | An administered object that encapsulates the identity of a message destination. 48 | 49 | #### 0.3.5 Delivery Semantics 50 | When it comes to describing the semantics of a delivery mechanism, there are three semantic guarantees between producer and consumer: 51 | **At least once**: a message will be consumed at least once. 52 | **At most once**: a message will be consumed at most once, in this semantics, messages may be lost. 53 | **Exactly once**: a message will be consumed once and only once. 54 | 55 | ## 1 Type System 56 | The following abstract data types are available for use in attributes. 57 | 58 | - `String` - Sequence of printable Unicode characters. 59 | - `Binary` - Sequence of bytes. 60 | - `KeyValue` - `String`-indexed dictionary of `String`-typed or `Binary`-typed or `Numeric`-typed values 61 | - `Numeric`: 62 | - `Short` - Integer in the range -(2^15) to 2^15 - 1 inclusive. 63 | - `Integer` - Integer in the range -(2^31) to 2^31 - 1 inclusive. 64 | - `Long` - Integer in the range -(2^63) to 2^63 - 1 inclusive. 65 | - `Float` - A 32-bit floating point number (binary32 [IEEE754](http://ieeexplore.ieee.org/servlet/opac?punumber=4610933)). 66 | - `Double` - A 64-bit floating point number (binary64 [IEEE754](http://ieeexplore.ieee.org/servlet/opac?punumber=4610933)). 67 | - `Object` - Either a `String`, or a `Binary`, or a `KeyValue`, or a `Numeric` 68 | - `URI` - String expression conforming to `URI-reference` 69 | as defined in 70 | [RFC 3986 §4.1](https://tools.ietf.org/html/rfc3986#section-4.1). 71 | 72 | The `Object` type is a variant type that can take the shape of either a 73 | `String` or a `Binary` or a `KeyValue` or a `Numeric`. The type system is intentionally 74 | abstract, and therefore it is left to implementations how to represent the 75 | variant type(Reference to the `Object` description of [CloudEvents](https://github.com/cloudevents/spec/blob/v0.1/spec.md)). 76 | 77 | ## 2 Message Model 78 | ### 2.1 Message Type 79 | #### 2.1.1 Bytes Message 80 | A message that whose body contains a stream of uninterpreted bytes. This message type is for literally encoding a body to match an existing message format. 81 | It will be use one of self-defining message types to encode the message body, and vendors are responsible for decode these bytes in a custom rules. 82 | 83 | ### 2.2 Message Format 84 | In the OpenMessaging, a message consists of 4 parts: the version, the credential, the system header, the user header and the message body. 85 | #### 2.2.1 version 86 | - Type: `String` 87 | - Description: The version of OpenMessaging standard. 88 | - Constraints: REQUIRED 89 | 90 | #### 2.2.2 header 91 | - Type: `KeyValue` 92 | - Description: All messages support the same set of header fields, and these header fields are used by system, which are usually used for such as identify and route messages. 93 | Specific fields can be found in the next [chapter](#2.3-message-header). 94 | - Constraints: REQUIRED 95 | 96 | #### 2.2.3 extensionHeader 97 | - Type: `KeyValue` 98 | - Description: This field contains extension metadata for message middleware, and these extension fields are not mandatory, but for the time being, most of the message middleware has been 99 | implemented related content more or less, and these fields have been well known and understood by many messaging and streaming developers, See the ExtFields document for a list of possible properties. 100 | See the [extension header document](./extensionHeader.md) for a list of possible fields. 101 | - Constraints: OPTIONAL 102 | 103 | #### 2.2.4 properties 104 | - Type: `KeyValue` 105 | - Description: In addition to the system header, OpenMessaging provides a built-in user properties for adding optional fields to a message, and these fields are represented as key-value forms. 106 | - Constraints: REQUIRED 107 | 108 | #### 2.2.5 data 109 | - Type: `Binary` 110 | - Description: This field is the part of transmitted data that is the actual intended message contains application data. 111 | The message body is completely transparent to the server, the server cannot view or modify the message body. 112 | - Constraints: OPTIONAL 113 | 114 | ### 2.3 Message Header 115 | #### 2.3.1 messageId 116 | - Type: `String` 117 | - Description: An unique identifier for a message. When a message is sent, messageId is ignored. When the send method returns it contains a provider-assigned value. 118 | - Constraints: REQUIRED and MUST be a non-empty `String`. 119 | 120 | #### 2.3.2 bornTimestamp 121 | - Type: `Long` 122 | - Description: The timestamp of the message sent by the client. 123 | It is not the time the message was actually transmitted because the actual send may occur later due to transactions or other client side queueing of messages. 124 | When a message is sent, `bornTimestamp` is ignored. When the send method returns, the field contains a time value somewhere in the interval between the call and the return. 125 | It is represented as a long value which is defined as the difference, measured in milliseconds, between this time and midnight, January 1, 1970 UTC. 126 | - Constraints: REQUIRED 127 | 128 | #### 2.3.3 bornHost 129 | - Type: `String` 130 | - Description: When a message is sent, this field will be set with the local host info of client. 131 | - Constraints: REQUIRED and MUST be a non-empty `String`. 132 | 133 | 134 | #### 2.3.4 qos 135 | - Type: `Integer` 136 | - Description: OpenMessaging defines three modes of message delivery as mentioned before: 137 | **At least once**: if this value set with 0, a message will be consumed at least once, and this value should be set as default value. 138 | **At most once**: if this value set with 1, a message will be consumed at most once, in this semantics, messages may be lost. 139 | **Exactly once**: if this value set with 2, a message will be consumed once and only once. 140 | 141 | 142 | #### 2.3.5 compression 143 | - Type: `String` 144 | - Description: This field represents the message body compress algorithm. 145 | vendors are free to choose the compression algorithm, but must ensure that the decompressed message is delivered to the user. 146 | - Constraints: OPTIONAL 147 | 148 | #### 2.3.6 destination 149 | -Type: `String` 150 | -Description: This filed contains the logic destination to which the message is being sent, such as a queue or a topic. 151 | When a message is sent this value is set to the right queue, then the message will be sent to the specified destination. 152 | When a message is received, its destination is equivalent to the queue where the message resides in. 153 | - Constraints: REQUIRED 154 | 155 | 156 | 157 | ### Notational Conventions 158 | The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", 159 | "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to 160 | be interpreted as described in [RFC 2119](https://tools.ietf.org/html/rfc2119). 161 | 162 | ## OpenMessaging Common UseCases 163 | 164 | This document lists the most of common use cases supported by OpenMessaging. 165 | 166 | 1. P2P 167 | 2. Publish/Subscribe 168 | 3. Broadcast 169 | 4. Highway 170 | 5. Streaming 171 | 6. Filter 172 | 7. Routing 173 | 8. RPC 174 | 175 | ### P2P 176 | 177 | ![](/assets/images/use_cases/15078677455707.jpg) 178 | 179 | P2P, point to point, the simplest one, in this case, Queue is the only involved resource of OpenMessaging which only has one partition. Simply, Producer send message to Queue, and consumed by Consumer later. 180 | 181 | ### Publish/Subscribe 182 | 183 | In this case, Producer send message to Queue with multiple partitions in Round-robin or Hash way. And these partitions will be assigned to consumers who has already subscribed the specified queue regularly. 184 | 185 | ![](/assets/images/use_cases/15078678095515.jpg) 186 | 187 | Topic and Routing model also can be imported to this case as shown below, if necessary. 188 | 189 | ![](/assets/images/use_cases/15078678261228.jpg) 190 | 191 | ### Broadcast 192 | 193 | ![](/assets/images/use_cases/15078678568510.jpg) 194 | 195 | In broadcast case, any message sent to the Queue will be consumed by all consumers. 196 | 197 | ### Highway 198 | 199 | ![](/assets/images/use_cases/15078678981991.jpg) 200 | 201 | In highway case, the only focus of SequenceProducer is speed, Producer always want to send abundant and less important messages to Queue. One of the Implementation ways is Batch. 202 | 203 | ### Streaming 204 | 205 | ![](/assets/images/use_cases/15078679330640.jpg) 206 | 207 | 208 | StreamingConsumer is for this use case, a stream-oriented consumer, to integrate messaging system with Streaming/BigData related platforms easily. StreamingConsumer supports consume messages from partitions of a specified queue like iterator. 209 | 210 | ### Filter 211 | 212 | In most cases, original messages can’t arouse the interests of consumers, and consumers always want to consume processed messages, the most useful processing method is Filter. 213 | 214 | As shown below, the Routing model of OpenMessaging can be applied to Filter easily. In this case, the message will be routed to Queue through two filter operators, which will keep the message with Student tag and has a property age between 18~23. 215 | 216 | ![](/assets/images/use_cases/15078679950623.jpg) 217 | 218 | ### Replication 219 | 220 | ![](/assets/images/use_cases/15078680221779.jpg) 221 | 222 | 223 | Sometimes, the producers and consumers are distributed among multiple data centers, OpenMessaging provides a simple way to route messages from one region to another region. 224 | 225 | ### RPC 226 | 227 | ![](/assets/images/use_cases/15078681271290.jpg) 228 | 229 | In OpenMessaging, RPC is equal to synchronous message, it isn’t traditional CS(Client2Server) model, but CSC(Client2Server2Client) model. 230 | 231 | 232 | ## Appendix 233 | ### Example of OpenMessaging API 234 | ```json 235 | { 236 | "message": { 237 | "version":"1.0.0", 238 | "header": { 239 | "messageId": "7F00000100002873000000000004F49C", 240 | "destination": "orderQueue", 241 | "bornTimestamp": 1533780827824, 242 | "bornHost": "172.24.0.101:10035", 243 | "compression": "gzip", 244 | "qos": 1 245 | }, 246 | "extensionHeader": { 247 | "partition": 1, 248 | "storeTimestamp": 1533780827825, 249 | "storeHost": "172.24.0.102:52511", 250 | "messageKey": "orderId-103368921567", 251 | "correlationId": "7F00000100002873000000000004F2B4", 252 | "delayTime": 30000, 253 | "transactionId": "1E0578887D3F18B4AAC22B64D2B40A62", 254 | "expireTime": 1533780830000, 255 | "traceId": "1E0578887D3F18B4AAC22B64D2B00A5E", 256 | "priority": 1 257 | }, 258 | "properties": { 259 | "service": "helloService" 260 | }, 261 | "data": {} 262 | } 263 | } 264 | ``` 265 | 266 | ### Change History 267 | 0.3.0 version created, be compatible with existent runtime API. 268 | 1.0.0-preview version created, change domain model to queue based model, add type system and schema. 269 | 1.0.0-alpha version created, simplify specification and add extension fields. -------------------------------------------------------------------------------- /usecase.md: -------------------------------------------------------------------------------- 1 | # OpenMessaging Common UseCases 2 | 3 | This document lists the most of common use cases supported by OpenMessaging. 4 | 5 | 1. P2P 6 | 2. Publish/Subscribe 7 | 3. Broadcast 8 | 4. Highway/assets/images/use_cases 9 | 5. Streaming 10 | 6. Filter 11 | 7. Routing 12 | 8. Online Test 13 | 9. Upgrade 14 | 10. RPC 15 | 16 | ## P2P 17 | 18 | ![](/assets/images/use_cases/15078677455707.jpg) 19 | 20 | P2P, point to point, the simplest one, in this case, Queue is the only involved resource of OpenMessaging which only has one partition. Simply, Producer send message to Queue, and consumed by Consumer later. 21 | 22 | ## Publish/Subscribe 23 | 24 | In this case, Producer send message to Queue with multiple partitions in Round-robin or Hash way. And these partitions will be assigned to consumers who has already subscribed the specified queue regularly. 25 | 26 | ![](/assets/images/use_cases/15078678095515.jpg) 27 | 28 | Topic and Routing model also can be imported to this case as shown below, if necessary. 29 | 30 | ![](/assets/images/use_cases/15078678261228.jpg) 31 | 32 | ## Broadcast 33 | 34 | ![](/assets/images/use_cases/15078678568510.jpg) 35 | 36 | In broadcast case, any message sent to the Queue will be consumed by all consumers. 37 | 38 | ## Highway 39 | 40 | ![](/assets/images/use_cases/15078678981991.jpg) 41 | 42 | In highway case, the only focus of SequenceProducer is speed, Producer always want to send abundant and less important messages to Queue. One of the Implementation ways is Batch. 43 | 44 | ## Streaming 45 | 46 | ![](/assets/images/use_cases/15078679330640.jpg) 47 | 48 | 49 | StreamingConsumer is for this use case, a stream-oriented consumer, to integrate messaging system with Streaming/BigData related platforms easily. StreamingConsumer supports consume messages from partitions of a specified queue like iterator. 50 | 51 | ## Filter 52 | 53 | In most cases, original messages can’t arouse the interests of consumers, and consumers always want to consume processed messages, the most useful processing method is Filter. 54 | 55 | As shown below, the Routing model of OpenMessaging can be applied to Filter easily. In this case, the message will be routed to Queue through two filter operators, which will keep the message with Student tag and has a property age between 18~23. 56 | 57 | ![](/assets/images/use_cases/15078679950623.jpg) 58 | 59 | ## Replication 60 | 61 | ![](/assets/images/use_cases/15078680221779.jpg) 62 | 63 | 64 | Sometimes, the producers and consumers are distributed among multiple data centers, OpenMessaging provides a simple way to route messages from one region to another region. 65 | 66 | ## Online Test 67 | 68 | ![](/assets/images/use_cases/15078680617233.jpg) 69 | 70 | Tests are important, many tests like A/B or pressure test need online environment. Create a test Queue with partial traffic can reach this goal. 71 | 72 | ## Upgrade 73 | 74 | ![](/assets/images/use_cases/15078680954998.jpg) 75 | 76 | Image that we want to release our Consumer version 2.0, which can handle messages with tag Staff or Student, while the version 1.0 consumer only can handle messages with tag Student. OpenMessaging can cover this case easily. 77 | 78 | ## RPC 79 | 80 | ![](/assets/images/use_cases/15078681271290.jpg) 81 | 82 | In OpenMessaging, RPC is equal to synchronous message, it isn’t traditional CS(Client2Server) model, but CSC(Client2Server2Client) model. 83 | --------------------------------------------------------------------------------