├── LICENSE ├── README.md ├── account-component-diagram-cross.png ├── account-component-diagram-through.png ├── cache1 ├── AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G.jwt └── ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4.jwt ├── cache2 ├── AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G.jwt └── ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4.jwt ├── cache3 ├── AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G.jwt └── ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4.jwt ├── cache4 ├── AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G.jwt └── ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4.jwt ├── cache5 ├── AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G.jwt └── ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4.jwt ├── cache6 ├── AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G.jwt └── ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4.jwt ├── cache7 ├── AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G.jwt └── ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4.jwt ├── cache8 ├── AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G.jwt └── ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4.jwt ├── cache9 ├── AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G.jwt └── ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4.jwt ├── cluster-hub-1.cfg ├── cluster-hub-2.cfg ├── cluster-hub-3.cfg ├── cluster-spoke-1-1.cfg ├── cluster-spoke-1-2.cfg ├── cluster-spoke-1-3.cfg ├── cluster-spoke-2-1.cfg ├── cluster-spoke-2-2.cfg ├── cluster-spoke-2-3.cfg ├── keys └── .gitignore ├── leafnode-remotes.cfg ├── main.go ├── nats-account-resolver.cfg ├── outline.txt ├── puml ├── topology1.puml ├── topology2-js-merged.puml ├── topology3-js-domains.puml ├── topology4-js-streams-mirror.puml ├── topology5-js-streams-source.puml ├── topology6-js-streams-backup.puml ├── topology7-mirror-cross-account.puml ├── topology8-consume-cross-account.puml ├── topology9-source-cross-domain-account.png └── topology9-source-cross-domain-account.puml ├── store └── OP │ ├── .nsc │ ├── OP.jwt │ └── accounts │ ├── SYS │ ├── SYS.jwt │ └── users │ │ └── sys.jwt │ └── TEST │ ├── TEST.jwt │ └── users │ └── leaf.jwt ├── topology1-server.png ├── topology2-js-merged.png ├── topology3-js-domains.png ├── topology4-js-streams-mirror.png ├── topology5-js-streams-source.png ├── topology6-js-streams-backup.png ├── topology7-mirror-cross-account.png ├── topology8-consume-cross-account.png └── topology9-source-cross-domain-account.png /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # JetStream-LeafNodes-Demo 2 | 3 | This repository contains the configuration for the [Persistence at the Edge == JetStream in Leaf Nodes demo](https://www.youtube.com/watch?v=0MkS_S7lyHk) as well as the [script](#video-script-and-commands) it is based on. 4 | The state is identical to the one the demo started with. 5 | 6 | To set up your `nsc` environment execute the followings commands in base directory: 7 | 8 | ```txt 9 | export NKEYS_PATH="`pwd`/keys" 10 | nsc env -s "`pwd`/store" 11 | nsc env --operator OP 12 | ``` 13 | 14 | The context used in the demo need to be created separately using 15 | 16 | ## Content of this repo 17 | 18 | This directory contains `nsc` directories `store` and `keys` containing jwt and creds. 19 | Server config files are of the format `cluster--.cfg` 20 | `nats-account-resolver.cfg` contains the account resolver setup shared by all server. 21 | The directories `CACHE*` are nats account resolver directories for each server. 22 | They contain already pushed account JWT so you are ready to go. 23 | `main.go` contains the source code shown during the presentation. 24 | `outline.txt` contains the outline of the presentation. 25 | The folder `puml` contains the plant uml files used to generate the png named `topology*` 26 | To generate install [plantuml](https://plantuml.com/) and execute `plantuml -tpng `. 27 | 28 | ## Server Startup 29 | 30 | To have a single nats account resolver config file each server needs the environment variable `CACHE` set. 31 | This variable is referenced in line four of the config file `nats-account-resolver.cfg`. 32 | 33 | To start the server execute the following commands: 34 | 35 | ```txt 36 | export CACHE='"./cache1"'; nats-server -c cluster-hub-1.cfg 37 | export CACHE='"./cache2"'; nats-server -c cluster-hub-2.cfg 38 | export CACHE='"./cache3"'; nats-server -c cluster-hub-3.cfg 39 | 40 | export CACHE='"./cache4"'; nats-server -c cluster-spoke-1-1.cfg 41 | export CACHE='"./cache5"'; nats-server -c cluster-spoke-1-2.cfg 42 | export CACHE='"./cache6"'; nats-server -c cluster-spoke-1-3.cfg 43 | 44 | export CACHE='"./cache7"'; nats-server -c cluster-spoke-2-1.cfg 45 | export CACHE='"./cache8"'; nats-server -c cluster-spoke-2-2.cfg 46 | export CACHE='"./cache9"'; nats-server -c cluster-spoke-2-3.cfg 47 | ``` 48 | 49 | Or all at once: 50 | 51 | ```txt 52 | i=0; for c in cluster*.cfg; do ((i=i+1)); export CACHE=cache$i; nats-server -c $c & ; done 53 | ``` 54 | 55 | ## Nats cli contexts 56 | 57 | To create the contexts used execute the commands below. The context will function in the current directory only. 58 | 59 | ``` 60 | nats context save sys --creds ./keys/creds/OP/SYS/sys.creds --server "nats://127.0.0.1:4222,nats://127.0.0.1:4232,nats://127.0.0.1:4242,nats://127.0.0.1:4252,nats://127.0.0.1:4262,nats://127.0.0.1:4272,nats://127.0.0.1:4282,nats://127.0.0.1:4292,nats://127.0.0.1:4202" 61 | nats context save hub --creds ./keys/creds/OP/TEST/leaf.creds --server "nats://127.0.0.1:4222,nats://127.0.0.1:4232,nats://127.0.0.1:4282" 62 | nats context save spoke-1 --creds ./keys/creds/OP/TEST/leaf.creds --server "nats://127.0.0.1:4242,nats://127.0.0.1:4252,nats://127.0.0.1:4292" 63 | nats context save spoke-2 --creds ./keys/creds/OP/TEST/leaf.creds --server "nats://127.0.0.1:4262,nats://127.0.0.1:4272,nats://127.0.0.1:4202" 64 | ``` 65 | 66 | ## Video Script and Commands 67 | 68 | Nats is the networking abstraction to finally free you and your apps from networking and the silos that plague us all. 69 | It allows you to refocus on your data and it’s flows on a global scale. 70 | We have recently added JetStream, our persistence layer. 71 | 72 | This video is about JetStream at the edge. 73 | 74 | Our server are light weight and can therefore run in resource constrained environments. 75 | This allows you to have persistence on a remote edge and have it function independently, without connectivity to the cloud. 76 | If you want to, you can configure JetStream such that your locally persisted data is automatically uploaded to the cloud as soon as you regain connectivity. 77 | 78 | This applies specifically to cases where the edge site itself moves in and out of network connectivity, 79 | or when you have a great number of sites and are guaranteed a network outage at any given time. 80 | 81 | Depending on your needs you would install one or more JetStream enabled server in your edge site and connect them to server in the cloud using leaf node connections. 82 | To demonstrate the principle, I am working with two cluster that are connected to a hub. But you can have as many as needed. 83 | 84 | This is going to be a whirlwind tour through features and use cases. 85 | Do not focus on every detail. Please focus on what our software can do and which use cases are relevant to you. 86 | I will provide [further links](#relevant-links) at the end. 87 | 88 | ### Outline 89 | 90 | In this video I want to: 91 | 92 | 1. [Introduce the leafnode setup used here](#setup-topology) 93 | 2. [Talk about the implications of connecting the system account as leaf node remote](#connected-system-account-implications) 94 | 3. [Introduce the concept of JetStream domains](#jetstream-domains) 95 | 4. [Use stream mirrors to connect a command and control stream across domains](#stream-mirrors-across-domains) 96 | 5. [Use stream source to aggregate streams across domains](#stream-source-across-domains) 97 | 6. [Demonstrate that domain connectivity is not tied to the underlying topology](#sourcemirror-not-dependent-on-topology) 98 | 7. [Connect streams cross accounts](#connect-streams-cross-accounts) 99 | 8. [Connect streams through accounts](#connect-streams-through-accounts) 100 | 9. [Relevant Documentation](#relevant-links) 101 | 102 | #### Setup Topology 103 | 104 | This is the topology of my setup. 105 | 106 | ![`imgcat topology1-server.png`](topology1-server.png) 107 | 108 | There is a central cluster to which two clusters `spoke-1` and `spoke-2` connect via leafnode connections 109 | I am using two of these to demonstrate the principle, but you can have as many as needed. 110 | To show it's possible, each cluster consists of 3 server. Use fewer if you do not need the redundancy. 111 | let's have a look at server one in the cluster hub. 112 | 113 | ```txt 114 | > cat cluster-hub-1.cfg 115 | listen: localhost:4222 116 | server_name: srv-4222 117 | jetstream { 118 | store_dir: "./s1-1" 119 | domain: hub 120 | } 121 | cluster { 122 | listen localhost:4223 123 | name cluster-hub 124 | routes = [ 125 | nats-route://localhost:4223 126 | nats-route://localhost:4233 127 | nats-route://localhost:4283 128 | ] 129 | } 130 | leafnodes { 131 | listen localhost:4224 132 | no_advertise: true 133 | } 134 | mqtt { 135 | port: 4225 136 | } 137 | http: localhost:8080 138 | include ./nats-account-resolver.cfg 139 | ``` 140 | 141 | This is your regular cluster and leafnode setup. 142 | JetStream has a new property called domain, I'll be talking about this in a moment. 143 | 144 | ```txt 145 | > cat nats-account-resolver.cfg 146 | operator: "./store/OP/OP.jwt" 147 | resolver: { 148 | type: full 149 | dir: $CACHE 150 | interval: "2m" 151 | allow_delete: true 152 | } 153 | resolver_preload: { 154 | ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4:eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiJMTFpUU0paUTNGTkZWNlhDVDNBRkdCSlZWU0FaSk9IN0JSNlFETUdNVUdETktVWUlQR0hRIiwiaWF0IjoxNjI0NDc5NDgyLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJTWVMiLCJzdWIiOiJBREVDQ05CVUVCV1o3MjdPTUJGU043T01LMkZQWVJNNTJUSlMyNVRGUVdZUzc2TlBPSkJOM0tVNCIsIm5hdHMiOnsibGltaXRzIjp7InN1YnMiOi0xLCJkYXRhIjotMSwicGF5bG9hZCI6LTEsImltcG9ydHMiOi0xLCJleHBvcnRzIjotMSwid2lsZGNhcmRzIjp0cnVlLCJjb25uIjotMSwibGVhZiI6LTF9LCJzaWduaW5nX2tleXMiOlsiQUFDWUlDT0FRTVE3MkVIVDM1UjdMVjZWRldNSVZXRktXRkU1UDJKSjJUVDY3NEVPN0RKVFVITU0iXSwiZGVmYXVsdF9wZXJtaXNzaW9ucyI6eyJwdWIiOnt9LCJzdWIiOnt9fSwidHlwZSI6ImFjY291bnQiLCJ2ZXJzaW9uIjoyfX0.tHteTcshVIInToM6LQ7G2AmmWfeKYCCjJdCC4ZfJ1WUtmY1Bk0sEwwbjb6uSycEb4ljohMQnQVgkbAYZsiqZDw 155 | } 156 | ``` 157 | 158 | Enabling multi tenancy, accounts are the secure nats isolation context. 159 | Unless explicitly defined, connections belonging to one account can not communicate with connections belonging to another. 160 | Because of this we generally recommend one account per application. However, their scope is up to you. 161 | 162 | In addition, this is an operator setup and uses the nats account resolver to retrieve accounts. 163 | Instead of defining an account in every server we provide a way to obtain it. 164 | On connect, the client provides a web token carrying user permissions as well as proof of possession of a corresponding private key. 165 | After the account web token is downloaded, the chain of trust, operator signs account token, account signs user token is verified and all relevant limits and settings are applied. 166 | 167 | What I will be demonstrating in this video also applies when you configure your accounts in server config files. 168 | 169 | ```txt 170 | > cat cluster-spoke-1-1.cfg 171 | listen: localhost:4242 172 | server_name: srv-4242 173 | jetstream { 174 | store_dir: "./s2-1" 175 | domain: spoke-1 176 | } 177 | cluster { 178 | listen localhost:4243 179 | name cluster-spoke-1 180 | routes = [ 181 | nats-route://localhost:4243 182 | nats-route://localhost:4253 183 | nats-route://localhost:4293 184 | ] 185 | } 186 | mqtt { 187 | port: 4245 188 | } 189 | http: localhost:8084 190 | include ./nats-account-resolver.cfg 191 | include ./leafnode-remotes.cfg 192 | ``` 193 | 194 | Leaf nodes are clustered as well. 195 | 196 | #### Connected System Account Implications 197 | 198 | Although leaf nodes bridge authentication domains, if you have the same operator setup on either end and connect system accounts, it will appear to you as one large authentication domain. 199 | You could use such a setup for example when you can't use super cluster due to fire wall rules and have to use leaf node connections instead. 200 | 201 | Another benefit of connecting system accounts is that you can obtain monitoring information from every server in this network. 202 | 203 | ``` 204 | > nats --context=sys server list 9 205 | ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ 206 | │ Server Overview │ 207 | ├──────────┬─────────────────┬───────────┬────────────┬─────┬───────┬──────┬────────┬─────┬─────────┬─────┬──────┬────────┬───────────┤ 208 | │ Name │ Cluster │ IP │ Version │ JS │ Conns │ Subs │ Routes │ GWs │ Mem │ CPU │ Slow │ Uptime │ RTT │ 209 | ├──────────┼─────────────────┼───────────┼────────────┼─────┼───────┼──────┼────────┼─────┼─────────┼─────┼──────┼────────┼───────────┤ 210 | │ srv-4242 │ cluster-spoke-1 │ localhost │ 2.3.3-beta │ yes │ 0 │ 283 │ 2 │ 0 │ 16 MiB │ 0.4 │ 0 │ 39.65s │ 975.35µs │ 211 | │ srv-4262 │ cluster-spoke-2 │ localhost │ 2.3.3-beta │ yes │ 0 │ 283 │ 2 │ 0 │ 17 MiB │ 0.3 │ 0 │ 39.65s │ 953.193µs │ 212 | │ srv-4202 │ cluster-spoke-2 │ localhost │ 2.3.3-beta │ yes │ 0 │ 283 │ 2 │ 0 │ 16 MiB │ 0.3 │ 0 │ 39.65s │ 933.849µs │ 213 | │ srv-4232 │ cluster-hub │ localhost │ 2.3.3-beta │ yes │ 0 │ 338 │ 2 │ 0 │ 18 MiB │ 0.3 │ 0 │ 39.65s │ 913.57µs │ 214 | │ srv-4272 │ cluster-spoke-2 │ localhost │ 2.3.3-beta │ yes │ 0 │ 297 │ 2 │ 0 │ 16 MiB │ 0.4 │ 0 │ 39.65s │ 890.216µs │ 215 | │ srv-4292 │ cluster-spoke-1 │ localhost │ 2.3.3-beta │ yes │ 0 │ 283 │ 2 │ 0 │ 16 MiB │ 0.3 │ 0 │ 39.65s │ 865.885µs │ 216 | │ srv-4252 │ cluster-spoke-1 │ localhost │ 2.3.3-beta │ yes │ 0 │ 311 │ 2 │ 0 │ 16 MiB │ 0.2 │ 0 │ 39.65s │ 844.531µs │ 217 | │ srv-4222 │ cluster-hub │ localhost │ 2.3.3-beta │ yes │ 0 │ 306 │ 2 │ 0 │ 17 MiB │ 0.3 │ 0 │ 39.65s │ 819.322µs │ 218 | │ srv-4282 │ cluster-hub │ localhost │ 2.3.3-beta │ yes │ 1 │ 279 │ 2 │ 0 │ 16 MiB │ 0.5 │ 0 │ 39.65s │ 791.504µs │ 219 | ├──────────┼─────────────────┼───────────┼────────────┼─────┼───────┼──────┼────────┼─────┼─────────┼─────┼──────┼────────┼───────────┤ 220 | │ │ 3 Clusters │ 9 Servers │ │ 9 │ 1 │ 2663 │ │ │ 149 MiB │ │ 0 │ │ │ 221 | ╰──────────┴─────────────────┴───────────┴────────────┴─────┴───────┴──────┴────────┴─────┴─────────┴─────┴──────┴────────┴───────────╯ 222 | 223 | ╭────────────────────────────────────────────────────────────────────────────────────╮ 224 | │ Cluster Overview │ 225 | ├─────────────────┬────────────┬───────────────────┬───────────────────┬─────────────┤ 226 | │ Cluster │ Node Count │ Outgoing Gateways │ Incoming Gateways │ Connections │ 227 | ├─────────────────┼────────────┼───────────────────┼───────────────────┼─────────────┤ 228 | │ cluster-spoke-1 │ 3 │ 0 │ 0 │ 0 │ 229 | │ cluster-spoke-2 │ 3 │ 0 │ 0 │ 0 │ 230 | │ cluster-hub │ 3 │ 0 │ 0 │ 1 │ 231 | ├─────────────────┼────────────┼───────────────────┼───────────────────┼─────────────┤ 232 | │ │ 9 │ 0 │ 0 │ 1 │ 233 | ╰─────────────────┴────────────┴───────────────────┴───────────────────┴─────────────╯ 234 | ``` 235 | 236 | Here you can see all the leaf nodes as well. 237 | However, if you should connect system accounts heavily depends on your security needs! 238 | 239 | ```txt 240 | > cat leafnode-remotes.cfg 241 | HUB-URLS=["nats-leaf://localhost:4224","nats-leaf://localhost:4234","nats-leaf://localhost:4284"] 242 | leafnodes { 243 | no_advertise: true 244 | remotes = [ 245 | { 246 | urls: $HUB-URLS 247 | account: ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4 248 | credentials: keys/creds/OP/SYS/sys.creds 249 | }, 250 | { 251 | urls: $HUB-URLS 252 | account: AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G 253 | credentials: keys/creds/OP/TEST/leaf.creds 254 | }, 255 | ] 256 | } 257 | ``` 258 | 259 | This is exactly what was done here. 260 | The first remote connects the system accounts 261 | The second remote connects the account we will be using 262 | 263 | If you have the system account connected but no domain specified, this is the JetStream topology you'd get. 264 | 265 | ![`imgcat topology2-js-merged.png`](topology2-js-merged.png) 266 | 267 | This is a singe JetStream spanning all of our clusters. 268 | JetStream has a meta data leader that is in charge of creating and placing streams and consumer. 269 | In a hub/spoke setup like the one here, this meta data leader will be pinned to server that have no leaf node remotes specified (meaning, the hub). 270 | This avoids having a leaf node server become a leader and thus forcing other leafs to take two hops to get to it. 271 | But, while leaf node connections are down, in the affected server, you could not create streams or consumer during that time. 272 | 273 | #### JetStream Domains 274 | 275 | However, if you specify a JetStream `domain` in your configs, each `domain` will become an independent JetStream. 276 | My config will result in this topology. 277 | 278 | ![`imgcat topology3-js-domains.png`](topology3-js-domains.png) 279 | 280 | If you do not connect system accounts, This will be the resulting topology as well. 281 | Presence of domains cuts off certain system account traffic along leaf node connections and makes JetStream addressable by domain name. 282 | 283 | The part I want to show you now is how to connect these independent JetStreams. 284 | 285 | On the left I am starting a watch command that repeatedly executes 3 `nats` commands. 286 | This cli command is our swiss army knife. Among other things it can be used to publish and subscribe, interact with JetStream and generate various reports. 287 | It supports a context that can be set once and subsequently reused. 288 | Here, each invocation of nats uses a context that uses appropriate credentials and connects it to the corresponding cluster. 289 | 290 | ```txt 291 | > watch -n 1 "nats --context=hub s report ; nats --context=spoke-1 s report; nats --context=spoke-2 s report" 292 | 293 | Obtaining Stream stats 294 | 295 | No Streams defined 296 | Obtaining Stream stats 297 | 298 | No Streams defined 299 | Obtaining Stream stats 300 | 301 | No Streams defined 302 | ``` 303 | 304 | This will be my primary tool to show you what happened. 305 | Right now, there are no stream yet. 306 | 307 | #### Stream Mirrors across Domains 308 | 309 | One way to connect streams across JS domains would be to have a command and control stream. 310 | Essentially to communicate from the hub to each spoke without loss, even when leaf nodes are disconnected. 311 | 312 | ![`imgcat topology4-js-streams-mirror.png`](topology4-js-streams-mirror.png) 313 | 314 | We have a stream called cnc and it is subscribing to a subject by the same name. 315 | This stream is located in the hub. 316 | It's content is then mirrored to a `recv-cnc` stream located in each node. 317 | 318 | ```txt 319 | > nats --context=hub stream add cnc --subjects cnc --replicas 3 320 | ? Storage backend file 321 | ? Retention Policy Limits 322 | ? Discard Policy Old 323 | ? Stream Messages Limit -1 324 | ? Per Subject Messages Limit -1 325 | ? Message size limit -1 326 | ? Maximum message age limit -1 327 | ? Maximum individual message size -1 328 | ? Duplicate tracking time window 2m 329 | Stream cnc was created 330 | 331 | Information for Stream cnc created 2021-07-28T12:25:07-04:00 332 | 333 | Configuration: 334 | 335 | Subjects: cnc 336 | Acknowledgements: true 337 | Retention: File - Limits 338 | Replicas: 3 339 | Discard Policy: Old 340 | Duplicate Window: 2m0s 341 | Maximum Messages: unlimited 342 | Maximum Bytes: unlimited 343 | Maximum Age: 0.00s 344 | Maximum Message Size: unlimited 345 | Maximum Consumers: unlimited 346 | 347 | 348 | Cluster Information: 349 | 350 | Name: cluster-hub 351 | Leader: srv-4222 352 | Replica: srv-4232, current, seen 0.00s ago 353 | Replica: srv-4282, current, seen 0.00s ago 354 | 355 | State: 356 | 357 | Messages: 0 358 | Bytes: 0 B 359 | FirstSeq: 0 360 | LastSeq: 0 361 | Active Consumers: 0 362 | ``` 363 | 364 | Streams offer a variety of settings like storage backend, various limits etc... 365 | I will only give arguments for values I want to be changed. 366 | Here stream replica count and subject to subscribe to. 367 | This way I can quickly go through the questionnaire. 368 | 369 | Let's also store a few messages: 370 | 371 | ```txt 372 | > nats --context=hub pub cnc "hello world" --count 10 373 | 12:25:50 Published 11 bytes to "cnc" 374 | 12:25:50 Published 11 bytes to "cnc" 375 | 12:25:50 Published 11 bytes to "cnc" 376 | 12:25:50 Published 11 bytes to "cnc" 377 | 12:25:50 Published 11 bytes to "cnc" 378 | 12:25:50 Published 11 bytes to "cnc" 379 | 12:25:50 Published 11 bytes to "cnc" 380 | 12:25:50 Published 11 bytes to "cnc" 381 | 12:25:50 Published 11 bytes to "cnc" 382 | 12:25:50 Published 11 bytes to "cnc" 383 | ``` 384 | 385 | In each leaf node cluster I want a stream named recv-cnc that is a mirror of the cnc stream. 386 | The `output` option causes `nats` to store the settings in a file named `recv-cnc` 387 | 388 | ```txt 389 | > nats --context=hub stream add recv-cnc --mirror cnc --replicas 3 --output recv-cnc 390 | ? Storage backend file 391 | ? Retention Policy Limits 392 | ? Discard Policy Old 393 | ? Stream Messages Limit -1 394 | ? Message size limit -1 395 | ? Maximum message age limit -1 396 | ? Maximum individual message size -1 397 | ? Adjust mirror start No 398 | ? Import mirror from a different JetStream domain Yes 399 | ? Foreign JetStream domain name hub 400 | ? Delivery prefix 401 | ``` 402 | 403 | now I'm creating that stream in each domain 404 | 405 | ```txt 406 | > nats --context=hub stream add --config recv-cnc --js-domain spoke-1 407 | Stream recv-cnc was created 408 | 409 | Information for Stream recv-cnc created 2021-07-28T12:30:51-04:00 410 | 411 | Configuration: 412 | 413 | Acknowledgements: true 414 | Retention: File - Limits 415 | Replicas: 3 416 | Discard Policy: Old 417 | Duplicate Window: 2m0s 418 | Maximum Messages: unlimited 419 | Maximum Bytes: unlimited 420 | Maximum Age: 0.00s 421 | Maximum Message Size: unlimited 422 | Maximum Consumers: unlimited 423 | Mirror: cnc, API Prefix: $JS.hub.API, Delivery Prefix: 424 | 425 | 426 | Cluster Information: 427 | 428 | Name: cluster-spoke-1 429 | Leader: srv-4252 430 | Replica: srv-4292, current, seen 0.00s ago 431 | Replica: srv-4242, current, seen 0.00s ago 432 | 433 | Mirror Information: 434 | 435 | Stream Name: cnc 436 | Lag: 0 437 | Last Seen: never 438 | Ext. API Prefix: $JS.hub.API 439 | 440 | State: 441 | 442 | Messages: 0 443 | Bytes: 0 B 444 | FirstSeq: 0 445 | LastSeq: 0 446 | Active Consumers: 0 447 | > nats --context=hub stream add --config recv-cnc --js-domain spoke-2 448 | Stream recv-cnc was created 449 | 450 | Information for Stream recv-cnc created 2021-07-28T12:30:58-04:00 451 | 452 | Configuration: 453 | 454 | Acknowledgements: true 455 | Retention: File - Limits 456 | Replicas: 3 457 | Discard Policy: Old 458 | Duplicate Window: 2m0s 459 | Maximum Messages: unlimited 460 | Maximum Bytes: unlimited 461 | Maximum Age: 0.00s 462 | Maximum Message Size: unlimited 463 | Maximum Consumers: unlimited 464 | Mirror: cnc, API Prefix: $JS.hub.API, Delivery Prefix: 465 | 466 | 467 | Cluster Information: 468 | 469 | Name: cluster-spoke-2 470 | Leader: srv-4272 471 | Replica: srv-4202, current, seen 0.00s ago 472 | Replica: srv-4262, current, seen 0.00s ago 473 | 474 | Mirror Information: 475 | 476 | Stream Name: cnc 477 | Lag: 0 478 | Last Seen: never 479 | Ext. API Prefix: $JS.hub.API 480 | 481 | State: 482 | 483 | Messages: 0 484 | Bytes: 0 B 485 | FirstSeq: 0 486 | LastSeq: 0 487 | Active Consumers: 0 488 | ``` 489 | 490 | It is important to point out that names do only need to be unique within a JetStream domain. 491 | On the left hand side you can see the streams created, and that our `recv-cnc` streams already copied all data from `cnc`. 492 | 493 | ```txt 494 | > watch -n 1 "nats --context=hub s report ; nats --context=spoke-1 s report; nats --context=spoke-2 s report" 495 | 496 | Obtaining Stream stats 497 | 498 | ╭──────────────────────────────────────────────────────────────────────────────────────────────────╮ 499 | │ Stream Report │ 500 | ├────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤ 501 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 502 | ├────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤ 503 | │ cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4222*, srv-4232, srv-4282 │ 504 | ╰────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯ 505 | 506 | Obtaining Stream stats 507 | 508 | ╭────────────────────────────────────────────────────────────────────────────────────────────────────╮ 509 | │ Stream Report │ 510 | ├──────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤ 511 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 512 | ├──────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤ 513 | │ recv-cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4242, srv-4252*, srv-4292 │ 514 | ╰──────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯ 515 | 516 | ╭────────────────────────────────────────────────────────────────────────╮ 517 | │ Replication Report │ 518 | ├──────────┬────────┬─────────────┬───────────────┬────────┬─────┬───────┤ 519 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 520 | ├──────────┼────────┼─────────────┼───────────────┼────────┼─────┼───────┤ 521 | │ recv-cnc │ Mirror │ $JS.hub.API │ cnc │ 1.69s │ 0 │ │ 522 | ╰──────────┴────────┴─────────────┴───────────────┴────────┴─────┴───────╯ 523 | 524 | Obtaining Stream stats 525 | 526 | ╭────────────────────────────────────────────────────────────────────────────────────────────────────╮ 527 | │ Stream Report │ 528 | ├──────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤ 529 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 530 | ├──────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤ 531 | │ recv-cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4202, srv-4262, srv-4272* │ 532 | ╰──────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯ 533 | 534 | ╭────────────────────────────────────────────────────────────────────────╮ 535 | │ Replication Report │ 536 | ├──────────┬────────┬─────────────┬───────────────┬────────┬─────┬───────┤ 537 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 538 | ├──────────┼────────┼─────────────┼───────────────┼────────┼─────┼───────┤ 539 | │ recv-cnc │ Mirror │ $JS.hub.API │ cnc │ 0.53s │ 0 │ │ 540 | ╰──────────┴────────┴─────────────┴───────────────┴────────┴─────┴───────╯ 541 | ``` 542 | 543 | #### Stream Source across Domains 544 | 545 | Doing this the other way around, collecting data in the spokes and aggregating them in a stream in the hub is possible as well. 546 | 547 | ![`imgcat topology5-js-streams-source.png`](topology5-js-streams-source.png) 548 | 549 | In each spoke JetStream domain I am going to create a stream named `test` that subscribes to `test`. 550 | 551 | ```txt 552 | > nats --context=hub stream add test --subjects test --replicas 3 --output test 553 | ? Storage backend file 554 | ? Retention Policy Limits 555 | ? Discard Policy Old 556 | ? Stream Messages Limit -1 557 | ? Per Subject Messages Limit -1 558 | ? Message size limit -1 559 | ? Maximum message age limit -1 560 | ? Maximum individual message size -1 561 | ? Duplicate tracking time window 2m 562 | ``` 563 | 564 | Again, I stored the output, so I can now easily create the stream multiple times. 565 | 566 | ```txt 567 | > nats --context=hub stream add --config test --js-domain spoke-1 568 | Stream test was created 569 | 570 | Information for Stream test created 2021-07-28T12:42:12-04:00 571 | 572 | Configuration: 573 | 574 | Subjects: test 575 | Acknowledgements: true 576 | Retention: File - Limits 577 | Replicas: 3 578 | Discard Policy: Old 579 | Duplicate Window: 2m0s 580 | Maximum Messages: unlimited 581 | Maximum Bytes: unlimited 582 | Maximum Age: 0.00s 583 | Maximum Message Size: unlimited 584 | Maximum Consumers: unlimited 585 | 586 | 587 | Cluster Information: 588 | 589 | Name: cluster-spoke-1 590 | Leader: srv-4252 591 | Replica: srv-4242, current, seen 0.00s ago 592 | Replica: srv-4292, current, seen 0.00s ago 593 | 594 | State: 595 | 596 | Messages: 0 597 | Bytes: 0 B 598 | FirstSeq: 0 599 | LastSeq: 0 600 | Active Consumers: 0 601 | > nats --context=hub stream add --config test --js-domain spoke-2 602 | Stream test was created 603 | 604 | Information for Stream test created 2021-07-28T12:42:14-04:00 605 | 606 | Configuration: 607 | 608 | Subjects: test 609 | Acknowledgements: true 610 | Retention: File - Limits 611 | Replicas: 3 612 | Discard Policy: Old 613 | Duplicate Window: 2m0s 614 | Maximum Messages: unlimited 615 | Maximum Bytes: unlimited 616 | Maximum Age: 0.00s 617 | Maximum Message Size: unlimited 618 | Maximum Consumers: unlimited 619 | 620 | 621 | Cluster Information: 622 | 623 | Name: cluster-spoke-2 624 | Leader: srv-4262 625 | Replica: srv-4272, current, seen 0.00s ago 626 | Replica: srv-4202, current, seen 0.00s ago 627 | 628 | State: 629 | 630 | Messages: 0 631 | Bytes: 0 B 632 | FirstSeq: 0 633 | LastSeq: 0 634 | Active Consumers: 0 635 | ``` 636 | 637 | Please note that I created the streams in the respective domain while having been connected to the hub. 638 | If you want to interact with a JetStream that is not in the domain you are connected to, you can provide the `js-domain` option. 639 | This can also be set in the context. 640 | 641 | Let's also send 10 messages 642 | 643 | ```txt 644 | > nats --context=hub pub test "hello world" --count 10 645 | 12:43:45 Published 11 bytes to "test" 646 | 12:43:45 Published 11 bytes to "test" 647 | 12:43:45 Published 11 bytes to "test" 648 | 12:43:45 Published 11 bytes to "test" 649 | 12:43:45 Published 11 bytes to "test" 650 | 12:43:45 Published 11 bytes to "test" 651 | 12:43:45 Published 11 bytes to "test" 652 | 12:43:45 Published 11 bytes to "test" 653 | 12:43:45 Published 11 bytes to "test" 654 | 12:43:45 Published 11 bytes to "test" 655 | ``` 656 | 657 | As you can see, the messages sent while connected to hub appear in both streams. 658 | This is because they use the same subject. 659 | JetStream domains do not constrict message flow. 660 | 661 | ```txt 662 | > watch -n 1 "nats --context=hub s report ; nats --context=spoke-1 s report; nats --context=spoke-2 s report" 663 | 664 | Obtaining Stream stats 665 | 666 | ╭──────────────────────────────────────────────────────────────────────────────────────────────────╮ 667 | │ Stream Report │ 668 | ├────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤ 669 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 670 | ├────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤ 671 | │ cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4222*, srv-4232, srv-4282 │ 672 | ╰────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯ 673 | 674 | Obtaining Stream stats 675 | 676 | ╭────────────────────────────────────────────────────────────────────────────────────────────────────╮ 677 | │ Stream Report │ 678 | ├──────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤ 679 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 680 | ├──────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤ 681 | │ recv-cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4242, srv-4252*, srv-4292 │ 682 | │ test │ File │ 0 │ 10 │ 450 B │ 0 │ 0 │ srv-4242, srv-4252*, srv-4292 │ 683 | ╰──────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯ 684 | 685 | ╭────────────────────────────────────────────────────────────────────────╮ 686 | │ Replication Report │ 687 | ├──────────┬────────┬─────────────┬───────────────┬────────┬─────┬───────┤ 688 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 689 | ├──────────┼────────┼─────────────┼───────────────┼────────┼─────┼───────┤ 690 | │ recv-cnc │ Mirror │ $JS.hub.API │ cnc │ 1.35s │ 0 │ │ 691 | ╰──────────┴────────┴─────────────┴───────────────┴────────┴─────┴───────╯ 692 | 693 | Obtaining Stream stats 694 | 695 | ╭────────────────────────────────────────────────────────────────────────────────────────────────────╮ 696 | │ Stream Report │ 697 | ├──────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤ 698 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 699 | ├──────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤ 700 | │ recv-cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4202, srv-4262, srv-4272* │ 701 | │ test │ File │ 0 │ 10 │ 450 B │ 0 │ 0 │ srv-4202, srv-4262*, srv-4272 │ 702 | ╰──────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯ 703 | 704 | ╭────────────────────────────────────────────────────────────────────────╮ 705 | │ Replication Report │ 706 | ├──────────┬────────┬─────────────┬───────────────┬────────┬─────┬───────┤ 707 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 708 | ├──────────┼────────┼─────────────┼───────────────┼────────┼─────┼───────┤ 709 | │ recv-cnc │ Mirror │ $JS.hub.API │ cnc │ 0.24s │ 0 │ │ 710 | ╰──────────┴────────┴─────────────┴───────────────┴────────┴─────┴───────╯ 711 | ``` 712 | 713 | If you are using a non overlapping set of subject names in each domain, this won't happen to you. 714 | Same if you are using a different set of accounts in each domain. 715 | Later I will show a third way of doing this. 716 | 717 | ```txt 718 | > nats --context=spoke-1 pub test "hello world" --count 10 719 | 12:46:27 Published 11 bytes to "test" 720 | 12:46:27 Published 11 bytes to "test" 721 | 12:46:27 Published 11 bytes to "test" 722 | 12:46:27 Published 11 bytes to "test" 723 | 12:46:27 Published 11 bytes to "test" 724 | 12:46:27 Published 11 bytes to "test" 725 | 12:46:27 Published 11 bytes to "test" 726 | 12:46:27 Published 11 bytes to "test" 727 | 12:46:27 Published 11 bytes to "test" 728 | 12:46:27 Published 11 bytes to "test" 729 | ``` 730 | 731 | Of course this also applies when you connect to a spoke as well. 732 | 733 | ```txt 734 | > watch -n 1 "nats --context=hub s report ; nats --context=spoke-1 s report; nats --context=spoke-2 s report" 735 | 736 | Obtaining Stream stats 737 | 738 | ╭──────────────────────────────────────────────────────────────────────────────────────────────────╮ 739 | │ Stream Report │ 740 | ├────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤ 741 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 742 | ├────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤ 743 | │ cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4222*, srv-4232, srv-4282 │ 744 | ╰────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯ 745 | 746 | Obtaining Stream stats 747 | 748 | ╭────────────────────────────────────────────────────────────────────────────────────────────────────╮ 749 | │ Stream Report │ 750 | ├──────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤ 751 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 752 | ├──────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤ 753 | │ recv-cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4242, srv-4252*, srv-4292 │ 754 | │ test │ File │ 0 │ 20 │ 900 B │ 0 │ 0 │ srv-4242, srv-4252*, srv-4292 │ 755 | ╰──────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯ 756 | 757 | ╭────────────────────────────────────────────────────────────────────────╮ 758 | │ Replication Report │ 759 | ├──────────┬────────┬─────────────┬───────────────┬────────┬─────┬───────┤ 760 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 761 | ├──────────┼────────┼─────────────┼───────────────┼────────┼─────┼───────┤ 762 | │ recv-cnc │ Mirror │ $JS.hub.API │ cnc │ 0.68s │ 0 │ │ 763 | ╰──────────┴────────┴─────────────┴───────────────┴────────┴─────┴───────╯ 764 | 765 | Obtaining Stream stats 766 | 767 | ╭────────────────────────────────────────────────────────────────────────────────────────────────────╮ 768 | │ Stream Report │ 769 | ├──────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤ 770 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 771 | ├──────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤ 772 | │ recv-cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4202, srv-4262, srv-4272* │ 773 | │ test │ File │ 0 │ 20 │ 900 B │ 0 │ 0 │ srv-4202, srv-4262*, srv-4272 │ 774 | ╰──────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯ 775 | 776 | ╭────────────────────────────────────────────────────────────────────────╮ 777 | │ Replication Report │ 778 | ├──────────┬────────┬─────────────┬───────────────┬────────┬─────┬───────┤ 779 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 780 | ├──────────┼────────┼─────────────┼───────────────┼────────┼─────┼───────┤ 781 | │ recv-cnc │ Mirror │ $JS.hub.API │ cnc │ 1.60s │ 0 │ │ 782 | ╰──────────┴────────┴─────────────┴───────────────┴────────┴─────┴───────╯ 783 | ``` 784 | 785 | Now let's create a stream named `aggregate`, sourcing these streams. 786 | Here I have to specify which domain to source from (`spoke-1`/`spoke-2`). 787 | 788 | ```txt 789 | > nats --context=hub stream add aggregate --replicas 3 --source test --source test 790 | ? Storage backend file 791 | ? Retention Policy Limits 792 | ? Discard Policy Old 793 | ? Stream Messages Limit -1 794 | ? Message size limit -1 795 | ? Maximum message age limit -1 796 | ? Maximum individual message size -1 797 | ? Duplicate tracking time window 2m 798 | ? Adjust source "test" start No 799 | ? Import "test" from a different JetStream domain Yes 800 | ? test Source foreign JetStream domain name spoke-1 801 | ? test Source foreign JetStream domain delivery prefix 802 | ? Adjust source "test" start No 803 | ? Import "test" from a different JetStream domain Yes 804 | ? test Source foreign JetStream domain name spoke-2 805 | ? test Source foreign JetStream domain delivery prefix 806 | Stream aggregate was created 807 | 808 | Information for Stream aggregate created 2021-07-28T12:48:52-04:00 809 | 810 | Configuration: 811 | 812 | Acknowledgements: true 813 | Retention: File - Limits 814 | Replicas: 3 815 | Discard Policy: Old 816 | Duplicate Window: 2m0s 817 | Maximum Messages: unlimited 818 | Maximum Bytes: unlimited 819 | Maximum Age: 0.00s 820 | Maximum Message Size: unlimited 821 | Maximum Consumers: unlimited 822 | Sources: test, API Prefix: $JS.spoke-1.API, Delivery Prefix: 823 | test, API Prefix: $JS.spoke-2.API, Delivery Prefix: 824 | 825 | 826 | Cluster Information: 827 | 828 | Name: cluster-hub 829 | Leader: srv-4282 830 | Replica: srv-4232, current, seen 0.00s ago 831 | Replica: srv-4222, current, seen 0.00s ago 832 | 833 | Source Information: 834 | 835 | Stream Name: test 836 | Lag: 0 837 | Last Seen: 0.00s 838 | Ext. API Prefix: $JS.spoke-1.API 839 | 840 | Stream Name: test 841 | Lag: 0 842 | Last Seen: 0.00s 843 | Ext. API Prefix: $JS.spoke-2.API 844 | 845 | State: 846 | 847 | Messages: 0 848 | Bytes: 0 B 849 | FirstSeq: 0 850 | LastSeq: 0 851 | Active Consumers: 0 852 | ``` 853 | 854 | Here are the corresponding stream reports. 855 | 856 | ```txt 857 | > watch -n 1 "nats --context=hub s report ; nats --context=spoke-1 s report; nats --context=spoke-2 s report" 858 | 859 | Obtaining Stream stats 860 | 861 | ╭───────────────────────────────────────────────────────────────────────────────────────────────────────╮ 862 | │ Stream Report │ 863 | ├───────────┬─────────┬───────────┬──────────┬─────────┬──────┬─────────┬───────────────────────────────┤ 864 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 865 | ├───────────┼─────────┼───────────┼──────────┼─────────┼──────┼─────────┼───────────────────────────────┤ 866 | │ cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4222*, srv-4232, srv-4282 │ 867 | │ aggregate │ File │ 0 │ 40 │ 3.8 KiB │ 0 │ 0 │ srv-4222, srv-4232, srv-4282* │ 868 | ╰───────────┴─────────┴───────────┴──────────┴─────────┴──────┴─────────┴───────────────────────────────╯ 869 | 870 | ╭─────────────────────────────────────────────────────────────────────────────╮ 871 | │ Replication Report │ 872 | ├───────────┬────────┬─────────────────┬───────────────┬────────┬─────┬───────┤ 873 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 874 | ├───────────┼────────┼─────────────────┼───────────────┼────────┼─────┼───────┤ 875 | │ aggregate │ Source │ $JS.spoke-1.API │ test │ 0.19s │ 0 │ │ 876 | │ aggregate │ Source │ $JS.spoke-2.API │ test │ 0.18s │ 0 │ │ 877 | ╰───────────┴────────┴─────────────────┴───────────────┴────────┴─────┴───────╯ 878 | 879 | Obtaining Stream stats 880 | 881 | ╭────────────────────────────────────────────────────────────────────────────────────────────────────╮ 882 | │ Stream Report │ 883 | ├──────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤ 884 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 885 | ├──────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤ 886 | │ recv-cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4242, srv-4252*, srv-4292 │ 887 | │ test │ File │ 0 │ 20 │ 900 B │ 0 │ 0 │ srv-4242, srv-4252*, srv-4292 │ 888 | ╰──────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯ 889 | 890 | ╭────────────────────────────────────────────────────────────────────────╮ 891 | │ Replication Report │ 892 | ├──────────┬────────┬─────────────┬───────────────┬────────┬─────┬───────┤ 893 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 894 | ├──────────┼────────┼─────────────┼───────────────┼────────┼─────┼───────┤ 895 | │ recv-cnc │ Mirror │ $JS.hub.API │ cnc │ 1.93s │ 0 │ │ 896 | ╰──────────┴────────┴─────────────┴───────────────┴────────┴─────┴───────╯ 897 | 898 | Obtaining Stream stats 899 | 900 | ╭────────────────────────────────────────────────────────────────────────────────────────────────────╮ 901 | │ Stream Report │ 902 | ├──────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤ 903 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 904 | ├──────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤ 905 | │ recv-cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4202, srv-4262, srv-4272* │ 906 | │ test │ File │ 0 │ 20 │ 900 B │ 0 │ 0 │ srv-4202, srv-4262*, srv-4272 │ 907 | ╰──────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯ 908 | 909 | ╭────────────────────────────────────────────────────────────────────────╮ 910 | │ Replication Report │ 911 | ├──────────┬────────┬─────────────┬───────────────┬────────┬─────┬───────┤ 912 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 913 | ├──────────┼────────┼─────────────┼───────────────┼────────┼─────┼───────┤ 914 | │ recv-cnc │ Mirror │ $JS.hub.API │ cnc │ 0.89s │ 0 │ │ 915 | ╰──────────┴────────┴─────────────┴───────────────┴────────┴─────┴───────╯ 916 | ``` 917 | 918 | #### Source/Mirror not dependent on Topology 919 | 920 | Let me also demonstrate that source and mirror stream relationships do not have to align with the underlying topology. 921 | 922 | ![`imgcat topology6-js-streams-backup.png`](topology6-js-streams-backup.png) 923 | 924 | Now I'm going to create a backup of the stream test in `spoke-1`. 925 | The backup itself is located in `spoke-2`. 926 | 927 | ``` 928 | > nats --context=hub stream add backup-test-spoke-1 --replicas 3 --mirror test --js-domain spoke-2 929 | ? Storage backend file 930 | ? Retention Policy Limits 931 | ? Discard Policy Old 932 | ? Stream Messages Limit -1 933 | ? Message size limit -1 934 | ? Maximum message age limit -1 935 | ? Maximum individual message size -1 936 | ? Adjust mirror start No 937 | ? Import mirror from a different JetStream domain Yes 938 | ? Foreign JetStream domain name spoke-1 939 | ? Delivery prefix 940 | Stream backup-test-spoke-1 was created 941 | 942 | Information for Stream backup-test-spoke-1 created 2021-07-28T12:59:07-04:00 943 | 944 | Configuration: 945 | 946 | Acknowledgements: true 947 | Retention: File - Limits 948 | Replicas: 3 949 | Discard Policy: Old 950 | Duplicate Window: 2m0s 951 | Maximum Messages: unlimited 952 | Maximum Bytes: unlimited 953 | Maximum Age: 0.00s 954 | Maximum Message Size: unlimited 955 | Maximum Consumers: unlimited 956 | Mirror: test, API Prefix: $JS.spoke-1.API, Delivery Prefix: 957 | 958 | 959 | Cluster Information: 960 | 961 | Name: cluster-spoke-2 962 | Leader: srv-4202 963 | Replica: srv-4262, current, seen 0.00s ago 964 | Replica: srv-4272, current, seen 0.00s ago 965 | 966 | Mirror Information: 967 | 968 | Stream Name: test 969 | Lag: 0 970 | Last Seen: never 971 | Ext. API Prefix: $JS.spoke-1.API 972 | 973 | State: 974 | 975 | Messages: 0 976 | Bytes: 0 B 977 | FirstSeq: 0 978 | LastSeq: 0 979 | Active Consumers: 0 980 | ``` 981 | 982 | And send 10 messages. 983 | 984 | ```txt 985 | > nats --context=spoke-1 pub test "hello world" --count 10 986 | 13:00:48 Published 11 bytes to "test" 987 | 13:00:48 Published 11 bytes to "test" 988 | 13:00:48 Published 11 bytes to "test" 989 | 13:00:48 Published 11 bytes to "test" 990 | 13:00:48 Published 11 bytes to "test" 991 | 13:00:48 Published 11 bytes to "test" 992 | 13:00:48 Published 11 bytes to "test" 993 | 13:00:48 Published 11 bytes to "test" 994 | 13:00:48 Published 11 bytes to "test" 995 | 13:00:48 Published 11 bytes to "test" 996 | ``` 997 | 998 | These messages also appear in `backup-test-spoke-1`. 999 | 1000 | ```txt 1001 | > watch -n 1 "nats --context=hub s report ; nats --context=spoke-1 s report; nats --context=spoke-2 s report" 1002 | 1003 | Obtaining Stream stats 1004 | 1005 | ╭───────────────────────────────────────────────────────────────────────────────────────────────────────╮ 1006 | │ Stream Report │ 1007 | ├───────────┬─────────┬───────────┬──────────┬─────────┬──────┬─────────┬───────────────────────────────┤ 1008 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 1009 | ├───────────┼─────────┼───────────┼──────────┼─────────┼──────┼─────────┼───────────────────────────────┤ 1010 | │ cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4222*, srv-4232, srv-4282 │ 1011 | │ aggregate │ File │ 0 │ 60 │ 5.8 KiB │ 0 │ 0 │ srv-4222, srv-4232, srv-4282* │ 1012 | ╰───────────┴─────────┴───────────┴──────────┴─────────┴──────┴─────────┴───────────────────────────────╯ 1013 | 1014 | ╭─────────────────────────────────────────────────────────────────────────────╮ 1015 | │ Replication Report │ 1016 | ├───────────┬────────┬─────────────────┬───────────────┬────────┬─────┬───────┤ 1017 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 1018 | ├───────────┼────────┼─────────────────┼───────────────┼────────┼─────┼───────┤ 1019 | │ aggregate │ Source │ $JS.spoke-2.API │ test │ 1.18s │ 0 │ │ 1020 | │ aggregate │ Source │ $JS.spoke-1.API │ test │ 1.18s │ 0 │ │ 1021 | ╰───────────┴────────┴─────────────────┴───────────────┴────────┴─────┴───────╯ 1022 | 1023 | Obtaining Stream stats 1024 | 1025 | ╭──────────────────────────────────────────────────────────────────────────────────────────────────────╮ 1026 | │ Stream Report │ 1027 | ├──────────┬─────────┬───────────┬──────────┬─────────┬──────┬─────────┬───────────────────────────────┤ 1028 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 1029 | ├──────────┼─────────┼───────────┼──────────┼─────────┼──────┼─────────┼───────────────────────────────┤ 1030 | │ recv-cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4242, srv-4252*, srv-4292 │ 1031 | │ test │ File │ 0 │ 30 │ 1.3 KiB │ 0 │ 0 │ srv-4242, srv-4252*, srv-4292 │ 1032 | ╰──────────┴─────────┴───────────┴──────────┴─────────┴──────┴─────────┴───────────────────────────────╯ 1033 | 1034 | ╭────────────────────────────────────────────────────────────────────────╮ 1035 | │ Replication Report │ 1036 | ├──────────┬────────┬─────────────┬───────────────┬────────┬─────┬───────┤ 1037 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 1038 | ├──────────┼────────┼─────────────┼───────────────┼────────┼─────┼───────┤ 1039 | │ recv-cnc │ Mirror │ $JS.hub.API │ cnc │ 1.76s │ 0 │ │ 1040 | ╰──────────┴────────┴─────────────┴───────────────┴────────┴─────┴───────╯ 1041 | 1042 | Obtaining Stream stats 1043 | 1044 | ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ 1045 | │ Stream Report │ 1046 | ├─────────────────────┬─────────┬───────────┬──────────┬─────────┬──────┬─────────┬───────────────────────────────┤ 1047 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 1048 | ├─────────────────────┼─────────┼───────────┼──────────┼─────────┼──────┼─────────┼───────────────────────────────┤ 1049 | │ recv-cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4202, srv-4262, srv-4272* │ 1050 | │ backup-test-spoke-1 │ File │ 0 │ 30 │ 1.3 KiB │ 0 │ 0 │ srv-4202*, srv-4262, srv-4272 │ 1051 | │ test │ File │ 0 │ 30 │ 1.3 KiB │ 0 │ 0 │ srv-4202, srv-4262*, srv-4272 │ 1052 | ╰─────────────────────┴─────────┴───────────┴──────────┴─────────┴──────┴─────────┴───────────────────────────────╯ 1053 | 1054 | ╭───────────────────────────────────────────────────────────────────────────────────────╮ 1055 | │ Replication Report │ 1056 | ├─────────────────────┬────────┬─────────────────┬───────────────┬────────┬─────┬───────┤ 1057 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 1058 | ├─────────────────────┼────────┼─────────────────┼───────────────┼────────┼─────┼───────┤ 1059 | │ recv-cnc │ Mirror │ $JS.hub.API │ cnc │ 0.67s │ 0 │ │ 1060 | │ backup-test-spoke-1 │ Mirror │ $JS.spoke-1.API │ test │ 1.25s │ 0 │ │ 1061 | ╰─────────────────────┴────────┴─────────────────┴───────────────┴────────┴─────┴───────╯ 1062 | ``` 1063 | 1064 | Let's quickly demonstrate that this works the way I explained by shutting down all server in the cluster hub. 1065 | Here our watch command for the hub can't connect any longer. 1066 | 1067 | ``` 1068 | > watch -n 1 "nats --context=hub s report ; nats --context=spoke-1 s report; nats --context=spoke-2 s report" 1069 | 1070 | nats: error: setup failed: nats: no servers available for connection 1071 | Obtaining Stream stats 1072 | 1073 | ╭──────────────────────────────────────────────────────────────────────────────────────────────────────╮ 1074 | │ Stream Report │ 1075 | ├──────────┬─────────┬───────────┬──────────┬─────────┬──────┬─────────┬───────────────────────────────┤ 1076 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 1077 | ├──────────┼─────────┼───────────┼──────────┼─────────┼──────┼─────────┼───────────────────────────────┤ 1078 | │ recv-cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4242, srv-4252*, srv-4292 │ 1079 | │ test │ File │ 0 │ 30 │ 1.3 KiB │ 0 │ 0 │ srv-4242, srv-4252*, srv-4292 │ 1080 | ╰──────────┴─────────┴───────────┴──────────┴─────────┴──────┴─────────┴───────────────────────────────╯ 1081 | 1082 | ╭────────────────────────────────────────────────────────────────────────╮ 1083 | │ Replication Report │ 1084 | ├──────────┬────────┬─────────────┬───────────────┬────────┬─────┬───────┤ 1085 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 1086 | ├──────────┼────────┼─────────────┼───────────────┼────────┼─────┼───────┤ 1087 | │ recv-cnc │ Mirror │ $JS.hub.API │ cnc │ 9.48s │ 0 │ │ 1088 | ╰──────────┴────────┴─────────────┴───────────────┴────────┴─────┴───────╯ 1089 | 1090 | Obtaining Stream stats 1091 | 1092 | ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ 1093 | │ Stream Report │ 1094 | ├─────────────────────┬─────────┬───────────┬──────────┬─────────┬──────┬─────────┬───────────────────────────────┤ 1095 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 1096 | ├─────────────────────┼─────────┼───────────┼──────────┼─────────┼──────┼─────────┼───────────────────────────────┤ 1097 | │ recv-cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4202, srv-4262, srv-4272* │ 1098 | │ backup-test-spoke-1 │ File │ 0 │ 30 │ 1.3 KiB │ 0 │ 0 │ srv-4202*, srv-4262, srv-4272 │ 1099 | │ test │ File │ 0 │ 30 │ 1.3 KiB │ 0 │ 0 │ srv-4202, srv-4262*, srv-4272 │ 1100 | ╰─────────────────────┴─────────┴───────────┴──────────┴─────────┴──────┴─────────┴───────────────────────────────╯ 1101 | 1102 | ╭───────────────────────────────────────────────────────────────────────────────────────╮ 1103 | │ Replication Report │ 1104 | ├─────────────────────┬────────┬─────────────────┬───────────────┬────────┬─────┬───────┤ 1105 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 1106 | ├─────────────────────┼────────┼─────────────────┼───────────────┼────────┼─────┼───────┤ 1107 | │ recv-cnc │ Mirror │ $JS.hub.API │ cnc │ 10.42s │ 0 │ │ 1108 | │ backup-test-spoke-1 │ Mirror │ $JS.spoke-1.API │ test │ 11.00s │ 0 │ │ 1109 | ╰─────────────────────┴────────┴─────────────────┴───────────────┴────────┴─────┴───────╯ 1110 | ``` 1111 | 1112 | Let's send 10 messages. 1113 | 1114 | ```txt 1115 | nats --context=spoke-1 pub test "hello world" --count 10 1116 | 13:08:32 Published 11 bytes to "test" 1117 | 13:08:32 Published 11 bytes to "test" 1118 | 13:08:32 Published 11 bytes to "test" 1119 | 13:08:32 Published 11 bytes to "test" 1120 | 13:08:32 Published 11 bytes to "test" 1121 | 13:08:32 Published 11 bytes to "test" 1122 | 13:08:32 Published 11 bytes to "test" 1123 | 13:08:32 Published 11 bytes to "test" 1124 | 13:08:32 Published 11 bytes to "test" 1125 | 13:08:32 Published 11 bytes to "test" 1126 | ``` 1127 | 1128 | And observe them being in the stream `test` but due to the missing connectivity via `hub` not yet in `backup-test-spoke-1`. 1129 | 1130 | ```txt 1131 | > watch -n 1 "nats --context=hub s report ; nats --context=spoke-1 s report; nats --context=spoke-2 s report" 1132 | 1133 | nats: error: setup failed: nats: no servers available for connection 1134 | Obtaining Stream stats 1135 | 1136 | ╭──────────────────────────────────────────────────────────────────────────────────────────────────────╮ 1137 | │ Stream Report │ 1138 | ├──────────┬─────────┬───────────┬──────────┬─────────┬──────┬─────────┬───────────────────────────────┤ 1139 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 1140 | ├──────────┼─────────┼───────────┼──────────┼─────────┼──────┼─────────┼───────────────────────────────┤ 1141 | │ recv-cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4242, srv-4252*, srv-4292 │ 1142 | │ test │ File │ 0 │ 40 │ 1.8 KiB │ 0 │ 0 │ srv-4242, srv-4252*, srv-4292 │ 1143 | ╰──────────┴─────────┴───────────┴──────────┴─────────┴──────┴─────────┴───────────────────────────────╯ 1144 | 1145 | ╭────────────────────────────────────────────────────────────────────────╮ 1146 | │ Replication Report │ 1147 | ├──────────┬────────┬─────────────┬───────────────┬────────┬─────┬───────┤ 1148 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 1149 | ├──────────┼────────┼─────────────┼───────────────┼────────┼─────┼───────┤ 1150 | │ recv-cnc │ Mirror │ $JS.hub.API │ cnc │ 53.84s │ 0 │ │ 1151 | ╰──────────┴────────┴─────────────┴───────────────┴────────┴─────┴───────╯ 1152 | 1153 | Obtaining Stream stats 1154 | 1155 | ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ 1156 | │ Stream Report │ 1157 | ├─────────────────────┬─────────┬───────────┬──────────┬─────────┬──────┬─────────┬───────────────────────────────┤ 1158 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 1159 | ├─────────────────────┼─────────┼───────────┼──────────┼─────────┼──────┼─────────┼───────────────────────────────┤ 1160 | │ recv-cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4202, srv-4262, srv-4272* │ 1161 | │ backup-test-spoke-1 │ File │ 0 │ 30 │ 1.3 KiB │ 0 │ 0 │ srv-4202*, srv-4262, srv-4272 │ 1162 | │ test │ File │ 0 │ 30 │ 1.3 KiB │ 0 │ 0 │ srv-4202, srv-4262*, srv-4272 │ 1163 | ╰─────────────────────┴─────────┴───────────┴──────────┴─────────┴──────┴─────────┴───────────────────────────────╯ 1164 | 1165 | ╭───────────────────────────────────────────────────────────────────────────────────────╮ 1166 | │ Replication Report │ 1167 | ├─────────────────────┬────────┬─────────────────┬───────────────┬────────┬─────┬───────┤ 1168 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 1169 | ├─────────────────────┼────────┼─────────────────┼───────────────┼────────┼─────┼───────┤ 1170 | │ recv-cnc │ Mirror │ $JS.hub.API │ cnc │ 54.78s │ 0 │ │ 1171 | │ backup-test-spoke-1 │ Mirror │ $JS.spoke-1.API │ test │ 55.37s │ 0 │ │ 1172 | ╰─────────────────────┴────────┴─────────────────┴───────────────┴────────┴─────┴───────╯ 1173 | ``` 1174 | 1175 | See there is a difference in message counts. 1176 | 1177 | Until I start the hub again, at which point, the message count of `backup-test-spoke-1` is identical to `test` in `spoke-1` 1178 | 1179 | ```txt 1180 | > watch -n 1 "nats --context=hub s report ; nats --context=spoke-1 s report; nats --context=spoke-2 s report" 1181 | 1182 | Obtaining Stream stats 1183 | 1184 | ╭───────────────────────────────────────────────────────────────────────────────────────────────────────╮ 1185 | │ Stream Report │ 1186 | ├───────────┬─────────┬───────────┬──────────┬─────────┬──────┬─────────┬───────────────────────────────┤ 1187 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 1188 | ├───────────┼─────────┼───────────┼──────────┼─────────┼──────┼─────────┼───────────────────────────────┤ 1189 | │ cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4222*, srv-4232, srv-4282 │ 1190 | │ aggregate │ File │ 0 │ 70 │ 6.8 KiB │ 0 │ 0 │ srv-4222, srv-4232*, srv-4282 │ 1191 | ╰───────────┴─────────┴───────────┴──────────┴─────────┴──────┴─────────┴───────────────────────────────╯ 1192 | 1193 | ╭─────────────────────────────────────────────────────────────────────────────╮ 1194 | │ Replication Report │ 1195 | ├───────────┬────────┬─────────────────┬───────────────┬────────┬─────┬───────┤ 1196 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 1197 | ├───────────┼────────┼─────────────────┼───────────────┼────────┼─────┼───────┤ 1198 | │ aggregate │ Source │ $JS.spoke-1.API │ test │ 1.24s │ 0 │ │ 1199 | │ aggregate │ Source │ $JS.spoke-2.API │ test │ 1.24s │ 0 │ │ 1200 | ╰───────────┴────────┴─────────────────┴───────────────┴────────┴─────┴───────╯ 1201 | 1202 | Obtaining Stream stats 1203 | 1204 | ╭──────────────────────────────────────────────────────────────────────────────────────────────────────╮ 1205 | │ Stream Report │ 1206 | ├──────────┬─────────┬───────────┬──────────┬─────────┬──────┬─────────┬───────────────────────────────┤ 1207 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 1208 | ├──────────┼─────────┼───────────┼──────────┼─────────┼──────┼─────────┼───────────────────────────────┤ 1209 | │ recv-cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4242, srv-4252*, srv-4292 │ 1210 | │ test │ File │ 0 │ 40 │ 1.8 KiB │ 0 │ 0 │ srv-4242, srv-4252*, srv-4292 │ 1211 | ╰──────────┴─────────┴───────────┴──────────┴─────────┴──────┴─────────┴───────────────────────────────╯ 1212 | 1213 | ╭────────────────────────────────────────────────────────────────────────╮ 1214 | │ Replication Report │ 1215 | ├──────────┬────────┬─────────────┬───────────────┬────────┬─────┬───────┤ 1216 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 1217 | ├──────────┼────────┼─────────────┼───────────────┼────────┼─────┼───────┤ 1218 | │ recv-cnc │ Mirror │ $JS.hub.API │ cnc │ 0.58s │ 0 │ │ 1219 | ╰──────────┴────────┴─────────────┴───────────────┴────────┴─────┴───────╯ 1220 | 1221 | Obtaining Stream stats 1222 | 1223 | ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ 1224 | │ Stream Report │ 1225 | ├─────────────────────┬─────────┬───────────┬──────────┬─────────┬──────┬─────────┬───────────────────────────────┤ 1226 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 1227 | ├─────────────────────┼─────────┼───────────┼──────────┼─────────┼──────┼─────────┼───────────────────────────────┤ 1228 | │ recv-cnc │ File │ 0 │ 10 │ 440 B │ 0 │ 0 │ srv-4202, srv-4262, srv-4272* │ 1229 | │ test │ File │ 0 │ 30 │ 1.3 KiB │ 0 │ 0 │ srv-4202*, srv-4262, srv-4272 │ 1230 | │ backup-test-spoke-1 │ File │ 0 │ 40 │ 1.8 KiB │ 0 │ 0 │ srv-4202*, srv-4262, srv-4272 │ 1231 | ╰─────────────────────┴─────────┴───────────┴──────────┴─────────┴──────┴─────────┴───────────────────────────────╯ 1232 | 1233 | ╭───────────────────────────────────────────────────────────────────────────────────────╮ 1234 | │ Replication Report │ 1235 | ├─────────────────────┬────────┬─────────────────┬───────────────┬────────┬─────┬───────┤ 1236 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 1237 | ├─────────────────────┼────────┼─────────────────┼───────────────┼────────┼─────┼───────┤ 1238 | │ recv-cnc │ Mirror │ $JS.hub.API │ cnc │ 0.32s │ 0 │ │ 1239 | │ backup-test-spoke-1 │ Mirror │ $JS.spoke-1.API │ test │ 0.51s │ 0 │ │ 1240 | ╰─────────────────────┴────────┴─────────────────┴───────────────┴────────┴─────┴───────╯ 1241 | ``` 1242 | 1243 | #### Connect Streams Cross Accounts 1244 | 1245 | Until now we have exchanged streams across JS domains, but we stayed in the same account. 1246 | Now let me show you how to exchange stream data across accounts. 1247 | 1248 | This will largely be an exercise in maintaining prefixes to avoid subject overlaps. 1249 | 1250 | I am using `nsc` to create accounts, and users, and modify them. 1251 | Because I am using the nats account resolver, when done, I can simply `push` my changes into the network. 1252 | The `nsc` directory is in the same directory where my servers got started. 1253 | Part of my server config just uses the creds files in the nsc keys directory. 1254 | I did this, so that for this demo I have everything in one place without having to copy files around. 1255 | However, the power of our jwt based approach is that you can have that `nsc` environment anywhere. 1256 | Provided you can connect to `push`, everything will work just the same. 1257 | 1258 | The commands I show next can be translated into accounts in regular config files as well. 1259 | 1260 | Here is the resulting topology. 1261 | 1262 | ![`imgcat topology7-mirror-cross-account.png`](topology7-mirror-cross-account.png) 1263 | 1264 | The account `TEST` is the account we have been using so far. 1265 | We want to mirror the stream `aggregate` that we just created into a stream named `crossacc`. 1266 | Mirroring that particular stream allows the importing stream in the other account to be independent of the actual number of spokes. 1267 | 1268 | But first we need another JS enabled account and user: 1269 | 1270 | ```txt 1271 | > nsc add account -n IMPORTER 1272 | [ OK ] generated and stored account key "ABPGFDEBTHRPZIPYEDUMLTPUXPCSEG6DVG5IKW4PS55GHWQSYVMZBROI" 1273 | [ OK ] added account "IMPORTER" 1274 | > nsc edit account --name IMPORTER --js-disk-storage -1 --js-streams -1 --js-consumer -1 1275 | [ OK ] edited account "IMPORTER" 1276 | > nsc add user --account IMPORTER -n iuser 1277 | [ OK ] generated and stored user key "UCRX3AG5BCSJPHJ3ZI3PH4FLL2CRUMCI4LOMLNDVEJEOERQX55IBMFIY" 1278 | [ OK ] generated user creds file `~/test/jetstream-leaf-nodes-demo/keys/creds/OP/IMPORTER/iuser.creds` 1279 | [ OK ] added user "iuser" to account "IMPORTER" 1280 | ``` 1281 | 1282 | Here I created the account, enabled JetStream and created a user. 1283 | 1284 | To connect our accounts we need to export the consumer API needed when using mirror. 1285 | The advantage of copying the data from one account into another is that this avoids having to explicitly create a consumer for one account in another. 1286 | The other advantage is that you don't have to write and run a program that does the copying. 1287 | 1288 | ```txt 1289 | > nsc add export --account TEST --name Consumer-API --service --response-type Stream --subject '$JS.hub.API.CONSUMER.>' 1290 | [ OK ] added public service export "Consumer-API" 1291 | ``` 1292 | 1293 | Here we are exporting the consumer API as public service with a stream as response (meaning more than one message as response). 1294 | This can also be done as private export which requires a token signed by the exporting account for the importing account. Therefore you have precise control on who can import. 1295 | You can also export the entire JetStream API by exporting `$JS.hub.API.>`. 1296 | If you do so, you are giving full control over JS to everyone importing. 1297 | I also export `$JS.hub.API` instead of `$JS.API`. 1298 | This is so that I can pin access to a particular JS domain and not just use to the one where I connect to. 1299 | 1300 | On import we change `$JS.hub.API` to `JS.test@hub.API`. 1301 | This is done to stay clear of the $JS prefix which may get additions as new features are added to JetStream. 1302 | We give it a different prefix and subsequently specify that prefix if we want to talk to that particular JetStream. 1303 | Btw this import renaming feature is generally available. 1304 | Different organizations working on different applications most likely have different naming schemes. 1305 | So when they clash, just rename on import. 1306 | As long as they same number/type of wildcards are present you are good. 1307 | Reordering of wildcards would be possible to, but that's for another time. 1308 | 1309 | ```txt 1310 | > nsc add import --account IMPORTER --src-account TEST --name Remote-Consumer-API --service --remote-subject '$JS.hub.API.CONSUMER.>' --local-subject 'JS.test@hub.API.CONSUMER.>' 1311 | [ OK ] added service import "$JS.hub.API.CONSUMER.>" 1312 | ``` 1313 | 1314 | We also need a subject to deliver our data on. 1315 | It is important to note that the subject we will be using in a bit is a lot longer than that. 1316 | Essentially, each mirror will need a unique subject. 1317 | 1318 | I am picking the common prefix on export and subsequently add parts. 1319 | 1320 | ```txt 1321 | > nsc add export --account TEST --name Data-Path --response-type Stream --subject 'deliver.>' 1322 | [ OK ] added public stream export "Data-Path" 1323 | ``` 1324 | 1325 | On import the importing account's name is added. 1326 | 1327 | ```txt 1328 | > nsc add import --account IMPORTER --src-account TEST --name Remote-Data-Path --remote-subject 'deliver.importer.>' 1329 | [ OK ] added stream import "deliver.importer.>" 1330 | ``` 1331 | 1332 | Upload the changes 1333 | 1334 | ```txt 1335 | > nsc push -A 1336 | [ OK ] push to nats-server "nats://localhost:4222,nats://localhost:4232,nats://localhost:4282" using system account "SYS": 1337 | [ OK ] push IMPORTER to nats-server with nats account resolver: 1338 | [ OK ] pushed "IMPORTER" to nats-server srv-4282: jwt updated 1339 | [ OK ] pushed "IMPORTER" to nats-server srv-4272: jwt updated 1340 | [ OK ] pushed "IMPORTER" to nats-server srv-4262: jwt updated 1341 | [ OK ] pushed "IMPORTER" to nats-server srv-4222: jwt updated 1342 | [ OK ] pushed "IMPORTER" to nats-server srv-4232: jwt updated 1343 | [ OK ] pushed "IMPORTER" to nats-server srv-4252: jwt updated 1344 | [ OK ] pushed "IMPORTER" to nats-server srv-4242: jwt updated 1345 | [ OK ] pushed "IMPORTER" to nats-server srv-4202: jwt updated 1346 | [ OK ] pushed "IMPORTER" to nats-server srv-4292: jwt updated 1347 | [ OK ] pushed to a total of 9 nats-server 1348 | [ OK ] push SYS to nats-server with nats account resolver: 1349 | [ OK ] pushed "SYS" to nats-server srv-4282: jwt updated 1350 | [ OK ] pushed "SYS" to nats-server srv-4272: jwt updated 1351 | [ OK ] pushed "SYS" to nats-server srv-4262: jwt updated 1352 | [ OK ] pushed "SYS" to nats-server srv-4232: jwt updated 1353 | [ OK ] pushed "SYS" to nats-server srv-4222: jwt updated 1354 | [ OK ] pushed "SYS" to nats-server srv-4202: jwt updated 1355 | [ OK ] pushed "SYS" to nats-server srv-4252: jwt updated 1356 | [ OK ] pushed "SYS" to nats-server srv-4292: jwt updated 1357 | [ OK ] pushed "SYS" to nats-server srv-4242: jwt updated 1358 | [ OK ] pushed to a total of 9 nats-server 1359 | [ OK ] push TEST to nats-server with nats account resolver: 1360 | [ OK ] pushed "TEST" to nats-server srv-4282: jwt updated 1361 | [ OK ] pushed "TEST" to nats-server srv-4272: jwt updated 1362 | [ OK ] pushed "TEST" to nats-server srv-4262: jwt updated 1363 | [ OK ] pushed "TEST" to nats-server srv-4232: jwt updated 1364 | [ OK ] pushed "TEST" to nats-server srv-4222: jwt updated 1365 | [ OK ] pushed "TEST" to nats-server srv-4242: jwt updated 1366 | [ OK ] pushed "TEST" to nats-server srv-4292: jwt updated 1367 | [ OK ] pushed "TEST" to nats-server srv-4252: jwt updated 1368 | [ OK ] pushed "TEST" to nats-server srv-4202: jwt updated 1369 | [ OK ] pushed to a total of 9 nats-server 1370 | ``` 1371 | 1372 | `nsc` also has a command to output account export/import relationships as plantuml file. 1373 | 1374 | ```txt 1375 | > nsc generate diagram component --detail --output-file account-component-diagram-cross.uml ; plantuml account-component-diagram-cross.uml 1376 | ``` 1377 | 1378 | We generate the diagram, process it with plantuml and show the generated png. 1379 | This usually gives a better overview of the relationships between accounts. 1380 | 1381 | ![`imgcat account-component-diagram-cross.png`](account-component-diagram-cross.png) 1382 | 1383 | When creating the mirror I have to import from a different account. 1384 | We also specify the prefix we set on import and add the stream name to our delivery subject. 1385 | 1386 | ```txt 1387 | > nats --context=hub --creds keys/creds/OP/IMPORTER/iuser.creds s add crossacc --mirror aggregate --replicas 3 1388 | ? Storage backend file 1389 | ? Retention Policy Limits 1390 | ? Discard Policy Old 1391 | ? Stream Messages Limit -1 1392 | ? Message size limit -1 1393 | ? Maximum message age limit -1 1394 | ? Maximum individual message size -1 1395 | ? Adjust mirror start No 1396 | ? Import mirror from a different JetStream domain No 1397 | ? Import mirror from a different account Yes 1398 | ? Foreign account API prefix JS.test@hub.API 1399 | ? Foreign account delivery prefix deliver.importer.crossacc 1400 | Stream crossacc was created 1401 | 1402 | Information for Stream crossacc created 2021-07-28T13:35:55-04:00 1403 | 1404 | Configuration: 1405 | 1406 | Acknowledgements: true 1407 | Retention: File - Limits 1408 | Replicas: 3 1409 | Discard Policy: Old 1410 | Duplicate Window: 2m0s 1411 | Maximum Messages: unlimited 1412 | Maximum Bytes: unlimited 1413 | Maximum Age: 0.00s 1414 | Maximum Message Size: unlimited 1415 | Maximum Consumers: unlimited 1416 | Mirror: aggregate, API Prefix: JS.test@hub.API, Delivery Prefix: deliver.importer.crossacc 1417 | 1418 | 1419 | Cluster Information: 1420 | 1421 | Name: cluster-hub 1422 | Leader: srv-4222 1423 | Replica: srv-4232, current, seen 0.00s ago 1424 | Replica: srv-4282, current, seen 0.00s ago 1425 | 1426 | Mirror Information: 1427 | 1428 | Stream Name: aggregate 1429 | Lag: 0 1430 | Last Seen: never 1431 | Ext. API Prefix: JS.test@hub.API 1432 | Ext. Delivery Prefix: deliver.importer.crossacc 1433 | 1434 | State: 1435 | 1436 | Messages: 0 1437 | Bytes: 0 B 1438 | FirstSeq: 0 1439 | LastSeq: 0 1440 | Active Consumers: 0 1441 | ``` 1442 | 1443 | Here you see, the new mirror already has all messages from before. 1444 | 1445 | ```txt 1446 | > nats --context=hub s report --creds keys/creds/OP/IMPORTER/iuser.creds 1447 | Obtaining Stream stats 1448 | 1449 | ╭──────────────────────────────────────────────────────────────────────────────────────────────────────╮ 1450 | │ Stream Report │ 1451 | ├──────────┬─────────┬───────────┬──────────┬─────────┬──────┬─────────┬───────────────────────────────┤ 1452 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 1453 | ├──────────┼─────────┼───────────┼──────────┼─────────┼──────┼─────────┼───────────────────────────────┤ 1454 | │ crossacc │ File │ 0 │ 70 │ 6.8 KiB │ 0 │ 0 │ srv-4222*, srv-4232, srv-4282 │ 1455 | ╰──────────┴─────────┴───────────┴──────────┴─────────┴──────┴─────────┴───────────────────────────────╯ 1456 | 1457 | ╭────────────────────────────────────────────────────────────────────────────╮ 1458 | │ Replication Report │ 1459 | ├──────────┬────────┬─────────────────┬───────────────┬────────┬─────┬───────┤ 1460 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 1461 | ├──────────┼────────┼─────────────────┼───────────────┼────────┼─────┼───────┤ 1462 | │ crossacc │ Mirror │ JS.test@hub.API │ aggregate │ 1.44s │ 0 │ │ 1463 | ╰──────────┴────────┴─────────────────┴───────────────┴────────┴─────┴───────╯ 1464 | ``` 1465 | 1466 | The api prefix is what we changed the api on import to. 1467 | This is so we can differentiate between our JetStream and the JetStream in the other account (in possibly the same domain). 1468 | For the delivery prefix we need to add the stream name. 1469 | Just consider, what if I wanted to mirror the same stream twice. 1470 | This is where the stream name helps to differentiate. 1471 | 1472 | While we recommend exchanging stream data via source and mirror, I have to show you how to share direct access of a durable pull consumer as well. 1473 | 1474 | ![`imgcat topology8-consume-cross-account.png`](topology8-consume-cross-account.png) 1475 | 1476 | Since exports in `TEST` exist already, I briefly clean them up to avoid warnings about overlapping subjects. 1477 | 1478 | ```txt 1479 | > nsc delete export --account TEST --subject '$JS.hub.API.CONSUMER.>' 1480 | [ OK ] deleted service export "$JS.hub.API.CONSUMER.>" 1481 | > nsc delete import --account IMPORTER --subject '$JS.hub.API.CONSUMER.>' 1482 | [ OK ] deleted service import "$JS.hub.API.CONSUMER.>" 1483 | ``` 1484 | 1485 | What needs to be exported as service responding with a stream is the consumer's NEXT subject 1486 | 1487 | ```txt 1488 | > nsc add export --account TEST --name Next-API --service --response-type Stream --subject '$JS.hub.API.CONSUMER.MSG.NEXT.aggregate.DUR' 1489 | [ OK ] added public service export "Next-API" 1490 | ``` 1491 | The `NEXT` subject consists of the Prefix (with domain), consumer message next, stream name and durable consumer name. 1492 | 1493 | On import we rename `$JS.hub.API` to a different prefix, say `from.test.API`. 1494 | 1495 | ```txt 1496 | > nsc add import --account IMPORTER --name Remote-Next-API --src-account TEST --remote-subject '$JS.hub.API.CONSUMER.MSG.NEXT.aggregate.DUR' --local-subject 'from.test.API.CONSUMER.MSG.NEXT.aggregate.DUR' --service 1497 | [ OK ] added service import "$JS.hub.API.CONSUMER.MSG.NEXT.aggregate.DUR" 1498 | ``` 1499 | 1500 | To acknowledge messages, the ack api needs to be exported/imported as well. We do so without name changes. 1501 | 1502 | ```txt 1503 | > nsc add export --account TEST --name Ack-API --service --response-type Stream --subject '$JS.ACK.aggregate.DUR.>' 1504 | [ OK ] added public service export "Ack-API" 1505 | > nsc add import --account IMPORTER --name Remote-Ack-API --src-account TEST --remote-subject '$JS.ACK.aggregate.DUR.>' --service 1506 | [ OK ] added service import "$JS.ACK.aggregate.DUR.>" 1507 | ``` 1508 | 1509 | And upload the changes: 1510 | 1511 | ```txt 1512 | > nsc push -A 1513 | [ OK ] push to nats-server "nats://localhost:4222,nats://localhost:4232,nats://localhost:4282" using system account "SYS": 1514 | [ OK ] push IMPORTER to nats-server with nats account resolver: 1515 | [ OK ] pushed "IMPORTER" to nats-server srv-4222: jwt updated 1516 | [ OK ] pushed "IMPORTER" to nats-server srv-4202: jwt updated 1517 | [ OK ] pushed "IMPORTER" to nats-server srv-4232: jwt updated 1518 | [ OK ] pushed "IMPORTER" to nats-server srv-4282: jwt updated 1519 | [ OK ] pushed "IMPORTER" to nats-server srv-4242: jwt updated 1520 | [ OK ] pushed "IMPORTER" to nats-server srv-4252: jwt updated 1521 | [ OK ] pushed "IMPORTER" to nats-server srv-4292: jwt updated 1522 | [ OK ] pushed "IMPORTER" to nats-server srv-4272: jwt updated 1523 | [ OK ] pushed "IMPORTER" to nats-server srv-4262: jwt updated 1524 | [ OK ] pushed to a total of 9 nats-server 1525 | [ OK ] push SYS to nats-server with nats account resolver: 1526 | [ OK ] pushed "SYS" to nats-server srv-4222: jwt updated 1527 | [ OK ] pushed "SYS" to nats-server srv-4202: jwt updated 1528 | [ OK ] pushed "SYS" to nats-server srv-4282: jwt updated 1529 | [ OK ] pushed "SYS" to nats-server srv-4232: jwt updated 1530 | [ OK ] pushed "SYS" to nats-server srv-4242: jwt updated 1531 | [ OK ] pushed "SYS" to nats-server srv-4262: jwt updated 1532 | [ OK ] pushed "SYS" to nats-server srv-4252: jwt updated 1533 | [ OK ] pushed "SYS" to nats-server srv-4272: jwt updated 1534 | [ OK ] pushed "SYS" to nats-server srv-4292: jwt updated 1535 | [ OK ] pushed to a total of 9 nats-server 1536 | [ OK ] push TEST to nats-server with nats account resolver: 1537 | [ OK ] pushed "TEST" to nats-server srv-4222: jwt updated 1538 | [ OK ] pushed "TEST" to nats-server srv-4202: jwt updated 1539 | [ OK ] pushed "TEST" to nats-server srv-4232: jwt updated 1540 | [ OK ] pushed "TEST" to nats-server srv-4282: jwt updated 1541 | [ OK ] pushed "TEST" to nats-server srv-4292: jwt updated 1542 | [ OK ] pushed "TEST" to nats-server srv-4242: jwt updated 1543 | [ OK ] pushed "TEST" to nats-server srv-4252: jwt updated 1544 | [ OK ] pushed "TEST" to nats-server srv-4272: jwt updated 1545 | [ OK ] pushed "TEST" to nats-server srv-4262: jwt updated 1546 | [ OK ] pushed to a total of 9 nats-server 1547 | ``` 1548 | 1549 | Create the consumer DUR that we already referenced in our exports/imports. 1550 | Consumer add, stream name, durable consumer name, type pull consumer, deliver all messages in the stream: 1551 | 1552 | ```txt 1553 | > nats --context=hub c add aggregate DUR --pull --deliver all 1554 | ? Replay policy instant 1555 | ? Filter Stream by subject (blank for all) 1556 | ? Maximum Allowed Deliveries -1 1557 | ? Maximum Acknowledgements Pending 0 1558 | Information for Consumer aggregate > DUR created 2021-07-28T13:56:16-04:00 1559 | 1560 | Configuration: 1561 | 1562 | Durable Name: DUR 1563 | Pull Mode: true 1564 | Deliver All: true 1565 | Ack Policy: Explicit 1566 | Ack Wait: 30s 1567 | Replay Policy: Instant 1568 | Max Ack Pending: 20,000 1569 | Max Waiting Pulls: 512 1570 | 1571 | Cluster Information: 1572 | 1573 | Name: cluster-hub 1574 | Leader: srv-4282 1575 | Replica: srv-4232, current, seen 0.00s ago 1576 | Replica: srv-4222, current, seen 0.00s ago 1577 | 1578 | State: 1579 | 1580 | Last Delivered Message: Consumer sequence: 0 Stream sequence: 0 1581 | Acknowledgment floor: Consumer sequence: 0 Stream sequence: 0 1582 | Outstanding Acks: 0 out of maximum 20000 1583 | Redelivered Messages: 0 1584 | Unprocessed Messages: 70 1585 | Waiting Pulls: 0 of maximum 512 1586 | 1587 | ``` 1588 | 1589 | To consume I am overwriting the credentials specified in the context. 1590 | This is the user we created just now, consumer, next stream, durable, and this is the important bit, `js-api-prefix` 1591 | For that We use the api prefix `from.test.API` that we set on import. 1592 | 1593 | And now we get our first message: 1594 | 1595 | ```txt 1596 | > nats --context=hub --creds keys/creds/OP/IMPORTER/iuser.creds consumer next aggregate DUR --js-api-prefix=from.test.API 1597 | [13:59:04] subj: test / tries: 1 / cons seq: 1 / str seq: 1 / pending: 69 1598 | 1599 | Headers: 1600 | 1601 | Nats-Stream-Source: test:J3WG6St1 1 1602 | 1603 | Data: 1604 | 1605 | 1606 | hello world 1607 | 1608 | Acknowledged message 1609 | 1610 | ``` 1611 | 1612 | Let's briefly look at what would be necessary to do in a program. 1613 | 1614 | ```txt 1615 | > nl -b a main.go 1616 | 1 package main 1617 | 2 1618 | 3 import ( 1619 | 4 "fmt" 1620 | 5 "os" 1621 | 6 "os/signal" 1622 | 7 "syscall" 1623 | 8 "time" 1624 | 9 1625 | 10 "github.com/nats-io/nats.go" 1626 | 11 ) 1627 | 12 1628 | 13 func main() { 1629 | 14 nc, err := nats.Connect(os.Args[1], nats.Name("JS sub test"), nats.UserCredentials(os.Args[2])) 1630 | 15 defer nc.Close() 1631 | 16 if err != nil { 1632 | 17 fmt.Printf("nats connect: %v\n", err) 1633 | 18 return 1634 | 19 } 1635 | 20 js, err := nc.JetStream(nats.APIPrefix("from.test.API")) 1636 | 21 if err != nil { 1637 | 22 fmt.Printf("JetStream: %v\n", err) 1638 | 23 if js == nil { 1639 | 24 return 1640 | 25 } 1641 | 26 } 1642 | 27 s, err := js.PullSubscribe("test", "DUR", nats.Bind("aggregate", "DUR")) 1643 | 28 if err != nil { 1644 | 29 fmt.Printf("PullSubscribe: %v\n", err) 1645 | 30 return 1646 | 31 } 1647 | 32 1648 | 33 shutdown := make(chan os.Signal, 1) 1649 | 34 signal.Notify(shutdown, os.Interrupt, syscall.SIGTERM) 1650 | 35 1651 | 36 fmt.Printf("starting\n") 1652 | 37 for { 1653 | 38 select { 1654 | 39 case <-shutdown: 1655 | 40 return 1656 | 41 default: 1657 | 42 if m, err := s.Fetch(1, nats.MaxWait(time.Second)); err != nil { 1658 | 43 fmt.Println(err) 1659 | 44 } else { 1660 | 45 1661 | 46 if meta, err := m[0].Metadata(); err == nil { 1662 | 47 fmt.Printf("%+v\n", meta) 1663 | 48 } 1664 | 49 fmt.Println(string(m[0].Data)) 1665 | 50 1666 | 51 if err := m[0].Ack(); err != nil { 1667 | 52 fmt.Printf("ack error: %+v\n", err) 1668 | 53 } 1669 | 54 } 1670 | 55 } 1671 | 56 } 1672 | 57 } 1673 | ``` 1674 | 1675 | Similar to the cli, we specify `nats.APIPrefix` in line `20`. 1676 | Due to very specific export, this program has only limited JS API access. 1677 | Therefore `nats.Bind` in line `27` provides stream and durable name explicitly. 1678 | 1679 | Let's run it, connecting to the hub and using our new user. 1680 | 1681 | ```txt 1682 | > ./main localhost:4222 keys/creds/OP/IMPORTER/iuser.creds 1683 | starting 1684 | &{Sequence:{Consumer:3 Stream:3} NumDelivered:1 NumPending:67 Timestamp:2021-07-28 12:48:52.466147 -0400 EDT Stream:aggregate Consumer:DUR} 1685 | hello world 1686 | &{Sequence:{Consumer:4 Stream:4} NumDelivered:1 NumPending:66 Timestamp:2021-07-28 12:48:52.46615 -0400 EDT Stream:aggregate Consumer:DUR} 1687 | hello world 1688 | &{Sequence:{Consumer:5 Stream:5} NumDelivered:1 NumPending:65 Timestamp:2021-07-28 12:48:52.466152 -0400 EDT Stream:aggregate Consumer:DUR} 1689 | hello world 1690 | ... 1691 | &{Sequence:{Consumer:68 Stream:68} NumDelivered:1 NumPending:2 Timestamp:2021-07-28 13:11:10.789185 -0400 EDT Stream:aggregate Consumer:DUR} 1692 | hello world 1693 | &{Sequence:{Consumer:69 Stream:69} NumDelivered:1 NumPending:1 Timestamp:2021-07-28 13:11:10.789186 -0400 EDT Stream:aggregate Consumer:DUR} 1694 | hello world 1695 | &{Sequence:{Consumer:70 Stream:70} NumDelivered:1 NumPending:0 Timestamp:2021-07-28 13:11:10.789188 -0400 EDT Stream:aggregate Consumer:DUR} 1696 | hello world 1697 | nats: timeout 1698 | ^Cnats: timeout 1699 | ``` 1700 | 1701 | There you go, messages are directly received from a durable in another account. 1702 | 1703 | #### Connect Streams Through Accounts 1704 | 1705 | Finally I want to show you an account setup that addresses the same subject issue mentioned earlier. 1706 | You will be able to use the same account on each leafnode and isolate it such that you don't have to worry about cross traffic. 1707 | This will clearly increase the complexity of your setup, so only do that if it otherwise makes your life simpler. 1708 | 1709 | ![`imgcat topology9-source-cross-domain-account.png`](topology9-source-cross-domain-account.png) 1710 | 1711 | Essentially we will selectively connect subjects in our accounts through a dedicated exchange account. 1712 | We will be needing a name that identifies a leaf node or leafnode cluster uniquely. 1713 | As I am demonstrating JetStream I am using domain names for that. 1714 | However you are not limited to JetStream and any of your subject can be connected in a similar way. 1715 | This image shows the direction leaf to hub, but the setup I am showing next only needs two extra imports to enable the other direction as well. 1716 | So, let's create these accounts: 1717 | 1718 | I'm creating the exchange account and a user. (No JetStream on purpose): 1719 | 1720 | ```txt 1721 | > nsc add account -n EXCACC 1722 | [ OK ] generated and stored account key "ACSESHB7CAGJ6R5ITCOO5GXZKA7JZ6J2BRUIIH2LDSGZSH26C3YO6N64" 1723 | [ OK ] added account "EXCACC" 1724 | > nsc add user --account EXCACC --name exp 1725 | [ OK ] generated and stored user key "UDDE7EROVEJ37LZD2RU4Z56N5X6Z45BZ7GEOTIUTI7V4AVG5LP27TDQZ" 1726 | [ OK ] generated user creds file `~/test/jetstream-leaf-nodes-demo/keys/creds/OP/EXCACC/exp.creds` 1727 | [ OK ] added user "exp" to account "EXCACC" 1728 | ``` 1729 | 1730 | Leaf account, with JetStream enabled and a user: 1731 | 1732 | ```txt 1733 | > nsc add account -n LEAFACC 1734 | [ OK ] generated and stored account key "ADDIPFFGFZNLMR4OSCWX2RHBHTZBBEEV7THVWKBNJ4M7L6TDJ4YAXHY6" 1735 | [ OK ] added account "LEAFACC" 1736 | > nsc edit account -n LEAFACC --js-disk-storage -1 --js-consumer -1 --js-streams -1 1737 | [ OK ] edited account "LEAFACC" 1738 | > nsc add user --account LEAFACC --name exp 1739 | [ OK ] generated and stored user key "UCPEVXUHMULNSEE4OZ7YS4EHLYT573CREUHJ57WDZNIMJS43WS5U4GU2" 1740 | [ OK ] generated user creds file `~/test/jetstream-leaf-nodes-demo/keys/creds/OP/LEAFACC/exp.creds` 1741 | [ OK ] added user "exp" to account "LEAFACC" 1742 | ``` 1743 | 1744 | Hub account, with JetStream enabled and a user: 1745 | 1746 | ```txt 1747 | > nsc add account -n HUBACC 1748 | [ OK ] generated and stored account key "AC25IY7ID2R6VNGHFROBD75EFCS7LTE3G52YRA6BPYIJDW7RVHX6IEBY" 1749 | [ OK ] added account "HUBACC" 1750 | > nsc edit account -n HUBACC --js-disk-storage -1 --js-consumer -1 --js-streams -1 1751 | [ OK ] edited account "HUBACC" 1752 | > nsc add user --account HUBACC --name imp 1753 | [ OK ] generated and stored user key "UANIPWE67ANES3FLPJMD626TL62BYSL2U3L2YJ5PWNZ76BHPXBPUFSGR" 1754 | [ OK ] generated user creds file `~/test/jetstream-leaf-nodes-demo/keys/creds/OP/HUBACC/imp.creds` 1755 | [ OK ] added user "imp" to account "HUBACC" 1756 | ``` 1757 | 1758 | And pushing all accounts: 1759 | 1760 | ```txt 1761 | > nsc push -A 1762 | [ OK ] push to nats-server "nats://localhost:4222,nats://localhost:4232,nats://localhost:4282" using system account "SYS": 1763 | [ OK ] push EXCACC to nats-server with nats account resolver: 1764 | [ OK ] pushed "EXCACC" to nats-server srv-4222: jwt updated 1765 | [ OK ] pushed "EXCACC" to nats-server srv-4202: jwt updated 1766 | [ OK ] pushed "EXCACC" to nats-server srv-4282: jwt updated 1767 | [ OK ] pushed "EXCACC" to nats-server srv-4232: jwt updated 1768 | [ OK ] pushed "EXCACC" to nats-server srv-4242: jwt updated 1769 | [ OK ] pushed "EXCACC" to nats-server srv-4262: jwt updated 1770 | [ OK ] pushed "EXCACC" to nats-server srv-4292: jwt updated 1771 | [ OK ] pushed "EXCACC" to nats-server srv-4272: jwt updated 1772 | [ OK ] pushed "EXCACC" to nats-server srv-4252: jwt updated 1773 | [ OK ] pushed to a total of 9 nats-server 1774 | [ OK ] push HUBACC to nats-server with nats account resolver: 1775 | [ OK ] pushed "HUBACC" to nats-server srv-4222: jwt updated 1776 | [ OK ] pushed "HUBACC" to nats-server srv-4282: jwt updated 1777 | [ OK ] pushed "HUBACC" to nats-server srv-4232: jwt updated 1778 | [ OK ] pushed "HUBACC" to nats-server srv-4202: jwt updated 1779 | [ OK ] pushed "HUBACC" to nats-server srv-4262: jwt updated 1780 | [ OK ] pushed "HUBACC" to nats-server srv-4292: jwt updated 1781 | [ OK ] pushed "HUBACC" to nats-server srv-4242: jwt updated 1782 | [ OK ] pushed "HUBACC" to nats-server srv-4272: jwt updated 1783 | [ OK ] pushed "HUBACC" to nats-server srv-4252: jwt updated 1784 | [ OK ] pushed to a total of 9 nats-server 1785 | [ OK ] push IMPORTER to nats-server with nats account resolver: 1786 | [ OK ] pushed "IMPORTER" to nats-server srv-4222: jwt updated 1787 | [ OK ] pushed "IMPORTER" to nats-server srv-4202: jwt updated 1788 | [ OK ] pushed "IMPORTER" to nats-server srv-4282: jwt updated 1789 | [ OK ] pushed "IMPORTER" to nats-server srv-4232: jwt updated 1790 | [ OK ] pushed "IMPORTER" to nats-server srv-4272: jwt updated 1791 | [ OK ] pushed "IMPORTER" to nats-server srv-4242: jwt updated 1792 | [ OK ] pushed "IMPORTER" to nats-server srv-4262: jwt updated 1793 | [ OK ] pushed "IMPORTER" to nats-server srv-4252: jwt updated 1794 | [ OK ] pushed "IMPORTER" to nats-server srv-4292: jwt updated 1795 | [ OK ] pushed to a total of 9 nats-server 1796 | [ OK ] push LEAFACC to nats-server with nats account resolver: 1797 | [ OK ] pushed "LEAFACC" to nats-server srv-4222: jwt updated 1798 | [ OK ] pushed "LEAFACC" to nats-server srv-4232: jwt updated 1799 | [ OK ] pushed "LEAFACC" to nats-server srv-4202: jwt updated 1800 | [ OK ] pushed "LEAFACC" to nats-server srv-4282: jwt updated 1801 | [ OK ] pushed "LEAFACC" to nats-server srv-4292: jwt updated 1802 | [ OK ] pushed "LEAFACC" to nats-server srv-4252: jwt updated 1803 | [ OK ] pushed "LEAFACC" to nats-server srv-4262: jwt updated 1804 | [ OK ] pushed "LEAFACC" to nats-server srv-4242: jwt updated 1805 | [ OK ] pushed "LEAFACC" to nats-server srv-4272: jwt updated 1806 | [ OK ] pushed to a total of 9 nats-server 1807 | [ OK ] push SYS to nats-server with nats account resolver: 1808 | [ OK ] pushed "SYS" to nats-server srv-4222: jwt updated 1809 | [ OK ] pushed "SYS" to nats-server srv-4282: jwt updated 1810 | [ OK ] pushed "SYS" to nats-server srv-4202: jwt updated 1811 | [ OK ] pushed "SYS" to nats-server srv-4232: jwt updated 1812 | [ OK ] pushed "SYS" to nats-server srv-4262: jwt updated 1813 | [ OK ] pushed "SYS" to nats-server srv-4272: jwt updated 1814 | [ OK ] pushed "SYS" to nats-server srv-4242: jwt updated 1815 | [ OK ] pushed "SYS" to nats-server srv-4252: jwt updated 1816 | [ OK ] pushed "SYS" to nats-server srv-4292: jwt updated 1817 | [ OK ] pushed to a total of 9 nats-server 1818 | [ OK ] push TEST to nats-server with nats account resolver: 1819 | [ OK ] pushed "TEST" to nats-server srv-4222: jwt updated 1820 | [ OK ] pushed "TEST" to nats-server srv-4202: jwt updated 1821 | [ OK ] pushed "TEST" to nats-server srv-4232: jwt updated 1822 | [ OK ] pushed "TEST" to nats-server srv-4282: jwt updated 1823 | [ OK ] pushed "TEST" to nats-server srv-4242: jwt updated 1824 | [ OK ] pushed "TEST" to nats-server srv-4292: jwt updated 1825 | [ OK ] pushed "TEST" to nats-server srv-4252: jwt updated 1826 | [ OK ] pushed "TEST" to nats-server srv-4262: jwt updated 1827 | [ OK ] pushed "TEST" to nats-server srv-4272: jwt updated 1828 | [ OK ] pushed to a total of 9 nats-server 1829 | ``` 1830 | 1831 | Let's briefly change our watch command on the left to make use of our new users. 1832 | I'm connecting to the hub, hubaccount user, stream report. 1833 | The spoke, leaf account user, stream report. 1834 | The second spoke is the same as first. 1835 | 1836 | ```txt 1837 | > watch -n 1 "nats --context=hub --creds keys/creds/OP/HUBACC/imp.creds s report ; \ 1838 | nats --context=spoke-1 --creds keys/creds/OP/LEAFACC/exp.creds s report ; \ 1839 | nats --context=spoke-2 --creds keys/creds/OP/LEAFACC/exp.creds s report" 1840 | 1841 | Obtaining Stream stats 1842 | 1843 | No Streams defined 1844 | Obtaining Stream stats 1845 | 1846 | No Streams defined 1847 | Obtaining Stream stats 1848 | 1849 | No Streams defined 1850 | 1851 | ``` 1852 | 1853 | We have nothing defined yet, which is why they are empty. 1854 | 1855 | Furthermore, ONLY put that exchange account `EXCACC` into the remotes. 1856 | Not listing the other accounts is what isolates them from the hub and each other! 1857 | I have looked up the account id earlier using `nsc list keys`. 1858 | Register account EXCACC as remote: 1859 | 1860 | ```txt 1861 | > nl -b a leafnode-remotes.cfg 1862 | 1 HUB-URLS=["nats-leaf://localhost:4224","nats-leaf://localhost:4234","nats-leaf://localhost:4284"] 1863 | 2 leafnodes { 1864 | 3 no_advertise: true 1865 | 4 remotes = [ 1866 | 5 { 1867 | 6 urls: $HUB-URLS 1868 | 7 account: ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4 1869 | 8 credentials: keys/creds/OP/SYS/sys.creds 1870 | 9 }, 1871 | 10 { 1872 | 11 urls: $HUB-URLS 1873 | 12 account: AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G 1874 | 13 credentials: keys/creds/OP/TEST/leaf.creds 1875 | 14 }, 1876 | 15 { 1877 | 16 urls: $HUB-URLS 1878 | 17 account: ACSESHB7CAGJ6R5ITCOO5GXZKA7JZ6J2BRUIIH2LDSGZSH26C3YO6N64 1879 | 18 credentials: keys/creds/OP/EXCACC/exp.creds 1880 | 19 }, 1881 | 20 ] 1882 | 21 } 1883 | ``` 1884 | 1885 | A side effect of only having the exchange account in the remotes is that you can control access via exports and imports that can be pushed by `nsc`. 1886 | You do not have to touch remotes in a config going forward. 1887 | To demonstrate this I pushed accounts without any exports and imports already. 1888 | Right now I have to restart nats-server such that the change is picked up. 1889 | 1890 | ```txt 1891 | > pkill nats-server 1892 | ``` 1893 | 1894 | Now we are connecting the accounts. 1895 | 1896 | Selectively, we export subjects we want to be able to send to and receive from in other accounts and domains. 1897 | For every domain and account I wish to connect, I'm exporting the entire domain specific JetStream API as well as a delivery subject. 1898 | You can also add dedicated subjects to communicate with regular nats. 1899 | 1900 | For every account, I export a delivery subject containing account name and domain as well the JetStream domain specific API. 1901 | 1902 | ```txt 1903 | > nsc add export --account HUBACC --service --response-type Stream --subject '$JS.hub.API.>' 1904 | [ OK ] added public service export "$JS.hub.API.>" 1905 | > nsc add export --account LEAFACC --service --response-type Stream --subject '$JS.spoke-1.API.>' 1906 | [ OK ] added public service export "$JS.spoke-1.API.>" 1907 | > nsc add export --account LEAFACC --service --response-type Stream --subject '$JS.spoke-2.API.>' 1908 | [ OK ] added public service export "$JS.spoke-2.API.>" 1909 | > nsc add export --account HUBACC --subject 'deliver.hubacc.hub.>' 1910 | [ OK ] added public stream export "deliver.hubacc.hub.>" 1911 | > nsc add export --account LEAFACC --subject 'deliver.leafacc.spoke-1.>' 1912 | [ OK ] added public stream export "deliver.leafacc.spoke-1.>" 1913 | > nsc add export --account LEAFACC --subject 'deliver.leafacc.spoke-2.>' 1914 | [ OK ] added public stream export "deliver.leafacc.spoke-2.>" 1915 | ``` 1916 | 1917 | All these exports are then imported into the exchange account. 1918 | 1919 | Please note that every subject already contains, a fixed portion, that functions as type, the exporting account name as well as the domain. 1920 | In that order! 1921 | 1922 | Where this is not the case, such as the JS API, i remap accordingly. 1923 | This allows us to properly identify streams and services in an account in a leafnode as identified by it's domain. 1924 | 1925 | ```txt 1926 | > nsc add import --account EXCACC --src-account HUBACC --service --remote-subject '$JS.hub.API.>' --local-subject '$JS.hubacc.hub.API.>' 1927 | [ OK ] added service import "$JS.hub.API.>" 1928 | > nsc add import --account EXCACC --src-account LEAFACC --service --remote-subject '$JS.spoke-1.API.>' --local-subject '$JS.leafacc.spoke-1.API.>' 1929 | [ OK ] added service import "$JS.spoke-1.API.>" 1930 | > nsc add import --account EXCACC --src-account LEAFACC --service --remote-subject '$JS.spoke-2.API.>' --local-subject '$JS.leafacc.spoke-2.API.>' 1931 | [ OK ] added service import "$JS.spoke-2.API.>" 1932 | > nsc add import --account EXCACC --src-account HUBACC --remote-subject 'deliver.hubacc.hub.>' 1933 | [ OK ] added stream import "deliver.hubacc.hub.>" 1934 | > nsc add import --account EXCACC --src-account LEAFACC --remote-subject 'deliver.leafacc.spoke-1.>' 1935 | [ OK ] added stream import "deliver.leafacc.spoke-1.>" 1936 | > nsc add import --account EXCACC --src-account LEAFACC --remote-subject 'deliver.leafacc.spoke-2.>' 1937 | [ OK ] added stream import "deliver.leafacc.spoke-2.>" 1938 | ``` 1939 | 1940 | Then we re-export all subjects together. This is where the type comes in handy with `type.>`. 1941 | 1942 | ```txt 1943 | > nsc add export --account EXCACC --service --response-type Stream --subject '$JS.>' 1944 | [ OK ] added public service export "$JS.>" 1945 | > nsc add export --account EXCACC --subject 'deliver.>' 1946 | [ OK ] added public stream export "deliver.>" 1947 | ``` 1948 | 1949 | Finally import into every account you want connected. 1950 | I'm more specific than in the previous export as I need to avoid a self import cycle with say `deliver.hubacc.>`. 1951 | If you want to be more specific and pin an import to a domain, just add it. 1952 | 1953 | ```txt 1954 | > nsc add import --account HUBACC --src-account EXCACC --service --remote-subject '$JS.leafacc.>' 1955 | [ OK ] added service import "$JS.leafacc.>" 1956 | > nsc add import --account HUBACC --src-account EXCACC --remote-subject 'deliver.leafacc.>' 1957 | [ OK ] added stream import "deliver.leafacc.>" 1958 | ``` 1959 | 1960 | Inspect the changes 1961 | 1962 | ```txt 1963 | > nsc generate diagram component --detail --output-file account-component-diagram-through.uml ; plantuml account-component-diagram-through.uml 1964 | ``` 1965 | 1966 | ![`imgcat account-component-diagram-through.png`](account-component-diagram-through.png) 1967 | 1968 | Upload all changes: 1969 | 1970 | ```txt 1971 | > nsc push -A 1972 | [ OK ] push to nats-server "nats://localhost:4222,nats://localhost:4232,nats://localhost:4282" using system account "SYS": 1973 | [ OK ] push EXCACC to nats-server with nats account resolver: 1974 | [ OK ] pushed "EXCACC" to nats-server srv-4282: jwt updated 1975 | [ OK ] pushed "EXCACC" to nats-server srv-4232: jwt updated 1976 | [ OK ] pushed "EXCACC" to nats-server srv-4222: jwt updated 1977 | [ OK ] pushed "EXCACC" to nats-server srv-4272: jwt updated 1978 | [ OK ] pushed "EXCACC" to nats-server srv-4202: jwt updated 1979 | [ OK ] pushed "EXCACC" to nats-server srv-4252: jwt updated 1980 | [ OK ] pushed "EXCACC" to nats-server srv-4262: jwt updated 1981 | [ OK ] pushed "EXCACC" to nats-server srv-4292: jwt updated 1982 | [ OK ] pushed "EXCACC" to nats-server srv-4242: jwt updated 1983 | [ OK ] pushed to a total of 9 nats-server 1984 | [ OK ] push HUBACC to nats-server with nats account resolver: 1985 | [ OK ] pushed "HUBACC" to nats-server srv-4282: jwt updated 1986 | [ OK ] pushed "HUBACC" to nats-server srv-4232: jwt updated 1987 | [ OK ] pushed "HUBACC" to nats-server srv-4222: jwt updated 1988 | [ OK ] pushed "HUBACC" to nats-server srv-4202: jwt updated 1989 | [ OK ] pushed "HUBACC" to nats-server srv-4262: jwt updated 1990 | [ OK ] pushed "HUBACC" to nats-server srv-4272: jwt updated 1991 | [ OK ] pushed "HUBACC" to nats-server srv-4252: jwt updated 1992 | [ OK ] pushed "HUBACC" to nats-server srv-4292: jwt updated 1993 | [ OK ] pushed "HUBACC" to nats-server srv-4242: jwt updated 1994 | [ OK ] pushed to a total of 9 nats-server 1995 | [ OK ] push IMPORTER to nats-server with nats account resolver: 1996 | [ OK ] pushed "IMPORTER" to nats-server srv-4282: jwt updated 1997 | [ OK ] pushed "IMPORTER" to nats-server srv-4222: jwt updated 1998 | [ OK ] pushed "IMPORTER" to nats-server srv-4232: jwt updated 1999 | [ OK ] pushed "IMPORTER" to nats-server srv-4242: jwt updated 2000 | [ OK ] pushed "IMPORTER" to nats-server srv-4292: jwt updated 2001 | [ OK ] pushed "IMPORTER" to nats-server srv-4262: jwt updated 2002 | [ OK ] pushed "IMPORTER" to nats-server srv-4252: jwt updated 2003 | [ OK ] pushed "IMPORTER" to nats-server srv-4272: jwt updated 2004 | [ OK ] pushed "IMPORTER" to nats-server srv-4202: jwt updated 2005 | [ OK ] pushed to a total of 9 nats-server 2006 | [ OK ] push LEAFACC to nats-server with nats account resolver: 2007 | [ OK ] pushed "LEAFACC" to nats-server srv-4282: jwt updated 2008 | [ OK ] pushed "LEAFACC" to nats-server srv-4232: jwt updated 2009 | [ OK ] pushed "LEAFACC" to nats-server srv-4222: jwt updated 2010 | [ OK ] pushed "LEAFACC" to nats-server srv-4272: jwt updated 2011 | [ OK ] pushed "LEAFACC" to nats-server srv-4252: jwt updated 2012 | [ OK ] pushed "LEAFACC" to nats-server srv-4202: jwt updated 2013 | [ OK ] pushed "LEAFACC" to nats-server srv-4262: jwt updated 2014 | [ OK ] pushed "LEAFACC" to nats-server srv-4292: jwt updated 2015 | [ OK ] pushed "LEAFACC" to nats-server srv-4242: jwt updated 2016 | [ OK ] pushed to a total of 9 nats-server 2017 | [ OK ] push SYS to nats-server with nats account resolver: 2018 | [ OK ] pushed "SYS" to nats-server srv-4282: jwt updated 2019 | [ OK ] pushed "SYS" to nats-server srv-4222: jwt updated 2020 | [ OK ] pushed "SYS" to nats-server srv-4232: jwt updated 2021 | [ OK ] pushed "SYS" to nats-server srv-4292: jwt updated 2022 | [ OK ] pushed "SYS" to nats-server srv-4242: jwt updated 2023 | [ OK ] pushed "SYS" to nats-server srv-4202: jwt updated 2024 | [ OK ] pushed "SYS" to nats-server srv-4252: jwt updated 2025 | [ OK ] pushed "SYS" to nats-server srv-4272: jwt updated 2026 | [ OK ] pushed "SYS" to nats-server srv-4262: jwt updated 2027 | [ OK ] pushed to a total of 9 nats-server 2028 | [ OK ] push TEST to nats-server with nats account resolver: 2029 | [ OK ] pushed "TEST" to nats-server srv-4282: jwt updated 2030 | [ OK ] pushed "TEST" to nats-server srv-4222: jwt updated 2031 | [ OK ] pushed "TEST" to nats-server srv-4232: jwt updated 2032 | [ OK ] pushed "TEST" to nats-server srv-4292: jwt updated 2033 | [ OK ] pushed "TEST" to nats-server srv-4242: jwt updated 2034 | [ OK ] pushed "TEST" to nats-server srv-4272: jwt updated 2035 | [ OK ] pushed "TEST" to nats-server srv-4202: jwt updated 2036 | [ OK ] pushed "TEST" to nats-server srv-4252: jwt updated 2037 | [ OK ] pushed "TEST" to nats-server srv-4262: jwt updated 2038 | [ OK ] pushed to a total of 9 nats-server 2039 | ``` 2040 | 2041 | Then we create a stream in each leaf domain. 2042 | Using the stream configuration test from earlier. 2043 | Because it is possible, I do so by connecting to the `hub`, using hub account credentials. 2044 | A domain essentially means we use the prefix `$JS..API` 2045 | 2046 | When connected to the hub, because I imported `$JS.leafacc.spoke-1.API` and I neglected to avoid `$JS`, I can use the domain name `leafacc.spoke-1` in the `nats` cli. 2047 | 2048 | ```txt 2049 | > nats --context=hub --creds keys/creds/OP/HUBACC/imp.creds s add --config test --js-domain leafacc.spoke-1 2050 | Stream test was created 2051 | 2052 | Information for Stream test created 2021-07-28T14:49:29-04:00 2053 | 2054 | Configuration: 2055 | 2056 | Subjects: test 2057 | Acknowledgements: true 2058 | Retention: File - Limits 2059 | Replicas: 3 2060 | Discard Policy: Old 2061 | Duplicate Window: 2m0s 2062 | Maximum Messages: unlimited 2063 | Maximum Bytes: unlimited 2064 | Maximum Age: 0.00s 2065 | Maximum Message Size: unlimited 2066 | Maximum Consumers: unlimited 2067 | 2068 | 2069 | Cluster Information: 2070 | 2071 | Name: cluster-spoke-1 2072 | Leader: srv-4242 2073 | Replica: srv-4252, current, seen 0.00s ago 2074 | Replica: srv-4292, current, seen 0.00s ago 2075 | 2076 | State: 2077 | 2078 | Messages: 0 2079 | Bytes: 0 B 2080 | FirstSeq: 0 2081 | LastSeq: 0 2082 | Active Consumers: 0 2083 | > nats --context=hub --creds keys/creds/OP/HUBACC/imp.creds s add --config test --js-domain leafacc.spoke-2 2084 | Stream test was created 2085 | 2086 | Information for Stream test created 2021-07-28T14:49:37-04:00 2087 | 2088 | Configuration: 2089 | 2090 | Subjects: test 2091 | Acknowledgements: true 2092 | Retention: File - Limits 2093 | Replicas: 3 2094 | Discard Policy: Old 2095 | Duplicate Window: 2m0s 2096 | Maximum Messages: unlimited 2097 | Maximum Bytes: unlimited 2098 | Maximum Age: 0.00s 2099 | Maximum Message Size: unlimited 2100 | Maximum Consumers: unlimited 2101 | 2102 | 2103 | Cluster Information: 2104 | 2105 | Name: cluster-spoke-2 2106 | Leader: srv-4272 2107 | Replica: srv-4262, current, seen 0.00s ago 2108 | Replica: srv-4202, current, seen 0.00s ago 2109 | 2110 | State: 2111 | 2112 | Messages: 0 2113 | Bytes: 0 B 2114 | FirstSeq: 0 2115 | LastSeq: 0 2116 | Active Consumers: 0 2117 | ``` 2118 | 2119 | There we go, streams are created 2120 | 2121 | ```txt 2122 | > watch -n 1 "nats --context=hub --creds keys/creds/OP/HUBACC/imp.creds s report ; \ 2123 | nats --context=spoke-1 --creds keys/creds/OP/LEAFACC/exp.creds s report ; \ 2124 | nats --context=spoke-2 --creds keys/creds/OP/LEAFACC/exp.creds s report" 2125 | 2126 | Obtaining Stream stats 2127 | 2128 | No Streams defined 2129 | Obtaining Stream stats 2130 | 2131 | ╭──────────────────────────────────────────────────────────────────────────────────────────────────╮ 2132 | │ Stream Report │ 2133 | ├────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤ 2134 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 2135 | ├────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤ 2136 | │ test │ File │ 0 │ 0 │ 0 B │ 0 │ 0 │ srv-4242*, srv-4252, srv-4292 │ 2137 | ╰────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯ 2138 | 2139 | Obtaining Stream stats 2140 | 2141 | ╭──────────────────────────────────────────────────────────────────────────────────────────────────╮ 2142 | │ Stream Report │ 2143 | ├────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤ 2144 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 2145 | ├────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤ 2146 | │ test │ File │ 0 │ 0 │ 0 B │ 0 │ 0 │ srv-4202, srv-4262, srv-4272* │ 2147 | ╰────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯ 2148 | ``` 2149 | 2150 | Now I publish messages to each leaf cluster, using credentials for the leaf account. 2151 | 2152 | ```txt 2153 | > nats --context=spoke-1 --creds keys/creds/OP/LEAFACC/exp.creds pub test "" --count 10 2154 | 14:51:39 Published 0 bytes to "test" 2155 | 14:51:39 Published 0 bytes to "test" 2156 | 14:51:39 Published 0 bytes to "test" 2157 | 14:51:39 Published 0 bytes to "test" 2158 | 14:51:39 Published 0 bytes to "test" 2159 | 14:51:39 Published 0 bytes to "test" 2160 | 14:51:39 Published 0 bytes to "test" 2161 | 14:51:39 Published 0 bytes to "test" 2162 | 14:51:39 Published 0 bytes to "test" 2163 | 14:51:39 Published 0 bytes to "test" 2164 | > nats --context=spoke-2 --creds keys/creds/OP/LEAFACC/exp.creds pub test "" --count 10 2165 | 14:51:45 Published 0 bytes to "test" 2166 | 14:51:45 Published 0 bytes to "test" 2167 | 14:51:45 Published 0 bytes to "test" 2168 | 14:51:45 Published 0 bytes to "test" 2169 | 14:51:45 Published 0 bytes to "test" 2170 | 14:51:45 Published 0 bytes to "test" 2171 | 14:51:45 Published 0 bytes to "test" 2172 | 14:51:45 Published 0 bytes to "test" 2173 | 14:51:45 Published 0 bytes to "test" 2174 | 14:51:45 Published 0 bytes to "test" 2175 | ``` 2176 | 2177 | Because of the isolation, the messages stay in the domain they originated in. 2178 | Every stream contains 10 messages: 2179 | 2180 | ```txt 2181 | > watch -n 1 "nats --context=hub --creds keys/creds/OP/HUBACC/imp.creds s report ; \ 2182 | nats --context=spoke-1 --creds keys/creds/OP/LEAFACC/exp.creds s report ; \ 2183 | nats --context=spoke-2 --creds keys/creds/OP/LEAFACC/exp.creds s report" 2184 | 2185 | Obtaining Stream stats 2186 | 2187 | No Streams defined 2188 | Obtaining Stream stats 2189 | 2190 | ╭──────────────────────────────────────────────────────────────────────────────────────────────────╮ 2191 | │ Stream Report │ 2192 | ├────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤ 2193 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 2194 | ├────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤ 2195 | │ test │ File │ 0 │ 10 │ 340 B │ 0 │ 0 │ srv-4242*, srv-4252, srv-4292 │ 2196 | ╰────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯ 2197 | 2198 | Obtaining Stream stats 2199 | 2200 | ╭──────────────────────────────────────────────────────────────────────────────────────────────────╮ 2201 | │ Stream Report │ 2202 | ├────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤ 2203 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 2204 | ├────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤ 2205 | │ test │ File │ 0 │ 10 │ 340 B │ 0 │ 0 │ srv-4202, srv-4262, srv-4272* │ 2206 | ╰────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯ 2207 | ``` 2208 | 2209 | As a final test let's connect to the hub using the same credentials. 2210 | These messages will not be received because of our isolation scheme. 2211 | 2212 | ```txt 2213 | > nats --context=hub --creds keys/creds/OP/LEAFACC/exp.creds pub test "" --count 10 2214 | 14:53:44 Published 0 bytes to "test" 2215 | 14:53:44 Published 0 bytes to "test" 2216 | 14:53:44 Published 0 bytes to "test" 2217 | 14:53:44 Published 0 bytes to "test" 2218 | 14:53:44 Published 0 bytes to "test" 2219 | 14:53:44 Published 0 bytes to "test" 2220 | 14:53:44 Published 0 bytes to "test" 2221 | 14:53:44 Published 0 bytes to "test" 2222 | 14:53:44 Published 0 bytes to "test" 2223 | 14:53:44 Published 0 bytes to "test" 2224 | ``` 2225 | 2226 | Message count is unaltered as spoke's are isolated from hub as well. 2227 | 2228 | ```txt 2229 | > watch -n 1 "nats --context=hub --creds keys/creds/OP/HUBACC/imp.creds s report ; \ 2230 | nats --context=spoke-1 --creds keys/creds/OP/LEAFACC/exp.creds s report ; \ 2231 | nats --context=spoke-2 --creds keys/creds/OP/LEAFACC/exp.creds s report" 2232 | 2233 | Obtaining Stream stats 2234 | 2235 | No Streams defined 2236 | Obtaining Stream stats 2237 | 2238 | ╭──────────────────────────────────────────────────────────────────────────────────────────────────╮ 2239 | │ Stream Report │ 2240 | ├────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤ 2241 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 2242 | ├────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤ 2243 | │ test │ File │ 0 │ 10 │ 340 B │ 0 │ 0 │ srv-4242*, srv-4252, srv-4292 │ 2244 | ╰────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯ 2245 | 2246 | Obtaining Stream stats 2247 | 2248 | ╭──────────────────────────────────────────────────────────────────────────────────────────────────╮ 2249 | │ Stream Report │ 2250 | ├────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤ 2251 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 2252 | ├────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤ 2253 | │ test │ File │ 0 │ 10 │ 340 B │ 0 │ 0 │ srv-4202, srv-4262, srv-4272* │ 2254 | ╰────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯ 2255 | ``` 2256 | 2257 | Now lets create the importing stream inside the hub and the hub account. 2258 | I'm importing from a different account with the prefix `$JS.leafacc.spoke-1.API`. 2259 | The delivery prefix is `deliver.leafacc.spoke-1.hubacc.hub.aggregate`. 2260 | For `spoke-2` the corresponding prefix and deliver prefix are picked. 2261 | 2262 | ```txt 2263 | > nats --context=hub --creds keys/creds/OP/HUBACC/imp.creds s add aggregate --replicas 3 --source test --source test 2264 | ? Storage backend file 2265 | ? Retention Policy Limits 2266 | ? Discard Policy Old 2267 | ? Stream Messages Limit -1 2268 | ? Message size limit -1 2269 | ? Maximum message age limit -1 2270 | ? Maximum individual message size -1 2271 | ? Duplicate tracking time window 2m 2272 | ? Adjust source "test" start No 2273 | ? Import "test" from a different JetStream domain No 2274 | ? Import "test" from a different account Yes 2275 | ? test Source foreign account API prefix $JS.leafacc.spoke-1.API 2276 | ? test Source foreign account delivery prefix deliver.leafacc.spoke-1.hubacc.hub.aggregate 2277 | ? Adjust source "test" start No 2278 | ? Import "test" from a different JetStream domain No 2279 | ? Import "test" from a different account Yes 2280 | ? test Source foreign account API prefix $JS.leafacc.spoke-2.API 2281 | ? test Source foreign account delivery prefix deliver.leafacc.spoke-2.hubacc.hub.aggregate 2282 | Stream aggregate was created 2283 | 2284 | Information for Stream aggregate created 2021-07-28T14:56:55-04:00 2285 | 2286 | Configuration: 2287 | 2288 | Acknowledgements: true 2289 | Retention: File - Limits 2290 | Replicas: 3 2291 | Discard Policy: Old 2292 | Duplicate Window: 2m0s 2293 | Maximum Messages: unlimited 2294 | Maximum Bytes: unlimited 2295 | Maximum Age: 0.00s 2296 | Maximum Message Size: unlimited 2297 | Maximum Consumers: unlimited 2298 | Sources: test, API Prefix: $JS.leafacc.spoke-1.API, Delivery Prefix: deliver.leafacc.spoke-1.hubacc.hub.aggregate 2299 | test, API Prefix: $JS.leafacc.spoke-2.API, Delivery Prefix: deliver.leafacc.spoke-2.hubacc.hub.aggregate 2300 | 2301 | 2302 | Cluster Information: 2303 | 2304 | Name: cluster-hub 2305 | Leader: srv-4222 2306 | Replica: srv-4282, current, seen 0.00s ago 2307 | Replica: srv-4232, current, seen 0.00s ago 2308 | 2309 | Source Information: 2310 | 2311 | Stream Name: test 2312 | Lag: 0 2313 | Last Seen: 0.00s 2314 | Ext. API Prefix: $JS.leafacc.spoke-1.API 2315 | Ext. Delivery Prefix: deliver.leafacc.spoke-1.hubacc.hub.aggregate 2316 | 2317 | Stream Name: test 2318 | Lag: 0 2319 | Last Seen: 0.00s 2320 | Ext. API Prefix: $JS.leafacc.spoke-2.API 2321 | Ext. Delivery Prefix: deliver.leafacc.spoke-2.hubacc.hub.aggregate 2322 | 2323 | State: 2324 | 2325 | Messages: 0 2326 | Bytes: 0 B 2327 | FirstSeq: 0 2328 | LastSeq: 0 2329 | Active Consumers: 0 2330 | ``` 2331 | 2332 | About the delivery prefix. You are essentially creating a stream that copies from one account in a domain to another account in another domain. 2333 | Thus I'd recommend that the delivery prefix consists of all these parts: type, from account, from domain, to account, to domain, importing stream name. 2334 | 2335 | There you go, it's working see on the left hand side. 2336 | 2337 | ```txt 2338 | > watch -n 1 "nats --context=hub --creds keys/creds/OP/HUBACC/imp.creds s report ; \ 2339 | nats --context=spoke-1 --creds keys/creds/OP/LEAFACC/exp.creds s report ; \ 2340 | nats --context=spoke-2 --creds keys/creds/OP/LEAFACC/exp.creds s report" 2341 | 2342 | Obtaining Stream stats 2343 | 2344 | ╭───────────────────────────────────────────────────────────────────────────────────────────────────────╮ 2345 | │ Stream Report │ 2346 | ├───────────┬─────────┬───────────┬──────────┬─────────┬──────┬─────────┬───────────────────────────────┤ 2347 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 2348 | ├───────────┼─────────┼───────────┼──────────┼─────────┼──────┼─────────┼───────────────────────────────┤ 2349 | │ aggregate │ File │ 0 │ 20 │ 1.7 KiB │ 0 │ 0 │ srv-4222*, srv-4232, srv-4282 │ 2350 | ╰───────────┴─────────┴───────────┴──────────┴─────────┴──────┴─────────┴───────────────────────────────╯ 2351 | 2352 | ╭─────────────────────────────────────────────────────────────────────────────────────╮ 2353 | │ Replication Report │ 2354 | ├───────────┬────────┬─────────────────────────┬───────────────┬────────┬─────┬───────┤ 2355 | │ Stream │ Kind │ API Prefix │ Source Stream │ Active │ Lag │ Error │ 2356 | ├───────────┼────────┼─────────────────────────┼───────────────┼────────┼─────┼───────┤ 2357 | │ aggregate │ Source │ $JS.leafacc.spoke-1.API │ test │ 0.24s │ 0 │ │ 2358 | │ aggregate │ Source │ $JS.leafacc.spoke-2.API │ test │ 0.24s │ 0 │ │ 2359 | ╰───────────┴────────┴─────────────────────────┴───────────────┴────────┴─────┴───────╯ 2360 | 2361 | Obtaining Stream stats 2362 | 2363 | ╭──────────────────────────────────────────────────────────────────────────────────────────────────╮ 2364 | │ Stream Report │ 2365 | ├────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤ 2366 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 2367 | ├────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤ 2368 | │ test │ File │ 0 │ 10 │ 340 B │ 0 │ 0 │ srv-4242*, srv-4252, srv-4292 │ 2369 | ╰────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯ 2370 | 2371 | Obtaining Stream stats 2372 | 2373 | ╭──────────────────────────────────────────────────────────────────────────────────────────────────╮ 2374 | │ Stream Report │ 2375 | ├────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤ 2376 | │ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │ 2377 | ├────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤ 2378 | │ test │ File │ 0 │ 10 │ 340 B │ 0 │ 0 │ srv-4202, srv-4262, srv-4272* │ 2379 | ╰────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯ 2380 | ``` 2381 | 2382 | Please be aware that it does not matter that the hub is actually a hub. 2383 | I want you to take away that you can chain accounts and connect through them. 2384 | Just think of a scenario where you have multiple leaf nodes into an ngs account, collecting data locally. 2385 | One of your leaf nodes is specked bigger and this is where you aggregate streams and do all your analytics. 2386 | 2387 | One thing I hope you noticed is the lack of host names and the flexibility that gives you. 2388 | Of course, they are needed in the server configuration to create the nats network. 2389 | They are also needed to connect to that network. 2390 | But you don't need different ones for each of your applications. 2391 | Instead you get to focus on your data and how it flows. 2392 | 2393 | We will continue to add improvements to the setups and workflows introduced in this video. 2394 | I certainly have noticed a few rough edges, so it is worthwhile checking back every once in a while. 2395 | If you have questions, reach out on our slack channel on: natsio.slack.com 2396 | 2397 | #### Relevant Links 2398 | 2399 | * get nats cli at: https://github.com/nats-io/natscli 2400 | * get nsc cli at: https://github.com/nats-io/nsc 2401 | * docs: https://docs.nats.io/ 2402 | * docs for config: https://docs.nats.io/nats-server/configuration 2403 | * doc for config based accounts: https://docs.nats.io/nats-server/configuration/securing_nats/accounts 2404 | * jwt deep dive: https://docs.nats.io/developing-with-nats/tutorials/jwt 2405 | -------------------------------------------------------------------------------- /account-component-diagram-cross.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nats-io/jetstream-leaf-nodes-demo/15a5057ba82448307fa2344334d8a4a4a7006263/account-component-diagram-cross.png -------------------------------------------------------------------------------- /account-component-diagram-through.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nats-io/jetstream-leaf-nodes-demo/15a5057ba82448307fa2344334d8a4a4a7006263/account-component-diagram-through.png -------------------------------------------------------------------------------- /cache1/AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiIyNjZKUEFFQlMzVUdWQVVJRElNWFlMSzdVQ0lXUFdWU0VTS1dDTkRMVFFNRlVMTFlJQTRRIiwiaWF0IjoxNjI2ODIzMTYxLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJURVNUIiwic3ViIjoiQUE1QzU2RkFFVEJUVUNZTTdOQzVCRkJZRlRLTE9BQklPSUZQUURITzRSVUVBUFNOM0ZUWTVSNEciLCJuYXRzIjp7ImxpbWl0cyI6eyJzdWJzIjotMSwiZGF0YSI6LTEsInBheWxvYWQiOi0xLCJpbXBvcnRzIjotMSwiZXhwb3J0cyI6LTEsIndpbGRjYXJkcyI6dHJ1ZSwiY29ubiI6LTEsImxlYWYiOi0xLCJtZW1fc3RvcmFnZSI6LTEsImRpc2tfc3RvcmFnZSI6LTEsInN0cmVhbXMiOi0xLCJjb25zdW1lciI6LTF9LCJkZWZhdWx0X3Blcm1pc3Npb25zIjp7InB1YiI6e30sInN1YiI6e319LCJ0eXBlIjoiYWNjb3VudCIsInZlcnNpb24iOjJ9fQ.hM-Mi1HkROLNTEPrZEs-XbN_TeTJ40ZoYrQYHwqlyzwx1sT4tM0ku37hAizslYH6Wf-tUA-GQWmEOCu_Ba0JDA -------------------------------------------------------------------------------- /cache1/ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiJMTFpUU0paUTNGTkZWNlhDVDNBRkdCSlZWU0FaSk9IN0JSNlFETUdNVUdETktVWUlQR0hRIiwiaWF0IjoxNjI0NDc5NDgyLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJTWVMiLCJzdWIiOiJBREVDQ05CVUVCV1o3MjdPTUJGU043T01LMkZQWVJNNTJUSlMyNVRGUVdZUzc2TlBPSkJOM0tVNCIsIm5hdHMiOnsibGltaXRzIjp7InN1YnMiOi0xLCJkYXRhIjotMSwicGF5bG9hZCI6LTEsImltcG9ydHMiOi0xLCJleHBvcnRzIjotMSwid2lsZGNhcmRzIjp0cnVlLCJjb25uIjotMSwibGVhZiI6LTF9LCJzaWduaW5nX2tleXMiOlsiQUFDWUlDT0FRTVE3MkVIVDM1UjdMVjZWRldNSVZXRktXRkU1UDJKSjJUVDY3NEVPN0RKVFVITU0iXSwiZGVmYXVsdF9wZXJtaXNzaW9ucyI6eyJwdWIiOnt9LCJzdWIiOnt9fSwidHlwZSI6ImFjY291bnQiLCJ2ZXJzaW9uIjoyfX0.tHteTcshVIInToM6LQ7G2AmmWfeKYCCjJdCC4ZfJ1WUtmY1Bk0sEwwbjb6uSycEb4ljohMQnQVgkbAYZsiqZDw -------------------------------------------------------------------------------- /cache2/AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiIyNjZKUEFFQlMzVUdWQVVJRElNWFlMSzdVQ0lXUFdWU0VTS1dDTkRMVFFNRlVMTFlJQTRRIiwiaWF0IjoxNjI2ODIzMTYxLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJURVNUIiwic3ViIjoiQUE1QzU2RkFFVEJUVUNZTTdOQzVCRkJZRlRLTE9BQklPSUZQUURITzRSVUVBUFNOM0ZUWTVSNEciLCJuYXRzIjp7ImxpbWl0cyI6eyJzdWJzIjotMSwiZGF0YSI6LTEsInBheWxvYWQiOi0xLCJpbXBvcnRzIjotMSwiZXhwb3J0cyI6LTEsIndpbGRjYXJkcyI6dHJ1ZSwiY29ubiI6LTEsImxlYWYiOi0xLCJtZW1fc3RvcmFnZSI6LTEsImRpc2tfc3RvcmFnZSI6LTEsInN0cmVhbXMiOi0xLCJjb25zdW1lciI6LTF9LCJkZWZhdWx0X3Blcm1pc3Npb25zIjp7InB1YiI6e30sInN1YiI6e319LCJ0eXBlIjoiYWNjb3VudCIsInZlcnNpb24iOjJ9fQ.hM-Mi1HkROLNTEPrZEs-XbN_TeTJ40ZoYrQYHwqlyzwx1sT4tM0ku37hAizslYH6Wf-tUA-GQWmEOCu_Ba0JDA -------------------------------------------------------------------------------- /cache2/ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiJMTFpUU0paUTNGTkZWNlhDVDNBRkdCSlZWU0FaSk9IN0JSNlFETUdNVUdETktVWUlQR0hRIiwiaWF0IjoxNjI0NDc5NDgyLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJTWVMiLCJzdWIiOiJBREVDQ05CVUVCV1o3MjdPTUJGU043T01LMkZQWVJNNTJUSlMyNVRGUVdZUzc2TlBPSkJOM0tVNCIsIm5hdHMiOnsibGltaXRzIjp7InN1YnMiOi0xLCJkYXRhIjotMSwicGF5bG9hZCI6LTEsImltcG9ydHMiOi0xLCJleHBvcnRzIjotMSwid2lsZGNhcmRzIjp0cnVlLCJjb25uIjotMSwibGVhZiI6LTF9LCJzaWduaW5nX2tleXMiOlsiQUFDWUlDT0FRTVE3MkVIVDM1UjdMVjZWRldNSVZXRktXRkU1UDJKSjJUVDY3NEVPN0RKVFVITU0iXSwiZGVmYXVsdF9wZXJtaXNzaW9ucyI6eyJwdWIiOnt9LCJzdWIiOnt9fSwidHlwZSI6ImFjY291bnQiLCJ2ZXJzaW9uIjoyfX0.tHteTcshVIInToM6LQ7G2AmmWfeKYCCjJdCC4ZfJ1WUtmY1Bk0sEwwbjb6uSycEb4ljohMQnQVgkbAYZsiqZDw -------------------------------------------------------------------------------- /cache3/AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiIyNjZKUEFFQlMzVUdWQVVJRElNWFlMSzdVQ0lXUFdWU0VTS1dDTkRMVFFNRlVMTFlJQTRRIiwiaWF0IjoxNjI2ODIzMTYxLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJURVNUIiwic3ViIjoiQUE1QzU2RkFFVEJUVUNZTTdOQzVCRkJZRlRLTE9BQklPSUZQUURITzRSVUVBUFNOM0ZUWTVSNEciLCJuYXRzIjp7ImxpbWl0cyI6eyJzdWJzIjotMSwiZGF0YSI6LTEsInBheWxvYWQiOi0xLCJpbXBvcnRzIjotMSwiZXhwb3J0cyI6LTEsIndpbGRjYXJkcyI6dHJ1ZSwiY29ubiI6LTEsImxlYWYiOi0xLCJtZW1fc3RvcmFnZSI6LTEsImRpc2tfc3RvcmFnZSI6LTEsInN0cmVhbXMiOi0xLCJjb25zdW1lciI6LTF9LCJkZWZhdWx0X3Blcm1pc3Npb25zIjp7InB1YiI6e30sInN1YiI6e319LCJ0eXBlIjoiYWNjb3VudCIsInZlcnNpb24iOjJ9fQ.hM-Mi1HkROLNTEPrZEs-XbN_TeTJ40ZoYrQYHwqlyzwx1sT4tM0ku37hAizslYH6Wf-tUA-GQWmEOCu_Ba0JDA -------------------------------------------------------------------------------- /cache3/ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiJMTFpUU0paUTNGTkZWNlhDVDNBRkdCSlZWU0FaSk9IN0JSNlFETUdNVUdETktVWUlQR0hRIiwiaWF0IjoxNjI0NDc5NDgyLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJTWVMiLCJzdWIiOiJBREVDQ05CVUVCV1o3MjdPTUJGU043T01LMkZQWVJNNTJUSlMyNVRGUVdZUzc2TlBPSkJOM0tVNCIsIm5hdHMiOnsibGltaXRzIjp7InN1YnMiOi0xLCJkYXRhIjotMSwicGF5bG9hZCI6LTEsImltcG9ydHMiOi0xLCJleHBvcnRzIjotMSwid2lsZGNhcmRzIjp0cnVlLCJjb25uIjotMSwibGVhZiI6LTF9LCJzaWduaW5nX2tleXMiOlsiQUFDWUlDT0FRTVE3MkVIVDM1UjdMVjZWRldNSVZXRktXRkU1UDJKSjJUVDY3NEVPN0RKVFVITU0iXSwiZGVmYXVsdF9wZXJtaXNzaW9ucyI6eyJwdWIiOnt9LCJzdWIiOnt9fSwidHlwZSI6ImFjY291bnQiLCJ2ZXJzaW9uIjoyfX0.tHteTcshVIInToM6LQ7G2AmmWfeKYCCjJdCC4ZfJ1WUtmY1Bk0sEwwbjb6uSycEb4ljohMQnQVgkbAYZsiqZDw -------------------------------------------------------------------------------- /cache4/AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiIyNjZKUEFFQlMzVUdWQVVJRElNWFlMSzdVQ0lXUFdWU0VTS1dDTkRMVFFNRlVMTFlJQTRRIiwiaWF0IjoxNjI2ODIzMTYxLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJURVNUIiwic3ViIjoiQUE1QzU2RkFFVEJUVUNZTTdOQzVCRkJZRlRLTE9BQklPSUZQUURITzRSVUVBUFNOM0ZUWTVSNEciLCJuYXRzIjp7ImxpbWl0cyI6eyJzdWJzIjotMSwiZGF0YSI6LTEsInBheWxvYWQiOi0xLCJpbXBvcnRzIjotMSwiZXhwb3J0cyI6LTEsIndpbGRjYXJkcyI6dHJ1ZSwiY29ubiI6LTEsImxlYWYiOi0xLCJtZW1fc3RvcmFnZSI6LTEsImRpc2tfc3RvcmFnZSI6LTEsInN0cmVhbXMiOi0xLCJjb25zdW1lciI6LTF9LCJkZWZhdWx0X3Blcm1pc3Npb25zIjp7InB1YiI6e30sInN1YiI6e319LCJ0eXBlIjoiYWNjb3VudCIsInZlcnNpb24iOjJ9fQ.hM-Mi1HkROLNTEPrZEs-XbN_TeTJ40ZoYrQYHwqlyzwx1sT4tM0ku37hAizslYH6Wf-tUA-GQWmEOCu_Ba0JDA -------------------------------------------------------------------------------- /cache4/ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiJMTFpUU0paUTNGTkZWNlhDVDNBRkdCSlZWU0FaSk9IN0JSNlFETUdNVUdETktVWUlQR0hRIiwiaWF0IjoxNjI0NDc5NDgyLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJTWVMiLCJzdWIiOiJBREVDQ05CVUVCV1o3MjdPTUJGU043T01LMkZQWVJNNTJUSlMyNVRGUVdZUzc2TlBPSkJOM0tVNCIsIm5hdHMiOnsibGltaXRzIjp7InN1YnMiOi0xLCJkYXRhIjotMSwicGF5bG9hZCI6LTEsImltcG9ydHMiOi0xLCJleHBvcnRzIjotMSwid2lsZGNhcmRzIjp0cnVlLCJjb25uIjotMSwibGVhZiI6LTF9LCJzaWduaW5nX2tleXMiOlsiQUFDWUlDT0FRTVE3MkVIVDM1UjdMVjZWRldNSVZXRktXRkU1UDJKSjJUVDY3NEVPN0RKVFVITU0iXSwiZGVmYXVsdF9wZXJtaXNzaW9ucyI6eyJwdWIiOnt9LCJzdWIiOnt9fSwidHlwZSI6ImFjY291bnQiLCJ2ZXJzaW9uIjoyfX0.tHteTcshVIInToM6LQ7G2AmmWfeKYCCjJdCC4ZfJ1WUtmY1Bk0sEwwbjb6uSycEb4ljohMQnQVgkbAYZsiqZDw -------------------------------------------------------------------------------- /cache5/AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiIyNjZKUEFFQlMzVUdWQVVJRElNWFlMSzdVQ0lXUFdWU0VTS1dDTkRMVFFNRlVMTFlJQTRRIiwiaWF0IjoxNjI2ODIzMTYxLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJURVNUIiwic3ViIjoiQUE1QzU2RkFFVEJUVUNZTTdOQzVCRkJZRlRLTE9BQklPSUZQUURITzRSVUVBUFNOM0ZUWTVSNEciLCJuYXRzIjp7ImxpbWl0cyI6eyJzdWJzIjotMSwiZGF0YSI6LTEsInBheWxvYWQiOi0xLCJpbXBvcnRzIjotMSwiZXhwb3J0cyI6LTEsIndpbGRjYXJkcyI6dHJ1ZSwiY29ubiI6LTEsImxlYWYiOi0xLCJtZW1fc3RvcmFnZSI6LTEsImRpc2tfc3RvcmFnZSI6LTEsInN0cmVhbXMiOi0xLCJjb25zdW1lciI6LTF9LCJkZWZhdWx0X3Blcm1pc3Npb25zIjp7InB1YiI6e30sInN1YiI6e319LCJ0eXBlIjoiYWNjb3VudCIsInZlcnNpb24iOjJ9fQ.hM-Mi1HkROLNTEPrZEs-XbN_TeTJ40ZoYrQYHwqlyzwx1sT4tM0ku37hAizslYH6Wf-tUA-GQWmEOCu_Ba0JDA -------------------------------------------------------------------------------- /cache5/ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiJMTFpUU0paUTNGTkZWNlhDVDNBRkdCSlZWU0FaSk9IN0JSNlFETUdNVUdETktVWUlQR0hRIiwiaWF0IjoxNjI0NDc5NDgyLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJTWVMiLCJzdWIiOiJBREVDQ05CVUVCV1o3MjdPTUJGU043T01LMkZQWVJNNTJUSlMyNVRGUVdZUzc2TlBPSkJOM0tVNCIsIm5hdHMiOnsibGltaXRzIjp7InN1YnMiOi0xLCJkYXRhIjotMSwicGF5bG9hZCI6LTEsImltcG9ydHMiOi0xLCJleHBvcnRzIjotMSwid2lsZGNhcmRzIjp0cnVlLCJjb25uIjotMSwibGVhZiI6LTF9LCJzaWduaW5nX2tleXMiOlsiQUFDWUlDT0FRTVE3MkVIVDM1UjdMVjZWRldNSVZXRktXRkU1UDJKSjJUVDY3NEVPN0RKVFVITU0iXSwiZGVmYXVsdF9wZXJtaXNzaW9ucyI6eyJwdWIiOnt9LCJzdWIiOnt9fSwidHlwZSI6ImFjY291bnQiLCJ2ZXJzaW9uIjoyfX0.tHteTcshVIInToM6LQ7G2AmmWfeKYCCjJdCC4ZfJ1WUtmY1Bk0sEwwbjb6uSycEb4ljohMQnQVgkbAYZsiqZDw -------------------------------------------------------------------------------- /cache6/AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiIyNjZKUEFFQlMzVUdWQVVJRElNWFlMSzdVQ0lXUFdWU0VTS1dDTkRMVFFNRlVMTFlJQTRRIiwiaWF0IjoxNjI2ODIzMTYxLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJURVNUIiwic3ViIjoiQUE1QzU2RkFFVEJUVUNZTTdOQzVCRkJZRlRLTE9BQklPSUZQUURITzRSVUVBUFNOM0ZUWTVSNEciLCJuYXRzIjp7ImxpbWl0cyI6eyJzdWJzIjotMSwiZGF0YSI6LTEsInBheWxvYWQiOi0xLCJpbXBvcnRzIjotMSwiZXhwb3J0cyI6LTEsIndpbGRjYXJkcyI6dHJ1ZSwiY29ubiI6LTEsImxlYWYiOi0xLCJtZW1fc3RvcmFnZSI6LTEsImRpc2tfc3RvcmFnZSI6LTEsInN0cmVhbXMiOi0xLCJjb25zdW1lciI6LTF9LCJkZWZhdWx0X3Blcm1pc3Npb25zIjp7InB1YiI6e30sInN1YiI6e319LCJ0eXBlIjoiYWNjb3VudCIsInZlcnNpb24iOjJ9fQ.hM-Mi1HkROLNTEPrZEs-XbN_TeTJ40ZoYrQYHwqlyzwx1sT4tM0ku37hAizslYH6Wf-tUA-GQWmEOCu_Ba0JDA -------------------------------------------------------------------------------- /cache6/ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiJMTFpUU0paUTNGTkZWNlhDVDNBRkdCSlZWU0FaSk9IN0JSNlFETUdNVUdETktVWUlQR0hRIiwiaWF0IjoxNjI0NDc5NDgyLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJTWVMiLCJzdWIiOiJBREVDQ05CVUVCV1o3MjdPTUJGU043T01LMkZQWVJNNTJUSlMyNVRGUVdZUzc2TlBPSkJOM0tVNCIsIm5hdHMiOnsibGltaXRzIjp7InN1YnMiOi0xLCJkYXRhIjotMSwicGF5bG9hZCI6LTEsImltcG9ydHMiOi0xLCJleHBvcnRzIjotMSwid2lsZGNhcmRzIjp0cnVlLCJjb25uIjotMSwibGVhZiI6LTF9LCJzaWduaW5nX2tleXMiOlsiQUFDWUlDT0FRTVE3MkVIVDM1UjdMVjZWRldNSVZXRktXRkU1UDJKSjJUVDY3NEVPN0RKVFVITU0iXSwiZGVmYXVsdF9wZXJtaXNzaW9ucyI6eyJwdWIiOnt9LCJzdWIiOnt9fSwidHlwZSI6ImFjY291bnQiLCJ2ZXJzaW9uIjoyfX0.tHteTcshVIInToM6LQ7G2AmmWfeKYCCjJdCC4ZfJ1WUtmY1Bk0sEwwbjb6uSycEb4ljohMQnQVgkbAYZsiqZDw -------------------------------------------------------------------------------- /cache7/AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiIyNjZKUEFFQlMzVUdWQVVJRElNWFlMSzdVQ0lXUFdWU0VTS1dDTkRMVFFNRlVMTFlJQTRRIiwiaWF0IjoxNjI2ODIzMTYxLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJURVNUIiwic3ViIjoiQUE1QzU2RkFFVEJUVUNZTTdOQzVCRkJZRlRLTE9BQklPSUZQUURITzRSVUVBUFNOM0ZUWTVSNEciLCJuYXRzIjp7ImxpbWl0cyI6eyJzdWJzIjotMSwiZGF0YSI6LTEsInBheWxvYWQiOi0xLCJpbXBvcnRzIjotMSwiZXhwb3J0cyI6LTEsIndpbGRjYXJkcyI6dHJ1ZSwiY29ubiI6LTEsImxlYWYiOi0xLCJtZW1fc3RvcmFnZSI6LTEsImRpc2tfc3RvcmFnZSI6LTEsInN0cmVhbXMiOi0xLCJjb25zdW1lciI6LTF9LCJkZWZhdWx0X3Blcm1pc3Npb25zIjp7InB1YiI6e30sInN1YiI6e319LCJ0eXBlIjoiYWNjb3VudCIsInZlcnNpb24iOjJ9fQ.hM-Mi1HkROLNTEPrZEs-XbN_TeTJ40ZoYrQYHwqlyzwx1sT4tM0ku37hAizslYH6Wf-tUA-GQWmEOCu_Ba0JDA -------------------------------------------------------------------------------- /cache7/ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiJMTFpUU0paUTNGTkZWNlhDVDNBRkdCSlZWU0FaSk9IN0JSNlFETUdNVUdETktVWUlQR0hRIiwiaWF0IjoxNjI0NDc5NDgyLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJTWVMiLCJzdWIiOiJBREVDQ05CVUVCV1o3MjdPTUJGU043T01LMkZQWVJNNTJUSlMyNVRGUVdZUzc2TlBPSkJOM0tVNCIsIm5hdHMiOnsibGltaXRzIjp7InN1YnMiOi0xLCJkYXRhIjotMSwicGF5bG9hZCI6LTEsImltcG9ydHMiOi0xLCJleHBvcnRzIjotMSwid2lsZGNhcmRzIjp0cnVlLCJjb25uIjotMSwibGVhZiI6LTF9LCJzaWduaW5nX2tleXMiOlsiQUFDWUlDT0FRTVE3MkVIVDM1UjdMVjZWRldNSVZXRktXRkU1UDJKSjJUVDY3NEVPN0RKVFVITU0iXSwiZGVmYXVsdF9wZXJtaXNzaW9ucyI6eyJwdWIiOnt9LCJzdWIiOnt9fSwidHlwZSI6ImFjY291bnQiLCJ2ZXJzaW9uIjoyfX0.tHteTcshVIInToM6LQ7G2AmmWfeKYCCjJdCC4ZfJ1WUtmY1Bk0sEwwbjb6uSycEb4ljohMQnQVgkbAYZsiqZDw -------------------------------------------------------------------------------- /cache8/AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiIyNjZKUEFFQlMzVUdWQVVJRElNWFlMSzdVQ0lXUFdWU0VTS1dDTkRMVFFNRlVMTFlJQTRRIiwiaWF0IjoxNjI2ODIzMTYxLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJURVNUIiwic3ViIjoiQUE1QzU2RkFFVEJUVUNZTTdOQzVCRkJZRlRLTE9BQklPSUZQUURITzRSVUVBUFNOM0ZUWTVSNEciLCJuYXRzIjp7ImxpbWl0cyI6eyJzdWJzIjotMSwiZGF0YSI6LTEsInBheWxvYWQiOi0xLCJpbXBvcnRzIjotMSwiZXhwb3J0cyI6LTEsIndpbGRjYXJkcyI6dHJ1ZSwiY29ubiI6LTEsImxlYWYiOi0xLCJtZW1fc3RvcmFnZSI6LTEsImRpc2tfc3RvcmFnZSI6LTEsInN0cmVhbXMiOi0xLCJjb25zdW1lciI6LTF9LCJkZWZhdWx0X3Blcm1pc3Npb25zIjp7InB1YiI6e30sInN1YiI6e319LCJ0eXBlIjoiYWNjb3VudCIsInZlcnNpb24iOjJ9fQ.hM-Mi1HkROLNTEPrZEs-XbN_TeTJ40ZoYrQYHwqlyzwx1sT4tM0ku37hAizslYH6Wf-tUA-GQWmEOCu_Ba0JDA -------------------------------------------------------------------------------- /cache8/ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiJMTFpUU0paUTNGTkZWNlhDVDNBRkdCSlZWU0FaSk9IN0JSNlFETUdNVUdETktVWUlQR0hRIiwiaWF0IjoxNjI0NDc5NDgyLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJTWVMiLCJzdWIiOiJBREVDQ05CVUVCV1o3MjdPTUJGU043T01LMkZQWVJNNTJUSlMyNVRGUVdZUzc2TlBPSkJOM0tVNCIsIm5hdHMiOnsibGltaXRzIjp7InN1YnMiOi0xLCJkYXRhIjotMSwicGF5bG9hZCI6LTEsImltcG9ydHMiOi0xLCJleHBvcnRzIjotMSwid2lsZGNhcmRzIjp0cnVlLCJjb25uIjotMSwibGVhZiI6LTF9LCJzaWduaW5nX2tleXMiOlsiQUFDWUlDT0FRTVE3MkVIVDM1UjdMVjZWRldNSVZXRktXRkU1UDJKSjJUVDY3NEVPN0RKVFVITU0iXSwiZGVmYXVsdF9wZXJtaXNzaW9ucyI6eyJwdWIiOnt9LCJzdWIiOnt9fSwidHlwZSI6ImFjY291bnQiLCJ2ZXJzaW9uIjoyfX0.tHteTcshVIInToM6LQ7G2AmmWfeKYCCjJdCC4ZfJ1WUtmY1Bk0sEwwbjb6uSycEb4ljohMQnQVgkbAYZsiqZDw -------------------------------------------------------------------------------- /cache9/AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiIyNjZKUEFFQlMzVUdWQVVJRElNWFlMSzdVQ0lXUFdWU0VTS1dDTkRMVFFNRlVMTFlJQTRRIiwiaWF0IjoxNjI2ODIzMTYxLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJURVNUIiwic3ViIjoiQUE1QzU2RkFFVEJUVUNZTTdOQzVCRkJZRlRLTE9BQklPSUZQUURITzRSVUVBUFNOM0ZUWTVSNEciLCJuYXRzIjp7ImxpbWl0cyI6eyJzdWJzIjotMSwiZGF0YSI6LTEsInBheWxvYWQiOi0xLCJpbXBvcnRzIjotMSwiZXhwb3J0cyI6LTEsIndpbGRjYXJkcyI6dHJ1ZSwiY29ubiI6LTEsImxlYWYiOi0xLCJtZW1fc3RvcmFnZSI6LTEsImRpc2tfc3RvcmFnZSI6LTEsInN0cmVhbXMiOi0xLCJjb25zdW1lciI6LTF9LCJkZWZhdWx0X3Blcm1pc3Npb25zIjp7InB1YiI6e30sInN1YiI6e319LCJ0eXBlIjoiYWNjb3VudCIsInZlcnNpb24iOjJ9fQ.hM-Mi1HkROLNTEPrZEs-XbN_TeTJ40ZoYrQYHwqlyzwx1sT4tM0ku37hAizslYH6Wf-tUA-GQWmEOCu_Ba0JDA -------------------------------------------------------------------------------- /cache9/ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiJMTFpUU0paUTNGTkZWNlhDVDNBRkdCSlZWU0FaSk9IN0JSNlFETUdNVUdETktVWUlQR0hRIiwiaWF0IjoxNjI0NDc5NDgyLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJTWVMiLCJzdWIiOiJBREVDQ05CVUVCV1o3MjdPTUJGU043T01LMkZQWVJNNTJUSlMyNVRGUVdZUzc2TlBPSkJOM0tVNCIsIm5hdHMiOnsibGltaXRzIjp7InN1YnMiOi0xLCJkYXRhIjotMSwicGF5bG9hZCI6LTEsImltcG9ydHMiOi0xLCJleHBvcnRzIjotMSwid2lsZGNhcmRzIjp0cnVlLCJjb25uIjotMSwibGVhZiI6LTF9LCJzaWduaW5nX2tleXMiOlsiQUFDWUlDT0FRTVE3MkVIVDM1UjdMVjZWRldNSVZXRktXRkU1UDJKSjJUVDY3NEVPN0RKVFVITU0iXSwiZGVmYXVsdF9wZXJtaXNzaW9ucyI6eyJwdWIiOnt9LCJzdWIiOnt9fSwidHlwZSI6ImFjY291bnQiLCJ2ZXJzaW9uIjoyfX0.tHteTcshVIInToM6LQ7G2AmmWfeKYCCjJdCC4ZfJ1WUtmY1Bk0sEwwbjb6uSycEb4ljohMQnQVgkbAYZsiqZDw -------------------------------------------------------------------------------- /cluster-hub-1.cfg: -------------------------------------------------------------------------------- 1 | listen: localhost:4222 2 | server_name: srv-4222 3 | jetstream { 4 | store_dir: "./s1-1" 5 | domain: hub 6 | } 7 | cluster { 8 | listen localhost:4223 9 | name cluster-hub 10 | routes = [ 11 | nats-route://localhost:4223 12 | nats-route://localhost:4233 13 | nats-route://localhost:4283 14 | ] 15 | } 16 | leafnodes { 17 | listen localhost:4224 18 | no_advertise: true 19 | } 20 | mqtt { 21 | port: 4225 22 | } 23 | http: localhost:8080 24 | include ./nats-account-resolver.cfg 25 | -------------------------------------------------------------------------------- /cluster-hub-2.cfg: -------------------------------------------------------------------------------- 1 | listen: localhost:4232 2 | server_name: srv-4232 3 | jetstream { 4 | store_dir: "./s1-2" 5 | domain: hub 6 | } 7 | cluster { 8 | listen localhost:4233 9 | name cluster-hub 10 | routes = [ 11 | nats-route://localhost:4223 12 | nats-route://localhost:4233 13 | nats-route://localhost:4283 14 | ] 15 | } 16 | leafnodes { 17 | listen localhost:4234 18 | no_advertise: true 19 | } 20 | mqtt { 21 | port: 4235 22 | } 23 | http: localhost:8081 24 | include ./nats-account-resolver.cfg 25 | -------------------------------------------------------------------------------- /cluster-hub-3.cfg: -------------------------------------------------------------------------------- 1 | listen: localhost:4282 2 | server_name: srv-4282 3 | jetstream { 4 | store_dir: "./s1-3" 5 | domain: hub 6 | } 7 | cluster { 8 | listen localhost:4283 9 | name cluster-hub 10 | routes = [ 11 | nats-route://localhost:4223 12 | nats-route://localhost:4233 13 | nats-route://localhost:4283 14 | ] 15 | } 16 | leafnodes { 17 | listen localhost:4284 18 | no_advertise: true 19 | } 20 | mqtt { 21 | port: 4285 22 | } 23 | http: localhost:8083 24 | include ./nats-account-resolver.cfg 25 | -------------------------------------------------------------------------------- /cluster-spoke-1-1.cfg: -------------------------------------------------------------------------------- 1 | listen: localhost:4242 2 | server_name: srv-4242 3 | jetstream { 4 | store_dir: "./s2-1" 5 | domain: spoke-1 6 | } 7 | cluster { 8 | listen localhost:4243 9 | name cluster-spoke-1 10 | routes = [ 11 | nats-route://localhost:4243 12 | nats-route://localhost:4253 13 | nats-route://localhost:4293 14 | ] 15 | } 16 | mqtt { 17 | port: 4245 18 | } 19 | http: localhost:8084 20 | include ./nats-account-resolver.cfg 21 | include ./leafnode-remotes.cfg 22 | -------------------------------------------------------------------------------- /cluster-spoke-1-2.cfg: -------------------------------------------------------------------------------- 1 | listen: localhost:4252 2 | server_name: srv-4252 3 | jetstream { 4 | store_dir: "./s2-2" 5 | domain: spoke-1 6 | } 7 | cluster { 8 | listen localhost:4253 9 | name cluster-spoke-1 10 | routes = [ 11 | nats-route://localhost:4243 12 | nats-route://localhost:4253 13 | nats-route://localhost:4293 14 | ] 15 | } 16 | mqtt { 17 | port: 4255 18 | } 19 | http: localhost:8085 20 | include ./nats-account-resolver.cfg 21 | include ./leafnode-remotes.cfg 22 | -------------------------------------------------------------------------------- /cluster-spoke-1-3.cfg: -------------------------------------------------------------------------------- 1 | listen: localhost:4292 2 | server_name: srv-4292 3 | jetstream { 4 | store_dir: "./s2-3" 5 | domain: spoke-1 6 | } 7 | cluster { 8 | listen localhost:4293 9 | name cluster-spoke-1 10 | routes = [ 11 | nats-route://localhost:4243 12 | nats-route://localhost:4253 13 | nats-route://localhost:4293 14 | ] 15 | } 16 | mqtt { 17 | port: 4295 18 | } 19 | http: localhost:8086 20 | include ./nats-account-resolver.cfg 21 | include ./leafnode-remotes.cfg 22 | -------------------------------------------------------------------------------- /cluster-spoke-2-1.cfg: -------------------------------------------------------------------------------- 1 | listen: localhost:4262 2 | server_name: srv-4262 3 | jetstream { 4 | store_dir: "./s3-1" 5 | domain: spoke-2 6 | } 7 | cluster { 8 | listen localhost:4263 9 | name cluster-spoke-2 10 | routes = [ 11 | nats-route://localhost:4263 12 | nats-route://localhost:4273 13 | nats-route://localhost:4203 14 | ] 15 | } 16 | mqtt { 17 | port: 4265 18 | } 19 | http: localhost:8087 20 | include ./nats-account-resolver.cfg 21 | include ./leafnode-remotes.cfg 22 | -------------------------------------------------------------------------------- /cluster-spoke-2-2.cfg: -------------------------------------------------------------------------------- 1 | listen: localhost:4272 2 | server_name: srv-4272 3 | jetstream { 4 | store_dir: "./s3-2" 5 | domain: spoke-2 6 | } 7 | cluster { 8 | listen localhost:4273 9 | name cluster-spoke-2 10 | routes = [ 11 | nats-route://localhost:4263 12 | nats-route://localhost:4273 13 | nats-route://localhost:4203 14 | ] 15 | } 16 | mqtt { 17 | port: 4275 18 | } 19 | http: localhost:8088 20 | include ./nats-account-resolver.cfg 21 | include ./leafnode-remotes.cfg 22 | -------------------------------------------------------------------------------- /cluster-spoke-2-3.cfg: -------------------------------------------------------------------------------- 1 | listen: localhost:4202 2 | server_name: srv-4202 3 | jetstream { 4 | store_dir: "./s3-3" 5 | domain: spoke-2 6 | } 7 | cluster { 8 | listen localhost:4203 9 | name cluster-spoke-2 10 | routes = [ 11 | nats-route://localhost:4263 12 | nats-route://localhost:4273 13 | nats-route://localhost:4203 14 | ] 15 | } 16 | mqtt { 17 | port: 4205 18 | } 19 | http: localhost:8089 20 | include ./nats-account-resolver.cfg 21 | include ./leafnode-remotes.cfg 22 | -------------------------------------------------------------------------------- /keys/.gitignore: -------------------------------------------------------------------------------- 1 | # ignore all nk files 2 | **/*.nk 3 | 4 | # ignore all creds files 5 | **/*.creds 6 | -------------------------------------------------------------------------------- /leafnode-remotes.cfg: -------------------------------------------------------------------------------- 1 | HUB-URLS=["nats-leaf://localhost:4224","nats-leaf://localhost:4234","nats-leaf://localhost:4284"] 2 | leafnodes { 3 | no_advertise: true 4 | remotes = [ 5 | { 6 | urls: $HUB-URLS 7 | account: ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4 8 | credentials: keys/creds/OP/SYS/sys.creds 9 | }, 10 | { 11 | urls: $HUB-URLS 12 | account: AA5C56FAETBTUCYM7NC5BFBYFTKLOABIOIFPQDHO4RUEAPSN3FTY5R4G 13 | credentials: keys/creds/OP/TEST/leaf.creds 14 | }, 15 | ] 16 | } 17 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "os" 6 | "os/signal" 7 | "syscall" 8 | "time" 9 | 10 | "github.com/nats-io/nats.go" 11 | ) 12 | 13 | func main() { 14 | nc, err := nats.Connect(os.Args[1], nats.Name("JS sub test"), nats.UserCredentials(os.Args[2])) 15 | defer nc.Close() 16 | if err != nil { 17 | fmt.Printf("nats connect: %v\n", err) 18 | return 19 | } 20 | js, err := nc.JetStream(nats.APIPrefix("from.test.API")) 21 | if err != nil { 22 | fmt.Printf("JetStream: %v\n", err) 23 | if js == nil { 24 | return 25 | } 26 | } 27 | s, err := js.PullSubscribe("test", "DUR", nats.Bind("aggregate", "DUR")) 28 | if err != nil { 29 | fmt.Printf("PullSubscribe: %v\n", err) 30 | return 31 | } 32 | 33 | shutdown := make(chan os.Signal, 1) 34 | signal.Notify(shutdown, os.Interrupt, syscall.SIGTERM) 35 | 36 | fmt.Printf("starting\n") 37 | for { 38 | select { 39 | case <-shutdown: 40 | return 41 | default: 42 | if m, err := s.Fetch(1, nats.MaxWait(time.Second)); err != nil { 43 | fmt.Println(err) 44 | } else { 45 | 46 | if meta, err := m[0].Metadata(); err == nil { 47 | fmt.Printf("%+v\n", meta) 48 | } 49 | fmt.Println(string(m[0].Data)) 50 | 51 | if err := m[0].Ack(); err != nil { 52 | fmt.Printf("ack error: %+v\n", err) 53 | } 54 | } 55 | } 56 | } 57 | } 58 | -------------------------------------------------------------------------------- /nats-account-resolver.cfg: -------------------------------------------------------------------------------- 1 | operator: "./store/OP/OP.jwt" 2 | resolver: { 3 | type: full 4 | dir: $CACHE 5 | interval: "2m" 6 | allow_delete: true 7 | } 8 | resolver_preload: { 9 | ADECCNBUEBWZ727OMBFSN7OMK2FPYRM52TJS25TFQWYS76NPOJBN3KU4:eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiJMTFpUU0paUTNGTkZWNlhDVDNBRkdCSlZWU0FaSk9IN0JSNlFETUdNVUdETktVWUlQR0hRIiwiaWF0IjoxNjI0NDc5NDgyLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJTWVMiLCJzdWIiOiJBREVDQ05CVUVCV1o3MjdPTUJGU043T01LMkZQWVJNNTJUSlMyNVRGUVdZUzc2TlBPSkJOM0tVNCIsIm5hdHMiOnsibGltaXRzIjp7InN1YnMiOi0xLCJkYXRhIjotMSwicGF5bG9hZCI6LTEsImltcG9ydHMiOi0xLCJleHBvcnRzIjotMSwid2lsZGNhcmRzIjp0cnVlLCJjb25uIjotMSwibGVhZiI6LTF9LCJzaWduaW5nX2tleXMiOlsiQUFDWUlDT0FRTVE3MkVIVDM1UjdMVjZWRldNSVZXRktXRkU1UDJKSjJUVDY3NEVPN0RKVFVITU0iXSwiZGVmYXVsdF9wZXJtaXNzaW9ucyI6eyJwdWIiOnt9LCJzdWIiOnt9fSwidHlwZSI6ImFjY291bnQiLCJ2ZXJzaW9uIjoyfX0.tHteTcshVIInToM6LQ7G2AmmWfeKYCCjJdCC4ZfJ1WUtmY1Bk0sEwwbjb6uSycEb4ljohMQnQVgkbAYZsiqZDw 10 | } 11 | -------------------------------------------------------------------------------- /outline.txt: -------------------------------------------------------------------------------- 1 | Outline: 2 | 1. Introduce the leafnode setup used here 3 | 2. Talk about the implications of connecting the system account as leaf node remote 4 | 3. Introduce the concept of JetStream domains 5 | 4. Use stream mirrors to connect a command and control stream across domains 6 | 5. Use stream source to aggregate streams across domains 7 | 6. Demonstrate that domain connectivity is not tied to the underlying topology 8 | 7. Connect streams across accounts 9 | 8. Connect streams through accounts 10 | -------------------------------------------------------------------------------- /puml/topology1.puml: -------------------------------------------------------------------------------- 1 | @startuml 2 | left to right direction 3 | frame "cluster-hub" as hub { 4 | node "srv-4222" as s1 5 | node "srv-4232" as s2 6 | node "srv-4282" as s3 7 | 8 | s1<--->s2 : route 9 | s2<--->s3 : route 10 | s3<--->s1 : route 11 | } 12 | frame "cluster-spoke-1" as spoke1 { 13 | node "srv-4242" as s4 14 | node "srv-4252" as s5 15 | node "srv-4292" as s6 16 | 17 | s4<--->s5 : route 18 | s5<--->s6 : route 19 | s6<--->s4 : route 20 | } 21 | frame "cluster-spoke-2" as spoke2 { 22 | node "srv-4262" as s7 23 | node "srv-4272" as s8 24 | node "srv-4202" as s9 25 | 26 | s7<--->s8 : route 27 | s8<--->s9 : route 28 | s9<--->s7 : route 29 | } 30 | hub <---> spoke1 : leafnode remote 31 | hub <---> spoke2 : leafnode remote 32 | @enduml -------------------------------------------------------------------------------- /puml/topology2-js-merged.puml: -------------------------------------------------------------------------------- 1 | @startuml 2 | left to right direction 3 | database "JetStream" as js { 4 | frame "cluster-hub" as hub { 5 | node "srv-4222" as s1 6 | node "srv-4232" as s2 7 | node "srv-4282" as s3 8 | 9 | s1<--->s2 : route 10 | s2<--->s3 : route 11 | s3<--->s1 : route 12 | } 13 | frame "cluster-spoke-1" as spoke1 { 14 | node "srv-4242" as s4 15 | node "srv-4252" as s5 16 | node "srv-4292" as s6 17 | 18 | s4<--->s5 : route 19 | s5<--->s6 : route 20 | s6<--->s4 : route 21 | } 22 | frame "cluster-spoke-2" as spoke2 { 23 | node "srv-4262" as s7 24 | node "srv-4272" as s8 25 | node "srv-4202" as s9 26 | 27 | s7<--->s8 : route 28 | s8<--->s9 : route 29 | s9<--->s7 : route 30 | } 31 | 32 | hub <---> spoke1 : leafnode remote 33 | hub <---> spoke2 : leafnode remote 34 | } 35 | 36 | @enduml -------------------------------------------------------------------------------- /puml/topology3-js-domains.puml: -------------------------------------------------------------------------------- 1 | @startuml 2 | left to right direction 3 | database "JetStream domain hub" as jshub { 4 | frame "cluster-hub" as hub { 5 | node "srv-4222" as s1 6 | node "srv-4232" as s2 7 | node "srv-4282" as s3 8 | 9 | s1<--->s2 : route 10 | s2<--->s3 : route 11 | s3<--->s1 : route 12 | } 13 | } 14 | 15 | database "JetStream domain spoke-1" as jsspoke1 { 16 | frame "cluster-spoke-1" as spoke1 { 17 | node "srv-4242" as s4 18 | node "srv-4252" as s5 19 | node "srv-4292" as s6 20 | 21 | s4<--->s5 : route 22 | s5<--->s6 : route 23 | s6<--->s4 : route 24 | } 25 | } 26 | 27 | database "JetStream domain spoke-2" as jsspoke2 { 28 | frame "cluster-spoke-2" as spoke2 { 29 | node "srv-4262" as s7 30 | node "srv-4272" as s8 31 | node "srv-4202" as s9 32 | 33 | s7<--->s8 : route 34 | s8<--->s9 : route 35 | s9<--->s7 : route 36 | } 37 | } 38 | hub <---> spoke1 : leafnode remote 39 | hub <---> spoke2 : leafnode remote 40 | @enduml -------------------------------------------------------------------------------- /puml/topology4-js-streams-mirror.puml: -------------------------------------------------------------------------------- 1 | @startuml 2 | left to right direction 3 | database "JetStream domain hub" as jshub { 4 | queue cnc 5 | } 6 | 7 | database "JetStream domain spoke-1" as jsspoke1 { 8 | queue "recv-cnc" as test1 9 | } 10 | 11 | database "JetStream domain spoke-2" as jsspoke2 { 12 | queue "recv-cnc" as test2 13 | } 14 | 15 | cnc ---> test1 : mirror 16 | cnc ---> test2 : mirror 17 | @enduml -------------------------------------------------------------------------------- /puml/topology5-js-streams-source.puml: -------------------------------------------------------------------------------- 1 | @startuml 2 | left to right direction 3 | database "JetStream domain hub" as jshub { 4 | queue aggregate 5 | } 6 | 7 | database "JetStream domain spoke-1" as jsspoke1 { 8 | queue "test" as test1 9 | } 10 | 11 | database "JetStream domain spoke-2" as jsspoke2 { 12 | queue "test" as test2 13 | } 14 | 15 | aggregate <--- test1 : source 16 | aggregate <--- test2 : source 17 | @enduml -------------------------------------------------------------------------------- /puml/topology6-js-streams-backup.puml: -------------------------------------------------------------------------------- 1 | @startuml 2 | left to right direction 3 | database "JetStream domain hub" as jshub { 4 | queue aggregate 5 | } 6 | 7 | database "JetStream domain spoke-1" as jsspoke1 { 8 | queue "test" as test1 9 | } 10 | 11 | database "JetStream domain spoke-2" as jsspoke2 { 12 | queue "test" as test2 13 | queue "backup-test-spoke-1" as backup 14 | } 15 | 16 | aggregate <--- test1 : source 17 | aggregate <--- test2 : source 18 | test1 ---> backup : mirror 19 | @enduml -------------------------------------------------------------------------------- /puml/topology7-mirror-cross-account.puml: -------------------------------------------------------------------------------- 1 | @startuml 2 | left to right direction 3 | database "JetStream domain hub" as jshub { 4 | package "account: TEST" { 5 | queue aggregate 6 | } 7 | package "account: IMPORTER" { 8 | queue crossacc 9 | } 10 | } 11 | 12 | crossacc <--- aggregate : mirror 13 | @enduml 14 | -------------------------------------------------------------------------------- /puml/topology8-consume-cross-account.puml: -------------------------------------------------------------------------------- 1 | @startuml 2 | left to right direction 3 | package "account: IMPORTER" { 4 | actor main 5 | } 6 | database "JetStream domain hub" as jshub { 7 | package "account: TEST" { 8 | queue aggregate 9 | } 10 | } 11 | 12 | main <--- aggregate : mirror 13 | @enduml 14 | -------------------------------------------------------------------------------- /puml/topology9-source-cross-domain-account.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nats-io/jetstream-leaf-nodes-demo/15a5057ba82448307fa2344334d8a4a4a7006263/puml/topology9-source-cross-domain-account.png -------------------------------------------------------------------------------- /puml/topology9-source-cross-domain-account.puml: -------------------------------------------------------------------------------- 1 | @startuml 2 | left to right direction 3 | database "JetStream domain hub" as jshub { 4 | package "account: HUBACC" { 5 | queue aggregate 6 | } 7 | } 8 | 9 | package "account: EXCACC" as dmz #white { 10 | label "listed in\nleafnode remotes" as l1 11 | } 12 | 13 | database "JetStream domain spoke-1" as spoke1 { 14 | package "account: LEAFACC" as e1 { 15 | queue test as test1 16 | } 17 | dmz <--- test1 : "export: \nJetStream leafacc.spoke-1" 18 | } 19 | 20 | database "JetStream domain spoke-2" as spoke2 { 21 | package "account: LEAFACC" as e2 { 22 | queue test as test2 23 | } 24 | dmz <--- test2 : "export: \nJetStream leafacc.spoke-2" 25 | } 26 | 27 | aggregate <--- dmz : "import & source: \nJetStream leafacc.spoke-1" 28 | aggregate <--- dmz : "import & source: \nJetStream leafacc.spoke-2" 29 | @enduml 30 | -------------------------------------------------------------------------------- /store/OP/.nsc: -------------------------------------------------------------------------------- 1 | {"managed":false,"name":"OP","kind":"operator","version":"1"} -------------------------------------------------------------------------------- /store/OP/OP.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiJUM0FCQlVDUkZLRFBRSjJBRU9YUFIzWEg2UFZLR0RRWFVCQ1VPVURXQVgzTFhSNjZCWVZBIiwiaWF0IjoxNjI0NDgwNTkxLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJPUCIsInN1YiI6Ik9ETkVCN0RJS0w2VDRRNjJNU0VSMkQySENDTk5JNTVaRkxQSlU2QzQ0QVFURDVPSU9QSFRMRDVRIiwibmF0cyI6eyJhY2NvdW50X3NlcnZlcl91cmwiOiJuYXRzOi8vbG9jYWxob3N0OjQyMjIsbmF0czovL2xvY2FsaG9zdDo0MjMyLG5hdHM6Ly9sb2NhbGhvc3Q6NDI4MiIsInN5c3RlbV9hY2NvdW50IjoiQURFQ0NOQlVFQldaNzI3T01CRlNON09NSzJGUFlSTTUyVEpTMjVURlFXWVM3Nk5QT0pCTjNLVTQiLCJ0eXBlIjoib3BlcmF0b3IiLCJ2ZXJzaW9uIjoyfX0.8m4AdKEgqyAVyeF0qBIOk21802JXZmM6qBrDyezXQZvawLqiidnKEfg0OYnhbnAoDGgekNuq9ZAtGFsYlLC_AA -------------------------------------------------------------------------------- /store/OP/accounts/SYS/SYS.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiJMTFpUU0paUTNGTkZWNlhDVDNBRkdCSlZWU0FaSk9IN0JSNlFETUdNVUdETktVWUlQR0hRIiwiaWF0IjoxNjI0NDc5NDgyLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJTWVMiLCJzdWIiOiJBREVDQ05CVUVCV1o3MjdPTUJGU043T01LMkZQWVJNNTJUSlMyNVRGUVdZUzc2TlBPSkJOM0tVNCIsIm5hdHMiOnsibGltaXRzIjp7InN1YnMiOi0xLCJkYXRhIjotMSwicGF5bG9hZCI6LTEsImltcG9ydHMiOi0xLCJleHBvcnRzIjotMSwid2lsZGNhcmRzIjp0cnVlLCJjb25uIjotMSwibGVhZiI6LTF9LCJzaWduaW5nX2tleXMiOlsiQUFDWUlDT0FRTVE3MkVIVDM1UjdMVjZWRldNSVZXRktXRkU1UDJKSjJUVDY3NEVPN0RKVFVITU0iXSwiZGVmYXVsdF9wZXJtaXNzaW9ucyI6eyJwdWIiOnt9LCJzdWIiOnt9fSwidHlwZSI6ImFjY291bnQiLCJ2ZXJzaW9uIjoyfX0.tHteTcshVIInToM6LQ7G2AmmWfeKYCCjJdCC4ZfJ1WUtmY1Bk0sEwwbjb6uSycEb4ljohMQnQVgkbAYZsiqZDw -------------------------------------------------------------------------------- /store/OP/accounts/SYS/users/sys.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiJSVk1PWExSM0tRNTNNRjJYS0NUVkw0NlpCVFpZT0RUWDdFNDVEWUxOUVpON0pJM05MQTRRIiwiaWF0IjoxNjI0NDc5NDgyLCJpc3MiOiJBQUNZSUNPQVFNUTcyRUhUMzVSN0xWNlZGV01JVldGS1dGRTVQMkpKMlRUNjc0RU83REpUVUhNTSIsIm5hbWUiOiJzeXMiLCJzdWIiOiJVQUhONTdEUVdHTE9KWUVON0U3SVdTSDRCNzVTRElVQzZZTVFWRTNHNkMySVBPVlE0NEc2M0dUTSIsIm5hdHMiOnsicHViIjp7fSwic3ViIjp7fSwic3VicyI6LTEsImRhdGEiOi0xLCJwYXlsb2FkIjotMSwiaXNzdWVyX2FjY291bnQiOiJBREVDQ05CVUVCV1o3MjdPTUJGU043T01LMkZQWVJNNTJUSlMyNVRGUVdZUzc2TlBPSkJOM0tVNCIsInR5cGUiOiJ1c2VyIiwidmVyc2lvbiI6Mn19.CD3eqMOyrRw6cDNoKGisHbLtCCw8D9wxtvNbbz1UcZcPRjf2XBtpd1pbVcartxakADiyQAK5Rhx7-TViTie4Dw -------------------------------------------------------------------------------- /store/OP/accounts/TEST/TEST.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiIyNjZKUEFFQlMzVUdWQVVJRElNWFlMSzdVQ0lXUFdWU0VTS1dDTkRMVFFNRlVMTFlJQTRRIiwiaWF0IjoxNjI2ODIzMTYxLCJpc3MiOiJPRE5FQjdESUtMNlQ0UTYyTVNFUjJEMkhDQ05OSTU1WkZMUEpVNkM0NEFRVEQ1T0lPUEhUTEQ1USIsIm5hbWUiOiJURVNUIiwic3ViIjoiQUE1QzU2RkFFVEJUVUNZTTdOQzVCRkJZRlRLTE9BQklPSUZQUURITzRSVUVBUFNOM0ZUWTVSNEciLCJuYXRzIjp7ImxpbWl0cyI6eyJzdWJzIjotMSwiZGF0YSI6LTEsInBheWxvYWQiOi0xLCJpbXBvcnRzIjotMSwiZXhwb3J0cyI6LTEsIndpbGRjYXJkcyI6dHJ1ZSwiY29ubiI6LTEsImxlYWYiOi0xLCJtZW1fc3RvcmFnZSI6LTEsImRpc2tfc3RvcmFnZSI6LTEsInN0cmVhbXMiOi0xLCJjb25zdW1lciI6LTF9LCJkZWZhdWx0X3Blcm1pc3Npb25zIjp7InB1YiI6e30sInN1YiI6e319LCJ0eXBlIjoiYWNjb3VudCIsInZlcnNpb24iOjJ9fQ.hM-Mi1HkROLNTEPrZEs-XbN_TeTJ40ZoYrQYHwqlyzwx1sT4tM0ku37hAizslYH6Wf-tUA-GQWmEOCu_Ba0JDA -------------------------------------------------------------------------------- /store/OP/accounts/TEST/users/leaf.jwt: -------------------------------------------------------------------------------- 1 | eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiJDS1JJN0hTTUw1SlJYQVZXVFVKN1dMQ0ROWDdaTzZWUEtZNjZYWDMyTkwyU0ZKU05MNEhRIiwiaWF0IjoxNjI0NDc5NTQ0LCJpc3MiOiJBQTVDNTZGQUVUQlRVQ1lNN05DNUJGQllGVEtMT0FCSU9JRlBRREhPNFJVRUFQU04zRlRZNVI0RyIsIm5hbWUiOiJsZWFmIiwic3ViIjoiVUFUVFNCTzVDWU9TVUpaTVoyM0YzNDVHVU9RRldBNVdGQzVTMkUyUUdUS1JMUjc2QkpURU5ORFoiLCJuYXRzIjp7InB1YiI6e30sInN1YiI6e30sInN1YnMiOi0xLCJkYXRhIjotMSwicGF5bG9hZCI6LTEsInR5cGUiOiJ1c2VyIiwidmVyc2lvbiI6Mn19.-SbW7JhA5prumxJyQy7rn-YwY5qcjTk2lOC4iI2hyqYR_1qEYfzJ0j9J5vPiY40kBrrB09is_F1tQFpW_8InDw -------------------------------------------------------------------------------- /topology1-server.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nats-io/jetstream-leaf-nodes-demo/15a5057ba82448307fa2344334d8a4a4a7006263/topology1-server.png -------------------------------------------------------------------------------- /topology2-js-merged.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nats-io/jetstream-leaf-nodes-demo/15a5057ba82448307fa2344334d8a4a4a7006263/topology2-js-merged.png -------------------------------------------------------------------------------- /topology3-js-domains.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nats-io/jetstream-leaf-nodes-demo/15a5057ba82448307fa2344334d8a4a4a7006263/topology3-js-domains.png -------------------------------------------------------------------------------- /topology4-js-streams-mirror.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nats-io/jetstream-leaf-nodes-demo/15a5057ba82448307fa2344334d8a4a4a7006263/topology4-js-streams-mirror.png -------------------------------------------------------------------------------- /topology5-js-streams-source.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nats-io/jetstream-leaf-nodes-demo/15a5057ba82448307fa2344334d8a4a4a7006263/topology5-js-streams-source.png -------------------------------------------------------------------------------- /topology6-js-streams-backup.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nats-io/jetstream-leaf-nodes-demo/15a5057ba82448307fa2344334d8a4a4a7006263/topology6-js-streams-backup.png -------------------------------------------------------------------------------- /topology7-mirror-cross-account.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nats-io/jetstream-leaf-nodes-demo/15a5057ba82448307fa2344334d8a4a4a7006263/topology7-mirror-cross-account.png -------------------------------------------------------------------------------- /topology8-consume-cross-account.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nats-io/jetstream-leaf-nodes-demo/15a5057ba82448307fa2344334d8a4a4a7006263/topology8-consume-cross-account.png -------------------------------------------------------------------------------- /topology9-source-cross-domain-account.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nats-io/jetstream-leaf-nodes-demo/15a5057ba82448307fa2344334d8a4a4a7006263/topology9-source-cross-domain-account.png --------------------------------------------------------------------------------