├── README.md ├── _assets ├── emqx-4.4.16-fanin-result-charts.png ├── emqx-4.4.16-fanout-result-charts.png ├── emqx-4.4.16-p2p-result-charts.png ├── emqx-5.0.21-fanin-result-charts.png ├── emqx-5.0.21-fanout-result-charts.png ├── emqx-5.0.21-p2p-result-charts.png ├── fanin.png ├── fanout.png ├── mosquitto-2.0.15-fanout-result-charts.png ├── mosquitto-2.0.15-p2p-result-charts.png ├── mqtt_model.png ├── nano-0.17.0-fanin-result-charts.png ├── nano-0.17.0-fanout-result-charts.png ├── nano-0.17.0-p2p-result-charts.png └── p2p.png └── results ├── singlenode-conn-tcp-1M-5K.md ├── singlenode-fanout-5-1000-5-250K.md ├── singlenode-p2p-50K-50K-50K-50K.md └── singlenode-sharesub-50K-500-50K-50K.md /README.md: -------------------------------------------------------------------------------- 1 | # Open MQTT Benchmark Suite 2 | ## Introduction 3 | 4 | MQTT is a communication protocol based on Publish/Subscribe model. It has the following features: 5 | 6 | - Simple and easy to implement 7 | - QoS support for complex device network environment. 8 | - Lightweight and designed specially for unstable network connections and bandwidth saving 9 | - Data irrelevant 10 | - Continuous session awareness 11 | 12 | Those characteristics make it suited for IoT applications, and become one of the most popular protocols in IoT domain. 13 | 14 | In a typical MQTT Publish/Subscribe architecture, there are publishers, subscribers, and a broker in the system. A publisher sends a message to the broker on a specific topic and then the broker sends that message to all subscribers which are subscribed to the same topic. 15 | 16 | ![mqtt-model](_assets/mqtt_model.png) 17 | 18 | As a broker is the central component in an MQTT based IoT application, it plays an important role in the scalability and availability of the application. There have been many research papers and technical writeups on the performance evaluation and comparison of various MQTT brokers. However, the existing benchmarks often measure particular factors only, with scenarios not matching the realistic large-scale IoT applications. Through development and customer support experience of the past decade, we have learnt that many factors impact MQTT broker performance. 19 | 20 | For one thing, MQTT protocol states many features improving reliability and security, which bring overhead cost on broker side as well. Some of the features are: 21 | 22 | - QoS 23 | - Keep Alive 24 | - Clean Session 25 | - Retain 26 | - Topic wildcards 27 | - Authentication method 28 | - TLS authentication enablement 29 | 30 | In addition, different MQTT versions suggest different functions. For example, shared subscription is supported in MQTT 5.0 but not 3.1.1 or 3.1. 31 | 32 | For another thing, MQTT brokers deployment ways influence the performance evaluation. The brokers can be deployed on single node, or in a cluster way. If it is deployed in a cluster way, it can be put behind a load balancer or not. To be comparable, the brokers should be deployed in the similar ways. 33 | 34 | What is more, environment conditions like hardware, OS, system and network configuration also impact the evaluation result. Some of the important conditions are: 35 | 36 | - receive buffer size 37 | - network round trip time 38 | 39 | The rapid development of IoT applications brings new challenges. As a hub of devices and data, the capability of the MQTT broker to support massive device connectivity and data throughput is critical to the IoT business. Consequently, a thorough evaluation on its performance is important in the full IoT solution. The intent of this work is suggesting metrics and evaluation criteria, presenting real-world scenarios and use cases, and in the future, providing a tool that fully compatible with the benchmark suite. To this end, we hope to establish a practical base for benchmarking MQTT brokers, iteratively optimize its coverage, and help users find the most suitable MQTT broker for their needs and workloads. 40 | 41 | ## Methodology 42 | 43 | To measure performance of an MQTT broker, we need to define metrics, which can be looked at in a variety of use cases. Two primary perspectives of measurement are considered. One focuses on the computing resources perspective. The other focuses on the capabilities provided by the MQTT broker itself, and divides into three detailed aspects - scalability, availability, and latency, as suggested in Google “Cloud Pub/Sub” product guide. It should be noticed that different aspects may contradict each other under different circumstances. To achieve different evaluation purposes, user needs to pick the suitable metrics. 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 55 | 59 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 |
Computing resources metricsBroker capability metrics
52 |

Scalability

53 |

the ability to handle increases in load

54 | 		56 |

Availability

57 |

the ability to handle problems and failures

58 | 		60 |

Latency

61 |

the amount of time to deliver a published message to a subscriber

62 |
CPU load and usageMemory usageDisk I/O ratePackets receiving and sending rateRate of new established connectionsNumber of concurrent connectionsNumber of publishersNumber of subscribersNumber of topicsSize of messagesRate of messages publishedRate of messages subscribedSuccess rateReconnection delayAverage latency time90th percentile latency time
83 | 84 | A benchmark tool then is used to simulate publishers and/or subscribers based on use cases. It first generates dataset for benchmark, and loads the dataset to construct a series of MQTT requests, then executes to perform publishing and subscribing. The tool should work in a consistent way regardless of what the MQTT broker is. As many practical use cases involve large number of publishers/subscribers, the tool should also be implemented in a sophisticated way in order to generate desired large-scale load. 85 | 86 | As part of the benchmark submisison, a full disclosure report is suggested. The intent of the report is to simplify comparison between results of different brokers, and to provide enough information so other users can replicate the results of this benchmark. The report should identify the benchmark tool it uses, explain the detailed use cases, and list the hardware, OS and network conditions to perform the benchmark. It should also cover comparison data regarding the measurement metrics, both including computing resources perspective and broker capability perspective. 87 | 88 | ## Scenarios and use cases 89 | 90 | By analyzing realistic requirements from our customers, we categorize the benchmark into the following scenarios: 91 | 92 | - Connection 93 | - Fan-out 94 | - Point-to-point 95 | - Fan-in 96 | - Composite 97 | 98 | Next, we will describe each scenario and give detailed use cases for each scenario. To meet different user needs, different use cases are demanded. For example, some of them can be used for basic verification purpose, while some of them aim for enterprise level verification. Therefore, we group use cases of the same scalability level into one set, to help user pick up the use cases they need more easily. The use cases in each set will be introduced in details immediately after. 99 | 100 | ### Basic set 101 | 102 | This set is for small-scale. It consists of the following use cases: 103 | 104 | | Use case | Description | 105 | | ------------------------------- | ------------------------------------------------------------ | 106 | | singlenode-conn-tcp-10K-100 | Connection scenario for 1,000 clients | 107 | | singlenode-fanout-1-1K-1-1K | Fan-out scenario for 1 pub message per second and 1,000 sub messages per second | 108 | | singlenode-p2p-1K-1K-1K-1K | Point-to-point scenario for 1,000 pub messages per second and 1,000 sub messages per second | 109 | | singlenode-sharedsub-1K-5-1K-1K | Fan-in scenario for 1,000 pub messages per second and 1,000 sub messages per second in shared subscription way | 110 | 111 | ### Enterprise set 112 | 113 | This set is for large-scale. It consists of the following use cases: 114 | 115 | | Use case | Description | 116 | | ------------------------------------ | ------------------------------------------------------------ | 117 | | singlenode-conn-tcp-1M-5K | Connection scenario for 1,000,000 clients | 118 | | singlenode-fanout-5-1000-5-250K | Fan-out scenario for 5 pub messages per second and 250,000 sub messages per second | 119 | | singlenode-p2p-50K-50K-50K-50K | Point-to-point scenario for 50,000 pub messages per second and 50,000 sub messages per second | 120 | | singlenode-sharedsub-50K-500-50K-50K | Fan-in scenario for 50,000 pub messages per second and 50,000 sub messages per second in shared subscription way | 121 | 122 | Here comes the detailed use cases. The primary use cases focus on those executed on single-node-brokers, and in the future, we will add more supplemental use cases for clustering-brokers. 123 | 124 | ### Connection 125 | 126 | In a connection scenario, a batch of clients connect to the broker within a period of time, and keep the connections with the broker for quite a while. 127 | 128 | 2 use cases of this scenario for brokers deployed on single node are presented as: 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 147 | 148 | 149 | 150 | 157 | 158 | 159 | 160 | 167 | 168 |
Use casesinglenode-conn-tcp-10K-100
Description10,000 clients simultaneously connect to broker
Details 142 |
    143 |
  1. 10,000 clients connect to the broker within 100 seconds. Each client connects to the TCP port of the broker, using MQTT 3.1.1 protocol. The Keep Alive property is set as 300, and Clean Session is set as 1.
    2. 144 |
  2. Keep all clients online for 30 minutes.
    3. 145 |
146 |
Computing resource metrics of interest 151 |
    152 |
  • CPU load and usage
    • 153 |
  • Memory usage
    • 154 |
  • Disk I/O rate
    • 155 |
156 |
Broker capability metrics of interest 161 |
    162 |
  • Rate of new established connections
    • 163 |
  • Number of concurrent connections
    • 164 |
  • Success rate
    • 165 |
166 |
169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 187 | 188 | 189 | 190 | 197 | 198 | 199 | 200 | 207 | 208 |
Use casesinglenode-conn-tcp-1M-5K
Description1,000,000 clients simultaneously connect to broker
Details 182 |
    183 |
  1. 1,000,000 clients connect to the broker within 200 seconds. Each client connects to the TCP port of the broker, using MQTT 3.1.1 protocol. The Keep Alive property is set as 300, and Clean Session is set as 1.
    2. 184 |
  2. Keep all clients online for 30 minutes.
    3. 185 |
186 |
Computing resource metrics of interest 191 |
    192 |
  • CPU load and usage
    • 193 |
  • Memory usage
    • 194 |
  • Disk I/O rate
    • 195 |
196 |
Broker capability metrics of interest 201 |
    202 |
  • Rate of new established connections
    • 203 |
  • Number of concurrent connections
    • 204 |
  • Success rate
    • 205 |
206 |
209 | 210 | ### Fan-out 211 | 212 | In a fan-out scenario, a large number of clients act as subscribers, with only a few or a single publisher. 213 | 214 | ![fan-out](_assets/fanout.png) 215 | 216 | 2 use cases of this scenario for brokers deployed on single node are presented as: 217 | 218 | 219 | 220 | 221 | 222 | 223 | 224 | 225 | 226 | 227 | 228 | 229 | 238 | 239 | 240 | 241 | 249 | 250 | 251 | 252 | 265 | 266 |
Use casesinglenode-fanout-1-1K-1-1K
Description1 publisher publishes messages to 1 topic which are subscribed by 1,000 subscribers
Details 230 |
    231 |
  1. 1,001 clients are divided into publishers and subscribers: 1 as publisher and 1,000 as subscribers. The publishers and subscribers will work on 1 topic.
    2. 232 |
  2. All publishers and subscribers connect to the TCP port of the broker, using MQTT 3.1.1 protocol. The Keep Alive property is set as 300, and Clean Session is set as 1.
    3. 233 |
  3. Once the connection of a subscriber is established, the subscriber immediately subscribes to the topics using QoS 1.
    4. 234 |
  4. When all the connections are established, the publisher publishes message to the topic using QoS 1 with Retain as 0. The publish rate for each publisher is 1 message per second. The payload size of each message is 16 bytes.
    5. 235 |
  5. Keep the publish and subscribe for 30 minutes. The expected total publish rate is 1 message per second, and the expected total subscribe rate is 1,000 messages per second.
    6. 236 |
237 |
Computing resource metrics of interest 242 |
    243 |
  • CPU load and usage
    • 244 |
  • Memory usage
    • 245 |
  • Disk I/O rate
    • 246 |
  • Packets receiving and sending rate
    • 247 |
248 |
Broker capability metrics of interest 253 |
    254 |
  • Number of publishers
    • 255 |
  • Number of subscribers
    • 256 |
  • Number of topics
    • 257 |
  • Size of messages
    • 258 |
  • Rate of messages published
    • 259 |
  • Rate of messages subscribed
    • 260 |
  • Success rate
    • 261 |
  • Average latency time
    • 262 |
  • 90th percentile latency time
    • 263 |
264 |
267 | 268 | 269 | 270 | 271 | 272 | 273 | 274 | 275 | 276 | 277 | 278 | 279 | 288 | 289 | 290 | 291 | 299 | 300 | 301 | 302 | 315 | 316 |
Use casesinglenode-fanout-5-1000-5-250K
Description50 publishers publish messages to 5 topics which are subscribed by 1,000 subscribers
Details 280 |
    281 |
  1. 1,005 clients are divided into publishers and subscribers: 5 as publishers and 1,000 as subscribers. The publishers and subscribers will work on 5 topics.
    2. 282 |
  2. All publishers and subscribers connect to the TCP port of the broker, using MQTT 3.1.1 protocol. The Keep Alive property is set as 300, and Clean Session is set as 1.
    3. 283 |
  3. Once the connection of a subscriber is established, the subscriber immediately subscribes to all 5 topics using QoS 1.
    4. 284 |
  4. When all the connections are established, each publisher publishes message to an exclusive topic using QoS 1 with Retain as 0. The publish rate for each publisher is 50 messages per second. The payload size of each message is 16 bytes.
    5. 285 |
  5. Keep the publish and subscribe for 30 minutes. The expected total publish rate is 250 messages per second, and the expected total subscribe rate is 250,000 messages per second.
    6. 286 |
287 |
Computing resource metrics of interest 292 |
    293 |
  • CPU load and usage
    • 294 |
  • Memory usage
    • 295 |
  • Disk I/O rate
    • 296 |
  • Packets receiving and sending rate
    • 297 |
298 |
Broker capability metrics of interest 303 |
    304 |
  • Number of publishers
    • 305 |
  • Number of subscribers
    • 306 |
  • Number of topics
    • 307 |
  • Size of messages
    • 308 |
  • Rate of messages published
    • 309 |
  • Rate of messages subscribed
    • 310 |
  • Success rate
    • 311 |
  • Average latency time
    • 312 |
  • 90th percentile latency time
    • 313 |
314 |
317 | 318 | ### Point-to-point 319 | 320 | In a point-to-point scenario, the equal number of clients act as publishers and subscribers respectively. 321 | 322 | ![point-to-point](_assets/p2p.png) 323 | 324 | 2 use cases of this scenario for brokers deployed on single node are presented as: 325 | 326 | 327 | 328 | 329 | 330 | 331 | 332 | 333 | 334 | 335 | 336 | 337 | 347 | 348 | 349 | 350 | 358 | 359 | 360 | 361 | 374 | 375 |
Use casesinglenode-p2p-1K-1K-1K-1K
Description1,000 publishers publish messages to 1,000 topics which are subscribed by 1,000 subscribers
Details 338 |
    339 |
  1. 2,000 clients are divided into publishers and subscribers: 1,000 as publishers and 1,000 as subscribers. The publishers and subscribers will work on 1,000 topics.
    2. 340 | 341 | ​
  2. All publishers and subscribers connect to the TCP port of the broker, using MQTT 3.1.1 protocol. The Keep Alive property is set as 300, and Clean Session is set as 1.
    3. 342 |
  3. Once the connection of a subscriber is established, the subscriber immediately subscribes a topic using QoS 1. Different subscribers subscribe to different topics.
    4. 343 |
  4. When all the connections are established, each publisher publishes messages to a topic using QoS 1 with Retain as 0. Different publishers publish to different topics. The publish rate for each publisher is 1 message per second. The payload size of each message is 16 bytes.
    5. 344 |
  5. Keep the publish and subscribe for 30 minutes. The expected total publish rate is 1,000 messages per second, and the expected total subscribe rate is 1,000 messages per second.
    6. 345 |
346 |
Computing resource metrics of interest 351 |
    352 |
  • CPU load and usage
    • 353 |
  • Memory usage
    • 354 |
  • Disk I/O rate
    • 355 |
  • Packets receiving and sending rate
    • 356 |
357 |
Broker capability metrics of interest 362 |
    363 |
  • Number of publishers
    • 364 |
  • Number of subscribers
    • 365 |
  • Number of topics
    • 366 |
  • Size of messages
    • 367 |
  • Rate of messages published
    • 368 |
  • Rate of messages subscribed
    • 369 |
  • Success rate
    • 370 |
  • Average latency time
    • 371 |
  • 90th percentile latency time
    • 372 |
373 |
376 | 377 | 378 | 379 | 380 | 381 | 382 | 383 | 384 | 385 | 386 | 387 | 388 | 398 | 399 | 400 | 401 | 409 | 410 | 411 | 412 | 425 | 426 |
Use casesinglenode-p2p-50K-50K-50K-50K
Description50,000 publishers publish messages to 50,000 topics which are subscribed by 50,000 subscribers
Details 389 |
    390 |
  1. 100,000 clients are divided into publishers and subscribers: 50,000 as publishers and 50,000 as subscribers. The publishers and subscribers will work on 50,000 topics.
    2. 391 | 392 | ​
  2. All publishers and subscribers connect to the TCP port of the broker, using MQTT 3.1.1 protocol. The Keep Alive property is set as 300, and Clean Session is set as 1.
    3. 393 |
  3. Once the connection of a subscriber is established, the subscriber immediately subscribes a topic using QoS 1. Different subscribers subscribe to different topics.
    4. 394 |
  4. When all the connections are established, each publisher publishes messages to a topic using QoS 1 with Retain as 0. Different publishers publish to different topics. The publish rate for each publisher is 1 message per second. The payload size of each message is 16 bytes.
    5. 395 |
  5. Keep the publish and subscribe for 30 minutes. The expected total publish rate is 50,000 messages per second, and the expected total subscribe rate is 50,000 messages per second.
    6. 396 |
397 |
Computing resource metrics of interest 402 |
    403 |
  • CPU load and usage
    • 404 |
  • Memory usage
    • 405 |
  • Disk I/O rate
    • 406 |
  • Packets receiving and sending rate
    • 407 |
408 |
Broker capability metrics of interest 413 |
    414 |
  • Number of publishers
    • 415 |
  • Number of subscribers
    • 416 |
  • Number of topics
    • 417 |
  • Size of messages
    • 418 |
  • Rate of messages published
    • 419 |
  • Rate of messages subscribed
    • 420 |
  • Success rate
    • 421 |
  • Average latency time
    • 422 |
  • 90th percentile latency time
    • 423 |
424 |
427 | 428 | ### Fan-in 429 | 430 | A fan-in scenario is the opposite of fan-out. In a fan-in scenario, a large number of clients act as publishers, with only a few or a single subscriber. 431 | 432 | ![fan-in](_assets/fanin.png) 433 | 434 | 2 use cases of this scenario for brokers deployed on single node are presented as: 435 | 436 | 437 | 438 | 439 | 440 | 441 | 442 | 443 | 444 | 445 | 446 | 447 | 457 | 458 | 459 | 460 | 468 | 469 | 470 | 471 | 484 | 485 |
Use casesinglenode-sharedsub-1K-5-1K-1K
Description1,000 publishers publish messages to 1,000 topics which are subscribed in shared subscription way by 5 subscribers
Details 448 |
    449 |
  1. 1,005 clients are divided into publishers and subscribers: 1,000 as publishers and 5 as subscribers. The publishers and subscribers will work on 1,000 topics. The topics are like: test/1, test/2, ..., test/1000.
    2. 450 | 451 | ​
  2. All publishers and subscribers connect to the TCP port of the broker which should support MQTT 5.0 protocol. The Keep Alive property is set as 300, and Clean Session is set as 1.
    3. 452 |
  3. Once the connection of a subscriber is established, the subscriber immediately subscribes to all the topics via shared subscription way using QoS 1. The shared subscription topic used is: $share/benchmark/test/#
    4. 453 |
  4. When all the connections are established, each publisher publishes message to a topic using QoS 1 with Retain as 0. Different publishers publish to different topics. The publish rate for each publisher is 1 message per second. The payload size of each message is 16 bytes.
    5. 454 |
  5. Keep the publish and subscribe for 30 minutes. The expected total publish rate is 1,000 messages per second, and the expected total subscribe rate is 1,000 messages per second.
    6. 455 |
456 |
Computing resource metrics of interest 461 |
    462 |
  • CPU load and usage
    • 463 |
  • Memory usage
    • 464 |
  • Disk I/O rate
    • 465 |
  • Packets receiving and sending rate
    • 466 |
467 |
Broker capability metrics of interest 472 |
    473 |
  • Number of publishers
    • 474 |
  • Number of subscribers
    • 475 |
  • Number of topics
    • 476 |
  • Size of messages
    • 477 |
  • Rate of messages published
    • 478 |
  • Rate of messages subscribed
    • 479 |
  • Success rate
    • 480 |
  • Average latency time
    • 481 |
  • 90th percentile latency time
    • 482 |
483 |
486 | 487 | 488 | 489 | 490 | 491 | 492 | 493 | 494 | 495 | 496 | 497 | 498 | 508 | 509 | 510 | 511 | 519 | 520 | 521 | 522 | 535 | 536 |
Use casesinglenode-sharedsub-50K-500-50K-50K
Description50,000 publishers publish messages to 50,000 topics which are subscribed in shared subscription way by 500 subscribers
Details 499 |
    500 |
  1. 50,500 clients are divided into publishers and subscribers: 50,000 as publishers and 500 as subscribers. The publishers and subscribers will work on 50,000 topics. The topics are like: test/1, test/2, ..., test/50000.
    2. 501 | 502 | ​
  2. All publishers and subscribers connect to the TCP port of the broker which should support MQTT 5.0 protocol. The Keep Alive property is set as 300, and Clean Session is set as 1.
    3. 503 |
  3. Once the connection of a subscriber is established, the subscriber immediately subscribes to all the topics via shared subscription way using QoS 1. The shared subscription topic used is: $share/benchmark/test/#
    4. 504 |
  4. When all the connections are established, each publisher publishes message to a topic using QoS 1 with Retain as 0. Different publishers publish to different topics. The publish rate for each publisher is 1 message per second. The payload size of each message is 16 bytes.
    5. 505 |
  5. Keep the publish and subscribe for 30 minutes. The expected total publish rate is 50,000 messages per second, and the expected total subscribe rate is 50,000 messages per second.
    6. 506 |
507 |
Computing resource metrics of interest 512 |
    513 |
  • CPU load and usage
    • 514 |
  • Memory usage
    • 515 |
  • Disk I/O rate
    • 516 |
  • Packets receiving and sending rate
    • 517 |
518 |
Broker capability metrics of interest 523 |
    524 |
  • Number of publishers
    • 525 |
  • Number of subscribers
    • 526 |
  • Number of topics
    • 527 |
  • Size of messages
    • 528 |
  • Rate of messages published
    • 529 |
  • Rate of messages subscribed
    • 530 |
  • Success rate
    • 531 |
  • Average latency time
    • 532 |
  • 90th percentile latency time
    • 533 |
534 |
537 | 538 | ### Composite 539 | 540 | A composite scenario combines scenarios involving connection/publish/subscribe. Usually, a large number of clients connect to the broker, with most of them playing roles of background connections, and the rest of them perform fan-out, point-to-point, or fan-in scenario. 541 | 542 | We welcome contributions from the community to enrich use cases! 543 | 544 | ## Benchmarking results 545 | 546 | We execute benchmarking on several MQTT brokers (including EMQX, Mosquitto, NanoMQ, etc.), using XMeter as the benchmark tool. For each use case involved, XMeter simulates a huge number of MQTT clients, executes expected publishing/subscription, and generates reports covering the primary performance metrics. 547 | 548 | For detailed benchmarking information and result, please refer to the following pages: 549 | 550 | [Benchmarking results of concurrent connection](results/singlenode-conn-tcp-1M-5K.md) 551 | 552 | [Benchmarking results of fan-out](results/singlenode-fanout-5-1000-5-250K.md) 553 | 554 | [Benchmarking results of point-to-point](results/singlenode-p2p-50K-50K-50K-50K.md) 555 | 556 | [Benchmarking results of fan-in](results/singlenode-sharesub-50K-500-50K-50K.md) -------------------------------------------------------------------------------- /_assets/emqx-4.4.16-fanin-result-charts.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/emqx-4.4.16-fanin-result-charts.png -------------------------------------------------------------------------------- /_assets/emqx-4.4.16-fanout-result-charts.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/emqx-4.4.16-fanout-result-charts.png -------------------------------------------------------------------------------- /_assets/emqx-4.4.16-p2p-result-charts.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/emqx-4.4.16-p2p-result-charts.png -------------------------------------------------------------------------------- /_assets/emqx-5.0.21-fanin-result-charts.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/emqx-5.0.21-fanin-result-charts.png -------------------------------------------------------------------------------- /_assets/emqx-5.0.21-fanout-result-charts.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/emqx-5.0.21-fanout-result-charts.png -------------------------------------------------------------------------------- /_assets/emqx-5.0.21-p2p-result-charts.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/emqx-5.0.21-p2p-result-charts.png -------------------------------------------------------------------------------- /_assets/fanin.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/fanin.png -------------------------------------------------------------------------------- /_assets/fanout.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/fanout.png -------------------------------------------------------------------------------- /_assets/mosquitto-2.0.15-fanout-result-charts.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/mosquitto-2.0.15-fanout-result-charts.png -------------------------------------------------------------------------------- /_assets/mosquitto-2.0.15-p2p-result-charts.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/mosquitto-2.0.15-p2p-result-charts.png -------------------------------------------------------------------------------- /_assets/mqtt_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/mqtt_model.png -------------------------------------------------------------------------------- /_assets/nano-0.17.0-fanin-result-charts.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/nano-0.17.0-fanin-result-charts.png -------------------------------------------------------------------------------- /_assets/nano-0.17.0-fanout-result-charts.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/nano-0.17.0-fanout-result-charts.png -------------------------------------------------------------------------------- /_assets/nano-0.17.0-p2p-result-charts.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/nano-0.17.0-p2p-result-charts.png -------------------------------------------------------------------------------- /_assets/p2p.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/p2p.png -------------------------------------------------------------------------------- /results/singlenode-conn-tcp-1M-5K.md: -------------------------------------------------------------------------------- 1 | # Benchmark results of singlenode-conn-tcp-1M-5K 2 | 3 | ## Brokers 4 | 5 | | Opensource **Broker** | **Version** | 6 | | --------------------- | ----------- | 7 | | EMQX 4 | 4.4.16 | 8 | | EMQX 5 | 5.0.21 | 9 | | Mosquitto | 2.0.15 | 10 | | NanoMQ | 0.17.0 | 11 | 12 | ## Benchmark use case: singlenode-conn-tcp-1M-5K 13 | 14 | In this testcase, 1,000,000 clients connect to the broker within 200 seconds (e.g. cps 5K) and all clients are kept online for 30 minutes. 15 | 16 | MQTT protocol 3.1.1 is used for the use case, and other configured MQTT properties are: 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 |
TLS authentication enablementNo
Authentication enablementNo
Keep Alive (seconds)300
Clean Session1
36 | 37 | ## Testbed 38 | 39 | The use case is executed on single node. XMeter (version 3.2.4) is used as the benchmark tool. 40 | 41 | **HW Details** 42 | 43 | - public cloud: AWS 44 | - instance type: c5.4xlarge 16C32G 45 | - OS: Ubuntu 22.04.1 amd64 46 | 47 | ## Metrics 48 | 49 | | | Average latency (ms) | Max CPU user+system | Avg CPU user+system | Max memory used | Avg memory used | 50 | | ------------- | --------------------- | ------------------- | ------------------- | --------------- | --------------- | 51 | | **EMQX 4** | 2.21 | 44% | 31% | 10.87G | 8.68G | 52 | | **EMQX 5** | 2.25 | 52% | 24% | 16.1G | 12.1G | 53 | | **Mosquitto** | 5.61 | 2% | 2% | 1G | 1G | 54 | | **NanoMQ** | 2.99 | 5% | 4% | 10.6G | 10.6G | 55 | 56 | ### Appendix I: result charts 57 | 58 | #### EMQX 4 59 | 60 | #### EMQX 5 61 | 62 | #### Mosquitto 63 | 64 | #### NanoMQ 65 | 66 | 67 | 68 | ### Appendix II: monitoring charts 69 | 70 | #### Mosquitto 71 | 72 | 截屏2023-04-10上午9 55 53 73 | 74 | #### NanoMQ 75 | 76 | 截屏2023-04-10上午9 57 59 77 | 78 | -------------------------------------------------------------------------------- /results/singlenode-fanout-5-1000-5-250K.md: -------------------------------------------------------------------------------- 1 | # Benchmark results of UC: singlenode-fanout-5-1000-5-250K 2 | 3 | ## Brokers 4 | 5 | | Opensource **Broker** | **Version** | 6 | | --------------------- | ----------- | 7 | | EMQX 4 | 4.4.16 | 8 | | EMQX 5 | 5.0.21 | 9 | | Mosquitto | 2.0.15 | 10 | | NanoMQ | 0.17.0 | 11 | 12 | ## Benchmark use case: singlenode-fanout-5-1000-5-250K 13 | 14 | In this use case, 5 publishers publish messages to 5 topics, and 1,000 subscribers subscribe messages (each subscriber subscribs to all 5 topics). Specially, the use case is executed in the following way: 15 | 16 | 1. All publishers and subscribers connect to the TCP port of the broker. 17 | 2. Once the connection of a subscriber is established, the subscriber immediately subscribes to all 5 topics using QoS 1. 18 | 3. When all the connections are established, each publisher publishes message to a topic using QoS 1 with Retain as 0. Different publishers subscribe to different topics. The publish rate for each publisher is 50 messages per second. The payload size of each message is 16 bytes. 19 | 4. Keep the publish and subscribe for 30 minutes. The expected total publish rate is 250 messages per second, and the expected total subscribe rate is 250,000 messages per second. 20 | 21 | 22 | 23 | MQTT protocol 3.1.1 is used for the use case, and other configured MQTT properties are: 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 |
TLS authentication enablementNo
Authentication enablementNo
Keep Alive (seconds)300
Clean Session1
43 | 44 | ## Testbed 45 | 46 | The use case is executed on single node. XMeter (version 3.2.4) is used as the benchmark tool. 47 | 48 | **HW Details** 49 | 50 | - public cloud: AWS 51 | - instance type: c5.4xlarge 16C32G 52 | - OS: Ubuntu 22.04.1 amd64 53 | 54 | ## Metrics 55 | 56 | | | Actual msg rate | Average pub-to-sub latency (ms) | Max CPU user+system | Avg CPU user+system | Max memory used | Avg memory used | 57 | | ------------- | --------------- | ------------------------------- | ------------------- | ------------------- | --------------- | --------------- | 58 | | **EMQX 4** | 250k | 1.98 | 83% | 79% | 517M | 486M | 59 | | **EMQX 5** | 250k | 2.27 | 81% | 79% | 540M | 494M | 60 | | **Mosquitto** | 81k | 12,445 | 7% | 6% | 366M | 351M | 61 | | **NanoMQ** | 250k | 14.07 | 72% | 70% | 824M | 685M | 62 | 63 | > in this scenario, Mosquitto couldn't reach to the designed message rate. The throughput has been fluctuating around 80,000/s. 64 | > 65 | > EMQX 4, EMQX 5 and NanoMQ kept the stable rate at 250,000/s during the test. 66 | 67 | ### Appendix: result charts 68 | 69 | #### EMQX 4 70 | 71 | ![EMQX 4](../_assets/emqx-4.4.16-fanout-result-charts.png) 72 | 73 | #### EMQX 5 74 | 75 | ![EMQX 5](../_assets/emqx-5.0.21-fanout-result-charts.png) 76 | 77 | #### Mosquitto 78 | 79 | ![Mosquitto](../_assets/mosquitto-2.0.15-fanout-result-charts.png) 80 | 81 | #### NanoMQ 82 | 83 | ![NanoMQ](../_assets/nano-0.17.0-fanout-result-charts.png) -------------------------------------------------------------------------------- /results/singlenode-p2p-50K-50K-50K-50K.md: -------------------------------------------------------------------------------- 1 | # Benchmark results of UC: singlenode-p2p-50K-50K-50K-50K 2 | 3 | ## Brokers 4 | 5 | | Opensource **Broker** | **Version** | 6 | | --------------------- | ----------- | 7 | | EMQX 4 | 4.4.16 | 8 | | EMQX 5 | 5.0.21 | 9 | | Mosquitto | 2.0.15 | 10 | | NanoMQ | 0.17.0 | 11 | 12 | ## Benchmark use case: singlenode-p2p-50K-50K-50K-50K 13 | 14 | In this use case, 50,000 publishers publish messages to 50,000 topics, and 50,000 subscribers subscribe messages. Specially, the use case is executed in the following way: 15 | 16 | 1. All publishers and subscribers connect to the TCP port of the broker. 17 | 2. Once the connection of a subscriber is established, the subscriber immediately subscribes to a topic using QoS 1. Different subscribers subscribe to different topics. 18 | 3. When all the connections are established, each publisher publishes message to a topic using QoS 1 with Retain as 0. Different publishers subscribe to different topics. The publish rate for each publisher is 1 message per second. The payload size of each message is 16 bytes. 19 | 4. Keep the publish and subscribe for 30 minutes. The expected total publish rate is 50,000 messages per second, and the expected total subscribe rate is 50,000 messages per second. 20 | 21 | 22 | 23 | MQTT protocol 3.1.1 is used for the use case, and other configured MQTT properties are: 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 |
TLS authentication enablementNo
Authentication enablementNo
Keep Alive (seconds)300
Clean Session1
43 | 44 | ## Testbed 45 | 46 | The use case is executed on single node. XMeter (version 3.2.4) is used as the benchmark tool. 47 | 48 | **HW Details** 49 | 50 | - public cloud: AWS 51 | - instance type: c5.4xlarge 16C32G 52 | - OS: Ubuntu 22.04.1 amd64 53 | 54 | ## Metrics 55 | 56 | | | Actual msg rate | Average pub-to-sub latency (ms) | Max CPU user+system | Avg CPU user+system | Max memory used | Avg memory used | 57 | | ------------- | --------------- | ------------------------------- | ------------------- | ------------------- | --------------- | --------------- | 58 | | **EMQX 4** | 50k:50k | 1.68 | 88% | 80% | 6.19G | 5.22G | 59 | | **EMQX 5** | 50k:50k | 2.17 | 89% | 82% | 6.7G | 5.9G | 60 | | **Mosquitto** | 37k:37k | 335.94 | 7% | 6% | 354M | 328M | 61 | | **NanoMQ** | 50k:50k | 64.48 | 34% | 31% | 1.3G | 1.3G | 62 | 63 | > in this scenario, Mosquitto couldn't reach to the designed message rate. It stabilized at 37300/s for both pub and sub. 64 | > 65 | > EMQX 4, EMQX 5 and NanoMQ kept the stable pub & sub rate at 50000/s during the test. 66 | 67 | ### Appendix: result charts 68 | 69 | #### EMQX 4 70 | 71 | ![EMQX 4](../_assets/emqx-4.4.16-p2p-result-charts.png) 72 | 73 | #### EMQX 5 74 | 75 | ![EMQX 5](../_assets/emqx-5.0.21-p2p-result-charts.png) 76 | 77 | #### Mosquitto 78 | 79 | ![Mosquitto](../_assets/mosquitto-2.0.15-p2p-result-charts.png) 80 | 81 | #### NanoMQ 82 | 83 | ![NanoMQ](../_assets/nano-0.17.0-p2p-result-charts.png) 84 | 85 | -------------------------------------------------------------------------------- /results/singlenode-sharesub-50K-500-50K-50K.md: -------------------------------------------------------------------------------- 1 | # Benchmark results of UC: singlenode-sharesub-50K-500-50K-50K 2 | 3 | ## Brokers 4 | 5 | | Opensource **Broker** | **Version** | 6 | | --------------------- | ----------- | 7 | | EMQX 4 | 4.4.16 | 8 | | EMQX 5 | 5.0.21 | 9 | | Mosquitto | 2.0.15 | 10 | | NanoMQ | 0.17.0 | 11 | 12 | ## Benchmark use case: singlenode-p2p-50K-50K-50K-50K 13 | 14 | In this use case, 50,000 publishers publish messages to 50,000 topics, and 500 subscribers subscribe messages in shared subscription way. Specially, the use case is executed in the following way: 15 | 16 | 1. All publishers and subscribers connect to the TCP port of the broker. 17 | 2. Once the connection of a subscriber is established, the subscriber immediately subscribes to topics via shared subscription way using QoS 1. The shared subscription topic used is: $share/benchmark/test/# 18 | 3. When all the connections are established, each publisher publishes message to a topic using QoS 1 with Retain as 0. Different publishers subscribe to different topics. The topics are test/1, test/2, ..., test/50000. The publish rate for each publisher is 1 message per second. The payload size of each message is 16 bytes. 19 | 4. Keep the publish and subscribe for 30 minutes. The expected total publish rate is 50,000 messages per second, and the expected total subscribe rate is 50,000 messages per second. 20 | 21 | 22 | 23 | MQTT protocol 5.0 is used for the use case as shared subscription feature is required, and other configured MQTT properties are: 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 |
TLS authentication enablementNo
Authentication enablementNo
Keep Alive (seconds)300
Clean Session1
43 | 44 | ## Testbed 45 | 46 | The use case is executed on single node. XMeter (version 3.2.4) is used as the benchmark tool. 47 | 48 | **HW Details** 49 | 50 | - public cloud: AWS 51 | - instance type: c5.4xlarge 16C32G 52 | - OS: Ubuntu 22.04.1 amd64 53 | 54 | ## Metrics 55 | 56 | | | Actual msg rate | Average pub-to-sub latency (ms) | Max CPU user+system | Avg CPU user+system | Max memory used | Avg memory used | 57 | | ------------- | ---------------- | ------------------------------- | ------------------- | ------------------- | --------------- | --------------- | 58 | | **EMQX 4** | pub: 50ksub: 50k | 1.58 | 94% | 93% | 8.27G | 6.78G | 59 | | **EMQX 5** | pub: 50ksub: 50k | 2.51 | 94% | 93% | 6.6G | 5.9G | 60 | | **Mosquitto** | pub: 50ksub: 40k | 12,476.34 | 7% | 7% | 488M | 466M | 61 | | **NanoMQ** | pub: 50ksub: 50k | 2.76 | 34% | 34% | 795M | 783M | 62 | 63 | > in this scenario, the consumption rate of Mosquitto couldn't reach to the designed rate. It stabilized at 41,000/s. 64 | > 65 | > EMQX 4, EMQX 5 and NanoMQ kept the stable pub & sub rate at 50,000/s during the test. 66 | 67 | ### Appendix: result charts 68 | 69 | #### EMQX 4 70 | 71 | ![EMQX 4](../_assets/emqx-4.4.16-fanin-result-charts.png) 72 | 73 | #### EMQX 5 74 | 75 | ![EMQX 5](../_assets/emqx-5.0.21-fanin-result-charts.png) 76 | 77 | #### Mosquitto 78 | 79 | 80 | 81 | #### NanoMQ 82 | 83 | ![NanoMQ](../_assets/nano-0.17.0-fanin-result-charts.png) 84 | --------------------------------------------------------------------------------