├── README.md
├── _assets
├── emqx-4.4.16-fanin-result-charts.png
├── emqx-4.4.16-fanout-result-charts.png
├── emqx-4.4.16-p2p-result-charts.png
├── emqx-5.0.21-fanin-result-charts.png
├── emqx-5.0.21-fanout-result-charts.png
├── emqx-5.0.21-p2p-result-charts.png
├── fanin.png
├── fanout.png
├── mosquitto-2.0.15-fanout-result-charts.png
├── mosquitto-2.0.15-p2p-result-charts.png
├── mqtt_model.png
├── nano-0.17.0-fanin-result-charts.png
├── nano-0.17.0-fanout-result-charts.png
├── nano-0.17.0-p2p-result-charts.png
└── p2p.png
└── results
├── singlenode-conn-tcp-1M-5K.md
├── singlenode-fanout-5-1000-5-250K.md
├── singlenode-p2p-50K-50K-50K-50K.md
└── singlenode-sharesub-50K-500-50K-50K.md
/README.md:
--------------------------------------------------------------------------------
1 | # Open MQTT Benchmark Suite
2 | ## Introduction
3 |
4 | MQTT is a communication protocol based on Publish/Subscribe model. It has the following features:
5 |
6 | - Simple and easy to implement
7 | - QoS support for complex device network environment.
8 | - Lightweight and designed specially for unstable network connections and bandwidth saving
9 | - Data irrelevant
10 | - Continuous session awareness
11 |
12 | Those characteristics make it suited for IoT applications, and become one of the most popular protocols in IoT domain.
13 |
14 | In a typical MQTT Publish/Subscribe architecture, there are publishers, subscribers, and a broker in the system. A publisher sends a message to the broker on a specific topic and then the broker sends that message to all subscribers which are subscribed to the same topic.
15 |
16 | 
17 |
18 | As a broker is the central component in an MQTT based IoT application, it plays an important role in the scalability and availability of the application. There have been many research papers and technical writeups on the performance evaluation and comparison of various MQTT brokers. However, the existing benchmarks often measure particular factors only, with scenarios not matching the realistic large-scale IoT applications. Through development and customer support experience of the past decade, we have learnt that many factors impact MQTT broker performance.
19 |
20 | For one thing, MQTT protocol states many features improving reliability and security, which bring overhead cost on broker side as well. Some of the features are:
21 |
22 | - QoS
23 | - Keep Alive
24 | - Clean Session
25 | - Retain
26 | - Topic wildcards
27 | - Authentication method
28 | - TLS authentication enablement
29 |
30 | In addition, different MQTT versions suggest different functions. For example, shared subscription is supported in MQTT 5.0 but not 3.1.1 or 3.1.
31 |
32 | For another thing, MQTT brokers deployment ways influence the performance evaluation. The brokers can be deployed on single node, or in a cluster way. If it is deployed in a cluster way, it can be put behind a load balancer or not. To be comparable, the brokers should be deployed in the similar ways.
33 |
34 | What is more, environment conditions like hardware, OS, system and network configuration also impact the evaluation result. Some of the important conditions are:
35 |
36 | - receive buffer size
37 | - network round trip time
38 |
39 | The rapid development of IoT applications brings new challenges. As a hub of devices and data, the capability of the MQTT broker to support massive device connectivity and data throughput is critical to the IoT business. Consequently, a thorough evaluation on its performance is important in the full IoT solution. The intent of this work is suggesting metrics and evaluation criteria, presenting real-world scenarios and use cases, and in the future, providing a tool that fully compatible with the benchmark suite. To this end, we hope to establish a practical base for benchmarking MQTT brokers, iteratively optimize its coverage, and help users find the most suitable MQTT broker for their needs and workloads.
40 |
41 | ## Methodology
42 |
43 | To measure performance of an MQTT broker, we need to define metrics, which can be looked at in a variety of use cases. Two primary perspectives of measurement are considered. One focuses on the computing resources perspective. The other focuses on the capabilities provided by the MQTT broker itself, and divides into three detailed aspects - scalability, availability, and latency, as suggested in Google “Cloud Pub/Sub” product guide. It should be noticed that different aspects may contradict each other under different circumstances. To achieve different evaluation purposes, user needs to pick the suitable metrics.
44 |
45 |
46 |
47 | Computing resources metrics |
48 | Broker capability metrics |
49 |
50 |
51 |
52 | Scalability
53 | the ability to handle increases in load
54 | |
55 |
56 | Availability
57 | the ability to handle problems and failures
58 | |
59 |
60 | Latency
61 | the amount of time to deliver a published message to a subscriber
62 | |
63 |
64 |
65 | CPU load and usage |
66 | Memory usage |
67 | Disk I/O rate |
68 | Packets receiving and sending rate |
69 | Rate of new established connections |
70 | Number of concurrent connections |
71 | Number of publishers |
72 | Number of subscribers |
73 | Number of topics |
74 | Size of messages |
75 | Rate of messages published |
76 | Rate of messages subscribed |
77 | Success rate |
78 | Reconnection delay |
79 | Average latency time |
80 | 90th percentile latency time |
81 |
82 |
83 |
84 | A benchmark tool then is used to simulate publishers and/or subscribers based on use cases. It first generates dataset for benchmark, and loads the dataset to construct a series of MQTT requests, then executes to perform publishing and subscribing. The tool should work in a consistent way regardless of what the MQTT broker is. As many practical use cases involve large number of publishers/subscribers, the tool should also be implemented in a sophisticated way in order to generate desired large-scale load.
85 |
86 | As part of the benchmark submisison, a full disclosure report is suggested. The intent of the report is to simplify comparison between results of different brokers, and to provide enough information so other users can replicate the results of this benchmark. The report should identify the benchmark tool it uses, explain the detailed use cases, and list the hardware, OS and network conditions to perform the benchmark. It should also cover comparison data regarding the measurement metrics, both including computing resources perspective and broker capability perspective.
87 |
88 | ## Scenarios and use cases
89 |
90 | By analyzing realistic requirements from our customers, we categorize the benchmark into the following scenarios:
91 |
92 | - Connection
93 | - Fan-out
94 | - Point-to-point
95 | - Fan-in
96 | - Composite
97 |
98 | Next, we will describe each scenario and give detailed use cases for each scenario. To meet different user needs, different use cases are demanded. For example, some of them can be used for basic verification purpose, while some of them aim for enterprise level verification. Therefore, we group use cases of the same scalability level into one set, to help user pick up the use cases they need more easily. The use cases in each set will be introduced in details immediately after.
99 |
100 | ### Basic set
101 |
102 | This set is for small-scale. It consists of the following use cases:
103 |
104 | | Use case | Description |
105 | | ------------------------------- | ------------------------------------------------------------ |
106 | | singlenode-conn-tcp-10K-100 | Connection scenario for 1,000 clients |
107 | | singlenode-fanout-1-1K-1-1K | Fan-out scenario for 1 pub message per second and 1,000 sub messages per second |
108 | | singlenode-p2p-1K-1K-1K-1K | Point-to-point scenario for 1,000 pub messages per second and 1,000 sub messages per second |
109 | | singlenode-sharedsub-1K-5-1K-1K | Fan-in scenario for 1,000 pub messages per second and 1,000 sub messages per second in shared subscription way |
110 |
111 | ### Enterprise set
112 |
113 | This set is for large-scale. It consists of the following use cases:
114 |
115 | | Use case | Description |
116 | | ------------------------------------ | ------------------------------------------------------------ |
117 | | singlenode-conn-tcp-1M-5K | Connection scenario for 1,000,000 clients |
118 | | singlenode-fanout-5-1000-5-250K | Fan-out scenario for 5 pub messages per second and 250,000 sub messages per second |
119 | | singlenode-p2p-50K-50K-50K-50K | Point-to-point scenario for 50,000 pub messages per second and 50,000 sub messages per second |
120 | | singlenode-sharedsub-50K-500-50K-50K | Fan-in scenario for 50,000 pub messages per second and 50,000 sub messages per second in shared subscription way |
121 |
122 | Here comes the detailed use cases. The primary use cases focus on those executed on single-node-brokers, and in the future, we will add more supplemental use cases for clustering-brokers.
123 |
124 | ### Connection
125 |
126 | In a connection scenario, a batch of clients connect to the broker within a period of time, and keep the connections with the broker for quite a while.
127 |
128 | 2 use cases of this scenario for brokers deployed on single node are presented as:
129 |
130 |
131 |
132 | Use case |
133 | singlenode-conn-tcp-10K-100 |
134 |
135 |
136 | Description |
137 | 10,000 clients simultaneously connect to broker |
138 |
139 |
140 | Details |
141 |
142 |
143 | - 10,000 clients connect to the broker within 100 seconds. Each client connects to the TCP port of the broker, using MQTT 3.1.1 protocol. The Keep Alive property is set as 300, and Clean Session is set as 1.
144 | - Keep all clients online for 30 minutes.
145 |
146 | |
147 |
148 |
149 | Computing resource metrics of interest |
150 |
151 |
152 | - CPU load and usage
153 | - Memory usage
154 | - Disk I/O rate
155 |
156 | |
157 |
158 |
159 | Broker capability metrics of interest |
160 |
161 |
162 | - Rate of new established connections
163 | - Number of concurrent connections
164 | - Success rate
165 |
166 | |
167 |
168 |
169 |
170 |
171 |
172 | Use case |
173 | singlenode-conn-tcp-1M-5K |
174 |
175 |
176 | Description |
177 | 1,000,000 clients simultaneously connect to broker |
178 |
179 |
180 | Details |
181 |
182 |
183 | - 1,000,000 clients connect to the broker within 200 seconds. Each client connects to the TCP port of the broker, using MQTT 3.1.1 protocol. The Keep Alive property is set as 300, and Clean Session is set as 1.
184 | - Keep all clients online for 30 minutes.
185 |
186 | |
187 |
188 |
189 | Computing resource metrics of interest |
190 |
191 |
192 | - CPU load and usage
193 | - Memory usage
194 | - Disk I/O rate
195 |
196 | |
197 |
198 |
199 | Broker capability metrics of interest |
200 |
201 |
202 | - Rate of new established connections
203 | - Number of concurrent connections
204 | - Success rate
205 |
206 | |
207 |
208 |
209 |
210 | ### Fan-out
211 |
212 | In a fan-out scenario, a large number of clients act as subscribers, with only a few or a single publisher.
213 |
214 | 
215 |
216 | 2 use cases of this scenario for brokers deployed on single node are presented as:
217 |
218 |
219 |
220 | Use case |
221 | singlenode-fanout-1-1K-1-1K |
222 |
223 |
224 | Description |
225 | 1 publisher publishes messages to 1 topic which are subscribed by 1,000 subscribers |
226 |
227 |
228 | Details |
229 |
230 |
231 | - 1,001 clients are divided into publishers and subscribers: 1 as publisher and 1,000 as subscribers. The publishers and subscribers will work on 1 topic.
232 | - All publishers and subscribers connect to the TCP port of the broker, using MQTT 3.1.1 protocol. The Keep Alive property is set as 300, and Clean Session is set as 1.
233 | - Once the connection of a subscriber is established, the subscriber immediately subscribes to the topics using QoS 1.
234 | - When all the connections are established, the publisher publishes message to the topic using QoS 1 with Retain as 0. The publish rate for each publisher is 1 message per second. The payload size of each message is 16 bytes.
235 | - Keep the publish and subscribe for 30 minutes. The expected total publish rate is 1 message per second, and the expected total subscribe rate is 1,000 messages per second.
236 |
237 | |
238 |
239 |
240 | Computing resource metrics of interest |
241 |
242 |
243 | - CPU load and usage
244 | - Memory usage
245 | - Disk I/O rate
246 | - Packets receiving and sending rate
247 |
248 | |
249 |
250 |
251 | Broker capability metrics of interest |
252 |
253 |
254 | - Number of publishers
255 | - Number of subscribers
256 | - Number of topics
257 | - Size of messages
258 | - Rate of messages published
259 | - Rate of messages subscribed
260 | - Success rate
261 | - Average latency time
262 | - 90th percentile latency time
263 |
264 | |
265 |
266 |
267 |
268 |
269 |
270 | Use case |
271 | singlenode-fanout-5-1000-5-250K |
272 |
273 |
274 | Description |
275 | 50 publishers publish messages to 5 topics which are subscribed by 1,000 subscribers |
276 |
277 |
278 | Details |
279 |
280 |
281 | - 1,005 clients are divided into publishers and subscribers: 5 as publishers and 1,000 as subscribers. The publishers and subscribers will work on 5 topics.
282 | - All publishers and subscribers connect to the TCP port of the broker, using MQTT 3.1.1 protocol. The Keep Alive property is set as 300, and Clean Session is set as 1.
283 | - Once the connection of a subscriber is established, the subscriber immediately subscribes to all 5 topics using QoS 1.
284 | - When all the connections are established, each publisher publishes message to an exclusive topic using QoS 1 with Retain as 0. The publish rate for each publisher is 50 messages per second. The payload size of each message is 16 bytes.
285 | - Keep the publish and subscribe for 30 minutes. The expected total publish rate is 250 messages per second, and the expected total subscribe rate is 250,000 messages per second.
286 |
287 | |
288 |
289 |
290 | Computing resource metrics of interest |
291 |
292 |
293 | - CPU load and usage
294 | - Memory usage
295 | - Disk I/O rate
296 | - Packets receiving and sending rate
297 |
298 | |
299 |
300 |
301 | Broker capability metrics of interest |
302 |
303 |
304 | - Number of publishers
305 | - Number of subscribers
306 | - Number of topics
307 | - Size of messages
308 | - Rate of messages published
309 | - Rate of messages subscribed
310 | - Success rate
311 | - Average latency time
312 | - 90th percentile latency time
313 |
314 | |
315 |
316 |
317 |
318 | ### Point-to-point
319 |
320 | In a point-to-point scenario, the equal number of clients act as publishers and subscribers respectively.
321 |
322 | 
323 |
324 | 2 use cases of this scenario for brokers deployed on single node are presented as:
325 |
326 |
327 |
328 | Use case |
329 | singlenode-p2p-1K-1K-1K-1K |
330 |
331 |
332 | Description |
333 | 1,000 publishers publish messages to 1,000 topics which are subscribed by 1,000 subscribers |
334 |
335 |
336 | Details |
337 |
338 |
339 | - 2,000 clients are divided into publishers and subscribers: 1,000 as publishers and 1,000 as subscribers. The publishers and subscribers will work on 1,000 topics.
340 |
341 | - All publishers and subscribers connect to the TCP port of the broker, using MQTT 3.1.1 protocol. The Keep Alive property is set as 300, and Clean Session is set as 1.
342 | - Once the connection of a subscriber is established, the subscriber immediately subscribes a topic using QoS 1. Different subscribers subscribe to different topics.
343 | - When all the connections are established, each publisher publishes messages to a topic using QoS 1 with Retain as 0. Different publishers publish to different topics. The publish rate for each publisher is 1 message per second. The payload size of each message is 16 bytes.
344 | - Keep the publish and subscribe for 30 minutes. The expected total publish rate is 1,000 messages per second, and the expected total subscribe rate is 1,000 messages per second.
345 |
346 | |
347 |
348 |
349 | Computing resource metrics of interest |
350 |
351 |
352 | - CPU load and usage
353 | - Memory usage
354 | - Disk I/O rate
355 | - Packets receiving and sending rate
356 |
357 | |
358 |
359 |
360 | Broker capability metrics of interest |
361 |
362 |
363 | - Number of publishers
364 | - Number of subscribers
365 | - Number of topics
366 | - Size of messages
367 | - Rate of messages published
368 | - Rate of messages subscribed
369 | - Success rate
370 | - Average latency time
371 | - 90th percentile latency time
372 |
373 | |
374 |
375 |
376 |
377 |
378 |
379 | Use case |
380 | singlenode-p2p-50K-50K-50K-50K |
381 |
382 |
383 | Description |
384 | 50,000 publishers publish messages to 50,000 topics which are subscribed by 50,000 subscribers |
385 |
386 |
387 | Details |
388 |
389 |
390 | - 100,000 clients are divided into publishers and subscribers: 50,000 as publishers and 50,000 as subscribers. The publishers and subscribers will work on 50,000 topics.
391 |
392 | - All publishers and subscribers connect to the TCP port of the broker, using MQTT 3.1.1 protocol. The Keep Alive property is set as 300, and Clean Session is set as 1.
393 | - Once the connection of a subscriber is established, the subscriber immediately subscribes a topic using QoS 1. Different subscribers subscribe to different topics.
394 | - When all the connections are established, each publisher publishes messages to a topic using QoS 1 with Retain as 0. Different publishers publish to different topics. The publish rate for each publisher is 1 message per second. The payload size of each message is 16 bytes.
395 | - Keep the publish and subscribe for 30 minutes. The expected total publish rate is 50,000 messages per second, and the expected total subscribe rate is 50,000 messages per second.
396 |
397 | |
398 |
399 |
400 | Computing resource metrics of interest |
401 |
402 |
403 | - CPU load and usage
404 | - Memory usage
405 | - Disk I/O rate
406 | - Packets receiving and sending rate
407 |
408 | |
409 |
410 |
411 | Broker capability metrics of interest |
412 |
413 |
414 | - Number of publishers
415 | - Number of subscribers
416 | - Number of topics
417 | - Size of messages
418 | - Rate of messages published
419 | - Rate of messages subscribed
420 | - Success rate
421 | - Average latency time
422 | - 90th percentile latency time
423 |
424 | |
425 |
426 |
427 |
428 | ### Fan-in
429 |
430 | A fan-in scenario is the opposite of fan-out. In a fan-in scenario, a large number of clients act as publishers, with only a few or a single subscriber.
431 |
432 | 
433 |
434 | 2 use cases of this scenario for brokers deployed on single node are presented as:
435 |
436 |
437 |
438 | Use case |
439 | singlenode-sharedsub-1K-5-1K-1K |
440 |
441 |
442 | Description |
443 | 1,000 publishers publish messages to 1,000 topics which are subscribed in shared subscription way by 5 subscribers |
444 |
445 |
446 | Details |
447 |
448 |
449 | - 1,005 clients are divided into publishers and subscribers: 1,000 as publishers and 5 as subscribers. The publishers and subscribers will work on 1,000 topics. The topics are like: test/1, test/2, ..., test/1000.
450 |
451 | - All publishers and subscribers connect to the TCP port of the broker which should support MQTT 5.0 protocol. The Keep Alive property is set as 300, and Clean Session is set as 1.
452 | - Once the connection of a subscriber is established, the subscriber immediately subscribes to all the topics via shared subscription way using QoS 1. The shared subscription topic used is: $share/benchmark/test/#
453 | - When all the connections are established, each publisher publishes message to a topic using QoS 1 with Retain as 0. Different publishers publish to different topics. The publish rate for each publisher is 1 message per second. The payload size of each message is 16 bytes.
454 | - Keep the publish and subscribe for 30 minutes. The expected total publish rate is 1,000 messages per second, and the expected total subscribe rate is 1,000 messages per second.
455 |
456 | |
457 |
458 |
459 | Computing resource metrics of interest |
460 |
461 |
462 | - CPU load and usage
463 | - Memory usage
464 | - Disk I/O rate
465 | - Packets receiving and sending rate
466 |
467 | |
468 |
469 |
470 | Broker capability metrics of interest |
471 |
472 |
473 | - Number of publishers
474 | - Number of subscribers
475 | - Number of topics
476 | - Size of messages
477 | - Rate of messages published
478 | - Rate of messages subscribed
479 | - Success rate
480 | - Average latency time
481 | - 90th percentile latency time
482 |
483 | |
484 |
485 |
486 |
487 |
488 |
489 | Use case |
490 | singlenode-sharedsub-50K-500-50K-50K |
491 |
492 |
493 | Description |
494 | 50,000 publishers publish messages to 50,000 topics which are subscribed in shared subscription way by 500 subscribers |
495 |
496 |
497 | Details |
498 |
499 |
500 | - 50,500 clients are divided into publishers and subscribers: 50,000 as publishers and 500 as subscribers. The publishers and subscribers will work on 50,000 topics. The topics are like: test/1, test/2, ..., test/50000.
501 |
502 | - All publishers and subscribers connect to the TCP port of the broker which should support MQTT 5.0 protocol. The Keep Alive property is set as 300, and Clean Session is set as 1.
503 | - Once the connection of a subscriber is established, the subscriber immediately subscribes to all the topics via shared subscription way using QoS 1. The shared subscription topic used is: $share/benchmark/test/#
504 | - When all the connections are established, each publisher publishes message to a topic using QoS 1 with Retain as 0. Different publishers publish to different topics. The publish rate for each publisher is 1 message per second. The payload size of each message is 16 bytes.
505 | - Keep the publish and subscribe for 30 minutes. The expected total publish rate is 50,000 messages per second, and the expected total subscribe rate is 50,000 messages per second.
506 |
507 | |
508 |
509 |
510 | Computing resource metrics of interest |
511 |
512 |
513 | - CPU load and usage
514 | - Memory usage
515 | - Disk I/O rate
516 | - Packets receiving and sending rate
517 |
518 | |
519 |
520 |
521 | Broker capability metrics of interest |
522 |
523 |
524 | - Number of publishers
525 | - Number of subscribers
526 | - Number of topics
527 | - Size of messages
528 | - Rate of messages published
529 | - Rate of messages subscribed
530 | - Success rate
531 | - Average latency time
532 | - 90th percentile latency time
533 |
534 | |
535 |
536 |
537 |
538 | ### Composite
539 |
540 | A composite scenario combines scenarios involving connection/publish/subscribe. Usually, a large number of clients connect to the broker, with most of them playing roles of background connections, and the rest of them perform fan-out, point-to-point, or fan-in scenario.
541 |
542 | We welcome contributions from the community to enrich use cases!
543 |
544 | ## Benchmarking results
545 |
546 | We execute benchmarking on several MQTT brokers (including EMQX, Mosquitto, NanoMQ, etc.), using XMeter as the benchmark tool. For each use case involved, XMeter simulates a huge number of MQTT clients, executes expected publishing/subscription, and generates reports covering the primary performance metrics.
547 |
548 | For detailed benchmarking information and result, please refer to the following pages:
549 |
550 | [Benchmarking results of concurrent connection](results/singlenode-conn-tcp-1M-5K.md)
551 |
552 | [Benchmarking results of fan-out](results/singlenode-fanout-5-1000-5-250K.md)
553 |
554 | [Benchmarking results of point-to-point](results/singlenode-p2p-50K-50K-50K-50K.md)
555 |
556 | [Benchmarking results of fan-in](results/singlenode-sharesub-50K-500-50K-50K.md)
--------------------------------------------------------------------------------
/_assets/emqx-4.4.16-fanin-result-charts.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/emqx-4.4.16-fanin-result-charts.png
--------------------------------------------------------------------------------
/_assets/emqx-4.4.16-fanout-result-charts.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/emqx-4.4.16-fanout-result-charts.png
--------------------------------------------------------------------------------
/_assets/emqx-4.4.16-p2p-result-charts.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/emqx-4.4.16-p2p-result-charts.png
--------------------------------------------------------------------------------
/_assets/emqx-5.0.21-fanin-result-charts.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/emqx-5.0.21-fanin-result-charts.png
--------------------------------------------------------------------------------
/_assets/emqx-5.0.21-fanout-result-charts.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/emqx-5.0.21-fanout-result-charts.png
--------------------------------------------------------------------------------
/_assets/emqx-5.0.21-p2p-result-charts.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/emqx-5.0.21-p2p-result-charts.png
--------------------------------------------------------------------------------
/_assets/fanin.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/fanin.png
--------------------------------------------------------------------------------
/_assets/fanout.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/fanout.png
--------------------------------------------------------------------------------
/_assets/mosquitto-2.0.15-fanout-result-charts.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/mosquitto-2.0.15-fanout-result-charts.png
--------------------------------------------------------------------------------
/_assets/mosquitto-2.0.15-p2p-result-charts.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/mosquitto-2.0.15-p2p-result-charts.png
--------------------------------------------------------------------------------
/_assets/mqtt_model.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/mqtt_model.png
--------------------------------------------------------------------------------
/_assets/nano-0.17.0-fanin-result-charts.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/nano-0.17.0-fanin-result-charts.png
--------------------------------------------------------------------------------
/_assets/nano-0.17.0-fanout-result-charts.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/nano-0.17.0-fanout-result-charts.png
--------------------------------------------------------------------------------
/_assets/nano-0.17.0-p2p-result-charts.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/nano-0.17.0-p2p-result-charts.png
--------------------------------------------------------------------------------
/_assets/p2p.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/emqx/mqttbs/9a946c8c66ec3e8d1459fa5665def06f7b7d7eca/_assets/p2p.png
--------------------------------------------------------------------------------
/results/singlenode-conn-tcp-1M-5K.md:
--------------------------------------------------------------------------------
1 | # Benchmark results of singlenode-conn-tcp-1M-5K
2 |
3 | ## Brokers
4 |
5 | | Opensource **Broker** | **Version** |
6 | | --------------------- | ----------- |
7 | | EMQX 4 | 4.4.16 |
8 | | EMQX 5 | 5.0.21 |
9 | | Mosquitto | 2.0.15 |
10 | | NanoMQ | 0.17.0 |
11 |
12 | ## Benchmark use case: singlenode-conn-tcp-1M-5K
13 |
14 | In this testcase, 1,000,000 clients connect to the broker within 200 seconds (e.g. cps 5K) and all clients are kept online for 30 minutes.
15 |
16 | MQTT protocol 3.1.1 is used for the use case, and other configured MQTT properties are:
17 |
18 |
19 |
20 | TLS authentication enablement |
21 | No |
22 |
23 |
24 | Authentication enablement |
25 | No |
26 |
27 |
28 | Keep Alive (seconds) |
29 | 300 |
30 |
31 |
32 | Clean Session |
33 | 1 |
34 |
35 |
36 |
37 | ## Testbed
38 |
39 | The use case is executed on single node. XMeter (version 3.2.4) is used as the benchmark tool.
40 |
41 | **HW Details**
42 |
43 | - public cloud: AWS
44 | - instance type: c5.4xlarge 16C32G
45 | - OS: Ubuntu 22.04.1 amd64
46 |
47 | ## Metrics
48 |
49 | | | Average latency (ms) | Max CPU user+system | Avg CPU user+system | Max memory used | Avg memory used |
50 | | ------------- | --------------------- | ------------------- | ------------------- | --------------- | --------------- |
51 | | **EMQX 4** | 2.21 | 44% | 31% | 10.87G | 8.68G |
52 | | **EMQX 5** | 2.25 | 52% | 24% | 16.1G | 12.1G |
53 | | **Mosquitto** | 5.61 | 2% | 2% | 1G | 1G |
54 | | **NanoMQ** | 2.99 | 5% | 4% | 10.6G | 10.6G |
55 |
56 | ### Appendix I: result charts
57 |
58 | #### EMQX 4
59 |
60 | #### EMQX 5
61 |
62 | #### Mosquitto
63 |
64 | #### NanoMQ
65 |
66 |
67 |
68 | ### Appendix II: monitoring charts
69 |
70 | #### Mosquitto
71 |
72 |
73 |
74 | #### NanoMQ
75 |
76 |
77 |
78 |
--------------------------------------------------------------------------------
/results/singlenode-fanout-5-1000-5-250K.md:
--------------------------------------------------------------------------------
1 | # Benchmark results of UC: singlenode-fanout-5-1000-5-250K
2 |
3 | ## Brokers
4 |
5 | | Opensource **Broker** | **Version** |
6 | | --------------------- | ----------- |
7 | | EMQX 4 | 4.4.16 |
8 | | EMQX 5 | 5.0.21 |
9 | | Mosquitto | 2.0.15 |
10 | | NanoMQ | 0.17.0 |
11 |
12 | ## Benchmark use case: singlenode-fanout-5-1000-5-250K
13 |
14 | In this use case, 5 publishers publish messages to 5 topics, and 1,000 subscribers subscribe messages (each subscriber subscribs to all 5 topics). Specially, the use case is executed in the following way:
15 |
16 | 1. All publishers and subscribers connect to the TCP port of the broker.
17 | 2. Once the connection of a subscriber is established, the subscriber immediately subscribes to all 5 topics using QoS 1.
18 | 3. When all the connections are established, each publisher publishes message to a topic using QoS 1 with Retain as 0. Different publishers subscribe to different topics. The publish rate for each publisher is 50 messages per second. The payload size of each message is 16 bytes.
19 | 4. Keep the publish and subscribe for 30 minutes. The expected total publish rate is 250 messages per second, and the expected total subscribe rate is 250,000 messages per second.
20 |
21 |
22 |
23 | MQTT protocol 3.1.1 is used for the use case, and other configured MQTT properties are:
24 |
25 |
26 |
27 | TLS authentication enablement |
28 | No |
29 |
30 |
31 | Authentication enablement |
32 | No |
33 |
34 |
35 | Keep Alive (seconds) |
36 | 300 |
37 |
38 |
39 | Clean Session |
40 | 1 |
41 |
42 |
43 |
44 | ## Testbed
45 |
46 | The use case is executed on single node. XMeter (version 3.2.4) is used as the benchmark tool.
47 |
48 | **HW Details**
49 |
50 | - public cloud: AWS
51 | - instance type: c5.4xlarge 16C32G
52 | - OS: Ubuntu 22.04.1 amd64
53 |
54 | ## Metrics
55 |
56 | | | Actual msg rate | Average pub-to-sub latency (ms) | Max CPU user+system | Avg CPU user+system | Max memory used | Avg memory used |
57 | | ------------- | --------------- | ------------------------------- | ------------------- | ------------------- | --------------- | --------------- |
58 | | **EMQX 4** | 250k | 1.98 | 83% | 79% | 517M | 486M |
59 | | **EMQX 5** | 250k | 2.27 | 81% | 79% | 540M | 494M |
60 | | **Mosquitto** | 81k | 12,445 | 7% | 6% | 366M | 351M |
61 | | **NanoMQ** | 250k | 14.07 | 72% | 70% | 824M | 685M |
62 |
63 | > in this scenario, Mosquitto couldn't reach to the designed message rate. The throughput has been fluctuating around 80,000/s.
64 | >
65 | > EMQX 4, EMQX 5 and NanoMQ kept the stable rate at 250,000/s during the test.
66 |
67 | ### Appendix: result charts
68 |
69 | #### EMQX 4
70 |
71 | 
72 |
73 | #### EMQX 5
74 |
75 | 
76 |
77 | #### Mosquitto
78 |
79 | 
80 |
81 | #### NanoMQ
82 |
83 | 
--------------------------------------------------------------------------------
/results/singlenode-p2p-50K-50K-50K-50K.md:
--------------------------------------------------------------------------------
1 | # Benchmark results of UC: singlenode-p2p-50K-50K-50K-50K
2 |
3 | ## Brokers
4 |
5 | | Opensource **Broker** | **Version** |
6 | | --------------------- | ----------- |
7 | | EMQX 4 | 4.4.16 |
8 | | EMQX 5 | 5.0.21 |
9 | | Mosquitto | 2.0.15 |
10 | | NanoMQ | 0.17.0 |
11 |
12 | ## Benchmark use case: singlenode-p2p-50K-50K-50K-50K
13 |
14 | In this use case, 50,000 publishers publish messages to 50,000 topics, and 50,000 subscribers subscribe messages. Specially, the use case is executed in the following way:
15 |
16 | 1. All publishers and subscribers connect to the TCP port of the broker.
17 | 2. Once the connection of a subscriber is established, the subscriber immediately subscribes to a topic using QoS 1. Different subscribers subscribe to different topics.
18 | 3. When all the connections are established, each publisher publishes message to a topic using QoS 1 with Retain as 0. Different publishers subscribe to different topics. The publish rate for each publisher is 1 message per second. The payload size of each message is 16 bytes.
19 | 4. Keep the publish and subscribe for 30 minutes. The expected total publish rate is 50,000 messages per second, and the expected total subscribe rate is 50,000 messages per second.
20 |
21 |
22 |
23 | MQTT protocol 3.1.1 is used for the use case, and other configured MQTT properties are:
24 |
25 |
26 |
27 | TLS authentication enablement |
28 | No |
29 |
30 |
31 | Authentication enablement |
32 | No |
33 |
34 |
35 | Keep Alive (seconds) |
36 | 300 |
37 |
38 |
39 | Clean Session |
40 | 1 |
41 |
42 |
43 |
44 | ## Testbed
45 |
46 | The use case is executed on single node. XMeter (version 3.2.4) is used as the benchmark tool.
47 |
48 | **HW Details**
49 |
50 | - public cloud: AWS
51 | - instance type: c5.4xlarge 16C32G
52 | - OS: Ubuntu 22.04.1 amd64
53 |
54 | ## Metrics
55 |
56 | | | Actual msg rate | Average pub-to-sub latency (ms) | Max CPU user+system | Avg CPU user+system | Max memory used | Avg memory used |
57 | | ------------- | --------------- | ------------------------------- | ------------------- | ------------------- | --------------- | --------------- |
58 | | **EMQX 4** | 50k:50k | 1.68 | 88% | 80% | 6.19G | 5.22G |
59 | | **EMQX 5** | 50k:50k | 2.17 | 89% | 82% | 6.7G | 5.9G |
60 | | **Mosquitto** | 37k:37k | 335.94 | 7% | 6% | 354M | 328M |
61 | | **NanoMQ** | 50k:50k | 64.48 | 34% | 31% | 1.3G | 1.3G |
62 |
63 | > in this scenario, Mosquitto couldn't reach to the designed message rate. It stabilized at 37300/s for both pub and sub.
64 | >
65 | > EMQX 4, EMQX 5 and NanoMQ kept the stable pub & sub rate at 50000/s during the test.
66 |
67 | ### Appendix: result charts
68 |
69 | #### EMQX 4
70 |
71 | 
72 |
73 | #### EMQX 5
74 |
75 | 
76 |
77 | #### Mosquitto
78 |
79 | 
80 |
81 | #### NanoMQ
82 |
83 | 
84 |
85 |
--------------------------------------------------------------------------------
/results/singlenode-sharesub-50K-500-50K-50K.md:
--------------------------------------------------------------------------------
1 | # Benchmark results of UC: singlenode-sharesub-50K-500-50K-50K
2 |
3 | ## Brokers
4 |
5 | | Opensource **Broker** | **Version** |
6 | | --------------------- | ----------- |
7 | | EMQX 4 | 4.4.16 |
8 | | EMQX 5 | 5.0.21 |
9 | | Mosquitto | 2.0.15 |
10 | | NanoMQ | 0.17.0 |
11 |
12 | ## Benchmark use case: singlenode-p2p-50K-50K-50K-50K
13 |
14 | In this use case, 50,000 publishers publish messages to 50,000 topics, and 500 subscribers subscribe messages in shared subscription way. Specially, the use case is executed in the following way:
15 |
16 | 1. All publishers and subscribers connect to the TCP port of the broker.
17 | 2. Once the connection of a subscriber is established, the subscriber immediately subscribes to topics via shared subscription way using QoS 1. The shared subscription topic used is: $share/benchmark/test/#
18 | 3. When all the connections are established, each publisher publishes message to a topic using QoS 1 with Retain as 0. Different publishers subscribe to different topics. The topics are test/1, test/2, ..., test/50000. The publish rate for each publisher is 1 message per second. The payload size of each message is 16 bytes.
19 | 4. Keep the publish and subscribe for 30 minutes. The expected total publish rate is 50,000 messages per second, and the expected total subscribe rate is 50,000 messages per second.
20 |
21 |
22 |
23 | MQTT protocol 5.0 is used for the use case as shared subscription feature is required, and other configured MQTT properties are:
24 |
25 |
26 |
27 | TLS authentication enablement |
28 | No |
29 |
30 |
31 | Authentication enablement |
32 | No |
33 |
34 |
35 | Keep Alive (seconds) |
36 | 300 |
37 |
38 |
39 | Clean Session |
40 | 1 |
41 |
42 |
43 |
44 | ## Testbed
45 |
46 | The use case is executed on single node. XMeter (version 3.2.4) is used as the benchmark tool.
47 |
48 | **HW Details**
49 |
50 | - public cloud: AWS
51 | - instance type: c5.4xlarge 16C32G
52 | - OS: Ubuntu 22.04.1 amd64
53 |
54 | ## Metrics
55 |
56 | | | Actual msg rate | Average pub-to-sub latency (ms) | Max CPU user+system | Avg CPU user+system | Max memory used | Avg memory used |
57 | | ------------- | ---------------- | ------------------------------- | ------------------- | ------------------- | --------------- | --------------- |
58 | | **EMQX 4** | pub: 50ksub: 50k | 1.58 | 94% | 93% | 8.27G | 6.78G |
59 | | **EMQX 5** | pub: 50ksub: 50k | 2.51 | 94% | 93% | 6.6G | 5.9G |
60 | | **Mosquitto** | pub: 50ksub: 40k | 12,476.34 | 7% | 7% | 488M | 466M |
61 | | **NanoMQ** | pub: 50ksub: 50k | 2.76 | 34% | 34% | 795M | 783M |
62 |
63 | > in this scenario, the consumption rate of Mosquitto couldn't reach to the designed rate. It stabilized at 41,000/s.
64 | >
65 | > EMQX 4, EMQX 5 and NanoMQ kept the stable pub & sub rate at 50,000/s during the test.
66 |
67 | ### Appendix: result charts
68 |
69 | #### EMQX 4
70 |
71 | 
72 |
73 | #### EMQX 5
74 |
75 | 
76 |
77 | #### Mosquitto
78 |
79 |
80 |
81 | #### NanoMQ
82 |
83 | 
84 |
--------------------------------------------------------------------------------