├── .gitignore ├── README.md ├── common ├── pom.xml └── src │ └── main │ └── java │ └── com │ └── hsmq │ └── common │ ├── exception │ └── FileException.java │ └── utils │ ├── IpUtils.java │ ├── MessageIdUtils.java │ └── MurmurHash.java ├── io ├── pom.xml ├── protocol │ ├── pom.xml │ └── src │ │ └── main │ │ └── java │ │ └── com │ │ └── hsmq │ │ ├── data │ │ ├── Head.java │ │ ├── HsReq.java │ │ ├── HsResp.java │ │ └── message │ │ │ ├── Confirm.java │ │ │ ├── MessageQueueData.java │ │ │ ├── Pull.java │ │ │ ├── PullMessage.java │ │ │ ├── PullMessageResp.java │ │ │ ├── SendMessage.java │ │ │ ├── SyncOffsetMessage.java │ │ │ └── TopicData.java │ │ ├── decode │ │ ├── LengthObjectDecode.java │ │ └── ObjectDecode.java │ │ ├── encode │ │ ├── LengthObjectEncode.java │ │ └── ObjectEncode.java │ │ ├── enums │ │ ├── MessageEnum.java │ │ ├── MessageSendEnum.java │ │ ├── OperationEnum.java │ │ ├── ResultEnum.java │ │ └── SendResultEnum.java │ │ ├── error │ │ └── HsError.java │ │ ├── protocol │ │ ├── HsBaseData.java │ │ ├── HsDecodeData.java │ │ ├── HsEecodeData.java │ │ └── HsMessage.java │ │ └── utils │ │ └── ObjectByteUtils.java └── storage │ ├── pom.xml │ └── src │ ├── main │ └── java │ │ └── com │ │ └── hsmq │ │ └── storage │ │ ├── config │ │ ├── StorageConfig.java │ │ └── TopicConfig.java │ │ ├── data │ │ ├── MessageDurabilityStorage.java │ │ ├── MessageStorage.java │ │ └── QueueOffsetStorage.java │ │ ├── durability │ │ ├── MessageDurability.java │ │ └── TopicConsumerData.java │ │ └── file │ │ └── FileOperation.java │ └── test │ └── java │ └── com │ └── hs │ └── filer │ ├── FIleMappTest.java │ ├── FIleTest.java │ └── FIleTest2.java ├── mq-client ├── pom.xml └── src │ ├── main │ ├── java │ │ └── com │ │ │ └── hsms │ │ │ └── mqclient │ │ │ ├── ClientStartup.java │ │ │ ├── ConfigTest.java │ │ │ ├── consumer │ │ │ ├── config │ │ │ │ ├── ClientConfig.java │ │ │ │ ├── ExecutorService.java │ │ │ │ └── RegisteredConsumer.java │ │ │ ├── consumer │ │ │ │ ├── AbstractConsumer.java │ │ │ │ ├── ConsumerHandlerManger.java │ │ │ │ └── sub │ │ │ │ │ ├── TopicAConsumer.java │ │ │ │ │ └── TopicBConsumer.java │ │ │ ├── executos │ │ │ │ ├── CommitOffsetTask.java │ │ │ │ ├── ExecutorMessageTask.java │ │ │ │ └── PullMessageTask.java │ │ │ ├── handle │ │ │ │ └── ConsumerHandel.java │ │ │ ├── loadbalance │ │ │ │ └── ConsumerLoadBalance.java │ │ │ └── message │ │ │ │ └── ConsumerMessageQueue.java │ │ │ ├── producer │ │ │ ├── dto │ │ │ │ └── SendMessageResult.java │ │ │ ├── handle │ │ │ │ ├── NormalHandel.java │ │ │ │ └── ObjectOutHandle.java │ │ │ └── send │ │ │ │ └── MessageClient.java │ │ │ └── reactor │ │ │ └── ClientReactor.java │ └── resources │ │ ├── check │ │ └── hsmq_schema.json │ │ ├── hsmq.json │ │ └── logback.xml │ └── test │ └── java │ └── com │ └── mqclient │ ├── BtyVufTest.java │ ├── ClientA.java │ └── ClientB.java ├── mq-server ├── pom.xml └── src │ └── main │ ├── java │ └── com │ │ └── hsms │ │ └── mqserver │ │ ├── ServerStartup.java │ │ ├── config │ │ └── ServerConfig.java │ │ ├── data │ │ ├── ConsumerQueueManger.java │ │ ├── MessageStore.java │ │ └── TopicListener.java │ │ ├── handle │ │ └── ServerInHandel.java │ │ ├── reactor │ │ └── ObjectReactor.java │ │ ├── server │ │ └── MqServerBootstrap.java │ │ └── strategy │ │ ├── MessageStrategy.java │ │ └── executors │ │ ├── BaseExecutor.java │ │ └── impl │ │ ├── CommitOffsetExecutor.java │ │ ├── PullExecutor.java │ │ ├── SendMessageExecutor.java │ │ └── TopicDataExecutor.java │ └── resources │ └── logback.xml ├── pom.xml ├── zimage ├── consumer.png ├── hs.jpeg ├── hsmq.png ├── image.png └── server.png └── znote ├── Lock和Condition.md ├── Netty.md ├── Nio文件读写.md ├── 线程的生命周期.md └── 网络通信原理.md /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | *dependency-reduced-pom.xml 3 | .classpath 4 | .project 5 | .settings/ 6 | target/ 7 | devenv 8 | *.log* 9 | *.iml 10 | .idea/ 11 | *.versionsBackup 12 | !NOTICE-BIN 13 | !LICENSE-BIN 14 | .DS_Store 15 | localbin 16 | nohup.out 17 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## HeShenMQ 2 | [![version](https://img.shields.io/badge/java--version-1.8-orange)](https://www.java.com) 3 | [![version](https://img.shields.io/badge/io-netty-blue)](https://github.com/netty/netty) 4 | [![version](https://img.shields.io/badge/license-MIT-green)](https://github.com/MortyCode/HsMq) 5 | [![version](https://img.shields.io/badge/version-1.0--SNAPSHOT-brightgreen)](https://github.com/MortyCode/HsMq) 6 | ``` 7 | _ _ _____ ___ ___ _____ 8 | | | | | / ___/ / |/ | / _ \ 9 | | |_| | | |___ / /| /| | | | | | 10 | | _ | \___ \ / / |__/ | | | | | | 11 | | | | | ___| | / / | | | |_| |_ 12 | |_| |_| /_____/ /_/ |_| \_______| 13 | ``` 14 | ### 功能简述 15 | 1. 基于Netty来实现的消息中间件 16 | 2. 使用Server作为服务端 17 | 3. 生产者发送消息到服务端后将消息投递到多个MessageQueue中，消费者订阅一个或者多个Queue 18 | 4. 消费者每次取拉取10条消息后放入队列，异步消费 19 | 5. 消费后会提交消费记录到本地，定时任务会定时的同步到服务端 20 | 6. 在重启后，消费记录也会同步到消费者端 21 | 22 | ### 待开发 23 | 1. 集群模式 24 | 2. 负载均衡 25 | 26 | ### 一点小记录 27 | 项目位置：[https://github.com/MortyCode/HsMq.git](https://github.com/MortyCode/HsMq.git) 28 | # 1. 协议 29 | | 字段 | Length | HeadLength | head | DataLength | data | 30 | | --- | --- | --- | --- | --- | --- | 31 | | 解释 | 总消息体长度 | 消息头长度 | 消息体 | | | 32 | | 长度 | int(4个字节) | int(4个字节) | HeadLength的值 | int(4个字节) | DataLength的值 | 33 | 34 | - 解码 LengthObjectDecode 35 | - 编码 LengthObjectEncode 36 | ### Head 37 | 38 | - msgType: 请求类型 Req/Resp 39 | ### Data 40 | 41 | - 协议消息体序列化 Req对象/Resp对象 42 | # 2 . 服务端 43 | ## 2.1 服务端Reactor Handel 44 | ### 2.1.1 基于模板模式实现一个简单命令处理 45 | ```java 46 | private static final Map> enumExecutorMap ; 47 | 48 | ....策略初始化操作 49 | 50 | public static HsEecodeData executor(HsDecodeData hsDecodeData){ 51 | HsReq hsReq = (HsReq) hsDecodeData.getData(); 52 | 53 | OperationEnum operationEnum = hsReq.getOperationEnum(); 54 | if (operationEnum==null){ 55 | ... 异常处理 56 | } 57 | BaseExecutor baseExecutor = enumExecutorMap.get(operationEnum); 58 | return baseExecutor.executorReq(hsReq); 59 | } 60 | ``` 61 | ### 2.1.2 处理模板 62 | 63 | - 这里有一个细节,这里有两个方法，**executor** 方法的泛型是 **T 标识一个确定的类型**,**executor0** 方法泛型的是 **? 标识一个不确定的类型** . 64 | - 因为从HsDecodeData获取的对象是Object类型,并不知道此时类型是什么，所以convertReq的意义就是校验请求对象的数据是不是处理器需要的数据类型，并且转换。 65 | ```java 66 | public abstract class BaseExecutor { 67 | 68 | public static MessageStore messageStore = new MessageStore(); 69 | public abstract HsResp executor(HsReq hsReq); 70 | public abstract HsReq convertReq(HsReq hsReq); 71 | 72 | public HsEecodeData executorReq(HsReq hsReq){ 73 | HsReq fixedHsReq = convertReq(hsReq); 74 | if (fixedHsReq==null){ 75 | return HsEecodeData.typeError(); 76 | } 77 | 78 | HsResp hsResp = executor(fixedHsReq); 79 | hsResp.setReqType(hsReq.getOperation()); 80 | hsResp.setReqId(hsReq.getReqId()); 81 | return HsEecodeData.resp(hsResp); 82 | } 83 | 84 | } 85 | ``` 86 | 87 | 88 | ### 2.1.3 目前处理器 89 | 90 | - SendMessageExecutor 接受发送消息处理 91 | - PullExecutor 拉去消息处理 92 | - CommitOffsetExecutor 提交偏移量处理 93 | - TopicDataExecutor 获取Topic的信息处理 94 | 95 | 96 | 97 | ## 2.2. 消息存储 SendMessageExecutor 98 | ### 2.2.1 消息存储 99 | 100 | - 消息体首先存储到一个公共存储消息位置，目前是在 mq_1文件之中 101 | - 然后将返回的消息偏移量等信息存储到消息队列中，队列数为初始化的时候指定, 位置为/queue/目录下 102 | - 然后消息消费的时候，会从queue中拉取消息 103 | ```java 104 | public HsResp saveMessage(SendMessage sendMessage){ 105 | .... 106 | MessageDurability messageDurability = MessageStorage.saveMessage(sendMessage); 107 | boolean push = ConsumerQueueManger.pushConsumerQueue(sendMessage, messageDurability); 108 | ..... 109 | return resp; 110 | } 111 | ``` 112 | 113 | 114 | ### 2.2.1 消息体存储 115 | #### 存储格式 116 | 117 | 118 | | - | Length | Data | Length | Data | ...... | 119 | | --- | --- | --- | --- | --- | --- | 120 | | 解释 | 下一条数据长度 | 消息内容 | 下一条数据长度 | 消息内容 | ...... | 121 | | 占用 | int(4个字节) | Length长度 | int(4个字节) | Length长度 | ...... | 122 | 123 | #### 消息内容 124 | ```java 125 | public class SendMessage implements Serializable { 126 | private static final long serialVersionUID = -20210610L; 127 | private String msgId; 128 | private String topic; 129 | private String tag; 130 | private String key; 131 | private String body; 132 | } 133 | ``` 134 | #### 简介 135 | 136 | - 使用RandomAccessFile来进行存储，返回消息位点，以及偏移量 137 | - RandomAccessFile可以在任意位置进行快速的读写 138 | #### 存储代码 139 | ```java 140 | public static synchronized MessageDurability save(String fileName,Object object) 141 | throws IOException, InterruptedException{ 142 | 143 | RandomAccessFile rws = new RandomAccessFile(fileName, "rw"); 144 | FileChannel fileChannel = rws.getChannel(); 145 | 146 | byte[] bytes = ObjectByteUtils.toByteArray(object); 147 | if (bytes==null){ 148 | throw new FileException("文件转化异常：object not can cast bytes"); 149 | } 150 | 151 | ByteBuffer byteBuffer = ByteBuffer.allocate(bytes.length+4); 152 | byteBuffer.putInt(bytes.length); 153 | byteBuffer.put(bytes); 154 | byteBuffer.flip(); 155 | 156 | MessageDurability messageDurability = new MessageDurability(); 157 | messageDurability.setLength(byteBuffer.limit()); 158 | messageDurability.setOffset(fileChannel.size()); 159 | 160 | fileChannel.position(fileChannel.size()); 161 | fileChannel.write(byteBuffer); 162 | fileChannel.force(true); 163 | fileChannel.close(); 164 | 165 | return messageDurability; 166 | } 167 | ``` 168 | ### 2.2.2 消息消费队列存储 169 | #### 存储格式 170 | 171 | | -| Offset | Length | TagHashcode | Index | ...... | 172 | | --- | --- | --- | --- | --- | --- | 173 | | 解释 | 偏移量 | 消息长度 | 消息tagHashCode | 这个队列里面的第几个消息 | ...... | 174 | | 占用 | long(8个字节) | int(4个字节) | int(4个字节) | long(8个字节) | ...... | 175 | 176 | 177 | 178 | ## 2.3 消息拉取 PullExecutor 179 | ### 2.3.1 消息队列queue 180 | 181 | - 消息在存储到 **mq_n** 之后，会将消息分配到 消息队列之后，然后消费者在拉取消息的时候，会指定queueId来进行拉取数据。 182 | - 拉取消息的话，会首先读取queue的信息，读取出指定偏移量的n条数据的信息，然后去 **mq_n** 去查询 183 | ```java 184 | public List pullMessage(Pull pull){ 185 | //读取n条数据 186 | List data = 187 | MessageDurabilityStorage.readMessageQueue(pull.getQueueId(), 188 | pull.getTopic(), 189 | pull.getOffset()); 190 | if (data.size()==0){ 191 | return null; 192 | } 193 | //然后在去根据元信息读取消息 194 | return MessageStorage.readMessages(data); 195 | } 196 | ``` 197 | ## 2.4 消息消费位点保存 CommitOffsetExecutor 198 | ### 2.4.1 消费位点存储 199 | 200 | - 消费位点存储使用JSON格式存储，格式为offSetMap下面的key代表消费者,Value代表每个队列的消费点 201 | ```java 202 | { 203 | "offSetMap": { 204 | "CConsumer": { 205 | "0": 358, 206 | "1": 363, 207 | "2": 363, 208 | "3": 359 209 | }, 210 | "AConsumer": { 211 | "0": 331, 212 | "1": 336, 213 | "2": 335, 214 | "3": 332 215 | }, 216 | "RConsumer": { 217 | "0": 486, 218 | "1": 492, 219 | "2": 492, 220 | "3": 487 221 | }, 222 | "DConsumer": { 223 | "0": 358, 224 | "1": 363, 225 | "2": 363, 226 | "3": 359 227 | }, 228 | "BConsumer": { 229 | "0": 350, 230 | "1": 355, 231 | "2": 355, 232 | "3": 351 233 | } 234 | } 235 | } 236 | ``` 237 | ## 2.4 Topic信息返回 TopicDataExecutor 238 | 239 | - 将Topic对应的队列数量，消费量等返回给客户端 240 | ```java 241 | messageQueueData.setTopic(data.getTopic()); 242 | messageQueueData.setConsumerKey(data.getConsumerKey()); 243 | messageQueueData.setQueueSize(topicListener.getQueueSize()); 244 | messageQueueData.setOffSetMap(QueueOffsetStorage.getOffSetMap(data.getTopic(),data.getConsumerGroup())); 245 | ``` 246 | ## 2.5 基础架构图 247 | # ![image.png](https://github.com/MortyCode/HsMq/blob/master/zimage/server.png?raw=true) 248 | 249 | # 3. 客户端 250 | ## 3.1 消息发送 251 | 252 | - 消息发送目前很简单，组装数据后直接调用netty的writeAndFlush，添加了一个 Listener 以及一个超时的判断。 253 | - 目前实现比较简单，因为这里其实并不是MQ系统的重点 254 | ```java 255 | public static SendMessageResult sendMsg(SendMessage sendMessage){ 256 | SendMessageResult sendMessageResult = new SendMessageResult(); 257 | try { 258 | HsEecodeData hsEecodeData = new HsEecodeData(); 259 | .... 260 | 261 | resultMap.put(hsReq.getReqId(), sendMessageResult); 262 | 263 | MessageClient.channelFuture 264 | .channel() 265 | .writeAndFlush(hsEecodeData) 266 | .addListener((future)->{ 267 | if (future.isSuccess()) { 268 | sendMessageResult.setSendDone(true); 269 | } else { 270 | sendMessageResult.setSendDone(false); 271 | sendMessageResult.setRespDesc("消息发送失败"); 272 | } 273 | }); 274 | 275 | long nanosTimeout = TimeUnit.SECONDS.toNanos(3); 276 | final long deadline = System.nanoTime() + nanosTimeout; 277 | 278 | while (true) { 279 | if (nanosTimeout<0){ 280 | sendMessageResult.setRespDesc("发送超时"); 281 | sendMessageResult.setMessageResult(-1); 282 | break; 283 | } 284 | if (sendMessageResult.getMessageResult() != null) { 285 | break; 286 | } 287 | nanosTimeout = deadline - System.nanoTime(); 288 | } 289 | return sendMessageResult; 290 | } catch (Exception e) { 291 | sendMessageResult.setMessageResult(-2); 292 | sendMessageResult.setSendDone(false); 293 | e.printStackTrace(); 294 | } 295 | return sendMessageResult; 296 | } 297 | ``` 298 | ## 3.2 消息消费 299 | ### 3.2.1 消费者初始化 300 | #### 目前消费者存储结构 301 | ```java 302 | Map> 303 | 304 | { 305 | "consumerGroup": { 306 | "Topic": Consumer, 307 | "Topic": Consumer, 308 | }, 309 | "consumerGroup": { 310 | "Topic": Consumer, 311 | "Topic": Consumer, 312 | }, 313 | } 314 | 315 | 系统内有多个消费组，一个消费组内有多个Topic对应的消费者 316 | ``` 317 | #### 初始化消费者 318 | 319 | - 消息管理器 ConsumerMessageQueue 320 | - 注册拉取消息任务 321 | - 注册执行器任务,异步的从消息管理器中拉取数据进行消费 322 | - 注册定时任务,定时向服务端提交消费偏移量 323 | ```java 324 | public static void registeredConsumer(String topic,String consumerGroup){ 325 | 326 | ThreadPoolExecutor executor = ExecutorService.getExecutor(); 327 | //创建消费者 328 | ConsumerMessageQueue consumerMessageQueue = new ConsumerMessageQueue(topic,consumerGroup); 329 | //注册到管理器中 330 | consumerMessageQueueMap.put(consumerKey(topic,consumerGroup),consumerMessageQueue); 331 | //注册拉取消息任务 332 | executor.execute(new PullMessageTask(channelFuture , consumerGroup, consumerMessageQueue)); 333 | //注册执行器任务 334 | executor.execute(new ExecutorMessageTask(channelFuture ,consumerMessageQueue)); 335 | //定时任务 336 | channelFuture.channel().eventLoop().scheduleWithFixedDelay(()->{ 337 | //定时任务,定时向服务端提交消费偏移量 338 | new Thread(new CommitOffsetTask(channelFuture,consumerMessageQueue)).start(); 339 | },1, 3L, TimeUnit.SECONDS); 340 | 341 | } 342 | ``` 343 | 344 | 345 | #### 初始化消费位点 346 | 347 | - 首先向服务发出获取对应topic对应消费组的 TopicData 的请求 348 | ```java 349 | public static void initConsumerQueue(String consumerGroup){ 350 | 351 | consumerMessageQueueMap.forEach((consumerKey,queue)->{ 352 | HsEecodeData hsEecodeData = new HsEecodeData(); 353 | ..... 354 | hsReq.setOperation(OperationEnum.TopicData.getOperation()); 355 | hsEecodeData.setData(hsReq); 356 | channelFuture.channel().writeAndFlush(hsEecodeData).sync(); 357 | }); 358 | } 359 | ``` 360 | 361 | - 然后根据服务端的返回同步消费数据 362 | - 目前是为所有的队列都创建消息队列，然后把获取的queueId , offset 对应的信息初始化到 消费者里面，这样消费者就可以根据偏移量去服务端拉取数据 363 | ```java 364 | public void initQueue(MessageQueueData messageQueueData){ 365 | Integer queueSize = messageQueueData.getQueueSize(); 366 | for (int i=0;i()); 368 | } 369 | Map serverOffSetMap = messageQueueData.getOffSetMap(); 370 | if (serverOffSetMap!=null&&serverOffSetMap.size()>0){ 371 | offSetMap.putAll(serverOffSetMap); 372 | lastMessageMap.putAll(serverOffSetMap); 373 | } 374 | } 375 | ``` 376 | 377 | ## 3.3 消费者整体结构图 378 | # ![image.png](https://github.com/MortyCode/HsMq/blob/master/zimage/consumer.png?raw=true) 379 | 380 | 381 | ## 3.4 目前还没有做消费的负载均衡 382 | ## ​ 383 | 384 | 385 | 386 | 387 | -------------------------------------------------------------------------------- /common/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 5 | 6 | hsmq 7 | com.heshen 8 | 1.0-SNAPSHOT 9 | 10 | 4.0.0 11 | 12 | common 13 | 14 | 15 | -------------------------------------------------------------------------------- /common/src/main/java/com/hsmq/common/exception/FileException.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.common.exception; 2 | 3 | /** 4 | * @author ：河神 5 | * @date ：Created in 2021/6/12 1:50 下午 6 | */ 7 | public class FileException extends RuntimeException{ 8 | 9 | public FileException(String message) { 10 | super(message); 11 | } 12 | } 13 | -------------------------------------------------------------------------------- /common/src/main/java/com/hsmq/common/utils/IpUtils.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.common.utils; 2 | 3 | import java.net.InetAddress; 4 | import java.net.UnknownHostException; 5 | 6 | /** 7 | * @author ：河神 8 | * @date ：Created in 2021/7/10 4:11 下午 9 | */ 10 | public class IpUtils { 11 | 12 | public static String getIp(){ 13 | try { 14 | return InetAddress.getLocalHost().getHostAddress(); 15 | } catch (UnknownHostException e) { 16 | e.printStackTrace(); 17 | } 18 | return null; 19 | } 20 | 21 | 22 | } 23 | -------------------------------------------------------------------------------- /common/src/main/java/com/hsmq/common/utils/MessageIdUtils.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.common.utils; 2 | 3 | import java.lang.management.ManagementFactory; 4 | 5 | /** 6 | * SnowFlake 算法 生产messageID 7 | * @author ：河神 8 | * @date ：Created in 2021/7/10 3:53 下午 9 | */ 10 | public class MessageIdUtils { 11 | 12 | public static String newMsgId(String topic){ 13 | String name = ManagementFactory.getRuntimeMXBean().getName(); 14 | String s = name.split("@")[0]; 15 | 16 | return topic+":"+s+":"+System.nanoTime(); 17 | } 18 | 19 | } 20 | -------------------------------------------------------------------------------- /common/src/main/java/com/hsmq/common/utils/MurmurHash.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.common.utils; 2 | 3 | 4 | /** 5 | * A fast non-encrypted hash 6 | * Suitable for occasions that do not require high confidentiality and do not care about hash collision attacks 7 | */ 8 | public class MurmurHash { 9 | 10 | public static int hash(Object o) { 11 | if (o == null) { 12 | return 0; 13 | } 14 | if (o instanceof Long) { 15 | return hashLong((Long) o); 16 | } 17 | if (o instanceof Integer) { 18 | return hashLong((Integer) o); 19 | } 20 | if (o instanceof Double) { 21 | return hashLong(Double.doubleToRawLongBits((Double) o)); 22 | } 23 | if (o instanceof Float) { 24 | return hashLong(Float.floatToRawIntBits((Float) o)); 25 | } 26 | if (o instanceof String) { 27 | return hash(((String) o).getBytes()); 28 | } 29 | if (o instanceof byte[]) { 30 | return hash((byte[]) o); 31 | } 32 | return hash(o.toString()); 33 | } 34 | 35 | public static int hash(byte[] data) { 36 | return hash(data, data.length, -1); 37 | } 38 | 39 | public static int hash(byte[] data, int seed) { 40 | return hash(data, data.length, seed); 41 | } 42 | 43 | public static int hash(byte[] data, int length, int seed) { 44 | int m = 0x5bd1e995; 45 | int r = 24; 46 | 47 | int h = seed ^ length; 48 | 49 | int len_4 = length >> 2; 50 | 51 | for (int i = 0; i < len_4; i++) { 52 | int i_4 = i << 2; 53 | int k = data[i_4 + 3]; 54 | k = k << 8; 55 | k = k | (data[i_4 + 2] & 0xff); 56 | k = k << 8; 57 | k = k | (data[i_4 + 1] & 0xff); 58 | k = k << 8; 59 | k = k | (data[i_4 + 0] & 0xff); 60 | k *= m; 61 | k ^= k >>> r; 62 | k *= m; 63 | h *= m; 64 | h ^= k; 65 | } 66 | 67 | // avoid calculating modulo 68 | int len_m = len_4 << 2; 69 | int left = length - len_m; 70 | 71 | if (left != 0) { 72 | if (left >= 3) { 73 | h ^= (int) data[length - 3] << 16; 74 | } 75 | if (left >= 2) { 76 | h ^= (int) data[length - 2] << 8; 77 | } 78 | if (left >= 1) { 79 | h ^= (int) data[length - 1]; 80 | } 81 | 82 | h *= m; 83 | } 84 | 85 | h ^= h >>> 13; 86 | h *= m; 87 | h ^= h >>> 15; 88 | 89 | return h; 90 | } 91 | 92 | public static int hashLong(long data) { 93 | int m = 0x5bd1e995; 94 | int r = 24; 95 | 96 | int h = 0; 97 | 98 | int k = (int) data * m; 99 | k ^= k >>> r; 100 | h ^= k * m; 101 | 102 | k = (int) (data >> 32) * m; 103 | k ^= k >>> r; 104 | h *= m; 105 | h ^= k * m; 106 | 107 | h ^= h >>> 13; 108 | h *= m; 109 | h ^= h >>> 15; 110 | 111 | return h; 112 | } 113 | 114 | public static long hash64(Object o) { 115 | if (o == null) { 116 | return 0l; 117 | } else if (o instanceof String) { 118 | final byte[] bytes = ((String) o).getBytes(); 119 | return hash64(bytes, bytes.length); 120 | } else if (o instanceof byte[]) { 121 | final byte[] bytes = (byte[]) o; 122 | return hash64(bytes, bytes.length); 123 | } 124 | return hash64(o.toString()); 125 | } 126 | 127 | // 64 bit implementation copied from here: https://github.com/tnm/murmurhash-java 128 | 129 | /** 130 | * Generates 64 bit hash from byte array with default seed value. 131 | * 132 | * @param data byte array to hash 133 | * @param length length of the array to hash 134 | * @return 64 bit hash of the given string 135 | */ 136 | public static long hash64(final byte[] data, int length) { 137 | return hash64(data, length, 0xe17a1465); 138 | } 139 | 140 | 141 | /** 142 | * Generates 64 bit hash from byte array of the given length and seed. 143 | * 144 | * @param data byte array to hash 145 | * @param length length of the array to hash 146 | * @param seed initial seed value 147 | * @return 64 bit hash of the given array 148 | */ 149 | public static long hash64(final byte[] data, int length, int seed) { 150 | final long m = 0xc6a4a7935bd1e995L; 151 | final int r = 47; 152 | 153 | long h = (seed & 0xffffffffl) ^ (length * m); 154 | 155 | int length8 = length / 8; 156 | 157 | for (int i = 0; i < length8; i++) { 158 | final int i8 = i * 8; 159 | long k = ((long) data[i8 + 0] & 0xff) + (((long) data[i8 + 1] & 0xff) << 8) 160 | + (((long) data[i8 + 2] & 0xff) << 16) + (((long) data[i8 + 3] & 0xff) << 24) 161 | + (((long) data[i8 + 4] & 0xff) << 32) + (((long) data[i8 + 5] & 0xff) << 40) 162 | + (((long) data[i8 + 6] & 0xff) << 48) + (((long) data[i8 + 7] & 0xff) << 56); 163 | 164 | k *= m; 165 | k ^= k >>> r; 166 | k *= m; 167 | 168 | h ^= k; 169 | h *= m; 170 | } 171 | 172 | switch (length % 8) { 173 | case 7: 174 | h ^= (long) (data[(length & ~7) + 6] & 0xff) << 48; 175 | case 6: 176 | h ^= (long) (data[(length & ~7) + 5] & 0xff) << 40; 177 | case 5: 178 | h ^= (long) (data[(length & ~7) + 4] & 0xff) << 32; 179 | case 4: 180 | h ^= (long) (data[(length & ~7) + 3] & 0xff) << 24; 181 | case 3: 182 | h ^= (long) (data[(length & ~7) + 2] & 0xff) << 16; 183 | case 2: 184 | h ^= (long) (data[(length & ~7) + 1] & 0xff) << 8; 185 | case 1: 186 | h ^= (long) (data[length & ~7] & 0xff); 187 | h *= m; 188 | } 189 | ; 190 | 191 | h ^= h >>> r; 192 | h *= m; 193 | h ^= h >>> r; 194 | 195 | return h; 196 | } 197 | } -------------------------------------------------------------------------------- /io/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 5 | 6 | hsmq 7 | com.heshen 8 | 1.0-SNAPSHOT 9 | 10 | 4.0.0 11 | pom 12 | io 13 | 14 | 15 | protocol 16 | storage 17 | 18 | 19 | 20 | -------------------------------------------------------------------------------- /io/protocol/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 5 | 6 | io 7 | com.heshen 8 | 1.0-SNAPSHOT 9 | 10 | 4.0.0 11 | protocol 12 | 13 | 14 | 15 | io.netty 16 | netty-all 17 | 18 | 19 | 20 | ch.qos.logback 21 | logback-classic 22 | 23 | 24 | 25 | 26 | 27 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/data/Head.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.data; 2 | 3 | import com.hsmq.enums.MessageEnum; 4 | import com.hsmq.utils.ObjectByteUtils; 5 | 6 | import java.io.Serializable; 7 | 8 | /** 9 | * @author ：河神 10 | * @date ：Created in 2021/6/8 8:26 下午 11 | */ 12 | public class Head implements Serializable { 13 | 14 | private static final long serialVersionUID = -20210610L; 15 | 16 | 17 | /** 18 | * @see MessageEnum 19 | * 消息类型：Req 和 Resp 20 | */ 21 | private String msgType; 22 | 23 | public String getMsgType() { 24 | return msgType; 25 | } 26 | 27 | public MessageEnum getMsgTypeEnum() { 28 | return MessageEnum.getByCode(msgType); 29 | } 30 | 31 | 32 | public void setMsgType(String msgType) { 33 | this.msgType = msgType; 34 | } 35 | 36 | 37 | public static Head toHead(byte[] headData){ 38 | return (Head) ObjectByteUtils.toObject(headData); 39 | } 40 | 41 | public static Head toHead(MessageEnum messageEnum){ 42 | Head head = new Head(); 43 | head.setMsgType(messageEnum.getCode()); 44 | return head; 45 | } 46 | } 47 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/data/HsReq.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.data; 2 | 3 | import com.hsmq.enums.OperationEnum; 4 | 5 | import java.io.Serializable; 6 | 7 | /** 8 | * @author ：河神 9 | * @date ：Created in 2021/6/8 8:12 下午 10 | */ 11 | public class HsReq implements Serializable { 12 | 13 | private static final long serialVersionUID = -20210610L; 14 | 15 | 16 | /** 17 | * @see OperationEnum 18 | */ 19 | private String operation; 20 | 21 | private T data; 22 | 23 | private Integer reqId; 24 | 25 | public String getOperation() { 26 | return operation; 27 | } 28 | 29 | public OperationEnum getOperationEnum() { 30 | return OperationEnum.getByCode(operation); 31 | } 32 | 33 | public void setOperation(String operation) { 34 | this.operation = operation; 35 | } 36 | 37 | public T getData() { 38 | return data; 39 | } 40 | 41 | public void setData(T data) { 42 | this.data = data; 43 | } 44 | 45 | public Integer getReqId() { 46 | return reqId; 47 | } 48 | 49 | public void setReqId(Integer reqId) { 50 | this.reqId = reqId; 51 | } 52 | 53 | public HsReq convertReq(){ 54 | HsReq data = new HsReq<>(); 55 | data.setData((T)data); 56 | data.setOperation(getOperation()); 57 | return data; 58 | 59 | } 60 | } 61 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/data/HsResp.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.data; 2 | 3 | import com.hsmq.enums.ResultEnum; 4 | 5 | import java.io.Serializable; 6 | import java.util.List; 7 | 8 | /** 9 | * @author ：河神 10 | * @date ：Created in 2021/6/8 8:12 下午 11 | */ 12 | public class HsResp implements Serializable { 13 | 14 | private static final long serialVersionUID = -20210610L; 15 | 16 | 17 | private Integer result; 18 | 19 | private String operation; 20 | 21 | private String reqType; 22 | 23 | private Integer reqId; 24 | 25 | private boolean success = true; 26 | 27 | private T data; 28 | 29 | private List datas; 30 | 31 | public Integer getResult() { 32 | return result; 33 | } 34 | 35 | public void setResult(Integer result) { 36 | this.result = result; 37 | } 38 | 39 | public String getOperation() { 40 | return operation; 41 | } 42 | 43 | public void setOperation(String operation) { 44 | this.operation = operation; 45 | } 46 | 47 | public T getData() { 48 | return data; 49 | } 50 | 51 | public void setData(T data) { 52 | this.data = data; 53 | } 54 | 55 | public List getDatas() { 56 | return datas; 57 | } 58 | 59 | public boolean isSuccess() { 60 | return success; 61 | } 62 | 63 | public void setSuccess(boolean success) { 64 | this.success = success; 65 | } 66 | 67 | public void setDatas(List datas) { 68 | this.datas = datas; 69 | } 70 | 71 | public String getReqType() { 72 | return reqType; 73 | } 74 | 75 | public Integer getReqId() { 76 | return reqId; 77 | } 78 | 79 | public void setReqId(Integer reqId) { 80 | this.reqId = reqId; 81 | } 82 | 83 | public void setReqType(String reqType) { 84 | this.reqType = reqType; 85 | } 86 | 87 | public static HsResp typeError(){ 88 | HsResp resp = new HsResp<>(); 89 | resp.setResult(ResultEnum.ParameterWrongType.getCode()); 90 | resp.setSuccess(false); 91 | return resp; 92 | } 93 | 94 | public static HsResp topicNotExistsError(){ 95 | HsResp resp = new HsResp<>(); 96 | resp.setResult(ResultEnum.TopicNotExists.getCode()); 97 | resp.setSuccess(false); 98 | return resp; 99 | } 100 | 101 | public static HsResp error(ResultEnum resultEnum){ 102 | HsResp resp = new HsResp<>(); 103 | resp.setResult(ResultEnum.ParameterWrongType.getCode()); 104 | resp.setSuccess(false); 105 | return resp; 106 | } 107 | } 108 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/data/message/Confirm.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.data.message; 2 | 3 | /** 4 | * @author ：河神 5 | * @date ：Created in 2021/6/12 10:50 下午 6 | */ 7 | public class Confirm { 8 | 9 | private int offset; 10 | private int queueId; 11 | 12 | public int getOffset() { 13 | return offset; 14 | } 15 | 16 | public void setOffset(int offset) { 17 | this.offset = offset; 18 | } 19 | 20 | public int getQueueId() { 21 | return queueId; 22 | } 23 | 24 | public void setQueueId(int queueId) { 25 | this.queueId = queueId; 26 | } 27 | } 28 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/data/message/MessageQueueData.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.data.message; 2 | 3 | import java.io.Serializable; 4 | import java.util.Map; 5 | 6 | /** 7 | * @author ：河神 8 | * @date ：Created in 2021/6/7 3:05 下午 9 | */ 10 | public class MessageQueueData implements Serializable { 11 | 12 | private static final long serialVersionUID = -20210610L; 13 | 14 | private String topic; 15 | 16 | private String consumerKey; 17 | 18 | private Integer queueSize; 19 | 20 | private Map offSetMap; 21 | 22 | public Integer getQueueSize() { 23 | return queueSize; 24 | } 25 | 26 | public void setQueueSize(Integer queueSize) { 27 | this.queueSize = queueSize; 28 | } 29 | 30 | public Map getOffSetMap() { 31 | return offSetMap; 32 | } 33 | 34 | public void setOffSetMap(Map offSetMap) { 35 | this.offSetMap = offSetMap; 36 | } 37 | 38 | public String getTopic() { 39 | return topic; 40 | } 41 | 42 | public void setTopic(String topic) { 43 | this.topic = topic; 44 | } 45 | 46 | public String getConsumerKey() { 47 | return consumerKey; 48 | } 49 | 50 | public void setConsumerKey(String consumerKey) { 51 | this.consumerKey = consumerKey; 52 | } 53 | } 54 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/data/message/Pull.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.data.message; 2 | 3 | import java.io.Serializable; 4 | 5 | /** 6 | * @author ：河神 7 | * @date ：Created in 2021/6/7 3:05 下午 8 | */ 9 | public class Pull implements Serializable { 10 | 11 | private static final long serialVersionUID = -20210610L; 12 | 13 | private String consumerGroup; 14 | private int queueId; 15 | private String topic; 16 | private int size; 17 | private long offset; 18 | private String tags; 19 | 20 | public String getTopic() { 21 | return topic; 22 | } 23 | 24 | public void setTopic(String topic) { 25 | this.topic = topic; 26 | } 27 | 28 | public String getConsumerGroup() { 29 | return consumerGroup; 30 | } 31 | 32 | public void setConsumerGroup(String consumerGroup) { 33 | this.consumerGroup = consumerGroup; 34 | } 35 | 36 | public String getTags() { 37 | return tags; 38 | } 39 | 40 | public void setTags(String tags) { 41 | this.tags = tags; 42 | } 43 | 44 | public int getSize() { 45 | return size; 46 | } 47 | 48 | public void setSize(int size) { 49 | this.size = size; 50 | } 51 | 52 | public int getQueueId() { 53 | return queueId; 54 | } 55 | 56 | public void setQueueId(int queueId) { 57 | this.queueId = queueId; 58 | } 59 | 60 | public long getOffset() { 61 | return offset; 62 | } 63 | 64 | public void setOffset(long offset) { 65 | this.offset = offset; 66 | } 67 | } 68 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/data/message/PullMessage.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.data.message; 2 | 3 | import java.io.Serializable; 4 | import java.util.StringJoiner; 5 | 6 | /** 7 | * @author ：河神 8 | * @date ：Created in 2021/6/7 3:05 下午 9 | */ 10 | public class PullMessage implements Serializable { 11 | 12 | private static final long serialVersionUID = -20210610L; 13 | 14 | private String msgId; 15 | private String topic; 16 | private String tag; 17 | private String key; 18 | private String body; 19 | private Long index; 20 | 21 | 22 | 23 | public String getMsgId() { 24 | return msgId; 25 | } 26 | 27 | public void setMsgId(String msgId) { 28 | this.msgId = msgId; 29 | } 30 | 31 | public String getTopic() { 32 | return topic; 33 | } 34 | 35 | public void setTopic(String topic) { 36 | this.topic = topic; 37 | } 38 | 39 | public String getTag() { 40 | return tag; 41 | } 42 | 43 | public void setTag(String tag) { 44 | this.tag = tag; 45 | } 46 | 47 | public String getKey() { 48 | return key; 49 | } 50 | 51 | public void setKey(String key) { 52 | this.key = key; 53 | } 54 | 55 | public String getBody() { 56 | return body; 57 | } 58 | 59 | public void setBody(String body) { 60 | this.body = body; 61 | } 62 | 63 | @Override 64 | public String toString() { 65 | return new StringJoiner(", ", PullMessage.class.getSimpleName() + "[", "]") 66 | .add("msgId='" + msgId + "'") 67 | .add("topic='" + topic + "'") 68 | .add("tag='" + tag + "'") 69 | .add("key='" + key + "'") 70 | .add("body='" + body + "'") 71 | .add("index=" + index) 72 | .toString(); 73 | } 74 | 75 | public Long getIndex() { 76 | return index; 77 | } 78 | 79 | public void setIndex(Long index) { 80 | this.index = index; 81 | } 82 | 83 | } 84 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/data/message/PullMessageResp.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.data.message; 2 | 3 | import java.io.Serializable; 4 | import java.util.List; 5 | 6 | /** 7 | * @author ：河神 8 | * @date ：Created in 2021/10/2 3:00 下午 9 | */ 10 | public class PullMessageResp implements Serializable { 11 | private static final long serialVersionUID = 6307404206182632758L; 12 | 13 | private List pullMessages; 14 | 15 | private String topic; 16 | 17 | private String consumerGroup; 18 | 19 | private Integer queueId; 20 | 21 | private Long lastIndex; 22 | 23 | public List getPullMessages() { 24 | return pullMessages; 25 | } 26 | 27 | public void setPullMessages(List pullMessages) { 28 | this.pullMessages = pullMessages; 29 | } 30 | 31 | public String getTopic() { 32 | return topic; 33 | } 34 | 35 | public void setTopic(String topic) { 36 | this.topic = topic; 37 | } 38 | 39 | public Integer getQueueId() { 40 | return queueId; 41 | } 42 | 43 | public void setQueueId(Integer queueId) { 44 | this.queueId = queueId; 45 | } 46 | 47 | public Long getLastIndex() { 48 | return lastIndex; 49 | } 50 | 51 | public void setLastIndex(Long lastIndex) { 52 | this.lastIndex = lastIndex; 53 | } 54 | 55 | public String getConsumerGroup() { 56 | return consumerGroup; 57 | } 58 | 59 | public void setConsumerGroup(String consumerGroup) { 60 | this.consumerGroup = consumerGroup; 61 | } 62 | } 63 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/data/message/SendMessage.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.data.message; 2 | 3 | import java.io.Serializable; 4 | import java.util.StringJoiner; 5 | 6 | /** 7 | * @author ：河神 8 | * @date ：Created in 2021/6/7 3:05 下午 9 | */ 10 | public class SendMessage implements Serializable { 11 | 12 | private static final long serialVersionUID = -20210610L; 13 | 14 | 15 | private String msgId; 16 | private String topic; 17 | private String tag; 18 | private String key; 19 | private String body; 20 | 21 | 22 | public String getMsgId() { 23 | return msgId; 24 | } 25 | 26 | public void setMsgId(String msgId) { 27 | this.msgId = msgId; 28 | } 29 | 30 | public String getTopic() { 31 | return topic; 32 | } 33 | 34 | public void setTopic(String topic) { 35 | this.topic = topic; 36 | } 37 | 38 | public String getTag() { 39 | return tag; 40 | } 41 | 42 | public void setTag(String tag) { 43 | this.tag = tag; 44 | } 45 | 46 | public String getKey() { 47 | return key; 48 | } 49 | 50 | public void setKey(String key) { 51 | this.key = key; 52 | } 53 | 54 | public String getBody() { 55 | return body; 56 | } 57 | 58 | public void setBody(String body) { 59 | this.body = body; 60 | } 61 | 62 | @Override 63 | public String toString() { 64 | return new StringJoiner(", ", SendMessage.class.getSimpleName() + "[", "]") 65 | .add("msgId='" + msgId + "'") 66 | .add("topic='" + topic + "'") 67 | .add("tag='" + tag + "'") 68 | .add("key='" + key + "'") 69 | .add("body='" + body + "'") 70 | .toString(); 71 | } 72 | } 73 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/data/message/SyncOffsetMessage.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.data.message; 2 | 3 | import java.io.Serializable; 4 | import java.util.Map; 5 | 6 | /** 7 | * @author ：河神 8 | * @date ：Created in 2021/6/7 3:05 下午 9 | */ 10 | public class SyncOffsetMessage implements Serializable { 11 | 12 | private static final long serialVersionUID = -20210610L; 13 | 14 | private String consumer; 15 | private String topic; 16 | private Map offSetMap; 17 | 18 | public String getConsumer() { 19 | return consumer; 20 | } 21 | 22 | public void setConsumer(String consumer) { 23 | this.consumer = consumer; 24 | } 25 | 26 | public String getTopic() { 27 | return topic; 28 | } 29 | 30 | public void setTopic(String topic) { 31 | this.topic = topic; 32 | } 33 | 34 | public Map getOffSetMap() { 35 | return offSetMap; 36 | } 37 | 38 | public void setOffSetMap(Map offSetMap) { 39 | this.offSetMap = offSetMap; 40 | } 41 | } 42 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/data/message/TopicData.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.data.message; 2 | 3 | import java.io.Serializable; 4 | 5 | /** 6 | * @author ：河神 7 | * @date ：Created in 2021/6/7 3:05 下午 8 | */ 9 | public class TopicData implements Serializable { 10 | 11 | private static final long serialVersionUID = -20210610L; 12 | 13 | private String consumerKey; 14 | private String consumerGroup; 15 | private String topic; 16 | 17 | public String getConsumerKey() { 18 | return consumerKey; 19 | } 20 | 21 | public void setConsumerKey(String consumerKey) { 22 | this.consumerKey = consumerKey; 23 | } 24 | 25 | public String getConsumerGroup() { 26 | return consumerGroup; 27 | } 28 | 29 | public void setConsumerGroup(String consumerGroup) { 30 | this.consumerGroup = consumerGroup; 31 | } 32 | 33 | public String getTopic() { 34 | return topic; 35 | } 36 | 37 | public void setTopic(String topic) { 38 | this.topic = topic; 39 | } 40 | } 41 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/decode/LengthObjectDecode.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.decode; 2 | 3 | import com.hsmq.data.Head; 4 | import com.hsmq.enums.MessageEnum; 5 | import com.hsmq.protocol.HsDecodeData; 6 | import com.hsmq.utils.ObjectByteUtils; 7 | import io.netty.buffer.ByteBuf; 8 | import io.netty.channel.ChannelHandlerContext; 9 | import io.netty.handler.codec.LengthFieldBasedFrameDecoder; 10 | 11 | import java.nio.ByteBuffer; 12 | 13 | /** 14 | * @author: 河神 15 | * @date:2020-04-19 16 | */ 17 | public class LengthObjectDecode extends LengthFieldBasedFrameDecoder { 18 | 19 | 20 | private static final int FRAME_MAX_LENGTH = 21 | Integer.parseInt(System.getProperty("com.hsmq.frameMaxLength", "16777216")); 22 | 23 | public LengthObjectDecode() { 24 | super(FRAME_MAX_LENGTH, 0, 4, 0, 4); 25 | } 26 | 27 | @Override 28 | public Object decode(ChannelHandlerContext ctx, ByteBuf in) throws Exception { 29 | 30 | ByteBuf frame = null; 31 | 32 | try { 33 | frame = (ByteBuf) super.decode(ctx, in); 34 | if (null == frame) { 35 | return null; 36 | 37 | } 38 | ByteBuffer byteBuffer = frame.nioBuffer(); 39 | 40 | int headLength = byteBuffer.getInt(); 41 | byte[] headData = new byte[headLength]; 42 | byteBuffer.get(headData); 43 | 44 | Head head = Head.toHead(headData); 45 | if (head==null){ 46 | return null; 47 | } 48 | 49 | int dataLength = byteBuffer.getInt(); 50 | byte[] dataData = new byte[dataLength]; 51 | byteBuffer.get(dataData); 52 | 53 | MessageEnum msgTypeEnum = head.getMsgTypeEnum(); 54 | if (msgTypeEnum==null){ 55 | return null; 56 | } 57 | 58 | HsDecodeData decodeData = new HsDecodeData(); 59 | decodeData.setHead(head); 60 | decodeData.setData(ObjectByteUtils.toObject(dataData)); 61 | decodeData.setMsgTypeEnum(msgTypeEnum); 62 | return decodeData; 63 | }catch (Exception e){ 64 | e.printStackTrace(); 65 | ctx.channel().close(); 66 | }finally { 67 | if (frame!=null){ 68 | frame.release(); 69 | } 70 | } 71 | return null; 72 | 73 | } 74 | 75 | } 76 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/decode/ObjectDecode.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.decode; 2 | 3 | import com.hsmq.protocol.HsMessage; 4 | import io.netty.buffer.ByteBuf; 5 | import io.netty.channel.ChannelHandlerContext; 6 | import io.netty.handler.codec.ByteToMessageDecoder; 7 | 8 | import java.util.List; 9 | 10 | /** 11 | * @author: 河神 12 | * @date:2020-04-19 13 | */ 14 | public class ObjectDecode extends ByteToMessageDecoder { 15 | 16 | 17 | private HsMessage hsMessage; 18 | 19 | public ObjectDecode() { 20 | this.hsMessage = new HsMessage<>(); 21 | } 22 | 23 | @Override 24 | protected void decode(ChannelHandlerContext ctx, ByteBuf byteBuf, List out) throws Exception { 25 | 26 | while (byteBuf.isReadable()&&byteBuf.readableBytes()>0) { 27 | if (hsMessage.getLength() == null) { 28 | if (byteBuf.readableBytes()>=4){ 29 | int head = byteBuf.readInt(); 30 | hsMessage.setLength(head); 31 | }else { 32 | break; 33 | } 34 | } else { 35 | Integer length = hsMessage.getLength(); 36 | if (hsMessage.getDataArray() != null) { 37 | length = hsMessage.getLength() - hsMessage.getDataArray().length; 38 | } 39 | if (length>byteBuf.readableBytes()){ 40 | length = byteBuf.readableBytes(); 41 | } 42 | byte[] b2 = new byte[length]; 43 | byteBuf.readBytes(b2); 44 | hsMessage.appendArray(b2); 45 | } 46 | if (hsMessage.arrayFinish()) { 47 | out.add(hsMessage.toData()); 48 | hsMessage.setNull(); 49 | break; 50 | } 51 | } 52 | } 53 | } 54 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/encode/LengthObjectEncode.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.encode; 2 | 3 | import com.hsmq.protocol.HsEecodeData; 4 | import io.netty.buffer.ByteBuf; 5 | import io.netty.channel.ChannelHandlerContext; 6 | import io.netty.handler.codec.MessageToByteEncoder; 7 | 8 | /** 9 | * @author ：河神 10 | * @date ：Created in 2021/6/6 6:38 下午 11 | */ 12 | public class LengthObjectEncode extends MessageToByteEncoder { 13 | 14 | 15 | @Override 16 | protected void encode(ChannelHandlerContext ctx, HsEecodeData encodeData, ByteBuf out) throws Exception { 17 | out.writeInt(encodeData.getLength()); 18 | out.writeInt(encodeData.getHeadLength()); 19 | out.writeBytes(encodeData.getHead()); 20 | out.writeInt(encodeData.getDataLength()); 21 | out.writeBytes(encodeData.getData()); 22 | } 23 | } 24 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/encode/ObjectEncode.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.encode; 2 | 3 | import com.hsmq.data.message.SendMessage; 4 | import com.hsmq.protocol.HsMessage; 5 | import io.netty.buffer.ByteBuf; 6 | import io.netty.channel.ChannelHandlerContext; 7 | import io.netty.handler.codec.MessageToByteEncoder; 8 | 9 | /** 10 | * @author ：河神 11 | * @date ：Created in 2021/6/6 6:38 下午 12 | */ 13 | public class ObjectEncode extends MessageToByteEncoder { 14 | 15 | 16 | @Override 17 | protected void encode(ChannelHandlerContext ctx, SendMessage msg, ByteBuf out) throws Exception { 18 | 19 | HsMessage hsMessage = new HsMessage<>(msg); 20 | 21 | out.writeInt(hsMessage.getLength()); 22 | out.writeBytes(hsMessage.getDataArray()); 23 | } 24 | } 25 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/enums/MessageEnum.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.enums; 2 | 3 | import com.hsmq.data.HsReq; 4 | import com.hsmq.data.HsResp; 5 | 6 | /** 7 | * @author ：河神 8 | * @date ：Created in 2021/6/7 11:20 下午 9 | */ 10 | public enum MessageEnum { 11 | 12 | /** 13 | * 请求 14 | */ 15 | Req("Req", HsReq.class), 16 | /** 17 | * 返回值 18 | */ 19 | Resp("Resp", HsResp.class); 20 | 21 | private String code; 22 | private Class clazz; 23 | 24 | MessageEnum(String code, Class clazz) { 25 | this.code = code; 26 | this.clazz = clazz; 27 | } 28 | 29 | public String getCode() { 30 | return code; 31 | } 32 | 33 | public static MessageEnum getByCode(String code){ 34 | for (MessageEnum value : values()) { 35 | if (value.getCode().equals(code)){ 36 | return value; 37 | } 38 | } 39 | return null; 40 | } 41 | } 42 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/enums/MessageSendEnum.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.enums; 2 | 3 | /** 4 | * @author ：河神 5 | * @date ：Created in 2021/6/7 11:20 下午 6 | */ 7 | public enum MessageSendEnum { 8 | 9 | /** 10 | * 发送成功 11 | */ 12 | SendOK("SendOK"); 13 | 14 | private String code; 15 | 16 | MessageSendEnum(String code) { 17 | this.code = code; 18 | } 19 | public String getCode() { 20 | return code; 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/enums/OperationEnum.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.enums; 2 | 3 | import com.hsmq.data.message.Pull; 4 | 5 | /** 6 | * @author ：河神 7 | * @date ：Created in 2021/6/8 8:38 下午 8 | */ 9 | public enum OperationEnum { 10 | 11 | /** 12 | * 发送消息 13 | */ 14 | SendMessage("SendMessage", com.hsmq.data.message.SendMessage.class), 15 | 16 | /** 17 | * 拉取消息 18 | */ 19 | Pull("Pull", Pull.class), 20 | 21 | /** 22 | * 拉取消息 23 | */ 24 | TopicData("TopicData", Pull.class), 25 | 26 | /** 27 | * 同步确认消息消费位点 28 | */ 29 | CommitOffset("CommitOffset", com.hsmq.data.message.SendMessage.class), 30 | 31 | /** 32 | * 请求返回resp 33 | */ 34 | Resp("Resp", null), 35 | 36 | 37 | 38 | ; 39 | 40 | private String operation; 41 | private Class clazz; 42 | 43 | OperationEnum(String operation, Class clazz) { 44 | this.operation = operation; 45 | this.clazz = clazz; 46 | } 47 | 48 | public String getOperation() { 49 | return operation; 50 | } 51 | 52 | public void setOperation(String operation) { 53 | this.operation = operation; 54 | } 55 | 56 | public Class getClazz() { 57 | return clazz; 58 | } 59 | 60 | public void setClazz(Class clazz) { 61 | this.clazz = clazz; 62 | } 63 | 64 | public static OperationEnum getByCode(String code){ 65 | for (OperationEnum value : values()) { 66 | if (value.getOperation().equals(code)){ 67 | return value; 68 | } 69 | } 70 | return null; 71 | } 72 | 73 | } 74 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/enums/ResultEnum.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.enums; 2 | 3 | /** 4 | * @author ：河神 5 | * @date ：Created in 2021/6/7 11:20 下午 6 | */ 7 | public enum ResultEnum { 8 | 9 | /** 10 | * 发送成功 11 | */ 12 | SendOK(200,"SendOK"), 13 | ParameterWrongType(1000,"Wrong Parameter type"), 14 | TopicNotExists(1001,"Topic Not Exists type"), 15 | 16 | 17 | 18 | ; 19 | 20 | private Integer code; 21 | private String dec; 22 | 23 | ResultEnum(Integer code, String dec) { 24 | this.code = code; 25 | this.dec = dec; 26 | } 27 | 28 | public Integer getCode() { 29 | return code; 30 | } 31 | 32 | public String getDec() { 33 | return dec; 34 | } 35 | } 36 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/enums/SendResultEnum.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.enums; 2 | 3 | /** 4 | * @author ：河神 5 | * @date ：Created in 2021/10/11 3:22 下午 6 | */ 7 | public enum SendResultEnum { 8 | 9 | 10 | 11 | } 12 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/error/HsError.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.error; 2 | 3 | import com.hsmq.data.HsResp; 4 | import com.hsmq.enums.ResultEnum; 5 | 6 | /** 7 | * @author ：河神 8 | * @date ：Created in 2021/6/9 2:30 下午 9 | */ 10 | public class HsError { 11 | 12 | public static final HsResp ParameterWrongType = HsResp.error(ResultEnum.ParameterWrongType); 13 | 14 | 15 | } 16 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/protocol/HsBaseData.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.protocol; 2 | 3 | import java.util.StringJoiner; 4 | 5 | /** 6 | * @author ：河神 7 | * @date ：Created in 2021/6/6 6:42 下午 8 | */ 9 | public class HsBaseData { 10 | 11 | private Integer length; 12 | private String body; 13 | 14 | public boolean isNotNull(){ 15 | return length!=null&&body!=null; 16 | } 17 | 18 | public void setNull(){ 19 | length = null; 20 | body = null; 21 | } 22 | 23 | public HsBaseData copy(){ 24 | HsBaseData hsBaseData = new HsBaseData(); 25 | hsBaseData.setBody(body); 26 | hsBaseData.setLength(length); 27 | return hsBaseData; 28 | } 29 | 30 | public Integer getLength() { 31 | return length; 32 | } 33 | 34 | public void setLength(Integer length) { 35 | this.length = length; 36 | } 37 | 38 | public String getBody() { 39 | return body; 40 | } 41 | 42 | public void setBody(String body) { 43 | this.body = body; 44 | } 45 | 46 | public void appendBody(String body){ 47 | if (this.body==null){ 48 | this.body = body; 49 | }else{ 50 | this.body = this.body+ body; 51 | } 52 | } 53 | 54 | public HsBaseData(String body) { 55 | this.body = body; 56 | this.length = body.getBytes().length; 57 | } 58 | 59 | public HsBaseData() { 60 | } 61 | 62 | @Override 63 | public String toString() { 64 | return new StringJoiner(", ", HsBaseData.class.getSimpleName() + "[", "]") 65 | .add("length=" + length) 66 | .add("body='" + body + "'") 67 | .toString(); 68 | } 69 | } 70 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/protocol/HsDecodeData.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.protocol; 2 | 3 | import com.hsmq.data.Head; 4 | import com.hsmq.data.HsReq; 5 | import com.hsmq.enums.MessageEnum; 6 | 7 | /** 8 | * @author ：河神 9 | * @date ：Created in 2021/6/8 8:50 下午 10 | */ 11 | public class HsDecodeData { 12 | 13 | private Head head; 14 | private MessageEnum msgTypeEnum; 15 | private Object data; 16 | 17 | public HsDecodeData() { 18 | } 19 | 20 | public HsDecodeData(Head head) { 21 | this.head = head; 22 | } 23 | 24 | public Head getHead() { 25 | return head; 26 | } 27 | 28 | public void setHead(Head head) { 29 | this.head = head; 30 | } 31 | 32 | public MessageEnum getMsgTypeEnum() { 33 | return msgTypeEnum; 34 | } 35 | 36 | public void setMsgTypeEnum(MessageEnum msgTypeEnum) { 37 | this.msgTypeEnum = msgTypeEnum; 38 | } 39 | 40 | public Object getData() { 41 | return data; 42 | } 43 | 44 | public void setData(Object data) { 45 | this.data = data; 46 | } 47 | 48 | 49 | public static HsDecodeData req(HsReq data){ 50 | HsDecodeData respHsDecodeData = new HsDecodeData(); 51 | respHsDecodeData.setHead(Head.toHead(MessageEnum.Resp)); 52 | respHsDecodeData.setData(data); 53 | return respHsDecodeData; 54 | } 55 | 56 | 57 | 58 | } 59 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/protocol/HsEecodeData.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.protocol; 2 | 3 | import com.hsmq.data.Head; 4 | import com.hsmq.data.HsResp; 5 | import com.hsmq.enums.MessageEnum; 6 | import com.hsmq.utils.ObjectByteUtils; 7 | 8 | import java.io.Serializable; 9 | 10 | /** 11 | * @author ：河神 12 | * @date ：Created in 2021/6/8 8:50 下午 13 | */ 14 | public class HsEecodeData { 15 | 16 | private byte[] head; 17 | private byte[] data; 18 | 19 | public byte[] getHead() { 20 | return head; 21 | } 22 | 23 | public void setHead(Head head) { 24 | this.head = ObjectByteUtils.toByteArray(head); 25 | } 26 | 27 | public byte[] getData() { 28 | return data; 29 | } 30 | 31 | public void setData(Serializable data) { 32 | this.data = ObjectByteUtils.toByteArray(data); 33 | } 34 | 35 | public int getHeadLength(){ 36 | return head.length; 37 | } 38 | 39 | public int getDataLength(){ 40 | return data.length; 41 | } 42 | 43 | public int getLength(){ 44 | return 4 * 2 + getDataLength() + getHeadLength(); 45 | } 46 | 47 | 48 | public static HsEecodeData resp(HsResp data){ 49 | HsEecodeData respHsDecodeData = new HsEecodeData(); 50 | respHsDecodeData.setHead(Head.toHead(MessageEnum.Resp)); 51 | respHsDecodeData.setData(data); 52 | return respHsDecodeData; 53 | } 54 | 55 | public static HsEecodeData typeError(){ 56 | HsEecodeData respHsDecodeData = new HsEecodeData(); 57 | respHsDecodeData.setHead(Head.toHead(MessageEnum.Resp)); 58 | respHsDecodeData.setData(HsResp.typeError()); 59 | return respHsDecodeData; 60 | } 61 | 62 | } 63 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/protocol/HsMessage.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.protocol; 2 | 3 | import com.hsmq.data.message.SendMessage; 4 | import com.hsmq.utils.ObjectByteUtils; 5 | 6 | import java.lang.reflect.Array; 7 | 8 | /** 9 | * @author ：河神 10 | * @date ：Created in 2021/6/7 2:25 下午 11 | */ 12 | public class HsMessage { 13 | 14 | public static void main(String[] args) { 15 | 16 | HsMessage hsMessage = new HsMessage(); 17 | 18 | hsMessage.appendArray(new byte[]{1,2,3}); 19 | hsMessage.appendArray(new byte[]{4,5,6}); 20 | 21 | SendMessage sendMessage = new SendMessage(); 22 | sendMessage.setTopic("TopicA"); 23 | sendMessage.setTag("TagA"); 24 | sendMessage.setBody("撒撒打算打算打算打算打算打算的撒打算打算的"); 25 | 26 | HsMessage hsMessage2 = new HsMessage<>(sendMessage); 27 | 28 | } 29 | 30 | 31 | private Integer length; 32 | private T data; 33 | private byte[] dataArray; 34 | 35 | public boolean isNotNull(){ 36 | return length!=null&&dataArray!=null; 37 | } 38 | 39 | public void setNull(){ 40 | length = null; 41 | data = null; 42 | dataArray = null; 43 | } 44 | 45 | public boolean arrayFinish(){ 46 | return isNotNull()&&length == dataArray.length; 47 | } 48 | 49 | public void appendArray(byte[] addArray){ 50 | if (addArray==null||addArray.length==0){ 51 | return; 52 | } 53 | if (dataArray==null){ 54 | this.dataArray = addArray; 55 | }else{ 56 | Class type1 = dataArray.getClass().getComponentType(); 57 | byte[] joinedArray = (byte[]) Array.newInstance(type1, dataArray.length + addArray.length); 58 | System.arraycopy(dataArray, 0, joinedArray, 0, dataArray.length); 59 | System.arraycopy(addArray, 0, joinedArray, dataArray.length, addArray.length); 60 | this.dataArray = joinedArray; 61 | } 62 | } 63 | 64 | public HsMessage(T data) { 65 | setData(data); 66 | } 67 | 68 | public HsMessage() { 69 | } 70 | 71 | public Integer getLength() { 72 | return length; 73 | } 74 | 75 | public void setLength(Integer length) { 76 | this.length = length; 77 | } 78 | 79 | public T toData() { 80 | if (data!=null){ 81 | return data; 82 | } 83 | if (dataArray==null){ 84 | return null; 85 | } 86 | return (T) ObjectByteUtils.toObject(dataArray); 87 | } 88 | 89 | public void setData(T data) { 90 | this.data = data; 91 | byte[] bytes = ObjectByteUtils.toByteArray(data); 92 | this.dataArray = bytes; 93 | this.length = bytes.length; 94 | } 95 | 96 | public byte[] getDataArray() { 97 | return dataArray; 98 | } 99 | } 100 | -------------------------------------------------------------------------------- /io/protocol/src/main/java/com/hsmq/utils/ObjectByteUtils.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.utils; 2 | 3 | import org.slf4j.Logger; 4 | import org.slf4j.LoggerFactory; 5 | 6 | import java.io.ByteArrayInputStream; 7 | import java.io.ByteArrayOutputStream; 8 | import java.io.IOException; 9 | import java.io.ObjectInputStream; 10 | import java.io.ObjectOutputStream; 11 | 12 | /** 13 | * @author ：河神 14 | * @date ：Created in 2021/6/7 5:40 下午 15 | */ 16 | public class ObjectByteUtils { 17 | 18 | final static Logger log = LoggerFactory.getLogger(ObjectByteUtils.class); 19 | 20 | 21 | 22 | /** 23 | * 对象转数组 24 | * @param obj 25 | * @return 26 | */ 27 | public static byte[] toByteArray (Object obj) { 28 | byte[] bytes = null; 29 | ByteArrayOutputStream bos = new ByteArrayOutputStream(); 30 | try { 31 | ObjectOutputStream oos = new ObjectOutputStream(bos); 32 | oos.writeObject(obj); 33 | oos.flush(); 34 | bytes = bos.toByteArray (); 35 | oos.close(); 36 | bos.close(); 37 | } catch (IOException ex) { 38 | ex.printStackTrace(); 39 | return null; 40 | } 41 | return bytes; 42 | } 43 | 44 | /** 45 | * 数组转对象 46 | * @param bytes 47 | * @return 48 | */ 49 | public static Object toObject (byte[] bytes) { 50 | Object obj = null; 51 | try { 52 | ByteArrayInputStream bis = new ByteArrayInputStream (bytes); 53 | ObjectInputStream ois = new ObjectInputStream (bis); 54 | obj = ois.readObject(); 55 | ois.close(); 56 | bis.close(); 57 | } catch (Exception ex) { 58 | log.error("ex",ex); 59 | return null; 60 | } 61 | return obj; 62 | } 63 | 64 | } 65 | -------------------------------------------------------------------------------- /io/storage/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 5 | 6 | io 7 | com.heshen 8 | 1.0-SNAPSHOT 9 | 10 | 4.0.0 11 | 12 | storage 13 | 14 | 15 | 16 | io.netty 17 | netty-all 18 | 19 | 20 | 21 | com.heshen 22 | protocol 23 | 24 | 25 | 26 | com.heshen 27 | common 28 | 29 | 30 | 31 | ch.qos.logback 32 | logback-classic 33 | 34 | 35 | com.alibaba 36 | fastjson 37 | 1.2.76 38 | compile 39 | 40 | 41 | 42 | 43 | -------------------------------------------------------------------------------- /io/storage/src/main/java/com/hsmq/storage/config/StorageConfig.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.storage.config; 2 | 3 | /** 4 | * @author ：河神 5 | * @date ：Created in 2021/6/12 3:32 下午 6 | */ 7 | public class StorageConfig { 8 | 9 | public static final String MessagePath = System.getProperty("user.home")+"/data/"; 10 | 11 | public static final String Queue = "/queue/"; 12 | 13 | public static String rootPath = System.getProperty("user.home")+"/hsmq/"; 14 | 15 | public static final String MessageStorage = rootPath+"storage"; 16 | 17 | public static final String MessageQueue = rootPath+"message_queue"; 18 | 19 | public static final String ConsumerMessageQueue = rootPath+"consumer_message_queue"; 20 | 21 | 22 | 23 | 24 | } 25 | -------------------------------------------------------------------------------- /io/storage/src/main/java/com/hsmq/storage/config/TopicConfig.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.storage.config; 2 | 3 | /** 4 | * @author ：河神 5 | * @date ：Created in 2021/6/18 11:09 上午 6 | */ 7 | public class TopicConfig { 8 | 9 | private String topicName; 10 | private int messageQueueSize = 4; 11 | 12 | public static TopicConfig getDefault(String topicName){ 13 | TopicConfig topicConfig = new TopicConfig(); 14 | topicConfig.setTopicName(topicName); 15 | topicConfig.setMessageQueueSize(4); 16 | return topicConfig; 17 | } 18 | 19 | public String getTopicName() { 20 | return topicName; 21 | } 22 | 23 | public void setTopicName(String topicName) { 24 | this.topicName = topicName; 25 | } 26 | 27 | public int getMessageQueueSize() { 28 | return messageQueueSize; 29 | } 30 | 31 | public void setMessageQueueSize(int messageQueueSize) { 32 | this.messageQueueSize = messageQueueSize; 33 | } 34 | } 35 | -------------------------------------------------------------------------------- /io/storage/src/main/java/com/hsmq/storage/data/MessageDurabilityStorage.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.storage.data; 2 | 3 | import com.hsmq.storage.config.StorageConfig; 4 | import com.hsmq.storage.durability.MessageDurability; 5 | import com.hsmq.storage.file.FileOperation; 6 | 7 | import java.io.File; 8 | import java.io.FileNotFoundException; 9 | import java.io.IOException; 10 | import java.nio.ByteBuffer; 11 | import java.util.ArrayList; 12 | import java.util.List; 13 | 14 | /** 15 | * @author ：河神 16 | * @date ：Created in 2021/10/3 11:02 上午 17 | */ 18 | public class MessageDurabilityStorage { 19 | 20 | 21 | public static void saveMessageQueue(Integer queueId,String topic, List data) throws IOException { 22 | String queueFileName = StorageConfig.MessagePath + topic +StorageConfig.Queue + queueId; 23 | new File(StorageConfig.MessagePath + topic +StorageConfig.Queue ).mkdirs(); 24 | try { 25 | FileOperation.save(queueFileName, data); 26 | } catch (FileNotFoundException e) { 27 | e.printStackTrace(); 28 | } 29 | } 30 | 31 | public static List readMessageQueue(Integer queueId,String topic,long index){ 32 | String queueFileName = StorageConfig.MessagePath + topic +StorageConfig.Queue + queueId; 33 | new File(StorageConfig.MessagePath + topic +StorageConfig.Queue ).mkdirs(); 34 | try { 35 | return FileOperation.readMessageQueue(queueFileName, index); 36 | } catch (IOException e) { 37 | e.printStackTrace(); 38 | } 39 | return new ArrayList<>(); 40 | } 41 | 42 | } 43 | -------------------------------------------------------------------------------- /io/storage/src/main/java/com/hsmq/storage/data/MessageStorage.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.storage.data; 2 | 3 | import com.hsmq.data.message.PullMessage; 4 | import com.hsmq.data.message.SendMessage; 5 | import com.hsmq.storage.config.StorageConfig; 6 | import com.hsmq.storage.durability.MessageDurability; 7 | import com.hsmq.storage.file.FileOperation; 8 | import org.slf4j.Logger; 9 | import org.slf4j.LoggerFactory; 10 | 11 | import java.io.IOException; 12 | import java.util.List; 13 | 14 | /** 15 | * 持久化消息 16 | * @author ：河神 17 | * @date ：Created in 2021/6/11 3:12 下午 18 | */ 19 | public class MessageStorage { 20 | 21 | final static Logger log = LoggerFactory.getLogger(MessageStorage.class); 22 | 23 | /** 24 | * 存储消息 25 | * @param sendMessage 26 | * @return 27 | */ 28 | public static MessageDurability saveMessage(SendMessage sendMessage){ 29 | try { 30 | synchronized (MessageStorage.class){ 31 | return FileOperation.save(StorageConfig.MessagePath + "mq_1", sendMessage); 32 | } 33 | } catch (IOException | InterruptedException e) { 34 | log.error("save filer error",e); 35 | } 36 | return null; 37 | } 38 | 39 | public static List readMessages(List messageDurabilitys){ 40 | return FileOperation.readMessages(StorageConfig.MessagePath + "mq_1",messageDurabilitys); 41 | } 42 | 43 | public static List readMessageDurability(long offset,int size,String topic){ 44 | return FileOperation.readMessageDurability(StorageConfig.MessagePath + "mq_1",offset,size,topic); 45 | } 46 | 47 | 48 | } 49 | -------------------------------------------------------------------------------- /io/storage/src/main/java/com/hsmq/storage/data/QueueOffsetStorage.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.storage.data; 2 | 3 | import com.alibaba.fastjson.JSON; 4 | import com.alibaba.fastjson.JSONObject; 5 | import com.hsmq.storage.config.StorageConfig; 6 | import com.hsmq.storage.durability.MessageDurability; 7 | import com.hsmq.storage.durability.TopicConsumerData; 8 | import com.hsmq.storage.file.FileOperation; 9 | 10 | import java.io.File; 11 | import java.io.FileNotFoundException; 12 | import java.io.IOException; 13 | import java.nio.file.Files; 14 | import java.nio.file.Path; 15 | import java.nio.file.Paths; 16 | import java.util.ArrayList; 17 | import java.util.HashMap; 18 | import java.util.List; 19 | import java.util.Map; 20 | 21 | /** 22 | * @author ：河神 23 | * @date ：Created in 2021/10/8 2:24 下午 24 | */ 25 | public class QueueOffsetStorage { 26 | 27 | public static void saveConsumer(String topic,String consumer, Map offSetMap) { 28 | 29 | String consumerDir = StorageConfig.MessagePath + topic; 30 | String consumerFileName = consumerDir +"/ConsumerData"; 31 | boolean mkdirs = new File(consumerDir).mkdirs(); 32 | if (mkdirs){ 33 | throw new RuntimeException("创建文件夹失败"); 34 | } 35 | TopicConsumerData topicConsumerData = readConsumer(topic); 36 | if (topicConsumerData==null){ 37 | topicConsumerData = new TopicConsumerData(); 38 | topicConsumerData.setOffSetMap(new HashMap<>()); 39 | } 40 | Map> offSetAllMap = topicConsumerData.getOffSetMap(); 41 | Map stringLongMap = offSetAllMap.get(consumer); 42 | if (stringLongMap==null){ 43 | stringLongMap = new HashMap<>(); 44 | } 45 | Map finalStringLongMap = stringLongMap; 46 | offSetMap.forEach((k, v)->{ 47 | finalStringLongMap.put(k.toString(),v); 48 | }); 49 | offSetAllMap.put(consumer,finalStringLongMap); 50 | 51 | FileOperation.saveString(consumerFileName, JSON.toJSONString(topicConsumerData)); 52 | } 53 | 54 | public static Map getOffSetMap(String topic,String consumer){ 55 | 56 | TopicConsumerData topicConsumerData = readConsumer(topic); 57 | if (topicConsumerData==null){ 58 | return null; 59 | } 60 | Map> offSetMap = topicConsumerData.getOffSetMap(); 61 | if (offSetMap==null){ 62 | return null; 63 | } 64 | Map stringLongMap = offSetMap.get(consumer); 65 | if (stringLongMap==null){ 66 | return null; 67 | } 68 | Map data = new HashMap<>(); 69 | stringLongMap.forEach((k,v)->{ 70 | data.put(Integer.valueOf(k),v); 71 | }); 72 | return data; 73 | } 74 | 75 | public static TopicConsumerData readConsumer(String topic){ 76 | String consumerDir = StorageConfig.MessagePath + topic; 77 | String consumerFileName = consumerDir +"/ConsumerData"; 78 | boolean mkdirs = new File(consumerDir).mkdirs(); 79 | if (mkdirs){ 80 | throw new RuntimeException("创建文件夹失败"); 81 | } 82 | if (!Files.exists(Paths.get(consumerFileName))){ 83 | return null; 84 | } 85 | 86 | String s = FileOperation.readString(consumerFileName); 87 | if (s.length()==0){ 88 | return null; 89 | } 90 | 91 | return JSON.parseObject(s,TopicConsumerData.class); 92 | } 93 | 94 | 95 | 96 | } 97 | -------------------------------------------------------------------------------- /io/storage/src/main/java/com/hsmq/storage/durability/MessageDurability.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.storage.durability; 2 | 3 | import java.io.Serializable; 4 | import java.util.StringJoiner; 5 | 6 | /** 7 | * @author ：河神 8 | * @date ：Created in 2021/6/11 4:14 下午 9 | */ 10 | public class MessageDurability implements Serializable { 11 | 12 | 13 | private static final long serialVersionUID = -7827672105039234315L; 14 | private long offset; 15 | 16 | private int length; 17 | 18 | private int tagHashcode; 19 | 20 | private long index; 21 | 22 | public MessageDurability() { 23 | } 24 | 25 | public MessageDurability(long offset, int length, int tagHashcode) { 26 | this.offset = offset; 27 | this.length = length; 28 | this.tagHashcode = tagHashcode; 29 | } 30 | 31 | public long getOffset() { 32 | return offset; 33 | } 34 | 35 | public void setOffset(long offset) { 36 | this.offset = offset; 37 | } 38 | 39 | public int getLength() { 40 | return length; 41 | } 42 | 43 | public void setLength(int length) { 44 | this.length = length; 45 | } 46 | 47 | public int getTagHashcode() { 48 | return tagHashcode; 49 | } 50 | 51 | public void setTagHashcode(int tagHashcode) { 52 | this.tagHashcode = tagHashcode; 53 | } 54 | 55 | public long getIndex() { 56 | return index; 57 | } 58 | 59 | public void setIndex(long index) { 60 | this.index = index; 61 | } 62 | 63 | @Override 64 | public String toString() { 65 | return new StringJoiner(", ", MessageDurability.class.getSimpleName() + "[", "]") 66 | .add("offset=" + offset) 67 | .add("length=" + length) 68 | .add("tagHashcode=" + tagHashcode) 69 | .add("index=" + index) 70 | .toString(); 71 | } 72 | } 73 | -------------------------------------------------------------------------------- /io/storage/src/main/java/com/hsmq/storage/durability/TopicConsumerData.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.storage.durability; 2 | 3 | import java.io.Serializable; 4 | import java.util.Map; 5 | 6 | /** 7 | * @author ：河神 8 | * @date ：Created in 2021/10/8 11:37 上午 9 | */ 10 | public class TopicConsumerData { 11 | 12 | private Map> offSetMap; 13 | 14 | public Map> getOffSetMap() { 15 | return offSetMap; 16 | } 17 | 18 | public void setOffSetMap(Map> offSetMap) { 19 | this.offSetMap = offSetMap; 20 | } 21 | 22 | 23 | } 24 | -------------------------------------------------------------------------------- /io/storage/src/main/java/com/hsmq/storage/file/FileOperation.java: -------------------------------------------------------------------------------- 1 | package com.hsmq.storage.file; 2 | 3 | import com.hsmq.common.exception.FileException; 4 | import com.hsmq.data.message.PullMessage; 5 | import com.hsmq.data.message.SendMessage; 6 | import com.hsmq.storage.durability.MessageDurability; 7 | import com.hsmq.utils.ObjectByteUtils; 8 | import org.slf4j.Logger; 9 | import org.slf4j.LoggerFactory; 10 | 11 | import java.io.IOException; 12 | import java.io.RandomAccessFile; 13 | import java.nio.ByteBuffer; 14 | import java.nio.channels.FileChannel; 15 | import java.nio.file.Files; 16 | import java.nio.file.Paths; 17 | import java.util.ArrayList; 18 | import java.util.List; 19 | 20 | /** 21 | * @author ：河神 22 | * @date ：Created in 2021/6/12 1:46 下午 23 | */ 24 | public class FileOperation { 25 | 26 | final static Logger log = LoggerFactory.getLogger(FileOperation.class); 27 | 28 | private static final int MessageDurabilityLength = 24; 29 | 30 | public static void save(String fileName,List data) throws IOException { 31 | if (data==null||data.size()==0){ 32 | return; 33 | } 34 | 35 | RandomAccessFile rws = new RandomAccessFile(fileName, "rw"); 36 | FileChannel fileChannel = rws.getChannel(); 37 | ByteBuffer byteBuffer = ByteBuffer.allocate(data.size()* MessageDurabilityLength); 38 | 39 | long index = fileChannel.size()/MessageDurabilityLength; 40 | for (MessageDurability messageDurability : data) { 41 | byte[] bytes = ObjectByteUtils.toByteArray(messageDurability); 42 | if (bytes==null){ 43 | throw new FileException("文件转化异常：object not can cast bytes"); 44 | } 45 | messageDurability.setIndex(index++); 46 | 47 | byteBuffer.putLong(messageDurability.getOffset()); 48 | byteBuffer.putInt(messageDurability.getLength()); 49 | byteBuffer.putInt(messageDurability.getTagHashcode()); 50 | byteBuffer.putLong(messageDurability.getIndex()); 51 | } 52 | 53 | byteBuffer.flip(); 54 | fileChannel.position(fileChannel.size()); 55 | fileChannel.write(byteBuffer); 56 | 57 | fileChannel.force(true); 58 | fileChannel.close(); 59 | } 60 | 61 | 62 | public static List readMessageQueue(String fileName,long index) throws IOException { 63 | List data = new ArrayList<>(); 64 | 65 | RandomAccessFile rws = new RandomAccessFile(fileName, "rw"); 66 | long offset = index * MessageDurabilityLength; 67 | 68 | FileChannel fileChannel = rws.getChannel(); 69 | while (true){ 70 | ByteBuffer byteBuffer = ByteBuffer.allocate(MessageDurabilityLength); 71 | int read = fileChannel.read(byteBuffer, offset); 72 | if (read<=0){ 73 | break; 74 | } 75 | byteBuffer.flip(); 76 | MessageDurability messageDurability = new MessageDurability(); 77 | messageDurability.setOffset(byteBuffer.getLong()); 78 | messageDurability.setLength(byteBuffer.getInt()); 79 | messageDurability.setTagHashcode(byteBuffer.getInt()); 80 | messageDurability.setIndex(byteBuffer.getLong()); 81 | 82 | offset+=MessageDurabilityLength; 83 | 84 | data.add(messageDurability); 85 | } 86 | fileChannel.close(); 87 | return data; 88 | } 89 | 90 | 91 | 92 | public static synchronized MessageDurability save(String fileName,Object object) throws IOException, InterruptedException{ 93 | 94 | RandomAccessFile rws = new RandomAccessFile(fileName, "rw"); 95 | FileChannel fileChannel = rws.getChannel(); 96 | 97 | byte[] bytes = ObjectByteUtils.toByteArray(object); 98 | if (bytes==null){ 99 | throw new FileException("文件转化异常：object not can cast bytes"); 100 | } 101 | 102 | ByteBuffer byteBuffer = ByteBuffer.allocate(bytes.length+4); 103 | byteBuffer.putInt(bytes.length); 104 | byteBuffer.put(bytes); 105 | byteBuffer.flip(); 106 | 107 | MessageDurability messageDurability = new MessageDurability(); 108 | messageDurability.setLength(byteBuffer.limit()); 109 | messageDurability.setOffset(fileChannel.size()); 110 | 111 | fileChannel.position(fileChannel.size()); 112 | fileChannel.write(byteBuffer); 113 | fileChannel.force(true); 114 | fileChannel.close(); 115 | 116 | return messageDurability; 117 | } 118 | 119 | public static byte[] read(FileChannel fileChannel,long offset,int length) throws IOException{ 120 | ByteBuffer byteBuffer = ByteBuffer.allocate(length); 121 | fileChannel.read(byteBuffer,offset+4); 122 | byteBuffer.flip(); 123 | byte[] bytes = new byte[length-4]; 124 | byteBuffer.get(bytes); 125 | return bytes; 126 | } 127 | 128 | public static List readMessages(String fileName, List messageDurability){ 129 | List data = new ArrayList<>(); 130 | try { 131 | FileChannel fileChannel = 132 | new RandomAccessFile(fileName, "r").getChannel(); 133 | for (MessageDurability durability : messageDurability) { 134 | byte[] read = read(fileChannel, durability.getOffset(), durability.getLength()); 135 | Object object = ObjectByteUtils.toObject(read); 136 | if (object==null){ 137 | continue; 138 | } 139 | if (object instanceof SendMessage){ 140 | SendMessage sendMessage = (SendMessage) object; 141 | PullMessage pullMessage = new PullMessage(); 142 | pullMessage.setMsgId(sendMessage.getMsgId()); 143 | pullMessage.setBody(sendMessage.getBody()); 144 | pullMessage.setKey(sendMessage.getKey()); 145 | pullMessage.setTopic(sendMessage.getTopic()); 146 | pullMessage.setTag(sendMessage.getTag()); 147 | pullMessage.setIndex(durability.getIndex()); 148 | data.add(pullMessage); 149 | } 150 | } 151 | fileChannel.close(); 152 | return data; 153 | } catch (Exception e) { 154 | log.error("read filer error",e); 155 | } 156 | return data; 157 | } 158 | 159 | 160 | 161 | public static List readMessageDurability(String fileName, long offset,int size,String topic){ 162 | List data = new ArrayList<>(); 163 | try { 164 | FileChannel fileChannel = 165 | new RandomAccessFile(fileName, "r").getChannel(); 166 | for (int i=0;i=map.limit()){ 34 | break; 35 | } 36 | int length = map.getInt(); 37 | byte[] bytes = new byte[length]; 38 | map.get(bytes); 39 | System.out.println(length+" : "+ObjectByteUtils.toObject(bytes).toString()); 40 | } 41 | map.clear(); 42 | fileChannel.close(); 43 | 44 | 45 | } 46 | 47 | public static void saveAndRead() throws IOException, InterruptedException{ 48 | MessageDurability messageDurability1 = FileOperation.save(StorageConfig.MessagePath+"data-0001", "长颈鹿吃树叶"); 49 | 50 | FileChannel fileChannel = 51 | new RandomAccessFile(StorageConfig.MessagePath+"data-0001", "r").getChannel(); 52 | 53 | MappedByteBuffer map = fileChannel.map(FileChannel.MapMode.READ_ONLY, messageDurability1.getOffset(), 54 | messageDurability1.getLength()); 55 | 56 | int anInt = map.getInt(); 57 | System.out.println(anInt); 58 | byte[] bytes = new byte[messageDurability1.getLength()-4]; 59 | map.get(bytes); 60 | 61 | System.out.println(ObjectByteUtils.toObject(bytes).toString()); 62 | 63 | fileChannel.close(); 64 | } 65 | 66 | } 67 | -------------------------------------------------------------------------------- /io/storage/src/test/java/com/hs/filer/FIleTest.java: -------------------------------------------------------------------------------- 1 | package com.hs.filer; 2 | 3 | import com.hsmq.storage.config.StorageConfig; 4 | import com.hsmq.storage.durability.MessageDurability; 5 | import com.hsmq.storage.file.FileOperation; 6 | import com.hsmq.utils.ObjectByteUtils; 7 | import io.netty.util.internal.logging.InternalLogger; 8 | import io.netty.util.internal.logging.InternalLoggerFactory; 9 | 10 | import java.io.IOException; 11 | import java.io.RandomAccessFile; 12 | import java.nio.channels.FileChannel; 13 | import java.util.ArrayList; 14 | import java.util.List; 15 | 16 | /** 17 | * 零拷贝 18 | * @author ：河神 19 | * @date ：Created in 2021/4/4 7:13 下午 20 | */ 21 | public class FIleTest { 22 | 23 | private static InternalLogger logger = InternalLoggerFactory.getInstance(FIleTest.class); 24 | 25 | 26 | public static void main(String[] args) throws IOException, InterruptedException { 27 | 28 | MessageDurability messageDurability1 = FileOperation.save(StorageConfig.MessagePath+"data-0001", "长颈鹿吃树叶"); 29 | MessageDurability messageDurability2 = FileOperation.save(StorageConfig.MessagePath+"data-0001", "长颈鹿吃树叶"); 30 | MessageDurability messageDurability3 = FileOperation.save(StorageConfig.MessagePath+"data-0001", "长颈鹿吃树叶"); 31 | 32 | FileChannel fileChannel = 33 | new RandomAccessFile(StorageConfig.MessagePath+"data-0001", "r").getChannel(); 34 | 35 | List data = new ArrayList<>(); 36 | 37 | data.add(messageDurability1); 38 | data.add(messageDurability2); 39 | data.add(messageDurability3); 40 | 41 | for (MessageDurability datum : data) { 42 | byte[] read1 = FileOperation.read(fileChannel, datum.getOffset(),datum.getLength()); 43 | logger.info(ObjectByteUtils.toObject(read1).toString()); 44 | } 45 | fileChannel.close(); 46 | 47 | 48 | } 49 | 50 | } 51 | -------------------------------------------------------------------------------- /io/storage/src/test/java/com/hs/filer/FIleTest2.java: -------------------------------------------------------------------------------- 1 | package com.hs.filer; 2 | 3 | import com.hsmq.common.exception.FileException; 4 | import com.hsmq.storage.config.StorageConfig; 5 | import com.hsmq.storage.durability.MessageDurability; 6 | import com.hsmq.storage.file.FileOperation; 7 | import com.hsmq.utils.ObjectByteUtils; 8 | import io.netty.util.internal.logging.InternalLogger; 9 | import io.netty.util.internal.logging.InternalLoggerFactory; 10 | 11 | import java.io.IOException; 12 | import java.io.RandomAccessFile; 13 | import java.nio.ByteBuffer; 14 | import java.nio.channels.FileChannel; 15 | import java.util.ArrayList; 16 | import java.util.List; 17 | 18 | /** 19 | * 零拷贝 20 | * @author ：河神 21 | * @date ：Created in 2021/4/4 7:13 下午 22 | */ 23 | public class FIleTest2 { 24 | 25 | private static InternalLogger logger = InternalLoggerFactory.getInstance(FIleTest2.class); 26 | 27 | 28 | public static void main(String[] args) throws IOException, InterruptedException { 29 | 30 | 31 | MessageDurability messageDurability3 = 32 | FileOperation.save(StorageConfig.MessagePath+"data-0001", "长颈鹿吃树叶"); 33 | 34 | FileChannel fileChannel = 35 | new RandomAccessFile(StorageConfig.MessagePath+"data-0001", "r").getChannel(); 36 | 37 | List data = new ArrayList<>(); 38 | 39 | data.add(messageDurability3); 40 | 41 | for (MessageDurability datum : data) { 42 | byte[] read1 = FileOperation.read(fileChannel, datum.getOffset(),datum.getLength()); 43 | logger.info(ObjectByteUtils.toObject(read1).toString()); 44 | } 45 | fileChannel.close(); 46 | } 47 | 48 | public static void init() throws IOException, InterruptedException{ 49 | 50 | String fileName = StorageConfig.MessagePath+"data-0001"; 51 | String object = "InterruptedException"; 52 | 53 | RandomAccessFile rws = new RandomAccessFile(fileName, "rw"); 54 | FileChannel fileChannel = rws.getChannel(); 55 | 56 | byte[] bytes = ObjectByteUtils.toByteArray(object); 57 | if (bytes==null){ 58 | throw new FileException("文件转化异常：object not can cast bytes"); 59 | } 60 | 61 | ByteBuffer byteBuffer = ByteBuffer.allocate(bytes.length+4); 62 | byteBuffer.putInt(bytes.length); 63 | byteBuffer.put(bytes); 64 | byteBuffer.flip(); 65 | 66 | MessageDurability messageDurability = new MessageDurability(); 67 | messageDurability.setLength(byteBuffer.limit()); 68 | messageDurability.setOffset(fileChannel.size()); 69 | 70 | fileChannel.position(fileChannel.size()); 71 | fileChannel.write(byteBuffer); 72 | fileChannel.force(true); 73 | fileChannel.close(); 74 | } 75 | 76 | } 77 | -------------------------------------------------------------------------------- /mq-client/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 5 | 6 | hsmq 7 | com.heshen 8 | 1.0-SNAPSHOT 9 | 10 | 4.0.0 11 | 12 | mq-client 13 | 14 | 15 | 8 16 | 8 17 | 18 | 19 | 20 | 21 | 22 | com.heshen 23 | protocol 24 | 25 | 26 | 27 | com.heshen 28 | common 29 | 30 | 31 | 32 | 33 | org.apache.commons 34 | commons-lang3 35 | 3.12.0 36 | 37 | 38 | 39 | 40 | org.apache.commons 41 | commons-collections4 42 | 4.4 43 | 44 | 45 | 46 | ch.qos.logback 47 | logback-classic 48 | 49 | 50 | 51 | com.alibaba 52 | fastjson 53 | 1.2.76 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/ClientStartup.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient; 2 | 3 | import com.hsmq.data.message.SendMessage; 4 | import com.hsms.mqclient.consumer.config.RegisteredConsumer; 5 | import com.hsms.mqclient.consumer.consumer.ConsumerHandlerManger; 6 | import com.hsms.mqclient.producer.dto.SendMessageResult; 7 | import com.hsms.mqclient.producer.send.MessageClient; 8 | import com.hsms.mqclient.reactor.ClientReactor; 9 | import org.slf4j.Logger; 10 | import org.slf4j.LoggerFactory; 11 | 12 | import java.io.IOException; 13 | 14 | /** 15 | * @author ：河神 16 | * @date ：Created in 2021/6/6 6:25 下午 17 | */ 18 | public class ClientStartup { 19 | 20 | final static Logger log = LoggerFactory.getLogger(ClientStartup.class); 21 | 22 | 23 | public static void main(String[] args) throws InterruptedException, IOException { 24 | 25 | String start = 26 | "\n" + 27 | " _ _ _____ _____ _ _ _ \n" + 28 | "| | | |/ ___| / __ \\ (_) | | \n" + 29 | "| |_| |\\ `--. ______| / \\/ |_ ___ _ __ | |_ \n" + 30 | "| _ | `--. \\______| | | | |/ _ \\ '_ \\| __|\n" + 31 | "| | | |/\\__/ / | \\__/\\ | | __/ | | | |_ \n" + 32 | "\\_| |_/\\____/ \\____/_|_|\\___|_| |_|\\__|\n"; 33 | log.info(start); 34 | 35 | //记录启动时间 36 | RegisteredConsumer.setStopWatch(); 37 | 38 | //注册Netty 39 | ClientReactor clientReactor = new ClientReactor("127.0.0.1", 9001); 40 | //启动Netty 41 | clientReactor.start(); 42 | 43 | //初始化消费者 44 | args = new String[]{"TopicB"}; 45 | ConsumerHandlerManger.initConsumer("BBAConsumer",args,clientReactor.getChannelFuture()); 46 | ConsumerHandlerManger.initConsumer("BBCConsumer",args,clientReactor.getChannelFuture()); 47 | 48 | //初始化生产者 49 | MessageClient.setChannelFuture(clientReactor.getChannelFuture()); 50 | 51 | 52 | SendMessage sendMessage = new SendMessage(); 53 | sendMessage.setTopic("TopicB"); 54 | sendMessage.setTag("tagB"); 55 | sendMessage.setBody("消息---1"); 56 | 57 | for (int i=0;i<2;i++){ 58 | SendMessageResult b = MessageClient.sendMsg(sendMessage); 59 | 60 | if (!Integer.valueOf(200).equals(b.getMessageResult())){ 61 | System.out.println("消息发送失败 : "+b); 62 | break; 63 | } 64 | System.out.println("消息发送 : "+b); 65 | Thread.sleep(100L); 66 | } 67 | 68 | //发送消息 69 | // for (int i=5000;;i++){ 70 | // SendMessage sendMessage = new SendMessage(); 71 | // sendMessage.setTopic("TopicB"); 72 | // sendMessage.setTag("tagB"); 73 | // sendMessage.setBody("消息---"+i); 74 | // MessageClient.sendMsg(sendMessage); 75 | // Thread.sleep(100L); 76 | // } 77 | 78 | } 79 | 80 | } 81 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/ConfigTest.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient; 2 | 3 | import com.alibaba.fastjson.JSON; 4 | import com.hsms.mqclient.consumer.config.ClientConfig; 5 | 6 | import java.io.IOException; 7 | import java.nio.file.Files; 8 | import java.nio.file.Paths; 9 | 10 | /** 11 | * @author ：河神 12 | * @date ：Created in 2021/10/9 3:21 下午 13 | */ 14 | public class ConfigTest { 15 | public static void main(String[] args) throws IOException { 16 | 17 | String url4 = ConfigTest.class.getClassLoader().getResource("").getFile(); 18 | ClientConfig o = 19 | JSON.parseObject(Files.readAllBytes(Paths.get(url4 + "/hsmq.json")), ClientConfig.class); 20 | 21 | 22 | System.out.println(JSON.toJSONString(o)); 23 | } 24 | 25 | } 26 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/consumer/config/ClientConfig.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient.consumer.config; 2 | 3 | import java.util.HashMap; 4 | import java.util.List; 5 | 6 | /** 7 | * @author ：河神 8 | * @date ：Created in 2021/10/9 3:45 下午 9 | */ 10 | public class ClientConfig { 11 | 12 | public ServerConfig server; 13 | public HashMap> consumer; 14 | 15 | public static class ServerConfig { 16 | public String host = "127.0.0.1"; 17 | public Integer post = 9001; 18 | 19 | public String getHost() { 20 | return host; 21 | } 22 | 23 | public void setHost(String host) { 24 | this.host = host; 25 | } 26 | 27 | public Integer getPost() { 28 | return post; 29 | } 30 | 31 | public void setPost(Integer post) { 32 | this.post = post; 33 | } 34 | } 35 | 36 | public static class ConsumerGroup{ 37 | public String topic; 38 | 39 | public String getTopic() { 40 | return topic; 41 | } 42 | 43 | public void setTopic(String topic) { 44 | this.topic = topic; 45 | } 46 | } 47 | 48 | public ServerConfig getServer() { 49 | return server; 50 | } 51 | 52 | public void setServer(ServerConfig server) { 53 | this.server = server; 54 | } 55 | 56 | public HashMap> getConsumer() { 57 | return consumer; 58 | } 59 | 60 | public void setConsumer(HashMap> consumer) { 61 | this.consumer = consumer; 62 | } 63 | } 64 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/consumer/config/ExecutorService.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient.consumer.config; 2 | 3 | import java.util.concurrent.ArrayBlockingQueue; 4 | import java.util.concurrent.ThreadFactory; 5 | import java.util.concurrent.ThreadPoolExecutor; 6 | import java.util.concurrent.TimeUnit; 7 | 8 | /** 9 | * @author ：河神 10 | * @date ：Created in 2021/7/11 2:51 下午 11 | */ 12 | public class ExecutorService { 13 | 14 | 15 | public static ThreadPoolExecutor threadPoolExecutor; 16 | 17 | static { 18 | threadPoolExecutor = new ThreadPoolExecutor(20, 100, 19 | 30, TimeUnit.SECONDS, new ArrayBlockingQueue<>(5), new ThreadFactory() { 20 | @Override 21 | public Thread newThread(Runnable r) { 22 | return new Thread(r, "ConsumerExecutor"); 23 | } 24 | }); 25 | } 26 | 27 | public static ThreadPoolExecutor getExecutor(){ 28 | return threadPoolExecutor; 29 | } 30 | 31 | } 32 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/consumer/config/RegisteredConsumer.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient.consumer.config; 2 | 3 | import com.hsmq.data.Head; 4 | import com.hsmq.data.HsReq; 5 | import com.hsmq.data.message.MessageQueueData; 6 | import com.hsmq.data.message.TopicData; 7 | import com.hsmq.enums.MessageEnum; 8 | import com.hsmq.enums.OperationEnum; 9 | import com.hsmq.protocol.HsEecodeData; 10 | import com.hsms.mqclient.consumer.executos.CommitOffsetTask; 11 | import com.hsms.mqclient.consumer.executos.ExecutorMessageTask; 12 | import com.hsms.mqclient.consumer.executos.PullMessageTask; 13 | import com.hsms.mqclient.consumer.message.ConsumerMessageQueue; 14 | import io.netty.channel.ChannelFuture; 15 | import org.apache.commons.lang3.time.StopWatch; 16 | import org.slf4j.Logger; 17 | import org.slf4j.LoggerFactory; 18 | 19 | import java.util.HashMap; 20 | import java.util.Map; 21 | import java.util.concurrent.ThreadPoolExecutor; 22 | import java.util.concurrent.TimeUnit; 23 | import java.util.concurrent.atomic.AtomicInteger; 24 | 25 | /** 26 | * @author ：河神 27 | * @date ：Created in 2021/7/11 2:55 下午 28 | */ 29 | public class RegisteredConsumer { 30 | 31 | final static Logger log = LoggerFactory.getLogger(RegisteredConsumer.class); 32 | 33 | private static AtomicInteger initFlag = new AtomicInteger(0); 34 | private static ChannelFuture channelFuture; 35 | private static StopWatch stopWatch; 36 | private static Map consumerMessageQueueMap = new HashMap<>(); 37 | 38 | public static void setStopWatch() { 39 | RegisteredConsumer.stopWatch = new StopWatch(); 40 | RegisteredConsumer.stopWatch.start(); 41 | } 42 | 43 | public static ConsumerMessageQueue getConsumerMessageQueue(String topic,String consumerGroup){ 44 | return consumerMessageQueueMap.get(consumerKey(topic,consumerGroup)); 45 | } 46 | 47 | public static String consumerKey(String topic,String consumerGroup){ 48 | return topic+":"+consumerGroup; 49 | } 50 | 51 | public static void setChannelFuture(ChannelFuture channelFuture) { 52 | RegisteredConsumer.channelFuture = channelFuture; 53 | } 54 | 55 | public static void init(String consumerGroup, String[] topics){ 56 | //注册消费者 57 | for (String topic : topics) { 58 | RegisteredConsumer.registeredConsumer(topic,consumerGroup); 59 | } 60 | //根据服务端数据初始化消费者 61 | RegisteredConsumer.initConsumerQueue(consumerGroup); 62 | } 63 | 64 | public static void registeredConsumer(String topic,String consumerGroup){ 65 | log.info("registeredConsumer topic:[{}]",topic); 66 | ThreadPoolExecutor executor = ExecutorService.getExecutor(); 67 | //创建消费者 68 | ConsumerMessageQueue consumerMessageQueue = new ConsumerMessageQueue(topic,consumerGroup); 69 | //注册到管理器中 70 | consumerMessageQueueMap.put(consumerKey(topic,consumerGroup),consumerMessageQueue); 71 | //注册拉取消息任务 72 | executor.execute(new PullMessageTask(channelFuture , consumerGroup, consumerMessageQueue)); 73 | //注册执行器任务 74 | executor.execute(new ExecutorMessageTask(channelFuture ,consumerMessageQueue)); 75 | 76 | //定时任务 77 | channelFuture.channel().eventLoop().scheduleWithFixedDelay(()->{ 78 | //定时任务 79 | new Thread(new CommitOffsetTask(channelFuture,consumerMessageQueue)).start(); 80 | },1, 3L, TimeUnit.SECONDS); 81 | 82 | initFlag.incrementAndGet(); 83 | log.info("registeredConsumer end consumer size :{} ",initFlag.get()); 84 | } 85 | 86 | public static boolean isInit() { 87 | return initFlag.get()<=0; 88 | } 89 | 90 | public static void initConsumerQueue(String consumerGroup){ 91 | 92 | consumerMessageQueueMap.forEach((consumerKey,queue)->{ 93 | HsEecodeData hsEecodeData = new HsEecodeData(); 94 | hsEecodeData.setHead(Head.toHead(MessageEnum.Req)); 95 | HsReq hsReq = new HsReq<>(); 96 | 97 | TopicData topicData = new TopicData(); 98 | topicData.setTopic(queue.getTopic()); 99 | topicData.setConsumerGroup(queue.getConsumerGroup()); 100 | topicData.setConsumerKey(consumerKey); 101 | 102 | hsReq.setData(topicData); 103 | hsReq.setOperation(OperationEnum.TopicData.getOperation()); 104 | hsEecodeData.setData(hsReq); 105 | 106 | try { 107 | channelFuture.channel().writeAndFlush(hsEecodeData).sync(); 108 | log.info("consumer init by req server - topic:{}",queue.getTopic()); 109 | } catch (InterruptedException e) { 110 | e.printStackTrace(); 111 | } 112 | 113 | }); 114 | } 115 | 116 | public static void initConsumerQueueHandle(MessageQueueData messageQueueData){ 117 | ConsumerMessageQueue consumerMessageQueue = consumerMessageQueueMap.get(messageQueueData.getConsumerKey()); 118 | if (consumerMessageQueue==null){ 119 | return; 120 | } 121 | //初始化消费者 122 | consumerMessageQueue.initQueue(messageQueueData); 123 | //设置初始化完成 124 | initFlag.decrementAndGet(); 125 | 126 | if (isInit()){ 127 | // stopWatch.stop(); 128 | log.info("Consumer Start End"); 129 | } 130 | 131 | } 132 | 133 | } 134 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/consumer/consumer/AbstractConsumer.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient.consumer.consumer; 2 | 3 | import com.hsmq.data.message.PullMessage; 4 | 5 | /** 6 | * @author ：河神 7 | * @date ：Created in 2021/10/1 2:44 下午 8 | */ 9 | public abstract class AbstractConsumer { 10 | 11 | protected abstract boolean consumeMessage(PullMessage pullMessage); 12 | 13 | public boolean consumer(PullMessage pullMessage){ 14 | //前置处理 15 | boolean b = consumeMessage(pullMessage); 16 | //后置处理 17 | return b; 18 | } 19 | } 20 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/consumer/consumer/ConsumerHandlerManger.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient.consumer.consumer; 2 | 3 | import com.hsmq.data.message.PullMessage; 4 | import com.hsms.mqclient.consumer.config.RegisteredConsumer; 5 | import com.hsms.mqclient.consumer.consumer.sub.TopicAConsumer; 6 | import com.hsms.mqclient.consumer.consumer.sub.TopicBConsumer; 7 | import io.netty.channel.ChannelFuture; 8 | import org.slf4j.Logger; 9 | import org.slf4j.LoggerFactory; 10 | 11 | import java.util.HashMap; 12 | import java.util.Map; 13 | 14 | /** 15 | * @author ：河神 16 | * @date ：Created in 2021/10/1 2:45 下午 17 | */ 18 | public class ConsumerHandlerManger { 19 | 20 | private static final Map> allConsumerMap = new HashMap<>(); 21 | 22 | final static Logger log = LoggerFactory.getLogger(TopicBConsumer.class); 23 | 24 | 25 | public static void initConsumer(String consumerGroup, String[] topics, ChannelFuture channelFuture){ 26 | log.info("initConsumer start"); 27 | 28 | //注册消费组对应的消费者 29 | Map consumerMap = new HashMap<>(); 30 | consumerMap.put("TopicA",new TopicAConsumer()); 31 | consumerMap.put("TopicB",new TopicBConsumer()); 32 | allConsumerMap.put(consumerGroup,consumerMap); 33 | 34 | //初始化消费者 35 | RegisteredConsumer.setChannelFuture(channelFuture); 36 | RegisteredConsumer.init(consumerGroup,topics); 37 | 38 | log.info("initConsumer end"); 39 | } 40 | 41 | public static boolean consumer(String consumerGroup,PullMessage pullMessage){ 42 | 43 | Map consumerMap = allConsumerMap.get(consumerGroup); 44 | if (consumerMap==null){ 45 | log.error("ConsumerGroup:{} Not Exists Consumer",pullMessage.getTopic()); 46 | return false; 47 | } 48 | AbstractConsumer abstractConsumer = consumerMap.get(pullMessage.getTopic()); 49 | if (abstractConsumer==null){ 50 | log.error("Topic:{} Not Exists Consumer",pullMessage.getTopic()); 51 | return false; 52 | } 53 | return abstractConsumer.consumer(pullMessage); 54 | } 55 | 56 | } 57 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/consumer/consumer/sub/TopicAConsumer.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient.consumer.consumer.sub; 2 | 3 | import com.hsmq.data.message.PullMessage; 4 | import com.hsms.mqclient.consumer.consumer.AbstractConsumer; 5 | import org.slf4j.Logger; 6 | import org.slf4j.LoggerFactory; 7 | 8 | /** 9 | * @author ：河神 10 | * @date ：Created in 2021/10/1 2:47 下午 11 | */ 12 | public class TopicAConsumer extends AbstractConsumer { 13 | 14 | final static Logger log = LoggerFactory.getLogger(TopicAConsumer.class); 15 | 16 | @Override 17 | protected boolean consumeMessage(PullMessage pullMessage) { 18 | 19 | log.info("TopicAConsumer 消费:{}",pullMessage); 20 | return true; 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/consumer/consumer/sub/TopicBConsumer.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient.consumer.consumer.sub; 2 | 3 | import com.hsmq.data.message.PullMessage; 4 | import com.hsms.mqclient.consumer.consumer.AbstractConsumer; 5 | import org.slf4j.Logger; 6 | import org.slf4j.LoggerFactory; 7 | 8 | /** 9 | * @author ：河神 10 | * @date ：Created in 2021/10/1 2:47 下午 11 | */ 12 | public class TopicBConsumer extends AbstractConsumer { 13 | 14 | final static Logger log = LoggerFactory.getLogger(TopicBConsumer.class); 15 | 16 | @Override 17 | protected boolean consumeMessage(PullMessage pullMessage) { 18 | log.info("TopicBConsumer 消费:{}",pullMessage); 19 | return true; 20 | } 21 | } 22 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/consumer/executos/CommitOffsetTask.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient.consumer.executos; 2 | 3 | import com.hsmq.data.Head; 4 | import com.hsmq.data.HsReq; 5 | import com.hsmq.data.message.SyncOffsetMessage; 6 | import com.hsmq.enums.MessageEnum; 7 | import com.hsmq.enums.OperationEnum; 8 | import com.hsmq.protocol.HsEecodeData; 9 | import com.hsms.mqclient.ClientStartup; 10 | import com.hsms.mqclient.consumer.config.RegisteredConsumer; 11 | import com.hsms.mqclient.consumer.message.ConsumerMessageQueue; 12 | import io.netty.channel.ChannelFuture; 13 | import org.slf4j.Logger; 14 | import org.slf4j.LoggerFactory; 15 | 16 | import java.util.Map; 17 | 18 | /** 19 | * 定时提交偏移量任务 20 | * @author ：河神 21 | * @date ：Created in 2021/9/30 8:42 下午 22 | */ 23 | public class CommitOffsetTask implements Runnable{ 24 | 25 | final static Logger log = LoggerFactory.getLogger(ClientStartup.class); 26 | 27 | 28 | private final ConsumerMessageQueue consumerMessageQueue; 29 | private final ChannelFuture channelFuture; 30 | 31 | 32 | public CommitOffsetTask(ChannelFuture channelFuture,ConsumerMessageQueue consumerMessageQueue) { 33 | this.channelFuture = channelFuture; 34 | this.consumerMessageQueue = consumerMessageQueue; 35 | } 36 | 37 | @Override 38 | public void run() { 39 | Map offSetMap = consumerMessageQueue.getOffSetMap(); 40 | log.info("consumerGroupL:{} ,offSetMap:{}",consumerMessageQueue.getConsumerGroup(),offSetMap); 41 | while (true){ 42 | //尚未初始化完成 43 | if (!RegisteredConsumer.isInit()){ 44 | try { 45 | Thread.sleep(100L); 46 | } catch (InterruptedException e) { 47 | e.printStackTrace(); 48 | } 49 | continue; 50 | } 51 | 52 | try { 53 | syncIndex(); 54 | Thread.sleep(1000L); 55 | } catch (InterruptedException e) { 56 | e.printStackTrace(); 57 | } 58 | } 59 | 60 | } 61 | 62 | public void syncIndex() throws InterruptedException { 63 | HsEecodeData hsEecodeData = new HsEecodeData(); 64 | hsEecodeData.setHead(Head.toHead(MessageEnum.Req)); 65 | 66 | HsReq hsReq = new HsReq<>(); 67 | 68 | SyncOffsetMessage syncOffsetMessage = new SyncOffsetMessage(); 69 | syncOffsetMessage.setTopic(consumerMessageQueue.getTopic()); 70 | syncOffsetMessage.setConsumer(consumerMessageQueue.getConsumerGroup()); 71 | syncOffsetMessage.setOffSetMap(consumerMessageQueue.getOffSetMap()); 72 | 73 | hsReq.setData(syncOffsetMessage); 74 | hsReq.setOperation(OperationEnum.CommitOffset.getOperation()); 75 | hsEecodeData.setData(hsReq); 76 | 77 | channelFuture.channel().writeAndFlush(hsEecodeData).sync(); 78 | } 79 | } 80 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/consumer/executos/ExecutorMessageTask.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient.consumer.executos; 2 | 3 | import com.hsmq.data.message.PullMessage; 4 | import com.hsms.mqclient.consumer.config.ExecutorService; 5 | import com.hsms.mqclient.consumer.consumer.ConsumerHandlerManger; 6 | import com.hsms.mqclient.consumer.message.ConsumerMessageQueue; 7 | import io.netty.channel.ChannelFuture; 8 | 9 | import java.util.Map; 10 | import java.util.concurrent.ConcurrentLinkedQueue; 11 | import java.util.concurrent.ThreadPoolExecutor; 12 | 13 | /** 14 | * @author ：河神 15 | * @date ：Created in 2021/6/8 3:17 下午 16 | */ 17 | public class ExecutorMessageTask implements Runnable{ 18 | 19 | 20 | private final ConsumerMessageQueue consumerMessageQueue; 21 | private final ChannelFuture channelFuture; 22 | private final String topic; 23 | private final String consumerGroup; 24 | 25 | public ExecutorMessageTask(ChannelFuture channelFuture, 26 | ConsumerMessageQueue consumerMessageQueue) { 27 | this.channelFuture = channelFuture; 28 | this.topic = consumerMessageQueue.getTopic(); 29 | this.consumerGroup = consumerMessageQueue.getConsumerGroup(); 30 | this.consumerMessageQueue = consumerMessageQueue; 31 | } 32 | 33 | @Override 34 | public void run() { 35 | ThreadPoolExecutor executor = ExecutorService.getExecutor(); 36 | do { 37 | Map> queueMap = consumerMessageQueue.getQueueMap(); 38 | queueMap.forEach((queueId,queue)->{ 39 | PullMessage pullMessage = queue.poll(); 40 | if (pullMessage != null) { 41 | boolean consumer = ConsumerHandlerManger.consumer(consumerGroup,pullMessage); 42 | if (consumer) { 43 | consumerMessageQueue.confirmOffset(queueId,pullMessage); 44 | } 45 | } 46 | }); 47 | } while (!Thread.interrupted()); 48 | } 49 | 50 | 51 | 52 | } 53 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/consumer/executos/PullMessageTask.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient.consumer.executos; 2 | 3 | import com.hsmq.data.Head; 4 | import com.hsmq.data.HsReq; 5 | import com.hsmq.data.message.Pull; 6 | import com.hsmq.data.message.PullMessage; 7 | import com.hsmq.enums.MessageEnum; 8 | import com.hsmq.enums.OperationEnum; 9 | import com.hsmq.protocol.HsEecodeData; 10 | import com.hsms.mqclient.consumer.config.RegisteredConsumer; 11 | import com.hsms.mqclient.consumer.message.ConsumerMessageQueue; 12 | import io.netty.channel.ChannelFuture; 13 | 14 | import java.util.Map; 15 | import java.util.concurrent.ConcurrentLinkedQueue; 16 | 17 | /** 18 | * @author ：河神 19 | * @date ：Created in 2021/6/8 3:17 下午 20 | */ 21 | public class PullMessageTask implements Runnable{ 22 | 23 | private final ConsumerMessageQueue consumerMessageQueue; 24 | private final ChannelFuture channelFuture; 25 | private final String consumerGroup; 26 | private final String topic; 27 | 28 | public PullMessageTask(ChannelFuture channelFuture, String consumerGroup, ConsumerMessageQueue consumerMessageQueue) { 29 | this.channelFuture = channelFuture; 30 | this.consumerGroup = consumerGroup; 31 | this.topic = consumerMessageQueue.getTopic(); 32 | this.consumerMessageQueue = consumerMessageQueue; 33 | } 34 | 35 | @Override 36 | public void run() { 37 | while (true){ 38 | //尚未初始化完成 39 | if (!RegisteredConsumer.isInit()){ 40 | try { 41 | Thread.sleep(100L); 42 | } catch (InterruptedException e) { 43 | e.printStackTrace(); 44 | } 45 | continue; 46 | } 47 | Map> queueMap = consumerMessageQueue.getQueueMap(); 48 | for (Integer queueId : queueMap.keySet()) { 49 | try { 50 | pull(queueId); 51 | Thread.sleep(1L); 52 | } catch (InterruptedException e) { 53 | e.printStackTrace(); 54 | } 55 | } 56 | } 57 | } 58 | 59 | public void pull(Integer queueId) throws InterruptedException { 60 | HsEecodeData hsEecodeData = new HsEecodeData(); 61 | hsEecodeData.setHead(Head.toHead(MessageEnum.Req)); 62 | 63 | Map lastMessageMap = consumerMessageQueue.getLastMessageMap(); 64 | 65 | HsReq hsReq = new HsReq<>(); 66 | 67 | Pull pull = new Pull(); 68 | pull.setTopic(topic); 69 | pull.setQueueId(queueId); 70 | pull.setConsumerGroup(consumerGroup); 71 | pull.setSize(10); 72 | 73 | if (lastMessageMap.get(queueId)==null){ 74 | pull.setOffset(0); 75 | }else{ 76 | pull.setOffset(lastMessageMap.get(queueId)+1); 77 | } 78 | 79 | hsReq.setData(pull); 80 | hsReq.setOperation(OperationEnum.Pull.getOperation()); 81 | hsEecodeData.setData(hsReq); 82 | 83 | channelFuture.channel().writeAndFlush(hsEecodeData).sync(); 84 | } 85 | } 86 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/consumer/handle/ConsumerHandel.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient.consumer.handle; 2 | 3 | import com.hsmq.data.HsResp; 4 | import com.hsmq.data.message.MessageQueueData; 5 | import com.hsmq.data.message.PullMessageResp; 6 | import com.hsmq.enums.OperationEnum; 7 | import com.hsmq.protocol.HsDecodeData; 8 | import com.hsms.mqclient.consumer.config.RegisteredConsumer; 9 | import com.hsms.mqclient.consumer.message.ConsumerMessageQueue; 10 | import com.hsms.mqclient.producer.send.MessageClient; 11 | import io.netty.channel.ChannelHandlerContext; 12 | import io.netty.channel.SimpleChannelInboundHandler; 13 | import org.slf4j.Logger; 14 | import org.slf4j.LoggerFactory; 15 | 16 | /** 17 | * @author ：河神 18 | * @date ：Created in 2021/6/7 11:56 下午 19 | */ 20 | public class ConsumerHandel extends SimpleChannelInboundHandler { 21 | 22 | final static Logger log = LoggerFactory.getLogger(ConsumerHandel.class); 23 | 24 | @Override 25 | protected void channelRead0(ChannelHandlerContext ctx, HsDecodeData msg) throws Exception { 26 | HsResp data = (HsResp) msg.getData(); 27 | if (data.getData()!=null){ 28 | if (data.getData() instanceof PullMessageResp){ 29 | PullMessageResp pullMessageResp = (PullMessageResp) data.getData(); 30 | ConsumerMessageQueue queue = RegisteredConsumer.getConsumerMessageQueue(pullMessageResp.getTopic(),pullMessageResp.getConsumerGroup()); 31 | if (queue!=null){ 32 | queue.addMessage(pullMessageResp); 33 | }else{ 34 | log.info("not search topic queue:{}",pullMessageResp.getTopic()); 35 | } 36 | } 37 | if (data.getData() instanceof MessageQueueData){ 38 | RegisteredConsumer.initConsumerQueueHandle((MessageQueueData)data.getData()); 39 | } 40 | 41 | if (OperationEnum.SendMessage.getOperation().equals(data.getReqType())){ 42 | System.out.println("发送消息成功"); 43 | MessageClient.executorResp(data); 44 | } 45 | 46 | } 47 | } 48 | 49 | } 50 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/consumer/loadbalance/ConsumerLoadBalance.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient.consumer.loadbalance; 2 | 3 | /** 4 | * @author ：河神 5 | * @date ：Created in 2021/10/13 10:15 上午 6 | * 集群消息消费负载均衡 7 | */ 8 | public class ConsumerLoadBalance { 9 | 10 | // public static void loadBalanceConsumer(int queueSize,){ 11 | // 12 | // 13 | // } 14 | 15 | } 16 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/consumer/message/ConsumerMessageQueue.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient.consumer.message; 2 | 3 | import com.hsmq.data.message.MessageQueueData; 4 | import com.hsmq.data.message.PullMessage; 5 | import com.hsmq.data.message.PullMessageResp; 6 | import org.apache.commons.collections4.CollectionUtils; 7 | 8 | import java.util.Map; 9 | import java.util.concurrent.ConcurrentHashMap; 10 | import java.util.concurrent.ConcurrentLinkedQueue; 11 | 12 | /** 13 | * @author ：河神 14 | * @date ：Created in 2021/6/12 6:22 下午 15 | */ 16 | public class ConsumerMessageQueue { 17 | 18 | 19 | private final Map> queueMap = new ConcurrentHashMap<>(); 20 | 21 | private final Map offSetMap = new ConcurrentHashMap<>(); 22 | 23 | private final Map lastMessageMap = new ConcurrentHashMap<>(); 24 | 25 | /** 26 | * 主题 27 | */ 28 | private final String topic; 29 | 30 | /** 31 | * 消费组 32 | */ 33 | private final String consumerGroup; 34 | 35 | public ConsumerMessageQueue(String topic ,String consumerGroup) { 36 | this.topic = topic; 37 | this.consumerGroup = consumerGroup; 38 | } 39 | 40 | public void initQueue(MessageQueueData messageQueueData){ 41 | Integer queueSize = messageQueueData.getQueueSize(); 42 | for (int i=0;i()); 44 | } 45 | Map serverOffSetMap = messageQueueData.getOffSetMap(); 46 | if (serverOffSetMap!=null&&serverOffSetMap.size()>0){ 47 | offSetMap.putAll(serverOffSetMap); 48 | lastMessageMap.putAll(serverOffSetMap); 49 | } 50 | } 51 | 52 | /** 53 | * 将拉取的消息放入消费队列当中 54 | */ 55 | public void addMessage(PullMessageResp pullMessageResp){ 56 | if (CollectionUtils.isEmpty(pullMessageResp.getPullMessages())){ 57 | return; 58 | } 59 | ConcurrentLinkedQueue concurrentLinkedQueue = queueMap.get(pullMessageResp.getQueueId()); 60 | if (concurrentLinkedQueue==null){ 61 | return; 62 | } 63 | lastMessageMap.put(pullMessageResp.getQueueId(),pullMessageResp.getLastIndex()); 64 | concurrentLinkedQueue.addAll(pullMessageResp.getPullMessages()); 65 | } 66 | 67 | public Map> getQueueMap() { 68 | return queueMap; 69 | } 70 | 71 | public void confirmOffset(Integer queueId,PullMessage pullMessage) { 72 | offSetMap.put(queueId,pullMessage.getIndex()); 73 | } 74 | 75 | public Map getOffSetMap() { 76 | return offSetMap; 77 | } 78 | 79 | public Map getLastMessageMap() { 80 | return lastMessageMap; 81 | } 82 | 83 | public String getTopic() { 84 | return topic; 85 | } 86 | 87 | public String getConsumerGroup() { 88 | return consumerGroup; 89 | } 90 | } 91 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/producer/dto/SendMessageResult.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient.producer.dto; 2 | 3 | import java.util.StringJoiner; 4 | 5 | /** 6 | * @author ：河神 7 | * @date ：Created in 2021/10/10 2:20 下午 8 | */ 9 | public class SendMessageResult { 10 | 11 | 12 | /** 13 | * 是否消息发送成功 14 | */ 15 | private volatile Integer messageResult; 16 | 17 | /** 18 | * 是否消息发送出去 19 | */ 20 | private Boolean sendDone; 21 | 22 | /** 23 | * 消息id 24 | */ 25 | private String msgId; 26 | 27 | private String respDesc; 28 | 29 | public Integer getMessageResult() { 30 | return messageResult; 31 | } 32 | 33 | public void setMessageResult(Integer messageResult) { 34 | this.messageResult = messageResult; 35 | } 36 | 37 | public Boolean getSendDone() { 38 | return sendDone; 39 | } 40 | 41 | public void setSendDone(Boolean sendDone) { 42 | this.sendDone = sendDone; 43 | } 44 | 45 | public String getMsgId() { 46 | return msgId; 47 | } 48 | 49 | public void setMsgId(String msgId) { 50 | this.msgId = msgId; 51 | } 52 | 53 | public String getRespDesc() { 54 | return respDesc; 55 | } 56 | 57 | public void setRespDesc(String respDesc) { 58 | this.respDesc = respDesc; 59 | } 60 | 61 | @Override 62 | public String toString() { 63 | return new StringJoiner(", ", SendMessageResult.class.getSimpleName() + "[", "]") 64 | .add("messageResult=" + messageResult) 65 | .add("sendDone=" + sendDone) 66 | .add("msgId='" + msgId + "'") 67 | .add("respDesc='" + respDesc + "'") 68 | .toString(); 69 | } 70 | } 71 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/producer/handle/NormalHandel.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient.producer.handle; 2 | 3 | import com.alibaba.fastjson.JSON; 4 | import com.hsmq.data.HsResp; 5 | import com.hsmq.protocol.HsDecodeData; 6 | import io.netty.channel.ChannelHandlerContext; 7 | import io.netty.channel.SimpleChannelInboundHandler; 8 | import org.slf4j.Logger; 9 | import org.slf4j.LoggerFactory; 10 | 11 | /** 12 | * @author ：河神 13 | * @date ：Created in 2021/6/7 11:56 下午 14 | */ 15 | public class NormalHandel extends SimpleChannelInboundHandler { 16 | 17 | final static Logger log = LoggerFactory.getLogger(NormalHandel.class); 18 | 19 | @Override 20 | protected void channelRead0(ChannelHandlerContext ctx, HsDecodeData msg) throws Exception { 21 | HsResp data = (HsResp) msg.getData(); 22 | System.out.println("普通返回"); 23 | if (data.getData()!=null){ 24 | if (data.getData() instanceof String){ 25 | log.info("普通返回:{}：",JSON.toJSONString(msg.getData())); 26 | } 27 | } 28 | } 29 | 30 | } 31 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/producer/handle/ObjectOutHandle.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient.producer.handle; 2 | 3 | import io.netty.channel.ChannelHandlerContext; 4 | import io.netty.channel.ChannelOutboundHandlerAdapter; 5 | import io.netty.channel.ChannelPromise; 6 | 7 | import java.net.SocketAddress; 8 | 9 | /** 10 | * @author: 河神 11 | * @date:2020-04-18 12 | */ 13 | public class ObjectOutHandle extends ChannelOutboundHandlerAdapter { 14 | 15 | 16 | /** 17 | * 服务端执行bind时 18 | * @param ctx 19 | * @param localAddress 20 | * @param promise 21 | * @throws Exception 22 | */ 23 | @Override 24 | public void bind(ChannelHandlerContext ctx, SocketAddress localAddress, ChannelPromise promise) throws Exception { 25 | super.bind(ctx, localAddress, promise); 26 | } 27 | 28 | /** 29 | * 客户端执行connect连接服务端时进入 30 | * @param ctx 31 | * @param remoteAddress 32 | * @param localAddress 33 | * @param promise 34 | * @throws Exception 35 | */ 36 | @Override 37 | public void connect(ChannelHandlerContext ctx, SocketAddress remoteAddress, SocketAddress localAddress, ChannelPromise promise) throws Exception { 38 | super.connect(ctx, remoteAddress, localAddress, promise); 39 | } 40 | 41 | /** 42 | * 43 | * @param ctx 44 | * @param promise 45 | * @throws Exception 46 | */ 47 | @Override 48 | public void disconnect(ChannelHandlerContext ctx, ChannelPromise promise) throws Exception { 49 | super.disconnect(ctx, promise); 50 | } 51 | 52 | @Override 53 | public void close(ChannelHandlerContext ctx, ChannelPromise promise) throws Exception { 54 | super.close(ctx, promise); 55 | } 56 | 57 | @Override 58 | public void deregister(ChannelHandlerContext ctx, ChannelPromise promise) throws Exception { 59 | super.deregister(ctx, promise); 60 | } 61 | 62 | @Override 63 | public void read(ChannelHandlerContext ctx) throws Exception { 64 | super.read(ctx); 65 | } 66 | 67 | @Override 68 | public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) throws Exception { 69 | 70 | super.write(ctx,msg, promise); 71 | } 72 | 73 | @Override 74 | public void flush(ChannelHandlerContext ctx) throws Exception { 75 | super.flush(ctx); 76 | } 77 | 78 | 79 | } 80 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/producer/send/MessageClient.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient.producer.send; 2 | 3 | import com.hsmq.data.Head; 4 | import com.hsmq.data.HsReq; 5 | import com.hsmq.data.HsResp; 6 | import com.hsmq.data.message.SendMessage; 7 | import com.hsmq.enums.MessageEnum; 8 | import com.hsmq.enums.OperationEnum; 9 | import com.hsmq.protocol.HsEecodeData; 10 | import com.hsms.mqclient.producer.dto.SendMessageResult; 11 | import io.netty.channel.ChannelFuture; 12 | 13 | import java.util.concurrent.ConcurrentHashMap; 14 | import java.util.concurrent.TimeUnit; 15 | import java.util.concurrent.atomic.AtomicInteger; 16 | 17 | /** 18 | * @author ：河神 19 | * @date ：Created in 2021/10/9 11:08 下午 20 | */ 21 | public class MessageClient { 22 | 23 | private static ChannelFuture channelFuture; 24 | 25 | private static ConcurrentHashMap resultMap = new ConcurrentHashMap<>(); 26 | 27 | private static AtomicInteger reqId = new AtomicInteger(0); 28 | 29 | public static void setChannelFuture(ChannelFuture channelFuture) { 30 | MessageClient.channelFuture = channelFuture; 31 | } 32 | 33 | public static void executorResp(HsResp data){ 34 | SendMessageResult sendMessageResult = resultMap.get(data.getReqId()); 35 | if (sendMessageResult ==null){ 36 | return; 37 | } 38 | sendMessageResult.setMsgId((String) data.getData()); 39 | sendMessageResult.setMessageResult(data.getResult()); 40 | sendMessageResult.setRespDesc("发送成功"); 41 | 42 | resultMap.remove(data.getReqId()); 43 | } 44 | 45 | public static SendMessageResult sendMsg(SendMessage sendMessage){ 46 | SendMessageResult sendMessageResult = new SendMessageResult(); 47 | try { 48 | HsEecodeData hsEecodeData = new HsEecodeData(); 49 | hsEecodeData.setHead(Head.toHead(MessageEnum.Req)); 50 | 51 | HsReq hsReq = new HsReq<>(); 52 | hsReq.setOperation(OperationEnum.SendMessage.getOperation()); 53 | hsReq.setData(sendMessage); 54 | hsReq.setReqId(reqId.incrementAndGet()); 55 | hsEecodeData.setData(hsReq); 56 | 57 | resultMap.put(hsReq.getReqId(), sendMessageResult); 58 | 59 | MessageClient.channelFuture 60 | .channel() 61 | .writeAndFlush(hsEecodeData) 62 | .addListener((future)->{ 63 | if (future.isSuccess()) { 64 | sendMessageResult.setSendDone(true); 65 | } else { 66 | sendMessageResult.setSendDone(false); 67 | sendMessageResult.setRespDesc("消息发送失败"); 68 | } 69 | }); 70 | 71 | long nanosTimeout = TimeUnit.SECONDS.toNanos(3); 72 | final long deadline = System.nanoTime() + nanosTimeout; 73 | 74 | while (true) { 75 | if (nanosTimeout<0){ 76 | sendMessageResult.setRespDesc("发送超时"); 77 | sendMessageResult.setMessageResult(-1); 78 | break; 79 | } 80 | if (sendMessageResult.getMessageResult() != null) { 81 | break; 82 | } 83 | nanosTimeout = deadline - System.nanoTime(); 84 | } 85 | return sendMessageResult; 86 | } catch (Exception e) { 87 | sendMessageResult.setMessageResult(-2); 88 | sendMessageResult.setSendDone(false); 89 | e.printStackTrace(); 90 | } 91 | return sendMessageResult; 92 | } 93 | 94 | 95 | } 96 | -------------------------------------------------------------------------------- /mq-client/src/main/java/com/hsms/mqclient/reactor/ClientReactor.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqclient.reactor; 2 | 3 | 4 | import com.hsmq.decode.LengthObjectDecode; 5 | import com.hsmq.encode.LengthObjectEncode; 6 | import com.hsms.mqclient.consumer.handle.ConsumerHandel; 7 | import com.hsms.mqclient.producer.handle.NormalHandel; 8 | import io.netty.bootstrap.Bootstrap; 9 | import io.netty.channel.ChannelFuture; 10 | import io.netty.channel.ChannelInitializer; 11 | import io.netty.channel.ChannelOption; 12 | import io.netty.channel.ChannelPipeline; 13 | import io.netty.channel.EventLoopGroup; 14 | import io.netty.channel.nio.NioEventLoopGroup; 15 | import io.netty.channel.socket.nio.NioSocketChannel; 16 | import org.slf4j.Logger; 17 | import org.slf4j.LoggerFactory; 18 | 19 | /** 20 | * @author: 河神 21 | * @date:2020-04-18 22 | */ 23 | public class ClientReactor { 24 | 25 | final static Logger log = LoggerFactory.getLogger(ClientReactor.class); 26 | 27 | 28 | private String host; 29 | private int port; 30 | private Bootstrap bootstrap; 31 | 32 | public ClientReactor(String host, int port) { 33 | this.host = host; 34 | this.port = port; 35 | } 36 | 37 | public void start() { 38 | log.info("ConsumerClient#start"); 39 | bootstrap = new Bootstrap(); 40 | 41 | EventLoopGroup eventExecutors = new NioEventLoopGroup(); 42 | 43 | bootstrap.group(eventExecutors) 44 | .channel(NioSocketChannel.class) 45 | .option(ChannelOption.TCP_NODELAY, true); 46 | 47 | 48 | bootstrap.handler(new ChannelInitializer() { 49 | @Override 50 | protected void initChannel(NioSocketChannel socketChannel) throws Exception { 51 | ChannelPipeline pipeline = socketChannel.pipeline(); 52 | pipeline.addLast(new LengthObjectEncode()); 53 | pipeline.addLast(new LengthObjectDecode()); 54 | pipeline.addLast(new ConsumerHandel()); 55 | pipeline.addLast(new NormalHandel()); 56 | } 57 | }); 58 | 59 | } 60 | 61 | public ChannelFuture getChannelFuture() throws InterruptedException { 62 | return bootstrap.connect(host, port).sync(); 63 | } 64 | 65 | } 66 | -------------------------------------------------------------------------------- /mq-client/src/main/resources/check/hsmq_schema.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "object", 3 | "required":["consumer","server"], 4 | "properties": { 5 | "consumer": { 6 | "type": "object" 7 | }, 8 | "server": { 9 | "type": "object" 10 | } 11 | } 12 | } -------------------------------------------------------------------------------- /mq-client/src/main/resources/hsmq.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "check/hsmq_schema.json", 3 | "consumer": { 4 | "consumerB": [ 5 | { 6 | "topic":"TopicA" 7 | }, 8 | { 9 | "topic":"TopicB" 10 | } 11 | ] 12 | }, 13 | "server": { 14 | "host": "127.0.0.1", 15 | "post": 9001 16 | } 17 | } -------------------------------------------------------------------------------- /mq-client/src/main/resources/logback.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | %d{yyy-MM-dd HH:mm:ss,GMT+8}[%p][%t][%file:%M %line][%X] - %m%n 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | -------------------------------------------------------------------------------- /mq-client/src/test/java/com/mqclient/BtyVufTest.java: -------------------------------------------------------------------------------- 1 | package com.mqclient; 2 | 3 | import io.netty.buffer.ByteBuf; 4 | import io.netty.buffer.ByteBufAllocator; 5 | import io.netty.buffer.CompositeByteBuf; 6 | import io.netty.buffer.Unpooled; 7 | 8 | /** 9 | * @author ：河神 10 | * @date ：Created in 2021/10/20 4:34 下午 11 | */ 12 | public class BtyVufTest { 13 | 14 | public static void main(String[] args) { 15 | 16 | 17 | CompositeByteBuf compositeByteBuf = new CompositeByteBuf(ByteBufAllocator.DEFAULT, true, 10); 18 | 19 | ByteBuf buffer = Unpooled.buffer(12); 20 | buffer.writeBytes("测试".getBytes()); 21 | compositeByteBuf.addComponent(buffer); 22 | 23 | ByteBuf buffer2 = Unpooled.buffer(12); 24 | buffer2.writeBytes("同学".getBytes()); 25 | compositeByteBuf.addComponent(buffer2); 26 | 27 | 28 | 29 | 30 | 31 | } 32 | } 33 | -------------------------------------------------------------------------------- /mq-client/src/test/java/com/mqclient/ClientA.java: -------------------------------------------------------------------------------- 1 | package com.mqclient; 2 | 3 | import com.hsmq.data.Head; 4 | import com.hsmq.data.HsReq; 5 | import com.hsmq.data.message.SendMessage; 6 | import com.hsmq.enums.MessageEnum; 7 | import com.hsmq.enums.OperationEnum; 8 | import com.hsmq.protocol.HsEecodeData; 9 | import com.hsms.mqclient.consumer.config.RegisteredConsumer; 10 | import com.hsms.mqclient.consumer.consumer.ConsumerHandlerManger; 11 | import com.hsms.mqclient.reactor.ClientReactor; 12 | 13 | /** 14 | * @author ：河神 15 | * @date ：Created in 2021/10/9 2:22 下午 16 | */ 17 | public class ClientA { 18 | 19 | 20 | public static void main(String[] args) throws InterruptedException { 21 | 22 | args = new String[]{"TopicB"}; 23 | 24 | //记录启动时间 25 | RegisteredConsumer.setStopWatch(); 26 | //注册Netty 27 | ClientReactor clientReactor = new ClientReactor("127.0.0.1", 9001); 28 | //启动Netty 29 | clientReactor.start(); 30 | //初始化消费者 31 | ConsumerHandlerManger.initConsumer("AConsumer",args,clientReactor.getChannelFuture()); 32 | 33 | //发送消息 34 | for (int i=5000;;i++){ 35 | HsEecodeData hsEecodeData = new HsEecodeData(); 36 | hsEecodeData.setHead(Head.toHead(MessageEnum.Req)); 37 | 38 | HsReq hsReq = new HsReq<>(); 39 | SendMessage sendMessage = new SendMessage(); 40 | 41 | hsReq.setOperation(OperationEnum.SendMessage.getOperation()); 42 | 43 | sendMessage.setTopic("TopicA"); 44 | sendMessage.setTag("tagA"); 45 | sendMessage.setBody("消息---"+i); 46 | hsReq.setData(sendMessage); 47 | hsReq.setReqId(i); 48 | hsEecodeData.setData(hsReq); 49 | 50 | Thread.sleep(1000L); 51 | clientReactor.getChannelFuture().channel().writeAndFlush(hsEecodeData).sync(); 52 | } 53 | 54 | } 55 | 56 | } 57 | -------------------------------------------------------------------------------- /mq-client/src/test/java/com/mqclient/ClientB.java: -------------------------------------------------------------------------------- 1 | package com.mqclient; 2 | 3 | import com.hsms.mqclient.consumer.config.RegisteredConsumer; 4 | import com.hsms.mqclient.consumer.consumer.ConsumerHandlerManger; 5 | import com.hsms.mqclient.reactor.ClientReactor; 6 | 7 | /** 8 | * @author ：河神 9 | * @date ：Created in 2021/10/9 2:22 下午 10 | */ 11 | public class ClientB { 12 | 13 | 14 | public static void main(String[] args) throws InterruptedException { 15 | 16 | args = new String[]{"TopicA"}; 17 | 18 | 19 | //记录启动时间 20 | RegisteredConsumer.setStopWatch(); 21 | //注册Netty 22 | ClientReactor clientReactor = new ClientReactor("127.0.0.1", 9001); 23 | //启动Netty 24 | clientReactor.start(); 25 | //初始化消费者 26 | ConsumerHandlerManger.initConsumer("RConsumer",args,clientReactor.getChannelFuture()); 27 | 28 | //发送消息 29 | // for (int i=5000;;i++){ 30 | // HsEecodeData hsEecodeData = new HsEecodeData(); 31 | // hsEecodeData.setHead(Head.toHead(MessageEnum.Req)); 32 | // 33 | // HsReq hsReq = new HsReq<>(); 34 | // SendMessage sendMessage = new SendMessage(); 35 | // 36 | // hsReq.setOperation(OperationEnum.SendMessage.getOperation()); 37 | // 38 | // sendMessage.setTopic("TopicB"); 39 | // sendMessage.setTag("tagB"); 40 | // sendMessage.setBody("消息---"+i); 41 | // hsReq.setData(sendMessage); 42 | // hsEecodeData.setData(hsReq); 43 | // 44 | // Thread.sleep(100L); 45 | // clientReactor.getChannelFuture().channel().writeAndFlush(hsEecodeData).sync(); 46 | // } 47 | 48 | } 49 | 50 | } 51 | -------------------------------------------------------------------------------- /mq-server/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 5 | 6 | hsmq 7 | com.heshen 8 | 1.0-SNAPSHOT 9 | 10 | 4.0.0 11 | 12 | mqserver 13 | 14 | 15 | 16 | com.google.guava 17 | guava 18 | 19 | 20 | 21 | ch.qos.logback 22 | logback-classic 23 | 24 | 25 | 26 | com.heshen 27 | protocol 28 | 29 | 30 | 31 | com.heshen 32 | common 33 | 34 | 35 | 36 | com.heshen 37 | storage 38 | 39 | 40 | 41 | 42 | -------------------------------------------------------------------------------- /mq-server/src/main/java/com/hsms/mqserver/ServerStartup.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqserver; 2 | 3 | 4 | import com.hsmq.storage.config.TopicConfig; 5 | import com.hsms.mqserver.config.ServerConfig; 6 | import com.hsms.mqserver.server.MqServerBootstrap; 7 | import org.slf4j.Logger; 8 | import org.slf4j.LoggerFactory; 9 | 10 | /** 11 | * @author ：河神 12 | * @date ：Created in 2021/6/4 3:46 下午 13 | */ 14 | public class ServerStartup { 15 | 16 | final static Logger log = LoggerFactory.getLogger(ServerStartup.class); 17 | 18 | public static void main(String[] args) { 19 | start0(args); 20 | } 21 | 22 | public static void start0(String[] args){ 23 | String start = 24 | "\n _ _ _____ _____ \n" + 25 | "| | | |/ ___| / ___| \n" + 26 | "| |_| |\\ `--. ______\\ `--. ___ _ ____ _____ _ __ \n" + 27 | "| _ | `--. \\______|`--. \\/ _ \\ '__\\ \\ / / _ \\ '__|\n" + 28 | "| | | |/\\__/ / /\\__/ / __/ | \\ V / __/ | \n" + 29 | "\\_| |_/\\____/ \\____/ \\___|_| \\_/ \\___|_| \n"; 30 | log.info(start); 31 | //默认配置 32 | buildConfig(); 33 | //启动Reactor Server 34 | new MqServerBootstrap().start(); 35 | } 36 | 37 | 38 | public static void buildConfig(){ 39 | ServerConfig.Port = 9001; 40 | ServerConfig.TopicConfig.put("TopicA", TopicConfig.getDefault("TopicA")); 41 | ServerConfig.TopicConfig.put("TopicB", TopicConfig.getDefault("TopicB")); 42 | } 43 | 44 | 45 | 46 | } 47 | -------------------------------------------------------------------------------- /mq-server/src/main/java/com/hsms/mqserver/config/ServerConfig.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqserver.config; 2 | 3 | import com.hsmq.storage.config.TopicConfig; 4 | 5 | import java.util.concurrent.ConcurrentHashMap; 6 | 7 | /** 8 | * @author ：河神 9 | * @date ：Created in 2021/7/24 2:04 下午 10 | */ 11 | public class ServerConfig { 12 | 13 | public static int Port = 9001; 14 | 15 | public static ConcurrentHashMap TopicConfig = new ConcurrentHashMap<>(); 16 | 17 | } 18 | -------------------------------------------------------------------------------- /mq-server/src/main/java/com/hsms/mqserver/data/ConsumerQueueManger.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqserver.data; 2 | 3 | import com.hsmq.data.message.SendMessage; 4 | import com.hsmq.storage.config.TopicConfig; 5 | import com.hsmq.storage.durability.MessageDurability; 6 | 7 | import java.util.concurrent.ConcurrentHashMap; 8 | import java.util.concurrent.ConcurrentMap; 9 | 10 | /** 11 | * @author ：河神 12 | * @date ：Created in 2021/6/12 5:20 下午 13 | */ 14 | public class ConsumerQueueManger { 15 | 16 | private static final ConcurrentMap DATA = new ConcurrentHashMap<>(); 17 | 18 | public static void registerTopic(TopicConfig topicConfig){ 19 | DATA.put(topicConfig.getTopicName(), new TopicListener(topicConfig)); 20 | } 21 | 22 | public static TopicListener getTopicListener(String topic){ 23 | return DATA.get(topic); 24 | } 25 | 26 | public static boolean existsTopic(SendMessage sendMessage){ 27 | return getTopicListener(sendMessage.getTopic())!=null; 28 | } 29 | 30 | public static boolean pushConsumerQueue(SendMessage sendMessage, MessageDurability messageDurability){ 31 | TopicListener topicListener = getTopicListener(sendMessage.getTopic()); 32 | return topicListener.addMsg2Queue(sendMessage,messageDurability); 33 | } 34 | 35 | } 36 | -------------------------------------------------------------------------------- /mq-server/src/main/java/com/hsms/mqserver/data/MessageStore.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqserver.data; 2 | 3 | import com.hsmq.data.HsResp; 4 | import com.hsmq.data.message.SendMessage; 5 | import com.hsmq.enums.OperationEnum; 6 | import com.hsmq.enums.ResultEnum; 7 | import com.hsmq.storage.data.MessageStorage; 8 | import com.hsmq.storage.durability.MessageDurability; 9 | import org.slf4j.Logger; 10 | import org.slf4j.LoggerFactory; 11 | 12 | /** 13 | * @author ：河神 14 | * @date ：Created in 2021/6/7 11:10 下午 15 | */ 16 | public class MessageStore { 17 | 18 | final static Logger log = LoggerFactory.getLogger(MessageStore.class); 19 | 20 | public HsResp saveMessage(SendMessage sendMessage){ 21 | boolean existsTopic = ConsumerQueueManger.existsTopic(sendMessage); 22 | if (!existsTopic){ 23 | return HsResp.topicNotExistsError(); 24 | } 25 | 26 | MessageDurability messageDurability = MessageStorage.saveMessage(sendMessage); 27 | boolean push = ConsumerQueueManger.pushConsumerQueue(sendMessage, messageDurability); 28 | if (!push){ 29 | return HsResp.topicNotExistsError(); 30 | } 31 | log.info("saveMessage#messageDurability:{}",messageDurability); 32 | HsResp resp = new HsResp<>(); 33 | resp.setData(sendMessage.getMsgId()); 34 | resp.setOperation(OperationEnum.Resp.getOperation()); 35 | resp.setResult(ResultEnum.SendOK.getCode()); 36 | return resp; 37 | } 38 | 39 | } 40 | -------------------------------------------------------------------------------- /mq-server/src/main/java/com/hsms/mqserver/data/TopicListener.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqserver.data; 2 | 3 | import com.hsmq.data.message.Pull; 4 | import com.hsmq.data.message.PullMessage; 5 | import com.hsmq.data.message.SendMessage; 6 | import com.hsmq.storage.config.TopicConfig; 7 | import com.hsmq.storage.data.MessageDurabilityStorage; 8 | import com.hsmq.storage.data.MessageStorage; 9 | import com.hsmq.storage.durability.MessageDurability; 10 | 11 | import java.io.IOException; 12 | import java.util.Collections; 13 | import java.util.HashMap; 14 | import java.util.List; 15 | 16 | /** 17 | * @author ：河神 18 | * @date ：Created in 2021/7/24 5:18 下午 19 | */ 20 | public class TopicListener { 21 | 22 | private TopicConfig topicConfig; 23 | private volatile int queueId = 0; 24 | public int queueSize ; 25 | 26 | public TopicListener(TopicConfig topicConfig) { 27 | this.topicConfig = topicConfig; 28 | this.queueSize = topicConfig.getMessageQueueSize(); 29 | } 30 | 31 | 32 | public List pullMessage(Pull pull){ 33 | List data = MessageDurabilityStorage.readMessageQueue(pull.getQueueId(), pull.getTopic(), pull.getOffset()); 34 | if (data.size()==0){ 35 | return null; 36 | } 37 | return MessageStorage.readMessages(data); 38 | } 39 | 40 | public boolean addMsg2Queue(SendMessage sendMessage,MessageDurability messageDurability){ 41 | int queueId = getQueueId(); 42 | try { 43 | MessageDurabilityStorage.saveMessageQueue(queueId,sendMessage.getTopic(), Collections.singletonList(messageDurability)); 44 | } catch (IOException e) { 45 | e.printStackTrace(); 46 | } 47 | return true; 48 | } 49 | 50 | 51 | public synchronized int getQueueId(){ 52 | queueId++; 53 | if (queueId>10000){ 54 | queueId = 0; 55 | } 56 | return queueId%queueSize; 57 | } 58 | 59 | public int getQueueSize() { 60 | return queueSize; 61 | } 62 | } 63 | -------------------------------------------------------------------------------- /mq-server/src/main/java/com/hsms/mqserver/handle/ServerInHandel.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqserver.handle; 2 | 3 | import com.hsmq.protocol.HsDecodeData; 4 | import com.hsmq.protocol.HsEecodeData; 5 | import com.hsms.mqserver.strategy.MessageStrategy; 6 | import io.netty.channel.ChannelHandlerContext; 7 | import io.netty.channel.SimpleChannelInboundHandler; 8 | import org.slf4j.Logger; 9 | import org.slf4j.LoggerFactory; 10 | 11 | /** 12 | * @author: 河神 13 | * @date:2020-04-18 14 | */ 15 | public class ServerInHandel extends SimpleChannelInboundHandler { 16 | 17 | final static Logger log = LoggerFactory.getLogger(ServerInHandel.class); 18 | 19 | 20 | @Override 21 | protected void channelRead0(ChannelHandlerContext ctx, HsDecodeData decodeData) throws Exception { 22 | if (decodeData==null){ 23 | log.warn("decodeData is null"); 24 | return; 25 | } 26 | HsEecodeData executor = MessageStrategy.executor(decodeData); 27 | ctx.channel().writeAndFlush(executor).sync(); 28 | } 29 | 30 | } 31 | -------------------------------------------------------------------------------- /mq-server/src/main/java/com/hsms/mqserver/reactor/ObjectReactor.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqserver.reactor; 2 | 3 | import com.hsmq.decode.LengthObjectDecode; 4 | import com.hsmq.encode.LengthObjectEncode; 5 | import com.hsms.mqserver.handle.ServerInHandel; 6 | import io.netty.bootstrap.ServerBootstrap; 7 | import io.netty.channel.ChannelFuture; 8 | import io.netty.channel.ChannelInitializer; 9 | import io.netty.channel.ChannelOption; 10 | import io.netty.channel.ChannelPipeline; 11 | import io.netty.channel.EventLoopGroup; 12 | import io.netty.channel.nio.NioEventLoopGroup; 13 | import io.netty.channel.socket.SocketChannel; 14 | import io.netty.channel.socket.nio.NioServerSocketChannel; 15 | 16 | import java.util.concurrent.TimeUnit; 17 | 18 | /** 19 | * @author ：河神 20 | * @date ：Created in 2021/6/4 3:53 下午 21 | */ 22 | public class ObjectReactor { 23 | 24 | private final int port; 25 | public ObjectReactor(int port ) { 26 | this.port = port; 27 | } 28 | 29 | public void start() { 30 | try { 31 | start0(); 32 | } catch (InterruptedException e) { 33 | e.printStackTrace(); 34 | } 35 | } 36 | 37 | public void start0() throws InterruptedException { 38 | ServerBootstrap serverBootstrap = new ServerBootstrap(); 39 | EventLoopGroup boss = new NioEventLoopGroup(); 40 | EventLoopGroup work = new NioEventLoopGroup(); 41 | serverBootstrap.group(boss,work); 42 | 43 | serverBootstrap.channel(NioServerSocketChannel.class); 44 | 45 | serverBootstrap.childHandler(new ChannelInitializer() { 46 | @Override 47 | protected void initChannel(SocketChannel socketChannel) { 48 | ChannelPipeline pipeline = socketChannel.pipeline(); 49 | pipeline.addLast(new LengthObjectDecode()); 50 | pipeline.addLast(new LengthObjectEncode()); 51 | pipeline.addLast(new ServerInHandel()); 52 | } 53 | }); 54 | 55 | serverBootstrap.option(ChannelOption.SO_BACKLOG,128); 56 | serverBootstrap.childOption(ChannelOption.SO_KEEPALIVE,true); 57 | 58 | ChannelFuture channelFuture = serverBootstrap.bind(port).sync(); 59 | 60 | //定时任务 61 | channelFuture.channel().eventLoop().scheduleWithFixedDelay(()->{ 62 | //定时任务 63 | },1, 3L,TimeUnit.SECONDS); 64 | 65 | // channelFuture.channel().eventLoop().schedule(()->{}, 66 | // 1, 3L,TimeUnit.SECONDS); 67 | 68 | channelFuture.channel().closeFuture().sync(); 69 | } 70 | } 71 | -------------------------------------------------------------------------------- /mq-server/src/main/java/com/hsms/mqserver/server/MqServerBootstrap.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqserver.server; 2 | 3 | import com.hsms.mqserver.config.ServerConfig; 4 | import com.hsms.mqserver.data.ConsumerQueueManger; 5 | import com.hsms.mqserver.reactor.ObjectReactor; 6 | 7 | /** 8 | * @author ：河神 9 | * @date ：Created in 2021/7/24 2:04 下午 10 | */ 11 | public class MqServerBootstrap { 12 | 13 | 14 | public void start() { 15 | //启动注册topic 16 | ServerConfig.TopicConfig.forEach((topic,config)->{ 17 | ConsumerQueueManger.registerTopic(config); 18 | }); 19 | //启动服务 20 | new ObjectReactor(ServerConfig.Port).start(); 21 | 22 | } 23 | } 24 | -------------------------------------------------------------------------------- /mq-server/src/main/java/com/hsms/mqserver/strategy/MessageStrategy.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqserver.strategy; 2 | 3 | import com.hsmq.data.HsReq; 4 | import com.hsmq.data.HsResp; 5 | import com.hsmq.enums.OperationEnum; 6 | import com.hsmq.error.HsError; 7 | import com.hsmq.protocol.HsDecodeData; 8 | import com.hsmq.protocol.HsEecodeData; 9 | import com.hsms.mqserver.strategy.executors.BaseExecutor; 10 | import com.hsms.mqserver.strategy.executors.impl.CommitOffsetExecutor; 11 | import com.hsms.mqserver.strategy.executors.impl.PullExecutor; 12 | import com.hsms.mqserver.strategy.executors.impl.SendMessageExecutor; 13 | import com.hsms.mqserver.strategy.executors.impl.TopicDataExecutor; 14 | import org.slf4j.Logger; 15 | import org.slf4j.LoggerFactory; 16 | 17 | import java.util.HashMap; 18 | import java.util.Map; 19 | 20 | /** 21 | * @author ：河神 22 | * @date ：Created in 2021/6/9 1:41 下午 23 | */ 24 | public class MessageStrategy { 25 | 26 | final static Logger log = LoggerFactory.getLogger(MessageStrategy.class); 27 | 28 | private static final Map> enumExecutorMap ; 29 | 30 | static { 31 | enumExecutorMap = new HashMap<>(); 32 | enumExecutorMap.put(OperationEnum.SendMessage,new SendMessageExecutor()); 33 | enumExecutorMap.put(OperationEnum.Pull,new PullExecutor()); 34 | enumExecutorMap.put(OperationEnum.CommitOffset,new CommitOffsetExecutor()); 35 | enumExecutorMap.put(OperationEnum.TopicData,new TopicDataExecutor()); 36 | log.info("Init MessageStrategy"); 37 | } 38 | 39 | public static HsEecodeData executor(HsDecodeData hsDecodeData){ 40 | HsReq hsReq = (HsReq) hsDecodeData.getData(); 41 | 42 | OperationEnum operationEnum = hsReq.getOperationEnum(); 43 | if (operationEnum==null){ 44 | HsResp parameterWrongType = HsError.ParameterWrongType; 45 | parameterWrongType.setReqType(hsReq.getOperation()); 46 | parameterWrongType.setReqId(hsReq.getReqId()); 47 | return HsEecodeData.resp(parameterWrongType); 48 | } 49 | 50 | BaseExecutor baseExecutor = enumExecutorMap.get(operationEnum); 51 | return baseExecutor.executorReq(hsReq); 52 | } 53 | 54 | 55 | } 56 | -------------------------------------------------------------------------------- /mq-server/src/main/java/com/hsms/mqserver/strategy/executors/BaseExecutor.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqserver.strategy.executors; 2 | 3 | import com.hsmq.data.HsReq; 4 | import com.hsmq.data.HsResp; 5 | import com.hsmq.protocol.HsEecodeData; 6 | import com.hsms.mqserver.data.MessageStore; 7 | 8 | /** 9 | * @author ：河神 10 | * @date ：Created in 2021/6/9 1:52 下午 11 | */ 12 | public abstract class BaseExecutor { 13 | 14 | public static MessageStore messageStore = new MessageStore(); 15 | 16 | public abstract HsResp executor(HsReq hsReq); 17 | 18 | public abstract HsReq convertReq(HsReq hsReq); 19 | 20 | public HsEecodeData executorReq(HsReq hsReq){ 21 | HsReq fixedHsReq = convertReq(hsReq); 22 | if (fixedHsReq==null){ 23 | return HsEecodeData.typeError(); 24 | } 25 | 26 | HsResp hsResp = executor(fixedHsReq); 27 | hsResp.setReqType(hsReq.getOperation()); 28 | hsResp.setReqId(hsReq.getReqId()); 29 | return HsEecodeData.resp(hsResp); 30 | } 31 | 32 | } 33 | -------------------------------------------------------------------------------- /mq-server/src/main/java/com/hsms/mqserver/strategy/executors/impl/CommitOffsetExecutor.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqserver.strategy.executors.impl; 2 | 3 | import com.hsmq.data.HsReq; 4 | import com.hsmq.data.HsResp; 5 | import com.hsmq.data.message.SyncOffsetMessage; 6 | import com.hsmq.enums.OperationEnum; 7 | import com.hsmq.enums.ResultEnum; 8 | import com.hsmq.storage.data.QueueOffsetStorage; 9 | import com.hsms.mqserver.strategy.executors.BaseExecutor; 10 | import org.slf4j.Logger; 11 | import org.slf4j.LoggerFactory; 12 | 13 | /** 14 | * @author ：河神 15 | * @date ：Created in 2021/6/12 6:18 下午 16 | */ 17 | public class CommitOffsetExecutor extends BaseExecutor { 18 | 19 | final static Logger log = LoggerFactory.getLogger(CommitOffsetExecutor.class); 20 | 21 | 22 | @Override 23 | public HsResp executor(HsReq hsReq) { 24 | SyncOffsetMessage data = hsReq.getData(); 25 | QueueOffsetStorage.saveConsumer(data.getTopic(),data.getConsumer(), data.getOffSetMap()); 26 | 27 | log.info("CommitOffsetExecutor sync offset:{}",data.getOffSetMap()); 28 | 29 | HsResp resp = new HsResp<>(); 30 | resp.setData("OK"); 31 | resp.setOperation(OperationEnum.CommitOffset.getOperation()); 32 | resp.setResult(ResultEnum.SendOK.getCode()); 33 | return resp; 34 | } 35 | 36 | @Override 37 | public HsReq convertReq(HsReq hsReq) { 38 | if (hsReq.getData() instanceof SyncOffsetMessage){ 39 | HsReq data = new HsReq<>(); 40 | data.setData((SyncOffsetMessage)hsReq.getData()); 41 | data.setOperation(hsReq.getOperation()); 42 | return data; 43 | } 44 | return null; 45 | } 46 | } 47 | -------------------------------------------------------------------------------- /mq-server/src/main/java/com/hsms/mqserver/strategy/executors/impl/PullExecutor.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqserver.strategy.executors.impl; 2 | 3 | import com.hsmq.data.HsReq; 4 | import com.hsmq.data.HsResp; 5 | import com.hsmq.data.message.Pull; 6 | import com.hsmq.data.message.PullMessage; 7 | import com.hsmq.data.message.PullMessageResp; 8 | import com.hsmq.enums.OperationEnum; 9 | import com.hsmq.enums.ResultEnum; 10 | import com.hsms.mqserver.data.ConsumerQueueManger; 11 | import com.hsms.mqserver.data.TopicListener; 12 | import com.hsms.mqserver.strategy.executors.BaseExecutor; 13 | import org.slf4j.Logger; 14 | import org.slf4j.LoggerFactory; 15 | 16 | import java.util.Comparator; 17 | import java.util.List; 18 | import java.util.Optional; 19 | 20 | /** 21 | * @author ：河神 22 | * @date ：Created in 2021/6/9 1:50 下午 23 | */ 24 | public class PullExecutor extends BaseExecutor { 25 | 26 | final static Logger log = LoggerFactory.getLogger(PullExecutor.class); 27 | 28 | @Override 29 | public HsResp executor(HsReq hsReq) { 30 | 31 | Pull pull = hsReq.getData(); 32 | 33 | TopicListener topicListener = ConsumerQueueManger.getTopicListener(pull.getTopic()); 34 | if (topicListener==null){ 35 | return HsResp.typeError(); 36 | } 37 | 38 | List pullMessages = topicListener.pullMessage(pull); 39 | PullMessageResp pullMessageResp = new PullMessageResp(); 40 | pullMessageResp.setPullMessages(pullMessages); 41 | pullMessageResp.setTopic(pull.getTopic()); 42 | pullMessageResp.setQueueId(pull.getQueueId()); 43 | pullMessageResp.setConsumerGroup(pull.getConsumerGroup()); 44 | if (pullMessages!=null){ 45 | Optional first = pullMessages.stream().map(PullMessage::getIndex).max(Comparator.comparing(Long::longValue)); 46 | first.ifPresent(pullMessageResp::setLastIndex); 47 | } 48 | 49 | HsResp resp = new HsResp<>(); 50 | resp.setData(pullMessageResp); 51 | resp.setOperation(OperationEnum.Resp.getOperation()); 52 | resp.setResult(ResultEnum.SendOK.getCode()); 53 | return resp; 54 | } 55 | 56 | @Override 57 | public HsReq convertReq(HsReq hsReq) { 58 | if (hsReq.getData() instanceof Pull){ 59 | HsReq data = new HsReq<>(); 60 | data.setData((Pull)hsReq.getData()); 61 | data.setOperation(hsReq.getOperation()); 62 | return data; 63 | } 64 | return null; 65 | } 66 | } 67 | -------------------------------------------------------------------------------- /mq-server/src/main/java/com/hsms/mqserver/strategy/executors/impl/SendMessageExecutor.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqserver.strategy.executors.impl; 2 | 3 | import com.hsmq.common.utils.MessageIdUtils; 4 | import com.hsmq.data.HsReq; 5 | import com.hsmq.data.HsResp; 6 | import com.hsmq.data.message.SendMessage; 7 | import com.hsms.mqserver.strategy.executors.BaseExecutor; 8 | import org.slf4j.Logger; 9 | import org.slf4j.LoggerFactory; 10 | 11 | /** 12 | * @author ：河神 13 | * @date ：Created in 2021/6/9 1:50 下午 14 | */ 15 | public class SendMessageExecutor extends BaseExecutor { 16 | 17 | Logger logger = LoggerFactory.getLogger(PullExecutor.class); 18 | 19 | @Override 20 | public HsResp executor(HsReq hsReq) { 21 | SendMessage sendMessage = hsReq.getData(); 22 | sendMessage.setMsgId(MessageIdUtils.newMsgId(sendMessage.getTopic())); 23 | HsResp hsResp = messageStore.saveMessage(sendMessage); 24 | hsResp.setReqId(hsReq.getReqId()); 25 | return hsResp; 26 | } 27 | 28 | @Override 29 | public HsReq convertReq(HsReq hsReq) { 30 | if (hsReq.getData() instanceof SendMessage){ 31 | HsReq data = new HsReq<>(); 32 | data.setData((SendMessage)hsReq.getData()); 33 | data.setOperation(hsReq.getOperation()); 34 | return data; 35 | } 36 | return null; 37 | } 38 | 39 | } 40 | -------------------------------------------------------------------------------- /mq-server/src/main/java/com/hsms/mqserver/strategy/executors/impl/TopicDataExecutor.java: -------------------------------------------------------------------------------- 1 | package com.hsms.mqserver.strategy.executors.impl; 2 | 3 | import com.hsmq.data.HsReq; 4 | import com.hsmq.data.HsResp; 5 | import com.hsmq.data.message.MessageQueueData; 6 | import com.hsmq.data.message.TopicData; 7 | import com.hsmq.enums.OperationEnum; 8 | import com.hsmq.enums.ResultEnum; 9 | import com.hsmq.storage.data.QueueOffsetStorage; 10 | import com.hsms.mqserver.data.ConsumerQueueManger; 11 | import com.hsms.mqserver.data.TopicListener; 12 | import com.hsms.mqserver.strategy.executors.BaseExecutor; 13 | import org.slf4j.Logger; 14 | import org.slf4j.LoggerFactory; 15 | 16 | /** 17 | * @author ：河神 18 | * @date ：Created in 2021/10/2 3:54 下午 19 | */ 20 | public class TopicDataExecutor extends BaseExecutor { 21 | 22 | final static Logger log = LoggerFactory.getLogger(TopicDataExecutor.class); 23 | 24 | 25 | @Override 26 | public HsResp executor(HsReq hsReq) { 27 | 28 | TopicData data = hsReq.getData(); 29 | MessageQueueData messageQueueData = new MessageQueueData(); 30 | 31 | TopicListener topicListener = ConsumerQueueManger.getTopicListener(data.getTopic()); 32 | 33 | messageQueueData.setTopic(data.getTopic()); 34 | messageQueueData.setConsumerKey(data.getConsumerKey()); 35 | messageQueueData.setQueueSize(topicListener.getQueueSize()); 36 | messageQueueData.setOffSetMap(QueueOffsetStorage.getOffSetMap(data.getTopic(),data.getConsumerGroup())); 37 | 38 | HsResp resp = new HsResp<>(); 39 | resp.setData(messageQueueData); 40 | resp.setOperation(OperationEnum.Resp.getOperation()); 41 | resp.setResult(ResultEnum.SendOK.getCode()); 42 | return resp; 43 | 44 | } 45 | 46 | @Override 47 | public HsReq convertReq(HsReq hsReq) { 48 | if (hsReq.getData() instanceof TopicData){ 49 | HsReq data = new HsReq<>(); 50 | data.setData((TopicData)hsReq.getData()); 51 | data.setOperation(hsReq.getOperation()); 52 | return data; 53 | } 54 | return null; 55 | } 56 | } 57 | -------------------------------------------------------------------------------- /mq-server/src/main/resources/logback.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | %d{yyy-MM-dd HH:mm:ss,GMT+8}[%p][%t][%file:%M %line][%X] - %m%n 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | -------------------------------------------------------------------------------- /pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 5 | 4.0.0 6 | 7 | com.heshen 8 | hsmq 9 | 1.0-SNAPSHOT 10 | 11 | consumer 12 | producer 13 | io 14 | mqserver 15 | common 16 | mq-client 17 | 18 | pom 19 | 20 | 21 | 1.8 22 | 23 | 24 | 25 | 26 | 27 | 28 | com.heshen 29 | common 30 | ${project.version} 31 | 32 | 33 | 34 | com.heshen 35 | protocol 36 | ${project.version} 37 | 38 | 39 | 40 | 41 | com.heshen 42 | storage 43 | ${project.version} 44 | 45 | 46 | 47 | 48 | io.netty 49 | netty-all 50 | 4.1.59.Final 51 | 52 | 53 | 54 | com.heshen 55 | protocol 56 | 1.0-SNAPSHOT 57 | 58 | 59 | 60 | com.heshen 61 | client 62 | 1.0-SNAPSHOT 63 | 64 | 65 | 66 | commons-logging 67 | commons-logging 68 | 1.2 69 | 70 | 71 | 72 | com.google.guava 73 | guava 74 | 29.0-jre 75 | 76 | 77 | 78 | ch.qos.logback 79 | logback-classic 80 | 1.1.3 81 | compile 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | org.apache.maven.plugins 91 | maven-compiler-plugin 92 | 93 | 8 94 | 8 95 | 96 | 97 | 98 | 99 | 100 | -------------------------------------------------------------------------------- /zimage/consumer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MortyCode/HsMq/95b88d8fe479971c3cadce6b9269bd5eb98bcd4f/zimage/consumer.png -------------------------------------------------------------------------------- /zimage/hs.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MortyCode/HsMq/95b88d8fe479971c3cadce6b9269bd5eb98bcd4f/zimage/hs.jpeg -------------------------------------------------------------------------------- /zimage/hsmq.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MortyCode/HsMq/95b88d8fe479971c3cadce6b9269bd5eb98bcd4f/zimage/hsmq.png -------------------------------------------------------------------------------- /zimage/image.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MortyCode/HsMq/95b88d8fe479971c3cadce6b9269bd5eb98bcd4f/zimage/image.png -------------------------------------------------------------------------------- /zimage/server.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MortyCode/HsMq/95b88d8fe479971c3cadce6b9269bd5eb98bcd4f/zimage/server.png -------------------------------------------------------------------------------- /znote/Lock和Condition.md: -------------------------------------------------------------------------------- 1 | # Lock 和Condition 2 | 3 | ### > Lock 用于解决互斥问题， 4 | - 阻塞队列队列中的节点都想获取锁 5 | ### > Condition 用于解决同步问题。 6 | 7 | - 条件队列则是由condition形成的条件队列，线程被await操作挂起后就会被放入条件队列，这个队列中的节点都被挂起，他们都等待被signal进入阻塞队列再次获取锁 8 | - 当前线程被await后，就会被挂起放入条件队列中 9 | 10 | 11 | 12 | ## Synchronized无法解决的问题,为什么需要Lock？ 13 | 14 | - synchronized 无法中断 15 | - synchronized 无法设置超时 16 | - synchronized 无法非阻塞的获取锁 17 | 18 | 19 | 20 | ### Lock的可见行 21 | 22 | - Lock的可见性通过AQS的 state 变量来实现 23 | - Lock 的加锁和解锁 都会操作 state 变量，而state 变量是有volitile修饰，根据haper-before原则，多个线程之间共享资源是可见的。 24 | 25 | 26 | 27 | ### 可重入锁 28 | 29 | - 同一个线程可以重复获取同一把锁 30 | - 通过AQS的占用的线程id来实现，每次加锁 state +1 ,解锁 -1 31 | 32 | 33 | 34 | ### 公平锁 35 | 36 | - 公平锁，唤醒的策略就是谁等待的时间长，就唤醒谁，很公平； 37 | - 非公平锁，则不提供这个公平保证，有可能等待时间短的线程反而先被唤醒。 38 | - 通过CLH队列实现 39 | 40 | 41 | 42 | ### 读写锁 43 | 44 | - 允许多个读线程同时访问，但是读写互斥，写写互斥 45 | #### 实现原理 46 | 47 | - 将AQS的state 分为了两个部分，高位和低位，高位表示读，低位表示写 48 | 49 | 50 | 51 | - 如果当前没有写锁或读锁时，第一个获取锁的线程都会成功，无论该锁是写锁还是读锁 52 | - 如果当前已经有了读锁，那么这时获取写锁将失败，获取读锁有可能成功也有可能失败 53 | - 如果当前已经有了写锁，那么这时获取读锁或写锁，如果线程相同（可重入），那么成功；否则失败 54 | 55 | 56 | 57 | - **释放锁** 58 | - 如果当前是写锁被占有了，只有当写锁的数据降为0时才认为释放成功；否则失败。因为只要有写锁，那么除了占有写锁的那个线程，其他线程即不可以获得读锁，也不能获得写锁 59 | - 如果当前是读锁被占有了，那么只有在写锁的个数为0时才认为释放成功。因为一旦有写锁，别的任何线程都不应该再获得读锁了，除了获得写锁的那个线程。 60 | 61 | 62 | 63 | 64 | 65 | ## Condition 66 | ### 原理 67 | #### ConditionObject#await 68 | 69 | 1. 将当前线程构造为条件节点加入条件队列尾部 70 | 1. 释放锁 71 | 1. 挂起当前线程并等待进入阻塞队列 72 | 1. 线程被signal唤醒进入阻塞队列 73 | 1. 进入阻塞队列自旋获取锁，其实到这里应该明白了，上一节中所谓的资源就是锁 74 | #### ConditionObject#signal 75 | 76 | 1. 将节点从条件队列转移到阻塞队列后，然后将该节点自旋加入阻塞队列尾部，然后csa替换为状态为signal，然后执行unpark 唤醒线程 77 | 78 | 79 | 80 | #### 81 | #### Condition的方法 82 | | 方法 | | | 83 | | --- | --- | --- | 84 | | await() | 使当前线程加入 await() 等待队列中，并释放当锁，当其他线程调用signal()会重新请求锁。与Object.wait()类似。 | 此时调用thread.interrupt()会报错 | 85 | | awaitUninterruptibly() | 调用该方法的前提是，当前线程已经成功获得与该条件对象绑定的重入锁，否则调用该方法时会抛出IllegalMonitorStateException. 86 | 调用该方法后，结束等待的唯一方法是其它线程调用该条件对象的signal()或signalALL()方法。等待过程中如果当前线程被中断，该方法仍然会继续等待，同时保留该线程的中断状态。 | 调用thread.interrupt()则不会报错 | 87 | | await(long time, 88 | TimeUnit unit) | 与await()基本一致，唯一不同点在于，指定时间之内没有收到signal()或signalALL()信号或者线程中断时该方法会返回false;其它情况返回true | | 89 | | awaitUntil(Date deadline) | 适用条件与行为与awaitNanos(long nanosTimeout)完全一样，唯一不同点在于它不是等待指定时间，而是等待由参数指定的某一时刻。 90 | | | 91 | | signal() | 唤醒一个在 await()等待队列中的线程。与Object.notify()相似 92 | | | 93 | | signalAll() | 唤醒 await()等待队列中所有的线程。与object.notifyAll()相似 94 | | | 95 | 96 | 97 | 98 | ### wait 和 lock.newCondition().await()的不同点 99 | | 对比项 | synchronized | lock的condition | 100 | | --- | --- | --- | 101 | | 前置条件 | 获取对象锁 | 调用lock.lock() 获取锁 102 | 调用lock.newCondition() 获取Condition | 103 | | 等待队列个数 | 一个 | 多个队列， 104 | 可以创建多个等待队列，例如在阻塞队列中，可以创建 队列空condition 和 队列满condition 两个 condition | 105 | | 当前线程释放锁并进入等待队列，在等待过程中不响应中断 | 不支持 | 支持，使用awaitUninterruptibly()方法 | 106 | | 当前线程释放锁并进入等待队列到未来指定时间 | 不支持 | 支持 | 107 | 108 | #### 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | -------------------------------------------------------------------------------- /znote/Netty.md: -------------------------------------------------------------------------------- 1 | # 定义 2 | netty是一个异步的，事件驱动的，用来做高性能高可用的网络框架。 3 | ## 优点 4 | 5 | 1. 框架设计优雅，底层网络模型切换方便 6 | 1. 提供多种标准的协议，安全，编码解码器的支持 7 | 1. 在Dubbo RocketMQ 等等中都有使用 8 | # 高性能的原因 9 | ## 1. 异步非阻塞通信 10 | 11 | - 采用Reactor来设计和实现，可以同时处理多个channel，并且读写都是异步的，可以大幅度的提高io线程的允许效率 12 | ## 2. 零拷贝 13 | ### 1. DirectBuffer 14 | 15 | - Netty的接受和发送使用DirectBuffer，使用堆外内存直接进行进行Socket的读写，不需要字节缓冲区的二次拷贝。 16 | - 传统的堆存储进行Socket读写会将堆内存的数据复制到直接内存之后才可以写入Socket， 17 | - 相比于直接使用直接内存，少一次复制 18 | ### 2. 组合Buffer 19 | 20 | - Netty提供了组合Buffer的功能，可以方便的聚合Buffer,避免了传统内存拷贝的buffer合并方法 21 | ### 3. 文件传输采用transferTo方法 22 | 23 | - transferTo方法将数据从FileChannel对象传送到可写的字节通道 24 | - 由native方法实现，使用系统函数sendfile()调用，实现了数据直接从内核的读缓冲区传输到套接字缓冲区，避免了用户态(User-space) 与内核态(Kernel-space) 之间的数据拷贝。 25 | - 可以避免传统的循环写的内存拷贝问题，例如用户态和内核态的切换，文件过大等等问题 26 | ## 3. 内存池 27 | 28 | - ByteBuf的内存分配使用内存池机制，重复的利用缓冲区的内存，提高性能。 29 | - 在大数据量的情况下，性能于直接内存分配性能有明显提高 30 | ## 4. 多种的Reactor模型 31 | ### - Reactor单线程模型 32 | 33 | - 一个线程同时负责Socket的连接以及任务的分发处理 34 | ### - Reactor多线程模型 35 | 36 | - 一个线程成为Reactor处理器接受请求 37 | - 网络IO的读写由另一个线程池进行负责，由这些线程来负责处理读写，解码编码，发送。 38 | - 一个线程可以负责多个链路，但是一个链路只能由一个线程负责 39 | ### - 主从Reactor多线程模型 40 | 41 | - 主线程池处理客户端的连接请求，只负责客户端的登陆，验证，连接等操作。 42 | - 从线程池负责处理链路的读写，解码编码，发送等后续内容。 43 | # 44 | # Netty的使用 45 | ## 1. Bootstrap 46 | 47 | 48 | 49 | 50 | 51 | 52 | ### 2. ChannelOption 53 | | **option** | **desc** | **value** | 54 | | --- | --- | --- | 55 | | _TCP_NODELAY_ | 启动TCP_NODELAY，就意味着禁用了Nagle算法，允许小包的发送. 56 | Nagle算法通过减少需要传输的数据包，来优化网络。在内核实现中，数据包的发送和接受会先做缓存，分别对应于写缓存和读缓存。 | true/false | 57 | | _SO_KEEPALIVE_ | 该参数用于设置TCP连接，当设置该选项以后，连接会测试链接的状态，这个选项用于可能长时间没有数据交流的连接。 58 | 当设置该选项以后，如果在两小时内没有数据的通信时，TCP会自动发送一个活动探测数据报文。 | true/false | 59 | | _SO_SNDBUF_ | 操作发送缓冲区大小,发缓冲区用于保存发送数据，直到发送成功。 | 不同操作系统不同 | 60 | | _SO_RCVBUF_ | 接受缓冲区大小，接收缓冲区用于保存网络协议站内收到的数据，直到应用程序读取成功 | 不同操作系统不同 | 61 | | _CONNECT_TIMEOUT_MILLIS_ | 连接超时 | | 62 | 63 | ### 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | -------------------------------------------------------------------------------- /znote/Nio文件读写.md: -------------------------------------------------------------------------------- 1 | # NIO文件读写 2 | 3 | > FileChannel是一个用读写，映射和操作一个文件的通道。 4 | >出了读写操作之外，还有裁剪特定大小文件truncate()，强 5 | >制在内存中的数据刷新到硬盘中去force()，对通道上锁lock()等功能。 -------------------------------------------------------------------------------- /znote/线程的生命周期.md: -------------------------------------------------------------------------------- 1 | ## 线程生命周期 2 | ![image.png](../zimage/image.png) 3 | ## 线程的方法 4 | | **方法名** | **方法解释** | **其他** | 5 | | --- | --- | --- | 6 | | Thread.yield() | 让当前运行线程回到可运行状态，允许具有相同优先级的其他线程获得运行机会。但是不一定是别的线程执行，有可能执行yield()方法的线程也可能被调度。 | yield()不会导致线程转到等待/睡眠/阻塞状态。 | 7 | | join() | 阻塞调用此方法的线程, 进入 TIMED_WAITING 状态，直到线程t完成，此线程再继续； 8 | 例如主线程调用自线程的join方法，则主线程的在join之后的内容会在自线程执行完成之后执行 | 底层就是调用wait方法。当join的参数是0的时候，会不停的调用wait方法，即只要子线程是活的，主线程就不停的等待 | 9 | | sleep() | 使当前线程（即调用该方法的线程）暂停执行一段时间，让其他线程有机会继续执行,但它并不释放对象锁。也就是说如果有synchronized同步快，其他线程仍然不能访问共享数据。 10 | | sleep()可以使低优先级的线程得到执行的机会，当然也可以让同优先级、高优先级的线程有执行的机会。 11 | 可以在任何地方调用 | 12 | | **wait()** | 使此对象监视器当前执行的线程处于“等待(阻塞)状态”，“直到其他线程调用此对象的 notify() 方法或 notifyAll() 方法”，当前线程被唤醒(进入“就绪状态”)。 | 13 | 1. 会释放同步锁 14 | 1. 需要在同步代码块调用 15 | | 16 | | **wait(long timeout)** | 使此对象监视器当前执行的线程处于“等待(阻塞)状态”，“直到其他线程调用此对象的 notify() 方法或 notifyAll() 方法，或者超过指定的时间量”，当前线程被唤醒(进入“就绪状态”)。 | 17 | 1. 会释放同步锁 18 | 1. 需要在同步代码块调用 19 | | 20 | | notify() | 唤醒在此对象监视器上等待的单个线程。 | 从对象的等待池中移走一个任意的线程并放到锁标志等待池中，只有锁标志等待池中线程能够获取锁标志；如果锁标志等待池中没有线程，则notify()不起作用 | 21 | | notifyAll() | 唤醒在此对象监视器上等待的所有线程。 | 从对象等待池中移走所有等待那个对象的线程并放到锁标志等待池中。 | 22 | 23 | 24 | 25 | -------------------------------------------------------------------------------- /znote/网络通信原理.md: -------------------------------------------------------------------------------- 1 | 2 | | TCP/IP | TCP/IP | OSI | 内容 | 3 | | :---: | :---: | :---: | :---: | 4 | | 应用层 | 应用层 | 应用层 | HTTP/FTP/POP3 等等应用协议 | 5 | | | | 表示层 | | 6 | | | | 会话层 | | 7 | | 传输层 | 传输层 | 传输层 | TCP协议 | 8 | | 网络层 | 网络层 | 网络层 | IP协议 | 9 | | 网络接口层 | 数据链路层 | 数据链路层 | 将0 1 信号进行组封装位数据包 | 10 | | | 物理层 | 物理层 | 基于电信号发送高低位信号 | 11 | 12 | 13 | 14 | ## 三次握手 15 | 客户端发起请求 16 | 17 | | | client | | server | | 18 | | :---: | :---: | :---: | :---: | :---: | 19 | | syn_send | SYN=1   seq = x | -> | | | 20 | | | | <- | SYN = 1   ACK=1    seq=y ack= x+1 | syn_recv | 21 | | 连接成功 | ACK=1  ack = y+1  | -> | | 连接成功 | 22 | 23 | ## 数据传输 24 | | | 写 | | 读 | | 25 | | :---: | :---: | :---: | :---: | :---: | 26 | | | seq=x+1 ack=y+1 | -> | | | 27 | | | | <- | ack = x+2 | | 28 | 29 | ## 四次挥手 30 | 任意一方发起 31 | 32 | | | client | | server | | 33 | | :---: | :---: | :---: | :---: | :---: | 34 | | fin_wait_1 | FIN=1 seq=u | -> | | | 35 | | fin_wait_2 | | <- | ACK=1 ack=u+1  seq=v | close_wait | 36 | | time_walt | | <- | FIN =1 ACK=1 seq =w ack=u+1 | last_ack | 37 | | time_walt | ACK=1 seq=u+1 ack=w+1 | -> | | close() | 38 | 39 | 40 | 41 | 42 | 43 | 1. 客户端发起中断请求，FIN 请求，seq为(前面传送过来的最后一个数据字节+1)，意味着客户端没有数据发送了 44 | 1. 此时返回一个ACK确认，表示收到了，但是服务端如果有数据需要发送，还是可以继续发送，客户端也需要接受。并且服务端进行等待关闭。客户端接口之后，进入终止等待2状态 45 | 1. 当服务端的数据发送完成之后，服务端 发送FIN 请求，服务端进入最后确认状态 46 | 1. 当客户端收到最后确认之后，发送确认消息，服务端收到后立即 进入CLOSE 状态。但是客户端此时还没有关闭，必须等待（2MSL）2倍最长报文寿命的时候，才能进入CLOSE状态 ，（可以设置不等待，直接关闭） 47 | 48 | 49 | 50 | 51 | ## IO是什么 52 | 53 | 1. 对于Unix来说，一切皆文件，Socket,管道，FIFO,终端，都是文件，操作这些文件，就是IO操作 54 | 1. 文件描述符（FD）,文件描述符标识 文件流，让操作系统可以找到需要操作的流。 55 | ## 用户空间/内核空间 56 | 57 | 1. 内核空间存放的是内核代码 和 数据 58 | 1. 权限为0级 59 | 2. 用户空间存放的是用户代码 和 数据 60 | 1. 权限为3级 61 | 62 | 63 | 64 | ### 一般的网络操作为： 65 | 66 | 1. 等待数据到达网卡，然后将数据复制到内核缓冲区 67 | 1. 从内核缓冲区复制数据，然后拷贝到用户空间 68 | 69 | 70 | 71 | ## IO模型 72 | 73 | - IO模型一般分为两个步骤，1 等待 2.数据复制 74 | | IO模型 | 特点 | 优点 | 缺点 | 75 | | --- | --- | --- | --- | 76 | | 阻塞IO模型 | 在IO执行的两个阶段，都会被阻塞 | 1.阻塞进程不占用CPU 77 | 2.响应即时 | 1.需要为每一个请求分配一个线程，系统开销大，不适合大并发 | 78 | | 非阻塞IO模型 | 第一阶段不阻塞，由阻塞变为不断的轮询 79 | 第二阶段阻塞 | | 1.需要不断轮询，CPU消耗大，适合 80 | 2.需要为每一个请求分配一个线程，系统开销大，不适合大并发 81 | 3.响应不及时 | 82 | | 多路复用IO模型 | 第一阶段，多个进程的I/O注册到一个复用器（Selector）上面，当没有数据的时候Selector阻塞，当有数据的时候，会返回数据。 83 | 84 | 第二阶段阻塞 | 1.多个请求可以由一个线程进行管理，节省CPU 和 系统消耗。 85 | 2.适合高并发应用，性能好 | 开发难度大 | 86 | | 信号驱动IO模型 | 第一阶段由系统通过注册的信号来通知用户进程，是非阻塞的 87 | 88 | 第二阶段阻塞 | 应用场景少 | | 89 | | 异步IO模型 | 第一阶段 第二阶段 都不阻塞，一步完成后通知通知应用程序 | JAVA的aio就是这样 | | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | --------------------------------------------------------------------------------