├── .gitignore ├── LICENSE ├── README.md ├── docs ├── cn │ ├── advance.md │ ├── arch.md │ ├── consumer.md │ ├── design.md │ ├── install.md │ ├── message.md │ ├── producer.md │ ├── query.md │ ├── quickstart.md │ └── table.md └── images │ ├── arch.png │ ├── design.png │ ├── param1.png │ ├── param2.png │ ├── param3.png │ └── qq.png ├── pom.xml └── src └── main ├── java └── com │ └── qunar │ └── cm │ └── ic │ ├── Application.java │ ├── common │ └── exception │ │ ├── ExceptionEnum.java │ │ └── ICException.java │ ├── controller │ ├── AbstractController.java │ ├── EventController.java │ ├── ListenerController.java │ └── TypeController.java │ ├── dao │ ├── EventRepository.java │ ├── ListenerRepository.java │ ├── ProducerRepository.java │ ├── PropertyRepository.java │ ├── TypeRepository.java │ ├── converter │ │ ├── EventReadConverter.java │ │ ├── EventWriteConverter.java │ │ └── package-info.java │ └── page │ │ ├── FirstEventPage.java │ │ └── package-info.java │ ├── dto │ ├── EventResult.java │ ├── EventSaveResult.java │ ├── IpMatcher.java │ ├── ListenerFetchResult.java │ └── MessageResponse.java │ ├── model │ ├── Event.java │ ├── IdentityCounter.java │ ├── Listener.java │ ├── Producer.java │ ├── Property.java │ ├── Type.java │ └── jackson │ │ ├── EventDeserializer.java │ │ └── EventSerializer.java │ ├── package-info.java │ └── service │ ├── EventConsumerService.java │ ├── EventService.java │ ├── ListenerService.java │ ├── ProducerService.java │ ├── PropertyService.java │ ├── TypeService.java │ └── impl │ ├── EventConsumerServiceImpl.java │ ├── EventServiceImpl.java │ ├── ListenerServiceImpl.java │ ├── ProducerServiceImpl.java │ ├── PropertyServiceImpl.java │ └── TypeServiceImpl.java ├── resources ├── application.properties ├── base-type.json ├── logback.xml ├── mongodb.properties └── spring-mongodb.xml └── webapp ├── WEB-INF └── web.xml └── index.html /.gitignore: -------------------------------------------------------------------------------- 1 | ### Java template 2 | *.class 3 | 4 | # Mobile Tools for Java (J2ME) 5 | .mtj.tmp/ 6 | 7 | # Package Files # 8 | *.jar 9 | *.war 10 | *.ear 11 | 12 | # virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml 13 | hs_err_pid* 14 | ### Maven template 15 | /*.iml 16 | /.idea 17 | target/ 18 | pom.xml.tag 19 | pom.xml.releaseBackup 20 | pom.xml.versionsBackup 21 | pom.xml.next 22 | release.properties 23 | dependency-reduced-pom.xml 24 | buildNumber.properties 25 | .mvn/timing.properties 26 | /.sonar 27 | 28 | src/main/webapp/components 29 | src/test -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Qunar, Inc. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # IC 2 | 3 | IC是去哪儿公司内部CI、CD以及devops体系建设过程中使用的消息系统和数据中心。由于其基于HTTP协议的特性，具有跨平台、跨语言的优点。而devops体系搭建中，会引入各种开源工具，这些工具的语言差异也很大。基于IC，我们不仅快速实现了流程自动化，而且系统解耦，自动化进程大大提高。 4 | 5 | ## IC主要提供以下功能： 6 | * 消息接收 7 | * 消息监听（轮询方式实时返回，长轮询方式可能延时2s，可自行修改） 8 | * 历史消息按时间段查询 9 | * 历史消息按id查询 10 | 11 | ## 依赖 12 | * MongoDB 3.2.8 13 | * JDK 8 14 | * Tomcat 8 15 | 16 | 17 | 18 | 19 | 20 | ## 文档 21 | * [快速开始](docs/cn/quickstart.md) 22 | * [安装](docs/cn/install.md) 23 | * [设计背景](docs/cn/design.md) 24 | * [架构概览](docs/cn/arch.md) 25 | * [表结构介绍](docs/cn/table.md) 26 | * [配置消息](docs/cn/message.md) 27 | * [发送消息](docs/cn/producer.md) 28 | * [消费消息](docs/cn/consumer.md) 29 | * [查询消息](docs/cn/query.md) 30 | * [进阶说明](docs/cn/advance.md) 31 | 32 | 33 | ## 技术支持 34 | 35 | ### QQ群 36 | ![QQ](docs/images/qq.png) 37 | 38 | -------------------------------------------------------------------------------- /docs/cn/advance.md: -------------------------------------------------------------------------------- 1 | [上一页](query.md) 2 | [回目录](../../README.md) 3 | 4 | 5 | # 进阶说明 6 | 7 | ### consumer使用长轮询方式监听消息的实时性改进 8 | 在实际实际使用时，我们在去哪儿内部是集成了QMQ的，每个消息在可以被消费时会发出一个QMQ消息，所以在consumer发起长轮询时，IC随时监听QMQ消息，当有新消息时会立马返回给consumer，提高实时性。但在开源版本中为了提高IC服务部署的简洁性，去掉了对QMQ的依赖，当consumer发起长轮询时返回新消息可能会有两秒的延时。 9 | 10 | ### 新增消息类型和新增生产者的说明 11 | 在开源版本中，我们没有给管理员提供配置界面，都是直接操作的数据库，在使用上便捷性可能不是很友好，因为在去哪儿内部，我们开发了另一个系统最为管理员操作页面，这个系统直接操作IC的mongodb数据库，没有和IC集成到一起。 12 | 下图是消息从产生到消费的时序图: 13 | 14 | 15 | 16 | [上一页](query.md) 17 | [回目录](../../README.md) -------------------------------------------------------------------------------- /docs/cn/arch.md: -------------------------------------------------------------------------------- 1 | [上一页](design.md) 2 | [回目录](../../README.md) 3 | [下一页](table.md) 4 | 5 | 6 | # 架构概览 7 | 下图是消息从产生到消费的时序图: 8 | 9 | * IC 本服务 10 | * MongoDB 存储消息的数据库 11 | * producer 消息生产者 12 | * consumer 消息消费者 13 | 14 | ![时序图](../images/arch.png) 15 | 16 | 根据图中的编号描述一下其交互过程 17 | 1. producer向IC推送消息 18 | 2. IC将消息内容进行处理，并添加一些固定字段，其中将_hidden设置为true，即此时该消息不会被消费者消费 19 | 3. 另一个线程将连续的待被消费的消息的_hidden字段更新成false，从而保证消息会按顺序被消费，且不会丢失消息 20 | 4. 更新数据库中last_event_id，即最后一个可被消费的消息id 21 | 5. consumer轮询/长轮询监听消息 22 | 6. IC查询数据库获取新消息 23 | 7. IC返回新消息给consumer 24 | 25 | 26 | [上一页](design.md) 27 | [回目录](../../README.md) 28 | [下一页](table.md) 29 | -------------------------------------------------------------------------------- /docs/cn/consumer.md: -------------------------------------------------------------------------------- 1 | [上一页](producer.md) 2 | [回目录](../../README.md) 3 | [下一页](query.md) 4 | 5 | 6 | # 消费消息(consumer) 7 | 8 | ### 发送GET请求监听消息 9 | 10 | GET /api/v2/event/listener/{token}?type=student-chooseClasses&longPoll=true 11 | 12 | 请求参数说明 13 | 14 | ![请求参数](../images/param1.png) 15 | 16 | ps：新的消费者token可以自定义传入，只要和其他消费者的token不重复就可以 17 | 18 | 返回值 19 | ![返回参数](../images/param2.png) 20 | 21 | ```json 22 | { 23 | "code": "cd007e2e-fda5-4508-b4ca-588d326385ab", 24 | "list": [ 25 | { 26 | "id": 7683203, 27 | "type": "student-chooseClasses", 28 | "event": "student-chooseClasses", 29 | "operator": "san.zhang", 30 | "source": "courseSelectionSystem", 31 | "time": "2017-12-06T18:15:33+08:00", 32 | "timestamp": 1512555333139, 33 | "updated": "2018-12-17T19:59:27+08:00", 34 | "updatedTimestamp": 1545047967933, 35 | "userName": "san.zhang", 36 | "classes": [ 37 | "math", 38 | "english", 39 | "chinese" 40 | ], 41 | "age": 14, 42 | "sex": "male", 43 | "hobby": "basketball" 44 | } 45 | ], 46 | "maxResults": 100, 47 | "hasMore": false 48 | } 49 | ``` 50 | 51 | ## 消费成功发ACK确认 52 | POST /api/v2/event/listener/{token}/{code} 53 | 54 | * token是监听时传入的token 55 | * code是监听时返回的code 56 | 57 | 返回值 58 | 59 | ```json 60 | { 61 | "message": "success" 62 | } 63 | ``` 64 | 65 | [上一页](producer.md) 66 | [回目录](../../README.md) 67 | [下一页](query.md) 68 | -------------------------------------------------------------------------------- /docs/cn/design.md: -------------------------------------------------------------------------------- 1 | [上一页](install.md) 2 | [回目录](../../README.md) 3 | [下一页](arch.md) 4 | 5 | # 设计背景 6 | 7 | ### 真实使用场景 8 | 在我们工程效率团队，有大量的CI、CD、DevOps工具，如何能够实现工具解耦，且在短期实现自动化？其实只需要一个消息总线。 9 | 10 | ### 消息总线 11 | 消息总线，简单理解就是一个消息中心，众多微服务实例可以连接到总线上，实例可以往消息中心发送或接收消息（通过监听）。比如：实例1发送一条消息到总线上，总线上的实例2可以接收到消息（实例2监听了实例1发送的消息类型），这样的话，消息总线就充当一个中间者的角色，使得实例1和实例2解偶了，很方便。 12 | 13 | ![design](../images/design.png) 14 | 15 | 16 | ### 不直接使用现有的消息中间件原因 17 | 去哪儿内部已经存在了工程师广泛使用的消息中间件QMQ，java的工程能够很方便地通过引入jar包和注解的方式发送和接收QMQ消息，但是对于公司内部相对小众语言，比如：python、nodejs等开发人员来说，享受不到QMQ带来的便利，需要一个跨语言跨平台的消息系统。IC由此诞生，它不仅仅解决了跨语言的消息传递问题，还是一个大数据中心。而且在去哪儿内部和QMQ相结合，使python系统和java系统都能够便利地发送和接收消息（主要是python系统发送消息，java系统接收消息）。 18 | 19 | 20 | [上一页](install.md) 21 | [回目录](../../README.md) 22 | [下一页](arch.md) 23 | -------------------------------------------------------------------------------- /docs/cn/install.md: -------------------------------------------------------------------------------- 1 | [上一页](quickstart.md) 2 | [回目录](../../README.md) 3 | [下一页](design.md) 4 | 5 | # 安装 6 | 7 | ### 初始化数据库 8 | 创建mongo数据库 9 | ``` 10 | > use dc 11 | switched to db dc 12 | ``` 13 | 14 | 创建mongo数据库用户 15 | ``` 16 | > db.createUser({user:'username',pwd:'pwd',roles:['readWrite']}) 17 | Successfully added user: { "user" : "username", "roles" : [ "readWrite" ] } 18 | ``` 19 | 创建collection 20 | ``` 21 | > db.createCollection("eventinfos") 22 | { "ok" : 1 } 23 | > db.createCollection("identitycounters") 24 | { "ok" : 1 } 25 | > db.createCollection("listenerinfos") 26 | { "ok" : 1 } 27 | > db.createCollection("producerinfos") 28 | { "ok" : 1 } 29 | > db.createCollection("typeinfos") 30 | { "ok" : 1 } 31 | ``` 32 | ### 初始化数据 33 | ``` 34 | db.identitycounters.insert({ 35 | "model" : "EventInfo", 36 | "field" : "id", 37 | "count" : 0.0, 38 | "__v" : 0 39 | }) 40 | 41 | db.propertyinfos.insert({ 42 | "key" : "cache.schema", 43 | "version" : 0 44 | }) 45 | 46 | db.propertyinfos.insert({ 47 | "key" : "cache.producer", 48 | "version" : 0 49 | }) 50 | 51 | ``` 52 | 53 | ### 配置文件 54 | *mongodb.properties* 55 | ``` 56 | # 必填，mongo数据库连接地址 57 | mongo.hosts=
: 58 | # 必填，mongo数据库用户名 59 | mongo.username=userName 60 | # 必填，mongo数据库密码 61 | mongo.password=pwd 62 | # 必填，mongo数据库 63 | mongo.database=dc 64 | ``` 65 | 66 | ## 部署启动 67 | 进入ic_task目录，执行打包命令 68 | >mvn package -Dmaven.test.skip=true 69 | 70 | 进入ic_task/target，复制 ic_task.war 到%TOMCAT_HONE%/webapps下 71 | 72 | 进入%TOMCAT_HOME%/conf目录下，编辑server.xml,在host中添加 73 | > \ 74 | 75 | 进入%TOMCAT_HOME%/bin 目录下，运行startup.sh(startup.bat) 启动tomcat服务 76 | 77 | 进入%TOMCAT_HOME%/logs 目录下,查看catalina.out文件即可查看日志 78 | 79 | 80 | [上一页](quickstart.md) 81 | [回目录](../../README.md) 82 | [下一页](design.md) 83 | -------------------------------------------------------------------------------- /docs/cn/message.md: -------------------------------------------------------------------------------- 1 | [上一页](table.md) 2 | [回目录](../../README.md) 3 | [下一页](producer.md) 4 | 5 | # 配置消息 6 | 7 | ### 新增消息类型 8 | 在typeinfos表中添加一条记录，表示新增一种消息类型叫"student-chooseClasses"，表示学生选择了某些课程 9 | ``` 10 | > db.typeinfos.insert({ 11 | "name" : "student-chooseClasses", 12 | "detail" : "学生选课", 13 | "properties" : { 14 | "userName" : { 15 | "type" : "string" 16 | }, 17 | "classes" : { 18 | "type" : "array" 19 | }, 20 | "age" : { 21 | "type" : "integer" 22 | }, 23 | "sex" : { 24 | "enum" : [ 25 | "male", 26 | "female" 27 | ], 28 | "type" : "string" 29 | }, 30 | "hobby" : { 31 | "type" : "string" 32 | }, 33 | }, 34 | "required" : [ 35 | "userName", 36 | "classes", 37 | "age", 38 | "sex" 39 | ] 40 | }) 41 | 42 | ``` 43 | 44 | ### 配置消息来源 ip白名单 45 | 在typeinfos表中添加一条记录，表示ip 属于courseSelectionSystem，如果配置的是".*"，表示任何ip都可能是courseSelectionSystem的机器（用于测试环境） 46 | ps：当IC接收到一条消息时，会根据消息中的source值进行ip校验，可以避免producer乱发消息 47 | ``` 48 | > db.producerinfos.insert({ 49 | "name" : "courseSelectionSystem", 50 | "ips" : [ 51 | ".*" 52 | ], 53 | "detail" : "选课系统" 54 | }) 55 | 56 | ``` 57 | 58 | ### 更新消息类型配置版本,使消息类型的修改在IC服务中生效 59 | 将propertyinfos表中key="cache.schema"的记录的version+1 60 | ``` 61 | db.getCollection('propertyinfos').update({'key': "cache.schema"}, {'$inc': {'version': 1}}) 62 | ``` 63 | 64 | 65 | ### 更新producer版本，使producer的修改在IC服务中生效 66 | 将propertyinfos表中key="cache.producer"的记录的version+1 67 | ``` 68 | db.getCollection('propertyinfos').update({'key': "cache.producer"}, {'$inc': {'version': 1}}) 69 | ``` 70 | 71 | 72 | 配置完producer和consumer就可以发送和监听student-chooseClasses消息了 73 | 74 | [上一页](table.md) 75 | [回目录](../../README.md) 76 | [下一页](producer.md) 77 | -------------------------------------------------------------------------------- /docs/cn/producer.md: -------------------------------------------------------------------------------- 1 | [上一页](message.md) 2 | [回目录](../../README.md) 3 | [下一页](consumer.md) 4 | 5 | # 发送消息(producer) 6 | 7 | ### 发送POST请求 8 | 其中source、type、operator、timestamp是所有消息的公共字段，也是producer的必填字段，其他字段会根据消息类型配置表中的配置进行校验（包括必填字段和字段值类型的校验），消息发送成功会收到该消息在数据库中存储的id。 9 | * source表示消息来源 10 | * type表示消息类型，值必须已在消息配置表（typeinfos）中添加的 11 | * operator表示操作人 12 | * timstamp表示时间戳 13 | 14 | POST /api/v2/event 15 | 16 | *HEADER* 17 | ``` 18 | Content-Type: application/json 19 | ``` 20 | 21 | *请求参数* 22 | ```json 23 | { 24 | "userName": "san.zhang", 25 | "classes": [ 26 | "math", 27 | "english", 28 | "chinese" 29 | ], 30 | "age": 14, 31 | "sex": "male", 32 | "hobby": "basketball", 33 | "source": "courseSelectionSystem", 34 | "type": "student-chooseClasses", 35 | "operator": "san.zhang", 36 | "timestamp": 1512555333139 37 | } 38 | ``` 39 | *发送成功返回值* 40 | ```json 41 | { 42 | "message": "success", 43 | "event": { 44 | "id": 7576300 45 | } 46 | } 47 | 48 | ``` 49 | 50 | *发送失败返回值* 51 | 52 | 当producer没有传userName字段时表示userName是必填字段 53 | ```json 54 | { 55 | "message": "com.github.fge.jsonschema.core.report.ListProcessingReport: failure\n--- BEGIN MESSAGES ---\nerror: object has missing required properties ([\"userName\"])\n level: \"error\"\n schema: {\"loadingURI\":\"#\",\"pointer\":\"\"}\n instance: {\"pointer\":\"\"}\n domain: \"validation\"\n keyword: \"required\"\n required: [\"userName\",\"operator\",\"repo\",\"source\",\"type\"]\n missing: [\"userName\"]\n--- END MESSAGES ---\n" 56 | } 57 | ``` 58 | 59 | [上一页](message.md) 60 | [回目录](../../README.md) 61 | [下一页](consumer.md) 62 | -------------------------------------------------------------------------------- /docs/cn/query.md: -------------------------------------------------------------------------------- 1 | [上一页](consumer.md) 2 | [回目录](../../README.md) 3 | [下一页](advance.md) 4 | 5 | # 查询消息 6 | 7 | ## 根据时间范围查询 8 | GET /api/v2/event?type=student-chooseClasses&from=2018-01-30T17:31:51%2b08:00&to=2018-01-31T17:31:51%2b08:00 9 | 10 | 请求参数说明 11 | ![请求参数](../images/param3.png) 12 | 13 | 成功返回值 14 | 状态码 200 15 | 16 | ```json 17 | [ 18 | { 19 | "id": 7683203, 20 | "type": "student-chooseClasses", 21 | "event": "student-chooseClasses", 22 | "operator": "san.zhang", 23 | "source": "courseSelectionSystem", 24 | "time": "2017-12-06T18:15:33+08:00", 25 | "timestamp": 1512555333139, 26 | "updated": "2018-12-17T19:59:27+08:00", 27 | "updatedTimestamp": 1545047967933, 28 | "userName": "san.zhang", 29 | "classes": [ 30 | "math", 31 | "english", 32 | "chinese" 33 | ], 34 | "age": 14, 35 | "sex": "male", 36 | "hobby": "basketball" 37 | } 38 | ] 39 | 40 | ``` 41 | 失败返回值 42 | 状态码 400 43 | ```json 44 | { 45 | "message": "查询时间范围不能超过一天" 46 | } 47 | ``` 48 | 49 | ## 根据id查询事件 50 | GET /api/v2/event/{id} 51 | 52 | 成功返回值 53 | 状态码 200 54 | ```json 55 | { 56 | "id": 7683203, 57 | "type": "student-chooseClasses", 58 | "event": "student-chooseClasses", 59 | "operator": "san.zhang", 60 | "source": "courseSelectionSystem", 61 | "time": "2017-12-06T18:15:33+08:00", 62 | "timestamp": 1512555333139, 63 | "updated": "2018-12-17T19:59:27+08:00", 64 | "updatedTimestamp": 1545047967933, 65 | "userName": "san.zhang", 66 | "classes": [ 67 | "math", 68 | "english", 69 | "chinese" 70 | ], 71 | "age": 14, 72 | "sex": "male", 73 | "hobby": "basketball" 74 | } 75 | 76 | ``` 77 | 78 | 失败返回值 79 | 状态码 400 80 | ```json 81 | { 82 | "message": "[参数错误]事件166129不存在" 83 | } 84 | ``` 85 | 86 | [上一页](consumer.md) 87 | [回目录](../../README.md) 88 | [下一页](advance.md) -------------------------------------------------------------------------------- /docs/cn/quickstart.md: -------------------------------------------------------------------------------- 1 | [回目录](../../README.md) 2 | [下一页](install.md) 3 | 4 | # 快速入门 5 | 6 | ### 配置可接收的消息 7 | 8 | 参见 [配置消息](docs/cn/message.md) 9 | 10 | ### 发送消息 11 | 12 | ``` 13 | curl -i -X POST \ 14 | -H "Content-Type:application/json" \ 15 | -d \ 16 | '{ 17 | "userName": "san.zhang", 18 | "classes": [ 19 | "math", 20 | "english", 21 | "chinese" 22 | ], 23 | "age": 14, 24 | "sex": "male", 25 | "hobby": "basketball", 26 | "source": "courseSelectionSystem", 27 | "type": "student-chooseClasses", 28 | "operator": "san.zhang", 29 | "timestamp": 1512555333139 30 | }' \ 31 | 'http://ip:port/api/v2/event' 32 | ``` 33 | 34 | ## 消费消息 35 | 36 | ``` 37 | curl -i -X GET \ 38 | 'http://ip:port/api/v2/event/listener/demo2?type=student-chooseClasses' 39 | ``` 40 | 41 | [回目录](../../README.md) 42 | [下一页](install.md) -------------------------------------------------------------------------------- /docs/cn/table.md: -------------------------------------------------------------------------------- 1 | [上一页](arch.md) 2 | [回目录](../../README.md) 3 | [下一页](message.md) 4 | 5 | 6 | # 表结构介绍 7 | 8 | ### eventinfos表 9 | 10 | 用于保存所有的消息 11 | 12 | ### identitycounters表 13 | 14 | 用于保存消息总数 15 | 16 | ### listenerinfos表 17 | 18 | 用于保存consumer信息 19 | 20 | ### producerinfos表 21 | 22 | 用于保存producer信息 23 | 24 | ### typeinfos表 25 | 26 | 用于保存消息类型及格式信息 27 | 28 | ### propertyinfos表 29 | 30 | 用于保存配置版本信息,包括更新或添加producer（即修改producerinfos表），更新或添加消息类型（即修改typeinfos表） 31 | 当改动了producerinfos表时，要将key=cache.producer的version加1 32 | 当改动了typeinfos表时，要将key=cache.schema的version加1 33 | 34 | 35 | [上一页](arch.md) 36 | [回目录](../../README.md) 37 | [下一页](message.md) -------------------------------------------------------------------------------- /docs/images/arch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/qunarcorp/ic/3836c67c7d1cff6d268b000c6fb16a50134d75a2/docs/images/arch.png -------------------------------------------------------------------------------- /docs/images/design.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/qunarcorp/ic/3836c67c7d1cff6d268b000c6fb16a50134d75a2/docs/images/design.png -------------------------------------------------------------------------------- /docs/images/param1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/qunarcorp/ic/3836c67c7d1cff6d268b000c6fb16a50134d75a2/docs/images/param1.png -------------------------------------------------------------------------------- /docs/images/param2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/qunarcorp/ic/3836c67c7d1cff6d268b000c6fb16a50134d75a2/docs/images/param2.png -------------------------------------------------------------------------------- /docs/images/param3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/qunarcorp/ic/3836c67c7d1cff6d268b000c6fb16a50134d75a2/docs/images/param3.png -------------------------------------------------------------------------------- /docs/images/qq.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/qunarcorp/ic/3836c67c7d1cff6d268b000c6fb16a50134d75a2/docs/images/qq.png -------------------------------------------------------------------------------- /pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 4 | 4.0.0 5 | 6 | com.qunar.cm.ic 7 | ic_task 8 | 1.0.0 9 | war 10 | 11 | ic_task 12 | Qmq Connector for IC 13 | 14 | 15 | UTF-8 16 | 17 | 1.8 18 | 1.8 19 | 1.8 20 | 21 | 2.0.4.RELEASE 22 | 5.0.8.RELEASE 23 | 24 | 4.5.2 25 | 4.4.4 26 | 27 | 2.0.1.Final 28 | 6.0.11.Final 29 | 1.4 30 | 31 | 2.9.6 32 | 2.2.10 33 | 34 | 1.2.3 35 | 36 | 37 | 38 | 39 | 40 | 41 | org.springframework.boot 42 | spring-boot-dependencies 43 | ${springboot.version} 44 | pom 45 | import 46 | 47 | 48 | 49 | javax.validation 50 | validation-api 51 | ${validation-api.version} 52 | 53 | 54 | org.hibernate 55 | hibernate-validator 56 | ${hibernate-validator.version} 57 | 58 | 59 | commons-dbcp 60 | commons-dbcp 61 | ${dbcp.version} 62 | 63 | 64 | 65 | com.github.java-json-tools 66 | json-schema-validator 67 | ${json-schema.version} 68 | 69 | 70 | 71 | com.google.code.findbugs 72 | jsr305 73 | 3.0.1 74 | 75 | 76 | 77 | com.googlecode.libphonenumber 78 | libphonenumber 79 | 8.0.0 80 | 81 | 82 | 83 | ch.qos.logback 84 | logback-classic 85 | ${logback.version} 86 | 87 | 88 | ch.qos.logback 89 | logback-core 90 | ${logback.version} 91 | 92 | 93 | 94 | commons-io 95 | commons-io 96 | 2.4 97 | 98 | 99 | 100 | 101 | 102 | org.springframework.boot 103 | spring-boot-starter 104 | 105 | 106 | org.springframework.boot 107 | spring-boot-starter-web 108 | ${springboot.version} 109 | 110 | 111 | org.springframework.boot 112 | spring-boot-starter-tomcat 113 | provided 114 | 115 | 116 | org.springframework.boot 117 | spring-boot-starter-data-mongodb 118 | 119 | 120 | org.slf4j 121 | slf4j-api 122 | 123 | 124 | 125 | 126 | org.springframework.boot 127 | spring-boot-starter-test 128 | test 129 | 130 | 131 | 132 | commons-dbcp 133 | commons-dbcp 134 | 135 | 136 | 137 | mysql 138 | mysql-connector-java 139 | runtime 140 | 141 | 142 | 143 | org.apache.httpcomponents 144 | fluent-hc 145 | 4.5.2 146 | 147 | 148 | commons-logging 149 | commons-logging 150 | 151 | 152 | 153 | 154 | 155 | ch.qos.logback 156 | logback-classic 157 | 158 | 159 | ch.qos.logback 160 | logback-core 161 | 162 | 163 | 164 | javax.validation 165 | validation-api 166 | 167 | 168 | 169 | com.github.java-json-tools 170 | json-schema-validator 171 | 172 | 173 | commons-io 174 | commons-io 175 | 176 | 177 | 178 | ic_task 179 | 180 | 181 | maven-compiler-plugin 182 | 3.5.1 183 | 184 | ${java_source_version} 185 | ${java_target_version} 186 | UTF-8 187 | true 188 | true 189 | 190 | 191 | 192 | 193 | 194 | 195 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/Application.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic; 2 | 3 | import org.springframework.boot.autoconfigure.SpringBootApplication; 4 | import org.springframework.boot.builder.SpringApplicationBuilder; 5 | import org.springframework.boot.web.servlet.support.SpringBootServletInitializer; 6 | import org.springframework.context.annotation.Bean; 7 | import org.springframework.context.annotation.ImportResource; 8 | import org.springframework.context.annotation.PropertySource; 9 | import org.springframework.scheduling.annotation.EnableScheduling; 10 | import org.springframework.scheduling.annotation.SchedulingConfigurer; 11 | import org.springframework.scheduling.config.ScheduledTaskRegistrar; 12 | import org.springframework.validation.beanvalidation.MethodValidationPostProcessor; 13 | 14 | import java.util.concurrent.Executor; 15 | import java.util.concurrent.Executors; 16 | 17 | /** 18 | * Created by yu.qi on 2016/3/10. 19 | */ 20 | 21 | @SpringBootApplication 22 | @EnableScheduling 23 | @PropertySource(value = { 24 | "classpath:mongodb.properties", 25 | }) 26 | @ImportResource(value = { 27 | "classpath:spring-mongodb.xml", 28 | }) 29 | public class Application extends SpringBootServletInitializer implements SchedulingConfigurer { 30 | 31 | @Override 32 | protected SpringApplicationBuilder configure(SpringApplicationBuilder application) { 33 | return application.sources(Application.class); 34 | } 35 | 36 | @Bean 37 | public MethodValidationPostProcessor methodValidationPostProcessor() { 38 | return new MethodValidationPostProcessor(); 39 | } 40 | 41 | @Override 42 | public void configureTasks(ScheduledTaskRegistrar taskRegistrar) { 43 | taskRegistrar.setScheduler(taskExecutor()); 44 | } 45 | 46 | @Bean 47 | public Executor taskExecutor() { 48 | return Executors.newScheduledThreadPool(10); 49 | } 50 | } 51 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/common/exception/ExceptionEnum.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.common.exception; 2 | 3 | /** 4 | * Created by dandan.sha on 2018/08/24. 5 | */ 6 | public enum ExceptionEnum { 7 | 8 | DATA_CONVERTER_ERROR("数据转换失败"), 9 | IP_LIMITED("IP禁止访问"), 10 | PARAMS_INVALID("参数错误"); 11 | 12 | private String message; 13 | 14 | ExceptionEnum(String message) { 15 | this.message = message; 16 | } 17 | 18 | public String getMessage() { 19 | return message; 20 | } 21 | } 22 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/common/exception/ICException.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.common.exception; 2 | 3 | /** 4 | * Created by dandan.sha on 2018/08/24. 5 | */ 6 | public class ICException extends RuntimeException { 7 | 8 | 9 | public ICException(String message) { 10 | super(message); 11 | } 12 | 13 | public ICException(ExceptionEnum exceptionEnum) { 14 | 15 | super(exceptionEnum.getMessage()); 16 | } 17 | 18 | public ICException(ExceptionEnum exceptionEnum, String data) { 19 | 20 | super("[" + exceptionEnum.getMessage() + "]" + data); 21 | } 22 | 23 | public ICException(ExceptionEnum exceptionEnum, Throwable cause) { 24 | 25 | super(exceptionEnum.getMessage(), cause); 26 | } 27 | 28 | public ICException(String message, Throwable cause) { 29 | 30 | super(message, cause); 31 | } 32 | 33 | public ICException(ExceptionEnum exceptionEnum, String message, Throwable cause) { 34 | 35 | super("[" + exceptionEnum.getMessage() + "]" + message, cause); 36 | } 37 | 38 | } 39 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/controller/AbstractController.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.controller; 2 | 3 | import com.google.common.base.Strings; 4 | import com.qunar.cm.ic.common.exception.ICException; 5 | import com.qunar.cm.ic.dto.MessageResponse; 6 | import org.slf4j.Logger; 7 | import org.slf4j.LoggerFactory; 8 | import org.springframework.http.HttpStatus; 9 | import org.springframework.http.ResponseEntity; 10 | import org.springframework.web.bind.annotation.ExceptionHandler; 11 | 12 | import javax.annotation.Resource; 13 | import javax.servlet.http.HttpServletRequest; 14 | import javax.validation.ConstraintViolationException; 15 | 16 | /** 17 | * Created by yu.qi on 2018/08/31. 18 | */ 19 | public abstract class AbstractController { 20 | private static final Logger logger = LoggerFactory.getLogger(AbstractController.class); 21 | 22 | @Resource 23 | private HttpServletRequest request; 24 | 25 | String getClientIp() { 26 | String ip = request.getHeader("X-FORWARDED-FOR"); 27 | if (Strings.isNullOrEmpty(ip)) { 28 | ip = request.getRemoteAddr(); 29 | } 30 | return ip; 31 | } 32 | 33 | /* 异常处理，输出异常信息 */ 34 | @ExceptionHandler(RuntimeException.class) 35 | public ResponseEntity handlerRuntimeException(RuntimeException e) { 36 | logger.error("Controller运行时异常：{}", e.getMessage(), e); 37 | HttpStatus httpStatus = HttpStatus.INTERNAL_SERVER_ERROR; 38 | MessageResponse messageResponse = new MessageResponse("系统异常"); 39 | //使用@Validated注解对参数进行校验的时候，如果失败则会抛出ConstraintViolationException这个异常 40 | if (e instanceof ICException || e instanceof ConstraintViolationException) { 41 | httpStatus = HttpStatus.BAD_REQUEST; 42 | messageResponse = new MessageResponse(e.getMessage()); 43 | } 44 | return new ResponseEntity<>(messageResponse, httpStatus); 45 | } 46 | 47 | } 48 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/controller/EventController.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.controller; 2 | 3 | import com.qunar.cm.ic.dto.EventResult; 4 | import com.qunar.cm.ic.dto.EventSaveResult; 5 | import com.qunar.cm.ic.model.Event; 6 | import com.qunar.cm.ic.service.EventService; 7 | import com.qunar.cm.ic.service.ProducerService; 8 | import org.springframework.http.HttpStatus; 9 | import org.springframework.stereotype.Controller; 10 | import org.springframework.web.bind.annotation.*; 11 | 12 | import javax.annotation.Resource; 13 | import javax.validation.constraints.NotBlank; 14 | import javax.validation.constraints.NotEmpty; 15 | import java.util.List; 16 | 17 | /** 18 | * Created by yu.qi on 2018/08/27. 19 | */ 20 | @Controller 21 | public class EventController extends AbstractController { 22 | @Resource 23 | private EventService eventService; 24 | @Resource 25 | private ProducerService producerService; 26 | 27 | @GetMapping("/api/v2/event/{id}") 28 | @ResponseBody 29 | Event getEvent(@NotBlank @PathVariable Long id) { 30 | return eventService.queryById(id); 31 | } 32 | 33 | @GetMapping("/api/v2/event") 34 | @ResponseBody 35 | List getEventByTimeAndType(@RequestParam String type, @RequestParam String from, @RequestParam(required = false) String to) { 36 | return eventService.queryByTimeAndType(type, from, to); 37 | } 38 | 39 | @PostMapping("/api/v2/event") 40 | @ResponseBody 41 | EventSaveResult addEvent(@RequestBody Event event) { 42 | event.setIp(getClientIp()); 43 | //检查ip是否在白名单中 44 | producerService.checkIp(event.getSource(), getClientIp()); 45 | Event newEvent = eventService.checkAndSaveEvent(event); 46 | return new EventSaveResult(new EventResult(newEvent.getId())); 47 | } 48 | 49 | @ResponseStatus(HttpStatus.ACCEPTED) 50 | @PostMapping("/events/{type}") 51 | @ResponseBody 52 | EventSaveResult oldAddEvent( 53 | @NotEmpty 54 | @PathVariable String type, @RequestBody Event event) { 55 | event.setIp(getClientIp()); 56 | 57 | //成功时返回202而不是200，为了兼容早期的一些生产者而不校验ip 58 | event.setType(type); 59 | event.normalizeBody(); 60 | Event newEvent = eventService.checkAndSaveEvent(event); 61 | return new EventSaveResult(new EventResult(newEvent.getId())); 62 | } 63 | 64 | } 65 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/controller/ListenerController.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.controller; 2 | 3 | import com.google.common.base.Strings; 4 | import com.google.common.collect.Lists; 5 | import com.qunar.cm.ic.dto.ListenerFetchResult; 6 | import com.qunar.cm.ic.dto.MessageResponse; 7 | import com.qunar.cm.ic.service.ListenerService; 8 | import org.springframework.lang.NonNull; 9 | import org.springframework.stereotype.Controller; 10 | import org.springframework.validation.annotation.Validated; 11 | import org.springframework.web.bind.annotation.*; 12 | 13 | import javax.annotation.Resource; 14 | import javax.validation.constraints.Max; 15 | import javax.validation.constraints.Min; 16 | import javax.validation.constraints.NotBlank; 17 | import javax.validation.constraints.NotNull; 18 | import java.util.List; 19 | 20 | /** 21 | * Created by yu.qi on 2018/08/29. 22 | */ 23 | @Controller 24 | @Validated 25 | public class ListenerController extends AbstractController { 26 | 27 | @Resource 28 | private ListenerService listenerService; 29 | 30 | @GetMapping("/api/v2/event/listener/{token}") 31 | @ResponseBody 32 | public ListenerFetchResult getEvents( 33 | @NotBlank 34 | @PathVariable String token, 35 | @RequestParam(required = false) String type, 36 | @NonNull 37 | @RequestParam(required = false, defaultValue = "") List types, 38 | @Max(100) 39 | @Min(1) 40 | @NotNull 41 | @RequestParam(required = false, defaultValue = "100") int maxResults, 42 | @RequestParam(required = false, defaultValue = "false") boolean longPoll) { 43 | List targetTypes = Lists.newArrayList(types); 44 | if(!Strings.isNullOrEmpty(type)) { 45 | targetTypes.add(type); 46 | } 47 | return listenerService.consumeEvents(token, targetTypes, maxResults, longPoll, getClientIp()); 48 | } 49 | 50 | @PostMapping("/api/v2/event/listener/{token}/{code}") 51 | @ResponseBody 52 | public MessageResponse acknowledge( 53 | @NotBlank 54 | @PathVariable String token, 55 | @NotBlank 56 | @PathVariable String code) { 57 | listenerService.acknowledge(token, code, getClientIp()); 58 | return new MessageResponse("success"); 59 | } 60 | } 61 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/controller/TypeController.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.controller; 2 | 3 | import com.qunar.cm.ic.model.Type; 4 | import com.qunar.cm.ic.service.TypeService; 5 | import org.springframework.stereotype.Controller; 6 | import org.springframework.validation.annotation.Validated; 7 | import org.springframework.web.bind.annotation.GetMapping; 8 | import org.springframework.web.bind.annotation.PathVariable; 9 | import org.springframework.web.bind.annotation.ResponseBody; 10 | 11 | import javax.annotation.Resource; 12 | import javax.validation.constraints.NotBlank; 13 | import java.util.List; 14 | 15 | /** 16 | * Created by yu.qi on 2018/09/12. 17 | */ 18 | @Controller 19 | @Validated 20 | public class TypeController extends AbstractController { 21 | @Resource 22 | private TypeService typeService; 23 | 24 | @GetMapping("/api/v2/type") 25 | @ResponseBody 26 | public List getTypes() { 27 | return typeService.allTypes(); 28 | } 29 | 30 | @GetMapping("/api/v2/type/name/{name}") 31 | @ResponseBody 32 | public Type getType(@NotBlank 33 | @PathVariable String name) { 34 | return typeService.getType(name); 35 | } 36 | } 37 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/dao/EventRepository.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.dao; 2 | 3 | import com.qunar.cm.ic.model.Event; 4 | import org.springframework.data.domain.Pageable; 5 | import org.springframework.data.domain.Sort; 6 | import org.springframework.data.mongodb.repository.MongoRepository; 7 | import org.springframework.data.mongodb.repository.Query; 8 | 9 | import java.util.Date; 10 | import java.util.List; 11 | import java.util.Optional; 12 | 13 | /** 14 | * Created by dandan.sha on 2018/08/24. 15 | */ 16 | 17 | public interface EventRepository extends MongoRepository { 18 | Optional findFirstByOrderByIdDesc(); 19 | 20 | @Query("{_hidden: {$ne: true}, _dummy: {$ne: true}, id: {$gt: ?0}, event: {$in: ?1}}") 21 | List consumeEventByTypes(Long id, List types, Pageable pageable); 22 | 23 | @Query("{_hidden: {$ne: true}, _dummy: {$ne: true}, id: {$gt: ?0}}") 24 | List consumeEvent(Long id, Pageable pageable); 25 | 26 | @Query("{id: {$gt: ?0}}") 27 | List findGreaterThanId(Long id, Sort sort); 28 | 29 | @Query("{_hidden: {$ne: true}, _dummy: {$ne: true}, id: ?0}") 30 | Optional findOneById(Long id); 31 | 32 | @Query("{_hidden: {$ne: true}, _dummy: {$ne: true}, event: ?0, time: {$gte: ?1, $lt: ?2}}") 33 | List findByTypeAndTime(String type, Date from, Date to); 34 | } 35 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/dao/ListenerRepository.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.dao; 2 | 3 | import com.qunar.cm.ic.model.Listener; 4 | import org.bson.types.ObjectId; 5 | import org.springframework.data.mongodb.repository.MongoRepository; 6 | 7 | import java.util.Optional; 8 | 9 | /** 10 | * Created by yu.qi on 2018/08/29. 11 | */ 12 | public interface ListenerRepository extends MongoRepository { 13 | Optional findByToken(String token); 14 | } 15 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/dao/ProducerRepository.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.dao; 2 | 3 | import com.qunar.cm.ic.model.Producer; 4 | import org.bson.types.ObjectId; 5 | import org.springframework.data.mongodb.repository.MongoRepository; 6 | 7 | /** 8 | * Created by yu.qi on 2018/09/11. 9 | */ 10 | public interface ProducerRepository extends MongoRepository { 11 | } 12 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/dao/PropertyRepository.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.dao; 2 | 3 | import com.qunar.cm.ic.model.Property; 4 | import org.bson.types.ObjectId; 5 | import org.springframework.data.mongodb.repository.MongoRepository; 6 | 7 | import java.util.Optional; 8 | 9 | /** 10 | * Created by yu.qi on 2018/9/5. 11 | */ 12 | public interface PropertyRepository extends MongoRepository { 13 | Optional findByKey(String key); 14 | } 15 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/dao/TypeRepository.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.dao; 2 | 3 | import com.qunar.cm.ic.model.Type; 4 | import org.springframework.data.mongodb.repository.MongoRepository; 5 | import org.springframework.stereotype.Repository; 6 | 7 | import java.util.Optional; 8 | 9 | /** 10 | * Created by dandan.sha on 2018/08/28. 11 | */ 12 | @Repository 13 | public interface TypeRepository extends MongoRepository { 14 | Optional findOneByName(String name); 15 | } 16 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/dao/converter/EventReadConverter.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.dao.converter; 2 | 3 | import com.google.common.collect.Maps; 4 | import com.qunar.cm.ic.common.exception.ExceptionEnum; 5 | import com.qunar.cm.ic.common.exception.ICException; 6 | import com.qunar.cm.ic.model.Event; 7 | import org.bson.Document; 8 | import org.springframework.core.convert.converter.Converter; 9 | import org.springframework.data.convert.ReadingConverter; 10 | import org.springframework.lang.Nullable; 11 | import org.springframework.stereotype.Component; 12 | 13 | /** 14 | * Created by dandan.sha on 2018/08/24. 15 | */ 16 | 17 | @ReadingConverter 18 | @Component 19 | public class EventReadConverter implements Converter { 20 | @Nullable 21 | @Override 22 | public Event convert(Document source) { 23 | Event event = new Event(); 24 | event.setId(parseId(source)); 25 | event.setTime(source.getDate("time")); 26 | event.setUpdated(source.getDate("updated")); 27 | event.setOperator(source.getString("operator")); 28 | event.setSource(source.getString("source")); 29 | //数据库中存储的字段为event，展示给用户的是type 30 | event.setType(source.getString("event")); 31 | event.setIp(source.getString("_ip")); 32 | //可选字段，旧的数据没有这个字段 33 | event.setHidden(source.get("_hidden", false)); 34 | event.setHidden(source.get("_dummy", false)); 35 | event.setBody(Maps.newLinkedHashMap(source)); 36 | event.normalizeBody(); 37 | return event; 38 | } 39 | 40 | /** 41 | * 解析并将id转换成long类型,同时支持int和long两种格式 42 | */ 43 | private long parseId(Document source) { 44 | Object id = source.get("id"); 45 | if (id instanceof Integer) { 46 | return ((Integer) id).longValue(); 47 | } else if (id instanceof Long) { 48 | return (long) id; 49 | } else { 50 | throw new ICException(ExceptionEnum.DATA_CONVERTER_ERROR, source.toJson()); 51 | } 52 | } 53 | } 54 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/dao/converter/EventWriteConverter.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.dao.converter; 2 | 3 | import com.qunar.cm.ic.model.Event; 4 | import org.bson.Document; 5 | import org.springframework.core.convert.converter.Converter; 6 | import org.springframework.data.convert.WritingConverter; 7 | import org.springframework.lang.Nullable; 8 | import org.springframework.stereotype.Component; 9 | 10 | import java.util.Objects; 11 | 12 | /** 13 | * Created by yu.qi on 2018/08/27. 14 | */ 15 | @WritingConverter 16 | @Component 17 | public class EventWriteConverter implements Converter { 18 | 19 | @Nullable 20 | @Override 21 | public Document convert(Event event) { 22 | if (Objects.equals(event.getDummy(), true)) { 23 | return convertDummy(event); 24 | } 25 | Document document = new Document(event.getBody()); 26 | 27 | document.remove("type"); 28 | document.remove("timestamp"); 29 | document.remove("updatedTimestamp"); 30 | document.entrySet().removeIf(entry -> entry.getKey().startsWith("_")); 31 | 32 | document.put("id", event.getId()); 33 | document.put("operator", event.getOperator()); 34 | document.put("source", event.getSource()); 35 | document.put("time", event.getTime()); 36 | document.put("updated", event.getUpdated()); 37 | document.put("_ip", event.getIp()); 38 | document.put("_hidden", event.getHidden()); 39 | document.put("_dummy", event.getDummy()); 40 | //数据库中存储的字段为event，展示给用户的是type 41 | document.put("event", event.getType()); 42 | return document; 43 | } 44 | 45 | private Document convertDummy(Event event) { 46 | Document document = new Document(); 47 | document.put("_hidden", event.getHidden()); 48 | document.put("_dummy", true); 49 | document.put("id", event.getId()); 50 | return document; 51 | } 52 | } 53 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/dao/converter/package-info.java: -------------------------------------------------------------------------------- 1 | /** 2 | * Created by dandan.sha on 2018/08/24. 3 | */ 4 | @NonNullApi 5 | package com.qunar.cm.ic.dao.converter; 6 | 7 | import org.springframework.lang.NonNullApi; -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/dao/page/FirstEventPage.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.dao.page; 2 | 3 | import org.springframework.data.domain.AbstractPageRequest; 4 | import org.springframework.data.domain.Pageable; 5 | import org.springframework.data.domain.Sort; 6 | 7 | /** 8 | * Created by yu.qi on 2018/08/30. 9 | *

10 | * 这个类的作用主要是为了排序和限制返回的数量，而不是分页 11 | */ 12 | public final class FirstEventPage extends AbstractPageRequest { 13 | 14 | public static FirstEventPage of(int size) { 15 | //只返回第一页 16 | return new FirstEventPage(0, size); 17 | } 18 | 19 | private FirstEventPage(int page, int size) { 20 | super(page, size); 21 | } 22 | 23 | 24 | @Override 25 | public Sort getSort() { 26 | return Sort.by(Sort.Order.asc("id")); 27 | } 28 | 29 | @Override 30 | public Pageable next() { 31 | throw new UnsupportedOperationException(); 32 | } 33 | 34 | @Override 35 | public Pageable previous() { 36 | throw new UnsupportedOperationException(); 37 | } 38 | 39 | @Override 40 | public Pageable first() { 41 | throw new UnsupportedOperationException(); 42 | } 43 | } 44 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/dao/page/package-info.java: -------------------------------------------------------------------------------- 1 | /** 2 | * Created by yu.qi on 2018/08/30. 3 | */ 4 | @NonNullApi 5 | package com.qunar.cm.ic.dao.page; 6 | 7 | import org.springframework.lang.NonNullApi; -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/dto/EventResult.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.dto; 2 | 3 | /** 4 | * Created by dandan.sha on 2018/09/13. 5 | */ 6 | public class EventResult { 7 | private Long id; 8 | 9 | public EventResult(Long id) { 10 | this.id = id; 11 | } 12 | 13 | public Long getId() { 14 | return id; 15 | } 16 | 17 | public void setId(Long id) { 18 | this.id = id; 19 | } 20 | 21 | } 22 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/dto/EventSaveResult.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.dto; 2 | 3 | /** 4 | * Created by dandan.sha on 2018/09/13. 5 | */ 6 | public class EventSaveResult extends MessageResponse { 7 | private EventResult event; 8 | 9 | public EventSaveResult(EventResult event) { 10 | super("success"); 11 | this.event = event; 12 | } 13 | 14 | public EventResult getEvent() { 15 | return event; 16 | } 17 | 18 | public void setEvent(EventResult event) { 19 | this.event = event; 20 | } 21 | } 22 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/dto/IpMatcher.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.dto; 2 | 3 | import com.google.common.collect.Lists; 4 | 5 | import java.util.List; 6 | import java.util.regex.Pattern; 7 | import java.util.stream.Collectors; 8 | 9 | /** 10 | * Created by yu.qi on 2018/09/11. 11 | */ 12 | 13 | public class IpMatcher { 14 | private List ips; 15 | private List patterns; 16 | 17 | public IpMatcher(List ips) { 18 | this.ips = Lists.newArrayList(ips); 19 | this.patterns = ips.stream().map(Pattern::compile).collect(Collectors.toList()); 20 | } 21 | 22 | public boolean match(String ip) { 23 | return ips.contains(ip.trim()) || patterns.stream().anyMatch(pattern -> pattern.matcher(ip).matches()); 24 | } 25 | 26 | public List getIps() { 27 | return ips; 28 | } 29 | } 30 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/dto/ListenerFetchResult.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.dto; 2 | 3 | import com.fasterxml.jackson.annotation.JsonIgnore; 4 | import com.qunar.cm.ic.model.Event; 5 | 6 | import java.util.List; 7 | 8 | /** 9 | * Created by yu.qi on 2018/08/29. 10 | */ 11 | public class ListenerFetchResult { 12 | private String code; 13 | private List list; 14 | private Integer maxResults; 15 | private Boolean hasMore; 16 | @JsonIgnore 17 | private boolean filled; 18 | 19 | public boolean getFilled() { 20 | return filled; 21 | } 22 | 23 | public void setFilled(boolean filled) { 24 | this.filled = filled; 25 | } 26 | 27 | public ListenerFetchResult(String code, Integer maxResults) { 28 | this.code = code; 29 | this.maxResults = maxResults; 30 | } 31 | 32 | public String getCode() { 33 | return code; 34 | } 35 | 36 | 37 | public List getList() { 38 | return list; 39 | } 40 | 41 | public Integer getMaxResults() { 42 | return maxResults; 43 | } 44 | 45 | public Boolean getHasMore() { 46 | return hasMore; 47 | } 48 | 49 | @JsonIgnore 50 | public boolean isEmpty() { 51 | return list == null || list.isEmpty(); 52 | } 53 | 54 | public void setList(List list) { 55 | this.list = list; 56 | } 57 | 58 | public void setCode(String code) { 59 | this.code = code; 60 | } 61 | 62 | public void setHasMore(Boolean hasMore) { 63 | this.hasMore = hasMore; 64 | } 65 | } 66 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/dto/MessageResponse.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.dto; 2 | 3 | /** 4 | * Created by yu.qi on 2018/08/31. 5 | */ 6 | public class MessageResponse { 7 | private String message; 8 | 9 | public MessageResponse(String message) { 10 | this.message = message; 11 | } 12 | 13 | public String getMessage() { 14 | return message; 15 | } 16 | } 17 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/model/Event.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.model; 2 | 3 | import com.fasterxml.jackson.databind.annotation.JsonDeserialize; 4 | import com.fasterxml.jackson.databind.annotation.JsonSerialize; 5 | import com.qunar.cm.ic.model.jackson.EventDeserializer; 6 | import com.qunar.cm.ic.model.jackson.EventSerializer; 7 | import org.springframework.data.mongodb.core.mapping.Document; 8 | import org.springframework.data.mongodb.core.mapping.Field; 9 | 10 | import java.util.Date; 11 | import java.util.Map; 12 | 13 | /** 14 | * Created by dandan.sha on 2018/08/24. 15 | *

16 | * 事件有3个字段在数据库中的名称和类的字段名不同，分别是： 17 | * event->type 18 | * _ip->ip 19 | * _hidden->hidden 20 | * 使用Field注解表明字段在数据库中的类型，但实际上并没有作用，因为在读写数据库时使用的是Converter完成的数据转化 21 | */ 22 | 23 | @Document(collection = "eventinfos") 24 | @JsonSerialize(using = EventSerializer.class) 25 | @JsonDeserialize(using = EventDeserializer.class) 26 | public class Event { 27 | @Field("id") 28 | private Long id; 29 | 30 | @Field("event") 31 | private String type; 32 | private String operator; 33 | //事件来源 34 | private String source; 35 | //事件发生的时间 36 | private Date time; 37 | //事件被添加到IC的时间 38 | private Date updated; 39 | //用户定义的其它的字段，也可以包含上面定义的字段，但如果二者的值不同，则以上面定义的字段值为准 40 | private Map body; 41 | //事件发送者的ip 42 | @Field("_ip") 43 | private String ip; 44 | //是否发布事件 45 | @Field("_hidden") 46 | private Boolean hidden; 47 | //是否是一个假事件，用于占位 48 | @Field("_dummy") 49 | private Boolean dummy; 50 | 51 | public void normalizeBody() { 52 | body.put("timestamp", time.getTime()); 53 | body.put("type", type); 54 | 55 | body.remove("updated"); 56 | body.remove("updatedTimestamp"); 57 | body.remove("id"); 58 | body.remove("time"); 59 | body.remove("event"); 60 | body.entrySet().removeIf(next -> next.getKey().startsWith("_")); 61 | } 62 | 63 | 64 | public Long getId() { 65 | return id; 66 | } 67 | 68 | public void setId(Long id) { 69 | this.id = id; 70 | } 71 | 72 | public Date getTime() { 73 | return time; 74 | } 75 | 76 | public void setTime(Date time) { 77 | this.time = time; 78 | } 79 | 80 | public String getType() { 81 | return type; 82 | } 83 | 84 | public void setType(String type) { 85 | this.type = type; 86 | } 87 | 88 | public Map getBody() { 89 | return body; 90 | } 91 | 92 | public void setBody(Map body) { 93 | this.body = body; 94 | } 95 | 96 | public String getOperator() { 97 | return operator; 98 | } 99 | 100 | public void setOperator(String operator) { 101 | this.operator = operator; 102 | } 103 | 104 | public String getIp() { 105 | return ip; 106 | } 107 | 108 | public void setIp(String ip) { 109 | this.ip = ip; 110 | } 111 | 112 | public Boolean getHidden() { 113 | return hidden; 114 | } 115 | 116 | public void setHidden(Boolean hidden) { 117 | this.hidden = hidden; 118 | } 119 | 120 | public Date getUpdated() { 121 | return updated; 122 | } 123 | 124 | public void setUpdated(Date updated) { 125 | this.updated = updated; 126 | } 127 | 128 | public String getSource() { 129 | return source; 130 | } 131 | 132 | public void setSource(String source) { 133 | this.source = source; 134 | } 135 | 136 | public Boolean getDummy() { 137 | return dummy; 138 | } 139 | 140 | public void setDummy(Boolean dummy) { 141 | this.dummy = dummy; 142 | } 143 | 144 | @Override 145 | public String toString() { 146 | return "Event{" + 147 | "id=" + id + 148 | ", time=" + time + 149 | ", type='" + type + '\'' + 150 | ", body=" + body + 151 | '}'; 152 | } 153 | } 154 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/model/IdentityCounter.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.model; 2 | 3 | import org.springframework.data.mongodb.core.mapping.Document; 4 | 5 | /** 6 | * Created by dandan.sha on 2018/08/30. 7 | */ 8 | @Document(collection = "identitycounters") 9 | 10 | public class IdentityCounter { 11 | private Long count; 12 | 13 | public Long getCount() { 14 | return count; 15 | } 16 | 17 | public void setCount(Long count) { 18 | this.count = count; 19 | } 20 | 21 | @Override 22 | public String toString() { 23 | return "IdentityCounter{" + 24 | "count=" + count + 25 | '}'; 26 | } 27 | } 28 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/model/Listener.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.model; 2 | 3 | import org.bson.types.ObjectId; 4 | import org.springframework.data.mongodb.core.mapping.Document; 5 | 6 | import java.util.Date; 7 | import java.util.List; 8 | 9 | /** 10 | * Created by yu.qi on 2018/08/29. 11 | */ 12 | 13 | @Document(collection = "listenerinfos") 14 | public class Listener { 15 | //这个字段是必须的，在使用save方法时会使用这个字段的值判断是修改还是更新 16 | private ObjectId id; 17 | private String token; 18 | private String code; 19 | //token的描述信息 20 | private String detail; 21 | //true表示是用户直接通过接口访问添加的，false表示是我们提前在数据库中添加的token 22 | private Boolean temporary; 23 | //表示消费时要从这个事件开始，不包括这个事件本身 24 | private Long lastEventId; 25 | //表示上次用户获取到的最后一个事件 26 | private Long recordEventId; 27 | //最后一次consume的时间 28 | private Date readTime; 29 | //最后一次ack的时间 30 | private Date replyTime; 31 | private List ips; 32 | 33 | public String getToken() { 34 | return token; 35 | } 36 | 37 | public void setToken(String token) { 38 | this.token = token; 39 | } 40 | 41 | public String getCode() { 42 | return code; 43 | } 44 | 45 | public void setCode(String code) { 46 | this.code = code; 47 | } 48 | 49 | public String getDetail() { 50 | return detail; 51 | } 52 | 53 | public void setDetail(String detail) { 54 | this.detail = detail; 55 | } 56 | 57 | public Boolean getTemporary() { 58 | return temporary; 59 | } 60 | 61 | public void setTemporary(Boolean temporary) { 62 | this.temporary = temporary; 63 | } 64 | 65 | public Long getLastEventId() { 66 | return lastEventId; 67 | } 68 | 69 | public void setLastEventId(Long lastEventId) { 70 | this.lastEventId = lastEventId; 71 | } 72 | 73 | public Long getRecordEventId() { 74 | return recordEventId; 75 | } 76 | 77 | public void setRecordEventId(Long recordEventId) { 78 | this.recordEventId = recordEventId; 79 | } 80 | 81 | public Date getReadTime() { 82 | return readTime; 83 | } 84 | 85 | public void setReadTime(Date readTime) { 86 | this.readTime = readTime; 87 | } 88 | 89 | public Date getReplyTime() { 90 | return replyTime; 91 | } 92 | 93 | public void setReplyTime(Date replyTime) { 94 | this.replyTime = replyTime; 95 | } 96 | 97 | public ObjectId getId() { 98 | return id; 99 | } 100 | 101 | public void setId(ObjectId id) { 102 | this.id = id; 103 | } 104 | 105 | public List getIps() { 106 | return ips; 107 | } 108 | 109 | public void setIps(List ips) { 110 | this.ips = ips; 111 | } 112 | } 113 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/model/Producer.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.model; 2 | 3 | import org.bson.types.ObjectId; 4 | import org.springframework.data.mongodb.core.mapping.Document; 5 | 6 | import java.util.List; 7 | 8 | /** 9 | * Created by dandan.sha on 2018/09/11. 10 | */ 11 | @Document(collection = "producerinfos") 12 | public class Producer { 13 | 14 | private ObjectId id; 15 | private String name; 16 | private List ips; 17 | private String detail; 18 | 19 | public ObjectId getId() { 20 | return id; 21 | } 22 | 23 | public void setId(ObjectId id) { 24 | this.id = id; 25 | } 26 | 27 | public String getName() { 28 | return name; 29 | } 30 | 31 | public void setName(String name) { 32 | this.name = name; 33 | } 34 | 35 | public List getIps() { 36 | return ips; 37 | } 38 | 39 | public void setIps(List ips) { 40 | this.ips = ips; 41 | } 42 | 43 | public String getDetail() { 44 | return detail; 45 | } 46 | 47 | public void setDetail(String detail) { 48 | this.detail = detail; 49 | } 50 | } 51 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/model/Property.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.model; 2 | 3 | import org.bson.types.ObjectId; 4 | import org.springframework.data.mongodb.core.mapping.Document; 5 | 6 | /** 7 | * Created by yu.qi on 2018/9/5. 8 | */ 9 | @Document(collection = "propertyinfos") 10 | public class Property { 11 | private ObjectId id; 12 | private String key; 13 | private Long version; 14 | 15 | public String getKey() { 16 | return key; 17 | } 18 | 19 | public void setKey(String key) { 20 | this.key = key; 21 | } 22 | 23 | public Long getVersion() { 24 | return version; 25 | } 26 | 27 | public void setVersion(Long version) { 28 | this.version = version; 29 | } 30 | 31 | public ObjectId getId() { 32 | return id; 33 | } 34 | 35 | public void setId(ObjectId id) { 36 | this.id = id; 37 | } 38 | } 39 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/model/Type.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.model; 2 | 3 | import com.fasterxml.jackson.annotation.JsonIgnoreProperties; 4 | import org.springframework.data.mongodb.core.mapping.Document; 5 | 6 | import java.util.Date; 7 | import java.util.List; 8 | import java.util.Map; 9 | 10 | /** 11 | * Created by dandan.sha on 2018/08/28. 12 | */ 13 | @Document(collection = "typeinfos") 14 | public class Type { 15 | private String type = "object"; 16 | private String name; 17 | private String detail; 18 | private List required; 19 | private Map properties; 20 | private Boolean additionalProperties; 21 | private Date updated; 22 | private Date created; 23 | 24 | public Boolean getAdditionalProperties() { 25 | return additionalProperties; 26 | } 27 | 28 | public void setAdditionalProperties(Boolean additionalProperties) { 29 | this.additionalProperties = additionalProperties; 30 | } 31 | 32 | public String getDetail() { 33 | return detail; 34 | } 35 | 36 | public void setDetail(String detail) { 37 | this.detail = detail; 38 | } 39 | 40 | public String getName() { 41 | return name; 42 | } 43 | 44 | public void setName(String name) { 45 | this.name = name; 46 | } 47 | 48 | 49 | public List getRequired() { 50 | return required; 51 | } 52 | 53 | public void setRequired(List required) { 54 | this.required = required; 55 | } 56 | 57 | public Map getProperties() { 58 | return properties; 59 | } 60 | 61 | public void setProperties(Map properties) { 62 | this.properties = properties; 63 | } 64 | 65 | public Date getUpdated() { 66 | return updated; 67 | } 68 | 69 | public void setUpdated(Date updated) { 70 | this.updated = updated; 71 | } 72 | 73 | public Date getCreated() { 74 | return created; 75 | } 76 | 77 | public void setCreated(Date created) { 78 | this.created = created; 79 | } 80 | 81 | public String getType() { 82 | return type; 83 | } 84 | 85 | public void setType(String type) { 86 | this.type = type; 87 | } 88 | 89 | @JsonIgnoreProperties(value = {"name", "detail", "updated", "created"}) 90 | public interface SchemaMixIn { 91 | } 92 | } 93 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/model/jackson/EventDeserializer.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.model.jackson; 2 | 3 | import com.fasterxml.jackson.core.JsonParser; 4 | import com.fasterxml.jackson.databind.DeserializationContext; 5 | import com.fasterxml.jackson.databind.JsonDeserializer; 6 | import com.qunar.cm.ic.common.exception.ExceptionEnum; 7 | import com.qunar.cm.ic.common.exception.ICException; 8 | import com.qunar.cm.ic.model.Event; 9 | import joptsimple.internal.Strings; 10 | 11 | import java.io.IOException; 12 | import java.time.OffsetDateTime; 13 | import java.util.Date; 14 | import java.util.LinkedHashMap; 15 | import java.util.Map; 16 | 17 | /** 18 | * Created by yu.qi on 2018/08/27. 19 | */ 20 | public class EventDeserializer extends JsonDeserializer { 21 | @Override 22 | public Event deserialize(JsonParser jp, DeserializationContext ctxt) throws IOException { 23 | Event event = new Event(); 24 | Map body = readBody(jp); 25 | event.setBody(body); 26 | event.setType(parseType(body)); 27 | event.setTime(parseTime(body)); 28 | event.setOperator((String) body.get("operator")); 29 | event.setSource((String) body.get("source")); 30 | event.setHidden(true); 31 | event.setDummy(false); 32 | event.setUpdated(new Date()); 33 | event.normalizeBody(); 34 | return event; 35 | } 36 | 37 | 38 | @SuppressWarnings("unchecked") 39 | private Map readBody(JsonParser jp) throws IOException { 40 | return jp.getCodec().readValue(jp, LinkedHashMap.class); 41 | } 42 | 43 | private Date parseTime(Map properties) { 44 | return parseTime((String) properties.get("time"), (Long) properties.get("timestamp")); 45 | } 46 | 47 | private String parseType(Map properties) { 48 | return parseType((String) properties.get("event"), (String) properties.get("type")); 49 | } 50 | 51 | /** 52 | * 如果事件不包含类型信息，则返回空字符串 53 | */ 54 | private String parseType(String type, String event) { 55 | String result = Strings.EMPTY; 56 | if (type != null) { 57 | result = type; 58 | } else if (event != null) { 59 | result = event; 60 | } 61 | return result; 62 | } 63 | 64 | Date parseTime(String time, Long timestamp) { 65 | Date result; 66 | if (timestamp != null) { 67 | result = new Date(timestamp); 68 | } else if (time != null) { 69 | OffsetDateTime offsetDateTime = OffsetDateTime.parse(time); 70 | result = Date.from(offsetDateTime.toInstant()); 71 | } else { 72 | throw new ICException(ExceptionEnum.PARAMS_INVALID, "事件中必须包含timestamp字段"); 73 | } 74 | return result; 75 | } 76 | 77 | } 78 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/model/jackson/EventSerializer.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.model.jackson; 2 | 3 | import com.fasterxml.jackson.core.JsonGenerator; 4 | import com.fasterxml.jackson.databind.JsonSerializer; 5 | import com.fasterxml.jackson.databind.SerializerProvider; 6 | import com.google.common.base.Preconditions; 7 | import com.google.common.collect.Sets; 8 | import com.qunar.cm.ic.model.Event; 9 | 10 | import java.io.IOException; 11 | import java.time.OffsetDateTime; 12 | import java.time.ZoneId; 13 | import java.time.format.DateTimeFormatter; 14 | import java.util.Date; 15 | import java.util.Map; 16 | import java.util.Set; 17 | 18 | /** 19 | * Created by yu.qi on 2018/08/27. 20 | */ 21 | public class EventSerializer extends JsonSerializer { 22 | private static final Set reservedFieldNames = Sets.newHashSet( 23 | "id", "type", "event", "operator", "source", "time", 24 | "timestamp", "updated", "updatedTimestamp" 25 | ); 26 | private DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ssZZZZZ"); 27 | 28 | @Override 29 | public void serialize(Event event, JsonGenerator jsonGenerator, SerializerProvider serializerProvider) throws IOException { 30 | jsonGenerator.writeStartObject(); 31 | jsonGenerator.writeNumberField("id", event.getId()); 32 | jsonGenerator.writeStringField("type", event.getType()); 33 | jsonGenerator.writeStringField("event", event.getType()); 34 | jsonGenerator.writeStringField("operator", event.getOperator()); 35 | jsonGenerator.writeStringField("source", event.getSource()); 36 | jsonGenerator.writeStringField("time", formatDateTime(event.getTime())); 37 | jsonGenerator.writeNumberField("timestamp", event.getTime().getTime()); 38 | jsonGenerator.writeStringField("updated", formatDateTime(event.getUpdated())); 39 | jsonGenerator.writeNumberField("updatedTimestamp", event.getUpdated().getTime()); 40 | for (Map.Entry entry : event.getBody().entrySet()) { 41 | if (!entry.getKey().startsWith("_") && !reservedFieldNames.contains(entry.getKey())) { 42 | jsonGenerator.writeObjectField(entry.getKey(), entry.getValue()); 43 | } 44 | } 45 | jsonGenerator.writeEndObject(); 46 | } 47 | 48 | String formatDateTime(Date date) { 49 | Preconditions.checkNotNull(date); 50 | //调用truncatedTo是为了去掉毫秒，最终生成的格式如2018-08-28T14:28:21+08:00 51 | OffsetDateTime offsetDateTime = OffsetDateTime.ofInstant(date.toInstant(), ZoneId.of("UTC+8")); 52 | return offsetDateTime.format(formatter); 53 | } 54 | } 55 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/package-info.java: -------------------------------------------------------------------------------- 1 | /** 2 | * Created by yu.qi on 2018/09/14. 3 | */ 4 | @NonNullApi 5 | package com.qunar.cm.ic; 6 | 7 | import org.springframework.lang.NonNullApi; -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/service/EventConsumerService.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.service; 2 | 3 | import java.util.Date; 4 | import java.util.concurrent.TimeoutException; 5 | 6 | /** 7 | * Created by yu.qi on 2018/08/31. 8 | */ 9 | public interface EventConsumerService { 10 | 11 | long waitForNextEvent(String token, Date deadline) throws TimeoutException; 12 | } 13 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/service/EventService.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.service; 2 | 3 | import com.qunar.cm.ic.model.Event; 4 | 5 | import java.util.List; 6 | 7 | /** 8 | * Created by dandan.sha on 2018/08/24. 9 | */ 10 | public interface EventService { 11 | Event queryById(Long id); 12 | 13 | Event checkAndSaveEvent(Event event); 14 | 15 | List queryByTimeAndType(String type, String from, String to); 16 | } 17 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/service/ListenerService.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.service; 2 | 3 | import com.qunar.cm.ic.dto.ListenerFetchResult; 4 | 5 | import java.util.List; 6 | 7 | /** 8 | * Created by yu.qi on 2018/08/29. 9 | */ 10 | public interface ListenerService { 11 | ListenerFetchResult consumeEvents(String token, List types, int maxResults, boolean longPoll, String ip); 12 | 13 | void acknowledge(String token, String code, String ip); 14 | 15 | } 16 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/service/ProducerService.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.service; 2 | 3 | /** 4 | * Created by dandan.sha on 2018/09/11. 5 | */ 6 | public interface ProducerService { 7 | void checkIp(String name,String ip); 8 | } 9 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/service/PropertyService.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.service; 2 | 3 | /** 4 | * Created by yu.qi on 2018/9/5. 5 | */ 6 | public interface PropertyService { 7 | boolean changedSinceLastAccess(String key); 8 | } 9 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/service/TypeService.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.service; 2 | 3 | import com.qunar.cm.ic.model.Event; 4 | import com.qunar.cm.ic.model.Type; 5 | 6 | import java.util.List; 7 | 8 | /** 9 | * Created by dandan.sha on 2018/08/29. 10 | */ 11 | public interface TypeService { 12 | void checkEvent(Event event); 13 | 14 | List allTypes(); 15 | 16 | Type getType(String name); 17 | } 18 | 19 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/service/impl/EventConsumerServiceImpl.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.service.impl; 2 | 3 | import com.qunar.cm.ic.service.EventConsumerService; 4 | import org.slf4j.Logger; 5 | import org.slf4j.LoggerFactory; 6 | import org.springframework.stereotype.Service; 7 | 8 | import java.util.Date; 9 | import java.util.concurrent.TimeoutException; 10 | 11 | /** 12 | * Created by yu.qi on 2018/08/31. 13 | */ 14 | @Service 15 | public class EventConsumerServiceImpl implements EventConsumerService { 16 | private static final Logger logger = LoggerFactory.getLogger(EventConsumerServiceImpl.class); 17 | 18 | 19 | @Override 20 | public long waitForNextEvent(String token, Date deadline) throws TimeoutException { 21 | if (new Date().after(deadline)) { 22 | throw new TimeoutException(); 23 | } 24 | try { 25 | Thread.sleep(2000); 26 | } catch (InterruptedException e) { 27 | logger.error("Listener[token={}]消费事件时，等待事件失败", token, e); 28 | } 29 | return 0; 30 | } 31 | 32 | } 33 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/service/impl/EventServiceImpl.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.service.impl; 2 | 3 | import com.google.common.base.Preconditions; 4 | import com.google.common.cache.CacheBuilder; 5 | import com.google.common.cache.CacheLoader; 6 | import com.google.common.cache.LoadingCache; 7 | import com.google.common.collect.Lists; 8 | import com.google.common.util.concurrent.UncheckedExecutionException; 9 | import com.mongodb.client.result.UpdateResult; 10 | import com.qunar.cm.ic.common.exception.ExceptionEnum; 11 | import com.qunar.cm.ic.common.exception.ICException; 12 | import com.qunar.cm.ic.dao.EventRepository; 13 | import com.qunar.cm.ic.model.Event; 14 | import com.qunar.cm.ic.model.IdentityCounter; 15 | import com.qunar.cm.ic.service.EventService; 16 | import com.qunar.cm.ic.service.TypeService; 17 | import joptsimple.internal.Strings; 18 | import org.slf4j.Logger; 19 | import org.slf4j.LoggerFactory; 20 | import org.springframework.data.domain.Sort; 21 | import org.springframework.data.mongodb.core.FindAndModifyOptions; 22 | import org.springframework.data.mongodb.core.MongoTemplate; 23 | import org.springframework.data.mongodb.core.query.Criteria; 24 | import org.springframework.data.mongodb.core.query.Query; 25 | import org.springframework.data.mongodb.core.query.Update; 26 | import org.springframework.scheduling.annotation.Scheduled; 27 | import org.springframework.stereotype.Service; 28 | 29 | import javax.annotation.ParametersAreNonnullByDefault; 30 | import javax.annotation.Resource; 31 | import java.time.OffsetDateTime; 32 | import java.time.format.DateTimeParseException; 33 | import java.util.Date; 34 | import java.util.List; 35 | import java.util.Objects; 36 | import java.util.Optional; 37 | import java.util.concurrent.TimeUnit; 38 | import java.util.concurrent.atomic.AtomicLong; 39 | import java.util.concurrent.locks.Condition; 40 | import java.util.concurrent.locks.Lock; 41 | import java.util.concurrent.locks.ReentrantLock; 42 | import java.util.stream.Collectors; 43 | 44 | /** 45 | * Created by dandan.sha on 2018/08/24. 46 | */ 47 | 48 | @Service 49 | public class EventServiceImpl implements EventService { 50 | private static final Logger logger = LoggerFactory.getLogger(EventServiceImpl.class); 51 | 52 | private static final long MILLISECONDS_PER_DAY = 24 * 60 * 60 * 1000L; 53 | private static final Sort orderByIdAsc = Sort.by(Sort.Order.asc("id")); 54 | 55 | 56 | @Resource 57 | private EventRepository eventRepository; 58 | 59 | @Resource 60 | private MongoTemplate mongoTemplate; 61 | 62 | @Resource 63 | private TypeService typeService; 64 | 65 | 66 | private LoadingCache caches; 67 | 68 | private AtomicLong eventCount = new AtomicLong(); 69 | 70 | private volatile boolean running = true; 71 | private final Lock lock = new ReentrantLock(); 72 | private final Condition condition = lock.newCondition(); 73 | 74 | public EventServiceImpl() { 75 | Runtime.getRuntime().addShutdownHook(new Thread(() -> { 76 | running = false; 77 | logger.info("程序即将退出，publishEvents定时任务将终止"); 78 | })); 79 | caches = CacheBuilder.newBuilder() 80 | .maximumSize(10000) 81 | .recordStats() 82 | .build(new CacheLoader() { 83 | @Override 84 | @ParametersAreNonnullByDefault 85 | public Event load(Long key) throws Exception { 86 | Event event = queryByIdFromDb(key); 87 | logger.info("从数据库中加载事件{}到缓存中，添加前缓存大小为{}", key, caches.size()); 88 | return event; 89 | } 90 | }); 91 | } 92 | 93 | 94 | @Override 95 | public Event queryById(Long id) { 96 | //使用getUnchecked要求CacheLoader.load方法必须不能抛出任何checked的异常 97 | try { 98 | return caches.getUnchecked(id); 99 | } catch (UncheckedExecutionException e) { 100 | //如果load方法出现异常，取出原始的ICException异常对象 101 | if (e.getCause() instanceof ICException) { 102 | throw (ICException) e.getCause(); 103 | } 104 | throw e; 105 | } 106 | } 107 | 108 | private Event queryByIdFromDb(Long id) { 109 | Optional optionalEvent = eventRepository.findOneById(id); 110 | return optionalEvent.orElseThrow(() -> 111 | new ICException(ExceptionEnum.PARAMS_INVALID, "事件" + id + "不存在")); 112 | } 113 | 114 | @Override 115 | public Event checkAndSaveEvent(Event event) { 116 | typeService.checkEvent(event); 117 | IdentityCounter identityCounter = getIdentityCounter(); 118 | event.setId(identityCounter.getCount()); 119 | eventRepository.insert(event); 120 | notifyForInsertedEvent(); 121 | return event; 122 | } 123 | 124 | private void notifyForInsertedEvent() { 125 | eventCount.incrementAndGet(); 126 | lock.lock(); 127 | try { 128 | condition.signalAll(); 129 | } finally { 130 | lock.unlock(); 131 | } 132 | } 133 | 134 | private IdentityCounter getIdentityCounter() { 135 | Query query = new Query(Criteria.where("field").is("id")); 136 | Update update = new Update(); 137 | update.inc("count", 1); 138 | FindAndModifyOptions options = new FindAndModifyOptions(); 139 | options.returnNew(true); 140 | IdentityCounter identityCounter = mongoTemplate.findAndModify(query, update, options, IdentityCounter.class); 141 | assert identityCounter != null; 142 | return identityCounter; 143 | } 144 | 145 | @Override 146 | public List queryByTimeAndType(String type, String from, String to) { 147 | Date fromDate = parseDate(from); 148 | Date toDate; 149 | if (Strings.isNullOrEmpty(to)) { 150 | toDate = new Date(); 151 | } else { 152 | toDate = parseDate(to); 153 | } 154 | if (toDate.getTime() - fromDate.getTime() > MILLISECONDS_PER_DAY) { 155 | throw new ICException(ExceptionEnum.PARAMS_INVALID, "from和to时间跨度不能超过一天"); 156 | } 157 | return eventRepository.findByTypeAndTime(type, fromDate, toDate); 158 | } 159 | 160 | private Date parseDate(String from) { 161 | try { 162 | return Date.from(OffsetDateTime.parse(from).toInstant()); 163 | } catch (DateTimeParseException e) { 164 | throw new ICException(ExceptionEnum.PARAMS_INVALID, "时间格式" + from + "不合法", e); 165 | } 166 | } 167 | 168 | 169 | @Scheduled(fixedDelay = 1000L) 170 | public synchronized void publishEvents() { 171 | logger.info("publishEvents定时任务执行开始"); 172 | Long lastNotHiddenEventId = getLastNotHiddenEventId(); 173 | 174 | Long oldEventCount = 0L; 175 | Long newEventCount = eventCount.get(); 176 | 177 | NewEventFoundTime newEventFoundTime = new NewEventFoundTime(); 178 | //单位秒 179 | while (running) { 180 | while (Objects.equals(oldEventCount, newEventCount)) { 181 | lock.lock(); 182 | try { 183 | if (!condition.await(5, TimeUnit.SECONDS)) { 184 | break; 185 | } 186 | } catch (InterruptedException e) { 187 | logger.error("publishEvents定时任务被中断", e); 188 | Thread.currentThread().interrupt(); 189 | } finally { 190 | lock.unlock(); 191 | } 192 | newEventCount = eventCount.get(); 193 | } 194 | oldEventCount = newEventCount; 195 | 196 | List hiddenEvents = eventRepository.findGreaterThanId(lastNotHiddenEventId, orderByIdAsc); 197 | List sequentialEvents = getSequentialEvents(lastNotHiddenEventId, hiddenEvents); 198 | logger.info("publishEvents定时任务获取到{}个隐藏的事件，其中包含{}个连续事件，lastNotHiddenEventId为{}", 199 | hiddenEvents.size(), sequentialEvents.size(), lastNotHiddenEventId); 200 | 201 | //处理等待超时的逻辑 202 | if (sequentialEvents.isEmpty()) { 203 | if (!hiddenEvents.isEmpty()) { 204 | newEventFoundTime.set(); 205 | if (newEventFoundTime.timeout()) { 206 | createDummyEvent(++lastNotHiddenEventId); 207 | logger.warn("publishEvents定时任务没有找到事件{}且已经超时，已经跳过该事件", lastNotHiddenEventId); 208 | newEventFoundTime.unset(); 209 | } 210 | } 211 | } else { 212 | newEventFoundTime.unset(); 213 | publishEvents(sequentialEvents); 214 | lastNotHiddenEventId += sequentialEvents.size(); 215 | } 216 | } 217 | logger.info("publishEvents定时任务执行结束"); 218 | } 219 | 220 | private void publishEvents(List sequentialEvents) { 221 | Preconditions.checkState(!sequentialEvents.isEmpty()); 222 | //发布事件，先将事件hidden改为false,然后发qmq消息 223 | Long firstId = sequentialEvents.get(0).getId(); 224 | Long lastId = sequentialEvents.get(sequentialEvents.size() - 1).getId(); 225 | Query query = Query.query(Criteria.where("id").gte(firstId).lte(lastId)); 226 | Update update = Update.update("_hidden", false); 227 | UpdateResult updateResult = mongoTemplate.updateMulti(query, update, Event.class); 228 | Preconditions.checkState(updateResult.wasAcknowledged()); 229 | //getMatchedCount返回的是long，size返回的是int，不能使用equals 230 | Preconditions.checkState(updateResult.getMatchedCount() == sequentialEvents.size()); 231 | //TODO 232 | logger.info("publishEvents定时任务发布了事件{}", 233 | sequentialEvents.stream().map(Event::getId).collect(Collectors.toList())); 234 | } 235 | 236 | private List getSequentialEvents(Long startEventId, List events) { 237 | List sequentialEvents = Lists.newArrayList(); 238 | Long lastEventId = startEventId; 239 | for (Event event : events) { 240 | if (!Objects.equals(event.getId(), ++lastEventId)) { 241 | break; 242 | } 243 | sequentialEvents.add(event); 244 | } 245 | return sequentialEvents; 246 | } 247 | 248 | private void createDummyEvent(Long id) { 249 | Event dummyEvent = new Event(); 250 | dummyEvent.setId(id); 251 | dummyEvent.setHidden(false); 252 | dummyEvent.setDummy(true); 253 | //这里不能使用save方法，因为save方法对已经存在的id会直接执行更新操作 254 | eventRepository.insert(dummyEvent); 255 | notifyForInsertedEvent(); 256 | } 257 | 258 | private Long getLastNotHiddenEventId() { 259 | //因为_hidden可能为null,所以使用ne(true) 260 | Query query = new Query(Criteria.where("_hidden").ne(true)); 261 | query.with(Sort.by(Sort.Order.desc("id"))).limit(1); //按id进行 降序 262 | Event event = mongoTemplate.findOne(query, Event.class); 263 | if (event != null) { 264 | return event.getId(); 265 | } else { 266 | return 0L; 267 | } 268 | } 269 | 270 | private static class NewEventFoundTime { 271 | //单位为毫秒 272 | private static long newEventFoundTimeout = 5000; 273 | private Date date; 274 | 275 | private void set() { 276 | if (date == null) { 277 | date = new Date(); 278 | } 279 | } 280 | 281 | private void unset() { 282 | date = null; 283 | } 284 | 285 | private boolean timeout() { 286 | Preconditions.checkNotNull(date, "必须先调用set方法"); 287 | return new Date().getTime() - date.getTime() > newEventFoundTimeout; 288 | } 289 | } 290 | } 291 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/service/impl/ListenerServiceImpl.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.service.impl; 2 | 3 | import com.google.common.collect.Lists; 4 | import com.google.common.collect.Maps; 5 | import com.qunar.cm.ic.common.exception.ExceptionEnum; 6 | import com.qunar.cm.ic.common.exception.ICException; 7 | import com.qunar.cm.ic.dao.EventRepository; 8 | import com.qunar.cm.ic.dao.ListenerRepository; 9 | import com.qunar.cm.ic.dao.page.FirstEventPage; 10 | import com.qunar.cm.ic.dto.IpMatcher; 11 | import com.qunar.cm.ic.dto.ListenerFetchResult; 12 | import com.qunar.cm.ic.model.Event; 13 | import com.qunar.cm.ic.model.Listener; 14 | import com.qunar.cm.ic.service.EventConsumerService; 15 | import com.qunar.cm.ic.service.ListenerService; 16 | import org.slf4j.Logger; 17 | import org.slf4j.LoggerFactory; 18 | import org.springframework.stereotype.Service; 19 | 20 | import javax.annotation.Resource; 21 | import java.util.Calendar; 22 | import java.util.Collections; 23 | import java.util.Date; 24 | import java.util.List; 25 | import java.util.Objects; 26 | import java.util.Optional; 27 | import java.util.UUID; 28 | import java.util.concurrent.ConcurrentMap; 29 | import java.util.concurrent.TimeoutException; 30 | 31 | /** 32 | * Created by yu.qi on 2018/08/29. 33 | */ 34 | @Service 35 | public class ListenerServiceImpl implements ListenerService { 36 | private static final Logger logger = LoggerFactory.getLogger(ListenerServiceImpl.class); 37 | 38 | @Resource 39 | private EventRepository eventRepository; 40 | @Resource 41 | private ListenerRepository listenerRepository; 42 | @Resource 43 | private EventConsumerService eventConsumerService; 44 | 45 | private ConcurrentMap ipMatcherMap = Maps.newConcurrentMap(); 46 | 47 | 48 | @Override 49 | public ListenerFetchResult consumeEvents(String token, List types, int maxResults, boolean longPoll, String ip) { 50 | 51 | Optional optionalListener = listenerRepository.findByToken(token); 52 | Listener listener = optionalListener.orElseGet(() -> createTemporaryListener(token)); 53 | if (!createIpMatcher(listener).match(ip)) { 54 | throw new ICException(ExceptionEnum.IP_LIMITED, ip + "没有访问权限"); 55 | } 56 | ListenerFetchResult result = new ListenerFetchResult(listener.getCode(), maxResults); 57 | result.setList(Collections.emptyList()); 58 | 59 | fillResult(listener.getLastEventId(), types, maxResults, result); 60 | if (longPoll && result.isEmpty()) { 61 | Calendar calendar = Calendar.getInstance(); 62 | calendar.setTime(new Date()); 63 | calendar.add(Calendar.SECOND, 10); 64 | Date deadLine = calendar.getTime(); 65 | 66 | while (result.isEmpty()) { 67 | try { 68 | eventConsumerService.waitForNextEvent(token, deadLine); 69 | } catch (TimeoutException e) { 70 | break; 71 | } 72 | fillResult(listener.getLastEventId(), types, maxResults, result); 73 | } 74 | } 75 | 76 | if (!result.isEmpty()) { 77 | //重新生成code，如果返回的结果为空，则code不变 78 | listener.setCode(UUID.randomUUID().toString()); 79 | long recordEventId = result.getList().get(result.getList().size() - 1).getId(); 80 | listener.setRecordEventId(recordEventId); 81 | listener.setReadTime(new Date()); 82 | listenerRepository.save(listener); 83 | } 84 | result.setCode(listener.getCode()); 85 | return result; 86 | } 87 | 88 | 89 | private void fillResult(long lastEventId, List types, int maxResults, 90 | ListenerFetchResult result) { 91 | List events; 92 | //多获取一个，用于判断hasMore 93 | int actualQueryCount = maxResults + 1; 94 | if (types.isEmpty()) { 95 | events = eventRepository.consumeEvent( 96 | lastEventId, FirstEventPage.of(actualQueryCount)); 97 | } else { 98 | events = eventRepository.consumeEventByTypes( 99 | lastEventId, types, FirstEventPage.of(actualQueryCount)); 100 | } 101 | 102 | boolean hasMore = false; 103 | if (events.size() == actualQueryCount) { 104 | events.remove(actualQueryCount - 1); 105 | hasMore = true; 106 | } 107 | result.setHasMore(hasMore); 108 | result.setList(events); 109 | } 110 | 111 | /** 112 | * 不支持并发，即多个用户同时用一个不存在的token时，有一个会报错 113 | */ 114 | Listener createTemporaryListener(String token) { 115 | Listener listener = new Listener(); 116 | listener.setToken(token); 117 | listener.setDetail(token); 118 | listener.setCode(UUID.randomUUID().toString()); 119 | String matchAllIps = ".*"; 120 | listener.setIps(Lists.newArrayList(matchAllIps)); 121 | Optional optionalEvent = eventRepository.findFirstByOrderByIdDesc(); 122 | if (optionalEvent.isPresent()) { 123 | listener.setLastEventId(optionalEvent.get().getId()); 124 | } else { 125 | listener.setLastEventId(0L); 126 | } 127 | listener.setTemporary(true); 128 | //insert方法的返回值是一个包含id的listener，可以用于后续的save操作 129 | logger.info("Listener[token={}]不存在，本次自动创建，lastEventId为{}", token, listener.getLastEventId()); 130 | return listenerRepository.insert(listener); 131 | } 132 | 133 | 134 | @Override 135 | public void acknowledge(String token, String code, String ip) { 136 | Optional optionalListener = listenerRepository.findByToken(token); 137 | Listener listener = optionalListener.orElseThrow(() -> 138 | new ICException(ExceptionEnum.PARAMS_INVALID, "token的值无效")); 139 | if (!createIpMatcher(listener).match(ip)) { 140 | throw new ICException(ExceptionEnum.IP_LIMITED, ip + "没有访问权限"); 141 | } 142 | if (!Objects.equals(listener.getCode(), code)) { 143 | throw new ICException(ExceptionEnum.PARAMS_INVALID, "code的值不正确"); 144 | } 145 | if (listener.getRecordEventId() != null && !Objects.equals(listener.getLastEventId(), listener.getRecordEventId())) { 146 | listener.setLastEventId(listener.getRecordEventId()); 147 | listener.setReplyTime(new Date()); 148 | listenerRepository.save(listener); 149 | } 150 | } 151 | 152 | /** 153 | * 创建IpMatcher对象，用于校验ip权限，这个接口还会缓存结果 154 | */ 155 | private IpMatcher createIpMatcher(Listener listener) { 156 | return ipMatcherMap.compute(listener.getToken(), (key, value) -> { 157 | if (value != null && Objects.equals(value.getIps(), listener.getIps())) { 158 | return value; 159 | } 160 | logger.info("Listener[token={}]创建或更新IpMatcher对象", listener.getToken()); 161 | return new IpMatcher(listener.getIps()); 162 | }); 163 | } 164 | 165 | } 166 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/service/impl/ProducerServiceImpl.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.service.impl; 2 | 3 | import com.google.common.base.Preconditions; 4 | import com.google.common.collect.Maps; 5 | import com.qunar.cm.ic.common.exception.ExceptionEnum; 6 | import com.qunar.cm.ic.common.exception.ICException; 7 | import com.qunar.cm.ic.dao.ProducerRepository; 8 | import com.qunar.cm.ic.dto.IpMatcher; 9 | import com.qunar.cm.ic.model.Producer; 10 | import com.qunar.cm.ic.service.ProducerService; 11 | import com.qunar.cm.ic.service.PropertyService; 12 | import org.slf4j.Logger; 13 | import org.slf4j.LoggerFactory; 14 | import org.springframework.scheduling.annotation.Scheduled; 15 | import org.springframework.stereotype.Service; 16 | 17 | import javax.annotation.Resource; 18 | import java.util.List; 19 | import java.util.concurrent.ConcurrentMap; 20 | 21 | /** 22 | * Created by dandan.sha on 2018/09/11. 23 | */ 24 | @Service 25 | public class ProducerServiceImpl implements ProducerService { 26 | private static final Logger logger = LoggerFactory.getLogger(ProducerServiceImpl.class); 27 | @Resource 28 | private ProducerRepository producerRepository; 29 | @Resource 30 | private PropertyService propertyService; 31 | 32 | private static final String PRODUCER_CACHE_KEY = "cache.producer"; 33 | private volatile ConcurrentMap ipMatcherMap; 34 | 35 | @Override 36 | public void checkIp(String name, String ip) { 37 | Preconditions.checkNotNull(ipMatcherMap, "ProducerCache尚未初始化完成"); 38 | IpMatcher ipMatcher = ipMatcherMap.get(name); 39 | if (!ipMatcher.match(ip)) { 40 | throw new ICException(ExceptionEnum.IP_LIMITED, ip + "没有访问权限"); 41 | } 42 | } 43 | 44 | private synchronized void refreshIpMatcherMap() { 45 | List producers = producerRepository.findAll(); 46 | ConcurrentMap newIpMatcherMap = Maps.newConcurrentMap(); 47 | producers.forEach(producer -> 48 | newIpMatcherMap.put(producer.getName(), new IpMatcher(producer.getIps()))); 49 | ipMatcherMap = newIpMatcherMap; 50 | logger.info("更新Producer的ip列表成功，更新的Producer数量为{}", producers.size()); 51 | } 52 | 53 | @Scheduled(fixedDelay = 5000L) 54 | public synchronized void refreshIpMatcherMapOnChange() { 55 | if (propertyService.changedSinceLastAccess(PRODUCER_CACHE_KEY)) { 56 | refreshIpMatcherMap(); 57 | } 58 | } 59 | } 60 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/service/impl/PropertyServiceImpl.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.service.impl; 2 | 3 | import com.google.common.collect.Maps; 4 | import com.qunar.cm.ic.dao.PropertyRepository; 5 | import com.qunar.cm.ic.model.Property; 6 | import com.qunar.cm.ic.service.PropertyService; 7 | import org.springframework.stereotype.Service; 8 | 9 | import javax.annotation.Resource; 10 | import java.util.Map; 11 | import java.util.Objects; 12 | import java.util.Optional; 13 | 14 | /** 15 | * Created by yu.qi on 2018/9/5. 16 | */ 17 | @Service 18 | public class PropertyServiceImpl implements PropertyService { 19 | @Resource 20 | private PropertyRepository propertyRepository; 21 | 22 | //下面的方法都定义成了synchronized，这里可以使用Map 23 | private Map versions = Maps.newHashMap(); 24 | 25 | /** 26 | * 这个方法调用不频繁，添加synchronized不会影响性能 27 | */ 28 | @Override 29 | public synchronized boolean changedSinceLastAccess(String key) { 30 | Optional optionalProperty = propertyRepository.findByKey(key); 31 | Property property = optionalProperty.orElseGet(() -> createVersionProperty(key)); 32 | Long oldVersion = versions.put(property.getKey(), property.getVersion()); 33 | return !Objects.equals(oldVersion, property.getVersion()); 34 | } 35 | 36 | private synchronized Property createVersionProperty(String key) { 37 | Property property = new Property(); 38 | property.setKey(key); 39 | property.setVersion(0L); 40 | return property; 41 | } 42 | } 43 | -------------------------------------------------------------------------------- /src/main/java/com/qunar/cm/ic/service/impl/TypeServiceImpl.java: -------------------------------------------------------------------------------- 1 | package com.qunar.cm.ic.service.impl; 2 | 3 | import com.fasterxml.jackson.core.JsonProcessingException; 4 | import com.fasterxml.jackson.databind.JsonNode; 5 | import com.fasterxml.jackson.databind.ObjectMapper; 6 | import com.github.fge.jackson.JsonLoader; 7 | import com.github.fge.jsonschema.core.exceptions.ProcessingException; 8 | import com.github.fge.jsonschema.core.report.ProcessingReport; 9 | import com.github.fge.jsonschema.main.JsonSchema; 10 | import com.github.fge.jsonschema.main.JsonSchemaFactory; 11 | import com.google.common.base.Preconditions; 12 | import com.google.common.collect.Lists; 13 | import com.google.common.collect.Maps; 14 | import com.qunar.cm.ic.common.exception.ExceptionEnum; 15 | import com.qunar.cm.ic.common.exception.ICException; 16 | import com.qunar.cm.ic.dao.TypeRepository; 17 | import com.qunar.cm.ic.model.Event; 18 | import com.qunar.cm.ic.model.Type; 19 | import com.qunar.cm.ic.service.PropertyService; 20 | import com.qunar.cm.ic.service.TypeService; 21 | import org.slf4j.Logger; 22 | import org.slf4j.LoggerFactory; 23 | import org.springframework.scheduling.annotation.Scheduled; 24 | import org.springframework.stereotype.Service; 25 | 26 | import javax.annotation.Resource; 27 | import java.io.IOException; 28 | import java.net.URL; 29 | import java.util.List; 30 | import java.util.Map; 31 | import java.util.Optional; 32 | import java.util.stream.Collectors; 33 | 34 | /** 35 | * Created by dandan.sha on 2018/08/29. 36 | */ 37 | @Service 38 | public class TypeServiceImpl implements TypeService { 39 | private static Logger logger = LoggerFactory.getLogger(TypeServiceImpl.class); 40 | private static ObjectMapper objectMapper = new ObjectMapper(); 41 | private static ObjectMapper schemaObjectMapper = new ObjectMapper().addMixIn(Type.class, Type.SchemaMixIn.class); 42 | private static JsonSchemaFactory factory = JsonSchemaFactory.byDefault(); 43 | private static final String CACHE_KEY = "cache.schema"; 44 | private static final String BASE_TYPE_FILE_NAME = "base-type.json"; 45 | private Type baseType; 46 | 47 | @Resource 48 | private TypeRepository typeRepository; 49 | @Resource 50 | private PropertyService propertyService; 51 | 52 | private volatile Map caches; 53 | 54 | public TypeServiceImpl() { 55 | initializeBaseType(); 56 | } 57 | 58 | private void initializeBaseType() { 59 | URL resource = getClass().getClassLoader().getResource(BASE_TYPE_FILE_NAME); 60 | Preconditions.checkNotNull(resource, BASE_TYPE_FILE_NAME + "文件不存在"); 61 | try { 62 | baseType = objectMapper.readValue(resource, Type.class); 63 | } catch (IOException e) { 64 | throw new ICException("从" + BASE_TYPE_FILE_NAME + "中创建BaseType异常", e); 65 | } 66 | } 67 | 68 | @Override 69 | public void checkEvent(Event event) { 70 | JsonNode jsonNode; 71 | try { 72 | jsonNode = JsonLoader.fromString(objectMapper.writeValueAsString(event.getBody())); 73 | } catch (IOException e) { 74 | throw new ICException("序列化或反序列化Json失败", e); 75 | } 76 | Preconditions.checkNotNull(caches, "JsonSchemaCache尚未初始化完成"); 77 | JsonSchema schema; 78 | schema = caches.get(event.getType()); 79 | if (schema == null) { 80 | throw new ICException("事件" + event.getType() + "未定义"); 81 | } 82 | ProcessingReport report; 83 | try { 84 | report = schema.validate(jsonNode); 85 | } catch (ProcessingException e) { 86 | throw new ICException("校验Json格式异常", e); 87 | } 88 | if (!report.isSuccess()) { 89 | throw new ICException(report.toString()); 90 | } 91 | 92 | } 93 | 94 | @Override 95 | public List allTypes() { 96 | return mergeAllWithBaseType(typeRepository.findAll()); 97 | } 98 | 99 | private List mergeAllWithBaseType(List types) { 100 | return types.stream().map(this::mergeWithBaseType).collect(Collectors.toList()); 101 | } 102 | 103 | private Type mergeWithBaseType(Type type) { 104 | List newRequired = Lists.newArrayList(); 105 | newRequired.addAll(baseType.getRequired()); 106 | newRequired.addAll(type.getRequired()); 107 | type.setRequired(newRequired.stream().distinct().collect(Collectors.toList())); 108 | 109 | type.setAdditionalProperties(type.getAdditionalProperties() == null ? 110 | baseType.getAdditionalProperties() : type.getAdditionalProperties()); 111 | 112 | Map newProperties = Maps.newLinkedHashMap(baseType.getProperties()); 113 | newProperties.putAll(type.getProperties()); 114 | type.setProperties(newProperties); 115 | return type; 116 | } 117 | 118 | 119 | @Override 120 | public Type getType(String name) { 121 | Optional optionalType = typeRepository.findOneByName(name); 122 | return mergeWithBaseType(optionalType.orElseThrow(() -> new ICException(ExceptionEnum.PARAMS_INVALID, "不存在事件类型：" + name))); 123 | } 124 | 125 | 126 | @Scheduled(fixedDelay = 5000L) 127 | public synchronized void refreshOnChanged() { 128 | if (propertyService.changedSinceLastAccess(CACHE_KEY)) { 129 | refresh(); 130 | } 131 | } 132 | 133 | private synchronized void refresh() { 134 | List typeList = mergeAllWithBaseType(typeRepository.findAll()); 135 | Map newCaches = Maps.newHashMap(); 136 | typeList.forEach(type -> newCaches.put(type.getName(), createJsonSchema(type))); 137 | caches = newCaches; 138 | logger.info("更新Schema列表成功，更新的Schema数量为{}", typeList.size()); 139 | } 140 | 141 | /** 142 | * 创建JsonSchema对象 143 | */ 144 | private JsonSchema createJsonSchema(Type type) { 145 | String typeJson; 146 | try { 147 | typeJson = schemaObjectMapper.writeValueAsString(type); 148 | } catch (JsonProcessingException e) { 149 | throw new ICException(ExceptionEnum.DATA_CONVERTER_ERROR, "将对象转化成Json字符串失败：" + type, e); 150 | } 151 | JsonNode jsonNode; 152 | try { 153 | jsonNode = JsonLoader.fromString(typeJson); 154 | } catch (IOException e) { 155 | throw new ICException(ExceptionEnum.DATA_CONVERTER_ERROR, "从字符串中生成Json失败：" + typeJson, e); 156 | } 157 | try { 158 | return factory.getJsonSchema(jsonNode); 159 | } catch (ProcessingException e) { 160 | throw new ICException(ExceptionEnum.DATA_CONVERTER_ERROR, "创建JsonSchema失败：" + jsonNode, e); 161 | } 162 | } 163 | 164 | } 165 | -------------------------------------------------------------------------------- /src/main/resources/application.properties: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/qunarcorp/ic/3836c67c7d1cff6d268b000c6fb16a50134d75a2/src/main/resources/application.properties -------------------------------------------------------------------------------- /src/main/resources/base-type.json: -------------------------------------------------------------------------------- 1 | { 2 | "properties": { 3 | "type": { 4 | "type": "string", 5 | "minLength": 1, 6 | "description": "事件类型" 7 | }, 8 | "source": { 9 | "type": "string", 10 | "minLength": 1, 11 | "description": "事件来源" 12 | }, 13 | "operator": { 14 | "type": "string", 15 | "minLength": 1, 16 | "description": "事件操作人" 17 | }, 18 | "timestamp": { 19 | "type": "integer", 20 | "description": "事件发生时间timestamp" 21 | }, 22 | "refId": { 23 | "type": "integer", 24 | "minimum": 1, 25 | "description": "关联事件ID" 26 | } 27 | }, 28 | "required": [ 29 | "type", 30 | "source", 31 | "operator" 32 | ], 33 | "additionalProperties": true 34 | } -------------------------------------------------------------------------------- /src/main/resources/logback.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | ${std-pattern} 10 | ${encoding} 11 | 12 | 13 | 14 | 15 | 17 | ${log.dir}/file.log 18 | 19 | ${log.dir}/file.%d{yyyy-MM-dd}.log 20 | 30 21 | 22 | 23 | ${std-pattern} 24 | ${encoding} 25 | 26 | 27 | 28 | 30 | ${log.dir}/error.log 31 | 32 | ERROR 33 | 34 | 35 | ${log.dir}/error.%d{yyyy-MM-dd}.log 36 | 30 37 | 38 | 39 | ${std-pattern} 40 | ${encoding} 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | -------------------------------------------------------------------------------- /src/main/resources/mongodb.properties: -------------------------------------------------------------------------------- 1 | mongo.hosts=: 2 | mongo.username=username 3 | mongo.password=pwd 4 | mongo.database=dc -------------------------------------------------------------------------------- /src/main/resources/spring-mongodb.xml: -------------------------------------------------------------------------------- 1 | 2 | 7 | 8 | 9 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | -------------------------------------------------------------------------------- /src/main/webapp/WEB-INF/web.xml: -------------------------------------------------------------------------------- 1 | 2 | 5 | 6 | 7 | 8 | encodingFilter 9 | org.springframework.web.filter.CharacterEncodingFilter 10 | 11 | encoding 12 | UTF-8 13 | 14 | 15 | forceEncoding 16 | true 17 | 18 | 19 | 20 | 21 | encodingFilter 22 | /* 23 | 24 | 25 | -------------------------------------------------------------------------------- /src/main/webapp/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | IC 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | --------------------------------------------------------------------------------