├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── DESIGN.md ├── GO_STYLE_GUIDE.md ├── ISSUE_TEMPLATE.md ├── LICENSE ├── README.md ├── command ├── autoorient │ ├── autoorient.go │ └── command.go ├── command.go ├── crop │ ├── command.go │ └── crop.go ├── format │ ├── command.go │ └── format.go ├── quality │ ├── command.go │ └── quality.go ├── resize │ ├── command.go │ └── resize.go ├── rotate │ ├── command.go │ └── rotate.go ├── sharpen │ ├── command.go │ └── sharpen.go ├── strip │ ├── command.go │ └── strip.go └── watermark │ ├── command.go │ ├── exception.go │ ├── recode.go │ └── watermark.go ├── docs ├── cn │ ├── PREREQUISITE.md │ └── REQUIREMENT_CLASS.md └── en │ ├── CODE_OF_CONDUCT.md │ ├── LOG_COMPATIBILITY_STANDARD.md │ └── README.md ├── example ├── README.md └── minimal │ ├── README.md │ ├── default.toml │ ├── main.go │ └── server.toml ├── interpret ├── interpret.go └── neph │ └── neph.go ├── log ├── fake.go ├── log.go ├── log_test.go ├── logger.go ├── output │ ├── dump.go │ ├── dump_test.go │ ├── level.go │ ├── level_test.go │ ├── output.go │ └── stdout.go ├── trace.go └── trace_test.go ├── process ├── image.go └── process.go ├── server ├── get.go ├── ping │ └── ping.go └── server.go ├── storage ├── neph │ ├── file.go │ ├── neph.go │ └── storage.go └── storage.go ├── thirdparty └── graphicsmagick │ ├── README.md │ ├── download.vbs │ ├── setup_unix.sh │ └── setup_windows.bat ├── throttle ├── pool.go ├── throttle.go └── workflow.go ├── util ├── stringx.go └── util.go └── vendor └── vendor.json /.gitignore: -------------------------------------------------------------------------------- 1 | # Compiled Object files, Static and Dynamic libs (Shared Objects) 2 | *.o 3 | *.a 4 | *.so 5 | *.jpg 6 | *.png 7 | *.log 8 | 9 | # Folders 10 | *_bak 11 | _obj 12 | _test 13 | .idea/ 14 | 15 | # executables 16 | *.test 17 | *.exe 18 | *.tmp 19 | 20 | #main 21 | /example/*/* 22 | !/example/*/*.go 23 | !/example/*/*.md 24 | !/example/*/*.toml 25 | 26 | #vendor 27 | /vendor/* 28 | !/vendor/vendor.json 29 | 30 | # log file 31 | 32 | # config file 33 | 34 | # img4go 35 | /img4go/vendor/* 36 | !/img4go/vendor/vendor.json 37 | 38 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # 参与者公约 2 | 3 | ## 我们的保证 4 | 5 | 为了促进一个开放透明且友好的环境，我们作为贡献者和维护者保证：无论年龄、种族、民族、性别认同和表达（方式）、体型、身体健全与否、经验水平、国籍、个人表现、宗教或性别取向，参与者在我们项目和社区中都免于骚扰。 6 | 7 | ## 我们的标准 8 | 9 | 有助于创造正面环境的行为包括但不限于： 10 | * 使用友好和包容性语言 11 | * 尊重不同的观点和经历 12 | * 耐心地接受建设性批评 13 | * 关注对社区最有利的事情 14 | * 友善对待其他社区成员 15 | 16 | 身为参与者不能接受的行为包括但不限于： 17 | * 使用与性有关的言语或是图像，以及不受欢迎的性骚扰 18 | * 捣乱/煽动/造谣的行为或进行侮辱/贬损的评论，人身攻击及政治攻击 19 | * 公开或私下的骚扰 20 | * 未经许可地发布他人的个人资料，例如住址或是电子地址 21 | * 其他可以被合理地认定为不恰当或者违反职业操守的行为 22 | 23 | ## 我们的责任 24 | 25 | 项目维护者有责任为「可接受的行为」标准做出诠释，以及对已发生的不被接受的行为采取恰当且公平的纠正措施。 26 | 27 | 项目维护者有权利及责任去删除、编辑、拒绝与本行为标准有所违背的评论(comments)、提交(commits)、代码、wiki 编辑、问题(issues)和其他贡献，以及项目维护者可暂时或永久性的禁止任何他们认为有不适当、威胁、冒犯、有害行为的贡献者。 28 | 29 | ## 使用范围 30 | 31 | 当一个人代表该项目或是其社区时，本行为标准适用于其项目平台和公共平台。 32 | 33 | 代表项目或是社区的情况，举例来说包括使用官方项目的电子邮件地址、通过官方的社区媒体账号发布或线上或线下事件中担任指定代表。 34 | 35 | 该项目的呈现方式可由其项目维护者进行进一步的定义及解释。 36 | 37 | ## 强制执行 38 | 39 | 可以通过wucc@ctrip.com，来联系项目团队来举报滥用、骚扰或其他不被接受的行为。 40 | 41 | 任何维护团队认为有必要且适合的所有投诉都将进行审查及调查，并做出相对应的回应。项目小组有对事件回报者有保密的义务。具体执行的方针近一步细节可能会单独公布。 42 | 43 | 没有切实地遵守或是执行本行为标准的项目维护人员，可能会因项目领导人或是其他成员的决定，暂时或是永久地取消其参与资格。 44 | 45 | ## 来源 46 | 47 | 本行为标准改编自[贡献者公约][主页]，版本 1.4 48 | 可在此观看https://www.contributor-covenant.org/zh-cn/version/1/4/code-of-conduct.html 49 | 50 | [主页]: https://www.contributor-covenant.org 51 | 52 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | 2 | Contributing to Nephele 3 | ============================= 4 | 5 | 开源是一件很有意义的事，让我们结交志同道合的新朋友，学到意想不到的新知识。我们非常愿意见到更多的开发者加入到Nephele社区的大家庭。在这里首先，我们为每一位已经和即将参与到Nephele项目的开发者献上诚挚的敬意。志在云霄天庭，感谢路上有你。 6 | 7 | 下面的内容会是一些介绍如何为社区做贡献的文字或链接，其中开发者公约是必读的，余下的可以根据自己的需求选读。 8 | 9 | Code of Conduct 10 | ----------- 11 | [开发者公约](https://github.com/ctripcorp/nephele/blob/master/CODE_OF_CONDUCT.md) 12 | 13 | Report Bugs 14 | ----------- 15 | 如果你在阅读或使用过程中发现Nephele存在BUG，那么你要做的第一步应该是确认一个这个BUG是否已经在最新版本中被修复了。如果没有，那么你还要做一步，那就是在[问题列表](https://github.com/ctripcorp/nephele/issues)中搜索一下，看看有没有类似的已经打开的issue。 16 | 17 | 如果还是没有，那么你可以通过[Open a new issue](https://github.com/ctripcorp/nephele/issues/new)告诉我们你的发现。 18 | 19 | #### 如何更好的(在issue中)描述一个BUG 20 | 21 | * 你需要一个「简洁清晰的标题」。简洁的意思很简单，在于字数不要太多。而清晰的意义则在于这个标题可以让这个BUG比较容易的区别于其他BUG(而不是强求每一个寥寥数字的标题都能够完整的概括一个BUG)。 22 | 23 | * 你最好能够复现BUG，并给出具体的「环境」与「步骤」。比如，你是在什么系统上运行的，你做了怎样的配置，用什么样的命令启动了Nephele，调用了什么功能，又或者你用一些间接的方式触发了怎样的环境变化，等等，越具体越好。有时候生硬的文字描述可以不如代码来的直观，所以你也可以把这些步骤做成测试脚本并附以相应的文档说明上传至github上并把链接提供给我们。 24 | 25 | * 你可以简单描述一下具体步骤做完之后，你「预期的结果」，并给出你这样预期的根据。(比如你可以附上某个文档的链接，并告诉我们这个文档是这样这样写的。) 26 | 27 | * 你最好详细描述一下具体步骤做完之后，「实际发生的结果」。这包括与你使用的功能点直接相关的结果，也包括你观察到的一些间接的状态(比如CPU的占用率等等)。 28 | 29 | * 对于这个BUG，你最好能有「自己的观点」。如果是面对一些令人头疼的现象，你可以做一些猜测。当然如果你已经望眼欲穿了，那就直截了当的告诉我们为什么好了。 30 | 31 | * 会有一些BUG是用固定的模板无法描述的，可能你也无法用具体的步骤复现。如果是那样的话，也尽可能的按照[问题模板](https://github.com/ctripcorp/nephele/blob/master/ISSUE_TEMPLATE.md)来提交吧。 32 | 33 | Suggesting Enhancements 34 | ----------------------- 35 | 36 | -------------------------------------------------------------------------------- /DESIGN.md: -------------------------------------------------------------------------------- 1 | * [一、总体设计](#一、总体设计) 2 | * [1.1 基础模型](#基础模型) 3 | * [1.2 架构模块](#架构模块) 4 | * [1.3 各模块概要介绍](#各模块概要介绍) 5 | * [二、图片服务设计](#二、图片服务设计) 6 | * [三、性能和稳定性](#三、性能和稳定性) 7 | * [四、监控相关](#四、监控相关) 8 | 9 | # 一、总体设计 10 | 11 | ## 1.1 基础模型 12 | ![basic-architecture](https://github.com/ctripcorp/nephele/blob/master/docs/images/nephele_basic.png) 13 | 14 | 如果需要把图片服务暴露到公网，我们建议首要考虑引入CDN，因为国内CDN不仅能把回源率降到5%以下，这样做能极大减小对服务器的压力，并且也能解决用户最后一公里问题。 15 | 16 | ## 1.2 架构模块 17 | ![overall-architecture](https://github.com/ctripcorp/nephele/blob/master/docs/images/nephele_overall.png) 18 | 19 | ## 1.3 各模块概要介绍 20 | 21 | ### 1.3.1 Workbench 22 | 23 | * 提供在线图片管理 24 | * 提供在线图片编辑 25 | * 图片测试和监控 26 | 27 | ### 1.3.2 ImageService 28 | 29 | * 提供所有图片相关api 30 | * 支持认证，限流，跨域，原图保护，样式解析等功能。 31 | * 依赖imgcore模块对图片进行处理 32 | 33 | ### 1.3.3 ImageCore 34 | 35 | * 支持裁剪，缩放，旋转，转换，滤镜，水印 36 | * 底层依赖GraphicsMagick，ImageMagick库。 37 | 38 | ### 1.3.4 存储 39 | 40 | * 支持多种存储，本地文件，FastDFS，阿里云OSS，AWS S3 41 | 42 | ### 1.3.5 监控 43 | 44 | * 所有监控异常日志落本地，通过logagent转发到ES集群上。 45 | * 通过Workbench查询聚合数据做定制展示和告警。 46 | 47 | 48 | # 二、图片服务设计 49 | 50 | 下面展示了图片服务各模块之间的关系. 51 | 52 | ![overall-architecture](https://github.com/ctripcorp/nephele/blob/master/docs/images/nephele_module.png) 53 | 54 | 1. 图片服务包含多个过滤器，分别提供限流，认证，跨域，缓存时效，埋点等功能。 55 | 2. 采用异步模式处理图片，发送队列用于控制图片处理并行度，返回队列用于阻塞图片处理响应直到超时时间过期，启动快速失败想响应客户端。 56 | 3. 多协程处理图片，协程数相当于CPU核数，这样做能够最大程度有效利用CPU。 57 | 4. 核心图片处理模块提供对GM，IM组件的封装，并加入 58 | 5. 提供多种存储Driver，支持多个主流媒体存储服务。 59 | 60 | 61 | 62 | # 三、性能和稳定性 63 | 64 | 65 | 66 | # 四、监控相关 67 | 68 | Nephele采用本地磁盘写日志的方式记录异常以及监控埋点，这样做的好处是通过格式化日志的方式可以编写不同Logagent实现向不同监控体系传输对应数据，能够对图片服务内部的代码侵入性做到最低，并且使用基于本地持久化的技术也能够最大程度保证日志数据不丢，在携程内部，我们基于CAT做性能调优和监控，开源版本我们将采用更通用的ES作为监控手段。 69 | -------------------------------------------------------------------------------- /GO_STYLE_GUIDE.md: -------------------------------------------------------------------------------- 1 | # Nephele团队代码风格指南。 2 | 3 | ## 背景 4 | 5 | Nephele是一套企业级的图片解决方案。Go是Nephele团队主要使用的编程语言。Go语言简单，强大。其官方提供了大量值得参考遵循的规范。但Go始终是一门相对年轻，受众相对小的语言，会在一部分场景因缺乏相应约束造成代码可读性低下，形成学习壁垒，阻碍团队效率的进一步提升。 6 | 7 | ## 目标 8 | 9 | 一部分的目标不会有具体实例，但它们将作为一种理念被保留。 10 | 11 | **与go fmt tool无冲突** 12 | 13 | 遵循该指南的代码经由go fmt tool格式化之后将依然遵循该指南。 14 | 15 | **更在乎阅读者的体验** 16 | 17 | 可以要求开发者完成一些额外的编码工作，这些工作对于功能的完成而言可能是不必要的，但有利于阅读者的学习。 18 | 19 | **代码拒绝小聪明** 20 | 21 | 将提倡直白的编码方式。 22 | 23 | **注释拒绝浪漫** 24 | 25 | 将提倡严肃呆板的注释风格。 26 | 27 | **清晰的作用范围** 28 | 29 | 可以通过制定命名的约定，限制变量和函数的调用与声明，达到可以快速判断变量函数出处的目的。 30 | 31 | **最小化对于接口的冲击** 32 | 33 | 现有工程的改造需要一个循序渐进的过程。 34 | 该指南提供的规则将尽可能达到这样的效果： 35 | 不遵循该指南的代码被更改为遵循该指南之后，其调用方的代码结构无需变动。 36 | 37 | **顺从不可避免的性能优化** 38 | 39 | 一些引发质变的性能优化可能引入部分含蓄的代码，这些代码是被允许并鼓励的。 40 | 41 | ## 目录 42 | 43 | * [Import Package](#import-package) 44 | 45 | * [Package](#package) 46 | 47 | * [Indent](#indent) 48 | 49 | * [Blank line](#blank-line) 50 | 51 | * [Naming](#naming) 52 | 53 | * [Variables](#variables) 54 | 55 | * [Functions](#functions) 56 | 57 | * [Lock](#lock) 58 | 59 | 60 | ## Import Package 61 | 62 | **For example:** 63 | ```go 64 | import ( 65 | "time" 66 | "net/http" 67 | ) 68 | ``` 69 | 70 | **Not:** 71 | ```go 72 | import "time" 73 | import "net/http" 74 | ``` 75 | 76 | ## Package 77 | 78 | 一个包下面必须有一个与包同名的文件。为了方便，这里称之为主文件(master file)，其他的与包不同名的文件，这里称之为分支文件(branch file)。 79 | 80 | ## Indent 81 | 82 | 缩进请务必使用tab。 83 | 84 | ## Blank line 85 | 86 | 在变量(variable)，方法(function)，接口(interface)，结构体(struct)之间插入空行。 87 | 88 | **For example:** 89 | ```go 90 | var a int 91 | 92 | func b() { 93 | ... 94 | } 95 | 96 | type c struct { 97 | ... 98 | } 99 | ``` 100 | 101 | **Not:** 102 | ```go 103 | var a int 104 | func b() { 105 | ... 106 | } 107 | type c struct { 108 | ... 109 | } 110 | ``` 111 | 112 | ## Naming 113 | 114 | 包名全部小写。 115 | 其他使用驼峰命名法，并有一些额外的要求： 116 | 117 | **尽可能少的介词:** 118 | ```go 119 | func SetUsername() { 120 | ... 121 | } 122 | 123 | func (name *Username) String() string { 124 | ... 125 | } 126 | ``` 127 | 128 | **Not:** 129 | ```go 130 | func SetNameOfUser() { 131 | ... 132 | } 133 | 134 | func (name *Username) ToString() string { 135 | ... 136 | } 137 | ``` 138 | 139 | **既可以视为两个单词又可以视为一个单词的视为一个单词，以谷歌翻译为准:** 140 | ```go 141 | func SetUsername() { 142 | ... 143 | } 144 | 145 | func SetClientCode() { 146 | ... 147 | } 148 | ``` 149 | 150 | **Not:** 151 | ```go 152 | func SetUserName() { 153 | ... 154 | } 155 | 156 | func SetClientcode() { 157 | ... 158 | } 159 | ``` 160 | 161 | **动词在名词之前:** 162 | ```go 163 | func SetName() { 164 | ... 165 | } 166 | ``` 167 | 168 | **Not:** 169 | ```go 170 | func NameSet() { 171 | ... 172 | } 173 | ``` 174 | 175 | 形参使用完整词，函数内可以使用简写变量: 176 | 177 | **Example:** 178 | ```go 179 | func SetUsername(username string) { 180 | u := username 181 | ... 182 | } 183 | 184 | func SetClientcode(clientCode string) { 185 | cc := clientCode 186 | ... 187 | } 188 | ``` 189 | 190 | **Not:** 191 | ```go 192 | func SetUsername(u string) { 193 | ... 194 | } 195 | 196 | func SetClientcode(cc string) { 197 | ... 198 | } 199 | ``` 200 | 201 | 字母变量尽可能使用原变量名首字母组合，避免由类型演变的简写: 202 | 203 | **Not:** 204 | ```go 205 | func SetUsername(username string) { 206 | var s string 207 | ... 208 | } 209 | 210 | func SetUsername(username string) { 211 | var str string 212 | ... 213 | } 214 | 215 | ``` 216 | 217 | 面对类型转换或数据解析时需要区分类型或格式的，将类型或格式完整名作为后缀， 218 | 这里建议一个数据都是在同一类型或格式下参与业务计算，该类型或格式下的数据变量名可以没有后缀： 219 | 220 | **Example:** 221 | ```go 222 | //var usernameBytes []byte 223 | username := string(usernameBytes) 224 | //do something with username 225 | 226 | //var dataJson []byte 227 | //var data Data 228 | //dataJson := json.Marshal(data) 229 | json.Unmarshal(dataJson, &data) 230 | //do something with data 231 | ``` 232 | 233 | ## Variables 234 | 235 | 禁止在分支文件中声明公用变量。 236 | 237 | **For example:** 238 | ```go 239 | package foo 240 | //foo.go 241 | 242 | var A int 243 | ``` 244 | 245 | **Banned:** 246 | ```go 247 | package foo 248 | //goo.go 249 | 250 | var A int 251 | ``` 252 | 253 | 在分支文件中, 全局私有变量名必须带上与分之文件名相同的前缀。该前缀不视为修饰词，前缀之后的单词首字母需要大写。 254 | 255 | **For example:** 256 | ```go 257 | package foo 258 | //goo.go 259 | 260 | var gooA int 261 | const gooB = 1 262 | ``` 263 | 264 | **Not:** 265 | ```go 266 | package foo 267 | //goo.go 268 | 269 | var a int 270 | const b = 1 271 | ``` 272 | 273 | **Not:** 274 | ```go 275 | package foo 276 | //goo.go 277 | 278 | var gooa int 279 | const goob = 1 280 | ``` 281 | 282 | 在实践中我们会发现, 分支文件命名的好坏和代码可读性强弱有着相当大的关系。可以说分之文件名是需要着重设计的。 283 | 284 | ## Functions 285 | 286 | 非方法的公有函数只能在主文件中定义。 287 | 288 | **Allowed:** 289 | ```go 290 | package foo 291 | //foo.go 292 | 293 | func A() { 294 | ... 295 | } 296 | ``` 297 | 298 | **Banned:** 299 | ```go 300 | package foo 301 | //goo.go 302 | 303 | func B() { 304 | ... 305 | } 306 | ``` 307 | 308 | 在分支文件中, 非方法的私有函数名必须带上与分之文件名相同的前缀。该前缀不视为修饰词，前缀之后的单词首字母需要大写。 309 | 原本在分支文件中定义的函数，通常会以如下形式再现。 310 | 311 | **For example:** 312 | ```go 313 | package foo 314 | //goo.go 315 | 316 | type goo struct { 317 | a int 318 | } 319 | 320 | const gooA = 0 321 | const gooB = 1 322 | 323 | var gooC int = 2 324 | var gooD int = 3 325 | 326 | var gooInstance *goo = &goo{} 327 | 328 | func (g goo) step1() { 329 | ... 330 | } 331 | 332 | func (g goo) step2() { 333 | ... 334 | } 335 | 336 | func (g goo) do() { 337 | g.step1() 338 | g.step2() 339 | } 340 | ``` 341 | 342 | **or:** 343 | ```go 344 | package foo 345 | //goo.go 346 | 347 | func gooStep1() { 348 | ... 349 | } 350 | 351 | func gooStep2() { 352 | ... 353 | } 354 | 355 | func gooDo() { 356 | gooStep1() 357 | gooStep2() 358 | } 359 | ``` 360 | 361 | **Not:** 362 | ```go 363 | package foo 364 | //goo.go 365 | 366 | func step1() { 367 | ... 368 | } 369 | 370 | func step2() { 371 | ... 372 | } 373 | 374 | func do() { 375 | step1() 376 | step2() 377 | } 378 | ``` 379 | 380 | ## Lock 381 | 382 | 用go sdk提供的sync包。别用channel。 383 | 384 | **Example:** 385 | ```go 386 | var fooLock sync.Mutex 387 | 388 | func foo() { 389 | fooLock.Lock() 390 | ... 391 | fooLock.Unlock() 392 | } 393 | ``` 394 | 395 | **Not:** 396 | ```go 397 | var fooLock chan int = make(chan int, 1) 398 | 399 | func foo() { 400 | fooLock<-0 401 | ... 402 | <-fooLock 403 | } 404 | ``` 405 | -------------------------------------------------------------------------------- /ISSUE_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | ### 描述 2 | 3 | [大致描述一下这个问题，包括你的一些猜想和理解，也可以谈谈你的一些与之相关的理念] 4 | 5 | ### 重现步骤 6 | 7 | 1. [第一步] 8 | 2. [第二步] 9 | 3. [然后...] 10 | 11 | **预期的情况:** [本应该发生什么] 12 | 13 | **实际的情况:** [然而却发生了什么] 14 | 15 | **这个问题出现的频率:** [这个现象是稳定出现的还是偶尔出现的，如果是偶尔的话那么出现频繁么，有多频繁] 16 | 17 | ### 版本 18 | 19 | [这个可以附上Nephele的版本号，如果你觉得必要还可以带上GO语言或者操作系统的版本信息] 20 | 21 | ### 补充 22 | 23 | [一些你觉得还需要告诉我们的重要信息] 24 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Nephele 2 | 3 | [English](https://github.com/ctripcorp/nephele/blob/master/docs/en/README.md) [C1](https://github.com/ctripcorp/nephele/blob/master/docs/cn/REQUIREMENT_CLASS.md) 4 | 5 | ## 简介 6 | 7 | 2015年，携程框架研发部启动了Nephele工程，以应对公司内部的图片相关场景。现已开源。 8 | 9 | **项目包括:** 10 | 11 | * 现成的简单图片服务。它具备了市面上大部分简单或复杂的功能，并可以快速部署启动。 12 | * 定制的图片服务。通过更换插件以应对各种复杂的应用场景。 13 | * 清晰的管理界面。在管理界面中，用户可以编辑图片处理的流程，开发者可以对服务状态实行监控。 14 | * 实用的小工具。可以低成本解决诸多实际应用中遇到的困难。 15 | 16 | ## 功能 17 | 18 | **你最关心的功能:** 19 | 20 | * 基础图片处理 21 | * 多存储支持 22 | * 监控与日志 23 | 24 | **其他功能:** 25 | 26 | * 数字水印 27 | * 调优的图片编码 28 | 29 | ## 指导 30 | 31 | 如果你没有明确自己需要什么，参阅[Getting Started]。 32 | 33 | 需要快速启动现成的图片服务并了解其相关功能的，参阅[The Example Nephele Program](https://github.com/ctripcorp/nephele/blob/master/example/README.md)。 34 | 35 | 示例图片服务不能满足全部需求，需要自定化的，参阅[Pluggable Framework of Nephele]。 36 | 37 | 现成的插件无法满足需求，需要开发插件的，参阅[How to Deploy A Plugin for Nephele]。 38 | 39 | Nephele社区非常欢迎你，[Contributing](https://github.com/ctripcorp/nephele/blob/master/CONTRIBUTING.md)了解一下。 40 | 41 | ## License 42 | 43 | Nephele is licensed under the Apache License 2.0. See [LICENSE](https://github.com/ctripcorp/nephele/blob/master/LICENSE) for the full license text. 44 | 45 | -------------------------------------------------------------------------------- /command/autoorient/autoorient.go: -------------------------------------------------------------------------------- 1 | package autoorient 2 | 3 | import ( 4 | "github.com/ctripcorp/nephele/command" 5 | ) 6 | 7 | func init() { 8 | command.Register("autoorient", func() command.Command { 9 | return &Command{} 10 | }) 11 | } 12 | -------------------------------------------------------------------------------- /command/autoorient/command.go: -------------------------------------------------------------------------------- 1 | package autoorient 2 | 3 | import ( 4 | "context" 5 | 6 | "github.com/ctrip-nephele/gmagick" 7 | ) 8 | 9 | type Command struct { 10 | AutoOrient string 11 | } 12 | 13 | const ( 14 | commandKeyA string = "autoOrient" 15 | ) 16 | 17 | func (c *Command) Support() string { 18 | return "wand" 19 | } 20 | 21 | //Verify autoorient verify 22 | func (c *Command) Verify(ctx context.Context, params map[string]string) error { 23 | //log.Debugf(ctx, "autoOrient verification") 24 | return nil 25 | } 26 | 27 | func (c *Command) ExecuteOnBlob(ctx context.Context, blob []byte) ([]byte, error) { 28 | return nil, nil 29 | } 30 | 31 | //Exec autoorient exec 32 | func (c *Command) ExecuteOnWand(ctx context.Context, wand *gmagick.MagickWand) error { 33 | //log.TraceBegin(ctx, "", "URL.Command", "autoOrient", "") 34 | //defer log.TraceEnd(ctx, nil) 35 | orientation := wand.GetImageOrientation() 36 | return wand.AutoOrientImage(orientation) 37 | } 38 | -------------------------------------------------------------------------------- /command/command.go: -------------------------------------------------------------------------------- 1 | package command 2 | 3 | import ( 4 | "context" 5 | "github.com/ctrip-nephele/gmagick" 6 | ) 7 | 8 | var list map[string]func() Command 9 | 10 | type Command interface { 11 | Support() string 12 | Verify(context.Context, map[string]string) error 13 | ExecuteOnBlob(context.Context, []byte) ([]byte, error) 14 | ExecuteOnWand(context.Context, *gmagick.MagickWand) error 15 | } 16 | 17 | func List() map[string]func() Command { 18 | return list 19 | } 20 | 21 | func Register(name string, command func() Command) { 22 | if list == nil { 23 | list = make(map[string]func() Command) 24 | } 25 | if _, ok := list[name]; ok { 26 | panic("processer name conflict") 27 | } 28 | list[name] = command 29 | } 30 | 31 | var ErrorInvalidOptionFormat = "invalid %s option: %v" 32 | -------------------------------------------------------------------------------- /command/crop/command.go: -------------------------------------------------------------------------------- 1 | package crop 2 | 3 | import ( 4 | "errors" 5 | "fmt" 6 | "math" 7 | "strconv" 8 | 9 | "context" 10 | 11 | "github.com/ctrip-nephele/gmagick" 12 | "github.com/ctripcorp/nephele/command" 13 | ) 14 | 15 | type Command struct { 16 | Width uint 17 | Height uint 18 | Method string 19 | X, Y int 20 | Limit bool 21 | Percentage int 22 | } 23 | 24 | const ( 25 | commandKeyW string = "w" 26 | commandKeyH string = "h" 27 | commandKeyM string = "m" 28 | commandKeyP string = "p" 29 | commandKeyX string = "x" 30 | commandKeyY string = "y" 31 | commandKeyLimit string = "limit" 32 | ) 33 | 34 | const ( 35 | commandKeyMT string = "t" 36 | commandKeyMB string = "b" 37 | commandKeyML string = "l" 38 | commandKeyMR string = "r" 39 | commandKeyMWC string = "wc" 40 | commandKeyMHC string = "hc" 41 | commandKeyMC string = "c" 42 | commandKeyMRESIZE string = "resize" 43 | commandKeyMCROP string = "crop" 44 | ) 45 | 46 | func (c *Command) Support() string { 47 | return "wand" 48 | } 49 | 50 | //Verify crop Verify 51 | func (c *Command) Verify(ctx context.Context, params map[string]string) error { 52 | //log.Debugf(ctx, "crop verification") 53 | for k, v := range params { 54 | if k == commandKeyW { 55 | width, e := strconv.Atoi(v) 56 | if e != nil { 57 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 58 | } 59 | c.Width = uint(width) 60 | } 61 | if k == commandKeyH { 62 | height, e := strconv.Atoi(v) 63 | if e != nil { 64 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 65 | } 66 | c.Height = uint(height) 67 | } 68 | if k == commandKeyM { 69 | if v != commandKeyMT && v != commandKeyMB && v != commandKeyMC && v != commandKeyMCROP && 70 | v != commandKeyMHC && v != commandKeyML && v != commandKeyMR && v != commandKeyMRESIZE && v != commandKeyMWC { 71 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 72 | } 73 | c.Method = v 74 | } 75 | if k == commandKeyLimit { 76 | if v != "0" && v != "1" { 77 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 78 | } 79 | c.Limit = v == "1" 80 | } 81 | if k == commandKeyP { 82 | p, e := strconv.Atoi(v) 83 | if e != nil || p < 0 || p > 10000 { 84 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 85 | } 86 | c.Percentage = p 87 | } 88 | if k == commandKeyX { 89 | x, e := strconv.Atoi(v) 90 | if e != nil { 91 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 92 | } 93 | c.X = x 94 | } 95 | if k == commandKeyY { 96 | y, e := strconv.Atoi(v) 97 | if e != nil { 98 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 99 | } 100 | c.Y = y 101 | } 102 | } 103 | if c.Percentage < 0 || c.Percentage >= 100 { 104 | return fmt.Errorf(command.ErrorInvalidOptionFormat, commandKeyP, c.Percentage) 105 | } 106 | if (c.Method == commandKeyMT || c.Method == commandKeyMB || c.Method == commandKeyMHC) && 107 | c.Height < 1 && c.Percentage < 1 { 108 | return errors.New("m,h,p is invalid.") 109 | } 110 | if (c.Method == commandKeyML || c.Method == commandKeyMR || c.Method == commandKeyMWC) && 111 | c.Width < 1 && c.Percentage < 1 { 112 | return errors.New("m,w,p is invalid.") 113 | } 114 | if c.Method == commandKeyMC && c.Percentage < 1 && c.Width < 1 && c.Height < 1 { 115 | return errors.New("m,w,h,p is invalid.") 116 | } 117 | if (c.Method == commandKeyMRESIZE || c.Method == commandKeyMCROP) && 118 | c.Percentage < 1 && (c.Width < 1 || c.Height < 1) { 119 | return errors.New("m,w,h,p is invalid.") 120 | } 121 | return nil 122 | } 123 | 124 | func (c *Command) ExecuteOnBlob(ctx context.Context, blob []byte) ([]byte, error) { 125 | return nil, nil 126 | } 127 | 128 | //Exec crop exec 129 | func (c *Command) ExecuteOnWand(ctx context.Context, wand *gmagick.MagickWand) error { 130 | var err error 131 | //log.TraceBegin(ctx, "", "URL.Command", "crop", "method", c.Method, "width", c.Width, "height", c.Height, "x", c.X, "y", c.Y) 132 | //defer log.TraceEnd(ctx, err) 133 | 134 | srcW, srcH := wand.GetImageWidth(), wand.GetImageHeight() 135 | var width, height uint 136 | var x, y int 137 | if c.Method == commandKeyMRESIZE { 138 | var isResize bool 139 | width, height, x, y, isResize = cropMRE(c.Width, c.Height, srcW, srcH, c.Limit) 140 | if width == 0 && height == 0 && x == 0 && y == 0 { 141 | return nil 142 | } 143 | fmt.Println(width, height, x, y, isResize) 144 | if !(width == srcW && height == srcH && x == 0 && y == 0) { 145 | if err = wand.CropImage(width, height, x, y); err != nil { 146 | return err 147 | } 148 | if !isResize { 149 | return nil 150 | } 151 | if err = wand.ResizeImage(width, height, gmagick.FILTER_LANCZOS, 1.0); err != nil { 152 | return err 153 | } 154 | } 155 | return nil 156 | } 157 | 158 | switch c.Method { 159 | case commandKeyMB: 160 | width, height, x, y = cropMB(c.Height, srcW, srcH, c.Percentage) 161 | case commandKeyMC: 162 | width, height, x, y = cropMC(c.Width, c.Height, srcW, srcH, c.Percentage) 163 | case commandKeyMCROP: 164 | width, height = cropMCR(c.Width, c.Height, srcW, srcH, c.Percentage) 165 | x, y = c.X, c.Y 166 | case commandKeyMHC: 167 | width, height, x, y = cropMHC(c.Height, srcW, srcH, c.Percentage) 168 | case commandKeyML: 169 | width, height, x, y = cropML(c.Width, srcW, srcH, c.Percentage) 170 | case commandKeyMR: 171 | width, height, x, y = cropMR(c.Width, srcW, srcH, c.Percentage) 172 | case commandKeyMT: 173 | width, height, x, y = cropMT(c.Height, srcW, srcH, c.Percentage) 174 | case commandKeyMWC: 175 | width, height, x, y = cropMWC(c.Width, srcW, srcH, c.Percentage) 176 | } 177 | if width < 1 || height < 1 || x >= int(srcW) || y >= int(srcH) { 178 | err = errors.New("param is invalid.") 179 | return err 180 | } 181 | err = wand.CropImage(width, height, x, y) 182 | return err 183 | } 184 | 185 | // crop,m_resize,w_100,h_100 , first equal comparison cut image, finally resize 186 | func cropMRE(w, h, srcW, srcH uint, limit bool) (width, height uint, x, y int, resize bool) { 187 | resize = false 188 | if !limit && w > srcW && h > srcH { 189 | return 190 | } 191 | 192 | width = srcW 193 | height = srcH 194 | if !limit && !(w < srcW && h < srcH) { 195 | if srcW > w { 196 | x = int((srcW - w) / 2) 197 | width = w 198 | return 199 | } 200 | if srcH > h { 201 | y = int((srcH - h) / 2) 202 | height = h 203 | return 204 | } 205 | } 206 | resize = true 207 | dstP := float64(w) / float64(h) 208 | srcP := float64(srcW) / float64(srcH) 209 | if math.Abs(dstP-srcP) > 0.0001 { 210 | if srcP > dstP { //以高缩小 211 | height = srcH 212 | width = uint(math.Floor(float64(height) * dstP)) 213 | x = int((srcW - width) / 2) 214 | } 215 | if srcP < dstP { //以宽缩小 216 | width = srcW 217 | height = uint(math.Floor(float64(width) / dstP)) 218 | y = int((srcH - height) / 2) 219 | } 220 | } 221 | return 222 | } 223 | 224 | //crop,m_t,h_100 crop top height 100 225 | func cropMT(h, srcW, srcH uint, p int) (width, height uint, x, y int) { 226 | if p > 0 { 227 | h = srcH * uint(p) / 100 228 | } 229 | width = srcW 230 | height = srcH - h 231 | y = int(h) 232 | return 233 | } 234 | 235 | //crop,m_b,h_100 , crop bottom 236 | func cropMB(h, srcW, srcH uint, p int) (width, height uint, x, y int) { 237 | if p > 0 { 238 | h = srcH * uint(p) / 100 239 | } 240 | width = srcW 241 | height = srcH - h 242 | return 243 | } 244 | 245 | //crop,m_hc,h_100 , cut the upper and lower sides 246 | func cropMHC(h, srcW, srcH uint, p int) (width, height uint, x, y int) { 247 | if p > 0 { 248 | h = srcH * uint(p) / 100 249 | } 250 | width = srcW 251 | height = srcH - h 252 | y = int(h) / 2 253 | return 254 | } 255 | 256 | //crop,m_l,w_100, crop left 257 | func cropML(w, srcW, srcH uint, p int) (width, height uint, x, y int) { 258 | if p > 0 { 259 | w = srcW * uint(p) / 100 260 | } 261 | width = srcW - w 262 | height = srcH 263 | x = int(w) 264 | return 265 | } 266 | 267 | //crop,m_r,w_100, crop right 268 | func cropMR(w, srcW, srcH uint, p int) (width, height uint, x, y int) { 269 | if p > 0 { 270 | w = srcW * uint(p) / 100 271 | } 272 | width = srcW - w 273 | height = srcH 274 | return 275 | } 276 | 277 | //crop,m_wc,w_100, cut the left and right sides 278 | func cropMWC(w, srcW, srcH uint, p int) (width, height uint, x, y int) { 279 | if p > 0 { 280 | w = srcW * uint(p) / 100 281 | } 282 | width = srcW - w 283 | height = srcH 284 | x = int(w) / 2 285 | return 286 | } 287 | 288 | //crop,m_c,w_100 cut image for center 289 | func cropMC(w, h, srcW, srcH uint, p int) (width, height uint, x, y int) { 290 | if p > 0 { 291 | w = srcW * uint(p) / 100 292 | h = srcH * uint(p) / 100 293 | } 294 | 295 | width, height = srcW, srcH 296 | if srcW > w && w != 0 { 297 | width = w 298 | x = int((srcW - w) / 2) 299 | } 300 | if srcH > h && h != 0 { 301 | height = h 302 | y = int((srcH - h) / 2) 303 | } 304 | return 305 | } 306 | 307 | // crop,m_crop,w_100, crop cut image 308 | func cropMCR(w, h, srcW, srcH uint, p int) (width, height uint) { 309 | if p > 0 { 310 | w = srcW * uint(p) / 100 311 | h = srcH * uint(p) / 100 312 | } 313 | if w == 0 { 314 | w = srcW 315 | } 316 | if h == 0 { 317 | h = srcH 318 | } 319 | width = w 320 | height = h 321 | return 322 | } 323 | -------------------------------------------------------------------------------- /command/crop/crop.go: -------------------------------------------------------------------------------- 1 | package crop 2 | 3 | import ( 4 | "github.com/ctripcorp/nephele/command" 5 | ) 6 | 7 | func init() { 8 | command.Register("crop", func() command.Command { 9 | return &Command{} 10 | }) 11 | } 12 | -------------------------------------------------------------------------------- /command/format/command.go: -------------------------------------------------------------------------------- 1 | package format 2 | 3 | import ( 4 | "context" 5 | "fmt" 6 | "github.com/ctrip-nephele/gmagick" 7 | "github.com/ctripcorp/nephele/command" 8 | "github.com/ctripcorp/nephele/util" 9 | ) 10 | 11 | type Command struct { 12 | format string 13 | } 14 | 15 | func (c *Command) Support() string { 16 | return "wand" 17 | } 18 | 19 | func (c *Command) Verify(ctx context.Context, option map[string]string) error { 20 | //log.Debugf(ctx, "format verification") 21 | for k, v := range option { 22 | if k == "v" { 23 | if !util.InArray(v, []string{"jpg", "png", "webp", "gif"}) { 24 | return fmt.Errorf(command.ErrorInvalidOptionFormat, "format", option) 25 | } 26 | c.format = v 27 | return nil 28 | } 29 | } 30 | return fmt.Errorf(command.ErrorInvalidOptionFormat, "format", option) 31 | } 32 | 33 | func (c *Command) ExecuteOnBlob(ctx context.Context, blob []byte) ([]byte, error) { 34 | return nil, nil 35 | } 36 | 37 | func (c *Command) ExecuteOnWand(ctx context.Context, wand *gmagick.MagickWand) error { 38 | var err error 39 | //log.TraceBegin(ctx, "execute on wand", "url.command", "format", "format", c.format) 40 | //defer log.TraceEnd(ctx, err) 41 | err = wand.SetImageFormat(c.format) 42 | return err 43 | } 44 | -------------------------------------------------------------------------------- /command/format/format.go: -------------------------------------------------------------------------------- 1 | package format 2 | 3 | import ( 4 | "github.com/ctripcorp/nephele/command" 5 | ) 6 | 7 | func init() { 8 | command.Register("format", func() command.Command { 9 | return &Command{} 10 | }) 11 | } 12 | -------------------------------------------------------------------------------- /command/quality/command.go: -------------------------------------------------------------------------------- 1 | package quality 2 | 3 | import ( 4 | "fmt" 5 | "strconv" 6 | 7 | "context" 8 | 9 | "github.com/ctrip-nephele/gmagick" 10 | "github.com/ctripcorp/nephele/command" 11 | ) 12 | 13 | type Command struct { 14 | Quality uint 15 | } 16 | 17 | const ( 18 | commandKeyV string = "v" 19 | ) 20 | 21 | func (c *Command) Support() string { 22 | return "wand" 23 | } 24 | 25 | //Verify verify quality 26 | func (c *Command) Verify(ctx context.Context, params map[string]string) error { 27 | //log.Debugf(ctx, "quality verification") 28 | for k, v := range params { 29 | if k == commandKeyV { 30 | quality, e := strconv.Atoi(v) 31 | if e != nil || quality < 0 || quality > 100 { 32 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 33 | } 34 | c.Quality = uint(quality) 35 | return nil 36 | } 37 | } 38 | return fmt.Errorf(command.ErrorInvalidOptionFormat, "quality", params) 39 | } 40 | 41 | func (c *Command) ExecuteOnBlob(ctx context.Context, blob []byte) ([]byte, error) { 42 | return nil, nil 43 | } 44 | 45 | //Exec exec 46 | func (c *Command) ExecuteOnWand(ctx context.Context, wand *gmagick.MagickWand) error { 47 | var err error 48 | //log.TraceBegin(ctx, "", "URL.Command", "quality", "quality", q.Quality) 49 | //defer log.TraceEnd(ctx, err) 50 | if c.Quality == 0 { 51 | err = wand.SetImageOption("jpeg", "preserve-settings", "true") 52 | } else { 53 | err = wand.SetCompressionQuality(uint(c.Quality)) 54 | } 55 | return err 56 | } 57 | -------------------------------------------------------------------------------- /command/quality/quality.go: -------------------------------------------------------------------------------- 1 | package quality 2 | 3 | import ( 4 | "github.com/ctripcorp/nephele/command" 5 | ) 6 | 7 | func init() { 8 | command.Register("quality", func() command.Command { 9 | return &Command{} 10 | }) 11 | } 12 | -------------------------------------------------------------------------------- /command/resize/command.go: -------------------------------------------------------------------------------- 1 | package resize 2 | 3 | import ( 4 | "errors" 5 | "fmt" 6 | "math" 7 | "strconv" 8 | 9 | "context" 10 | 11 | "github.com/ctrip-nephele/gmagick" 12 | "github.com/ctripcorp/nephele/command" 13 | ) 14 | 15 | //Resize resize command 16 | type Command struct { 17 | Width uint 18 | Height uint 19 | Method string //lfit/fixed 20 | Limit bool 21 | Percentage int 22 | } 23 | 24 | func (c *Command) Support() string { 25 | return "wand" 26 | } 27 | 28 | const ( 29 | commandKeyW string = "w" 30 | commandKeyH string = "h" 31 | commandKeyM string = "m" 32 | commandKeyP string = "p" 33 | commandKeyLimit string = "limit" 34 | ) 35 | const ( 36 | commandKeyMFIXED string = "fixed" 37 | commandKeyMLFIT string = "lfit" 38 | ) 39 | 40 | //Verify resize Verify 41 | func (c *Command) Verify(ctx context.Context, params map[string]string) error { 42 | //log.Debugf(ctx, "resize verification") 43 | for k, v := range params { 44 | if k == commandKeyW { 45 | width, e := strconv.Atoi(v) 46 | if e != nil { 47 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 48 | } 49 | c.Width = uint(width) 50 | } 51 | if k == commandKeyH { 52 | height, e := strconv.Atoi(v) 53 | if e != nil { 54 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 55 | } 56 | c.Height = uint(height) 57 | } 58 | if k == commandKeyM { 59 | if v != commandKeyMFIXED && v != commandKeyMLFIT { 60 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 61 | } 62 | c.Method = v 63 | } 64 | if k == commandKeyLimit { 65 | if v != "0" && v != "1" { 66 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 67 | } 68 | c.Limit = v == "1" 69 | } 70 | if k == commandKeyP { 71 | p, e := strconv.Atoi(v) 72 | if e != nil || p < 0 || p > 10000 { 73 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 74 | } 75 | c.Percentage = p 76 | } 77 | } 78 | if c.Method == commandKeyMFIXED && c.Width < 1 && c.Height < 1 { 79 | return errors.New("m, w, h is invalid.") 80 | } 81 | if c.Width < 1 && c.Height < 1 && c.Percentage < 1 { 82 | return errors.New("w,h,p is invalid.") 83 | } 84 | return nil 85 | } 86 | 87 | //ExecuteOnBlob 88 | func (c *Command) ExecuteOnBlob(ctx context.Context, blob []byte) ([]byte, error) { 89 | return nil, nil 90 | } 91 | 92 | //Exec resize exec 93 | func (c *Command) ExecuteOnWand(ctx context.Context, wand *gmagick.MagickWand) error { 94 | var err error 95 | //log.TraceBegin(ctx, "", "URL.Command", "resize", "method", c.Method, "width", c.Width, "height", c.Height, "percentage", c.Percentage, "limit", c.Limit) 96 | //defer log.TraceEnd(ctx, err) 97 | if (c.Width > wand.GetImageWidth() && c.Height > wand.GetImageHeight() && !c.Limit) || 98 | (c.Method != commandKeyMFIXED && c.Percentage > 100 && !c.Limit) { 99 | return nil 100 | } 101 | srcW, srcH := wand.GetImageWidth(), wand.GetImageHeight() 102 | if srcW == c.Width && srcH == c.Height { 103 | return nil 104 | } 105 | var width, height uint 106 | if c.Method == commandKeyMFIXED { 107 | width, height = resizeFixed(c.Width, c.Height, srcW, srcH) 108 | } else if c.Percentage != 0 { 109 | width, height = resizePercentage(c.Percentage, srcW, srcH) 110 | } else { 111 | width, height = resizeLfit(c.Width, c.Height, srcW, srcH) 112 | } 113 | err = wand.ResizeImage(width, height, gmagick.FILTER_LANCZOS, 1.0) 114 | return err 115 | } 116 | 117 | //Lfit: isotropic scaling with fixed width and height, which tends to disable one of the inputs(width or height) to feed a larger aspect ratio 118 | func resizeLfit(dstW, dstH, srcW, srcH uint) (width, height uint) { 119 | //auto compute weight or height 120 | if dstW == 0 { 121 | width = dstH * srcW / srcH 122 | height = dstH 123 | return 124 | } 125 | if dstH == 0 { 126 | width = dstW 127 | height = dstW * srcH / srcW 128 | return 129 | } 130 | 131 | dstP := float64(dstW) / float64(dstH) 132 | srcP := float64(srcW) / float64(srcH) 133 | 134 | if srcP > dstP { 135 | width = dstW 136 | if uint(math.Abs(float64(dstH-srcH))) < 3 { 137 | height = dstH 138 | } else { 139 | height = uint(math.Floor(float64(dstW) / srcP)) 140 | } 141 | } else { 142 | height = dstH 143 | if uint(math.Abs(float64(dstW-srcW))) < 3 { 144 | width = dstW 145 | } else { 146 | width = uint(math.Floor(float64(dstH) * srcP)) 147 | } 148 | } 149 | return 150 | } 151 | 152 | //Fixed: forced scaling with fixed width and height 153 | func resizeFixed(dstW, dstH, srcW, srcH uint) (width, height uint) { 154 | width, height = dstW, dstH 155 | if dstW < 1 { 156 | width = srcW 157 | } 158 | if dstH < 1 { 159 | height = srcH 160 | } 161 | return 162 | } 163 | 164 | //Percentage: isotropic scaling by multiplicator(%) 165 | func resizePercentage(p int, srcW, srcH uint) (width, height uint) { 166 | width = srcW * uint(p) / 100 167 | height = srcH * uint(p) / 100 168 | return 169 | } 170 | -------------------------------------------------------------------------------- /command/resize/resize.go: -------------------------------------------------------------------------------- 1 | package resize 2 | 3 | import ( 4 | "github.com/ctripcorp/nephele/command" 5 | ) 6 | 7 | func init() { 8 | command.Register("resize", func() command.Command { 9 | return &Command{} 10 | }) 11 | } 12 | -------------------------------------------------------------------------------- /command/rotate/command.go: -------------------------------------------------------------------------------- 1 | package rotate 2 | 3 | import ( 4 | "fmt" 5 | "strconv" 6 | 7 | "context" 8 | 9 | "github.com/ctrip-nephele/gmagick" 10 | "github.com/ctripcorp/nephele/command" 11 | ) 12 | 13 | //Rotate rotate command 14 | type Command struct { 15 | Degree int 16 | } 17 | 18 | const ( 19 | commandKeyV string = "v" 20 | ) 21 | 22 | func (c *Command) Support() string { 23 | return "wand" 24 | } 25 | 26 | //Verify rotate verify params 27 | func (c *Command) Verify(ctx context.Context, params map[string]string) error { 28 | //log.Debugf(ctx, "rotate verification") 29 | for k, v := range params { 30 | if k == commandKeyV { 31 | degree, e := strconv.Atoi(v) 32 | if e != nil || degree < 0 || degree > 360 { 33 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 34 | } 35 | c.Degree = degree 36 | return nil 37 | } 38 | } 39 | return fmt.Errorf(command.ErrorInvalidOptionFormat, "rotate", params) 40 | } 41 | 42 | func (c *Command) ExecuteOnBlob(ctx context.Context, blob []byte) ([]byte, error) { 43 | return nil, nil 44 | } 45 | 46 | //Exec rotate ExecOnWand 47 | func (c *Command) ExecuteOnWand(ctx context.Context, wand *gmagick.MagickWand) error { 48 | var err error 49 | //log.TraceBegin(ctx, "", "URL.Command", "rotate", "degree", c.Degree) 50 | //defer log.TraceEnd(ctx, err) 51 | background := gmagick.NewPixelWand() 52 | background.SetColor("#000000") 53 | err = wand.RotateImage(background, float64(c.Degree)) 54 | return err 55 | } 56 | -------------------------------------------------------------------------------- /command/rotate/rotate.go: -------------------------------------------------------------------------------- 1 | package rotate 2 | 3 | import ( 4 | "github.com/ctripcorp/nephele/command" 5 | ) 6 | 7 | func init() { 8 | command.Register("rotate", func() command.Command { 9 | return &Command{} 10 | }) 11 | } 12 | -------------------------------------------------------------------------------- /command/sharpen/command.go: -------------------------------------------------------------------------------- 1 | package sharpen 2 | 3 | import ( 4 | "fmt" 5 | "strconv" 6 | 7 | "context" 8 | 9 | "github.com/ctrip-nephele/gmagick" 10 | "github.com/ctripcorp/nephele/command" 11 | ) 12 | 13 | type Command struct { 14 | Radius float64 15 | Sigma float64 16 | } 17 | 18 | const ( 19 | commandKeyR string = "r" 20 | commandKeyS string = "s" 21 | ) 22 | 23 | func (c *Command) Support() string { 24 | return "wand" 25 | } 26 | 27 | //Verify sharpen Verify 28 | func (c *Command) Verify(ctx context.Context, params map[string]string) error { 29 | //log.Debugf(ctx, "sharpen verification") 30 | for k, v := range params { 31 | if k == commandKeyR { 32 | radius, e := strconv.ParseFloat(v, 64) 33 | if e != nil { 34 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 35 | } 36 | c.Radius = radius 37 | } 38 | if k == commandKeyS { 39 | sigma, e := strconv.ParseFloat(v, 64) 40 | if e != nil { 41 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 42 | } 43 | c.Sigma = sigma 44 | } 45 | } 46 | return nil 47 | } 48 | 49 | func (c *Command) ExecuteOnBlob(ctx context.Context, blob []byte) ([]byte, error) { 50 | return nil, nil 51 | } 52 | 53 | //Exec sharpen exec 54 | func (c *Command) ExecuteOnWand(ctx context.Context, wand *gmagick.MagickWand) error { 55 | //log.TraceBegin(ctx, "", "URL.Command", "sharpen", "radius", c.Radius, "sigma", c.Sigma) 56 | //defer log.TraceEnd(ctx, err) 57 | return wand.SharpenImage(c.Radius, c.Sigma) 58 | } 59 | -------------------------------------------------------------------------------- /command/sharpen/sharpen.go: -------------------------------------------------------------------------------- 1 | package sharpen 2 | 3 | import ( 4 | "github.com/ctripcorp/nephele/command" 5 | ) 6 | 7 | func init() { 8 | command.Register("sharpen", func() command.Command { 9 | return &Command{} 10 | }) 11 | } 12 | -------------------------------------------------------------------------------- /command/strip/command.go: -------------------------------------------------------------------------------- 1 | package strip 2 | 3 | import ( 4 | "context" 5 | 6 | "github.com/ctrip-nephele/gmagick" 7 | ) 8 | 9 | //Strip strip command 10 | type Command struct { 11 | } 12 | 13 | func (c *Command) Support() string { 14 | return "wand" 15 | } 16 | 17 | //Verify strip Verify params 18 | func (c *Command) Verify(ctx context.Context, params map[string]string) error { 19 | //log.Debugf(ctx, "strip verification") 20 | return nil 21 | } 22 | 23 | func (c *Command) ExecuteOnBlob(ctx context.Context, blob []byte) ([]byte, error) { 24 | return nil, nil 25 | } 26 | 27 | // Exec strip exec 28 | func (c *Command) ExecuteOnWand(ctx context.Context, wand *gmagick.MagickWand) error { 29 | //log.TraceBegin(ctx, "", "URL.Command", "strip") 30 | //defer log.TraceEnd(ctx, err) 31 | return wand.StripImage() 32 | } 33 | -------------------------------------------------------------------------------- /command/strip/strip.go: -------------------------------------------------------------------------------- 1 | package strip 2 | 3 | import ( 4 | "github.com/ctripcorp/nephele/command" 5 | ) 6 | 7 | func init() { 8 | command.Register("strip", func() command.Command { 9 | return &Command{} 10 | }) 11 | } 12 | -------------------------------------------------------------------------------- /command/watermark/command.go: -------------------------------------------------------------------------------- 1 | package watermark 2 | 3 | import ( 4 | "encoding/base64" 5 | "fmt" 6 | "strconv" 7 | 8 | "github.com/ctripcorp/nephele/command" 9 | 10 | "github.com/ctrip-nephele/gmagick" 11 | "github.com/ctripcorp/nephele/storage" 12 | 13 | "context" 14 | 15 | "github.com/ctripcorp/nephele/util" 16 | ) 17 | 18 | //Watermark watermark command 19 | type Command struct { 20 | Name string 21 | Dissolve int //1-100 22 | Location string 23 | Minwidth, Minheight uint 24 | X, Y int 25 | } 26 | 27 | const ( 28 | commandKeyN string = "n" 29 | commandKeyD string = "d" 30 | commandKeyL string = "l" 31 | commandKeyX string = "x" 32 | commandKeyY string = "y" 33 | commandKeyMW string = "mw" 34 | commandKeyMH string = "mh" 35 | ) 36 | 37 | var watermarkLocations = []string{"nw", "north", "ne", "west", "center", "east", "sw", "south", "se"} 38 | 39 | func (c *Command) Support() string { 40 | return "wand" 41 | } 42 | 43 | //verify watermark verify 44 | func (c *Command) Verify(ctx context.Context, params map[string]string) error { 45 | //log.Debugf(ctx, "watermark verification") 46 | for k, v := range params { 47 | if k == commandKeyN { 48 | vByte, e := base64.StdEncoding.DecodeString(v) 49 | if e != nil { 50 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 51 | } 52 | if string(vByte) == "" { 53 | return fmt.Errorf("name of watermark must be provided") 54 | } 55 | c.Name = string(vByte) 56 | } 57 | if k == commandKeyD { 58 | dissolve, e := strconv.Atoi(v) 59 | if e != nil || dissolve < 0 || dissolve > 100 { 60 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 61 | } 62 | c.Dissolve = dissolve 63 | } 64 | if k == commandKeyL { 65 | if !util.InArray(v, watermarkLocations) && v != "" { 66 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 67 | } 68 | c.Location = v 69 | } 70 | if k == commandKeyX { 71 | x, e := strconv.Atoi(v) 72 | if e != nil || x < 0 { 73 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 74 | } 75 | c.X = x 76 | } 77 | if k == commandKeyY { 78 | y, e := strconv.Atoi(v) 79 | if e != nil || y < 0 { 80 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 81 | } 82 | c.Y = y 83 | } 84 | if k == commandKeyMW { 85 | mw, e := strconv.Atoi(v) 86 | if e != nil || mw < 0 { 87 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 88 | } 89 | c.Minwidth = uint(mw) 90 | } 91 | if k == commandKeyMH { 92 | mh, e := strconv.Atoi(v) 93 | if e != nil || mh < 0 { 94 | return fmt.Errorf(command.ErrorInvalidOptionFormat, k, v) 95 | } 96 | c.Minheight = uint(mh) 97 | } 98 | } 99 | return nil 100 | } 101 | 102 | func (c *Command) ExecuteOnBlob(ctx context.Context, blob []byte) ([]byte, error) { 103 | return nil, nil 104 | } 105 | 106 | //Exec watermark exec 107 | func (c *Command) ExecuteOnWand(ctx context.Context, wand *gmagick.MagickWand) error { 108 | var err error 109 | //log.TraceBegin(ctx, "", "URL.Command", "watermark", "watermarkName", c.Name, "location", c.Location, "dissolve", c.Dissolve, "x", c.X, "y", c.Y) 110 | //defer log.TraceEnd(ctx, err) 111 | if wand.GetImageWidth() < c.Minwidth || wand.GetImageHeight() < c.Minheight { 112 | return nil 113 | } 114 | var logoWand *gmagick.MagickWand 115 | logoWand, err = watermarkGetLogoWand(ctx, c.Name, c.Dissolve) 116 | if err != nil { 117 | return err 118 | } 119 | var x, y int 120 | if watermarkDecideLocationType(c.Location, c.X, c.Y) { 121 | x, y, err = watermarkGetCustomLocation(c.X, c.Y, wand, logoWand) 122 | } else { 123 | x, y, err = watermarkGetLocation(c.Location, wand, logoWand) 124 | } 125 | if err != nil { 126 | return err 127 | } 128 | err = wand.CompositeImage(logoWand, gmagick.COMPOSITE_OP_OVER, x, y) 129 | return err 130 | } 131 | 132 | //GetCustomLocation: get custom location by coordinate x,y 133 | func watermarkGetCustomLocation(x, y int, wand, logo *gmagick.MagickWand) (int, int, error) { 134 | width := wand.GetImageWidth() 135 | height := wand.GetImageHeight() 136 | logowidth := logo.GetImageWidth() 137 | logoheight := logo.GetImageHeight() 138 | if x >= 0 && y >= 0 && uint(x)+logowidth < width && uint(y)+logoheight < height { 139 | return x, y, nil 140 | } 141 | return 0, 0, nil 142 | } 143 | 144 | //GetLocation: get location via Sudoku 145 | func watermarkGetLocation(location string, wand, logo *gmagick.MagickWand) (int, int, error) { 146 | var ( 147 | x uint = 0 148 | y uint = 0 149 | ) 150 | width := wand.GetImageWidth() 151 | height := wand.GetImageHeight() 152 | logowidth := logo.GetImageWidth() 153 | logoheight := logo.GetImageHeight() 154 | switch location { 155 | case "nw": 156 | x, y = 0, 0 157 | case "north": 158 | x, y = (width-logowidth)/2, 0 159 | case "ne": 160 | x, y = width-logowidth, 0 161 | case "west": 162 | x, y = 0, (height-logoheight)/2 163 | case "center": 164 | x, y = (width-logowidth)/2, (height-logoheight)/2 165 | case "east": 166 | x, y = width-logowidth, (height-logoheight)/2 167 | case "sw": 168 | x, y = 0, height-logoheight 169 | case "south": 170 | x, y = (width-logowidth)/2, height-logoheight 171 | default: 172 | x, y = width-logowidth, height-logoheight 173 | } 174 | if x < 0 { 175 | x = 0 176 | } 177 | if y < 0 { 178 | y = 0 179 | } 180 | return int(x), int(y), nil 181 | } 182 | 183 | //GetLogoWand: get magickwand of logoImage 184 | func watermarkGetLogoWand(ctx context.Context, watermarkName string, dissolve int) (*gmagick.MagickWand, error) { 185 | bt, _, err := storage.Download(ctx, watermarkName) 186 | if err != nil { 187 | return nil, err 188 | } 189 | logoWand := gmagick.NewMagickWand() 190 | err = logoWand.ReadImageBlob(bt) 191 | if err != nil { 192 | return nil, err 193 | } 194 | if dissolve == 0 || dissolve == 100 { 195 | return logoWand, nil 196 | } 197 | logoWand.Dissolve(dissolve) 198 | if err != nil { 199 | return nil, err 200 | } 201 | return logoWand, nil 202 | } 203 | 204 | //DecideLocationType: decide function getLocation or function getCustomLocation to use 205 | func watermarkDecideLocationType(location string, dstX, dstY int) bool { 206 | if location != "" { 207 | return false 208 | } 209 | if dstX != 0 || dstY != 0 { 210 | return true 211 | } 212 | return false 213 | } 214 | -------------------------------------------------------------------------------- /command/watermark/exception.go: -------------------------------------------------------------------------------- 1 | package watermark 2 | 3 | import ( 4 | "strings" 5 | ) 6 | 7 | // exceptionType returns gmagick's error type, gmagick returns error with format: error_type: error_message 8 | func exceptionType(err error) string { 9 | i := strings.Index(err.Error(), ":") 10 | if i == -1 { 11 | return "UnknownError" 12 | } 13 | return err.Error()[:i] 14 | } 15 | -------------------------------------------------------------------------------- /command/watermark/recode.go: -------------------------------------------------------------------------------- 1 | package watermark 2 | 3 | import ( 4 | "bytes" 5 | "image/png" 6 | ) 7 | 8 | // recodePNG do a decode and encode routine of src image blob 9 | func recodePNG(blob []byte) ([]byte, error) { 10 | img, err := png.Decode(bytes.NewBuffer(blob)) 11 | if err != nil { 12 | return nil, err 13 | } 14 | buf := bytes.NewBuffer(nil) 15 | if err = png.Encode(buf, img); err != nil { 16 | return nil, err 17 | } 18 | return buf.Bytes(), nil 19 | } 20 | -------------------------------------------------------------------------------- /command/watermark/watermark.go: -------------------------------------------------------------------------------- 1 | package watermark 2 | 3 | import ( 4 | "github.com/ctripcorp/nephele/command" 5 | ) 6 | 7 | func init() { 8 | command.Register("watermark", func() command.Command { 9 | return &Command{} 10 | }) 11 | } 12 | -------------------------------------------------------------------------------- /docs/cn/PREREQUISITE.md: -------------------------------------------------------------------------------- 1 | # Prerequisite 2 | 3 | [Class 2](https://github.com/ctripcorp/nephele/blob/master/docs/cn/REQUIREMENT_CLASS.md) 4 | 5 | ## 目录 6 | 7 | * [安装 Go (>=1.10.2)](#安装-go-1102) 8 | 9 | * [安装 Govendor](#安装-govendor) 10 | 11 | * [安装 GraphicsMaigck](#安装-graphicsmaigck) 12 | 13 | ## 安装 Go (>=1.10.2) 14 | 15 | 从[https://golang.org/dl/](https://golang.org/dl/)下载安装包，并从[https://golang.org/doc/install](https://golang.org/doc/install)了解详细的安装信息。 16 | 17 | **以在Centos7上安装go1.10.2为例** 18 | 19 | * 在获取go1.10.2.linux-amd64.tar.gz (126MB)的前提下，输入以下命令(通常要加sudo)： 20 | 21 | ```bash 22 | tar -C /usr/local -xzf go1.10.2.linux-amd64.tar.gz 23 | ``` 24 | 25 | * 把/usr/local/go/bin添加到环境变量PATH。为此你可以将下面这行加入到/etc/profile或者$HOME/.profile中去。 26 | 27 | ```bash 28 | export PATH=$PATH:/usr/local/go/bin 29 | ``` 30 | 31 | ## 安装 Govendor 32 | 33 | 前往[github.com/kardianos/govendor](https://github.com/kardianos/govendor)了解详情。 34 | 35 | 或者运行下面的命令。 36 | 37 | ```bash 38 | go get -u github.com/kardianos/govendor 39 | ``` 40 | 41 | ## 安装 GraphicsMaigck 42 | 43 | 随着Nephele开源版本进度的推进，GraphicsMagick或将不再是必要的环境因子，转而作为一款插件存在。 44 | 45 | 但目前，[安装GraphicsMagick](https://github.com/ctripcorp/nephele/tree/master/thirdparty/graphicsmagick)依然是必要的，Nephele也将默认调用GraphicsMagick处理图片。 46 | -------------------------------------------------------------------------------- /docs/cn/REQUIREMENT_CLASS.md: -------------------------------------------------------------------------------- 1 | # Nephele Document Requirement Class 2 | 3 | 我们为Nephele文档提供的内容定义了4个需求等级(Requirement Class)。各个需求等级没有强依赖关系，但有道是：高级需求的产生是低级需求相对满足的结果。 4 | 5 | **Class 1 「了解」需要精简的陈述以填补基本认知的空白。** 6 | 7 | 标记为C1的内容通常是“是什么”，“能做什么”，“能做到什么程度”等常见问题的答案。 8 | 9 | **Class 2 「使用」需要具体的要求与建议指导实战。** 10 | 11 | 标记为C2的内容会是严肃的，全面的，细致的。这部分内容会占据整个文档群的绝大部分篇幅，有时会不可避免的高频的出现一些重复的文字。因此也会设置目录方便查阅。（还是建议有至少一次的完整浏览） 12 | 13 | **Class 3 「理解」需要学习内在机制以满足好奇心，或借鉴并构建类似的，相关的或其他的系统。** 14 | 15 | 标记为C3的内容通常会更加枯燥，冗长。它们可能会给读者一些启发与灵感，但也可能不会。对于那些把关注点放在实际应用的读者而言，这部分内容或许能够产生一些安全感吧，但总的来说，会显得不是那么重要。 16 | 17 | **Class 4 「交流」需要参与贡献社区或者需要挑战作者的理念。** 18 | 19 | 互相理解是合作的坚盾，知己知彼是争论的长矛。标记为C4的内容是作者团队的心里话，会是他们的一些展望，也可能会是总结。这些内容一定是天马行空，充满活力的。 20 | 21 | -------------------------------------------------------------------------------- /docs/en/CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation. 6 | 7 | ## Our Standards 8 | 9 | Examples of behavior that contributes to creating a positive environment include: 10 | 11 | * Using welcoming and inclusive language 12 | * Being respectful of differing viewpoints and experiences 13 | * Gracefully accepting constructive criticism 14 | * Focusing on what is best for the community 15 | * Showing empathy towards other community members 16 | 17 | Examples of unacceptable behavior by participants include: 18 | 19 | * The use of sexualized language or imagery and unwelcome sexual attention or advances 20 | * Trolling, insulting/derogatory comments, and personal or political attacks 21 | * Public or private harassment 22 | * Publishing others' private information, such as a physical or electronic address, without explicit permission 23 | * Other conduct which could reasonably be considered inappropriate in a professional setting 24 | 25 | ## Our Responsibilities 26 | 27 | Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior. 28 | 29 | Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful. 30 | 31 | ## Scope 32 | 33 | This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers. 34 | 35 | ## Enforcement 36 | 37 | Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at wucc@ctrip.com. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately. 38 | 39 | Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership. 40 | 41 | ## Attribution 42 | 43 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version] 44 | 45 | [homepage]: http://contributor-covenant.org 46 | [version]: http://contributor-covenant.org/version/1/4/ 47 | -------------------------------------------------------------------------------- /docs/en/LOG_COMPATIBILITY_STANDARD.md: -------------------------------------------------------------------------------- 1 | # The Nephele Log Compatibility Standard 2 | 3 | ## Log Suite & Log Interface 4 | 5 | Designed to be a compelete solution, Nephele provide a log suite compatible with [ES](https://www.elastic.co/products/elasticsearch) and [CAT](https://github.com/dianping/CAT). The log suite has two parts: develop kit and sync agent. 6 | 7 | The log develop kit is a go package providing nephele log programming interface, abbreviated as NLPI. Calling NLPI from your code forces your program to dump log files within some given constraints: specific alignment, particular layout, mangling names, etc. All these details are called the Nephele Log Binary interface, or NLBI. And when the moment is ripe, a background program will collect and decompose theses dumped log files. And that is the sync agent. It translates the content of log files into corresponding log units(CAT message or ES document or something similar) and send them to the target servers. 8 | 9 | **Keep reading through, you will learn about NLBI. Yet NLPI is documented in The Nephele Programming Interface() in detail, not here.** 10 | 11 | ## Compatibility 12 | 13 | The NLBI compatibility is defined with the equation above: given a log file generated with NLBI, it is able to be decomposed into single logs and every single log contains adequate infomation to be recognized as a given log unit. 14 | 15 | Currently NLBI is compatible with ES and CAT. 16 | 17 | ## Versioning 18 | 19 | The NLBI and the Nephele program are independant from versioning. 20 | 21 | The NLBI version consists of two numbers connected with a dot. 22 | 23 | This document(The Nephele Log Compatibility Standard) and The NLBI share a concert version number. 24 | 25 | ## Allowed Changes 26 | 27 | The following will cause the minor version number to increase, say from "1.0" to "1.1". 28 | 29 | ## Prohibited Changes 30 | 31 | The following will cause the major version number to increase, say from "0.9" to "1.0". 32 | 33 | ## Implementation 34 | 35 | ## FAQ 36 | 37 | ## Contact 38 | -------------------------------------------------------------------------------- /docs/en/README.md: -------------------------------------------------------------------------------- 1 | # Nephele 2 | 3 | [English](https://github.com/ctripcorp/nephele/blob/master/docs/cn/README.md) [C1](https://github.com/ctripcorp/nephele/blob/master/docs/en/REQUIREMENT_CLASS.md) 4 | 5 | ## Introduction 6 | 7 | Nephele is an open-source project created by Ctrip R&D to address image-related problems. 8 | 9 | **it provides:** 10 | 11 | * a basic server program with wide-ranging functionalities, plain or complicated 12 | * a pluggable framework surrounded with a bundle of ready-made plugins 13 | * a clear administration where users edit workflow and developers track server state 14 | * plenty of useful tools 15 | 16 | ## Feature 17 | 18 | **features you may concern instinctively:** 19 | 20 | * basic image processing 21 | * multiple storage support 22 | * monitor & logging 23 | * playground 24 | 25 | **features beyond intuition:** 26 | 27 | * digit watermark 28 | * optimized image encoding 29 | 30 | ## Intention 31 | 32 | Read [The Example Nephele Program]() to start up from the nephele project and see what nephele can be. 33 | 34 | Read [Pluggable Framework of Nephele]() to make customizations to the example. 35 | 36 | Read [How to Deploy A Plugin for Nephele]() when the ready-made plugin does not fit the needs. 37 | 38 | Read [Contributing]() to join the nephele community. 39 | 40 | ## License 41 | 42 | Nephele is licensed under the Apache License 2.0. See [LICENSE](https://github.com/ctripcorp/nephele/blob/master/LICENSE) for the full license text. 43 | 44 | 45 | 46 | 47 | -------------------------------------------------------------------------------- /example/README.md: -------------------------------------------------------------------------------- 1 | # The Example Nephele Program 2 | 3 | [Class 1](https://github.com/ctripcorp/nephele/blob/master/docs/cn/REQUIREMENT_CLASS.md) 4 | 5 | ## 介绍 6 | 7 | 基于Nephele Pluggable Framework(NPF)，Nephele团队针对不同场景的需求的强弱，开发了多款**现成**的图片服务产品。我们把它们称为[The Example Nephele Program]。 8 | 9 | * 用户或开发者只需要通过简单的配置部署它们的预编译文件，无需进行二次开发，便可以享受到完整的图片处理功能。 10 | 11 | * 同时，它们又是NPF的示例代码，是使用NPF开发最好的参考。 12 | 13 | 现在已有的两款分别是[Minimal](https://github.com/ctripcorp/nephele/blob/master/example/minimal/README.md)和[Pluggable]。 14 | 15 | ## 选择 16 | 17 | 如果你需要的功能非常简单，比如为自己的博客搭建一个动态图片处理服务以弱化版本更新引入的排版问题。你可能会希望这个服务最好能和你的博客网站部署在同一台虚机上并占用最少的服务器资源。那么Mininal对你而言就很不错。 18 | 19 | 如果你关心更多东西，比如你在纠结利用数据库中已有的数据来左右图片处理的流程，或者你需要考虑图片服务和一些主流存储系统的兼容性，又或者你必须把图片服务的工作状态写入到一个现有的日志系统中。那么直接使用Pluggable肯定是比只搭建Minimal并在其周遭再封装一层要来得方便的。 20 | -------------------------------------------------------------------------------- /example/minimal/README.md: -------------------------------------------------------------------------------- 1 | # The Example Nephele Program: Minimal 2 | 3 | [Class 2](https://github.com/ctripcorp/nephele/blob/master/docs/cn/REQUIREMENT_CLASS.md) 4 | 5 | ## 目录 6 | 7 | * [介绍](#介绍) 8 | 9 | * [安装](#安装) 10 | 11 | * [启动](#启动) 12 | 13 | * [试用](#试用) 14 | 15 | ## 介绍 16 | 17 | 这里简短罗列一下Minimal的现状和未来的规划。 18 | * 因为资源的原因，现阶段的Minimal还未引入上传功能，将在短期内引入。 19 | * Minimal的日志是直接落到本地的，近期计划开发相应的日志同步工具，将日志同步到[ElasticSearch](https://www.elastic.co/products/elasticsearch)和[CAT](https://github.com/dianping/cat)。 20 | * Minimal使用本地磁盘作为存储，如果要使用OSS或者S3之类的分布式存储的，应该转用Pluggable。 21 | * 基本的图片处理功能都已经有了。 22 | 23 | ## 安装 24 | 25 | **下载预编译文件** 26 | 27 | * [Centos 7](http://file.c-ctrip.com/files/3/nep/zip/82f/2de/e47/671aab3821dd4dab9aec14bcad8c2fd4.zip) 28 | 29 | * [Centos 6](http://file.c-ctrip.com/files/3/nep/zip/88d/e2c/185/2158e329294346f4870aec40c630b4f9.zip) 30 | 31 | * [Ubuntu](http://file.c-ctrip.com/files/3/nep/zip/300/cd7/235/a915c23e981e42a590633b7c273e5385.zip) 32 | 33 | * [Windows](http://file.c-ctrip.com/files/3/nep/zip/7e5/993/749/72c668fa4df142179d1833ae1a4a81a7.zip) 34 | 35 | 36 | **下载源代码** 37 | 38 | ```bash 39 | git clone https://github.com/ctripcorp/nephele.git $GOPATH/src/github.com/ctripcorp/nephele 40 | ``` 41 | 42 | **下载依赖** 43 | 44 | ```bash 45 | cd nephele 46 | govendor sync 47 | ``` 48 | 49 | 你可能需要前往[github.com/golang](https://github.com/golang)手动下载墙外包，并将它们移至正确路径。 50 | 51 | **编译** 52 | 53 | ```bash 54 | cd example/minimal 55 | go build 56 | ``` 57 | 58 | ## 启动 59 | 60 | **运行** 61 | 62 | [须知]执行下面的命令后，在用户根目录下会生成一个名为nephele的文件夹。 63 | 64 | ```bash 65 | ./minimal --open 66 | ``` 67 | 68 | 如果你看到类似下面这样的打印信息，那么你已经成功启动了Nephele Minimal： 69 | 70 | ``` 71 | [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production. 72 | - using env: export GIN_MODE=release 73 | - using code: gin.SetMode(gin.ReleaseMode) 74 | 75 | [GIN-debug] GET /image/*imagepath --> github.com/ctripcorp/nephele/service/handler.(*HandlerFactory).Build.func1 (3 handlers) 76 | [GIN-debug] POST /image --> github.com/ctripcorp/nephele/service/handler.(*HandlerFactory).Build.func1 (3 handlers) 77 | [GIN-debug] GET /healthcheck --> github.com/ctripcorp/nephele/service/handler.(*HandlerFactory).Build.func1 (3 handlers) 78 | ``` 79 | 80 | 同时，来到~/nephele/log下，你会看到Nephele Minimal生成的日志文件。 81 | 82 | ## 简单实例 83 | 84 | 详细使用方法请参考[Nephele API]()。 85 | 86 | 在~/nephele/image目录下添加一张名为1.jpg的图片。 87 | 88 | **查看原图 :8080/1.jpg** 89 | 90 | [原图](https://dimg08.c-ctrip.com/images/w20e0s000000hvulz1A96.jpg) 91 | 92 | **图片被等比缩放至目标区域区域内 :8080/1.jpg?x-nephele-process=image/resize,w_200,h_200** 93 | 94 | [保持长宽比不变，缩放至200X200的区域内的最大图片](https://dimg08.c-ctrip.com/images/w20h0s000000hufjr466E.jpg) 95 | 96 | **图片底部被裁剪，最终高为200 :8080/1.jpg?x-nephele-process=image/crop,m_b,h_200** 97 | 98 | [裁剪底部直至高为200](https://dimg08.c-ctrip.com/images/w2040s000000hrubd8B2F.jpg) 99 | 100 | **图片顺时针旋转90度 :8080/1.jpg?x-nephele-process=image/rotate,v_90** 101 | 102 | [顺时针旋转90度](https://dimg08.c-ctrip.com/images/w20q0s000000hued11162.jpg) 103 | 104 | **降低图片质量 :8080/1.jpg?x-nephele-process=image/quality,v_5** 105 | 106 | [图片质量降低至5](https://dimg08.c-ctrip.com/images/w20c0s000000hvr5p3621.jpg) 107 | 108 | **图片格式转换为png :8080/1.jpg?x-nephele-process=image/format,v_png** 109 | 110 | [图片格式转为png](https://dimg08.c-ctrip.com/images/w20m0s000000i2hswD841.png) 111 | 112 | **锐化图片 :8080/1.jpg?x-nephele-process=image/sharpen,r_44,s_3** 113 | 114 | [特定参数锐化](https://dimg08.c-ctrip.com/images/w2030s000000huu7l6749.jpg) 115 | 116 | **在~/nephele/image目录下添加用作水印的图片wm1.png** 117 | 118 | [用作水印的图片](https://dimg08.c-ctrip.com/images/w20u0s000000hteatB945.png) 119 | 120 | **图片打上水印（水印图片名字经base64编码处理） :8080/1.jpg?x-nephele-process=image/watermark,n_d20xLnBuZw==** 121 | 122 | [图片被打上水印](https://dimg08.c-ctrip.com/images/w20d0s000000hrxuf69D9.jpg) 123 | -------------------------------------------------------------------------------- /example/minimal/default.toml: -------------------------------------------------------------------------------- 1 | server_config_path = "server.toml" 2 | 3 | [interpret] 4 | type = "inline" 5 | 6 | #type = "plugin" 7 | #path = "[your-interpret].so" 8 | 9 | [process] 10 | 11 | [storage] 12 | type = "inline" 13 | root = "[to-your-path]" 14 | 15 | #path = "[your-storage].so" 16 | #endpoint = "[endpoint]" 17 | #bucketname = "[bucketname]" 18 | #accessKeyId = "[access-key-id]" 19 | #accessKeySecret = "[access-key-secret]" 20 | -------------------------------------------------------------------------------- /example/minimal/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "github.com/ctripcorp/nephele/interpret" 5 | "github.com/ctripcorp/nephele/process" 6 | "github.com/ctripcorp/nephele/storage" 7 | 8 | "github.com/ctripcorp/nephele/server" 9 | "github.com/ctripcorp/nephele/util" 10 | 11 | _ "github.com/ctripcorp/nephele/command/autoorient" 12 | _ "github.com/ctripcorp/nephele/command/crop" 13 | _ "github.com/ctripcorp/nephele/command/format" 14 | _ "github.com/ctripcorp/nephele/command/quality" 15 | _ "github.com/ctripcorp/nephele/command/resize" 16 | _ "github.com/ctripcorp/nephele/command/rotate" 17 | _ "github.com/ctripcorp/nephele/command/sharpen" 18 | _ "github.com/ctripcorp/nephele/command/strip" 19 | _ "github.com/ctripcorp/nephele/command/watermark" 20 | 21 | _ "github.com/ctripcorp/nephele/interpret/neph" 22 | _ "github.com/ctripcorp/nephele/storage/neph" 23 | _ "github.com/ctripcorp/nephele/server/ping" 24 | ) 25 | 26 | var Config = struct { 27 | ServerConfigPath string `toml:"server_config_path"` 28 | Interpret *map[string]string 29 | Process *map[string]string 30 | Storage *map[string]string 31 | }{ 32 | Interpret: &interpret.Config, 33 | Process: &process.Config, 34 | Storage: &storage.Config, 35 | } 36 | 37 | func main() { 38 | util.FromToml("default.toml", &Config) 39 | util.FromToml(Config.ServerConfigPath, &server.Config) 40 | 41 | interpret.Init() 42 | process.Init() 43 | storage.Init() 44 | 45 | server.Run() 46 | } 47 | -------------------------------------------------------------------------------- /example/minimal/server.toml: -------------------------------------------------------------------------------- 1 | port = ":8080" 2 | 3 | [cors] 4 | allow_all_origins = false 5 | max_age = 30 6 | option_pass_through = true 7 | allowed_origins = ["http://foo.com"] 8 | allowed_methods = ["GET", "PUT"] 9 | allowed_headers = ["Origin"] 10 | exposed_headers = ["Content-Length"] 11 | 12 | [entrance] 13 | request_buffer = 200 14 | request_timeout_millisecond = 3000 15 | max_concurrency = 8 16 | -------------------------------------------------------------------------------- /interpret/interpret.go: -------------------------------------------------------------------------------- 1 | package interpret 2 | 3 | import ( 4 | "github.com/gin-gonic/gin" 5 | ) 6 | 7 | var Config map[string]string 8 | 9 | var registeredInterpreter func(*gin.Context) (string, string, error) 10 | 11 | func Init() {} 12 | 13 | func Register(interpreter func(*gin.Context) (string, string, error)) { 14 | registeredInterpreter = interpreter 15 | } 16 | 17 | func Do(c *gin.Context) (string, string, error) { 18 | return registeredInterpreter(c) 19 | } 20 | -------------------------------------------------------------------------------- /interpret/neph/neph.go: -------------------------------------------------------------------------------- 1 | package neph 2 | 3 | import ( 4 | "github.com/ctripcorp/nephele/interpret" 5 | "github.com/gin-gonic/gin" 6 | ) 7 | 8 | func init() { 9 | interpret.Register(inter) 10 | } 11 | 12 | func inter(c *gin.Context) (string, string, error) { 13 | return c.Param("key"), c.Query("x-nephele-process"), nil 14 | } 15 | -------------------------------------------------------------------------------- /log/fake.go: -------------------------------------------------------------------------------- 1 | package log 2 | 3 | import ( 4 | "github.com/ctripcorp/nephele/context" 5 | ) 6 | 7 | type fakeLogger struct{} 8 | 9 | func (l *fakeLogger) Printf(ctx *context.Context, 10 | level string, format string, values ...interface{}) { 11 | } 12 | func (l *fakeLogger) Printw(ctx *context.Context, 13 | level string, message string, keysAndValues ...interface{}) { 14 | } 15 | -------------------------------------------------------------------------------- /log/log.go: -------------------------------------------------------------------------------- 1 | package log 2 | 3 | import ( 4 | "fmt" 5 | "os" 6 | "path/filepath" 7 | 8 | "github.com/ctripcorp/nephele/context" 9 | "github.com/ctripcorp/nephele/log/output" 10 | "github.com/ctripcorp/nephele/util" 11 | ) 12 | 13 | type Config interface { 14 | Build() (Logger, error) 15 | } 16 | 17 | var instance Logger = &fakeLogger{} 18 | 19 | func Init(loggerConfig Config) (err error) { 20 | instance, err = loggerConfig.Build() 21 | return 22 | } 23 | 24 | func DefaultConfig() (*LoggerConfig, error) { 25 | var err error 26 | var hd string 27 | var d string 28 | 29 | hd, err = util.HomePath() 30 | if err != nil { 31 | return nil, err 32 | } 33 | 34 | d = filepath.Join(hd, "log/") 35 | err = os.MkdirAll(d, 0777) 36 | if err != nil { 37 | fmt.Println(err) 38 | return nil, err 39 | } 40 | 41 | return &LoggerConfig{ 42 | "", 43 | &output.StdoutConfig{"debug"}, 44 | &output.DumpConfig{"info", d, 60}, 45 | }, nil 46 | } 47 | 48 | func Debugf(ctx *context.Context, format string, values ...interface{}) { 49 | instance.Printf(ctx, "debug", format, values...) 50 | } 51 | 52 | func Infof(ctx *context.Context, format string, values ...interface{}) { 53 | instance.Printf(ctx, "info", format, values...) 54 | } 55 | 56 | func Warnf(ctx *context.Context, format string, values ...interface{}) { 57 | instance.Printf(ctx, "warn", format, values...) 58 | } 59 | 60 | func Errorf(ctx *context.Context, format string, values ...interface{}) { 61 | instance.Printf(ctx, "error", format, values...) 62 | } 63 | 64 | func Fatalf(ctx *context.Context, format string, values ...interface{}) { 65 | instance.Printf(ctx, "fatal", format, values...) 66 | } 67 | 68 | func Debugw(ctx *context.Context, message string, keysAndValues ...interface{}) { 69 | instance.Printw(ctx, "debug", message, keysAndValues...) 70 | } 71 | 72 | func Infow(ctx *context.Context, message string, keysAndValues ...interface{}) { 73 | instance.Printw(ctx, "info", message, keysAndValues...) 74 | } 75 | 76 | func Warnw(ctx *context.Context, message string, keysAndValues ...interface{}) { 77 | instance.Printw(ctx, "warn", message, keysAndValues...) 78 | } 79 | 80 | func Errorw(ctx *context.Context, message string, keysAndValues ...interface{}) { 81 | instance.Printw(ctx, "error", message, keysAndValues...) 82 | } 83 | 84 | func Fatalw(ctx *context.Context, message string, keysAndValues ...interface{}) { 85 | instance.Printw(ctx, "fatal", message, keysAndValues...) 86 | } 87 | 88 | func TraceBegin(ctx *context.Context, message string, keysAndValues ...interface{}) { 89 | traceBegin(ctx.ID(), message, keysAndValues...) 90 | instance.Printw(ctx, "trace/begin", message, keysAndValues...) 91 | } 92 | 93 | func TraceEnd(ctx *context.Context, state interface{}) { 94 | traceEnd(ctx.ID(), state) 95 | instance.Printw(ctx, "trace/end", fmt.Sprintf("%v", state)) 96 | } 97 | 98 | func TraceEndRoot(ctx *context.Context, state interface{}) { 99 | traceEndRoot(ctx.ID(), state) 100 | message, namesAndDurations := traceSum(ctx.ID()) 101 | instance.Printw(ctx, "trace/endroot", fmt.Sprintf("%v", state)) 102 | instance.Printw(ctx, "trace/summary", message, namesAndDurations...) 103 | } 104 | -------------------------------------------------------------------------------- /log/log_test.go: -------------------------------------------------------------------------------- 1 | package log 2 | 3 | import ( 4 | "errors" 5 | "testing" 6 | 7 | "github.com/ctripcorp/nephele/context" 8 | ) 9 | 10 | func TestLogger(t *testing.T) { 11 | dc, _ := DefaultConfig() 12 | Init(dc) 13 | 14 | TraceBegin(&context.Context{}, "we are going to gather some personal infomations", "Info", "Collect") 15 | TraceBegin(&context.Context{}, "yes, lets start", "Info", "DelayCollect") 16 | 17 | Debugf(&context.Context{}, "%s!!!", "lets start") 18 | Errorf(&context.Context{}, "%s...", "so just wait a minute") 19 | Infow(&context.Context{}, "this is a info list", 20 | "name", "mag", 21 | "gender", "male") 22 | 23 | Errorw(&context.Context{}, "this is a error list", 24 | "name", "invalid last name", 25 | "gender", "not male nor female") 26 | 27 | TraceEndRoot(&context.Context{}, errors.New("seems something wrong")) 28 | } 29 | -------------------------------------------------------------------------------- /log/logger.go: -------------------------------------------------------------------------------- 1 | package log 2 | 3 | import ( 4 | "bytes" 5 | "fmt" 6 | "runtime" 7 | "strings" 8 | "sync" 9 | "time" 10 | 11 | "github.com/ctripcorp/nephele/context" 12 | "github.com/ctripcorp/nephele/log/output" 13 | ) 14 | 15 | type Logger interface { 16 | Printf(ctx *context.Context, level string, format string, values ...interface{}) 17 | Printw(ctx *context.Context, level string, message string, keysAndValues ...interface{}) 18 | } 19 | 20 | type logger struct { 21 | outputs []output.Output 22 | bufferPool *sync.Pool 23 | } 24 | 25 | func (l *logger) Printf(ctx *context.Context, level string, format string, values ...interface{}) { 26 | var rb *bytes.Buffer = l.bufferPool.Get().(*bytes.Buffer) 27 | fmt.Fprintf(rb, "[%s] %s ", level, ctx.ID()) 28 | if level == "error" { 29 | _, file, line, ok := runtime.Caller(2) 30 | if ok { 31 | fmt.Fprintf(rb, "%s line:%d ", file[strings.LastIndex(file, "nephele"):], line) 32 | } 33 | } 34 | fmt.Fprintf(rb, "[%s]", time.Now().Format(time.RFC3339Nano)) 35 | fmt.Fprintf(rb, "\t\""+format+"\"\n", values...) 36 | 37 | for _, o := range l.outputs { 38 | o.Write(rb.Bytes(), level) 39 | } 40 | 41 | rb.Reset() 42 | l.bufferPool.Put(rb) 43 | } 44 | 45 | func (l *logger) Printw(ctx *context.Context, level string, message string, keysAndValues ...interface{}) { 46 | var rb *bytes.Buffer = l.bufferPool.Get().(*bytes.Buffer) 47 | fmt.Fprintf(rb, "[%s] %s ", level, ctx.ID()) 48 | if level == "error" { 49 | _, file, line, ok := runtime.Caller(2) 50 | if ok { 51 | fmt.Fprintf(rb, "%s line:%d ", file[strings.LastIndex(file, "nephele"):], line) 52 | } 53 | } 54 | fmt.Fprintf(rb, "[%s]", time.Now().Format(time.RFC3339Nano)) 55 | for i := 0; i < len(keysAndValues)/2; i++ { 56 | fmt.Fprintf(rb, "\t\"%v\"", keysAndValues[i*2]) 57 | fmt.Fprintf(rb, "\t%v", keysAndValues[i*2+1]) 58 | } 59 | fmt.Fprintf(rb, "\t\"%s\"\n", message) 60 | 61 | for _, o := range l.outputs { 62 | o.Write(rb.Bytes(), level) 63 | } 64 | 65 | rb.Reset() 66 | l.bufferPool.Put(rb) 67 | } 68 | 69 | type LoggerConfig struct { 70 | ConfigPath string `toml:"config-path"` 71 | Stdout *output.StdoutConfig `toml:"stdout"` 72 | Dump *output.DumpConfig `toml:"dump"` 73 | } 74 | 75 | func (lc *LoggerConfig) Build() (Logger, error) { 76 | var stdout, dump output.Output 77 | var err error 78 | stdout, err = lc.Stdout.Build() 79 | dump, err = lc.Dump.Build() 80 | return &logger{ 81 | []output.Output{ 82 | stdout, 83 | dump, 84 | }, 85 | &sync.Pool{ 86 | New: func() interface{} { return new(bytes.Buffer) }, 87 | }, 88 | }, err 89 | } 90 | -------------------------------------------------------------------------------- /log/output/dump.go: -------------------------------------------------------------------------------- 1 | package output 2 | 3 | import ( 4 | "os" 5 | "path/filepath" 6 | "time" 7 | ) 8 | 9 | type DumpConfig struct { 10 | Level string `toml:"level"` 11 | Path string `toml:"path"` 12 | TimeBlock int `toml:"time-block"` 13 | } 14 | 15 | func (dc *DumpConfig) Build() (Output, error) { 16 | o, err := createBasicOutput( 17 | func() (WriteSyncer, error) { 18 | return os.OpenFile(filepath.Join(dc.Path, time.Now().Format("2006-01-02H15M04S05")), 19 | os.O_CREATE|os.O_APPEND|os.O_RDWR, 0644) 20 | }, 21 | dc.Level, 22 | ) 23 | if err != nil { 24 | return nil, err 25 | } 26 | go func(bo *basicOutput) { 27 | for { 28 | time.Sleep(time.Duration(dc.TimeBlock) * time.Minute) 29 | bo.Reset() 30 | } 31 | }(o) 32 | return o, err 33 | } 34 | -------------------------------------------------------------------------------- /log/output/dump_test.go: -------------------------------------------------------------------------------- 1 | package output 2 | 3 | import ( 4 | "testing" 5 | ) 6 | 7 | func TestDump(t *testing.T) { 8 | dc := &DumpConfig{ 9 | "info", 10 | "/Users/CINTS/nephele/log", 11 | 1, 12 | } 13 | o, err := dc.Build() 14 | if err != nil { 15 | t.Error(err) 16 | } 17 | for { 18 | o.Write([]byte("h1z1o4f3"), "info") 19 | } 20 | } 21 | -------------------------------------------------------------------------------- /log/output/level.go: -------------------------------------------------------------------------------- 1 | package output 2 | 3 | var levels []string = []string{ 4 | "fatal", 5 | "error", 6 | "warning", 7 | "info", 8 | "debug", 9 | } 10 | 11 | func levelInt(level string) int { 12 | var i = 0 13 | for ; i < 5; i++ { 14 | if levels[i] == level { 15 | return i 16 | } 17 | } 18 | return 3 19 | } 20 | -------------------------------------------------------------------------------- /log/output/level_test.go: -------------------------------------------------------------------------------- 1 | package output 2 | 3 | import ( 4 | "testing" 5 | ) 6 | 7 | func TestLevel(t *testing.T) { 8 | t2s := []struct { 9 | desc string 10 | f func() int 11 | want int 12 | }{ 13 | {"debug == 4", func() int { return levelInt("debug") }, 4}, 14 | {"info == 3", func() int { return levelInt("info") }, 3}, 15 | {"warning == 2", func() int { return levelInt("warning") }, 2}, 16 | {"error == 1", func() int { return levelInt("error") }, 1}, 17 | {"fatal == 0", func() int { return levelInt("fatal") }, 0}, 18 | {"unknown string == 5", func() int { return levelInt("unknown string") }, 5}, 19 | } 20 | 21 | for _, t2 := range t2s { 22 | t.Run(t2.desc, func(t *testing.T) { 23 | if t2.f() != t2.want { 24 | t.Error("level code unmatch") 25 | } 26 | }) 27 | } 28 | } 29 | -------------------------------------------------------------------------------- /log/output/output.go: -------------------------------------------------------------------------------- 1 | package output 2 | 3 | import ( 4 | "io" 5 | ) 6 | 7 | type WriteSyncer interface { 8 | io.Writer 9 | Sync() error 10 | } 11 | 12 | type Output interface { 13 | Write(p []byte, level string) (n int, err error) 14 | Sync() error 15 | } 16 | 17 | type basicOutput struct { 18 | internal WriteSyncer 19 | internalCreater func() (WriteSyncer, error) 20 | level string 21 | } 22 | 23 | func createBasicOutput(creater func() (WriteSyncer, error), level string) (bo *basicOutput, err error) { 24 | var internal WriteSyncer 25 | internal, err = creater() 26 | if err != nil { 27 | return nil, err 28 | } 29 | return &basicOutput{ 30 | internal, 31 | creater, 32 | level, 33 | }, nil 34 | } 35 | 36 | func (bo *basicOutput) Reset() (err error) { 37 | if bo.internalCreater == nil { 38 | return nil 39 | } 40 | bo.internal, err = bo.internalCreater() 41 | return err 42 | } 43 | 44 | func (bo *basicOutput) Write(p []byte, level string) (n int, err error) { 45 | if levelInt(level) <= levelInt(bo.level) { 46 | var internal WriteSyncer 47 | internal = bo.internal 48 | n, err = internal.Write(p) 49 | if err != nil { 50 | internal = bo.internal 51 | n, err = internal.Write(p) 52 | } 53 | if levelInt(level) <= levelInt("error") { 54 | defer internal.Sync() 55 | } 56 | return n, err 57 | } 58 | return 0, nil 59 | } 60 | 61 | func (bo *basicOutput) Sync() error { 62 | return bo.internal.Sync() 63 | } 64 | 65 | func (bo *basicOutput) Level() string { 66 | return bo.level 67 | } 68 | -------------------------------------------------------------------------------- /log/output/stdout.go: -------------------------------------------------------------------------------- 1 | package output 2 | 3 | import ( 4 | "os" 5 | ) 6 | 7 | type StdoutConfig struct { 8 | Level string `toml:"level"` 9 | } 10 | 11 | func (sc *StdoutConfig) Build() (Output, error) { 12 | return createBasicOutput( 13 | func() (WriteSyncer, error) { 14 | return os.Stdout, nil 15 | }, 16 | sc.Level, 17 | ) 18 | } 19 | -------------------------------------------------------------------------------- /log/trace.go: -------------------------------------------------------------------------------- 1 | package log 2 | 3 | import ( 4 | "fmt" 5 | "sync" 6 | "time" 7 | ) 8 | 9 | var traceTrees map[string]*traceTree 10 | var traceMutex sync.RWMutex 11 | 12 | func init() { 13 | traceTrees = make(map[string]*traceTree) 14 | } 15 | 16 | func traceBegin(contextID string, message string, keysAndValues ...interface{}) { 17 | traceMutex.RLock() 18 | t, ok := traceTrees[contextID] 19 | traceMutex.RUnlock() 20 | if !ok { 21 | traceMutex.Lock() 22 | traceTrees[contextID] = &traceTree{ 23 | make([]*trace, 0), 24 | nil, 25 | } 26 | traceMutex.Unlock() 27 | 28 | traceMutex.RLock() 29 | t = traceTrees[contextID] 30 | traceMutex.RUnlock() 31 | } 32 | t.begin(message, keysAndValues...) 33 | } 34 | 35 | func traceEnd(contextID string, state interface{}) { 36 | traceMutex.RLock() 37 | t, ok := traceTrees[contextID] 38 | traceMutex.RUnlock() 39 | if !ok { 40 | return 41 | } 42 | t.end(state) 43 | } 44 | 45 | func traceEndRoot(contextID string, state interface{}) { 46 | traceMutex.RLock() 47 | t, ok := traceTrees[contextID] 48 | traceMutex.RUnlock() 49 | if !ok { 50 | return 51 | } 52 | t.endRoot(state) 53 | } 54 | 55 | func traceSum(contextID string) (string, []interface{}) { 56 | defer func() { 57 | traceMutex.Lock() 58 | delete(traceTrees, contextID) 59 | traceMutex.Unlock() 60 | }() 61 | traceMutex.RLock() 62 | t, ok := traceTrees[contextID] 63 | traceMutex.RUnlock() 64 | if !ok { 65 | return "", nil 66 | } 67 | return t.sum() 68 | } 69 | 70 | type traceTree struct { 71 | stack []*trace 72 | root *trace 73 | } 74 | 75 | func (t *traceTree) begin(message string, keysAndValues ...interface{}) *trace { 76 | stk := t.stack 77 | trc := &trace{} 78 | trc.begin(message, keysAndValues...) 79 | l := len(stk) 80 | if l > 0 { 81 | parent := stk[l-1] 82 | parent.addChild(trc) 83 | } else { 84 | t.root = trc 85 | } 86 | t.stack = append(stk, trc) 87 | return trc 88 | } 89 | 90 | func (t *traceTree) end(state interface{}) { 91 | stk := t.stack 92 | current := len(stk) - 1 93 | if current == -1 { 94 | return 95 | } 96 | trc := stk[current] 97 | trc.end(state) 98 | t.stack = stk[:current] 99 | } 100 | 101 | func (t *traceTree) endRoot(state interface{}) { 102 | t.root.end(state) 103 | t.stack = t.stack[:0] 104 | } 105 | 106 | func (t *traceTree) sum() (string, []interface{}) { 107 | return t.root.sum() 108 | } 109 | 110 | type trace struct { 111 | message string 112 | alias string 113 | state interface{} 114 | startTime time.Time 115 | endTime time.Time 116 | children []*trace 117 | } 118 | 119 | func (t *trace) begin(message string, keysAndValues ...interface{}) { 120 | t.message = message 121 | if len(keysAndValues) <= 2 { 122 | t.alias = fmt.Sprintf("%v-%v", keysAndValues...) 123 | } else { 124 | t.alias = fmt.Sprintf("%v-%v", keysAndValues[0], keysAndValues[1]) 125 | } 126 | t.startTime = time.Now() 127 | } 128 | 129 | func (t *trace) end(state interface{}) { 130 | if state == nil { 131 | t.state = "done" 132 | } else { 133 | t.state = state 134 | } 135 | t.endTime = time.Now() 136 | } 137 | 138 | func (t *trace) addChild(child *trace) { 139 | if t.children == nil { 140 | t.children = make([]*trace, 0) 141 | } 142 | t.children = append(t.children, child) 143 | } 144 | 145 | func (t *trace) sum() (string, []interface{}) { 146 | var message string 147 | var namesAndDurations = make([]interface{}, 0) 148 | if t.state == nil { 149 | t.end("not a ended trace") 150 | message = t.message + "(not a ended trace)" 151 | namesAndDurations = []interface{}{ 152 | t.alias, 153 | "not a ended trace", 154 | } 155 | } else { 156 | message = t.message + fmt.Sprintf("(%v)", t.state) 157 | namesAndDurations = []interface{}{ 158 | t.alias, 159 | t.endTime.Sub(t.startTime), 160 | } 161 | } 162 | for i, child := range t.children { 163 | m, nad := child.sum() 164 | if i == 0 { 165 | message = message + ">>" + m 166 | } else { 167 | message = message + ">" + m 168 | } 169 | namesAndDurations = append(namesAndDurations, nad...) 170 | } 171 | return message, namesAndDurations 172 | } 173 | -------------------------------------------------------------------------------- /log/trace_test.go: -------------------------------------------------------------------------------- 1 | package log 2 | 3 | import ( 4 | "testing" 5 | ) 6 | 7 | func TestTraceRace(t *testing.T) { 8 | traceBegin("1", "hello", "my", "husband") 9 | go traceBegin("2", "hello", "my", "wift") 10 | } 11 | -------------------------------------------------------------------------------- /process/image.go: -------------------------------------------------------------------------------- 1 | package process 2 | 3 | import ( 4 | "github.com/ctrip-nephele/gmagick" 5 | ) 6 | 7 | const ( 8 | imageIsBlob = iota 9 | imageIsWand 10 | ) 11 | 12 | type image struct { 13 | state int 14 | blob []byte 15 | wand *gmagick.MagickWand 16 | } 17 | 18 | func (i *image) Bytes() ([]byte, error) { 19 | if i.state == imageIsWand { 20 | i.blob = i.wand.WriteImageBlob() 21 | if len(i.blob) == 0 { 22 | return nil, i.wand.GetLastError() 23 | } 24 | i.state = imageIsBlob 25 | } 26 | return i.blob, nil 27 | } 28 | 29 | func (i *image) SetBlob(blob []byte) { 30 | i.blob = blob 31 | } 32 | 33 | func (i *image) Wand() (*gmagick.MagickWand, error) { 34 | if i.state == imageIsBlob { 35 | i.wand = gmagick.NewMagickWand() 36 | err := i.wand.ReadImageBlob(i.blob) 37 | if err != nil { 38 | return nil, err 39 | } 40 | i.state = imageIsWand 41 | } 42 | return i.wand, nil 43 | } 44 | -------------------------------------------------------------------------------- /process/process.go: -------------------------------------------------------------------------------- 1 | package process 2 | 3 | import ( 4 | "github.com/ctripcorp/nephele/command" 5 | 6 | "context" 7 | "errors" 8 | "fmt" 9 | "strings" 10 | ) 11 | 12 | var Config map[string]string 13 | 14 | func Init() {} 15 | 16 | // 1. processString: "image/format,v_png/resize,w_100,h_200" 17 | // 18 | // 2. commandString: "image" | "format,v_png" | "resize,w_100,h_200" 19 | // 20 | // 3. word: "format" | "v_png" | "resize" | "w_100,h_200" 21 | // 22 | // 4. name: "format" | "resize" 23 | // 24 | // 5. k: "v" | "w" | "h" 25 | // 26 | // 6. v: "png" | "100" | "200" 27 | 28 | func Parse(ctx context.Context, processString string) ([]command.Command, error) { 29 | commands := make([]command.Command, 0) 30 | if processString == "" { 31 | return commands, nil 32 | } 33 | commandStrings := strings.Split(processString, "/") 34 | for i, commandString := range commandStrings { 35 | if i == 0 { 36 | if commandString != "image" { 37 | return nil, errors.New("invalid process category: " + commandString) 38 | } 39 | continue 40 | } 41 | words := strings.Split(commandString, ",") 42 | name := words[0] 43 | cc, ok := command.List()[name] 44 | if !ok { 45 | return nil, fmt.Errorf("invalid command name: %s", name) 46 | } 47 | option := make(map[string]string) 48 | for j := 1; j < len(words); j++ { 49 | kv := strings.Split(words[j], "_") 50 | if len(kv) != 2 { 51 | continue 52 | } 53 | if kv[0] == "" { 54 | continue 55 | } 56 | if kv[1] == "" { 57 | continue 58 | } 59 | option[kv[0]] = kv[1] 60 | } 61 | c := cc() 62 | err := c.Verify(ctx, option) 63 | if err != nil { 64 | return nil, err 65 | } 66 | commands = append(commands, c) 67 | } 68 | return commands, nil 69 | } 70 | 71 | func Do(ctx context.Context, blob []byte, commands []command.Command) ([]byte, error) { 72 | i := &image{ 73 | state: imageIsBlob, 74 | blob: blob, 75 | wand: nil, 76 | } 77 | for _, command := range commands { 78 | if command.Support() == "wand" { 79 | w, err := i.Wand() 80 | if err != nil { 81 | return nil, err 82 | } 83 | err = command.ExecuteOnWand(ctx, w) 84 | if err != nil { 85 | return nil, err 86 | } 87 | continue 88 | } 89 | if command.Support() == "blob" { 90 | b, err := i.Bytes() 91 | if err != nil { 92 | return nil, err 93 | } 94 | b, err = command.ExecuteOnBlob(ctx, b) 95 | if err != nil { 96 | return nil, err 97 | } 98 | i.SetBlob(b) 99 | continue 100 | } 101 | } 102 | return i.Bytes() 103 | } 104 | -------------------------------------------------------------------------------- /server/get.go: -------------------------------------------------------------------------------- 1 | package server 2 | 3 | import ( 4 | "github.com/gin-gonic/gin" 5 | 6 | "github.com/ctripcorp/nephele/interpret" 7 | "github.com/ctripcorp/nephele/process" 8 | "github.com/ctripcorp/nephele/storage" 9 | 10 | "context" 11 | ) 12 | 13 | func get(c *gin.Context) { 14 | 15 | var ( 16 | ctx = c.MustGet("context").(context.Context) 17 | ) 18 | 19 | key, proc, err := interpret.Do(c) 20 | if err != nil { 21 | c.String(400, err.Error()) 22 | return 23 | } 24 | 25 | commands, err := process.Parse(ctx, proc) 26 | if err != nil { 27 | c.String(400, err.Error()) 28 | return 29 | } 30 | 31 | blob, _, err := storage.Download(ctx, key) 32 | if err != nil { 33 | c.String(400, err.Error()) 34 | return 35 | } 36 | 37 | blob, err = process.Do(ctx, blob, commands) 38 | if err != nil { 39 | c.String(400, err.Error()) 40 | return 41 | } 42 | c.Writer.Write(blob) 43 | } 44 | -------------------------------------------------------------------------------- /server/ping/ping.go: -------------------------------------------------------------------------------- 1 | package ping 2 | 3 | import ( 4 | "github.com/ctripcorp/nephele/server" 5 | "github.com/gin-gonic/gin" 6 | ) 7 | 8 | func init() { 9 | server.Register(configurater) 10 | } 11 | 12 | func configurater(r *gin.Engine) { 13 | r.GET("/ping", func(c *gin.Context) { 14 | c.String(200, "pong") 15 | }) 16 | } 17 | -------------------------------------------------------------------------------- /server/server.go: -------------------------------------------------------------------------------- 1 | package server 2 | 3 | import ( 4 | "github.com/gin-gonic/gin" 5 | cors "github.com/rs/cors/wrapper/gin" 6 | "github.com/satori/go.uuid" 7 | 8 | "context" 9 | "time" 10 | ) 11 | 12 | var Config struct { 13 | Port string `toml:"port"` 14 | Recovery struct{} 15 | Entrance struct { 16 | RequestBuffer int `toml:"request_buffer"` 17 | RequestTimeoutMillisecond int `toml:"request_timeout_millisecond"` 18 | MaxConcurrency int `toml:"max_concurrency"` 19 | } 20 | CORS struct { 21 | AllowAllOrigins bool `toml:"allow_all_origins"` 22 | MaxAge int `toml:"max_age"` 23 | OptionPassThrough bool `toml:"option_pass_through"` 24 | AllowedOrigins []string `toml:"allowed_origins"` 25 | AllowedMethods []string `toml:"allowed_methods"` 26 | AllowedHeaders []string `toml:"allowed_headers"` 27 | ExposedHeaders []string `toml:"exposed_headers"` 28 | } 29 | } 30 | 31 | func Entrance() gin.HandlerFunc { 32 | var conf = Config.Entrance 33 | return func(c *gin.Context) { 34 | var ( 35 | ctx context.Context 36 | cancel context.CancelFunc 37 | uuidUUID uuid.UUID 38 | err error 39 | ) 40 | uuidUUID, err = uuid.NewV1() 41 | if err != nil { 42 | c.String(500, err.Error()) 43 | return 44 | } 45 | ctx = context.WithValue(context.Background(), "contextID", uuidUUID.String()) 46 | ctx = context.WithValue(ctx, "request", c.Request) 47 | ctx, cancel = context.WithTimeout(ctx, 48 | time.Duration(conf.RequestTimeoutMillisecond)*time.Millisecond) 49 | c.Set("context", ctx) 50 | c.Next() 51 | cancel() 52 | } 53 | } 54 | 55 | func Recovery() gin.HandlerFunc { 56 | return func(c *gin.Context) { 57 | defer func() { 58 | if err := recover(); err != nil { 59 | c.AbortWithStatus(500) 60 | } 61 | }() 62 | c.Next() 63 | } 64 | } 65 | 66 | func CORS() gin.HandlerFunc { 67 | var conf = Config.CORS 68 | 69 | if conf.AllowAllOrigins { 70 | return cors.AllowAll() 71 | } 72 | return cors.New(cors.Options{ 73 | MaxAge: conf.MaxAge, 74 | OptionsPassthrough: conf.OptionPassThrough, 75 | AllowedOrigins: conf.AllowedOrigins, 76 | AllowedMethods: conf.AllowedMethods, 77 | AllowedHeaders: conf.AllowedHeaders, 78 | ExposedHeaders: conf.ExposedHeaders, 79 | }) 80 | } 81 | 82 | func Run() { 83 | r := gin.New() 84 | 85 | r.Use(Entrance()) 86 | r.Use(Recovery()) 87 | r.Use(CORS()) 88 | 89 | registeredConfigurater(r) 90 | 91 | r.GET("/image/:key", get) 92 | r.Run(Config.Port) 93 | } 94 | 95 | var registeredConfigurater func(r *gin.Engine) 96 | 97 | func Register(configurater func(r *gin.Engine)) { 98 | registeredConfigurater = configurater 99 | } 100 | -------------------------------------------------------------------------------- /storage/neph/file.go: -------------------------------------------------------------------------------- 1 | package neph 2 | 3 | import ( 4 | "errors" 5 | "io/ioutil" 6 | "os" 7 | "path/filepath" 8 | 9 | stor "github.com/ctripcorp/nephele/storage" 10 | ) 11 | 12 | type file struct { 13 | root string 14 | key string 15 | } 16 | 17 | func (f *file) Key() string { 18 | return f.key 19 | } 20 | 21 | func (f *file) Exist() (bool, string, error) { 22 | _, err := os.Stat(filepath.Join(f.root, f.key)) 23 | if err == nil { 24 | return true, "", nil 25 | } 26 | if os.IsNotExist(err) { 27 | return false, "", nil 28 | } 29 | return false, "", err 30 | } 31 | 32 | func (f *file) Meta() (stor.Fetcher, error) { 33 | return nil, errors.New("Meta not supported") 34 | } 35 | 36 | func (f *file) Append(blob []byte, index int64, kvs ...stor.KV) (int64, string, error) { 37 | return index, "", errors.New("Append not supported") 38 | } 39 | 40 | func (f *file) Delete() (string, error) { 41 | return "", os.Remove(f.Key()) 42 | } 43 | 44 | func (f *file) Bytes() ([]byte, string, error) { 45 | blob, err := ioutil.ReadFile(filepath.Join(f.root, f.key)) 46 | return blob, "", err 47 | } 48 | 49 | func (f *file) SetMeta(...stor.KV) error { 50 | return errors.New("SetMeta not supported") 51 | } 52 | -------------------------------------------------------------------------------- /storage/neph/neph.go: -------------------------------------------------------------------------------- 1 | package neph 2 | 3 | import ( 4 | stor "github.com/ctripcorp/nephele/storage" 5 | ) 6 | 7 | func init() { 8 | stor.Register(New) 9 | } 10 | 11 | func New(config map[string]string) stor.Storage { 12 | return &storage{ 13 | root: config["root"], 14 | } 15 | } 16 | -------------------------------------------------------------------------------- /storage/neph/storage.go: -------------------------------------------------------------------------------- 1 | package neph 2 | 3 | import ( 4 | stor "github.com/ctripcorp/nephele/storage" 5 | "io/ioutil" 6 | "path/filepath" 7 | ) 8 | 9 | type storage struct { 10 | root string 11 | } 12 | 13 | func (s *storage) File(key string) stor.File { 14 | return &file{s.root, key} 15 | } 16 | 17 | func (s *storage) Iterator(prefix string, lastKey string) stor.Iterator { 18 | return nil 19 | } 20 | 21 | func (s *storage) StoreFile(key string, blob []byte, kvs ...stor.KV) (string, error) { 22 | return "", ioutil.WriteFile(filepath.Join(s.root, key), []byte(blob), 0666) 23 | } 24 | -------------------------------------------------------------------------------- /storage/storage.go: -------------------------------------------------------------------------------- 1 | package storage 2 | 3 | import ( 4 | "context" 5 | "errors" 6 | ) 7 | 8 | type Fetcher interface { 9 | Fetch(string) string 10 | } 11 | 12 | type KV [2]string 13 | 14 | type File interface { 15 | Key() string 16 | 17 | Exist() (bool, string, error) 18 | 19 | Meta() (Fetcher, error) 20 | 21 | Append([]byte, int64, ...KV) (int64, string, error) 22 | 23 | Delete() (string, error) 24 | 25 | Bytes() ([]byte, string, error) 26 | 27 | SetMeta(...KV) error 28 | } 29 | 30 | type Iterator interface { 31 | Next() (File, error) 32 | 33 | LastKey() string 34 | } 35 | 36 | type Storage interface { 37 | File(string) File 38 | 39 | Iterator(prefix string, lastKey string) Iterator 40 | 41 | StoreFile(string, []byte, ...KV) (string, error) 42 | } 43 | 44 | var Config map[string]string 45 | 46 | var instance Storage 47 | 48 | var NewStorage func(map[string]string) Storage 49 | 50 | func Init() { 51 | instance = NewStorage(Config) 52 | } 53 | 54 | func Register(ns func(map[string]string) Storage) { 55 | if Config["type"] == "inline" { 56 | panic("configured to apply inline storage") 57 | } 58 | NewStorage = ns 59 | } 60 | 61 | func Download(ctx context.Context, key string) ([]byte, string, error) { 62 | var ( 63 | blob []byte 64 | rid string 65 | err error 66 | done chan int = make(chan int) 67 | ) 68 | go func() { 69 | blob, rid, err = instance.File(key).Bytes() 70 | close(done) 71 | }() 72 | select { 73 | case <-ctx.Done(): 74 | return nil, "", errors.New("context timeout") 75 | case <-done: 76 | return blob, rid, err 77 | } 78 | } 79 | 80 | func Upload(ctx context.Context, key string, blob []byte, options ...KV) (string, error) { 81 | return instance.StoreFile(key, blob, options...) 82 | } 83 | -------------------------------------------------------------------------------- /thirdparty/graphicsmagick/README.md: -------------------------------------------------------------------------------- 1 | ## Nephele GraphicsMagick安装指南 2 | 3 | 4 | ### 安装 5 | * unix系统（centos,ubuntu,debian），执行setup_unix.sh 6 | * windows系统（win7及以上），执行setup_windows.bat 7 | 8 | ### 检查安装 9 | 在命令窗口输入gm,看到类似如下结果代表安装成功, 10 | ``` 11 | GraphicsMagick 1.3.21 2015-02-28 Q8 http://www.GraphicsMagick.org/ 12 | Copyright (C) 2002-2014 GraphicsMagick Group. 13 | Additional copyrights and licenses apply to this software. 14 | See http://www.GraphicsMagick.org/www/Copyright.html for details. 15 | Usage: gm command [options ...] 16 | 17 | Where commands include: 18 | animate - animate a sequence of images 19 | batch - issue multiple commands in interactive or batch mode 20 | benchmark - benchmark one of the other commands 21 | compare - compare two images 22 | composite - composite images together 23 | conjure - execute a Magick Scripting Language (MSL) XML script 24 | convert - convert an image or sequence of images 25 | display - display an image on a workstation running X 26 | help - obtain usage message for named command 27 | identify - describe an image or image sequence 28 | import - capture an application or X server screen 29 | mogrify - transform an image or sequence of images 30 | montage - create a composite image (in a grid) from separate images 31 | time - time one of the other commands 32 | version - obtain release version 33 | ``` 34 | 35 | ### 安装失败怎么办？ 36 | 如果未能通过脚本成功安装，可以通过源码编译安装http://www.graphicsmagick.org/README.html 37 | 38 | ### 注意： 39 | 脚本执行过程中如果被终止，那么临时文件epel-aliyun.repo可能会留在/etc/yum.repos.d/目录。 40 | -------------------------------------------------------------------------------- /thirdparty/graphicsmagick/download.vbs: -------------------------------------------------------------------------------- 1 | WSH.Echo "downloading,please wait!" 2 | Set xPost = CreateObject("Microsoft.XMLHTTP") 3 | xPost.Open "GET","http://file.c-ctrip.com/files/3/car22/zip/7a7/cf5/e08/a24538a1ed74420f95745097fa987fd8.zip",0 4 | xPost.Send() 5 | Set sGet = CreateObject("ADODB.Stream") 6 | sGet.Mode = 3 7 | sGet.Type = 1 8 | sGet.Open() 9 | sGet.Write(xPost.responseBody) 10 | sGet.SaveToFile "gm.zip",2 11 | WSH.Echo "download end" 12 | wscript.sleep 1000 13 | 14 | Class ZipCompressor 15 | 16 | Private objFileSystemObject 17 | Private objShellApplication 18 | Private objWScriptShell 19 | Private objScriptingDictionary 20 | Private objWMIService 21 | Private COPY_OPTIONS 22 | 23 | Private Sub Class_Initialize() 24 | Set objFileSystemObject = WSH.CreateObject("Scripting.FileSystemObject") 25 | Set objShellApplication = WSH.CreateObject("Shell.Application") 26 | Set objWScriptShell = WSH.CreateObject("WScript.Shell") 27 | Set objScriptingDictionary = WSH.CreateObject("Scripting.Dictionary") 28 | Dim strComputer 29 | strComputer = "." 30 | Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\cimv2") 31 | 32 | ' COPY_OPTIONS 33 | ' 4 Do not display a progress dialog box. 34 | ' 16 Respond with "Yes to All" for 35 | ' any dialog box that is displayed. 36 | ' 512 Do not confirm the creation of a new 37 | ' directory if the operation requires one to be created. 38 | ' 1024 Do not display a user interface if an error occurs. 39 | 40 | COPY_OPTIONS = 4 + 16 + 512 + 1024 41 | End Sub 42 | 43 | Private Sub Class_Terminate() 44 | Set objWMIService = Nothing 45 | objScriptingDictionary.RemoveAll 46 | Set objScriptingDictionary = Nothing 47 | Set objWScriptShell = Nothing 48 | Set objShellApplication = Nothing 49 | Set objFileSystemObject = Nothing 50 | End Sub 51 | 52 | 53 | Private Sub makeEmptyZipFile(pathToZipFile) 54 | Dim file 55 | Set file = objFileSystemObject.CreateTextFile(pathToZipFile) 56 | file.Write Chr(80) & Chr(75) & Chr(5) & Chr(6) & String(18, 0) 57 | file.Close 58 | End Sub 59 | 60 | Private Function pathToAbsolute(fileName) 61 | Dim i, file, files 62 | files = Split(fileName, ";") 63 | ReDim tmpFiles(UBound(files)) 64 | 65 | i = 0 66 | For Each file in files 67 | If file<>"" Then 68 | file = objWScriptShell.ExpandEnvironmentStrings(file) 69 | file = objFileSystemObject.GetAbsolutePathName(file) 70 | 71 | tmpFiles(i) = file 72 | i = i+1 73 | End If 74 | Next 75 | If i-1 > 0 And i-1 < UBound(files) Then ReDim Preserve tmpFiles(i-1) 76 | pathToAbsolute = Join(tmpFiles, ";") 77 | Erase tmpFiles 78 | End Function 79 | 80 | Private Function pathCombine(fileName, nextFileName) 81 | Dim files, lastIndex 82 | files = Split(fileName, "\") 83 | lastIndex = UBound(files) 84 | 85 | If files(lastIndex)<>"" Then 86 | lastIndex = lastIndex + 1 87 | ReDim Preserve files(lastIndex) 88 | End If 89 | 90 | files(lastIndex) = nextFileName 91 | 92 | pathCombine = Join(files, "\") 93 | Erase files 94 | End Function 95 | 96 | Private Function pathSplit(fileName) 97 | Dim fileSplitted(2) 98 | fileSplitted(0) = objFileSystemObject.GetDriveName(fileName) 99 | fileSplitted(2) = objFileSystemObject.GetFileName(fileName) 100 | fileSplitted(1) = Mid(fileName, Len(fileSplitted(0))+1, _ 101 | Len(fileName) - Len(fileSplitted(0)) - Len(fileSplitted(2))) 102 | 103 | pathSplit = fileSplitted 104 | End Function 105 | 106 | Private Function pathSplitForQuery(fileName) 107 | Dim fileSplitted 108 | fileSplitted = pathSplit(fileName) 109 | fileSplitted(1) = Replace(fileSplitted(1), "\", "\\") 110 | If Right(fileSplitted(1), 2) <> "\\" Then 111 | fileSplitted(1) = fileSplitted(1) & "\\" 112 | End If 113 | ' http://msdn.microsoft.com/en-us/library/windows/desktop/aa392263(v=vs.85).aspx 114 | fileSplitted(2) = Replace(fileSplitted(2), "_", "[_]") 115 | fileSplitted(2) = Replace(fileSplitted(2), "*", "%") 116 | fileSplitted(2) = Replace(fileSplitted(2), "?", "_") 117 | pathSplitForQuery = fileSplitted 118 | End Function 119 | 120 | Private Function buildQuerySQL(fileName) 121 | Dim fileSplitted, file, ext 122 | fileSplitted = pathSplitForQuery(fileName) 123 | 124 | Dim lastDotIndex 125 | 126 | file = "%" : ext = "%" 127 | If fileSplitted(2)<>"" Then 128 | lastDotIndex = InStrRev(fileSplitted(2), ".") 129 | file = fileSplitted(2) 130 | End If 131 | 132 | If lastDotIndex>0 Then 133 | ext = Mid(fileSplitted(2), _ 134 | lastDotIndex+1, Len(fileSplitted(2)) - lastDotIndex) 135 | file = Left(fileSplitted(2), Len(fileSplitted(2)) - Len(ext) - 1) 136 | End If 137 | 138 | ' http://msdn.microsoft.com/en-us/library/windows/desktop/aa387236(v=vs.85).aspx 139 | buildQuerySQL = "SELECT * FROM CIM_DataFile" & _ 140 | " WHERE Drive='" & fileSplitted(0) & "' AND" & _ 141 | " (FileName LIKE '" & file & "') AND" & _ 142 | " (Extension LIKE '" & ext & "') AND" & _ 143 | " (Path='" & fileSplitted(1) &"')" 144 | End Function 145 | 146 | Private Function deleteFile(fileName) 147 | deleteFile = False 148 | If objFileSystemObject.FileExists(fileName) Then 149 | objFileSystemObject.DeleteFile fileName 150 | deleteFile = True 151 | End If 152 | End Function 153 | 154 | Private Sub compress_(ByVal fileName, ByRef zipFile) 155 | Dim objFile, srcFile, srcFiles 156 | srcFiles = Split(fileName, ";") 157 | 158 | Dim colFiles 159 | 160 | ' http://msdn.microsoft.com/en-us/library/bb787866(VS.85).aspx 161 | For Each srcFile In srcFiles 162 | If objFileSystemObject.FolderExists(srcFile) Then 163 | Set objFile = objShellApplication.NameSpace(srcFile) 164 | If Not (objFile Is Nothing) Then 165 | zipFile.CopyHere objFile.Items, COPY_OPTIONS 166 | Do Until objFile.Items.Count <= zipFile.Items.Count 167 | WScript.Sleep(200) 168 | Loop 169 | End If 170 | Set objFile = Nothing 171 | ElseIf objFileSystemObject.FileExists(srcFile) Then 172 | zipFile.CopyHere srcFile, COPY_OPTIONS 173 | WScript.Sleep(200) 174 | Else 175 | Set colFiles = objWMIService.ExecQuery(buildQuerySQL(srcFile)) 176 | For Each objFile in colFiles 177 | srcFile = objFile.Name 178 | zipFile.CopyHere srcFile, COPY_OPTIONS 179 | WScript.Sleep(200) 180 | Next 181 | Set colFiles = Nothing 182 | End If 183 | Next 184 | End Sub 185 | 186 | Public Sub add(fileName) 187 | objScriptingDictionary.Add pathToAbsolute(fileName), "" 188 | End Sub 189 | 190 | ' Private Function makeTempDir() 191 | ' Dim tmpFolder, tmpName 192 | ' tmpFolder = objFileSystemObject.GetSpecialFolder(2) 193 | ' tmpName = objFileSystemObject.GetTempName() 194 | ' makeTempDir = pathCombine(tmpFolder, tmpName) 195 | ' objFileSystemObject.CreateFolder makeTempDir 196 | ' End Function 197 | 198 | Public Function compress(srcFileName, desFileName) 199 | Dim srcAbsFileName, desAbsFileName 200 | 201 | srcAbsFileName = "" 202 | If srcFileName<>"" Then 203 | srcAbsFileName = pathToAbsolute(srcFileName) 204 | End If 205 | 206 | desAbsFileName = pathToAbsolute(desFileName) 207 | 208 | If objFileSystemObject.FolderExists(desAbsFileName) Then 209 | compress = -1 210 | Exit Function 211 | End If 212 | 213 | ' That zip file already exists - deleting it. 214 | deleteFile desAbsFileName 215 | 216 | makeEmptyZipFile desAbsFileName 217 | 218 | Dim zipFile 219 | Set zipFile = objShellApplication.NameSpace(desAbsFileName) 220 | 221 | If srcAbsFileName<>"" Then 222 | compress_ srcAbsFileName, zipFile 223 | End If 224 | compress = zipFile.Items.Count 225 | 226 | Dim objKeys, i 227 | objKeys = objScriptingDictionary.Keys 228 | For i = 0 To objScriptingDictionary.Count -1 229 | compress_ objKeys(i), zipFile 230 | Next 231 | 232 | compress = compress + i 233 | 234 | Set zipFile = Nothing 235 | End Function 236 | 237 | Public Function decompress(srcFileName, desFileName) 238 | Dim srcAbsFileName, desAbsFileName 239 | srcAbsFileName = pathToAbsolute(srcFileName) 240 | desAbsFileName = pathToAbsolute(desFileName) 241 | 242 | If Not objFileSystemObject.FileExists(srcAbsFileName) Then 243 | decompress = -1 244 | Exit Function 245 | End If 246 | 247 | If Not objFileSystemObject.FolderExists(desAbsFileName) Then 248 | decompress = -1 249 | Exit Function 250 | End If 251 | 252 | Dim zipFile, objFile 253 | Set zipFile = objShellApplication.NameSpace(srcAbsFileName) 254 | Set objFile = objShellApplication.NameSpace(desAbsFileName) 255 | objFile.CopyHere zipFile.Items, COPY_OPTIONS 256 | Do Until zipFile.Items.Count <= objFile.Items.Count 257 | WScript.Sleep(200) 258 | Loop 259 | 260 | decompress = objFile.Items.Count 261 | Set objFile = Nothing 262 | Set zipFile = Nothing 263 | End Function 264 | End Class 265 | 266 | Dim zip 267 | Set zip = New ZipCompressor 268 | zip.decompress "gm.zip", "gm" 269 | Set zip = Nothing 270 | 271 | -------------------------------------------------------------------------------- /thirdparty/graphicsmagick/setup_unix.sh: -------------------------------------------------------------------------------- 1 | #! /bin/bash 2 | 3 | echo `uname -a` 4 | os=`uname -a` 5 | 6 | centos(){ 7 | sudo yum clean all 8 | sudo yum makecache 9 | sudo yum install -y GraphicsMagick-devel --enablerepo=epel 10 | sudo rm /etc/yum.repos.d/epel-aliyun.repo 11 | } 12 | 13 | centos7(){ 14 | sudo wget http://mirrors.aliyun.com/repo/epel-7.repo 15 | sudo mv epel-7.repo /etc/yum.repos.d/epel-aliyun.repo 16 | sudo chmod +x /etc/yum.repos.d/epel-aliyun.repo 17 | } 18 | 19 | centos6(){ 20 | sudo wget  http://mirrors.aliyun.com/repo/epel-6.repo 21 | sudo mv epel-6.repo /etc/yum.repos.d/epel-aliyun.repo 22 | sudo chmod +x /etc/yum.repos.d/epel-aliyun.repo 23 | } 24 | 25 | ubuntu_debian(){ 26 | sudo apt-get -y install gcc 27 | sudo apt-get -y install pkg-config 28 | 29 | sudo apt-get -y --purge remove graphicsmagick* 30 | sudo apt-get -y install libgraphicsmagick-dev #in fact libgraphicsmagick1-dev will be installed 31 | sudo apt-get -y install graphicsmagick* 32 | } 33 | 34 | macOS(){ 35 | brew install -y /GraphicsMagick*/ 36 | } 37 | 38 | [[ $os =~ "el7" ]] &¢os7&¢os 39 | [[ $os =~ "el6" ]] &¢os6&¢os 40 | [[ $os =~ "Ubuntu" ]] &&ubuntu_debian 41 | [[ $os =~ "Debian" ]] &&ubuntu_debian 42 | [[ $os =~ "Mac" ]] &&macOS 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | -------------------------------------------------------------------------------- /thirdparty/graphicsmagick/setup_windows.bat: -------------------------------------------------------------------------------- 1 | mkdir gm 2 | CScript download.vbs 3 | cd gm/ 4 | GraphicsMagick-1.3.28-Q8-win64-dll.exe -------------------------------------------------------------------------------- /throttle/pool.go: -------------------------------------------------------------------------------- 1 | package throttle 2 | 3 | import ( 4 | "errors" 5 | "time" 6 | ) 7 | 8 | type poolItem struct{} 9 | 10 | type pool struct { 11 | items chan *poolItem 12 | } 13 | 14 | func (p *pool) init(max int) { 15 | p.items = make(chan *poolItem, max) 16 | for i := 0; i < max; i++ { 17 | p.items <- &poolItem{} 18 | } 19 | } 20 | 21 | func (p *pool) get(timeout time.Duration) error { 22 | items := p.items 23 | if items == nil { 24 | return errors.New("pool is closed") 25 | } 26 | select { 27 | case <-items: 28 | return nil 29 | case <-time.After(timeout): 30 | return errors.New("timeout") 31 | } 32 | } 33 | 34 | func (p *pool) put() error { 35 | items := p.items 36 | if items == nil { 37 | return errors.New("pool is closed") 38 | } 39 | items <- &poolItem{} 40 | return nil 41 | } 42 | -------------------------------------------------------------------------------- /throttle/throttle.go: -------------------------------------------------------------------------------- 1 | package throttle 2 | 3 | import ( 4 | "fmt" 5 | "sync" 6 | ) 7 | 8 | var workflows sync.Map 9 | 10 | func Workflow(name string, maxConcurrency int, maxWait int) *workflow { 11 | id := fmt.Sprintf("%s-%d-%d", name, maxConcurrency, maxWait) 12 | l, ok := workflows.Load(id) 13 | if !ok { 14 | c := &pool{} 15 | c.init(maxConcurrency) 16 | w := &pool{} 17 | w.init(maxWait) 18 | 19 | workflows.Store(id, &workflow{ 20 | concurrency: c, 21 | wait: w, 22 | }) 23 | 24 | l, ok = workflows.Load(id) 25 | } 26 | if !ok { 27 | return nil 28 | } 29 | return l.(*workflow) 30 | } 31 | -------------------------------------------------------------------------------- /throttle/workflow.go: -------------------------------------------------------------------------------- 1 | package throttle 2 | 3 | import ( 4 | "errors" 5 | "time" 6 | ) 7 | 8 | type workflow struct { 9 | concurrency *pool 10 | wait *pool 11 | } 12 | 13 | func (l *workflow) Do(timeout time.Duration) error { 14 | if l.wait.get(0) != nil { 15 | return errors.New("fast fail") 16 | } 17 | defer l.wait.put() 18 | return l.concurrency.get(timeout) 19 | } 20 | 21 | func (l *workflow) Done() error { 22 | return l.concurrency.put() 23 | } 24 | -------------------------------------------------------------------------------- /util/stringx.go: -------------------------------------------------------------------------------- 1 | package util 2 | 3 | import ( 4 | "bytes" 5 | "strings" 6 | ) 7 | 8 | //SubString sub string 9 | func SubString(str string, start, length int) string { 10 | rs := []rune(str) 11 | rl := len(rs) 12 | end := 0 13 | if start < 0 { 14 | start = rl - 1 + start 15 | } 16 | end = start + length 17 | if start > end { 18 | start, end = end, start 19 | } 20 | if start < 0 { 21 | start = 0 22 | } 23 | if start > rl { 24 | start = rl 25 | } 26 | if end < 0 { 27 | end = 0 28 | } 29 | if end > rl { 30 | end = rl 31 | } 32 | return string(rs[start:end]) 33 | } 34 | 35 | // JoinString []string join 36 | func JoinString(args ...string) string { 37 | var buf bytes.Buffer 38 | for _, v := range args { 39 | buf.WriteString(v) 40 | } 41 | return buf.String() 42 | } 43 | 44 | //Cover cover string 45 | func Cover(s, converV string, length int) string { 46 | currentLen := len(s) 47 | for i := 0; i < length-currentLen; i++ { 48 | s = converV + s 49 | } 50 | return s 51 | } 52 | 53 | //TrimPrefixSlash trim prefix / 54 | func TrimPrefixSlash(path string) string { 55 | path = strings.Replace(path, "\\", "/", -1) 56 | if strings.HasPrefix(path, "/") { 57 | for { 58 | path = strings.TrimPrefix(path, "/") 59 | if !strings.HasPrefix(path, "/") { 60 | break 61 | } 62 | } 63 | } 64 | return path 65 | } 66 | -------------------------------------------------------------------------------- /util/util.go: -------------------------------------------------------------------------------- 1 | package util 2 | 3 | import ( 4 | "fmt" 5 | "io/ioutil" 6 | "os" 7 | "os/user" 8 | "path/filepath" 9 | 10 | "github.com/BurntSushi/toml" 11 | "golang.org/x/text/encoding/unicode" 12 | "golang.org/x/text/transform" 13 | ) 14 | 15 | func HomePath() (string, error) { 16 | var homeDir string 17 | // By default, store image and log files in current users home directory 18 | u, err := user.Current() 19 | if err == nil { 20 | homeDir = u.HomeDir 21 | } else if os.Getenv("HOME") != "" { 22 | homeDir = os.Getenv("HOME") 23 | } else { 24 | return homeDir, fmt.Errorf("failed to determine current user") 25 | } 26 | return filepath.Join(homeDir, "nephele"), nil 27 | } 28 | 29 | func FromToml(path string, v interface{}) error { 30 | var err error 31 | var blob []byte 32 | 33 | if blob, err = ioutil.ReadFile(path); err != nil { 34 | return err 35 | } 36 | 37 | // Handle any potential Byte-Order-Marks that may be in the config file. 38 | // This is for Windows compatibility only. 39 | bom := unicode.BOMOverride(transform.Nop) 40 | if blob, _, err = transform.Bytes(bom, blob); err != nil { 41 | return err 42 | } 43 | 44 | _, err = toml.Decode(string(blob), v) 45 | return err 46 | } 47 | 48 | func InArray(val string, arr []string) bool { 49 | for _, value := range arr { 50 | if val == value { 51 | return true 52 | break 53 | } 54 | } 55 | return false 56 | } 57 | -------------------------------------------------------------------------------- /vendor/vendor.json: -------------------------------------------------------------------------------- 1 | { 2 | "comment": "", 3 | "ignore": "test", 4 | "package": [ 5 | { 6 | "checksumSHA1": "Pc2ORQp+VY3Un/dkh4QwLC7R6lE=", 7 | "path": "github.com/BurntSushi/toml", 8 | "revision": "a368813c5e648fee92e5f6c30e3944ff9d5e8895", 9 | "revisionTime": "2017-06-26T11:06:00Z" 10 | }, 11 | { 12 | "checksumSHA1": "88ntCdB+fP/2EYKx+foTZ3Ja+rc=", 13 | "path": "github.com/ctrip-nephele/gmagick", 14 | "revision": "67ca701d3abae744bb5038485e42fa5e708002f0", 15 | "revisionTime": "2018-05-28T14:47:05Z" 16 | }, 17 | { 18 | "checksumSHA1": "53b5mH+hdqwDzlzTMGAOx4MLROc=", 19 | "path": "github.com/ctrip-nephele/gmagick/types", 20 | "revision": "67ca701d3abae744bb5038485e42fa5e708002f0", 21 | "revisionTime": "2018-05-28T14:47:05Z" 22 | }, 23 | { 24 | "checksumSHA1": "QeKwBtN2df+j+4stw3bQJ6yO4EY=", 25 | "path": "github.com/gin-contrib/sse", 26 | "revision": "22d885f9ecc78bf4ee5d72b937e4bbcdc58e8cae", 27 | "revisionTime": "2017-01-09T09:34:21Z" 28 | }, 29 | { 30 | "checksumSHA1": "ipkpl1oSI3ycEfTBdh0S6vDwrDY=", 31 | "path": "github.com/gin-gonic/gin", 32 | "revision": "5d3f30cfc81f9109850fa9043a19783cbbde68a5", 33 | "revisionTime": "2018-02-23T01:09:33Z" 34 | }, 35 | { 36 | "checksumSHA1": "3Ttl4CxOEl+NCstutCFyECv0CWs=", 37 | "path": "github.com/gin-gonic/gin/binding", 38 | "revision": "5d3f30cfc81f9109850fa9043a19783cbbde68a5", 39 | "revisionTime": "2018-02-23T01:09:33Z" 40 | }, 41 | { 42 | "checksumSHA1": "woO1qIxIeQ1bcbPSiMfAKk3r4xg=", 43 | "path": "github.com/gin-gonic/gin/json", 44 | "revision": "5d3f30cfc81f9109850fa9043a19783cbbde68a5", 45 | "revisionTime": "2018-02-23T01:09:33Z" 46 | }, 47 | { 48 | "checksumSHA1": "O+2Q4Uc9Vidvc/ctORliL315ZVI=", 49 | "path": "github.com/gin-gonic/gin/render", 50 | "revision": "5d3f30cfc81f9109850fa9043a19783cbbde68a5", 51 | "revisionTime": "2018-02-23T01:09:33Z" 52 | }, 53 | { 54 | "checksumSHA1": "Pyou8mceOASSFxc7GeXZuVdSMi0=", 55 | "path": "github.com/golang/protobuf/proto", 56 | "revision": "b4deda0973fb4c70b50d226b1af49f3da59f5265", 57 | "revisionTime": "2018-04-30T18:52:41Z" 58 | }, 59 | { 60 | "checksumSHA1": "w5RcOnfv5YDr3j2bd1YydkPiZx4=", 61 | "path": "github.com/mattn/go-isatty", 62 | "revision": "6ca4dbf54d38eea1a992b3c722a76a5d1c4cb25c", 63 | "revisionTime": "2017-11-07T05:05:31Z" 64 | }, 65 | { 66 | "checksumSHA1": "92dUWJPlL97eTKBNibbY8vPVxWY=", 67 | "path": "github.com/rs/cors", 68 | "revision": "12972ae4e7775151d0d53aaad277be85133ce445", 69 | "revisionTime": "2018-04-17T22:52:04Z" 70 | }, 71 | { 72 | "checksumSHA1": "S14onrwzwdOmTYTX+0w9hXwxPOw=", 73 | "path": "github.com/rs/cors/wrapper/gin", 74 | "revision": "12972ae4e7775151d0d53aaad277be85133ce445", 75 | "revisionTime": "2018-04-17T22:52:04Z" 76 | }, 77 | { 78 | "checksumSHA1": "eDQ6f1EsNf+frcRO/9XukSEchm8=", 79 | "path": "github.com/satori/go.uuid", 80 | "revision": "36e9d2ebbde5e3f13ab2e25625fd453271d6522e", 81 | "revisionTime": "2018-01-03T17:44:51Z" 82 | }, 83 | { 84 | "checksumSHA1": "WMhojMi3PWu3rn8vmXmUQuoOX/c=", 85 | "path": "github.com/ugorji/go/codec", 86 | "revision": "02537d3a3e32ef636a53519265d211bd208ca488", 87 | "revisionTime": "2018-03-07T15:23:41Z" 88 | }, 89 | { 90 | "checksumSHA1": "9/2qvuFPQxW7ZXzQSOYwqZb5cNg=", 91 | "path": "github.com/urfave/cli", 92 | "revision": "8e01ec4cd3e2d84ab2fe90d8210528ffbb06d8ff", 93 | "revisionTime": "2018-02-26T03:02:53Z" 94 | }, 95 | { 96 | "checksumSHA1": "wTEYbLPmy1zPu/SwZcQeltRBDZ4=", 97 | "path": "golang.org/x/sys/unix", 98 | "revision": "7dfd1290c7917b7ba22824b9d24954ab3002fe24", 99 | "revisionTime": "2018-05-09T21:44:12Z" 100 | }, 101 | { 102 | "checksumSHA1": "Mr4ur60bgQJnQFfJY0dGtwWwMPE=", 103 | "path": "golang.org/x/text/encoding", 104 | "revision": "8c34f848e18c4bd34d02db7f19a0ed1a0a8f5852", 105 | "revisionTime": "2018-03-01T21:24:51Z" 106 | }, 107 | { 108 | "checksumSHA1": "zeHyHebIZl1tGuwGllIhjfci+wI=", 109 | "path": "golang.org/x/text/encoding/internal", 110 | "revision": "8c34f848e18c4bd34d02db7f19a0ed1a0a8f5852", 111 | "revisionTime": "2018-03-01T21:24:51Z" 112 | }, 113 | { 114 | "checksumSHA1": "9cg4nSGfKTIWKb6bWV7U4lnuFKA=", 115 | "path": "golang.org/x/text/encoding/internal/identifier", 116 | "revision": "8c34f848e18c4bd34d02db7f19a0ed1a0a8f5852", 117 | "revisionTime": "2018-03-01T21:24:51Z" 118 | }, 119 | { 120 | "checksumSHA1": "G9LfJI9gySazd+MyyC6QbTHx4to=", 121 | "path": "golang.org/x/text/encoding/unicode", 122 | "revision": "8c34f848e18c4bd34d02db7f19a0ed1a0a8f5852", 123 | "revisionTime": "2018-03-01T21:24:51Z" 124 | }, 125 | { 126 | "checksumSHA1": "Qk7dljcrEK1BJkAEZguxAbG9dSo=", 127 | "path": "golang.org/x/text/internal/utf8internal", 128 | "revision": "8c34f848e18c4bd34d02db7f19a0ed1a0a8f5852", 129 | "revisionTime": "2018-03-01T21:24:51Z" 130 | }, 131 | { 132 | "checksumSHA1": "IV4MN7KGBSocu/5NR3le3sxup4Y=", 133 | "path": "golang.org/x/text/runes", 134 | "revision": "8c34f848e18c4bd34d02db7f19a0ed1a0a8f5852", 135 | "revisionTime": "2018-03-01T21:24:51Z" 136 | }, 137 | { 138 | "checksumSHA1": "ziMb9+ANGRJSSIuxYdRbA+cDRBQ=", 139 | "path": "golang.org/x/text/transform", 140 | "revision": "8c34f848e18c4bd34d02db7f19a0ed1a0a8f5852", 141 | "revisionTime": "2018-03-01T21:24:51Z" 142 | }, 143 | { 144 | "checksumSHA1": "P/k5ZGf0lEBgpKgkwy++F7K1PSg=", 145 | "path": "gopkg.in/go-playground/validator.v8", 146 | "revision": "5f1438d3fca68893a817e4a66806cea46a9e4ebf", 147 | "revisionTime": "2017-07-30T05:02:35Z" 148 | }, 149 | { 150 | "checksumSHA1": "Ww6wiSKdA8LwxoJz4DNUF+crH4g=", 151 | "path": "gopkg.in/yaml.v2", 152 | "revision": "" 153 | } 154 | ], 155 | "rootPath": "github.com/ctripcorp/nephele" 156 | } 157 | --------------------------------------------------------------------------------